Seven years ago, when traveling to Italy, I experienced the vagaries of data and its weird, unimaginative influence on our lives. Since then, the absurdity of what data-driven intelligence throws at us on a daily basis has increased exponentially. I wrote about it in an essay, 40 kilometers. It was part of a series of essays I wrote about data, its implications, and the emergence of limited-intelligence algorithms. If you are interested, here are some links to those articles in my archives.

Somehow that article, 40 kilometers, from seven years, ended up in the email inbox of my good friend Steve Crandall, who wrote a wonderful email reply in response. I thought it would be worth sharing and asked for his permission. Here it is:

The ‘data-driven world that we find all around us has little to do with science where data is highly contextualized and serendipity is welcomed and even hunted.  I think the notion of art is will be, or at least should be, important.

Operating as a simple person I like to make a distinction between awe and wonder. Both have multiple definitions, so I use my own.  Awe is a feeling of overwhelming majesty or even fear that seems to be beyond what we can understand or control. Wonder is a deep feeling of curiosity that leads to questions that can be addressed.  It’s scale may be big or small, but it can be consuming at any scale.  

Wonder is what I’m after and some of the paths have been decades long.  As a student in Pasadena I’d go on a long bike ride down to one of the beaches with the cycling club once or twice a month.  Being wasted from the ride and contemplating a more strenuous return I’d get lost watching gulls or the waves and surf.  I’d wonder about waves and that led me down a few paths.  The path I was taking wouldn’t naturally bump into fluid dynamics, but I started learning about the Navier-Stokes equation .. core in the study of fluid dynamics.  There were people to talk to and papers to read. The equations look simple, but are usually too difficult to solve analytically or exactly numerically in most real-world cases.  You learn tricks and the importance of the Reynolds Number as a guide for cheating.  I started to understand why the waves were doing what they did, but that led to other questions including the gulls.   

A few decades later I did some work on the flight of sports balls – particularly volleyballs as they’re one of the most interesting cases and that led to a friendship with Sarah Pavan and talks so far from my world that new sets of questions and thoughts sparkled into being.  Those waves were a long-term serendipity gateway and there have been dozens more.  I don’t know if a computer can help me in the wonder and initial serendipity part, but computer mediated communication, and synchronous is often the best kind, has certainly been an amplifier. So much of it is finding and bringing other wondering minds to the dance.

Steve’s right — what we called data-driven intelligence is not really intelligence. Instead, it is a somewhat simplistic rendering of the conclusions from the data. It lacks the ever-changing context and serendipity — something I experienced on that long drive to Siena.

July 7, 2021, San Francisco

Read article on 40 Kilometers

Big data intentionally creates a concentration of data and has a corrupting influence. It really concentrates the power in the hands of whoever holds that data — governments, companies. The PC revolution of the late 1970s and 1980s and the later early Internet (of the 1990s) seemed to hold so much promise and empowered the individual. Now with big data there is a shift of power in the other direction as it concentrates power in fewer hands.

Phil Zimmermann, creator of PGP

Why Hadoop’s time has come and gone

My former colleague, Derrick Harris, wrote more about Hadoop than any other reporter, and today, he wrote a wonderful obituary of Hadoop. He goes step by step, through what has made Hadoop quite irrelevant.

Hadoop’s path to ubiquity intersected a host of other technology shifts that as a whole would prove to be more impactful in the long run, in part by peeling off the most valuable promises of big data and making them more consumable…..The story of Hadoop can help us understand why the world of data looks how it does today. It also should be a valuable lesson for anybody trying to make sense of the next big thing in enterprise IT, and the next one after that…….Hadoop opened people’s eyes to what was possible with big data, but it’s also a reminder that no single technology is going to remake the world of enterprise IT— at least not anymore.

Way back in 2008, when writing GigaOm, I hosted a meet-up for Hadoop enthusiasts, inspired by what I had seen Google and Yahoo do with data. The age of large data sets was ascendant, and it made perfect sense for a distributed file-system called Hadoop to emerge. However, the world changed too fast. The emergence of smartphones, the growing dominance of cloud and other meta-changes in the technology landscape radically redefined what data could do. And just as quickly as it became hot, Hadoop found itself on the outs, with its logo, The Elephant, ironically telling the story of its lumbering legacy.

Hadoop’s rise and irrelevance is the new reality of our cloud-centric world. It is also a reminder that new technology emerges and spread at lightning-fast speed, only to become irrelevant or eventually be subsumed by larger players. What is hot today, is a stock market detritus a decade later. In many ways, Cloudera is a perfect proxy for Hadoop, and what has gone wrong with the Hadoop ecosystem.

Meanwhile, Wall Street shark Carl Icahn is hot and heavy for Cloudera. The company, which recently got rid of its chief executive, and lost one of its co-founders, is now a Wall Street piñata. The stock which in March 2017 was around $22 a share hit an all-time low of approximately $5 a share, though it has since rebounded to about $7 a share. I am all eyes and ears to see how Icahn will make the elephant dance.

Read article on Derrick Harris

We are all trapped in the “Feed”

Every afternoon, during lunch, I open up YouTube, and I find myself marveling at the sheer dumbness of its recommendations. Despite having all this viewing data of mine, world’s second most popular search engine is dumb as a brick. It shows me propaganda channels from two ends of the political spectrum. It surfaces some inane celebrity videos. It dredges up the worst material for me — considering I usually like watch science videos, long conversations and interviews, and photography-focused educational videos. Continue reading “We are all trapped in the “Feed””