Nothing is more frustrating to me than YouTube, which decides my front page based on my likes. It seems I can’t have multiple interests — variables — and thus, I must watch certain kinds of videos. In its infinite wisdom, Twitter believes that only the people whose content I like or share are the ones whose content I want to consume. And don’t get me started on online dating services — they could learn a thing or two from Sima Taparia

And that is because the post-social world of today is starting to coalesce around variables that are less humanistic and more biased towards corporate goals. “We live in a world that demands categorization,” I recently read in a newsletter, Tiny Revolutions. “We have to do some self-definition so the world knows what to do with us, and so that we can bond with others who share our interests, values, and concerns.”

While the writer, Sara Campbell, might have been talking about an individual’s desire not to be categorized, her words accurately describe our post-social society’s reality, dilemma, and futility in a handful of lines. 

Categorization is part of the human condition. Our brain uses categories to help us make sense of a lot of facts we experience. It is how we learn. As humans, we need categories to contextualize our world, and that includes each other. What is more important is the intent behind the categories. 

Categories, as such, have bias by intent. The bias allows us to ignore variables we don’t want to deal with and place boundaries around a category. It’s important because by ignoring them, we have to use fewer cognitive resources. The bias itself is not good or bad. It is the intent that leads you in different directions. That intent determines what variables we focus on and the ones we ‘choose’ to ignore. 

And a lot of that intent is determined by the human condition. For example, if you have grown up in a more traditional society, the category that defines you is your lineage for most of your life. The “intent” of that categorization is to find your place in the social hierarchy. Lineage isn’t a primary variable for Americans, but college and money are. That is why in more modern societies, such as America, the college you attend defines your place in society and the workplace.

Ever wondered why most conversations start with a question: what do you do? That question is not only reflective of our fading art of conversation, and it also is a way for us to define the variables and get a quick context on the person. By doing so, we quickly decide to assign a value-metric to the person who is the recipient of our attention. 

At best, in the pre-Internet world, categorization would rear its head in a social context, often giving us cues on how to engage with someone. An attractive single woman gets a different kind of attention from another woman versus a single man. Given the nature of modern consumerist society, it wasn’t a surprise that the emergence of databases allowed marketers to categorize us into “buckets” of those who may or may not buy some products. After all, the early usage of computers had been catalyzed by the demands from governmental agencies and corporations that wanted to use data to create categories. 

However, in our post-social society, these categories have become even more granular and metastasized. Just take Facebook as an example. School, location, gender, relationships, and many more variables have started to create a profile of us that can be bundled no different than the dastardly collateralized debt obligation (CDO.)

And it isn’t just Facebook that is alone in using so many variables. From online dating services to online marketing to banking, most of them feel both antiseptic and plastic. These data variables are what make up an algorithm whose sole job is categorization. At present, the algorithms are relatively simplistic. They lack the rationality and nuance that comes from social science.

The bigger question is, what if all these data variables picked by companies for their own needs don’t define you or your interests. I suspect all of us be trapped in a data prison — forced to live lives that an invisible black box algorithm will decide what is good for us.  


August 24, 2021, San Francisco

Seven years ago, when traveling to Italy, I experienced the vagaries of data and its weird, unimaginative influence on our lives. Since then, the absurdity of what data-driven intelligence throws at us on a daily basis has increased exponentially. I wrote about it in an essay, 40 kilometers. It was part of a series of essays I wrote about data, its implications, and the emergence of limited-intelligence algorithms. If you are interested, here are some links to those articles in my archives.

Somehow that article, 40 kilometers, from seven years, ended up in the email inbox of my good friend Steve Crandall, who wrote a wonderful email reply in response. I thought it would be worth sharing and asked for his permission. Here it is:


The ‘data-driven world that we find all around us has little to do with science where data is highly contextualized and serendipity is welcomed and even hunted.  I think the notion of art is will be, or at least should be, important.

Operating as a simple person I like to make a distinction between awe and wonder. Both have multiple definitions, so I use my own.  Awe is a feeling of overwhelming majesty or even fear that seems to be beyond what we can understand or control. Wonder is a deep feeling of curiosity that leads to questions that can be addressed.  It’s scale may be big or small, but it can be consuming at any scale.  

Wonder is what I’m after and some of the paths have been decades long.  As a student in Pasadena I’d go on a long bike ride down to one of the beaches with the cycling club once or twice a month.  Being wasted from the ride and contemplating a more strenuous return I’d get lost watching gulls or the waves and surf.  I’d wonder about waves and that led me down a few paths.  The path I was taking wouldn’t naturally bump into fluid dynamics, but I started learning about the Navier-Stokes equation .. core in the study of fluid dynamics.  There were people to talk to and papers to read. The equations look simple, but are usually too difficult to solve analytically or exactly numerically in most real-world cases.  You learn tricks and the importance of the Reynolds Number as a guide for cheating.  I started to understand why the waves were doing what they did, but that led to other questions including the gulls.   

A few decades later I did some work on the flight of sports balls – particularly volleyballs as they’re one of the most interesting cases and that led to a friendship with Sarah Pavan and talks so far from my world that new sets of questions and thoughts sparkled into being.  Those waves were a long-term serendipity gateway and there have been dozens more.  I don’t know if a computer can help me in the wonder and initial serendipity part, but computer mediated communication, and synchronous is often the best kind, has certainly been an amplifier. So much of it is finding and bringing other wondering minds to the dance.


Steve’s right — what we called data-driven intelligence is not really intelligence. Instead, it is a somewhat simplistic rendering of the conclusions from the data. It lacks the ever-changing context and serendipity — something I experienced on that long drive to Siena.

July 7, 2021, San Francisco

Read article on Om.co: 40 Kilometers

Big data intentionally creates a concentration of data and has a corrupting influence. It really concentrates the power in the hands of whoever holds that data — governments, companies. The PC revolution of the late 1970s and 1980s and the later early Internet (of the 1990s) seemed to hold so much promise and empowered the individual. Now with big data there is a shift of power in the other direction as it concentrates power in fewer hands.

Phil Zimmermann, creator of PGP

Why Hadoop’s time has come and gone

My former colleague, Derrick Harris, wrote more about Hadoop than any other reporter, and today, he wrote a wonderful obituary of Hadoop. He goes step by step, through what has made Hadoop quite irrelevant.

Hadoop’s path to ubiquity intersected a host of other technology shifts that as a whole would prove to be more impactful in the long run, in part by peeling off the most valuable promises of big data and making them more consumable…..The story of Hadoop can help us understand why the world of data looks how it does today. It also should be a valuable lesson for anybody trying to make sense of the next big thing in enterprise IT, and the next one after that…….Hadoop opened people’s eyes to what was possible with big data, but it’s also a reminder that no single technology is going to remake the world of enterprise IT— at least not anymore.

Way back in 2008, when writing GigaOm, I hosted a meet-up for Hadoop enthusiasts, inspired by what I had seen Google and Yahoo do with data. The age of large data sets was ascendant, and it made perfect sense for a distributed file-system called Hadoop to emerge. However, the world changed too fast. The emergence of smartphones, the growing dominance of cloud and other meta-changes in the technology landscape radically redefined what data could do. And just as quickly as it became hot, Hadoop found itself on the outs, with its logo, The Elephant, ironically telling the story of its lumbering legacy.

Hadoop’s rise and irrelevance is the new reality of our cloud-centric world. It is also a reminder that new technology emerges and spread at lightning-fast speed, only to become irrelevant or eventually be subsumed by larger players. What is hot today, is a stock market detritus a decade later. In many ways, Cloudera is a perfect proxy for Hadoop, and what has gone wrong with the Hadoop ecosystem.

Meanwhile, Wall Street shark Carl Icahn is hot and heavy for Cloudera. The company, which recently got rid of its chief executive, and lost one of its co-founders, is now a Wall Street piñata. The stock which in March 2017 was around $22 a share hit an all-time low of approximately $5 a share, though it has since rebounded to about $7 a share. I am all eyes and ears to see how Icahn will make the elephant dance.

Read article on Derrick Harris