Is AI a snake that eats itself?

Whenever you read a piece about OpenAI, ChatGPT, or ChatBots, the common refrain is that it is a massive manifestation of the concept of garbage-in-garbage-out. “One of the most important traits that a system can have is that one can reason about that system in a consistent and predictable way,” writes veteran software and tech writer Anil Dash, and adds, “a system should be predictable enough in its operation that we can then rely on it when building other systems upon it.”

In his essay, Today’s AI is unreasonable. Anil articulates the systematic risks posed by generative AI. “We have billions of dollars being invested into technologies where it is impossible to make falsifiable assertions.” Anil writes. “A system that you cannot debug through a logical, socratic process is a vulnerability.”

Paul Kedrosky highlights one such risk in his latest newsletter.

In the most recent JPM survey of large investors, more than half of them use (Machine Learning) algorithms, and the single largest source of content remains journalism. At the same time, the survey says that most investors believe that a large fraction of future journalism will be written by AI. If stories are written by AIs, and then ingested by pattern- and signal-seeking AIs, we should expect even more volatility in future given that more algorithms will be responding to the same signals. This will be risk-increasing and alpha-decreasing.

Paul Kedroksy

Anil and Paul’s comments made me wonder if AI is like that snake that eats itself. What do you think?

June 21, 2023. San Francisco