29 thoughts on “Ev Williams: The Challenges of a Web of Infinite Info”

  1. Though it really scares me to hear talk from people in our industry about there being “too much info on the internet” I do look forward to seeing algorithmic solutions to our need for recommendation, discovery, etc. However, if “social is the OS” I sure hope the OS doesn’t end up looking closed and shiny like iOS or Facebook at the expense of those of us who would like to push the envelope and discover new ways to process all these information streams and social connections for ourselves. New Twitter’s new pagination makes it much harder to scrape for research, analysis and journalism, for example, and the company’s policy for some time to block iFrames has inhibited some interesting things that small startups have tried to do. Twitter is an incredible collection of information, but every time it closes a little more for the convenience of Justin Bieber fans, it loses some value for the future of humanity. Good luck and thanks for all the Tweets.

    1. I think if “social is the OS” then I think the interface is really an API…but overall I of course love this conversation and direction since it’s very much what we are trying to tackle with http://knowabout.it (and yes we also will be offering up an API as well as a few other interfaces) =D

  2. Twitter is a fantastic place to discover new people, important news and get updates from all your connections. Ev and his colleagues were pioneers in terms of mastering reverse chronology at Blogger and now at Twitter, and the adoption has been fantastic. It is equally important, as he mentions, to find personal relevance in the streams. That’s what my6sense (where I am VP of Marketing) is trying to do with Twitter and all your other streams. We agree with Ev in finding the best stuff per individual.

    We were first to rank Twitter by relevance at the end of 2009, and more is coming. Just at your Twitter stream to the my6sense app on Android or iOS, and see what we mean. The challenge may be infinite info, but the challenge is also getting the right info at the right time for the right person.

    1. Louis

      Thanks for that summary and yes, you, I and Ev are on the same page. I think the challenge is to find individual signal in such a huge amount of data and then make it relevant to one. I am glad you guys are pursuing this, and will be keeping an eye on mysixsense!

  3. For what it’s worth, location based focus would not work for me. Twitter is valuable for me because it is global, it allows me to discover information beyond the limits of my own limited ability to search and be exposed to it. It is organic, which is why despite the torrent of information it seems second nature for me to use. It still needs to be up to the user to filter his/her own experience, otherwise the entire experience is limited. Algorythms, curation, the same principles that apply to successful media and info sites apply to Twitter. My two cents.

    1. Leandro,

      I think you are agreeing with Ev here. he is of the same opinion as well and during our chat we discussed that — twitter is many things to many people and the company has a challenge of finding the right relevance and use case that is a starting point for one and all.

      Not sure if that makes sense to you. If you want, I can dig through my notes.

      1. I’ve been following updates pretty closely, I think that the iPad app for example is a great direction, and the enhanced threading via conversations and RTs on the .com is also helpful. But, like I said previously, it’s the organic chaos mixed with my own curated follow list that keeps me coming back. The main problem I see at this juncture is the dilution of information via the follow. On one hand, building your own list brings pertinent info in, but unless a user is judicious with whom to follow the stream loses focus. Lists seem like the most obvious solution here, but at the same time one would want them to be present, but not dominating over the overall stream, which is what allows for the discovery factor. Longwinded I realize, it’s a complex problem and I think that inevitably new things must be tried out and modified if necessary. On location, geo info filtration beyond voluntary hash tag use may be needed, IMO. Thanks.

  4. Ev seems to be throwing out a paradox. On the one hand available information is scaling to approach infinity. On the other hand economies of scale naturally drive distributed systems towards consolidation. Implicitly, I believe he is also saying that consolidated platforms cannot scale infinitely, though distributed systems can. At least until the ensuing wave of economic consolidation occurs.
    The resolution of this paradox should be a new distributed system, that will do for information what the web did for documents. Like my6sense its principal use case will be personalised filtering, but its algos must be simple and open. This system should start out de-centralised and remain so as TimBL’s web actually has done. Though again consolidation would likely occur over time. The system should present a new UI metaphor that is native to this brave new world, that has such infinite information in’t.

    1. Henchan,

      Good points. I would love for you to elaborate on the UI metaphor. it seems you have some thoughts here and I for one, would love to know more.

      Also, can you elaborate on “This system should start out de-centralised and remain so as TimBL’s web actually has done.”

      1. Yes, I’ve designed the system that I describe above. It comprises just one data type and three algos. In private, I’d be very pleased to describe it in detail. It’s an open spec which is already available on the web. Few know about it as yet. I am currently prototyping a web app based on this system design. Every tag in the system’s data format has a distance (in units of Shannon entropy) from every other tag. UI takes the form of a ‘scope’ resembling a circular radar screen. My app lets one zoom in on data more related to the scope’s central tag or zoom out to max entropy and see all tags in the database.

        The only one of Ev’s points that I disagreed with was that the original distributed model of the web is dead. I can see why he might say so from his perspective, but I doubt Tim Berners Lee would agree. www remains a distributed application.

  5. If I read about thinking machines, I don’t know if I should laugh or cry.
    They don’t understand points, data and information and how they correlate and want to build something thinking.
    Thinking is about creation, which requires abstraction first, if your machine can not build it’s own abstractions, which is out of reach of “normal” algorithms, it will not think!
    But one does not need a thinking machine to organize points. But what he is talking about is shared context anyway, which is just a intermediate to personal context. If they don’t get that good luck.
    That’s the problem we are facing, a lot of blah without understanding what is what and how it’s created and correlates. Then there is the small problem that our programs are the equivalent of global variables in spaghetti code when used with context.

    1. Ronald,

      I think Ev is agreeing with you on the need for balance between human thinking and machine actions. If you mean otherwise, please elaborate, and I will try and answer your questions.

      1. So what is information? If we get infinite of it (which is not possible), we should at least know what it is. Or how do data and points relate to it, is it all interchangeable like the SV crowd likes to do?

        What’s the most overlooked problem to integrate human machine interaction for the background (model) of a system? Is it good enough to slap a new UI (algorithms) onto old models?

        All should not only apply to tweets, but also to playlists for example.

  6. @Ev – create a delicious-like ontology building tool for tweets. that way, users can not just “star” their tweets, but store them in folders. old tweets that have wisdom or helpfulness value or link into stories worth remembering should be treated as data objects with more persistent value than tweets get today. keep it up, though, twitter is rockin as far as #tinyvox is concerned ! 🙂

    1. I’m pretty sure this is the idea/motivation behind Twitter Annotations (which they announced at Chirp this past year, but as far as I know haven’t actually rolled out yet)…but yes, I agree with you and it’s one of the reasons I was so excited about annotations when they were first announced.

    2. Totally agree Srini… that would be awesome. But how about finding a way to integrate with delicious rather than create another tool? I love delicious and worry that it’s on the decline now… which is such a shame!

  7. Twitter latest round assures eventual failure and only sorry it will further embarrass one of the valley’s great VC firms rather than the Russians who are almost exclusively backing the rest of the ridiculousness.

  8. Om, This is a seminal interview in my books, for the future of information consumption at least.

    When Ev said “I think we need to design (our products) for a world of infinite information.”, do you think he meant “we” as Twitter Co., or “we” as “all of the rest of us”?

    There’s a big difference depending on who is the “we”. If it’s Twitter Co., they would be sending another mixed message to its ecosystem of developers.

    Who will be solving that connumdrum of optimal consumption of infinite information? Will it be Twitter or other apps that use Twitter’s and other social triggers data? This calls for another more elaborate post I think.

  9. The notion of infinite information is related to the advent of the real-time web, the move from web-pages as the destination to real-time data feeds and to the breakdown of the current generation of information filters that Clay Shirky talked about last year.

    The information filters in use today were invented over a decade ago for a growing web with static data. Organizations are continuing to use the old filters to solve the new real-time web problem and are coming a cropper. This is not surprising. Ev Williams is essentially looking for a new information filter to solve the personalization problem in the face of a real-time data firehose. Twitter recognize that if this problem isn’t fixed soon then people will move on as it becomes difficult to find things of interest. The problem is not restricted to Twitter but applies to all companies in the coming age of the real-time web with billions of mobile internet devices and trillions of pieces of data.

    This is the problem that we are addressing. Without a new tool you only have conventional and well travelled paths to address the problem.

  10. This issue isn’t the infinite information. It’s the availability and accessibility of that information and the limited ability to curate this information. We’re already struggling with the issues after only a decade. The bigger issue to me is that a lot of really valuable information is being stored in huge silos. Twitter isn’t really ‘indexed’ for global / open search. Neither is FB. But a lot of that information is going to be important. Twitter is at least open to enabling the firehose, and I think FB realizes the value / power of hoarding this information.

    So to me, the question remains in how ‘open’ the web remains going forward. I’ve no doubt that engineers will find ways to solve the Infinite Info issue, but I also think that humans will invariably end up being a critical part in curating information on the web. “Bloggers” were born out of a need for the masses on the web to help organize and find micro topics that broad media couldn’t cover. Twitter and FB have enabled highly micro-blogging to fill a same need. Twitter is more open, FB is semi-closed. But they are both great tools to unlock news and relevant information. I guess the question then becomes how relavant “tweets” are in a historical perspective. In any event, this is an interesting topic and very thought provoking… Thanks Om. Happy New Year.

  11. I wished Ev would come here and join this conversation. Om, can you help? There are some good rebuttals needed on this.

    Re: “It seems to me that almost all tools we rely on to manage information weren’t designed for a world of infinite info.” What tools is Ev referring to? No need to be subtle here. Need specifics. (is he implying Google?)

    “Twitter itself isn’t designed for this world of infinite information.” Is Ev implying that Twitter can solve this?

    In my opinion, the issue is not infinite information necessarily- it’s infinite attention. Let’s say it in layman terms: There’s lots of information, and we have little time to consume it. That’s it. That issue has been around for a while.

    Twitter (and social media) does a partial job at surfacing what’s popular and “sometimes” relevant, but it’s not a complete panacea to the issue, unless every single piece of content ever generated goes through the social media washing cycle, but then again, we’d be relying solely on the wisdom of the crowd to tell us what’s important to us. Well, that’s a sure way for mediocrity. We’d all be reading the same thing (which we are, seemingly in tech), so where is your differentiation, your edge?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.