Not a day goes by without someone bemoaning the evils of peer-to-peer networking, painting visions of a network apocalypse brought on by pimply-faced file stealers. And to make their case, naysayers typically present some hard-to-argue-with stats. This week, however, we came across a set of numbers that show more traditional video sources (streaming and flash video, for example) are now an increasing component of bandwidth on consumer-focused broadband networks.
As part of the research I’m doing for another piece, I had a long conversation with Danny McPherson, CTO of Arbor Networks, which makes all sorts of network-management and traffic-shaping tools. Arbor is used by dozens of ISPs around the planet and, as a result, McPherson is privy to details about traffic flows and usage patterns across many broadband networks.
McPherson shared with me some interesting stats and facts about broadband usage and peer-to-peer networking usage patterns. Given that Arbor makes a living selling its technology and products to carriers, it is prudent to maintain a degree of skepticism about the numbers. That said, they are nevertheless interesting enough to share.
On fixed and mobile broadband networks where consumer services are provided (i.e., NOT interprovider or typical dedicated Internet access for commercial enterprises):
- 10 percent of subscribers consume 80 percent of bandwidth.
- 0.5 percent of subscribers consume about 40 percent of total bandwidth
- 80 percent of subscribers use less than 10 percent of bandwidth
This supports the arguments made by some of the larger ISPs, including Comcast. In a recent interview, Comcast Cable CTO Tony Werner told me his company would try and deal with the tiny number of subscribers who use most of the bandwidth by slowing down their connections during peak times. (Personally, I find that to be a distasteful solution, and I believe that folks should learn from newer ISPs like Free.fr and better architect their networks so they can provide more bandwidth for all — without imposing any penalties.)
The P2P stats are the ones that came as a complete surprise. Like you, I have read many reports that suggest P2P applications account for the majority of the traffic on high-speed networks. But McPherson’s data suggests otherwise:
- 20 percent of traffic is P2P applications
- During peak-load times, 70 percent of subscribers use http while 20 percent are using P2P
- Http still makes up the majority of the total traffic, of which 45 percent is traditional web content that includes text and images. Streaming video and audio content from services like YouTube accounts for nearly 50 percent of the http traffic. It shouldn’t come as a surprise to anyone — streaming TV shows from Hulu and videos from YouTube have been on a major upswing, as noted by our colleagues over on NewTeeVee.
So, what do you make of these numbers?
Don’t the usage habits, if anything, show that punishing heavy users isn’t needed? After all, the vast majority of customers (in this case, 80 percent) wouldn’t even know if their connections were being hit by the teenager next door with a BitTorrent client up 24/7. And that remaining 10 percent (those between the very heavy users and the very light) still isn’t likely to notice.
It’s been my view that carriers continue to threaten caps, throttling, and other measures simply because they want to delay necessary upgrades as long as possible and soak in extra profits. At a certain point, Internet use will be heavy enough that Comcast and its kind won’t have P2P as a justification for slowing down subscribers’ connections.
P2P is the tool that enables certain behaviours. But what the heck are these super sharers doing? Downloading every film and album ever recorded? And then watching and listening? (And it is downloading as the upload capacity is limited.) Does anyone know a superfilesharer? What’s he like (and it is a he)? Figure out what they are doing with those files and you’ll figure out how to change behaviours.
Well, many P2P clients actually tunnel their traffic through HTTP. So, some percentage of that HTTP traffic is actually P2P traffic.
YouTube and other streamers have only been around for a few years and already they are accounting for 50% of the traffic? Wow…video is sapping Internet capacity. And we’re looking at HD video?
@ Prashanth
Arbor data suggests that ~12% of this is P2P protocols over tcp/80 and 8% if pure P2P. Hope that clarifies the issue you raised.
I believe the numbers.
Telling the carriers to provide more bandwidth doesn’t solve the problem.
The nature of P2P is the software increases speed to fill whatever capacity is available.
@ Don Jones
Don’t overlook the power of caching when talking Video. Everyone pounding the table about the video boom misses the impact a 40G cache in every DSLAM, CMTS headend, and OLT would have. It isn’t tough to implement, which means once it is a problem it will be implemented.
Wow… I am using twitterspeak. Om- you on twitter? You find it useful? It seems like too much noise to me…
Pareto rules again. 80/20
Focus on the 20% to solve your problems.
So the answer is….video not P2P! Short HULU.
This is very different to other research in this area and it’d be very interesting to know the samples here – even if they can’t name the ISPs, let us know where they are, how many subscribers, what traffic management techniques they use, etc.
The German firm iPoque measured P2P traffic at between 50% and 80% about six months ago; other surveys put p2p apps also somewhere in that range, and much greater at night. Is Arbor measuring encrypted p2p traffic or just readily identifiable protocols? How are they identifying that traffic? On port numbers or by packet inspection?
There are c.15-20m people using p2p apps at any one time; suggesting that they are only taking up 20% of bandwidth is v.surprising. It also sounds counter-intuitive to his other stats. On the one hand, he says that “10 percent of subscribers consume 80 percent of bandwidth.” But on the other, that only 20% of b/w is p2p. How is that 10% using all that bandwidth is not with at least a great deal of p2p? A YouTube video doesn’t saturate a broadband connection the way a bittorrent download easily can.
I find this surprising too. P2P users consume 100x more traffic, so I have trouble imagining that the streaming video demographic is 100x+ bigger than the P2P demographic, double-counting overlaps.
I have no doubt Comcast is throttling high bandwidth consumers. I was one. Note was. Our old ISP did not have an issue with the fact that I work from home ( constant VPN) and have 6 people with computers that eat bandwidth. We paid for 12 Meg down. We do not P2P here, except for very occaisional use of Joost. ( once a month??) We do not even use skype, we use sightspeed instead. That was not supposed to change when comcast took over, but it did. So I left Comcast for AT&T. It was an interesting story that got me calls from Comcast executive vice presidents. see: http://roguepuppet.blogspot.com/2008/04/comcast-gets-marching-orders.html and http://roguepuppet.blogspot.com/2008/04/comcast-take-notes.html for more history.
I agree with previous posts, sites like hulu are going to create a new paradigm.
Not all p2p fills whatever bandwidth is available.
Anonymous p2p depends on friends proxying media files for other friends and so only uses as much upload bandwidth as is available.
Such a system is http//www.Dargens.com an adult video friend to friend anonymous file sharing site.
By its nature anonymous p2p is self limiting with bandwidth usage dependant on what other friends allow.
Maybe ISPs should provide such systems as an alternative to traditional p2p to more control the amount of bandwith usage.
Excellent post, maybe it will put some sense in the erring ISPs.
“0.5 percent of subscribers consume about 40 percent of total bandwidth” – this is interesting!
Let’s say there are total of 200 subscribers each with 1Mbps internet connection. Now, we can say 0.5% i.e.
1 subscriber uses 40% of the ISP’s bandwidth. This means ISP’s bandwidth (to internet) is 2.5 Mbps per 200 customers each promised 1Mbps! Is this ratio an industry standard?
@Drew:
“Superfilesharers” I know don’t rely on their home connections (unless they had like 50mbit/s+ upload from home and even then…), and instead have colocated/dedicated hosting. I can think of many people I know who have multiple 100mbit/1gbit servers with softlayer etc. who blow through dozens of terabytes a month.
I suppose you could call me a “superfilesharer”? A few terabytes a month, mostly sharing and not downloading (bittorrent, ftp). My home connection is not even a tenth of my server’s, so I just grab stuff from my server and then pull from server to home. And I’m most definitely not male. I might note that if I had to try to do the same thing I do now from my home connection instead of a high speed server, I’d be up for shooting myself in the head because that’s just not possible thanks to the awful speeds. I can hardly share legitimate work (code, screenshots, things) at a reasonable speed via aim/msn/irc with my pathetic 1mbit/s upload.
Anyway, realize that any high quality hd rip is huge. I’m looking at a shoot ’em up bluray rip that is almost 50gb in size..that’s one movie, probably with commentary. Not many with average connections in the US can download that much quickly. Even with FiOS it can be a pain in the ass.
I can state emphatically and categorically that Arbor networks isn’t the provider of network management tools to many of the larger (or even smaller) ISPs (including Comcast) and that the numbers presented are incorrect and flawed.
Just one of the many incorrect assertions is the assumption that http = what folks think – web traffic. Note that some P2P traffic masquerades as http traffic. Not to mention some other applications.
P2P may only account for 20% of all traffic at peak but ONLY in a well managed network. If left alone, it is significantly more than 20% – particularly in the upstream.
I know this for a fact. I review the reports regularly…
Throwing more bandwidth at the p2p problem doesn’t fix it – it only makes it worse as it just eats up any bandwidth you can throw at it. And the very nature of the typical type of user that uses p2p (not all – but many) is that they are the type that refuse to pay for what they use (both in network usage and in content ownership/licensing rights etc.).
So should the 90% of users have to pay more for what the 10% do when the 10% won’t and you assert that only rate shaping the hogs isn’t the right thing to do?
I see lots of mud being slung by very naive folks in this net neutrality debate but no real financially sound solutions being offered by those slinging the mud and that includes here in this blog so far.
p2p great for sharing anything and i love it 🙂
Could it be the surge in investment $ flowing to CDNs is justified???
So much for the perceived glut of fiber.
meh…1% of my ISP’s network consumes 86% of the bandwidth…that number isn’t bad really. That’s why pay per usage and incremental revenue was invented.
Aren’t we comparing two different things – bandwidth and traffic? Most of the reports on P2P suggest that P2P consumes a great portion of the ‘bandwidth’. Traffic wise – P2P is a low proportion – doesn’t surprise me.
???
Got quite a thread started here
http://isp-lists.isp-planet.com/isp-bandwidth/0804/index.html
when I wrote
Om Malik supports throttling, gets flamed
and quoted some of what you’d said.
It’s a _very long_ thread.
One question the ISPs would like answered: why do the oil companies get a free pass on fixing prices?
I think it’s impossible to do justice on coverage of the topic in this short form. Many of the points made here are correct. Such as “what percentage” at “what time of day” or “upstream vs. downstream.” Like most statistics they can be manipulated to send a message that one wants to be sent. One thing is certain, no matter how you spin the numbers P2P applications are not the issue. Unattended applications are the issue. Basically the networks are desired for people to use them, not for machines to automatically do lots of stuff. When that starts happening, a single computer can be capable of creating more traffic than many many subscribers. Most of the Acceptable Use Policies (AUP) prohibit server functions on broadband residential. Many (but not all) of the P2P apps, mimmick that behavior acting as servers. And again, no matter how you spin the stats, it is true that some people abuse that shared resource and it DOES affect others. People do notice and do not like it.
Although I do not run P2P apps 24×7, I ocassionally use Pando to share large projects. I personally opted to pay my ISP $245/month (WiMax) for a business product instead of the residential $59 product. I therefore have the right to and expect to get my full bandwidth (2.5Mbps) 24×7.
That’s what people should do if they are running a business from home. It’s hardly a significant business expense when even a cell phone is at least $100/month.
-Victor
I watched the show on tivo and tracked down the story online to comment. If broadband company’s are going to advertise their service as “UNLIMITED”, just as dial up subscribers used to with the number of minutes you could be connected, then I think it should be unlimited in the sense of the word. if you don’t want certain types of traffic and usage, be upfront about it, and I will move to a different provider. ( and lets start upping the availability of more providers than just 2 to increase some competition here.) I think its just more whining from the cable provider because if everyone just used 10% of the available bandwidth then conversely there would be 90% left to be doled out to new subscribers by the Comcast gods .. whatever! and 10 plus 80 = 90 .. 10% headroom .. and did they do over long period studys to see if different users where “sucking” bandwith and then backing down?
There’s one line in this that bugs me a little: “Http still makes up the majority of the total traffic….” OK, so does HTTP represent a 51% majority according to MacPherson? Or 65%? Or 74%. If P2P is 20% and let’s assume everything else (e.g., non-http video, Voip,etc.) is 10 %, then could HTTP be 70%? I’d be grateful for any clarification.
Thanks, Om.
Bt considers 100 gig a month to be the max for an unlimited account
…
100 gig a month works out _ 1 month = 30.4368499 days = 3.2854911177913979856371404584809 gigs/pd (unlimated LMAO)
A 6 meg connection will do that in less than a hour if used well …..
For the past two months my broadband usage according to my provider has shot up by nearly 3 times. I am not even sure if the billing is right ‘cos I am have anything significantly different. In fact after the first shocker of a bill I actually cut down on all my you tube watching. But, again this month I got a bill for a similar amount. Most of the time I spend on twitter. So, my question – does twitter consume a lot of bandwidth?
Wildblue.com is bad about throttling down speeds so that one cannot even connect to the internet for 30 days! It’s their “Fair Use Policy.”
Unfortunately this is an issue that all satellite internet service providers have to deal with, because there is a limited amount of bandwidth that comes through the satellite. Satellite internet is not meant to go up against service like fiber optics of cable internet, but is meant to provide people that only have access to dial up with an option to get high speed broadband access. I actually have an entire blog about this at mybluedish.com/blog and would love for you to check it out.