If you are trying to get Netflix (s NFLX) and use Verizon’s broadband, then there is a good chance that your video performance is less than optimal. Some Verizon customers might even go as far as calling it a crappy Netflix experience. The reason: a behind-the-scenes power play between Verizon (s VZ) and Cogent Communications (s CCOI), one of the largest bandwidth providers. The head-butting between these two companies is over an arcane concept known as peering.
Peering is essentially an arrangement between two bandwidth providers where they send and receive traffic from each other for free. The logic is that the data sent from one network to another is reciprocated. Verizon runs one of the largest last mile networks and owns the descendants of MCI. Cogent is one of the largest bandwidth providers, and its network is spread across the globe in hundreds of cities.
Cogent and Verizon peer to each other at about ten locations and they exchange traffic through several ports. These ports typically send and receive data at speeds of around 10 gigabit per second. When the ports start to fill up (usually at 50 percent of their capacity), the internet companies add more ports. In this case, through, Verizon is allowing the ports that connect to Cogent to get crammed. “They are allowing the peer connections to degrade,” said Dave Schaeffer, chief executive officer of Cogent said in an interview. “Today some of the ports are at 100 percent capacity.”
“Think of it as the on-ramp to the freeway being log-jammed,” Schaeffer said. And that means your Netflix content, especially content sent by Netflix’s content delivery network, slows down, and you get pixelated pictures and buffering.
While not naming Netflix directly, Verizon has indicated to Cogent that the reason behind its actions is that Cogent is moving traffic for a large video provider. Schaeffer confirmed the Netflix is one of their largest customers. “Over the past year Netflix has become a big partner for us. This is a business model problem, not an engineering problem,” Schaeffer said.
Our sources tell us that Netflix recently bought 2 Terabits of bandwidth capacity in part to get around such cramming that was happening in places where it sends traffic directly to certain internet service providers.
When we called Verizon about this story asking if Verizon was having a problem with Cogent over peering issues associated with Netflix, Verizon spokesman Bill Kula said he’d get back to us. A few minutes later he sent the following reply that didn’t answer our question:
Verizon operates one of America’s lowest-latency, highest capacity networks. The various classes of Internet speeds we offer are among the fastest in the nation. Time and again, customers rate us best in class in various reports and surveys. Our customers enjoy a consistently superior Internet experience because our networks can adapt and grow with their use.
Netflix has been growing like crazy and it now accounts for a whole lot of Internet traffic — almost one out of every 3 bits (32.3 percent) sent downstream to users in North America is Netflix traffic according to Sandvine, a company that makes traffic monitoring gear for ISPs. That’s a lot of congested ports.
Netflix’s growing popularity has made it a target of ISP (internet service providers) vitriol and anger, especially those who offer competitive services. Verizon, for instance owns 50 percent of Redbox, a video-over-the-Internet service that is competitive with Netflix. Time Warner Cable and Comcast are other large providers that has allowed degradation of the online video experience on its networks — after all the logic is that as people start to have a bad Netflix experience, they start to look for alternatives — perhaps the ISP’s own pay TV offering.
This isn’t the first application last mile network operators have tried to degrade — last year the wrath of the Baby Bells and cable companies fell on Megaupload, a file sharing company started by Kim Dotcom, Schaeffer said. That too was one of the big bandwidth-hungry services popular with the end customers of the ISPs — actual consumers.
This post was updated on Wednesday June 19th to correct the spelling of Dave Schaeffer’s name.
Just reinforces why Verizon is among the most hated companies in America.
Why is it that every single Internet peering dispute seems to involve Cogent?
Maybe because they are big and are offering their bandwidth at a really low price that doesn’t fit into the plans of the carriers and ISPs who want to keep an artificial premium? Or that would be guess at least.
The Wikipedia page for Peering ( https://en.wikipedia.org/wiki/Peering#Depeering ) lists these “notable” disputes:
BBN Planet vs Exodus Communications[6]
PSINet vs Cable & Wireless[7]
AOL Transit Data Network (ATDN) vs Cogent Communications[8]
Teleglobe vs Cogent Communications[citation needed]
France Telecom vs Cogent Communications[9]
France Telecom (Wanadoo) vs Proxad (Free)[10]
Level 3 Communications vs XO Communications[citation needed]
Level 3 Communications vs Cogent Communications[11]
Telecom/Telefónica/Impsat/Prima vs CABASE (Argentina) [12]
Cogent Communications vs TeliaSonera[13]
Sprint-Nextel vs Cogent Communications[14]
For one reason or another, Cogent is more likely than not to be involved in these things.
Corrected this for you:
Maybe because they are big and are offering their peers bandwidth at a really low price that doesn’t fit into the plans of the carriers and ISPs who want to carry it more than 1 router card? Or that would be guess at least.
it would be better Peerguy if you wrote out your explanation as a separate comment that we could learn from IMHO.
Cogent (and a few other low-ball providers) are famous for selling 1-hop transit.
This is when you go to a big content owner saying you can sell transit for well below anyone’s cost as long as that content provider connects to you in Site A on Router B and delivers to ONLY Customer 1, 2 and 3 which happen to be peering on Site A in Router B and many times on the same Nx10G card.
Multiply that by 10 cities and a similar configuration which never requires any real network from the 1-hop transit supplier, combine that with overselling and congesting the existing capacity and you have a peering dispute
Thanks you PeerGuy
No problem. Education is needed around this topic. Chris Dickens below also clarifies the situation a bit further around the economics of peering and how it changes when someone abuses the relationship then publicly demand entitlements.
Now that you know more, perhaps you can correct your title around who’s actually in control of this issue and more directly to “blame” for issues.
I am going to stick to my headline for now. 🙂
PS: What is your affiliation @peerguy?
@peerguy – controversy sells papers^Wpage views.
I’m no fan of either Verizon or Cogent, but Peerguy nails it. We run into this problem all the time. It’s called “peering”. It should be between peers. If one side it dumping massive amounts of traffic onto your network for free, there’s a problem there. Cogent’s customers are paying them for bandwidth, not excuses. If Cogent’s free peering and begging for entitlements is not working for them, they should man up and purchase bandwidth. I’m sure Verizon would be happy to provision a circuit to Cogent at market rates. Again, no fan of Verizon, but Verizon’s not the bad guy here and they’re not doing anything nefarious at least in this case.
If you want to report the truth, have Cogent tell you on those points where they’re saturated (I know for a fact Chicago is one of them) which direction are the ports saturated. It’s a pipe. As you report, 10gb in each direction. Is the traffic load even, meaning that Cogent traffic is taking 5+gb AND the Verizon traffic is 5+GB, or is it one way, e.g. Cogent traffic is 5+gb, but Verizon is well under 50% capacity. I know (at least in Chicago) it’s all Cogent dumping traffic on Verizon.
Not all of them, just a lot of them. Why? Well they obviously can’t keep their traffic ratios in line. They should work on getting more eyeball traffic.
Verizon, Time Warner Cable and Comcast are all monopolies behaving in anticonsumer anticompetive ways.
Not true – as VOIPCEO said, it’s usually an unbalanced traffic pattern. I know with TWC and XO, XO was dumping approx 200G of traffic on TWCs network, while only 30-40G of traffic was being pushed onto XO’s. The majority of the traffic was and is Netflix related….After additional contract negotiations with XO, things are working well again….The disparity of traffic loading and balancing has to be managed – it costs a shitton of money to support this traffic.
Verizon’s peering is hot to a number of other carriers, including NTT. They are also doing some pretty scenic routing to Comcast.
Why aren’t we looking towards Verizon, and not Cogent, as the problem here?
David
The problem is that Cogent doesn’t have a competitive product to Netflix like Verizon and TW Cable do and the fact that Verizon has been nudging them to shut down their client as the story reports.
That’s an accusation without factual basis – how about some actual journalism and find a decision memo where a VZ exec says to run this peer specifically hot? Everyone in the industry knows that VZ bought UU and then handed the policy keys over to the people who wish to maintain the legacy “Tier 1” structure. They run hot to those who don’t pay them. Occam’s razor says this is just another example. Maybe the hype around Netflix will make it the example that helps to shatter the behavior, but to pretend that it is more than just consistent behavior demands proof, not conjecture.
Bring it.
Would love to use Netflix streaming, but we have 3Mpbs DSL here, from Frontier. A struggle to watch anything, because of such slow speeds.
I wonder how much of the USA is in the same boat, where Netflix streaming is just not practical to use.
Illinoi
I think the 3 Mbps is enough to watch Netflix on your iPad screen but I agree, you rarely get 3 Mbps and the networks are completely degraded.
One of the challenges is that the revenue model for service providers is getting upside-down. The all-we-can-eat bandwidth at a particular speed enables applications like Netflix, but requires investment to support this. Netflix reaps the rewards on the back of service provider networks.
^ what you said….I wish more people understood this.
I agree that Netflix is getting a great deal, but its to the benefit of the ISP.
ISP’s charge inflated prices for modest speeds which we are willing because bandwidth hogs like Netflix are so popular. Without these companies there would be no demand for $70 internet plans.
If I subscribe to higher tier plan, but I can only utilize a small portion of it due to congestion, what is the incentive to stay at that higher priced plan? Are they not shooting themselves in the foot?
It’s not Verizon’s responsibility to provide Cogent with bandwidth. There’s nothing “arcane” about Internet peering. It’s quite simple. When the balance of traffic between Cogent and Verizon are similar then the links can be provided at minimal to no cost to each other. If there becomes a bandwidth deficit (lop-sided, think like trade deficits between countries) then the peer pushing traffic much harder is forced to pay a fair market value to increase the capacity of the connection in the direction their traffic is going and the agreement switches to a “transit” agreement.
So, there are only one applicable question that needs to be asked before pointing a finger in the direction of Net Neutrality:
Is Verizon charging an unfair amount or even denying Cogent a transit agreement? (Unfair as meaning dissimilar to what Verizon charges other transit customers — not the amount Cogent thinks they should be charged)
Just to clarify the relationship does NOT have to switch to a “transit” agreement, it switches from a “settlement-free peer” read as free, to a “settled peer”. The routes exchanged stay identical, but the network sending (pushing) more traffic would pay the network who is receiving (transporting) said traffic.
Transit would indicate that the paying party would have their routes advertised to the peers non-paid customers and that wouldn’t be in pushing parties best interest. The majority of “peering relationships”, between ultra-heavy content providers and ultra-heavy eyeball networks are based on a settlement basis, just not typically disclosed. Think of the Level(3) / Comcast dispute.
cgucker – Thanks for clarification. I was unaware of the terminology of “settlement-free” versus “settled peer”. My input here is actually a direct result of lots of research I did back when the Level3 Comcast spat occurred some time back. I ultimately sided that as long as Comcast is not unfairly charging for the bandwidth versus other big data customers then they weren’t in the wrong. Similar situation applies here. If they are able to settle the peering agreement without providing transit (no additional routes) then the agreement may possibly be done more inexpensively than a transit agreement.
The pickle is ensuring that Verizon doesn’t play favorites and charging reduced rates to other settled peers where the incoming traffic doesn’t compete with their own products. I’m not a proponent of government oversight, but some very limited eye-balling can go a long way to keeping everyone honest and fair.
Verizon only has eyeballs. all content comes from Cogent, level 3, etc. if Verizon cared about their customers experience using the internet they would upgrade their capacity. that is how the internet was designed, to exchange information from one peer to the next. it is obvious that Verizon has an agenda.
“Verizon only has eyeballs” is false. Loads of enterprise, direct on-net hosting, and downstream datacenters/hosting providers are on their network because of the UUNet acquisition. All the corporate entities have agenda: maximize profit, minimize cost.
The rest, as with this supposed “reporting”, is conjecture. Pass the tinfoil…
Actually, by definition, it is arcane.
known or knowable only to the initiate : secret ; broadly : mysterious, obscure
If the peer pushing is the one that pays, then I want the ISP I own to peer with Verizon and have them pay me to take their traffic. After all, they’re the ones pushing it to me.
Mostly we have the alternative of dropping Verizon and such like companies and changing to a company that doesn’t throttle. However, who’s to know which alternative ISP doesn’t use Verizon? In some cases there is no alternative. This opens the question of whether to use Government lobby to legislate anti competition law. Will this interfere with free trade? A whole new bees nest.
Most of us have the option of changing from companies that do this. Unfortunately there are some that have no option other than to lobby government for legislative changes to stop this practice.
Is this legal if they have a competing service?
God forbid American companies should invest in America. T
But the tax breaks we give the rich are not invested here, so why should the big companies do better…
If Verizon really wanted to help their end customers in this situation, they’d install Netflix CDN servers on their network (for ‘free’). This would both reduce the peering traffic and provide better service to their customers. The only reason I see them not doing this is because of their own content services.
Why should Verizon allow Netflix free access to their network to deploy CDN caches? And what if Netflix is not popular in 2 years and its HBO2Go? Should they also allow that for free? Verizon still would have to invest in the network beyond the cache and would get no benefit from that other then helping out Netflix or HBO in these examples.
Because their customers want it. I can’t wait until I get free peering!
How are traffic ratios an indicator of fairness, or accurately assigning costs, even??
Why shouldn’t Verizon pay Cogent to receive the additional traffic, instead of the other way around? Cogent has to get the bits over into them somehow.
Because it costs considerably more to receive those bits, and deliver them to their final destination, than it does to send them.
When one carrier sends traffic to the customer of another carrier, they will usually try to hand that traffic off as close to the source as possible. The burden is placed on the receiver to carry the traffic for the long-haul.
Carriers often co-locate equipment in large Internet exchange facilities. This means that there might be a single sender router handling the traffic between the sending customer and the receiving carrier network. The sending carrier might only need to carry the traffic a few hundred feet from co-lo cage to co-lo cage.
The carrier receiving the traffic then has to haul the traffic hundreds or thousands of miles, across multiple hops: multiple large expensive routers, and multiple long-haul circuits.
Why should Cogent get a free ride from Verizon for long-haul transport?
If Cogent is heavy on the transmit, and Verizon isn’t allowing them to turn up more PAID peering ports, then that’s another issue entirely.
It doesn’t cost any different to send than it does to receive when you’re peers. Unfortunately, Cogent is mostly hosting companies and Verizon is mostly eyeballs… so they’re not peers. Verizon used to be mostly a balanced carrier (With MCIWorldcom), but with the explosion in cellular, DSL and FTTH data demands, their high priced transit has kept the hosting companies away from them.
I’m actually really surprised they only peered at 10 locations. I’d think they would peer at 10 locations in the US alone.
“This isn’t the first application last mile network operators have tried to degrade”
It’s not even an example of a network operator degrading an application.
Cogent is trying to cram more videos through a port than than that port can handle. When these streams compete with one another for bandwidth, the overall quality goes down.
If Cogent wants another port, Cogent should pay for another port. At the moment they expect someone else to pay their way.
Especially infuriating is how easily Cogent can take advantage of the public’s ignorance, manipulating hostility towards another party.
I’m actually surprised how sane the comments are here this time. Most of them are miles better than this conjecture and accusation branded as “reporting”.
I wouldn’t be so quick to place the blame on Verizon.
If Netflix wants their traffic to get through, maybe they should have purchased transit from someone with a better track record than Cogent, or at least have purchased transit from someone _in addition to_ Cogent to supplement for cases like these.
Even if Netflix/Cogent are in the wrong here, Verizon still might get burnt since Netflix is a high-profile service to be having chronic performance issues with.
My guess is that’s their angle.
I have Verizon FiOS and it is definitely not the lowest-latency (I have had comcast and got better latency). Also as for Netflix, they are definitely doing something to the stream as it buffers all the time (both PS3 and X360) and I have the 75/35Mbps plan….
I have noticed issues running Netflix and attempting to do ANYTHING else on a computer (sometimes just load a basic webpage like Yahoo! or Google). This began occurring approximately 1-2 months before Verizon began installing Verizon Fios in my neighborhood. I have been suspicious that the decreased bandwidth was intended to inspire customers to upgrade to Fios where “they could increase their internet speed and bandwidth considerably for a minimal cost increase”. Interesting.
Om, this post makes no sense. On one hand you are saying there is a problem, but then the title asks if people are having a problem. So which one is it? Are consumers having a problem or not? And if they are, where’s the proof of it? Your post basically says that ports on Cogent are getting full or are at capacity. That happens every day, that’s not news. Last mile providers buy transit all the time to upgrade their networks and add capacity.
And if this was a problem like you suggest, we’d see this reflected in the rankings that Netflix puts out on last mile providers and their ability to deliver good quality video. Last time I looked, Verizon was ranked number two by Netflix.
Having ports at 100% capacity is not un-normal and while people think you just “flip a switch” and turn up more ports, it’s not that quick. It’s not hard, but things fill up and more capacity is added. You are suggesting that Verizon isn’t turning up more ports on Cogent on purpose. Is there proof of that? Are they not turning up ports just in one city/location?
Cogent’s comment of, “today some of the ports are at 100 percent capacity” is not relevant as they don’t say what impact it’s having, on quality, or over what percentage of their network. How many ports are at 100% capacity and how long have they been at that? Days? Weeks? Months?
This story, as written, is incomplete and only raises more questions than answers. The biggest question of which is why you are implying that consumers are having problems getting good quality streaming from Verizon of Netflix content when you haven’t shown any evidence that it’s actually taking place.
A story like this, and it’s implications, will now mean that major media outlets who know nothing about video is delivered or how networks connect with one another will pick it up, regurgitate it, say there is a problem with Netflix streaming, as a factual statement, all while having zero evidence of that actually being the case. Nothing good comes of that.
That explains why everything I watch on Netflix is so dark!!!
Yet another big company playing dirty pool in order to profit. I bet a number of the comments here pointing the finger at Cogent are Verizon employees defending the actions of their sleazy employer.
In the end the only people getting screwed here are Verizon customers. Regardless of the business agreements and technical details Verizon is allowing their customers to get screwed so that they can push their own service to them. That’s just wrong no matter how you discuss the peering agreement.
Verizon statement:
“Recent reports, have raised questions about the performance of a couple of popular video streaming services. In response, we state unequivocally that Verizon’s broadband Internet access services deliver a pristine user experience to our customers at any time of day on every day of the week. This has been repeatedly proven through independent testing by the Federal Communications Commission (FCC) which has conclusively demonstrated that FiOS Internet consistently delivers both download and upload speeds in excess of what we advertise. In short, our Internet customers often get more than they pay for.
“How the Internet works can be complicated, and consumers should be aware of the fact that the integrity of their home Internet connection is only a portion of the streaming video quality equation. If their broadband connection is functioning correctly, the source of their frustration and the content they wish to see may be one in the same.”
Verizon blog:
http://publicpolicy.verizon.com/blog/entry/unbalanced-peering-and-the-real-story-behind-the-verizon-cogent-dispute
Unbalanced Peering, and the Real Story Behind the Verizon/Cogent Dispute
By David Young
An article appeared earlier this week at GigaOM, in which the authors “blame Verizon” for failing to accommodate the growing volume of traffic coming from Cogent Communications, another bandwidth provider. Unfortunately, GigaOM didn’t have the full story.
Routine Business Dispute
The authors accurately describe settlement-free peering as “essentially an arrangement between two bandwidth providers where they send and receive traffic from each other for free.” It goes on to say: “the logic is that the data sent from one network to another is reciprocated.”
What the article doesn’t say, however, is that Cogent is not compliant with one of the basic and long-standing requirements for most settlement-free peering arrangements: that traffic between the providers be roughly in balance. When the traffic loads are not symmetric, the provider with the heavier load typically pays the other for transit (see our ex parte filing [PDF] from the 2010 Comcast/Level 3 spat for more info on peering and transit agreements). This isn’t a story about Netflix, or about Verizon “letting” anybody’s traffic deteriorate. This is a fairly boring story about a bandwidth provider that is unhappy that they are out of balance and will have to make alternative arrangements for capacity enhancements, just like any other interconnecting ISP.
Readily Available Solutions
Verizon offers a number of readily available solutions for interconnecting providers who send significantly more traffic than they receive from Verizon’s networks. Solutions such as cloud, hosting, Partner Ports and others are designed specifically to provide a cost-effective means of delivering very large volumes of out-of-balance traffic. Other large streaming video providers (and/or network service providers carrying such one-way traffic) are already taking advantage of these solutions and seeing immediate benefits. These solutions are available today to Cogent, Netflix and any other content or network service provider with similar traffic profiles.
Repeat Offender
Most commercial peering/transit arrangements are able to be resolved without controversy. Unfortunately, some parties prefer to grandstand, repeatedly creating a perceived crisis that could affect Internet users in order to attract the attention of policymakers and gain leverage in negotiations (we would add that this type of behavior is detrimental to a free, well-functioning Internet, and the tech media should know better than to encourage it). Policymakers should recognize that the Internet interconnection market has seen very few major disruptions over the past twenty years, and should allow it to continue evolving to serve the needs of Internet users and content and service providers alike.
There’s no bias in this article, is there? Ahhh, GigaOm, sometimes you make me laugh…
Comparing Netflix to Megaupload seems a bit of stretch, even for telcos?
Quick clarification on Netflix traffic: according to Sandvine, it is 32% of peak time traffic load, not average. For purposes of this discussion peak time congestion is the relevant statistic anyway. Though it is worth keeping it in mind, as the ratio of peak to average increases (which I would assert Netflix also causes) it changes the economics of the equipment as well.