Update: Akamai spokesperson declined to comment. We are still waiting to hear from Apple.
Apple’s holly jolly Christmas, that resulted in extended outages and infuriating delays for customers trying to access the iTunes Store have been well reported. Some of us (guilty as charged) were amazed at how successful Apple was in squeezing billions of dollars out of its iPod franchise. According to Hitwise, there was a 1,222% increase in traffic to the iTunes store on Christmas day compared with the previous Monday. The traffic to the Apple Store was up 110%.
An AFX report on CNN Money website quoted Michael Gartnerberg of Jupiter Research as saying…
‘What you’re seeing is the tremendous success of the iPod …. No doubt it was a very, very popular gift, and no matter how well you plan on the server side of the equation, there are always times when you get caught short.’
Getting caught short just be the excuse of choice this season. Sony and Nintendo were caught short and did not have enough consoles. Their inability to meet the market demand for their consoles may be excusable since they were introducing new platforms, but Apple with a few successful holiday seasons under its belt did not have to face those issues. So why did they really have the problems at the iTunes store?
Forget what they tell the analysts – Apple is one of those companies which can pretty accurately forecast demand for its products. It is reasonable to assume that Cupertino knew that it would sell a lot iPods. And since the sales of iPods are in near direct correlation with visits to the iTunes store, there is a reasonable chance that Apple also knew that the traffic to the iTunes store would be up sharply.
In other words, someone did not plan properly for this holiday season. Of course, there is another explanation – one of their key infrastructure provider, Akamai Technologies failed to keep up with the iTunes crush.
Akamai’s content delivery network is supposed to keep things up and running without fail. (See how Akamai’s service works!) That is one of the reasons why Akamai can charge premium over others. They often talk about 20,000 servers in 71 countries to make the music delivery seamless and reliable.
If what Akamai promises is delivered then none of us should have experienced the iTunes Store outage. Akamai is betting big on digital music – streaming and download varieties – to make a lot of money for next few years. The iTunes store meltdown certainly is not going to help with that.
Update: Akamai spokesperson declined to comment. We are still waiting to hear from Apple.
So you’re just speculating that Akamai was to blame even though you really don’t know what the cause of the problem was? That’s not very responsible journalism.
you are an idiot! to make accusations with out knowing the facts
In the face of a lack of explanation from Apple, what is one to do but speculate? Om speculates, and I think it is fair.
Let me throw another speculation, though: could it be that Web Objects is finally cracking under the weight of scalability?
I doubt it, though —after all, Apple’s Web Objects software powers the 4th largest retailer, by dollars, on the internet. By comparison, it is also well known that Amazon (which does not use Web Objects) crashed for 20 minutes on Thanksgiving day, 2006.
Maybe Akamai does not deserve to be blamed. But Apple sure does. Om is right that Apple didn’t just this year discover what it means to sell loads of iPods for Christmas!
I also wish to comment on the AFX Report quotation: “no matter how well you plan on the server side of the equation, there are always times when you get caught short.”. Well, I reckon that is not true when you use Peer-to-Peer.
And let me add 2 speculations:
1- overload of the DRM servers, cause there’s no limit to the harm that DRM can do to online media distribution
2- sabotage
I haven’t seen any detailed descriptions of the problem, but it seems unlikely that there would be a global problem with Akamai limited to Apple.
I can’t believe suckers are still buying DRM encoded garbage from the iTMS. It would be amusing if that was where the bottleneck was- generating all of the new keys.
Another more basic problem is that iTunes 7 is perhaps the worst version of iTunes on Windows. Crashes all of the time anyway on my box- it can’t deal with networked drives at all.
1200% increase in traffic on xmas day?!?!?!?
Can you plan for that uptick for a short period of time? I wasn’t a big believer in ondemand cpu services but for apple, on xmas day, it would have been a savior…that is if it actually worked.
Om, I don’t think Akamai is to blame. Apple uses them and other CDNs and also buys transit and peers so if it was a network related issue, it should have healed itself.
How is it that Akamai can take credit for all of these media sites running so smoothly, but then when something goes wrong, not a peep?
Maybe we should ask Om whether he contacted Akamai and whether he got a response.
If Akamai did not respond with a clear statement that their link in the chain was not at fault, then I will bet my bottom dollar that Akamai was the culprit.
But maybe I don’t have to bet my bottom dollar. . . it’s likely that Apple will be moving their $20MM+ yearly payments for Akamai to somewhere else.
I’d watch the news for an Akamai CDN/bandwidth switch sometime soon.
I’ll type this slow so you can follow…
Just before christmas Apple needed a certain amount of bandwidth and for sake of discussion we will call this 1. Being a responsible company who expects to grow, they can handle a larger flow than that and knew that bandwidth would increase so they were able to take a bandwidth of 2. On christmas day demand jumped to 12 and in the days following it dropped back to 1.5.
Why the hell! do you think Apple would spend the money to be able to support a one day bandwidth jump to 12? Apple knew that some people would be slowed and others would have to wait and even that some people would be turned off to the service because of it. But the cost of trying to things ready to handle a 12 fold increase was just not worth it.
It is just business! It makes no sense to try to handle this one spike.
Just for reference, today I’m downloading a movie at 2x Real time. So It will take 30 minutes to download an hour of video. Apple seems to be handling demand just fine.
Well said, doug.
I see no reason to believe that Apple couldn’t fully predict the # of hits (the spike was much greater than in the past, the spike now includes larger forms of content that had been selling in limited number, and now also includes the need to download the application since most iPods do not feature a CD), plus the simple fact that demand can exceed the practical, and the fact that even if Apple had wished to meet that greater demand, could Akamai have met it?
Akamai could certainly have met the demand. They have a huge amount of untapped capacity. Part of it is available for situations just like this. After the Windows Vista Beta launch, Akamai went about securing even more bandwidth.
Taking a look at the source HTML for Apple.com, its clear that not everything is akam-ized. The front page, in particular, looks like it all lives on Apple, and is downstream from Internap. The deeper one gets in the Apple Store page, the more you get from Akamai. However, at all levels, there is a dependence on the Apple IT infrastructure.
As we have not heard of other akamized web sites having issues, I am deeply skeptical about blaming this on Akamai.
Here is the thing guys, you make good arguments but it is the content part – the one that is handled by Akamai which had problems. In other words, there was something strange going on with the entire infrastructure.
One of the reasons companies outsource is because they can predict that there will demand during certain times, and increase or expand their infrastructure to scale reliably.
That should have been done here as well.
tomo, you say that because it is one day, Apple and their vendors shouldn’t be prepared for the crush. Why not? This is a mega billion dollar business, one that has made Apple the stock market darling. Not spending on infrastructure is to kill the golden goose.
How many first time users might have left with a bad experience. Which is very un-Apple like!
What about all the iTunes gift cards sold at retailers – even my local grocery store carries the things – those alone could create a spike in demand at the iTS. So some poor kids didn’t get their instant gratification fix. My kids cry every time a server doesn’t load a flash game or web page quick enough and slam the poor G3 iMac for being slow. Everyone needs to learn to slow down some and be more patient! How soon they forget the horrors of 56k dial-up…
Now if there were widespread reports of failed transactions, lost money and incomplete downloads, then complaints would be justified.
Om, I think you are confusing my comment with Doug’s. I stated that I doubt it is an akamai issue, but i could be wrong, because akam isn’t the only path to the internet from apple’s datacenter in newark, ca where their infrastructure sits. In newark, they have boatloads of transit already installed. Probably commit to around 3 gigs from multiple providers and sustain 1.5gigs would be my guess(probably low) so actually the traffic would have hit 1800 Gigs? That doesn’t make sense either…was it maybe the # of users that spiked 1200%?
Now I have a big foot in my mouth TOMO 🙁
Guys, read Om’s update. Akamai has “No Comment”
If it wasn’t Akamai’s problem, wouldn’t Akamai comment on this??
something like – “Akamai’s service ran normally through the holidays, and experienced no network problems”
If they haven’t said this, it’s because they screwed up. There are few other reasonable explanations why Akamai would simply not comment. Just wait til WSJ picks this story up, or if AKAM loses the Apple account.
Maybe they’ll comment then! Until then you have to assume the worst. Akamai should know better and should respond quickly so that their customers don’t have to get their information from the rumor mill.
Sorry, but I have to disagree!
Sure you prepare for the crunch time but you don’t prepare for the spikes, there is no money in it. Being able to handle 2-3 times your normal flow to deal with surges and growth is fine but it is not worth while to have 12X your normal bandwidth sitting in reserve.
Look, it would be different if the iTMS was down for the week following christmas and was still slow today. But this was over sometime late on the 26th or early on the 27th. The store is handling demand just fine.
And for those who think they should be ready for that type of spike due the the “mega billions” of dollars…Sorry, the iTMS is above break even now but we are talking about pennies/song. This is a multi million dollar business for Apple each year but not a even a multi billion dollar, let alone a Mega billion dollar business.
BTW: Want to see Apples site brought to it’s knees again? Try to check it out around 10:00 Jan 8th. Steves keynote announcement will bring down the site for about an hour and then everything will be fine. It’s not Akami, it is not Apple, it is just the nature of the net!
There are two sites: the Apple.com website where the iTunes application is downloaded from, and the iTunes site (where media is downloaded from within the iTunes app). Are both serviced from the same Apple (or Akamai) infrastructure?
I had trouble on Christmas day (around 5pm Eastern time) with downloading a free song from the iTunes site (was only able to log in once in five tries and the one time I got in, the download didn’t properly proceed), but none at all with the Apple.com site (or for that matter, the Apple Store site) when using both almost concurrently.
Petrosky,
You’re just showing your ignorance of how bandwidth is bought and sold. In most cases, CDN bandwidth today is sold on a Terabyte basis, also known as the tonnage model. It’s not about the spike, it’s about how much data was delivered. In that way, your costs scale with your revenue. If it made sense for Apple to sell songs on November 26th, then it makes sense for Apple to sell songs on December 26th. Apple pays the same amount per MB, GB or TB regardless of the day or the spike.
Now you might be old school and are thinking of the 95th percentile measurements (for those unfamiliar it means to take all 5 minute averages during a month, shave off the top 5% of the data, and charge based on the next highest 5 minute average, the 95th percentile).
Even if Apple was on some old school contract, which it is very unlikely that they are (since Akamai sells scaling and spike mitigation), they have 36 hours in each month to spike as high as they want without having to pay for it. Apple could spike 100x higher than normal in that 36 hour period and not pay a nickel more than what they would if they had steady low bandwidth usage the rest of the month.
Since Akamai handles their downloads delivery then if Akamai wasn’t to blame, then you are saying that Apple artificially slowed the service down. The only way this is possible is if Apple specifically requested that Akamai not cache the content in their distributed network and instead redirect users to their servers. This is a pretty unlikely scenario, but certainly not impossible.
Bottom line is that there is no logical reason for Apple to cut the cord due to financial concerns as you explain (the pricing models simply wouldn’t demand such a drastic step.
And second, you still haven’t addressed why Akamai wouldn’t be all over this misinformation if they truly weren’t at fault. Service providers protect their reputation at all costs. If the allegations weren’t true, they would simply say so.
We have an acronym for this in mathematics,
Q.E.D.
I suspect that the ITMS volume had at least as much to do with the sales of iTMS Gift Cards (for which there was no seasonal precedent) as it did with the sales of iPods (for which there is a seasonal precendent). Given the proliferation of those Gift Cards this season, they were likely a tempting purchase and equally tempting to redeem right away.
The other piece that likely contributed to the perfect storm is the new swarm of users drawn by iTunes 7, which now downloads album covers even if you didn’t buy the songs from the iTMS. I’ll bet (and we’ll likely never really know) that there are at least 10x as many iTMS users (if not customers) as there were last year.. all busily banging away at the service network.
-Ian.
I wouldn’t take a “no comment” from AKAM as an indication of anything. They have strict NDAs with Apple. I’ve run into this before with both AKAM and Limelight – even when its not their fault, they won’t talk about it, as they are deathly afraid of criticizing a customer, even by inference.