[qi:066] Yesterday afternoon, I left my apartment, which also doubles as my part time office, in a mad rush, late for a meeting. As the elevator descended, the lights suddenly went out, and for a minute or two (it seemed like an eternity) I was suspended in a dark metal box. And then survival skills took over, as I pressed the emergency button.
And while on the phone, the lights came back on, and the elevator descended, and I rushed to my meeting, silently cursing the building owners for having shoddy infrastructure. It was only later in the day, I learned of massive power outages caused by snafus with at PG&E, our local power company. This resulted in a transformer blowing up, and causing even more disruptions, especially at 365 Main, one of the large co-lo/data center facilities situated in the SOMA area of San Francisco.
This resulted in massive outages at some of Web 2.0’s brand name companies – Six Apart, Facebook, Technorati and Yelp – knocking out their systems and web services out flat. Whatever the reasons behind the failure might be, yesterday was a rude reminder of how fragile our digital lives are.
The seemingly invincible web services (not to mention the notional wealth they signify) vanish within a blink of the eye. It was also a reminder, that all the hoopla around web services is just noise – for in the end the hardware, the plumbing, the pipes and more importantly, the power grid is the real show.
According to North American Electric Reliability Council (NERC) there has been a 50% decline in the capital expenditures by utilities over the last 15 years. The underground cables are crumbling. This report in Fresno Bee takes PG&E to task.
… San Francisco consulting firm KEMA Inc., which wrote in a report two years ago that PG&E distribution equipment “is getting older both in terms of average age and the percentage of very old equipment. … PG&E increasingly will have to become more proactive in addressing issues related to aging equipment.”
This is not just a problem with PG&E. Power grid across the country is aging. It is ironic because data centers/hosting business is one where US companies still dominate. Thanks to an abundance of long haul bandwidth and ample data center capacity, many overseas companies find it more secure to host their digital infrastructure in the US. Bebo might be big in Britain, but its websites are served up from 365 Main.
The reliable data center/colo facilities have been a distinct advantage for US start-ups, especially the Web 2.0 start-ups. And yet we continue to bet our future on this creaky house of cards. That’s like building a Taj Mahal on quick sand. And that is one sinking feeling – the same one I got in the 100 odd seconds I found myself stuck in the elevator to nowhere this afternoon.
You raise a very valid but much ignored point.
There is a downside to the “it doesn’t take any capital to run a Web 2.0 start-up” mantra.
I am stunned that some of these larger, well funded guys don’t have redundant servers somewhere else.
I’m very curious to hear who these guys host with as power outages should not be something that takes down their sites, especially heavyweights like facebook and yelp, etc.
When we started our business, we definitely were living what everyone chatters about these days: a few people in the company, built our app in Rails, our expenses being the $150/month for cheap hosting, etc.
But then we started to get some serious clients in the corporate space and it was just no longer worth the risk. We switched to a host (Logicworks) that is audited under SAS 70 to run their datacenter a certain way to ensure against stuff like power outages taking out the site. It’s a lot more expensive for sure, but it’s (relative) peace of mind.
I guess with consumer facing sites no one thinks about this stuff nearly as much but when these services become integral to our daily lives (I remember the howls when AOL AIM used to go down) it’s just as important they stay up even if the power company sucks.
Warning: It all goes in the fire in the last days. Jesus is the only way.
@Erik “I am stunned that some of these larger, well funded guys don’t have redundant servers somewhere else.”
Every data center out there has backup diesel generators for this very reason. It’s only the truly major outages (like this one apparently was) that prevent even the diesel generators from being effective, in which case it’s going to be fixed very quickly anyways (like, within minutes to one hour). It’s only the much bigger web 2.0 companies that tend to start housing all their data in their own data centers with their own network admins, etc. that would really start having to worry about making multiple-redundancy and off-site backup plans.
As with anything in life, shit happens. Even if you’re very well-prepared, shit still happens. The web is at the mercy of the power grid, but so are our entire daily lives here in America – It’s an accepted reality. As long as there are backup plans and systems in place, there’s really nothing else that can be done.
Here’s to sustainable, off-the-fossilized-forest energy!
Great post… I think the dated American power grid and transportation infrastructure are huge issues for the US – especially, in an era of terrorism.
Any thoughts on how the govt / power Cos could be encouraged to begin upgrading the infrastructure?
Just as worrisome (from a business perspective) is that almost none of these companies offer an actual uptime SLA. If you buy into the idea of Enterprise 2.0 and how SAAS is remaking enterprise IT then a failure like this is particularly frightening.
We’ve got some research on the topic over at http://www.yankeegroup.com – though I think most of it is password protected.
interestingly power is the only 365/24/7 auction. it makes ebay look like peanuts in terms of revenue. not sure what is the situation in s.f. but in n.y. it’s a lack of transmission lines to carry power and not a lack of generation, after all who would want a transmission line in their back yard… be surprised the x-mas lights come on…think excel spreadsheets and email…
I don’t think Bebo hosted in US for reliability, it is there to please potential auditorium with lowest delay. Tho choosing UK as location place doesn’t make lesser risks related to reliability issues, I guess.
Everyone understands minor disruptions. As long as no critical data is lost, most people can tolerate a very brief loss of service.
http://mikeelliottsblog.wordpress.com
The problem is that Web 2.0 applications are centralized applications (in data centers). Compare it with blogs, it is not possible that the blogosphere will be shut down by a power outage. 😉 And that’s simply because
a) many people host their blogs on their own servers in data centers around the world or
b) they use different blog hosters, spread in data centers around the world
I know, it’s not that easy to build an open source and decentralized Flickr clone, but it will come and it will increase availability of these services.
To me the scary thing is that, from an infrastructure point of view, utility computing is going to share all of these traits. The only saving grace will be computing capacity vendors who understand that geographic distribution is a necessary part of a high up-time offering.
365 Main is a great company…the fact they failed to deliver here should be symbolic of the fact that ANYONE can lose a data center despite best efforts. I guess my only question is why Facebook, Technorati and others didn’t prepare for this possibility by distributing their services among a variety of datacenter locations…
How could these Web 2.0 companies not have redundant servers? It’s a function of moving fast and not taking the necessary steps to plan for all contingencies. For those who heed the wake-up call, Phoenix is a great alternative. http://www.bizjournals.com/phoenix/stories/2007/07/02/story15.html
Of course I hope this creates more of an urgency for alternative power. But I think the real discussion is (or should be) around de-centralizing how power itself is distributed…
As we live more and more at the crossroads of the virtual and real, self powered Internet appliances, WiTricity and virtual (with no specific physical location) servers may be the next innovations that need attention. Why can’t all the gazillions of things off the grid with hard drives pitch in classic p2p fashion?
Business model and spectrum regulation concerns for the time being aside, is the idea really that far fetched?
Did I hear anyone say “midwest”? As all the shipping companies already know, it has lower cost per squarefootage and a much more reliable power infrastructure… or so the rumors go.
And Canadian datacenters could look good too, apart from the fact they are a bit “off the grid” as major peering exchage goes (but not that bad, just “not as good” as the main American peer points).
Have you seen Jericho, the CBS post-nuclear show? We are always at the brink of disaster – the electric grid is just one more method. Not that I am suggesting we should be paranoid…Rather part of life is living with the reality that our lives are that close to chaos.
This is a very interesting and scary topic…i’ve known about this for awhile and I was wondering when any web 2.0 people would actually take notice, thanks for writing this
I can’t see why you claim that data center/colo facilities have been a distinct advantage for US start-ups.
My start-up is based in Europe and I host it in a US data center, because it has better prices and service. I can see why that’s good for the US economy – money from start-ups abroad coming in, but how’s that a distinct advantage to US start-ups?