valleywag.com/tech/breakdowns/a-drunk-employee-kills-all-of-the-websites-you-care-about-282021.php
RSS 20 Partial Content (ad-free) You people in Silicon Valley are far too busy changing the world to care about sex, greed and hypocrisy. So you won't want to read a tech gossip rag like Valleywag.
com/tech_news/Drunk _guy_kills_internet Whoever it is, while we like how you roll in theory, in practice, we'd appreciate it if you laid off the servers running websites we actually use. We're sure 365 Main will deny that such a thing could ever happen.
But here's a question: When you have several levels of redundant power, what could bring your customers' servers down other than something like an employee physically ripping the plugs out of the wall? Or, with less effort, hitting the emergency-power-off switch that San Francisco's building codes require 365 Main install? Update: Technorati's Dave Sifry just sent this email: Folks, I just wanted to let you know, it looks like San Francisco is having a MAJOR power event, with outages from the Financial district all the way down to Daly City. One of our colos at 365 Main Street has experienced a power outage (never mind that they always swear up and down that this kind of event can't possibly happen, oh no, they have multiple redundant systems and they charge us up the wazoo to make sure that we'll have business continuity, so of course, this isn't really happening, oh yes) however, our other data centers are all up and running, so we hope to be back up and running as quickly as possible. I'll keep you all updated on progress, and I appreciate you bearing with us as we work our way through this...
The power's going off all over town (see twitter), not just at one data center. PG&E just told a co-worker that 10,000 people are affected, and it will be fixed between now and 4:30.
com: (who also has their servers @ 365 Main) "Customer Service Is our Top Priority. That's why we're working today to upgrade our system and make it easier to use."
This means that the TypePad application and your TypePad blog are not reachable at this time. This began at approximately 1:50 pm Pacific Daylight Time today, Tuesday July 24 2007. We are working closely with our hosting partner to bring TypePad back online as soon as possible. We sincerely apologize for the inconvenience this is causing, and we appreciate your patience. We will send another email update with more information as soon as possible.
I have a hard time believing that Craigslist, Typepad, Netflix, and the like are just in shared racks, rather than in they're own cage. Getting to the power plugs (or trashing servers) inside a cage is a pretty big PITA so unless you have the key/combo. My take is that unless it was a 365 Main employee that got into the battery room and went all Pete Townsend on the place I'm going to go ahead and believe them.
I don't see why 365Main couldn't have handled it the same way. Obviously, like this article suggests, something else is the reason of this giant blunder. I don't think they'll ever admit what has happened though.
Pacific Gas and Electric Company has recognized 365 Main's San Francisco data center for its noteworthy energy-reduction accomplishments while participating in PG&E's Critical Peak Pricing (CPP) program. The CPP program is designed to curtail energy load during critical peak days to offset the possibility of an energy emergency... The most significant power reduction, however, was attributed to an innovative testing procedure for the building's back-up generators. Each of 365 Main's five national data centers is equipped with powerful back-up generators to ensure customer uptime in the event of a power outage. In 365 Main's founding data center in San Francisco, the company maintains ten 21 MW (megawatt) generators manufactured by Hitec. These generators, known as Continuous Power System (CPS) generators, run 24 hours a day, ready to deliver 100 percent power to the data center in the event of an outage. As part of a comprehensive preventative maintenance program, 365 Main tests each Hitec generator once a month by running each of the 3000 horsepower diesel engines for two hours.
I'm sitting in Telecom One right now on the back-side of 365 Main, and we're still having some rippling issues. There are in fact, very large, redundant diesel generators under the foundation in the basement, so understandably form the outside it's possible that one might not hear them. However, I wouldn't put it past 365 to hire an incompetent, asshole employee. They go through at least a whole set of guards/NOC techs every few weeks or so. It's not a rotation either, most have said they've just seen people quit. Redundancy in all aspects is important in this regard as well.
Figaro: I think it's about 10 zillion times more likely to be a technical fuckup (they can't switch over to the generators due to stupid) than the Very, Very Emo Disgruntled Employee story. Who, coincidentally, went on his angry drunken rampage just as power issues hit the city?
To become a registered commenter on this site, you first need to be approved by our team. We're looking for comments that are interesting, substantial or highly amusing. So write a comment, polish up your words and choose a username and password below. Your comment will only appear once (or if) you're approved.
|
http://csua.com/feed/