Scratch

Here you will find older scratches, musings and other detritus that once were located on the front pages, but have now faded.

Filed away here to collect dust and cobwwwebs in perpetuity, links may break, facts may change and data may corrupt.

On occasion I may come down here to tidy, but for the most, I prefer to leave the past where it lies.

Darn Power

So, at 1205 this afternoon I got the first SMS (of many) from a piece of equipment saying there had been a mains failure. Approximately 10 minutes later that room was effectively dead. Some 10 minutes after that another four hundred or so servers died, and about five minutes later, the remaining 400 became uncontactable, presumably due to network links dying. Fingers crossed the generators have kicked in, but unless a thousand odd gallons of diesel materialise iin the tank in the next six hours, it becomes a mute point.

Up until the point where the UPS' finally gave out I was happily polling devices, checking electrical voltages, temperature, air flow, uptime, all across thirty/forty different locations. When everything runs out of UPS cover however, I am and might as well be in the dark.

Neither BBC News or News 24 list anything, but given the The Tube was reporting major problems and their live update has now fallen over, I am quite confident this isn't that small an incident anymore.

I suspect its gonna be a long night.

**Update 23/06/2006 16:20: It appears that the power problems stem from an EDF installation in St John's Wood which is effecting pretty much everything from Baker Street to Kings Cross, and as far south as Googe Street. As of about 45mins ago, EDF reckoned the power would be on by 16:00... Looking at the Tube's Realtime News it would seem that its all still down. Strangely, if you read the scrolling ticker at the top of the page, so is the live travel news which coincidentally is the page your looking at...

23 Jul 2006 14:16 | (2) comments | London

Comments

:( oh dear.
The highvoltage line serving lots of manchester tripped off on wednesday, just long enough to drop a thousand desktops and many more nodes in machine rooms around the university.

Manchester Computing's link to the backbone fell over completely so we were running through lancaster... I think their switch room aircon must have taken a pounding!

Hope you've got everything back up.

Posted by: chris | July 23, 2006 05:45 PM

I love electrical companies.

To be fair, we managed to maintain connectivity to Janet for about an hour, after which point the servers supported by generators just happily trundled on until the power resumed.

Some network problems with bits not autostarting. Few blown disk packs but generally it was ok.

This time ;)

Posted by: skitz | July 26, 2006 10:06 AM