Welcome to the High Availability Hosting Blog

For the first time we can say our standby diesel generator was used in anger over the weekend.

A 50 minute power cut in Holbrook caused the standby generator at the Sheffield Data Centre to fully take the load during the blackout.  Redundant power and other emergency systems ensured colocation racks stayed on, and no colo customers suffered any downtime.

The generator Automatic Transfer Switch and failover systems worked perfectly!

There was an issue with one UPS which tripped (the UPS pending new batteries, a victim of bad timing – now fixed, see other blog posts), which caused a VMware node and 3 switches to reboot.  We attended site, restarted the affected node and rebalanced the cluster manually. Some other Virtual Private Servers were restarted as a precaution due to network inter-dependancies.

Health Checks were done on all VMware and Hyper-V nodes, as well as visual inspections of each rack.

Just a quick one to say we today fired up the standby diesel generator at the Sheffield Data Centre – the genny fired up first time with no errors.

We’re planning another full failover test later in the month, watch out for updates!

(not particually exciting, but reassuring to know it works first time, every time!)

We’ve just done our final standby generator test for 2010, even in the cold Christmas Eve weather the generator fired up just fine  :-)

Merry Christmas from the Sheffield Data Centre!

Did anyone notice the power going off at the Sheffield Data Centre on Monday?  Nope?  That’s because our standby generator cut in and took over without interruption to service!

We had planned electrical works Monday (25th October) morning (Meter Plus onsite installing new equipment), and as part of a controlled failover the generator was started and mains power disconnected.  The mains was ws off for 16 minutes but service remained at 100%.

The work (which was planned well in advance) was pre-empted with a number of controlled start / stops of the generator and 2 complete data centre failovers beforehand, the longest being Sunday 24′th where the Sheffield Data Centre was running 100% on generator for 2 hours.

We’ve never had a utility power outage (we have a feed to a substation just outside the compound), but we’re confident that if we did there would not be a service outage!

Unlike other facilities which might have to get customers to shut down servers before maintenance, we can keep the lights on for up to 9 hours with the diesel onsite, and we’ve got fuel re-supply arrangements in case that’s not long enough!

…an example of High Availability Hosting at it’s best!

Just a quick one to say we’ve done another generator test today.  The generator fired up first time and ran for 5 minutes while we tested the exhaust and engine running.

The next test will be a failover test, scheduled for sometime within the next 2 weeks.

Yesterday (22/08/2010) we successfully tested and failed-over the Sheffield Data Centre to backup generator power.

The culmination of over a years planning, and many hundreds of man-hours to implement a N+1 power system without downtime or service interruption was finally completed just after 1pm yesterday. Utility power was disconnected, the Automatic Transfer Switchgear started the generator, waited until it had stabilised, and then automatically threw the switches transferring the feed.

Testing over the last three weeks, including a battery of fresh testing yesterday ahead of the fail-over included auto start/stop, continuity, voltage & frequency, load testing, UPS testing, engine servicing etc…

The failover was repeated twice and we had the building totally generator powered for 15 minutes before switching back to utility (mains) power.  The system is now set to automatic so in the (unlikely) event of a power cut we’ll fail-over to generator without human intervention.

Regular testing will ensure we’re always ready!

We did video the failover and take some stills, they’re in the process of being editited and we’ll put some images up on the blog soon.

The Sheffield Data Centre is now an N+1 power facility.

Just a quick update to say all the planned weekend works to cable and connect power to the generator charging system were successful with no service outages.

Ongoing works this week include installing a power usage meter and the emergncy stop breaker to the Automatic Transfer Switchgear. All non-service affecting.

Once the electrical works are complete we’ll post some pics!

Cabling works for our standby generator are finally underway, and should be completed over the weekend.

We have plans to connect the Automatic Transfer Switchgear to the generator TODAY (06/08/2010), and connect & test the power to the auto-start system over the weekend.

Next week we’ll run a series of tests of the auto-start system, transfer controllers, and the generator iteself (without switching supplies), with a view to a full failover test toward the end of August.

Later than originally planned – delays have been caused by human resources (lack of people time!!), weather, ongoing client projects, and other data centre engineering works.  We’d like to thank everyone who’s showed us their support and kept a keen interest in our expansion!

All client and core equipment is protected by UPS anyway, but the generator will add an extra layer of redundancy should there be an extended power outage.

Regards,
Stuart.

www.sheffielddatacentre.com

At 05:30 this morning a huge crane lifted our generator from it’s storage place nearby to the concrete pad and secure compound at the back of the Sheffield Data Centre.  It took a little time to plan, execute, beat the weather, and generally organise the lift – but it’s done and will be connected to the Automatic Transfer Switch shortly!

Not a great pic below, but it was practically dark when taken – the generator being lowered to it’s base:

Flying Generator

More pics and info will be available soon!

After several delays (mainly caused by bad weather, contractor delays, and Christmas!) we’ve finally had the footings dug out and concrete poured for the generator enclosure.

The grass banking was dug out on Tuesday 23rd, and concrete poured Thursday 25th.  Blockwork has been delivered and the enclosure itself is scheduled to be constructed next week.  Once the encloure is complete we’ll pull the thick generator cables through and move the generator itself into position.

We’re confident that (if there’s no more snow – which apparently doesn’t help concrete set!) the generator will be in position mid-March!

Digging out the footings

Ready for concrete

Concrete finally poured!

The generator, in storage. Waiting to be positioned.