Welcome to the High Availability Hosting Blog

As of yesterday morning the Level 3 fibre is now live with customer traffic, and takes over the default routing previously had by Virgin Media.

We’ve also enabled our “floating default gateway”, based on Cisco’s GLBP (Gateway Load Balancing Protocol), which provides outbound load-balancing, and allows us to “flip” between Level 3 and BT.

As always when you get a new internet connection, the first thing you do is speed test it!  We’ve not been able to find a UK server with enough bandwidth to fully test the 1Gbit link, the closest we could find was a server in London:

Customers may notice traffic coming in or going out of the Sheffield Data Centre via different paths, and perhaps not the path they were expecting.  This is normal.

DrayTek routers and Cisco GLBP

We’ve found that DrayTek routers don’t seem to like the “floating” part of the load balancer.  In basic terms, Cisco GLBP works by having a “master” ARP the MAC addresses of the “slave” routers to clients when they request the MAC of the default gateway to send traffic to.  That way the gateway and MAC can be switched to the other router for failover,  and the “master” can load-balance outbound traffic across the two links.  DrayTek routers don’t seem to respond to the ARP request properly and never get the MAC address of the router.

We have a fix in place, and are working with affected customers.  This only seems to affect customers with DrayTek hardware, all other routers, firewalls, and servers seem to be OK.

The Level 3 fibre connection to the Sheffield Data Centre went live yesterday, and following 24 hours of test traffic we’re ready to integrate the connection into BGP.

First tests of customer subnet traffic will commence tomorrow (13th June) from 07:00. If all testing completes successful the new “floating default gateway” (Cisco GLBP) will route live traffic.

We’ll post a speed test result tomorrow from a PC here once BGP is live :-)

This coming Monday (10th June), there are a number of Telco things happening…

Level3 Fibre Go-live
Monday is the Customer Commit Date for Level3, meaning peering will start and we’ll be able to begin routing test traffic. Nothing will change for customers until later in the week, and should be seamless.

ISDN Conversions
BT Openreach will be on site most of Monday afternoon converting our existing analogue lines to ISDN lines for our new ISDN & SIP phone system. During the conversion our usual 0114 number will not be in service. We will be posting an alternative number to contact us on, on Monday.

Yesterday we upgraded the hot air extraction in Aisle 2 at the Sheffield Data Centre, and today we had the hottest day of the year (so far) with no customer complaints and a successful go-live for the new air pump.

The upgrade gives a 20% improvement on airflow, and the old tube from Aisle 2 will be used to upgrade Aisle 1 in the near future.

As the Sheffield Data Centre expands, the cooling and air extraction are increased in a modular-fashion to meet the increased demand. The main advantage of the modular method is that we can complete most works in-situ and without interruption to service.

The cooling and hot air extraction at the Sheffield Data Centre, was pretty unique and radical when it was first implemented (although more common in different guises now). Evaporative Coolers humidify and chill the outside air, which is then pumped into the computer room and through the servers. Extraction takes place from contained hot aisles, through in-line fans and ducting, and direct to the outside. Energy is not wasted trying to re-chill the hot air, we simply get rid of it!

Above the computer room, away from the servers, lays a series of fans, ducting, cable trays and other equipment, where the new extraction was installed. The photo below taken during installation:

More on our cooling can be found here: Sheffield Data Centre – Cooling

We’ve completed the roll-out of CloudBackupUK 4.3 to all servers and clients.

New features in this version include faster streaming of data to our servers from the client machine, and the ability to restore a disk image backup as a Virtual Machine – something we’re very excited about here as it opens new opportunities for Disaster Recovery.

Clients (end users’ servers and desktops) automatically get the update when they connect to our server, no Administrator input is necessary as the process is entirely automated.

CloudBackupUK works by streaming data from the client server to our Cloud servers hosted in the Sheffield Data Centre.  Depending on the type of backup ran, users can opt for a disk image (where the entire disk is streamed for recovery), or file backups (where selected files get backed up).  The whole process then runs silently in the background, backing up data either continuously or on a schedule selected by the user.  Backups are also encrypted with AES encryption to which only the user has the decryption key.

CloudBackupUK – from HA Hosting

“Cloud” backup services are nothing new, but we have the advantage of guaranteeing your data will remain in the UK, and never be exported to 3rd party cloud storage hosted potentially anywhere in the world.  With this and the encryption, we ensure we comply with the UK Data Protection Act.

Cloud Backup UK is protecting many clients of HA Hosting already, if you want to give it a try email sales<at>hahosting.com and ask us about it!  Prices start from just £5 per month per licence.

We’re pleased to say we are bringing forward the go-live of our Level3 gigabit fibre to the 10th of June, less than two weeks away.  The go-live for customer traffic will be later that week.

We have the router on-site and are in the process of configuring it ready for racking, shown below with a fibre cable to a switch for testing.

Level3 CPE router being configured

We have just completed disconnecting a customer from our “73″ subnet as they were the recipient of lots of unwanted traffic which may have made connectivity for other customers slow. We will be working with the affected customer later today (Tuesday) on a resolution.

This is not the same customer who was the victim of a DDoS earlier this month.

(posted 28/05/2013 01:10)

Following on from last weeks’ DDoS, we worked to resolve several outstanding issues yesterday (Friday) where smaller but more frequent attacks were trying to get through to the customer who had already been disconnected from the network.

All our subnets except the one being targeted were moved yesterday from our BT fibre to our Virgin Media fibre which fixed problems for the majority of customers, however customers on our “76″ network were still seeing intermittent traffic levels up to 17:00 yesterday.

Working with BT, we have blacklisted the IP addresses being used as targets.

We will continue to monitor traffic levels over the weekend and re-assess on Monday re-balancing traffic over the 2 fibres.

With our Level3 Gigabit fibre going live next month, we will have more capacity and flexibility to re-route traffic even further in the future.  We will also be updating our Terms and Conditions.

One of our customers was on the unlucky end of a short DDoS attack this afternoon.  We’ve isolated the customer and traffic levels across the network returned to normal a short time later.

Openreach have been out and blown in the fibre for our Level 3 connection, and have been back to complete end-to-end testing.  Testing passed, and a man-in-a-van will be dispatched to install NTE’s later this week.

And because we’ve not done it for a while – here’s some pictures!

Openreach on site again!

Fibre splicing – hot stuff!

Testing underway

20Km end to end

The circuit end-to-end is just over 20Km, and Links to Level 3′s POP in Sheffield.  The circuit also puts us “on-net” and allows Level 3 to expand from the Sheffield Data Centre to other sites as required.  Go-live for the circuit it still scheduled for next month (June), although we’re hoping to route some test traffic down it soon to test latencies, routing tables etc…