Tuesday 28th February 2012

Network Core issue resolved

We have had a brief chat with the data centre team and the root cause of the downtime last night is believed to be down to a broad DDoS attack across a number of subnets - peripheral to our own network, but substantial enough to saturate the 10GB uplinks to our peers.

A formal investigation is under way at present, however, we have been assured our own connectivity should not be affected any more.

We would like to apologise for the outage last night, which spanned 11 minutes in total, but we hope our proactive response to the situation and information clarity throughout was of some benefit to concerned customers.

We are currently discussing means to prevent this happening again, however, as the attack was not directed at subnets within our own network, it will still be hard to mitigate.

For reliability and performance, we hand off BGP to our upstream provider who uses multiple peers and handles external (internet) routes on our behalf - however, this was our downfall, as when another customer of theirs fell victim to a DDoS attack, it saturated the common transit uplinks affecting the entire data centre.

We are not in doubt of our current peers/transit providers; as it has served us well, with 3 years of 100% network connectivity and we have full faith in their ability to deal with future issues.