Tuesday 6th December 2016

Network Interruption

Disruption

We are currently investigating a possible network incident.

  • Update (16:41): We believe that the issue has been identified, restored connectivity and are monitoring the situation. We will post a more detailed update shortly.
  • Update (16:56): The network has been stable for the past 15 minutes. An upstream transit provider suffered a large amount of packet loss. We have re-routed traffic around this provider and appear to see normal performance. We will continue to monitor with our connectivity to this provider shut down until we have confirmed it is healthy again.
  • Update (17:24): Upon initial diagnosis, we can confirm the loss of connectivity was caused by link saturation of a provider during a dDoS attack targetted at a single customer. A full post mortem will be available shortly.
  • Update (17:37): dDoS mitigation remains active, whilst some smaller attacks continue, it is not service affecting.

Post-Mortem

Our report from the incident is as follows.

Issue

Loss of connectivity from some ISPs.

Outage Length

The duration was 9 minutes.

Underlying cause

A large volume DOS attack targeted a single customer and was of such significant volume that it saturated the connectivity of one of our transit providers.

Symptoms

Our monitoring probes immediately reported the attack. Despite the attack being targeted at a single customer, the volume affected all our customers causing high levels of packet loss.

Resolution

Initially, without full information available, we interpreted the packet loss as an issue with a single transit provider and shut down our connectivity to said provider. At that point, traffic re-routed to our other (larger capacity) providers and the issue looked to be resolved as the larger capacity transit "absorbed" the attack. Moments later, our Level3 dDoS mitigation platform automatically activated and began scrubbing the malicious traffic and we restored connectivity to the original transit provider.

From start of attack to mitigation - the total time was 9 minutes. Our dDoS mitigation platform is a relatively new addition to the Sonassi network to offer an unprecedented level of protection to customers - and we are extremely happy that yet another large volume DOS attack was mitigated with only minimal disruption prior to activation.