Tuesday 25th October 2016

Network Interruption

Confirmed connectivity issues reported.

  • Update (15:10): An engineer is looking into this now, updates to follow.
  • Update (15:25): A upstream transit provider appears to be suffering some packet loss. We have re-routed traffic around this provider and appear to see normal performance. We will continue to monitor.

Post-Mortem

Our report from the incident is as follows.

Issue

Very brief loss of connectivity affecting around 20% of total traffic.

Outage Length

The duration was <4 minutes.

Underlying cause

An upstream provider suffered a router reboot, which caused traffic to be dropped both in/out of our network, for the small volume of traffic that passes via that provider.

Symptoms

Our monitoring probes immediately reported the incident. Customers would have experienced slow page load times through to a completely inaccessible site.

Resolution

The issue was resolved by temporarily dropping our connection to the respective provider and re-routing traffic over our other transit providers. The issue immediately subsided and normal traffic flows resumed. The upstream provider since has restored a previous configuration on their device and has it running stable again, we have since restored our connection to them and are operating at 100% capacity.