Tuesday 3rd November 2015

Network Network Interruption

Unconfirmed connectivity issues reported.

  • Update (14:36): We're seeing reachability issues to our legacy network IP ranges.
  • Update (14:37): The transit provider in question (that carries our legacy ranges) appears to have made a routing change and traffic is now flowing.
  • Update (14:51): Connectivity has remained stable for the past 5 minutes. A reason for outage will be provided once available.

Post-Mortem

Our report from the incident is as follows.

Issue

Loss of connectivity from some ISPs to our legacy IP ranges.

Outage Length

The duration was 30 seconds.

Underlying cause

The transit provider carrying our legacy IP range inbound traffic experienced an interface flap at LINX, triggering a re-route and re-convergence of routing.

Symptoms

Our external monitoring probes immediately reported the fault. A very small number of users would have been unable to access their servers.

Resolution

The issue resolved itself by automatically selecting another carrier (Level3) when the connection at LINX dropped. The cause of the downtime was merely the delay of end-user ISP route re-convergance.

All customers are already being migrated from our legacy IP range as part of our 2015 IP migration; giving us full control of all customer traffic, both inbound and outbound.