Thursday 27th June 2013

Network Some packet loss over UK routes

One of our carriers has reported a fault with routing in the UK.

  • Update (17:54): We have failed over to another carrier in the interim whilst this is investigated. Updates to follow.

  • Update (18:02): The issue stems from a major fault at LINX. We’re routing around LINX right now, but other ISPs may still be sending traffic via LINX and there might still be users experiencing packet loss.

  • Update (18:15): Traffic flows have reached normal levels again, the issue at [ One of our carriers has reported a fault with routing in the UK.

  • Update (17:54): We have failed over to another carrier in the interim whilst this is investigated. Updates to follow.

  • Update (18:02): The issue stems from a major fault at LINX. We’re routing around LINX right now, but other ISPs may still be sending traffic via LINX and there might still be users experiencing packet loss.

  • Update (18:15): Traffic flows have reached normal levels again, the issue at]5 looks to be resolved.

<!-- more -->

Post-Mortem

Our report from the incident is as follows.

IssuePacket loss affecting UK ISPs.

Underlying causeBroadcast traffic at LINX causing high CPU usage on core routers. SymptomsInaccessibility of our network from certain ISPs.ResolutionWe dropped our peering session with LINX whilst the issue was diagnosed and identified. The issue at LINX has reportedly been fixed and our network team will enable peering at LINX once we are confident their routers are once again stable.