Friday 14th June 2013

Network Some packet loss over international routes

We have a number of customers reporting some packet loss whilst accessing their servers from outside of the UK.

  • Update (22:07): This appears to be an issue with one of our upstream carriers. Updates to follow.
  • Update (22:20): Routes have been de-preferenced over L3 and Global Crossing;  Cogent is currently be prioritised higher and outbound routes are proving more stable.
  • Update (22:34): The majority of inbound routes over the selected carriers are now flowing via Cogent and all external health checks are reporting healthy.

<!-- more -->


Our report from the incident is as follows.

IssuePacket loss affecting a very small number users worldwide, resulting in slow page load times, or no response.

Underlying causeFibre break within Level 3 / Global Crossing’s core infrastructure. Our other carriers were unaffected, so only those ISPs peering with Level 3 saw issues.SymptomsUp to 100% packet loss from a limited number of national and international ISPs where traffic was coming in via Level 3.ResolutionWe temporarily stopped using L3/GBLX and prioritised our routes over Cogent until L3/GBLX had put repairs in place. We received confirmation that at 3:00am the broken fibre had been fixed, and routes began to flow back over L3/GBLX once again.