We have a number of customers reporting some packet loss whilst accessing their servers from outside of the UK.
- Update (22:07): This appears to be an issue with one of our upstream carriers. Updates to follow.
- Update (22:20): Routes have been de-preferenced over L3 and Global Crossing; Cogent is currently be prioritised higher and outbound routes are proving more stable.
- Update (22:34): The majority of inbound routes over the selected carriers are now flowing via Cogent and all external health checks are reporting healthy.
<!-- more -->
Post-Mortem
Our report from the incident is as follows.
IssuePacket loss affecting a very small number users worldwide, resulting in slow page load times, or no response.
Underlying causeFibre break within Level 3 / Global Crossing’s core infrastructure. Our other carriers were unaffected, so only those ISPs peering with Level 3 saw issues.SymptomsUp to 100% packet loss from a limited number of national and international ISPs where traffic was coming in via Level 3.ResolutionWe temporarily stopped using L3/GBLX and prioritised our routes over Cogent until L3/GBLX had put repairs in place. We received confirmation that at 3:00am the broken fibre had been fixed, and routes began to flow back over L3/GBLX once again.
