We have become aware of core network issue with one of our providers. Updates to follow.
- Update (14:10): The fault has been identified with a single provider, changes are being made to take this offline and to use an alternate until such time that it is fixed.
- Update (14:20): The affected provider has been isolated and temporarily dropped from our network and connectivity is restored. Updates to follow when this provider is restored again.
Post-Mortem
Our report from the incident is as follows.
Issue
Partial packet loss affecting global connectivity.
Outage Length
5 minutes.
Underlying cause
One of our carriers saw a large burst of traffic within their network resulting in a loss of connectivity.
Symptoms
Our internal and external monitoring probes immediately reported a fault.
Resolution
No active steps needed to be taken. Connectivity gracefully fell over to another carrier. A manual de-preference of that carrier was added to prevent its use until connectivity continuity was restored.
