We have become aware of core network issue with one of our providers. Updates to follow.
Update (14:10): The fault has been identified with a single provider, changes are being made to take this offline and to use an alternate until such time that it is fixed.
Update (14:20): The affected provider has been isolated and temporarily dropped from our network and connectivity is restored. Updates to follow when this provider is restored again.
Our report from the incident is as follows.
Partial packet loss affecting global connectivity.
One of our carriers saw a large burst of traffic within their network resulting in a loss of connectivity.
Our internal and external monitoring probes immediately reported a fault.
No active steps needed to be taken. Connectivity gracefully fell over to another carrier. A manual de-preference of that carrier was added to prevent its use until connectivity continuity was restored.