Experiencing packet loss within network core.
Our report from the incident is as follows.
Significant packet loss, effecting all servers at our Joule House location, causing a total service loss to all servers.
The duration was 97 minutes.
Currently under investigation with Juniper TAC to identify and isolate the issue. It appears to be a repeat incident whereby flooding within a single access switch caused significant control plane CPU consumption within other network devices.
Our external monitoring probes immediately reported the fault. End users will have noticed the issue as it had an effect on overall service.
Whilst the sympoms have been resolved, we believe the underlying issue to still be present and the result of firmware bug. The issue has been elevated to the vendor for in depth analysis and urgent review.
Our network and transit team is continuing to investigate, replicate and resolve this issue in an isolated environment in parallel with Juniper TACs efforts, so that a swift resolution can be reached.
The network status will remain as high until we are confident of a permanent resolution.