Intermittent packet loss.
- Update (00:27): One of the two core routers at Joule House is failing to respond, causing a network wide disruption.
- Update (00:29): The affected device is being power cycled.
- Update (00:51): The router has successfully powered back up.
Post-Mortem
Our report from the incident is as follows.
Issue
Minor intermittent packet loss, causing some packets to be dropped. This had negligible effect on servers at Joule House, the effects may not have even been noticeable to end users.
Outage Length
The intermittent packet loss duration was 2 minutes.
Underlying cause
The load on a core router suddenly increased with no known cause.
Symptoms
Our external monitoring probes immediately reported the fault. End users may not have noticed the issue as it had near negligible effect on overall service.
Resolution
As the router was non-responsive to input, it was deemed necessary to restart the device. Seamless failover completed to the other router whilst the device was powered down. There was no loss of service during failover.
The router was powered back on, underwent a consistency check and was added back into the routing pool.
