Confirmed connectivity issues reported.
- Update (15:10): An engineer is looking into this now, updates to follow.
- Update (15:25): A upstream transit provider appears to be suffering some packet loss. We have re-routed traffic around this provider and appear to see normal performance. We will continue to monitor.
Post-Mortem
Our report from the incident is as follows.
Issue
Very brief loss of connectivity affecting around 20% of total traffic.
Outage Length
The duration was <4 minutes.
Underlying cause
An upstream provider suffered a router reboot, which caused traffic to be dropped both in/out of our network, for the small volume of traffic that passes via that provider.
Symptoms
Our monitoring probes immediately reported the incident. Customers would have experienced slow page load times through to a completely inaccessible site.
Resolution
The issue was resolved by temporarily dropping our connection to the respective provider and re-routing traffic over our other transit providers. The issue immediately subsided and normal traffic flows resumed. The upstream provider since has restored a previous configuration on their device and has it running stable again, we have since restored our connection to them and are operating at 100% capacity.
