Our report from the incident on 15/06/2012 is as follows.
Issue
Complete loss of all connectivity to our entire network
Underlying cause
Our upstream transit provider’s switch failed at Reynolds House, Manchester, causing the loss of connectivity on one site feed. Simultaneously, a power breaker failed at Delta House, Manchester, rendering the redundant feed unavailable too.
Symptoms
Complete loss of all connectivity to our entire network
Resolution
A temporary alternative transit feed has been provisioned and routes are now sent via this (around 3 hops more than the previous feed). This is a semi-permanent fix whilst the switch is replaced in Reynolds House. There is expected to be planned maintenance to change the transit back to the primary feed; but nothing has been scheduled yet. The network is now operating as it should be and we apologise for the downtime. Downtime incurred was 180 minutes.
