Call us: +44 (0)330 043 1625
All systems are operational

Past Incidents

Monday 17th June 2013

No incidents reported

Sunday 16th June 2013

No incidents reported

Saturday 15th June 2013

No incidents reported

Friday 14th June 2013

Network Planned Maintenance

Brief loss of connectivity for all customers although it should be largely un-noticable

Estimated Downtime

Up to 120 seconds in small bursts across the course of the day

Actions

Following some repeated firmware issues on access switches, we are systematically replacing all access switches in every rack in Joule House.

As we run a redundant switching network, we will begin by powering down the B switching network, and replacing cabling and switches. Then powering it back up. Then repeating the same process on the A switching network.

When the switching networks go down and back up (4 times in total), it will cause a spanning tree calculation, which will result in momentary downtime (<30s).

Network Some packet loss over international routes

We have a number of customers reporting some packet loss whilst accessing their servers from outside of the UK.

  • Update (22:07): This appears to be an issue with one of our upstream carriers. Updates to follow.
  • Update (22:20): Routes have been de-preferenced over L3 and Global Crossing;  Cogent is currently be prioritised higher and outbound routes are proving more stable.
  • Update (22:34): The majority of inbound routes over the selected carriers are now flowing via Cogent and all external health checks are reporting healthy.

<!-- more -->

Post-Mortem

Our report from the incident is as follows.

IssuePacket loss affecting a very small number users worldwide, resulting in slow page load times, or no response.

Underlying causeFibre break within Level 3 / Global Crossing’s core infrastructure. Our other carriers were unaffected, so only those ISPs peering with Level 3 saw issues.SymptomsUp to 100% packet loss from a limited number of national and international ISPs where traffic was coming in via Level 3.ResolutionWe temporarily stopped using L3/GBLX and prioritised our routes over Cogent until L3/GBLX had put repairs in place. We received confirmation that at 3:00am the broken fibre had been fixed, and routes began to flow back over L3/GBLX once again.

Thursday 13th June 2013

No incidents reported

Wednesday 12th June 2013

No incidents reported

Tuesday 11th June 2013

No incidents reported