Past Incidents

We use two forms of monitoring, Pingdom (an external service) and our own monitoring platform (also external).

Within the last 15 minutes, we have received several Pingdom notifications reporting connectivity dropping and immediately coming back up. However, this does not correspond with our own monitoring reports.

Both Pingdom’s monitoring service and Pingdom’s FPT are showing strange results - however, other 3rd party services are reporting no issues.

At the moment, we are investigating what is going on, but it looks to be an issue with Pingdom rather than our connectivity. Enquiries are under way.

Our report from the incident on 27/02/2012 is as follows.
 
Issue

DDOS attack to our transit provider’s network

Underlying cause

External high volume attack from multiple sources targeting a customer subnet

Symptoms

Intermittent loss of service on multiple subnets

Resolution

1. 9:31pm 27/02/2012 the network monitoring and noc team saw a sustained DDOS attack to the network or around 3-4Gbit per second from around 2-3k of hosts. Traffic was received over all 4 carriers from both sites
2. 9:41pm port security violations limits were hit on our one of our carrier upstreams which took one of the carriers offline increasing the load on the remaining carriers   
3. 10:05pm affected IP subnet block was identified and network engineers began null routing the affected subnet from the network core
4. 10:30pm Delta house network connectivity was restored
5. 10:41pm partial traffic restored in Reynolds house network
6. 10:45pm full network was restored and DDOS traffic was being held back by the border routers and full customer services restored
7. 2:30am amended border routers to further drop packets from the attack.

From the information gathered so far the evidence points to a single attack to one customer.

The team are still looking through logs and progressing the incident with the relevant authorities and further measures are currently being invoked to reduce such attacks in future.