Thursday 1st March 2012

Network 27/02/2012 downtime explaination

Our report from the incident on 27/02/2012 is as follows.
 
Issue

DDOS attack to our transit provider’s network

Underlying cause

External high volume attack from multiple sources targeting a customer subnet

Symptoms

Intermittent loss of service on multiple subnets

Resolution

  1. 9:31pm 27/02/2012 the network monitoring and noc team saw a sustained DDOS attack to the network or around 3-4Gbit per second from around 2-3k of hosts. Traffic was received over all 4 carriers from both sites
  2. 9:41pm port security violations limits were hit on our one of our carrier upstreams which took one of the carriers offline increasing the load on the remaining carriers   
  3. 10:05pm affected IP subnet block was identified and network engineers began null routing the affected subnet from the network core
  4. 10:30pm Delta house network connectivity was restored
  5. 10:41pm partial traffic restored in Reynolds house network
  6. 10:45pm full network was restored and DDOS traffic was being held back by the border routers and full customer services restored
  7. 2:30am amended border routers to further drop packets from the attack.

From the information gathered so far the evidence points to a single attack to one customer.

The team are still looking through logs and progressing the incident with the relevant authorities and further measures are currently being invoked to reduce such attacks in future.