Past Incidents

sms-jay

Estimated Downtime

20 minutes

Actions

Since the restart, I/O wait has been higher than expected on sms-jay. We feel that a kernel update would be a wise manoeuvre given the age of the current release. 

• Update (11:26): The new kernel is installed and all tests are complete.

sms-jay has stopped responding.

• Update (11:30): sms-jay had restarted (cause currently unknown), upon reboot, as the server had been up for over 347 days without a reboot, a file system check must be carried out. 
  
• Update (11:32): Estimated completion time is approximately another 30 minutes.
• Update (11:39): Progress is currently at 87.8%
• Update (11:43): Automatic fsck failed, a manual fsck is now being run and overseen to correct file system errors.
• Update (11:51): The manual fsck is complete and the errors have been fixed. The server is now rebooting and should be up shortly.
• Update (11:56): The server is now up and operational and web services are being started.
• Update (11:59): Web services are operational, load will be high (and thus the server will be slower than usual) whilst the RAID array rebuilds (estimated 8 hours).