Experiencing some packet loss to some internal routes.
Our report from the incident is as follows.
A very small number of IP subnets encountered a routing loop and some customers servers were inaccessible.
The duration was 23 minutes.
The affected IP ranges are those we carry as a legacy from a historic provider - they do not form part of our multihomed BGP network and as such are subject to possible outages should the transit provider supplying them encouter problems.
The routing loop occured because of a configuration change on the upstream providers network.
Access to servers behind the affected subnets was impossible, a routing loop was visible on a traceroute.
The provider reverted the change and service was immediately restored.
The long-term plan is to renumber all these IP addresses into our own, so that they can be announced over our resilient, mutlihomed BGP network. This task is already underway and customers will be contacted soon to arrange for changeover dates.