Broadband & Leased Line Disruption

Incident Report for WNICares

Postmortem

We have received an official response from our carrier:
At 4:40AM on Friday 11th of February 2022, a fibre between THN & LD8 dropped during a non-service impacting audit on a fibre splice case near Telehouse. As this is part of a resilient network with a geographically diverse fibre to THW, traffic was re-routed autonomously and a high priority repair was scheduled.

While both these fibres are in separate and differently routed ducts, including different entry points within LD8 building, both connect to the same Zayo rack within LD8. At 11AM, an engineer was dispatched and was told to only work on the LD8 to THN fibre. The engineer needed to isolate the fibre before repairing, however at 12:10, they unplugged the LD8 to THW fibre accidently. The engineer noticed the mistake and reconnected within seconds.

This caused a short blip to leased line customers but was long enough to time out PPP sessions for all broadband xDSL subscribers. This took around 10 - 15 minutes to resolve. A contractor found a bend in the THN to LD8 fibre and repaired it by 12:52 on Saturday afternoon. We apologise for the inconvenience caused by this disruption. We are actively looking to introduce further alternative routes into our network to ensure that this cannot happen again.

Posted Feb 16, 2022 - 12:51 GMT

Resolved

This incident has been resolved.

Posted Feb 11, 2022 - 21:28 GMT

Monitoring

We have been made aware of a fibre outage between LD8 & Telehouse North. This was caused by overnight maintenance by one of our fibre carriers and is on a fibre not used by our service. We believe a network route may have been modified to bypass the broken fibre which caused our network PoP to drop out. We're monitoring closely and awaiting to hear back from our carriers engineers.

Posted Feb 11, 2022 - 13:10 GMT

Identified

Services are now restored following a router encountering issues at Equinix LD8 (HEX). Leased lines and broadband sessions not routed through LD8 will not have been affected. Affected leased lines saw around 60 seconds of downtime, broadband sessions saw around 4 minutes of downtime. We will continue to investigate the cause and provide more information shortly.

Posted Feb 11, 2022 - 12:59 GMT

Investigating

We are aware that some broadband and leased lines are currently encountering issues connecting to the internet. Engineers are investigating the cause. We apologise for any disruption caused.

Posted Feb 11, 2022 - 12:32 GMT

This incident affected: xDSL Broadband (SOGEA, SOGFAST) and Fibre Optic Broadband (Community FTTP).