Outage - Connectivity Issues
Incident Report for Beam Internet
Postmortem

Post Outage Event Report

Outage Overview:

At 9:30am CDST on the 11th of October our technicians detected a network outage causing heavily intermittent connectivity to most customers across the beam network.

Initial Response:

Our technical team responded and began investigation approximately 1 minute after receiving alerts. After attempting to restore stability remotely unsuccessfully we opted to attend our core network locations in Adelaide to isolate and restore segments of the network in an orderly fashion.

Initial Restoration Time frame:

98% of the network was restored by 3.30pm with an isolated pocket of 30 Residential and Business customers offline in the Kapunda area until 11:30pm. Whilst bringing the network back up to isolate the fault there was intermittent connectivity varying from 30 seconds to 30 minutes to already restored services until we narrowed down the fault to a single location.

What is Beam doing to ensure major outages are limited going forwards?

After we suffered the major outage in August, we identified and inherent bug in the network management protocol for a particular hardware vendor. The work around for the August outage was to disable the management protocol across the whole network to restore stability. Post the August outage, the decision was made to transition hardware vendors for all critical components in our network. The new hardware was in the process of being physically installed over the last week ready for network integration over the next 5 weeks. Unfortunately, the same vendors equipment we are in the process of replacing yet again let us down with a different software bug that caused the outage on the 11/10. The Implementation of the new hardware is being staged and completed between 3am and 6am in order to minimise any interruption with notifications sent out in advance to advise of the days we will be completing this. A hazard notification will also be published at status.beaminternet.com.au to advise when we will be completing work that has minimal risk of causing downtime.

Customer Alerting Platform

After our August Outage we implemented an alerting and status platform. The platform can be reached at http://status.beaminternet.com.au . From the status page, you will be able to see any current or past outages as well as subscribe for SMS (recommended) and email updates. Going forwards, this will be our primary communication method for outages as well as updates published to our Facebook page. If you need to get in contact with us during an outage the best method is to send us an email at support@beaminternet.com.au or send a message to our Facebook page. We make a huge effort to respond to ALL customer comments, messages and emails with staff working into the night to respond. Shortly we will also have a beam SMS number where you will be able to send us an SMS message to ask any questions or seek support.

SLA Rebates

Corporate customers are entitled to SLA Rebates. These will be calculated and applied over the coming week.

We apologies for any inconvenience caused and would like to thank our customers for their understanding. We are working our hardest alongside multiple consulting firms the remove the affected vendors hardware from our core network ASAP. If you have any further queries, questions or comments these can be emailed to support@beaminternet.com.au and we will respond early next week.

Regards,

Beam Internet

Posted Oct 12, 2019 - 16:32 ACDT

Resolved
This incident has been resolved.
Posted Oct 12, 2019 - 16:30 ACDT
Update
We are continuing to monitor for any further issues.
Posted Oct 12, 2019 - 16:29 ACDT
Update
Services have been restored, we will monitor over the coming days.
Posted Oct 11, 2019 - 23:58 ACDT
Update
Northern Barossa remains affected by the service outage with engineers completing onsite tower work. All other services have been restored.
Posted Oct 11, 2019 - 17:15 ACDT
Update
98% of services have been restored, we are stabilising connections and will continue to bring services online until fully restored.
Posted Oct 11, 2019 - 15:23 ACDT
Monitoring
We are isolating and bringing up segments of the network in an orderly fashion.
Posted Oct 11, 2019 - 14:00 ACDT
Update
We have identified the fault and have to travel to the data center to resolve.
Posted Oct 11, 2019 - 12:32 ACDT
Identified
We have narrowed the issue down to a core switch and we are working to resolve the issue.
Posted Oct 11, 2019 - 11:12 ACDT
Investigating
Hi, We're currently investigating a network outage affecting a large number of customers. We will post updates as they become available.

Regards,

Beam Internet
Posted Oct 11, 2019 - 10:46 ACDT
This incident affected: Area - Northern Barossa, Area - Mid North/ goyderCONNECT, Area - Kangaroo Island, Area - Fleurieu peninsula, Area - Murray Bridge, Mail Server - mail.beaminternet.com.au, Corporate Data/Datacentre Services, and Area - Southern Barossa.