Post Outage Event Report
At 9:30am CDST on the 11th of October our technicians detected a network outage causing heavily intermittent connectivity to most customers across the beam network.
Our technical team responded and began investigation approximately 1 minute after receiving alerts. After attempting to restore stability remotely unsuccessfully we opted to attend our core network locations in Adelaide to isolate and restore segments of the network in an orderly fashion.
Initial Restoration Time frame:
98% of the network was restored by 3.30pm with an isolated pocket of 30 Residential and Business customers offline in the Kapunda area until 11:30pm. Whilst bringing the network back up to isolate the fault there was intermittent connectivity varying from 30 seconds to 30 minutes to already restored services until we narrowed down the fault to a single location.
What is Beam doing to ensure major outages are limited going forwards?
After we suffered the major outage in August, we identified and inherent bug in the network management protocol for a particular hardware vendor. The work around for the August outage was to disable the management protocol across the whole network to restore stability. Post the August outage, the decision was made to transition hardware vendors for all critical components in our network. The new hardware was in the process of being physically installed over the last week ready for network integration over the next 5 weeks. Unfortunately, the same vendors equipment we are in the process of replacing yet again let us down with a different software bug that caused the outage on the 11/10. The Implementation of the new hardware is being staged and completed between 3am and 6am in order to minimise any interruption with notifications sent out in advance to advise of the days we will be completing this. A hazard notification will also be published at status.beaminternet.com.au to advise when we will be completing work that has minimal risk of causing downtime.
Customer Alerting Platform
After our August Outage we implemented an alerting and status platform. The platform can be reached at http://status.beaminternet.com.au . From the status page, you will be able to see any current or past outages as well as subscribe for SMS (recommended) and email updates. Going forwards, this will be our primary communication method for outages as well as updates published to our Facebook page. If you need to get in contact with us during an outage the best method is to send us an email at firstname.lastname@example.org or send a message to our Facebook page. We make a huge effort to respond to ALL customer comments, messages and emails with staff working into the night to respond. Shortly we will also have a beam SMS number where you will be able to send us an SMS message to ask any questions or seek support.
Corporate customers are entitled to SLA Rebates. These will be calculated and applied over the coming week.
We apologies for any inconvenience caused and would like to thank our customers for their understanding. We are working our hardest alongside multiple consulting firms the remove the affected vendors hardware from our core network ASAP. If you have any further queries, questions or comments these can be emailed to email@example.com and we will respond early next week.