Review of Optus emergency calls outage finds process, accountability gaps in Singtel unit

[SINGAPORE] An independent review into Optus’ major September outage has uncovered gaps in process, accountability, and escalation and information protocols.

The Sep 18 incident when Triple Zero calls – emergency numbers used for life-threatening situations or emergencies – were blocked due to an outage following a network upgrade led to two deaths, based on the review.

The report, which Optus’ board released on Thursday (Dec 18), found that “challenges in Optus’ culture” affected decision-making and response times.

It also reinforced the need for “further work to address the issues facing emergency call services and device behaviour, and the importance of industry-wide collaboration”, said Optus, a Singtel subsidiary and Australia’s second-largest telco provider.

Optus said that its board accepted all 21 recommendations put forth by the report and agreed to “move swiftly with their implementation”.

The recommendations build on the multi-year transformation plans that are under way and changes introduced to address shortcomings identified during the initial response to the incident, Optus added.

The independent review conducted a comprehensive examination of the causes of the incident that led to a failure of Triple Zero services in South Australia, Western Australia, Northern Territory and part of New South Wales. 

It was led by Dr Kerry Schott, a former chair for rail-related panels and reviews. She previously led a short review into incident management at Sydney Trains for the New South Wales Government Cabinet Office. 

What caused the outage?

First, the review found that mistakes made during the planning and implementing of a firewall upgrade led to the disconnection of Triple Zero calls from the network shortly after midnight. The services were restored only more than 14 hours later.

SEE ALSO

At least 10 actions were not properly executed, the report noted. Incorrect instructions about the upgrade’s implementation were provided, which appeared to be due to a lack of attention by firewall network engineers.

Second, the response was severely delayed as Optus failed to recognise the problem for around 13 hours despite receiving two earlier alerts, which did not lead to any corrective action.

Had the alerts received “more than cursory attention”, this would have allowed a correction about 30 minutes into the outage, the report said.

“The fact that alerts can be overlooked because they are related to ongoing equipment upgrade work is astounding, when the reason for those alerts may be unanticipated problems caused by that work,” it added.

Third, only 150 of the 605 call attempts to reach emergency call services were successfully connected, with the remaining 75 per cent not going through. This was despite how emergency calls are supposed to automatically divert to other available networks in the event of a network failure, the report stated.

Shares of Singtel ended Wednesday 0.2 per cent or S$0.01 lower at S$4.55.