'A miss on our side': Telstra outage caused by known issue

'A miss on our side': Telstra outage caused by known issue

Telstra failed to prioritise fixing known vulnerabilities in its systems despite being warned about them months before they triggered a major nationwide outage.An independent report investigating the July 8 outage, which affected about 45 per cent of all calls and data sessions on Telstra's mobile network and saw hundreds of Triple Zero (000) calls unable to connect, was released by the telco yesterday.The review by Technology Audit Partners (TAP) found that a lack of oversight and ownership of key elements of the network contributed to the outage, including that two of the telco's key engineers were on mandatory leave at the time it occurred.It found that the outage began at 2:50am — nearly an hour earlier than Telstra had previously disclosed — and was triggered after a faulty power supply was replaced that then reset a GPS card date to 2006, which then flowed through the rest of the mobile network.The review said the alarms that would usually alert staff to the error were only monitored during business hours by a "limited number" of people, and a lack of "knowledgeable staff" contributed to the telco taking "several hours to determine the root cause of the outage"."There were several reasons for this, primarily due to insufficient clarity of responsibility regarding NTP (Network Time Protocol) ownership and support, lack of visibility to the planned event to replace the Melbourne chassis that night, as well as a lack of understanding of the alarms that were the earliest indicators of a problem in the network," it said.The report also noted that the telco did not treat its timing system as a priority, nor did it treat it as a "high risk function" of the network.Telstra chief executive Vicki Brady said the company accepts the findings of the report and is working through its recommendations."The cause … was consistent with what we said at the time, so it was not due to infrastructure failing, it was due to an undocumented design change in October, and it was due to a software update having not been done on a GPS card," she told a media briefing."The overarching finding is that we did not prioritise our timing system inside our networks at the highest level as a critical capability, which is exactly what it should have been."That is a miss on our side. We should have had it prioritised at the highest level."The company has already reallocated resourcing to the team to improve its capability, Ms Brady added.The report's findings come after the ABC revealed in the days after the July outage that Telstra had been warned for months by federal government agencies and academics that it was vulnerable to the type of error it experienced.In 2024, Australia's government-run Cyber and Infrastructure Security Centre put out a public alert about how "Australia's critical infrastructure increasingly relies on the delivery of positioning, navigation and timing".In October last year, another alert from the centre singled out "critical communications assets" as being dependent on these timekeeping systems, including space-based satellites. It reiterated the legal obligations to ensure providers had backups if those systems failed.A review into Telstra's outage found it began almost an hour earlier than previously disclosed. (ABC News: Danielle Bonica)That same month, Telstra staff were working on addressing a timing issue that resulted in one of its clocks in Melbourne losing connectivity multiple times a day to another clock in Sydney, triggering NTP alarms."A 'Network at Risk' ticket should have been raised to find the real root cause of the frequent loss of the Sydney source and restore the NTP server back [to] the intended design but this did not happen," the report said."At the time of the October 2025 incident, the personnel involved did not recognise the meaning of Melbourne NTP at stratum 5 … this anomaly was observed but not noted and no problem ticket was created for further investigation."In hindsight, Ms Brady admitted that there "could have been more curiosity" about the incident."Particularly looking back in October last year when we did put a design change in to address an issue that didn't end up being documented and was not followed up," she said."That is a lesson we've got to take."Telstra chief executive Vicki Brady at a press conference following a nationwide Telstra outage 10 July 2026. (ABC News: Julia Andre)Clear 'preventable failures'The TAP report also found that there were eight additional Triple Zero calls that were unable to connect during the July outage.Telstra had previously told a Senate inquiry examining its outage that 604 Triple Zero calls failed.Of the additional eight missed calls, five did not need assistance or were "misdials", one successfully "camped on" via a different network, one connected after making a subsequent call, and one caller that could not be identified, even after police visited the premises.Ms Brady said those eight calls had since received the required welfare checks, and stressed that there had been "no adverse outcomes" reported."We know how important Triple Zero is, and that's why any time there is an issue we will focus very quickly on making sure we make the changes needed to ensure our processes are even more resilient," Ms Brady said.In a statement, Communications Minister Anika Wells said she expected Telstra to act on "the findings and recommendations without delay"."This was an outage that should not have happened, and it caused an extraordinary disruption across the country which impacted people, businesses and transport," she said."What this report has unearthed is concerning and clearly shows there have been preventable failures."Australians rightly expect to reach Triple Zero when they need it most, and telcos are legally obligated to ensure that connection occurs."The industry must learn from its mistakes."The Australian Communications and Media Authority is still investigating the outage, which could see Telstra fined up to $30 million.Ms Brady said it was "too early" and the telco had not yet made any budget allowance for any potential financial penalty.She also confirmed that "a little over" 30,000 customers have shared in roughly $1 million worth of compensation, with most of the compensated customers receiving a $15 credit, processed typically within three days of lodgement.For small businesses, the amounts paid have been determined on a case-by-case basis with requirements on some to show proof of business impact, Ms Brady said.

Original Source

Read the full article at Abc →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.