Concerns about the cooling system setup in one of the data centers used to host the NHS Foundation Trust’s major healthcare systems at Guy’s and St Thomas were raised in 2018 and were never fully addressed. bottom. Summer 2022 has been revealed.
As Computer Weekly previously reported, two data centers the trust relies on experienced cooling-related technical issues on Tuesday, July 19, 2022.
As confirmed in a 58-page review of the incident published in late January 2023, the extreme temperatures experienced by the UK on that day led to the Trust’s 371 legacy IT systems being used to host Two data centers overheated and malfunctioned.
The two sites, one in Guy’s Hospital and the other in St Thomas’, were designed to act as backups for each other in the event of an IT failure, but on July 19, 2022 At the time, both sites suffered outages. heat wave in England.
The impact of the incident was felt in the months that followed, and recovery was also hampered, according to reports, by an unrelated cyberattack on an external supplier the trust relied on to host its medical records system in August 2022. It has been.
“The Trust declared a major site incident on July 19 and transitioned to implementing a paper-based operating model to support clinical activities,” the report said. “Technical restoration of IT systems took significantly longer than originally anticipated, with several weeks to near-complete restoration. was quelled on September 21, including the management of an unrelated cycle attack on
The document also confirmed that the incident cost the Trust £1.4m in unexpected IT costs. This was because I had to extract the information stored on the servers damaged by the outage with the help of a third-party data recovery service. Create a new cloud-based data backup system.
Events that can be prevented
The report described the data center outage as a “potentially preventable event” and said it was “obvious” that the risk prediction, mitigation and reporting systems the trust had in place were inadequate. increase.
“This represents a failure of the trust’s risk management processes to effectively mitigate the risk of data center failures,” the review said.
Also, while the review did not find a “single major failure” to point to as the root cause, the study suggested a combination of factors that led to the “catastrophic failure” of the trust’s IT systems. says there is.
These factors include the aging technology infrastructure, the “overly” complex nature of data center assets, and “suboptimal cooling systems.”
The timeline of incidents detailed in the report reveals that concerns regarding the cooling system setup at the St. Thomas data center were first reported by a supplier in August 2018. air flow”.
Recommendations were made at the time to move the condenser, but other mitigation measures were put in place, but this change was not enacted.
The timeline showed that a review of Guy’s data center by the same supplier originally suggested that its air handling unit would end its life in “2021/2022”, but this assessment was later amended in February 2022 and extended by a further 12. Moon.
As a result of this evaluation, we requested £195,000 in funding from the Trust to allow us to install the replacement system in March 2022, which had not been approved at the time of the suspension. This funding request was then increased to his £360,000 and approved, with review documents confirmed.
“The trust must never again find itself in a situation where the recovery of core IT systems takes too long to complete as a result of infrastructure failures, cyberattacks or other causes,” the review said. I’m here.
“As a result, the Trust has developed a comprehensive strategic plan backed by appropriate investments to ensure that future computing and data storage requirements are robust, capable of meeting growing demand and able to withstand foreseeable risks. These plans should include regular and thorough testing of system recovery.”
The trust plans to roll out a new electronic medical records system in April 2023, according to the review. This paves the way for the “rationalization and consolidation” of legacy IT systems, which is expected.
In terms of resilience, the report says the Trust “must prepare for the fact that climate change means extreme weather events will occur.” In response, we confirmed that we have sought expert advice on how to ensure our systems can handle such threats in the future.
