Organized, cloud-accessible procedures for data center recovery made it possible to resuscitate the data center of an elite hospital system, and more than 100 of its sister units and component clinics, without serious consequences.
Houston Methodist Hospital is ranked by U.S. News & World Report as the best hospital in Texas and 15th in the nation. It provides health care from eight hospitals and 100 clinics throughout Houston, all networked from a central location on the Texas Medical Center campus.
A hospital must be prepared to offer state-of-the-art, split-second care at any moment. Its patients, doctors, and employees rely on an uninterrupted stream of reliable data for everything from operating cutting-edge medical devices during lifesaving procedures to scheduling, filing, reporting, and tracking employee hours. Any disruption to the transmission of this data can, at best, slow the ability to provide routine care by not making rooms, equipment, or medical personnel available quickly.
Challenges
Disasters happen
A modern hospital relies heavily on computerization. So late on a Friday afternoon in the fall of 2021, it was a jolt to Gene Gretzer, Disaster Recovery Architect for Houston Methodist Hospital, to receive notification on his device that the hospital’s main data center was on the brink of shutting down because of a power issue. Gretzer knew that when the data center lost power, all of HMH’s hospitals and clinics would be unable to access critical IT functions, except in the most basic way. Phones and internet access could also be affected.
Vital patient information would be difficult to access on a busy commuter Friday night, and perhaps deep into a weekend filled with injury potential from community activities like power-tool projects and amateur sports—not to mention that the data center also held human resources and financial data that would be needed for payroll and new hires for the upcoming week.
HM IT leadership called a yellow alert, meaning all hands were needed on deck, responding and connecting. While Gretzer was busy spooling up the DR recovery process at his desk, the HM IT teams were primarily working from home during the pandemic. Getting them organized to identify the cause, work on the problem, and develop a solution was the first priority.
With the power down, Gretzer had no access to the hospital’s installed internet (both wired and wireless). He quickly turned to his smartphone, using it as a hotspot to connect to the internet. Help was on the way.
Solutions
The cure for disaster
Years before, HMH had begun to plan for any sort of disaster that might affect the hospital. Two critical flooding issues had caused damage in the past, including the aftermath of Tropical Storm Allison in 2001, after which it took HMH five weeks to fully reopen. Knowing that the hospital would need to recover from another significant incident, HMH had built a disaster recovery plan with a variety of word processing documents and spreadsheets. Only recently had Gretzer led HMH to migrate all those application and infrastructure plans into Infinite Blue’s automated BC in the Cloud software, making it the hospital’s central recovery resource. Gretzer’s team had also conducted quarterly testing of the program, then used After Action Reports to identify necessary changes to the plan. In addition, they trained several times a year specifically on how to react if a power outage ever occurred. Everybody knew what to do, where to go, and how to start the recovery process.
This proved to be an advantage because, despite most of the team working from home and theoretically isolated, they were still able to easily log in to BC in the Cloud. With all of HMH’s information stored in the cloud by the application, that meant that everyone—no matter where they were, even Gretzer sitting in his dark medical center office—was able to access all the information they needed right when they needed it most.
Recovery efforts, status, and updates were defined and tested, but BC in the Cloud’s mobile accessibility kept the remote workforce updated in real time as much as possible. The task-based recovery plan outlined in BC in the Cloud months before was followed step by step. The team was able to determine the best course of action and get the recovery process underway.
They quickly determined that their network, the base system of their infrastructure, was completely down. Getting that restored was the first priority, and they knew they would have to bring in a few network engineers to get it going. With most of the team working from home, everyone joined different communication groups based on their role in the recovery process so that they were kept updated in real time as much as possible.
Based in order of significance—as determined by the software’s relational database and user inputs that ranked and scored the importance of each—systems were restored one at a time. In just two hours, the critical systems were recovered. It took just six hours to completely recover all clinical systems from the outage.
Results
Lessons learned and what’s next
The system at HMH was designed to be automated in order to protect the data center. This automation was part of the problem, though, because there were steps in the shutdown of the data center that the team could not control. This actually prolonged the recovery process. A new power panel system has been installed to prevent this from happening again.
This was also the first real incident with everyone working remotely, which Gretzer learned had major advantages. First among these advantages was that there were no issues connecting to BC in the Cloud with employees’ own Wi-Fi and power, away from the facility. But, because they were all remote, they needed to focus on communication even more than usual to keep everyone aware of the status of the recovery processes.
Ultimately, using BC in the Cloud in a real emergency and not a test provided HMH with new information it could leverage going forward. It also enabled the hospital to enact new procedures, which—written in advance—hastened the ability of the HM IT team to immediately get to work putting the extensive HMH computer system back on track.
