It was a Saturday in August 2023 when Pascagoula, Miss.-based Singing River Health System experienced an outage of its electronic health records and related IT systems. "You have to joke about these things as you move past it, but we called it 'Red Saturday,'" Susan Russell, MSN, RN, Singing River's chief nursing officer, told Becker's.
"Basically, all electronic systems had come to a standstill," she said. "We knew that we were undergoing a cyberattack. Our IT team came in, pulled everything offline to stop the infiltration. Everything, even our telephone system, is internet-based, so we lost phone service, intranet, and electronic medical records."
Even though the Singing River had experienced a smaller-scale outage in 2018, they were not prepared for one to span 13 days, as this one did.
Cybersecurity threats are becoming increasingly common and more severe, like the recent Ascension cyberattack or the Change Healthcare cyberattack. While the U.S. government is bolstering its response, ransomware attacks will likely continue on some scale.
Ms. Russell said one of the first things she did was run to an office supply store to bulk-purchase paper, pens, highlighters, binders and more. She raced to coordinate with nursing staff and other clinicians to get text message group chats going for offline communication.
"We didn't know how long we were going to be down, but when it became evident this was going to be more than a short-term outage, we needed to start long-term documentation of the care patients were receiving, the treatments that were going on, and also start the process of rebuilding a downtime record," Ms. Russell said. "Unfortunately, our downtime forms that we used for scheduled outages were also on the internet."
Quickly, Ms. Russell and her team realized that newer nurses were not trained in documenting care via hand-written notes or taking a written physician order, so the team had to deploy its nurse educators to help train as they went.
Pharmacy cabinets were electronically locked, so the pharmacists had to come in and work around the clock as well. There were no billing or chart reminders. Everything was manually recorded for nearly two weeks.
Any hospital employee who was not a clinician and unable to do their normal work due to downed systems were utilized for communication runs between departments, gathering supplies and helping wherever needed.
At the end of it, recording putting all of the handwritten documentation back into electronic systems also required double-checking information and teams that would correctly input it.
"Really, what matters is that initial assessment. When you see people who come in after a disaster — and I live in a hurricane zone — so I know what that's like," Ms. Russell said. "You come in, and you immediately assess the environment and see what do you have to work with? What are your resources? And I would advise everybody to look and see what is still working, and think about how you can capitalize on that."
As far as a checklist, she advises chief nursing officers to:
- Print out several documentation forms before you may not be able to print. Keep them on hand.
- Inventory the resources you do have and can use, like a fax machine or copy machine.
- Regularly train and refresh training of hand-written documentation processes.
- Have more than enough extra 'offline' medical record numbers built up to use as identifiers in case of an emergency lasting multiple days.
- Start to create team workflows and centralized communication "war rooms."
Check, check and triple check
Like Singing River Health system, Atlantic General Hospital's 2023 outage also fell on a weekend, and also lasted for about two weeks. At the time, Mandy Bounds, DNP, RN, had only been the system's chief nursing officer for a few months.
"It was in the middle of the night on a weekend that we were made aware that systems weren't performing, and learned that all of our systems had to be shut down," Dr. Bounds said. "So we went into complete downtime in the early hours of the Sunday morning and opened up our incident command, all of our leadership team, specifically, all the nursing leaders and clinical leaders were on-site, and we began to kind of work through what was going to be a much longer than expected downtime."
The hospital never had to shut down clinical services or cancel a procedure throughout the two-week period, she said. But before getting on the other side of that time frame, it took all hands on deck to keep patient care afloat.
One of the key strategies, in addition to calling in extra nursing support, was working to keep nurse-to-patient ratios lower than normal so they could have the bandwidth to care for patients and not be rushed with manual documentation processes.
"In the first five days, we really made a lot of drastic changes," Dr. Bounds said. "All of our documentation and forms, we replicated the electronic health record to make it easier for our nursing staff to document so that it was consistent with what their usual workflows and practices were. It was just on paper. We actually put together packets for each shift so that the nursing team was ready to go with the documentation forms that they were going to need, their care plans that they were going to need, forms to document medication administration records, and did a lot of prep work so that it was easier on them."
Dr. Bounds said they also used non-clincial staff to support the nursing staff to help with tasks like running to communicate information between departments quickly, getting orders to the lab and imaging, and communicating the other information that is normally communicated via computer systems.
Like Singing River, many of the newer nurses at Atlantic General had to be trained in or refresh training of manual documentation processes.
The most important piece in place, according to Dr. Bounds, was its system of checking and verifying information during the downtime.
"We deployed our quality nurses out, and they did chart reviews on every patient chart every day, to make sure that there wasn't an order that wasn't appropriate for the patient, that information wasn't on a different patient's chart, etc.," Dr. Bounds said. "Checks were being done on documentation to ensure things weren't being missed, like things that would potentially be picked up by an EHR. We also had our charge nurses do another 24 hour double chart check. So again, putting a couple of different layers of safety checks in place. We had our clinical pharmacy team engaged on double checks for medication."
Leadership was also very hands-on during this time, providing support to front-line staff as much as possible, she said.
Learn from others
Singing River Health System and Atlantic General's stories are not unique, but a reminder that these attacks can — and do — take systems down unexpectedly. There is never a good time for a cyberattack, but being prepared for a cyberattack can help smooth the edges that might scrape otherwise.
Following a shorter outage at Claxton-Hepburn Medical Center in Ogdensburg, N.Y., in September 2023, David Ferris, BSN, RN, chief nursing officer, said their team prepares EHR outage binders in advance now, which has been a big help.
"We were able to gain feedback from the other facilities to better prepare our front-line staff for an EHR outage," he told Becker's. "As an organization, we prepared downtime EHR outage binders. The binders included all order sets and downtime forms (documentation forms for every EHR aspect) that would be needed to continue care for our patients. These binders were kept in one central location and were updated as needed for document control to ensure we had the most current information. As an organization, we had learned from downtime drills and unexpected outages, which allowed us to be better prepared."
After the system recovered from the cyberattack, Mr. Ferris said each department was surveyed on how to improve downtime processes in their area. The information was collected for review to possibly implement ahead of the next cyber event to make things run even more smoothly.
"In many ways, paper documentation can often be quicker and our downtime binders were readily available," he said. "In order to minimize the additional workload on nurses, additional staff were deployed to the areas. We were very fortunate that our staff adapted to paper documentation quickly."