“We have a service level agreement in place to give Avogadro top preemptive priority. It looks like they must have wanted an emergency backup in case their own data centers were affected.”
Even Jan knew that Avogadro had more computer servers than any other company in the world. “Why would they want to use us? Don’t they have hundreds of their own data centers around the world?”
“Yes, but maybe they were anticipating a problem,” Helena responded. “According to this, we signed the contract with Avogadro just a few weeks ago.”
“So what are we running? Their email servers? Their search engine?” Jan wondered aloud.
“Well, it doesn’t look like we’re running any customer facing applications. If you look at the traffic profile,” and here Helena gestured to the second display. “You can see that the majority of the traffic is outbound. Looking at the ports and addresses, it seems like the Avogadro code is sending a ton of emails, and big ones too. They are getting some emails inbound, but not enough to account for all of their customers. It’s puzzling for sure. Could they be remotely restoring their servers via email?” Helena shook her head at the improbable notion.
She turned to the third computer on her desk, her personal Mac. “Let me see what happens when I visit Avogadro.” She launched two web browser windows, going to the Avogadro search page in one, and her personal Avogadro email account in the other. “Both email and search web servers are returning a not reachable error. That must mean Avogadro has a major outage.”
“What do we do?” Jan asked.
Helena paused and thought for a moment. “The application and traffic is legitimate. Avogadro paid us for top priority, including the ability to preempt anything else we’re running. They wanted this application, whatever it is, to run in the event that they had a major outage at their own data centers. I can’t peek at the actual code or traffic without violating our customer privacy policy. So I think we just babysit it and hope the servers don’t melt down under the load.”
She glanced at the load indicators, which showed that processing load was pegged at a hundred percent. Looking out the glass window of their enclosure, she glanced across the datacenter floor to see that every blinking indicator light on every rack-mount server and every router was a solid red. She’d never seen traffic loads like this.
“Look, I think there’s a few spare racks that aren’t powered up yet,” Helena said, heading for the door. “I’m going to turn on every spare piece of hardware I can find. I want you to go into the admin tool and throttle back any application that isn’t Avogadro. We’ve got to free up some capacity here.”
Helena headed out the door into the main room.
Jan swallowed hard, and sat down in front of Helena’s computer. His hands trembled slightly as he rested them on Helena’s keyboard. He summoned up his courage and got to work.
It was the third day since the attack that took ELOPe down. Across Avogadro, everyone was working around the clock to restore services and data. With no opportunity to alert the company ahead of time to the outage and with communications largely down, the best the Emergency Team could do was to have a point person at each site who had instructions on the proper process to restore computers to known good backups, backups free of ELOPe, and a signed letter giving them authority to oversee the restoration.
There were marketing managers pressed into service removing hard drives from computers, and administrative assistants running backups from USB drives. Towers of cardboard pizza boxes had sprung up through the hallways, like teetering skyscrapers. Employees worked sixteen and even eighteen hour shifts, some even sleeping under desks.
Yesterday, Gary Mitchell had finally shown up after being missing for two weeks. David had heard through the rumor mill that Gary had screamed bloody murder at the travel department. Apparently Gary had been on vacation in Tahiti as planned. On the last day of his vacation, he showed up at the airport for his flight, only to find that he had been bounced to a flight the following day. With the holiday ending, homeward bound vacationers had filled every last seat, and no amount of yelling at travel agents had gotten Gary onto the plane. He returned to his hotel only to find out that his cell phone was dead, and his computer refused to connect to the Avogadro network. When he returned to the airport the following day, his reservation had been moved out two days. And so it went for two weeks until the day after ELOPe had been killed, and only then was Gary finally able to get onto a flight. And of course Gary arrived right into the biggest operations nightmare the company had ever faced.
Hearing the story made David laugh, and even now he found himself thinking about it every couple of hours and smiling.
In a small silo of relative calm and isolation, David and Gene worked together in David’s office. They were part of a small team of people who were carefully monitoring all data traffic for any signs of ELOPe. Coffee cups and food plates were piled high around the room. David had been home twice for showers and clean clothes. The second time he had fallen asleep putting his shoes back on. He couldn’t remember what it was like to not be exhausted. He and Gene had tried to take shifts away from the office, but they both feared that something critical might happen when they were away. Now they took turns taking brief cat naps on the couch they had dragged over from the meeting room.
Christine had been understanding when he had to make the sudden trip to the East Coast. She had been accommodating when he worked sixteen-hour days during the emergency planning of the shutdown. The online gaming company where she worked had its own deadlines and big deals, and she’d pulled many all nighters before new releases, so it was nothing unexpected in their relationship. She had even helped out and brought food over to Sean’s several times. But now her patience with David was starting to run out. There were no more homemade food deliveries.
David was tired of takeout food, and dirty clothes, and his office chair. The chair was like some kind of modern prison cell.
“David! Look at this.”
David tiredly rolled his office chair — damn that chair — over to the small side table where Gene had set himself up to work and peered at Gene’s screen. After so much time watching Gene use only paper records, it felt odd to see the older man using a computer, but for all his talk, Gene was a quick, competent user.
Gene pointed to a heat map on the screen, showing network traffic. With a few clicks, he brought up a list of emails. In addition to scanning for ELOPe itself, David and Mike had written a tool that would look for signs of tampering with the emails. Through a heavily encrypted secondary channel, they sampled emails to see if the originating email sent by one user differed from the received email when it was read by the other user.
David and Gene had spent the morning reviewing records. David found that he had come to appreciate Gene’s distrust of technology, because Gene had an uncanny ability to spot gaps in processes or technology where data could be altered. Gene might distrust technology, but he understood it very well.
Gene manipulated the email list, drilling into the details of the email records. David’s first reaction was to yell “Oh shit, oh shit!” This brought him some puzzled looks through the open door to his office from passing coworkers. Although the members of the Emergency Team were back in the Avogadro offices now that ELOPe had been disabled, most of the employees still didn’t know the truth of what had happened. The official explanation was that a really bad computer virus had infected all of the company’s computers.