“We had tried everything we could to get performance improvements, but we didn’t think any other big gains were possible. I realized that if Gary was going to kick us off his servers because we couldn’t improve performance, then we needed to find other servers, or get new ones, and I didn’t think Gary would be willing to help us with either. So I resorted to the only option I could think of.”
David paused to drink his coffee. He glanced up to see that even though Mike and Gene had heard the story before, they were just as captivated as Sean.
“I decided there was no argument I could make that was compelling enough to change Gary’s mind. So I decided to let ELOPe do it. ELOPe was already running on the AvoMail servers, configured to ignore everything that wasn’t our test emails. I changed the configuration so that it would filter all company emails looking for any mention of the project. I configured it for the same settings we used for performance testing: The user interface was off, so the email sender would never see the modifications being made to the email.”
“In performance testing mode, we turn logging off, so we can get a realistic idea of real-world performance,” Mike said. “This also meant we didn’t have any log of the changes that ELOPe was making, which turns out to be important.”
“That’s right. Thanks, Mike,” David continued. “But that wasn’t the only change I made. To speed up analysis, and optimize performance on a handful of specific criteria, we already had some hand-tuned rules implemented in a lookup table that allow us to shortcut the full analysis process. If you remember that example I used of an email requesting resources for a project: rather than analyze that email and every other email requesting project resources, we cache and hand-tune some suggested language. Well, I added the subject ‘ELOPe’ to that table, and adjusted parameters to the algorithm to allow ELOPe the widest possible discretion in changing the email language to optimize the results for a positive outcome.”
“What we’ve guessed,” Mike jumped in again, “is that when David did this, the parameters he chose, in combination with the fact that the system is in performance test mode, allowed ELOPe not just to modify existing emails, but to autonomously generate emails on its own. It’s part of the test suite. Does this make sense?”
“Sure,” Sean said, nodding. “During testing, you don’t want people to have to hand-edit an email, and accept the changes interactively. You just want to batch process a bunch of emails. But why does it have the ability to generate emails on its own?”
“For one, it allowed us to test the natural language generation,” David answered. “Early on, the email analysis and language generation were two separate aspects of the project. The members of the team who were working on the code to generate natural sounding language wrote an email generator, so that they could independently test the ability of the system to mimic the way a person normally writes. We had hundreds of test subjects who rated emails, some of which were written by an actual person they knew, and some were generated by the system pretending to be that person. Our goal at the time was that ninety percent of ELOPe generated emails would pass as being written by the purported sender.”
“You met that goal?”
“Yes,” confirmed Mike. “Now ELOPe exceeds ninety-eight percent. In fact, on April First, there were endless practical jokes on the team as everyone played with the test system to generate prank emails. Both David and I fell for it.”
David smiled at the memory.
“Getting back to the problem,” Gene said, “unfortunately we now have evidence that ELOPe is manipulating others.”
“Yes.” David nodded, tearing himself away from the memory of happier times. “I was puzzled when the project was allocated five thousand servers on a priority exception, just a day or so after I tweaked the settings and turned on ELOPe company wide. Then shortly after that, we were assigned a team of contractors who specialized in high performance optimization, which was something we had chatted about informally, but never proposed. But that alone wasn’t what really convinced me.” He turned to Mike.
“The first clear evidence I saw,” Mike said, “occurred when I received an email, purportedly from my mother, telling me that my father had been admitted to the hospital for a heart attack. I flew to Wisconsin, only to find out that my mother never sent such an email.”
“What does that have to do with anything?” Sean asked, looking puzzled.
“ELOPe was getting Mike out of the way,” David said. “I had become nervous about what I had done, and why we were suddenly getting all these resources. I wanted to turn off ELOPe. I sent an email to Mike one evening asking for his help, since only he had the experience and permission to live patch the servers.”
“However, I never received that email,” Mike said. “Instead, I received an email that sent me more than a thousand miles away on a wild goose chase, and thanks to the winter storm, it was a week before I got back. When I did, I found that my access to the ELOPe project had been removed, and David was on vacation, off the grid in New Mexico.”
“Did ELOPe send you to New Mexico?” Sean asked, one eyebrow raised.
“No, no, that was a planned vacation we do every year. When I got back from vacation, Mike and I discussed what had happened to him. We also discovered that my access to the ELOPe code had been turned off as well. The first thing we did was try to find out who removed both Mike and I from the project access list,” David went on. “That investigation revealed the next big clue, which was an email sent to the Internal Tools department, which implied that you, Sean, were endorsing a request to have them implement an email-to-web bridge. Which I am guessing, you never heard of…”
“No, absolutely not,” Sean replied, shaking his head. “That could open up all kinds of security holes.”
“Meanwhile, just before the holiday vacation, I had found suspicious buying patterns across several departments,” Gene said. “What I found particularly unusual was how the purchases came within a single penny of the budget limits. In all my years auditing the purchasing department, I’ve never seen anything like it. Someone or something was making coordinated purchases across department budgets. They knew to avoid hitting the budget limits which would trigger reviews, but it seemed as though they never thought that leaving only a single penny in dozens of budgets would be suspicious. At first I thought some sort of fraud was occurring. I tracked down all the purchase orders. The line items included massive quantities of servers, which turned out to all be directly or indirectly allocated to ELOPe, contracts with external vendors for temporary software programmers, and extra parts for the offshore data centers, including auxiliary communication systems, backup power supplies and several particularly large line items related to defensive weaponized robots for the offshore data centers. I discussed the questionable items with procurement, and they told me that since they were in line with the types of purchases the departments usually made, they had approved everything.”
“You’re saying that ELOPe somehow made these purchases?”
“Exactly. As strange as that may seem.” Gene pulled out a sheaf of paper. “As part of my job, I can audit other people’s email accounts. And what I saw was that while David and Gary Mitchell were on vacations, their email accounts were still sending rapid fire emails, using this email to web bridge to direct the procurement department. Looking at the timestamps on the emails, I was able to figure out that it couldn’t be a human. It had to be a computer program.
Sean stared at the paper, frowning.