“Rocks don’t want anything.”
“Everything alive wants something,” Sydney said.
Damien shook his head. “I didn’t say it was alive. I said it was aware.”
“How can you be aware but not alive?”
“Do you believe in life after death?”
Sydney did not believe in life after death, but in her experience, admitting this could lead to long and complicated discussions in which people seemed to think that since she did not believe in God or the afterlife, there was nothing to stop her from becoming an ax murderer. She was pretty sure that Damien didn’t believe in God—he had a stridently pro-evolution T-shirt that said EVOLUTION How can 100 bazillion antibiotic-resistant bacteria be wrong?—but she wasn’t absolutely certain. “A ghost or a spirit was alive,” she said.
Damien shrugged and looked back at his monitor.
Which meant that Sydney should sit down, so she did.
After a minute Damien looked over her cube wall. His head was right above the Mardi Gras mask hung on her wall. She didn’t particularly want to go to Mardi Gras, which seemed to be mostly about blond girls flashing their tits; she just liked masks.
“I think DMS is aware but not alive,” Damien said.
“I don’t even know what that means,”
“Nobody does,” Damien said. Then he sat back down.
They decided to poke it. Or, rather, Damien did. Sydney pointed out that they didn’t know what it would do if they poked it—it could crash, it could shut down all the electrical systems, it could delete all the pharmaceutical records from the previous year.
“Then we’ll install from backup,” Damien said. “We’ll do it at 1:00 a.m. We’ll send out a system maintenance bulletin.” Hospitals don’t shut down, but they do a lot less at 1:00 a.m.
Sydney said, “But if we install from backup, we’ll be killing it.”
Damien leaned back in his chair. “Ah, the old transporter question.” They were sitting in his cube. His desk was a mess—stacked with papers, binders, a couple of manuals, and the remains of a dinner of Chinese food. “In Star Trek, if I beam you down to the planet, does that mean I have actually killed you and sent an exact replica in your place?”
“Yes,” Sydney said. She was wearing her If You’re Really a Goth, Where Were You When We Sacked Rome? T-shirt because Damien had laughed his ass off when she first wore it. Damien was wearing cut-off sweatpants and yellow flip-flops because even though the office was technically business casual, no one cared what you wore at 1:00 a.m. “Look, what if we shut it down, back it up, and it never comes back to consciousness?” She was thinking about the book The Moon Is a Harsh Mistress where the AI, Mike, is damaged during the war and after that never speaks again to Manny, the main character. Manny calls the secret phone number he has for Mike, but when he does, there is only silence. She’d read it when she was thirteen, and it had haunted her. She told herself that eventually Mike “woke up” again and called Manny.
“Why do you think it’s conscious?” Damien said.
“Why do I think you’re conscious?” Sydney said.
“You think I’m conscious because I’m like you, and you’re conscious,” Damien said. “DMS isn’t like us.”
“But if it’s aware, then it has consciousness,” Sydney said.
“Is a shark conscious?” Damien said.
“Yeah,” Sydney said.
“How about a cricket? How about a jellyfish? A sponge?”
“If we don’t know if DMS is conscious or not, then we pretty much have to assume it is,” Sydney said. “And if we back it up, we might kill it.”
Damien shook his head. “How can we kill it?”
Sydney said, “Because we will stop it and reinstall it.”
“So you think that the interruption of consciousness might be enough to kill it? You think it has a soul? Its consciousness is in the code. Its code and body are unchanged. If someone has a heart attack and you shock them back, they come back as themselves. Your body is you. DMS’s software and hardware is DMS.” Damien was very pleased with himself.
Sydney was pretty sure it wasn’t so simple. It wasn’t until the next day that she thought of a cogent argument, which was that organic systems are a lot less fragile than computer systems. Organic systems decay gracefully. Computer systems break easily. DMS was much more fragile than an animal. But that night she couldn’t think of anything.
The problem with poking the system to see if it was aware was to figure out what it could sense. DMS didn’t see or hear, didn’t eat or breathe. Its “senses” were all involved in interpreting data. So the “poke” needed to be something that it would recognize, that it would sense. And the poke needed to be something that it would sense as meaningful. The idea that Damien came up with was to feed it information in a way that it could recognize was a pattern but that wasn’t a pattern it expected.
DMS had several systems which regulated input and scanned for patterns. Epidemiological information was generated from ER, patient intake, and pharmaceutical information. Maintenance issues were anticipated from electrical usage. They picked the maintenance system, since DMS had been screwing with the electrical system, and input a thousand-character string of ones and zeroes. It was, Damien said, boring but clearly a pattern.
Sydney wasn’t sure it was the right kind of pattern. “Basically,” she said, “It’s like I flipped a coin and it came up heads a thousand times.”
“Yeah,” Damien said.
“If I did that, I might assume there was something wrong with the coin. But I wouldn’t assume aliens were trying to communicate with me through my coin toss.”
“DMS doesn’t have to recognize that we’re trying to communicate with it,” Damien explained. “It just has to notice that the information is not junk.”
DMS kicked the entry into the garbage column on its maintenance report.
They had written a program to do the entry, so they ran the program a thousand times.
If DMS noticed, it didn’t think anything of it. One thousand times it kicked the entry into the junk portion of the report.
“I don’t think it knows what we’re doing,” Sydney said. “You know, analyzing reports may be unconscious.”
“I don’t think consciousness is an issue here,” Damien said. “Remember the shark.”
“Okay. Maybe it’s involuntary. The shark has control over what fish it goes after, but it doesn’t have control over its kidney function. It doesn’t choose anything about kidney function. Maybe maintenance is involuntary.”
Damien looked at her. She thought he was going to say something dismissive, but after a moment he said, “Well, then, what parts of it would be voluntary?”
Sydney shrugged. “I don’t know. Epidemiologist, maybe. But we can’t screw too much with that.”
Screwing with maintenance was bad enough. But data from LEGBA went directly to the CDC and National Institute of Health through a weird subroutine called DAMBALLAH which did complicated pattern recognition and statistical stuff. Sydney worried about a couple of things. One was causing a system crash that meant someone ended up dying. The other was getting them in trouble with the CDC or the government. Of the two, she would have to choose getting in trouble, except she could imagine bad data to the CDC might mean someone ended up dying anyway. In her mind it unfolded: bad information seems to indicate a critical alert, Marburg virus reports in New York City seem to show that someone got off a plane and infected people with a hemorrhagic fever. The false epidemic pulls resources from a real outbreak of Legionnaires’ disease, and people who would have lived now die because she and Damien were poking DMS.