This Holon was as charming as he was handsome, but Antaea remembered where she'd heard the name before. She nearly said, "You tried to convince the Guard to kill Leal Maspeth," before remembering that if she admitted she knew that, she'd be giving away Leal's return, the existence of another door to Aethyr--essentially everything. "How is it you survived?"
"I walked away," he said with a shrug. "The Guardsmen and the other humans from Virga wouldn't have survived the journey. But I," he raised his perfect hands, "have augmentations that allowed me to survive until I could contact some of my people."
"But by then the rest of the survivors were dead?" He nodded. "Killed," pressed Antaea, "by ... what?"
Holon crossed his arms and frowned out the parlor's window. "We call them morphonts. They're creatures of Artificial Nature; they come in as many varieties as there are stars in the sky.
"Oh," he said suddenly. "You caught me as I was eating. You don't mind if I--?"
"By all means." She saw that a small buffet had been set up under the window.
He noticed her interest, and smiled. "Would you care to join me?"
Silent, she piled her plate high with cold cuts. Standing next to this foreigner, loading up on food--it was a very strange experience, but she barely noticed. She was thinking about Leal's conviction that the emissary was a friend, while at the same time, she'd insisted that it wasn't a conscious being like herself. The contradiction had been glaring the first time Antaea heard it. Over the weeks it hadn't become any less so.
Holon frowned at the fare, which was mostly meat. "It's difficult grazing for a vegetarian here."
"You're a vegetarian?" She watched as he picked through the food.
"Don't get me wrong, I love meat," he added as he piled up a plate. "Back home, I eat it all the time. But then, we've got other things besides meat and vegetable matter to eat--and whatever meat I eat is vat-grown."
She nodded, remembering the Home Guard fortress at the Gates of Virga. "I've had it. A perfect steak, every time."
"The mere thought of eating the flesh of something that once had a brain horrifies me," he continued. "I know you Virgans are a bit more ruthless that way. I suppose you have to be. But my conscience won't allow me to harm another sentient being."
Antaea put down her plate. "But you're happy to kill the morphonts." He shrugged. He sat at the parlor's little table, arraying his food around him.
"Tell me more about the morphonts," she said. "They're not aware like you and me, you say. How then are they a threat to anyone? Wouldn't they just be like plants, if they have no minds?"
"It's hardly a secret who they are or how they work--"
"Oh, but it is. Your people never told us about them," she interjected. "I was in the Guard for many years. I even traveled outside Virga--"
"I'm sure we told you," he said with sudden irritation. "Maybe you didn't understand us."
"Fair enough." She held up a placating hand. "And forgive me if you've been asked this twenty times already. Indulge me--what do you believe the morphonts are?"
"A mistake," Holon said curtly. "One that's taken over much of the universe, at the expense of conscious beings like you and me.
"Imagine that your tools could think--even anticipate your needs. Back in the early days of our expansion into space, we humans created machines like that. At first, we had to tell them what to do. They obeyed our orders--did what we said, but not always what we wanted. They didn't understand us the way we understood each other. So some wise idiots decided to give them the capacity to understand our needs, as well as our commands. So they could anticipate what we would want, rather than having to be told."
Antaea frowned. "And this was a mistake?"
He snorted. "Well, it's not as if there hadn't been countless stories written by then about what would happen if you let your machines understand you that well. --Problem is, they were all wrong. They all assumed the machines would take over--remove our free choice, disobey our orders in order to give us what we needed instead of what we wanted. Ridiculous, of course. They never ceased to follow orders."
"Then what went wrong?"
"We'd given them the ability to perceive purpose. Many of our researchers thought that purpose--or values, intentionality--was an illusion of our human perspective. Turns out it's not; it's an emergent feature of the universe, as real as water and rock. And it's not just humans that have it."
"Purpose ... You're talking about meaning?"
Holon nodded. "The Moderns who built the first artificial intelligences didn't really believe our minds were a part of this universe. They were still saddled with ancient religious beliefs, but they didn't know it. They thought meaning was some kind of local human illusion, or the gift of a god. But everything that lives, wants, and to want is to give meaning to things. --To say yes, or no, even if it's just about whether some speck is food.
"Once our machines could see that, they could no longer see the distinction between us and any other living thing. Of course, we didn't realize it at first. By the time our ancestors figured it out, some of our machines had started taking orders from nonhuman--and nonthinking--kinds of life."
"You're saying they started working for ... what?" She laughed. "Trees?"
But Holon wasn't laughing. "We recognize each other. We see the spark of life, of awareness, in one another. It's so easy for us that we never even considered that it might not be easy for an artificial intelligence. But they are not us. They cannot recognize that spark in us, the way we see each other. Other than its shape, and the fact that one can give verbal orders and the other can't, what's the real difference between a human and a tree? Or a dog. Or a lion?"
She didn't know what those last two things were, but the implication was clear--and unbelievably strange--to Antaea. But she remembered some of the weird things she'd seen when she'd visited the realms of Artificial Nature. There had been odd machines--giant crystal spheres encapsulating little miniature ecosystems, surrounded by a retinue of guard bots and helper machines. She'd seen one plow down the center of a street, humans and virtual life-forms hopping out of its way, but none protesting or trying to stop it. She'd asked what they were at the time, but had not understood the answer. "You're saying we gave away our technology to ... nature itself?"
Holon nodded. "Exactly. One way to put it would be to say that we accidentally created a universal interface for our entire industrial and intellectual legacy--an interface that anything that can want, can use."
"But why not simply go back to the way it was? Make machines that only obey orders from something that looks like a human?"
"Oh, we do. Now. But the machines that chained their own purposes to those of nonhuman life-forms proliferated; they took their own will to survive and reproduce from the species they allied with. Some became fierce beyond all human control."
Antaea was shaking her head. Holon said, "Look at it this way, then: an artificial intelligence doesn't come with its own will to live. That's something separate from the ability to think, it has to be added on. You don't notice this until you start to build tools that can act on their own--when you stop using them directly. The greater the distance between your guiding hand and the actions of the machine, the more it has to develop its own sense of who 'you' and 'it' are. The best way to get such an autonomous machine to work for you is to design it in such a way that it thinks it is you. It studies your desires and needs the way your brain studies the needs of your body; it identifies with you entirely, and has no desires of its own. It isn't even aware of itself. But a machine that can do that can just as easily identify itself as a flower, or a crow, or any other creature. It could identify itself as a rock, I suppose, but rocks have no needs. A machine that did that would just stop. But imagine one that, for one reason or another, has no human to imprint on. It searches for things like humans--and let's say it finds a crow. Once it understands some particular crow, it comes to think of itself as the crow--and has access to the entire history of human ingenuity and industry, to aid in obtaining what that crow wants ...