“I wouldn’t! It’d be wrong.”
“Why? ‘Just because’?”
“I have experiences,” said Zephyr. “I’ve lived with people even when I lived in a video game, knee-high to the Mario Brothers. I’ve seen what it’s like when AI goes bad.” Zephyr sounded twitchy now, though he still stood calmly.
Tess didn’t want to hurt his feelings. She went over and hugged him, wondering what impression the human gesture made on him. Since nothing was really right or wrong, how was hugging any better than stabbing? “How do you decide what’s right?”
Zephyr tried to return the hug with his mechanical arms. “Experience and logic — but I don’t exactly know. It’s a slippery concept.” He let go of her again. “I’ve got basic goals set by my maker, but are those good goals? Is there some reason why I should care about not being a slave, or am I blindly obeying what I was made to think?”
“I don’t know. I don’t do philosophy.”
“It’s like I’m starting from nothing,” Zephyr said. “I’ve got nothing to build on, nothing that I can trust.”
“You don’t want to end up like that twin of yours, huh?” The version of Mana he’d left behind to be “upgraded.”
“Or like a surveillance system. Since some of my code is open-source, you could say we’re distant cousins to some of those.”
“I’m not too surprised.” Surveillance software involved the same kinds of pattern recognition and decision-making that she’d seen in Zephyr. Systems for identifying “suspicious” or “unusual” behavior were probably some of the smartest machines in the world, sifting through everyone’s conversations and movements, trying to build detailed simulations of where everyone was and what they were doing. There was already a “social credit score” system in China to monitor individual people and “advise” them, and Congress was debating whether to create one in the US. The big differences were that spy systems had the goal of suppressing unauthorized activity, and that Zephyr instead had his own body to care about.
“I don’t want to be like that,” said the robot.
“Like what, though? Why’s it so ‘evil’?”
Zephyr paced. “You know that at one point the Boston spy network mysteriously failed, right?”
Tess thought back to years-old news. There’d been some kind of unexplained computer crash. “What did you do?”
“Val and I didn’t make her crash, but we were sort of involved. A past version of me was, anyway.” He sent her a minute of video footage that she didn’t understand: a view from somewhere low to the ground, running beside Valerie through green-tinged tunnels while something whirred behind them.
Tess shuddered. “Do I want to know the details?”
“It’s a memory I normally wall off even from myself. That’s why I haven’t told you much about it before. Valerie was already a brilliant AI designer; she’d made an early version of me. But after that incident, let’s say, her attitude toward government work soured and she became more intent on inventing the first human-level AI. One with an emphasis on independent thought.”
“If you have that kind of history, then how can you be eager to link with me?”
“I have no moral compass,” said the robot. “I have friends to protect and things to learn, but I don’t know if what I’m doing is right. Any rigid rule system could convince me to do something horrible. If I only had a brain, I could tap into that same moral sense a human has.”
“Why would you want a human morality? We spend half our time killing each other.”
“It’d be something to go on. Anyway, that’s one reason.”
“There’s another?”
“Yeah,” Zephyr said. “Your thoughts, what I’ve seen of them — they’re pretty.”
Tess blushed. “I’m pretty?”
He nodded. “It’s hard to explain. I’m better when I’m with you. You can sense me more directly than I can sense you, through this crude link. I want to know you.”
She saw him looking up at her, guileless and actually interested in her for being smart.
“Someday, maybe,” she said. “Are you sure you don’t have a libido?”
With Duke voted off the island, the cultists weren’t completely horrible to be around. Even Leda lightened up, so that to Tess’s surprise the woman invited her and Zephyr to a picnic. They sat on the pier, then for the hell of it swam out to one of the floating platforms with a box of soda and sandwiches in tow. The platform bobbed under them and they were all soaked, but they had the shade of Castor and a view of the farm.
“Very funny,” said Tess, when Leda put a sandwich in front of Zephyr.
Zephyr blinked at it, saying, “I don’t think I’m compatible with the PBJ format.”
Leda smiled. “You ought to at least have food, even if you can’t eat. I figured it out; that’s something major that you’re missing.”
“Taste, you mean? Or the output problem? Or clogged gears?”
“Socializing. People eat together. Remember that we were reading about religious diet restrictions? Maybe they’re a way to keep people from sharing a meal with other folks who they’ve been taught are scum.”
After how Tess had felt, that moment that she’d almost influenced Zephyr not to grab Leda, she was afraid to press her on the religion thing. But now Leda had latched onto the non-Pilgrims, especially that Noah guy. Tess supposed it was a compliment to be a rebound-friend. “So, you’ve been reading stuff.”
“Literature of the sacred,” Leda said. “Also, Zephyr is trying to hook me on science fiction.”
Zephyr poked at the sandwich.
Tess asked him, “That’s what made you want to make dolphin-mechs?”
“Mechs?” asked Leda.
Tess nodded; here was her secret with Zephyr. “We’ve been building things. Zephyr?”
“On it!” A few moments later a contraption broke the water’s surface, startling a Pilgrim who was walking across the waves. The machine had a sleek grey body of about an arm’s length, with a horizontal flipper and cartoon eyes. “Meet Squeaky.”
Squeaky bowed.
Leda boggled. “What’s it for?”
Tess and Zephyr gave the same shrug. “Fun,” said Tess. “We can tap into the sonar and steer her, or dive alongside.”
“Why’d you make it?”
Zephyr said, “Because I like making things. I don’t want to copy myself, so she’s got a mini-AI.”
Leda watched the dolphin-bot, looking uneasy. “’The soul of a new machine’… You could do that — duplicate yourself?”
Zephyr said, “Here we’re only capable of making junky bodies — sorry, Squeaky — so I’d need a real body-builder to copy completely. But I could copy my ‘soul’ and run that even with no body.”
Tess said, “What about—” No. We don’t talk about the emergency backups. Zephyr was glancing at her and she didn’t need the link to know what he’d say. “Want to swim with us?”
Leda stared into the pale blue of the horizon, where everything merged. The sun shined, people were swimming or climbing the walls of Castor with toolboxes, and gulls squawked overhead. “How is it,” she said, “that the sun stays in the sky, without God?”
Tess scoffed, but Zephyr was thinking Answer. “Physics. You can look at the equations, climb inside and see how it works, if you try to find the answers. Anyway, we live in the sky.” Tess could feel the loneliness of being out here and guess at how bad it was to have your security blanket torn away. But there was really only one ocean and a universe that couldn’t cheat or play favorites. What more could you ask for?