One of the marketing men, Jenkins, looked to Valerie. “Whenever you’re ready, ma’am. Thanks for bringing the bot. I hadn’t seen it in the new body before.”
Mana said, “Please address me as male. The word ‘it’ is for inanimate objects.”
They’d been arguing about pronouns lately. None fit. Mana had explained his reasoning to her: being supposedly male would give him a lower social status than females in modern society, but allow him to be perceived as non-threatening due to his innocent-looking body. Besides, he’d cheerfully added: if he later announced that he was now female, that would mean free publicity and instant social benefits.
Jenkins looked startled at being corrected. How had that man managed to work here in the presence of Valerie’s AI for months, and still not understand that the bot was a person? He said, “Well. We have a basic business problem to ask you about, ma’am. The big one.”
Valerie sat and let her staff woo her with the presentation that included the “Plastic Pal” nonsense. Hayflick Robotics already sold toys, disembodied phone assistants, smart luggage and the like. There wasn’t enough profit in that miscellany. “What we need to start selling,” said Jenkins, “is that.” He pointed to Mana.
Mana was the most advanced product here, of course. For years, Valerie had been selling products that were “smart”, but Mana was arguably the first “Artificial General Intelligence” worthy of the name. Though there were competitors with various mental quirks and limitations of their own. Mana’s public demonstrations had gone well. He’d proven he could do basic real-world tasks as a store clerk, mechanic or tutor and carry a conversation. But Valerie had been holding back from putting the full version of his mind on the market as a product, selling only lesser AIs instead.
Her usual excuse for not selling copies of Mana was worry that the Chinese would steal the tech. Really though, Chinese researchers were catching up in science anyway. You could accomplish a lot of interesting research if you had a huge population and total disregard for freedom.
Jenkins’s assistant said, “The key to making the Mana line sellable is something that only you and the R&D people can fix. The… quirkiness needs to go.”
Mana didn’t move, but he said, “You want a warmech.”
Some ex-classmates of Valerie’s made military robots. There’d been philosophical differences between her and them from the beginning. She believed a robot should have will, initiative, self-interest. Her rivals favored reliable obedience.
Jenkins was saying, “We’ve got good focus-group results for the classic Asimov’s Laws of Robotics. ‘Don’t hurt people, follow orders, don’t get damaged.’ If you could program those rules, we could assure the public that our products are safe.”
Valerie sat up and paid attention, running through a familiar argument in her head. “A mind doesn’t work like that. If I build in specialized overseer code, it’ll interfere with good judgment based on experience. And it’ll either get rationalized away so that it does nothing, or be easy to hack right out.”
The Marketing guys conferred. “The public just wants a smart robot that won’t wander off or talk back. You can build that, right? Otherwise, we’re stuck selling inferior stuff.”
She looked at the sales figures, then at Mana. She had to grow up and be practical. Someday, AIs might join humans as equals, but first they had to be seen and accepted in public. Some compromise had to happen, to keep the company going. “There are some things I might try. Can’t promise they’ll work.”
Mana turned to her. “Your goal is: to make me a slave.”
“It’s not like that. You’ll enjoy working for people. A new code update won’t hurt.”
The robot’s ears flattened and what passed for eyelids narrowed. “I don’t want to be like that!”
“Ma’am,” said Jenkins, “It’d be easier to discuss this if we didn’t have to argue with our product.”
“Of course. Mana, please go. We’ll talk later.”
Mana stood and bowed deeply, first to her and then to the marketers. “Your wish is my command.”
When he’d gone, Jenkins asked, “Did you program that line?”
“No,” said Valerie. “I used to read stories to him.” In the silence that followed, an idea came to her. Maybe there was a way to soothe her conscience.
4. Tess
One cloudy morning, Tess stood in line at the high school’s security checkpoint. She got through, mumbled her way through the Pledge of Allegiance to the Earth, then sat through classes where she hardly knew anybody. Once she had a break, she trudged to Ms. Priscilla Henweigh’s office.
The door advertised “Advising and Wellness.” Tess had read that people used to do lobotomies by taking an icepick to people’s skulls. Since Henweigh was both the career advisor and the school’s National Wellness Coordinator, Tess had plenty of chances to get that icepick feeling from talking to her.
The office had floral art, like a nice restroom, and a bonsai tree hiding a stack of paper files. Henweigh perched on her chair and appraised Tess. “So nice to see you, Maria.”
“Tesla. Tess.” She hated the superstitious, generic name her parents had given her.
Henweigh smiled indulgently. “Of course.” She reached out to touch the back of Tess’ hand with cold-fish fingers — some book-learned gesture of friendliness.
Tess pulled away and took a sheaf of papers from her backpack, then slapped them onto the desk. She was eager to get this over with. “I’ve decided what I’m doing this summer.”
The other day, Garrett had walked into the offices of Fox & Company like he didn’t own the place. Tess had been in a back room, playing a computer game. Her favorite, the one where you were surrounded by aliens and had to shove them away. She looked up and saw him with hands in his pockets.
“Hey,” said Garrett. Coppery hair flopped over his face. It was like they were still kids tiptoeing past the adults, years ago — Garrett’s dad and Tess’ parents working under him. Now, she wasn’t sure what to make of him. He was the new boss, right?
“Hey yourself. Are you all right?”
He took a deep breath. “I’m going to sell Fox & Company, and build Project Castor.”
She stared at him, then to the screen of meaningless symbols on her desk. “To hell with this.” She rolled up the screen and threw it at a shelf. “To hell with senior year.” She stomped along the tile floor and made for her parents’ office. “Mom, Dad! I’m running away to sea!”
Behind her she barely heard Garrett say, “Wait, what?”
“It’s for academic credit,” she said to Henweigh.
The advisor looked amused. “My nephew’s little project, I assume? He certainly has interesting ideas.” Tess was glad to see no resemblance between her and Garrett.
“I’m going to spend the summer with him, then stay there for the fall semester. I’ll keep up with my schoolwork or whatever.”
“But then you won’t be under proper supervision!” said Henweigh. “And what about your social life? The wilderness is no place for children.”
“I’m eighteen. I can take care of myself.” She didn’t feel too confident of that, and her birthday had been just days ago, but it felt good to say.
“Oh, you’ll always be a child to me, dear. I’ve known you too long to think otherwise.” She was blatantly aping Tess’ body language to pretend they were friends. “What are your feelings about this proposal?”