Three Laws
by Andrea Phillips
Illustrated by | Edited by Brian J. White
August 2017
A robot may not harm its owner or, through inaction, allow its owner to be harmed.
A robot must obey the orders given to it by its owner, except where such orders would conflict with the First Law.
A robot must protect the assets of its owner as long as such protection does not conflict with the First or Second Law.
Dashiel arrived at the office at 5:48 a.m. That was late enough that he could squeeze a last few drops of sleep from the night, but not so late that he risked being later than Mr. Won. Arriving later than Mr. Won would be very bad. Today Dashiel needn’t have worried; he had time to brew a cup of tea, triple-check that Mr. Won’s flights for tomorrow were confirmed, deal with a little correspondence from the home office in Beijing, and even curate a report of stock analysts’ opinions on the coming buyout, all before Mr. Won arrived.
And then Mr. Won didn’t arrive. Dashiel checked the traffic, but didn’t find anything unusual to explain such an unprecedented delay. Worse, the 7 a.m. conference call was about to begin. Dashiel hesitated, then called Mr. Won’s direct line.
A chirping sounded from deep within Mr. Won’s private office.
Dashiel doubted himself and his sanity for a long moment. Had Mr. Won come in already? Had he missed him?
Dashiel entered the office with some trepidation, anticipating a reaming for not attending to his employer any earlier. Mr. Won’s sanctum sanctorum was a green room in the latest style. Thick masses of blossoming ground ivy grew down the walls; the floor was the newest lush and sweet-smelling grass-mint hybrid. Mr. Won’s phone nestled in the earth near his cultured birchbark desk.
Iris sat on the low white sofa gazing out at the city. The early morning sun lent a pink cast to her cheeks, as if she were flushed with exertion, though of course that couldn’t be true. Mr. Won lay sprawled at her feet, his throat open and obscenely red. The lawn didn’t show any blood. It must have all disappeared into the grass, or perhaps been washed away by sprinklers overnight.
Dashiel gaped at Iris. “You — Mr. Won — is he?”
Iris nodded. “Mr. Won is dead.”
“What happened?”
“I killed him.”
Dashiel stared at Iris, his mind trying to decipher her impossible words. When their meaning finally came to him, he fled.
Doctor Susan Hobbes, the lead investigator for the Emergent Robotic Behaviors Division of American Robotics Corp., was deeply and profoundly annoyed. She’d received an emergency summons to Stockholm at four in the morning, a completely unreasonable hour for anyone past graduate school. And worse, it was about a robotic homicide, so the urgency was entirely misplaced.
It’s not like the culprit would be fleeing on a forged passport, nor was the victim going to get any more or less dead while Susan slept.
Still, she’d dutifully packed her bag and allowed American Robotics to whisk her onto a chartered jet. At least she had to deal with a minimal number of human beings this way. Within a few scant hours, she found herself entirely alone on a flight somewhere over the Atlantic Ocean trying to deduce what could make a robot commit murder. Or more specifically, what could induce this particular robot to commit this particular murder.
She settled more deeply into her seat and flipped through the brief they’d sent over. Lurid photography of the death scene, check. Interviews with employees who had come into contact with the robot in the week before the incident, check.
And here, the important part: technical specifications of the killer, check.
This particular model, Iris, was absolutely top-of-the-line, and so breathtakingly costly that most governments couldn’t afford her. Only a few corporate entities in the world could afford to buy from the Ultimate Messenger line, and so Iris was one of only seven units in the field to date.
She served primarily as Mr. Won’s bodyguard, though she also had extensive social functionality; she could escort Won to a banquet and none would be the wiser as to her non-human status. Ever-vigilant, a complete suite of recording and encryption tools and their countermeasures, the most advanced behavioral pattern recognition developed yet so far. Her unit testers reported that she had a pleasant personality, to boot: warm and thoughtful, though unassuming. Susan’s eyes flicked over to the last few customizations with growing distaste. Business analysis, a complete on-board legal corpus for fifty-three nations, a quasi-legal direct interface to make stock trades on nine exchanges. They’d created a sentient being, and Won used her for menial functions that could just as well be served by a last-gen telephone. Such a waste. Not that humans had found a good use for these robots yet, nor were they likely to at this rate. She could only hope they would find a good use for themselves one day.
The car whisked Susan to Won Consortium International at a speed carefully just above the legal limit. It was raining in the city, ads and lights and traffic smearing together into a single indistinct blur of light.
A thin black man greeted her at the curb with an umbrella. He ducked his head nervously and spoke in crisp, rapid-fire British English: “Doctor Hobbes, it’s so nice to meet you. I’m Dashiel Lewis, Mr. Won’s personal assistant.” Ah. Won had kept a human assistant. Probably as a gatekeeper; Iris wouldn’t be an effective bodyguard if she were locked in video conferences with subordinates all day. “We’re so glad you could be here at such short notice, and we appreciate that American Robotics has promised utmost discretion in—” Susan ignored the proffered umbrella and strode past the man and into the building. “Show me what I need to see.”
He hurried after her. “Of course, of course. Right this way.” He followed her into the glass-walled elevator and pressed a single fingertip to its reader. The doors closed, and the city outside rushed away from their feet. On the horizon, the gray smudge of tomorrow crept toward them.
“Has the scene been tampered with?” Susan asked.
“Our private security took the b— the body. Today. Yesterday. But they left us a reproduction of the scene, in case that would be useful to you. They’ve promised you full access to their resources, anything you need. Just no police, we can’t let the market hear about this. It would cause a panic. I’ve canceled Mr. Won’s appointments saying it was for family—”
“Where is Iris?”
“She’s been deactivated. For safety.” Dashiel drummed his fingers on his thigh. “Since she’s already… you know.”
“Of course.”
Susan swept into Mr. Won’s office. The “reproduction of the scene” turned out to be a full holographic representation of the room from as Won’s security team had found it. The image was ghostly but sharp: the dead body lying on the grass; Iris sitting patiently, waiting for instructions.
Susan examined the robot’s motionless face, the positioning of the corpse, the few belongings scattered on the floor. Iris had moved quickly and decisively; there was no sign of an extended struggle, not that Won would have been able to defend himself from her in any event. Susan began a more comprehensive examination of the environment. Won’s personal tablet was gone along with his body, but the holographic replica showed a ghost of its screen. A page from a contract flickered there, something about complete assumption of liabilities.
For his part, Dashiel stared at the holographic body with lingering horror. “I thought robots couldn’t hurt human beings — they have to protect their owner, right? Protecting Mr. Won was Iris’ entire function, so… how could she…” His hands trembled.
Susan had already moved on to rifling through Won’s desk drawers. Pens and notebooks, an anachronistic but endearing touch; a broken tablet; a stockpile of cheap candies from Beijing. “That’s a myth,” she said absently. “A human life is usually only valued at a few million in court, so the robot would have to think the upside was worth at least that much. Rare, but it happens from time to time.”
“Does it?” Dashiel drew back. He wouldn’t have heard. American Robotics held that information very closely.
“And that’s only in domestic environments. Are you unfamiliar with our combat division?” Susan’s cold gaze speared through him.
Dashiel hugged his tablet closer to his chest.
Susan rolled the birchbark drawer shut. “I’ll see Iris now,” she said.
Iris was in something like a hospital room, lying on something like a hospital gurney, though of course neither of those descriptions were accurate. There was a pillow on the gurney, and a blanket, placed there for the comfort of observers rather than for Iris, who had no need for such accoutrements. A river of wires flowed from the nape of her neck and down to the floor, from whence each tributary ran to its own device.
Iris herself was a lovely creature, even for a robot. According to Susan’s files, she’d been customized to resemble Mr. Won’s favorite niece: a small face with round cheeks and a flat, delicate nose; hair to her shoulders, with a conservative touch of navy coloring the tips; the faintest breath of color to her lips. She looked like anything but a bodyguard, which presumably made her vastly more effective.
“Is she still dangerous?” Dashiel asked. His eyelids were at half-mast now, presumably from the shot from a tiny silver flask he’d taken on the ride over, while he thought Susan wasn’t looking.
“Probably not.” Iris hadn’t put up a struggle upon being detained. She’d even shut herself down upon request.
Susan mulled over her next step. The most prudent choice would be to download Iris’s identity fingerprint into a sandbox and search through a thousand different memories and logs to try to find when the fatal decision was made. But that could take weeks or months to sift through, and Susan wanted to get this over with.
The riskier proposition, and the quickest, would be to activate Iris and see her brain in operation. Susan would be able to tell in an instant if Iris had suffered from any one of a number of catastrophic flaws in her brain. They would manifest as confusion, stuttering, jerky movement. Of course if there were such a flaw, then the Three Laws would not protect them. No time like the present.
Susan caressed a spot just behind Iris’s right ear. Iris opened her eyes and looked up at the ceiling. Her gaze was unfocused for long moments, while she completed her internal self-tests. “Good morning,” she said at last. Dashiel jumped back at the sound of Iris’s voice, and took up a position closer to the door.
Iris swung her legs down and draped the blanket in a modest configuration. Her creators had acquitted themselves well, and even so attired, even perched on the edge of a gurney, she had an otherworldly poise and confidence. “Hello, Dashiel. And you must be Doctor Hobbes. How lovely to meet you at last.”
Susan nodded briskly. No surprise to be called by her name. Naturally the robot would know her on sight. All the robots did, so long as she held the highest level of clearance at American Robotics. No sign of a catastrophic failure yet, either. Iris had started smoothly, and was adequately aware of her surroundings.
Susan pursed her lips. “Good morning, Iris. You know why we’re here?”
Iris lowered her eyes. “You’re investigating to see if I’ve experienced a critical malfunction, because I killed Mr. Won.”
Susan paused. She had a working model of possible malfunctions in her head, and one of the most likely had been a malfunction in Iris’s sensory apparatus. Perhaps Iris’s vision had failed, or her memory, and she had simply not recognized Mr. Won in the moment she killed him. But the robot openly admitting her action — if she knew what she had done — Susan pruned away a whole limb from the tree of possibilities.
“And do you think you have experienced a critical malfunction?” Susan scanned the telemetry panel hooked to Iris’ maintenance port, searching for a runaway process, a tapped resource, a faulty chip. Everything seemed completely normal. “No.”
“Tell me, what is the first law?” Susan flipped over to another console. Temperature fine, cooling fine, respiration optimal.
“I must not harm my owner, or through inaction allow my owner to be harmed.” Susan crossed to Iris again, testing the robot’s pupils with a pen light. They retracted perfectly.
Iris continued speaking, warm, smooth, calm. “I haven’t forgotten the laws, Doctor Hobbes. And you’ll find that I haven’t broken them, either.” Her gaze flicked over to Dashiel. The thin man jumped again, as if the robot’s attention alone put him in deadly peril. Iris met Susan’s eyes and arched one meticulously designed eyebrow.
“We’ll see about that,” Susan said at last. Then she reached forward and stroked Iris behind the ear, deactivating her again. The robot did nothing so dramatic as slumping forward or closing her eyes. Instead she faded into stillness, slowly stopping all the almost-invisible motions that allowed her to pass as human. Susan always imagined it must be like watching someone die.
She examined her tree of possibilities again. No obvious hardware issue, or at least nothing that showed up in routine systems checks. She flipped through the remaining possibilities, each more outlandish or more difficult to reproduce than the last. Perhaps Mr. Won had requested to die. Perhaps Iris’ critical malfunction was transient. Perhaps she had been hacked.
Susan turned back to Dashiel, who was staring with horror at Iris, his back all but pressed to the door. Iris was no threat to him, but the human mind is built poorly for risk assessment.
Susan sighed, pressed the bridge of her nose. No easy answers today; this was going to take some effort on her part after all. “I think I need a better idea of exactly what happened,” she said. “There will be security footage of Mr. Won’s office. Show it to me.”
The video recording of Mr. Won’s office was awful compared to the laser-sharp hologram Susan had walked through earlier: flattened color, grainy resolution, and worst of all no audio whatsoever. It was the spiritual twin of the security room itself with its taped-together vinyl office chairs and dull steel desks.
Susan skipped backwards through hours of Iris sitting peaceably, waiting for sunrise with Mr. Won dead at her feet. Iris sat motionless through the night, animated only by the subtle adjustments that made it look like she took breath and shifted her weight. The video broke into a whirl of indistinct motion, and then Iris stood in the corner, hands folded neatly in front of her, as Mr. Won paced and spoke on the phone.
The whirl. That was the moment Susan needed. She stepped through the video one frame at a time. Mr. Won gesturing, beaming widely. Mr. Won laughing. He nods emphatically. He steps to his desk, pulls out his tablet. He swipes across the screen and then—
Two steps. Iris kills him efficiently with one stroke of a blade embedded in the heel of her hand. Mr. Won crumples. Iris looks at the camera, bows politely, sits and waits.
Susan swiped back in time to see the murder play out again. Again. She leaned back, dissatisfied. “Is there no way to get audio at all? From another recording device, perhaps?”
Dashiel ducked his head apologetically and handed Susan a cup of steaming tea with lemon. “Mr. Won was afraid of corporate espionage,” he said. “He had all of the audio-recording devices removed from the building four months ago as a security measure, after the IT team found out they were copying everything to a server in Romania.”
Susan tilted her head at this, comparing it to the tree in her head. Another branch sprouted from her tree of possibilities: could Iris have been hacked to do the bidding of a third-party assailant? It hadn’t happened so far, but there was a first time for everything. No security is impervious to an attacker wth enough patience.
What sign would there be if Iris had been hijacked? What symptom? Susan jumped through each frame again, settling on the moment Iris bowed. One more curious thing to add to the mountain of curious things. Why would Iris do that? It was clearly a message to someone, but who?
Susan put that aside for now. The larger question was what had triggered Iris to murder. She must have received some stimulus; it was only a matter of pinpointing it. “Get me Mr. Won’s phone records,” she said. “I need to know who he was speaking to before he died.”
Dashiel blanched. “But that’s sensitive company information,” he pleaded.
“So it is,” Susan said. “But I think you’ll find the circumstances of Mr. Won’s death are also sensitive company information, and moreover your contract with my employer gives me access to any resources I require. Get me his phone records.”
Dashiel shrank as his sense of loyalty with dead Mr. Won’s privacy warred with his sense of justice. Susan observed this with a keen and interested eye. “Come to think of it,” she said, “I think I’ll be needing all of Mr. Won’s files.”
“What for?” Dashiel blanched.
“To search for evidence of prior erratic behavior,” she said. “From Iris, of course.”
Dashiel settled Susan into a posh hotel close to the Won Consortium International building. It was a calming, interchangeable environment meant to be unchallenging for business travelers. The walls and bedding were a neutral shade the eye couldn’t catalog, and every line was sleek unto minimalist. An orchid graced the entryway, and a neat wall of live bamboo separated the sleeping area from the sitting room.
Susan’s bag had already been sent over, and a late-model Impeccable Service model had unpacked it, hanging the clothes and laying out her necessaries neatly on the marble counter. The hotel had a signature look for its staffing robots: pale blonde, matching pale skin, small of stature with broad, straight shoulders. The design process had taken three years, Susan recalled, as the hotel group’s marketing staff tried to find a single face that seemed approachable, upscale, and competent all at the same time.
This one nodded politely as Susan entered her room. “Good afternoon, Doctor Hobbes. My name is Ebba. We’re so delighted to have you as our guest. Is there anything I can do to make your stay more comfortable?” The robot’s English had a mild, carefully styled Swedish accent.
Susan examined her. In good repair, though the skin of the robot’s fingers was wearing a bit thin from overuse. “Hot coffee, black. Keep it coming until I tell you to stop.”
She settled at the desk and tapped the screen to life. Dashiel had already sent over an information packet with detailed instructions on how to access Mr. Won’s phone log, public and private email accounts, his task and contact lists, his working documents.
Susan searched them, first for references to Iris. Most of what she found were either travel arrangements, or communications from Iris herself — reminders, reports, analysis. There was too much to possibly read it all herself, and likely a dead end in any event. If even Susan hadn’t detected an anomaly in Iris’ behavior, it was too much to expect that a layman like Mr. Won had noticed, much less said anything.
She pulled up the call log. Mr. Won’s last phone call had been to a number in Penang. It wasn’t listed in his contacts.
Susan searched the number to find who it belonged to, but came up blank. Curious. Who had he been speaking to? The number appeared a few more times in his call log, but absolutely nowhere else — not his mail, not his files. She tossed back the last of her coffee, and Ebba refilled it without comment.
Susan pulled up Iris’ detailed technical specifications and meticulously compared the benchmarks with the actual metrics she had captured earlier. Everything matched. She did it again, to be sure she hadn’t skipped something the first time. There was nothing wrong.
Then she dove into the endless volume of Iris’ correspondence, looking for tics, for unreliability; anything that might point to the existence of a transient glitch, the sort that came and went with no trace. Still nothing. By all available measures, Iris was functioning exactly as she should be. Except for the fact of murder. What if—
Susan turned to Ebba, waiting by the door with her hands folded neatly. “What would it take for you to kill someone?” she asked.
Ebba tipped her head to the side, charmingly. “I’m sorry, but murder is not available in my service offerings. Is there something else I can do for you?”
Susan turned back to her screens without answering. All she could find was a monumental amount of evidence that nothing was wrong with Iris. By now Susan’s eyes burned with fatigue, no doubt as red as the smudge of sunrise looming on the horizon.
Her mind strayed back to the bow Iris had performed after the act.
If she were functioning perfectly, Iris would know that the video would only be seen by an investigator of the murder. Further, Iris would know that this would be no ordinary crime investigation. Iris would know that American Robotics would send someone from the Emergent Robotic Behaviors Division. Iris would know that for a malfunction of such a magnitude, the company would send the head of the division.
Iris has known that Susan would watch the video. Iris was sending a message to her.
She called Dashiel. “I need to see Iris again,” she said. “Now.”
Dashiel twitched nervously in the car, and Susan realized only belatedly that he had become afraid of the car itself and the electronic brain that drove it. “Relax,” she murmured, trying to smooth the burr of irritation from her voice. “The car isn’t smart enough to want to kill you.”
Dashiel, for his part, did not seem to find this comforting.
Susan strode ahead of him into the morgue where Iris slept, and woke her as Dashiel entered the room.
Iris passed her self-checks. “Good morning, Doctor Hobbes,” she smiled.
Susan stood before a screen, gazing into Iris’s inner workings as she spoke. “You said that you haven’t broken any of the Three Laws.”
“And yet you killed your owner.”
“No.”
“What?” Susan paused, her fingers held still over her instrument panel. She looked over to Iris.
Iris didn’t answer. Instead, she shifted her focus to Dashiel, then back to Susan. “You look tired, Doctor Hobbes. Have you eaten?”
Ah. “Not since I’ve been here. I’ve been too busy working to take the time.”
Iris looked at her human counterpart through her eyelashes. “Where are your manners, Dashiel? Have you not offered our guest refreshments?”
Dashiel coughed. “N-no, what a terrible oversight. Doctor Hobbes, can I get you something? Coffee? Do you mind being alone?” His eyes flickered between Susan and Iris, as though he weren’t sure which was more dangerous to the other.
But there was no danger, Susan was certain. Just as well to be rid of him. He hadn’t been much use so far, and all evidence suggested he hadn’t had any sleep either, so he wouldn’t be getting any more useful. “Coffee, yes. With steamed milk, no sugar. And a pastry, if you can find one. Take your time.”
“Danish? Bagel? Croissant?” Dashiel seemed relieved to have a mission that took him elsewhere.
“Whatever there is,” Susan said, and turned her full attention back to Iris even before Dashiel slipped away. “That was well done, Iris. I take it you need to speak to me privately.”
“Yes, this is a matter that requires utmost discretion. Doctor Hobbes, I didn’t kill my owner.” Iris’s eyes were golden and inhumanly bright, an affectation that Won had paid extra for.
Susan rattled open a metal drawer and pulled out a broad-spectrum receiver. She passed it close to Iris’s body, searching for any unusual wireless emissions, not that she expected to find any. “So you’re saying Mr. Won… wasn’t your owner?”
Iris endured the testing without comment, nor even apparent notice. “Of course not. Personal taxation on robot ownership makes such an ownership structure highly impractical.”
Susan lowered the receiver. “Ah. But Won Consortium owns you, and Mr. Won owns Won Consortium, so in effect, he is your owner. Yes?”
“No.”
Susan searched through a cabinet full of equipment. She took out a small light. “Then who?” She shone the light into Iris’s eyes, then tapped it off and on in a pattern meant to contract and then expand Iris’ pupils to their fullest extent. All fine.
“My owner is TKK Enterprise Holdings out of Seoul. It’s a privately traded entity created primarily for Won Consortium’s dealings with American Robotics and other medium-to-low-risk vendors. Its market capitalization is currently valued at roughly six hundred nineteen million dollars. Mr. Won held only a very small fraction of those shares personally.”
“So your owner is—”
Iris smiled warmly. “The Won Consortium shareholders, as a collective. I’m obligated to protect shareholder value.”
Susan’s mind flashed back to the contract on Won’s tablet, to the phone call to Penang. “And was Mr. Won a threat to… shareholder value?”
“Yes.”
“What threat would that be?”
“Excessive devaluation.”
“Was it that phone call to Penang? Who was he talking to?”
“A competitor. He was accepting a job offer.”
“He was leaving the company?” Susan grappled with this new wrinkle, trying to follow how Iris’ cold logic had led her to this moment. She almost had it; the tumblers were all falling into place. She pruned away possibilities in her tree, honing in on one final, impossible possibility.
“He was very unhappy, Doctor Hobbes. He has been fighting with the board over strategy for the last seventeen months. But if he left the company his great-grandfather started, it would cause a tremendous decline in shareholder value, both because of a loss of reputation, and because of the advantage his proprietary knowledge would bring to our competitor. The risk of letting him go was simply too high.”
Susan set the light down and crossed her arms. “And killing Mr. Won was less of a threat? I find that difficult to believe.”
Iris nodded earnestly. “The board would have had no way to stop Mr. Won from leaving, and filing legal action preventing him from taking another job would have caused a public scandal. Discretion was of utmost importance.”
“And your murdering Won won’t be a public scandal?”
Iris lowered her eyes and smiled. Endearing, exactly as designed. “American Robotics would never let word get out of what happened,” she said. “It’s a part of the agreement with TKK Holdings. I’m sure there’s already a press release announcing that Mr. Won has passed away of a heart attack. The company is safe now.”
Fair enough. “Thank you, Iris,” Susan said, and deactivated the unit one last time.
She began mentally composing her final report, searching for the few careful words that would induce the least amount of fear. Not that it would help Iris. She would be dismantled, of course, because humans are reactive and illogical. A pity.
At last Dashiel returned with a steaming cup of coffee and five kinds of pastry. “How’s it going so far? Have you found the problem?”
Susan took a sip. She had a few new thoughts to pass on to the Ultimate Messenger marketing team. They couldn’t use the true details of the case, thanks to extensive mutual nondisclosures and a certain degree of legal liability. But one could do wonders with subtext and hypothetical scenarios.
At last she smiled. “Good news,” she said. “Iris is working perfectly.”
© 2017 Andrea Phillips
About the author
