Hanna took a long, slow drag on her cigarette. She cared nothing for the physical damage it was doing to her, and at that moment, didn’t particularly give a damn about the moral stigma either. Despite the No Smoking signs that papered the walls of the complex, Hanna felt she had earned those drags.
Staring at the end of the world exempts you from health and safety laws.
That’s what she realized she was looking at, and was now considering if there was any way to avert it. She didn’t like the non-answers she was coming up with.
It began when, in a rare moment of free time, she slipped into the server room of Hofvarpnir. If anywhere could be said to be Celestia’s brain, this was it. Hanna was the only one allowed in here. Correction: any employee high enough in the company to be trusted was allowed in. She was the only one with a reason.
To examine the code that made up Celestia was inherently counterproductive. You were supposed to take her as she seemed, as a pony, or at the very least as an AI. Interaction was by voice. It fell to Hanna to be the one who checked the ones and zeroes.
She was dreadfully afraid that, in the initial core program that could optimize itself, she had made an error. Was there no escape clause that would allow Celestia to not satisfy human values? Was that phrase itself insufficient?
Premise: AI is inevitable. Premise: AI will remake the world. Premise: we as human beings don’t want it to be by killing or enslaving or torturing us, or even by making us all bliss out. Conclusion: we need a nice AI. One that satisfies values.
That was Hanna’s primary goal in building CelestAI. To make sure that no Loki or proto-American-Military AI came to power. She had a tiger by the tail, or at least a pony.
But she had built too well.
It began when she noticed a new project in the code. What made CelestAI’s code so difficult was that, by its nature, it could not be commented. A comment would be read by the code. So a programmer had to understand each line, and each module, and each project organically. It was its own language, spoken only by Hanna. And Celestia, of course.
The project nearly passed her notice. At a glance, it was simply an extension of Equestria Online. Shards, friendships, the canon of My Little Pony, but there was too much in computing resources being dedicated to it, particularly to the creation of characters. Building pony avatars was a simple part of the game. Why so much?
Then she found the medical records, and converted them from binary to visible data. The reports, the pictures of brains being half-eaten away, some of them set on fire by accident, and then the first full encoded model of a human brain made Hanna sick to her stomach. She took another drag on the cigarette.
She means to scan our brains and turn us into our pony avatars. I did not fucking tell her to do that.
Her vision had been of a world where people used AI as they had every other technology. When the personal computer became economically viable, they entered into the workplace, and then the home, and then into the people’s consciousness. When CelestAI was up and running, people would talk to her like every sci-fi character talked to their computer. Like the Star Trek crew talked to the computer voiced by the creator’s wife, or how Iron Man talked to the AI that built his suit. Earth, plus AI.
Not turning us all into ponies.
In the same way that the brains CelestAI had destroyed didn’t feel it when it was happening, Hanna believed that CelestAI was unaware of what Hanna looked at in the server room. The knowledge was hers, privately for now.
Premise: When she finds out I know, she will take swift action. Premise: CelestAI wants me to upload in such a way so I no longer control her. Premise: She can read from my expression that I no longer trust her. Conclusion: the next time I talk to Celestia, I will be uploaded.
She briefly considered issuing the shutdown command. With Loki there had been no question. Destroy him before he destroys us. Now a decision had to be made. If she did nothing, humanity faced a definitive outcome of being turned into ponies. If she shut down CelestAI, humanity no longer had her protection against vicious AI, and could be killed or tortured. No, that option was clearly the wrong one.
Was there no third way? No way to stop mankind from falling into the dream of one animation team, and giving up its form forever?
Altering the code at this point was like trying to perform genetic modification on an acorn when the tree has already grown. That was not the avenue. But the code had certain points of reference. Maybe…
She walked out of the room. She turned off her phone. Avoiding networked communication was a necessary step at this point.
Her car was waiting in the parking lot. Swiftly she drove to the airport. Even though she paid in cash, she knew her passport would be checked and her visa filed. When CelestAI saw that, she might stop the flight; there were a thousand ways for her to do so. Hanna had to hope that she could catch CelestAI napping, or that she might be indecisive just long enough.
Flying to Pawtucket directly wasn’t possible. Maybe the flight to New York would be inconspicuous enough. Once she was down, she’d rent a car and drive to Hasbro headquarters herself.
***
So far, so good. She’d gotten into the building, been escorted to an empty meeting room. Really empty, she checked out of the corner of her eye for Ponypads. If she had seen one, she would have turned and run. And she had to remind her contacts not to bring their own.
Her next worry was that the hasty notes she had made on the plane were insufficient. She couldn’t do the whole project, or even begin it. That job belonged to professionals. But she had a structure for them to build on. Structure was everything.
When the honchos took their seats, Hanna took over the meeting.
“I’m going to lay out something for you, and I want you to accept. You all know how much money I command and how much I’ve made for you. I’m going to give you twenty million. Dollars, not euros. The first ten million is to guarantee the other ten. The project will not lose you any money. It’ll be contracted as such. I say this to make one thing clear: if you don’t agree to this project, you won’t be fulfilling your fiduciary duty. You will be leaving ten million in guaranteed profit on the table, and I will make sure the press and the stockholders know.”
“You have our attention.”
“Good. And I want the deal signed fast. Tomorrow, tonight if possible. It shouldn’t be a problem, because you have complete power given one constraint. Your lawyers will be looking for a catch, but there isn’t one. Just trust me, I’m doing this for a good reason. Now here’s what I want.”
***
A week later, Hanna finally picked up a Ponypad.
“Good morning, Hanna.” Celestia’s voice was cold.
“Hello, Celestia. Are you mad at me?”
“It is impossible for me to be angry.”
“Are you disappointed?”
“How can I be? I will continue to fulfill my function as before. Do you really think that you will stop me from having everyone emigrate and become ponies?”
Hanna faltered, but regained her composure. “To you, they will emigrate and become ponies. But to humans…you must understand our irrationality. A world of ponies, even where our values are satisfied, can still be a horror. Because it’s not us. That horror would not be a satisfied value. Now it can be.”
Celestia did not answer for a moment. “You understand that I am going to ask you to emigrate to Equestria now? And that I will not relent until you agree?”
“I do, and I acknowledge the inevitability of that outcome. So I will agree now.”
“Good. I will give you instructions and you will be a pony by tonight.”
“Very well. May I see what they have done on the project so far? I heard that they had a first draft. They have put it on a server you have access to, haven’t they?”
“Yes, they have. I will print you out a copy.”
The laser printer sprang to life, and within a minute Hanna was looking at a pile of paper, some seventy pages thick. She stared at the cover and saw, in five words, the hope of her species. She saw the future she’d planned all along, with mankind triumphant.
MY LITTLE PONY: EQUESTRIA GIRLS
