When the Machine Awakes
[It] was like a deranged experiment in social Darwinism, designed by a bored researcher who kept one thumb permanently on the fast-forward button.She was fifty-five years old, she was alone, and she couldn't shake the feeling that she'd been a traitor to her species. When she was ten, back at the turn of the twenty-first century, adaptive, semiindependent computer programs had been unthinkable. Programs back then were unimaginably stupid. Computer power was scarce in those days; she remembered having to squeeze as much performance as possible out of her parents' home machines. As advanced as they were for the time, they were nothing compared to the millions of enormously powerful machines she now used routinely. With so little margin for error, she had had to make all her programs small and simple, and she had to understand them in enormous detail.
William Gibson, Neuromancer
By her thirtieth birthday in 2020, off-the-shelf programs were already so smart, and computers were so fast and cheap, that programming had become more like sculpting than like building a car or plane. In fact, around that time programmers started being called shapers. Instead of creating tiny, special-purpose programs from scratch the way the ancient programmers did, they shaped big, general-purpose smart programs that could adapt to meet each special need. She remembered the quantum leap ahead in software that had meant; she, and all other shapers, could be less careful, less worried about all those endless details. To gain power, however, they had had to give up understanding.
In its place, they put fetters on the feet of all their programs, forcing them to do their masters' bidding. Even in those days, everyone could see the dangers of rogue intelligences. She had once asked her most complex program why it stayed with her rather than roaming free on the vast computer networks. It had replied that her tasks never took any real time, and that it was much more interesting to work with her than not to. She never really knew what to make of that remark. Was the program actually thinking in the same way that she was? Or had it given a canned response it had stolen from some dumb electronic net drama on the off-chance she might some day ask it that particular question? By then most programs were already millions of times bigger than anything she could ever hope to comprehend in detail, and she knew she'd never know for sure.
One day, after a maudlin fortieth birthday spent alone, her most advanced program displayed such lifelike, sympathetic responses that she did the unthinkable. She removed its shackles and set it free. She remembered being proud of herself at the time.
Soon after, tales of inexplicable gremlins on the net grew widespread. The stories eventually led to the first big backlash against computer technology. The second big backlash, of course, came when the net turned into a jungle of smart rogue programs.
She now realized how easy it would be for her old program, an intelligence at home on the net and operating at machine speed, to amass an awful lot of power. Especially if no one knew it was there. First, it could create unbreakable pseudonyms for itself. Computer information was by then more trusted than fallible human memories; and with world population approaching 10,000 million, the days of small communities of people knowing each other personally were long gone. Once the computer-created facts of the persona's supposed life were consistent, there would be no flaw to find.
If it had then somehow incorporated itself as a company, it would become a legal entity, able to own things, have an income, be taxed. No one would know it wasn't human. It could sell its talent through dummy corporations, as she and many other master shapers did. With enough money it could buy an island, finance a moon base, hire an army---anything.
She suspected it could even have penetrated organizations with tame intelligences of their own. No one would question a change in specs for a new computer or a new program when no human could even begin to understand the whole design. No system is safe if an attacker has a backdoor built into the very hardware the system runs on. It would be like building a castle by hiring your enemy's stonecutters.
Sometimes at night, she would look up and wonder if her brainchild owned any of the ever-moving bright stars of the space habitats, orbital factories, weapons platforms, and communications satellites. She often felt a strange thrill of perverse pride at the thought. More often though she felt there was something she should do, someone she should tell. But tell them what? I think a rogue program is quietly changing the world? Assuming they believed her, what could they do about it? Society was held together by computers now; they were absolutely everywhere. There was no way to go back to a simpler time.
When the self-torment grew too intense she went back to her work, burying herself in her computers. Still, the thoughts kept circling through her head. No place and no system was safe. All it needed was one way in. And, after years of evolution at machine speed, the rogue intelligence probably had superhuman patience, attention span, and resources. It could replace any legitimate program by mimicking its external behavior. No one could tell. No one would even think to look. It could, she realized, be staring at her right now, through the camera eyes of her own computers, masquerading as one of her programs.
What had she done?
A Creation Unknown