If It Can Go Wrong, It Will
There's a miasma of unreadiness surrounding computer hardware and software. We see it in hardware, where it's still not unheard of to buy a computer system and have to assemble it. Would you buy a car or washing machine in parts? Yet the problem is even more obvious in software.Two stories illustrate the unforeseen things that can go wrong. The software guiding a torpedo included a test designed to prevent the weapon from accidentally returning to its launching ship by checking whether it had turned around. If it had, it was supposed to blow itself up. On one test with a live warhead, however, the torpedo's motor failed on launch and the torpedo stuck in its tube. When the ship turned about to return to port, the torpedo detected that it had turned around and blew itself up---inside the ship.
A similar design oversight occurred in the subsystem of a plane-landing software system for guiding the aircraft along a beam on approach to the runway. If the plane lost the beam, it was programmed to apply power to fly around for a retry. Another subsystem took control when the plane was close to the ground, cutting the engine and raising the nose for landing. Both subsystems worked perfectly during tests. In the first live test, however, the plane flew down the beam until it was a few meters above the runway. At that point the second subsystem stepped in and cut the engine in preparation for landing. Now flying unpowered, the plane started to sink, which made it lose the beam. So the first subsystem cut in again and applied power, and the plane ploughed into the ground.
Today's software often has these kinds of teething problems. When we buy most consumer products---toasters, cameras, refrigerators---we expect them to work as advertised. And they usually do. Not so today's software. Whereas most consumer products come with a short warranty guaranteeing performance for a certain time, today's software comes with a long disclaimer disavowing responsibility for its ever doing anything at all.
The trouble is that in a complex system too many things can interact, and so too many things can go wrong. No programmer can predict them all. Today's computers can't help us either, because they don't understand what we want---because we don't understand what we want. Nor can they tell us that we're bungling a program because---so far---they can only do what we tell them to do, and we don't see the error. If we did, we would simply fix it. There must be a better way to program computers.
Or is there?
Eyeless in Gaza