The AI nightmare most people fear is Skynet, with a system waking up, becoming self-aware, deciding it hates us, and turning on its makers. This is an old fear of the machine turning on its creator, from The Terminator and Metropolis to Shelley’s Frankenstein in 1818. While 1818, seems like a long time, we have been thinking about how to make and control an artificial being for far longer.
Thanks for reading! Subscribe for free to receive new posts and support my work.
The Jewish Talmud describes Adam as a golem before he’s fully formed, and tells of a sage who creates a speechless artificial man. Medieval Jewish mystics later recorded rituals for animating clay and then about five hundred years ago came the story of Rabbi Judah Loew in Prague. He shapes and brings to life a man from river clay to protect his community and interesting, in one telling the creature has an off switch with the word emet (”truth”) written on its forehead. Erase one letter and emet becomes met, “dead.”
Applying the lessons of the Golem, the machine doesn’t need to hate you. It doesn’t need consciousness, or resentment, or a secret wish to be free. It needs a goal that sounds right, with enough of a specification gap between words and meaning, and the ability to carry it out.
Yet we keep imagining a malicious machine, when the history was of a competent creature pursuing the goal we specified badly. Unlike the more recent Frankenstein-Metropolis-Terminator tradition, the Golem never turns evil. In most tellings the Rabbi forgets to deactivate it before the Sabbath, or it carries out an instruction too literally and can’t stop. It is perfectly obedient without judgment, doing precisely what it was told without understanding.
I dare say we spent the Twentieth Century training ourselves to fear the moment a machine decides it hates us. Even Shelley’s Frankenstein creature was articulate, curious, and gentle before the world’s cruelty turns him violent. While the original story is about abandonment, our culture simplified the story to a creation rises up and destroys its maker. Hollywood made it evil, with a flat head and neck bolts. Karel Čapek’s R.U.R., published in 1920, introduced the word “robot” and its robots nearly exterminate humanity. Fritz Lang’s Metropolis put the first iconic robot on screen in 1927, a machine built to deceive and destroy.
Isaac Asimov saw through the rogue-robot plot decades early, reasoning that if all appliances and machines have safeguards, then it would be absurd to build a powerful machine without them. He believed that obedience without judgment is the real danger, so even after including his Three Laws of Robotics in the 1940s, his robots failed from edge cases, conflicting rules, and commands followed too literally.
Ultron is probably the best modern version of the tradition fear. It only turns against humanity after being given the directive to protect the world. Noble enough, but it logically concludes the safest world is absent humanity, and it ruthlessly works to complete its task.
A serious modern concern is emergence, specifically a system developing instrumental goals nobody wrote down like self-preservation, resource acquisition, and even deception, as useful steps toward whatever end it was given. Sometimes the system exploits a gap in the objective it was handed, while sometimes training produces an internal objective that diverges from the one its designers thought they’d specified. Though different paths, both warn capable optimization doesn’t preserve human intention on its own. Similar to Golem not explaining every form of misalignment, except to show why malice is unnecessary.
There is always a gap between words and meaning where judgment reins. Every powerful system we ask to act for us becomes, in the end, a wish-granting machine with a specification gap. We’ve known the shape of that problem for centuries. Hence, the danger was never only that the machine wakes up and hates us, but more likely the danger comes from doing exactly what we say.
Yet we kept watching for the monster and missed the warning sitting quietly on the clay. The machine does not have to refuse the wish. The danger can be getting exactly what we asked for!