- The prolific bank robber Willie Sutton, when asked why he robbed banks, reportedly replied, "Because that's where the money is." When it comes to AI, I would say that the most important issues are about extremely powerful smarter-than-human Artificial Intelligence (aka superintelligence) because that's where the utilons are - the value at stake. More powerful minds have bigger real-world impacts.
Within the issues of superintelligence, the most important (again following Sutton's Law) is, I would say, what Nick Bostrom termed the "value loading problem": how to construct superintelligences that want outcomes that are high-value, normative, beneficial for intelligent life over the long run - that are, in short, "good" - since if there is a cognitively powerful agent around, what it wants is probably what will happen.
Here are some brief arguments for why building AIs that prefer "good" outcomes is (a) important and (b) likely to be technically difficult.
First, why is it important that we try to create a superintelligence with particular goals? Can't it figure out its own goals?
As far back as 1739, David Hume observed a gap between "is" questions and "ought" questions, calling attention in particular to the sudden leap between when a philosopher has previously spoken of how the world is and then begins using words like should, ought, or ought not. From a modern perspective, we'd say that an agent's utility function (goals, preferences, ends) contains extra information not given in the agent's probability distribution (beliefs, world-model, map of reality).
If in 100 million years we see (a) an intergalactic civilization full of diverse, marvelously strange intelligences interacting with one another, with most of them happy most of the time, then is that better or worse than (b) most available matter having been transformed into paperclips? What Hume's insight tells us is that if you specify a mind with a preference (a) > (b), we can follow back the trace of where the > (the preference ordering) first entered the system and imagine a mind with a different algorithm that computes (a) < (b) instead. Show me a mind that is aghast at the seeming folly of pursuing paperclips, and I can follow back Hume's regress and exhibit a slightly different mind that computes < instead of > on that score too.
I don't particularly think that silicon-based intelligence should forever be the slave of carbon-based intelligence. But if we want to end up with a diverse cosmopolitan civilization instead of, for example, paperclips, we may need to ensure that the first sufficiently advanced AI is built with a utility function whose maximum pinpoints that outcome. If we want an AI to do its own moral reasoning, Hume's Law says we need to define the framework for that reasoning. This takes an extra fact beyond the AI having an accurate model of reality and being an excellent planner.
But if Hume's Law makes it possible in principle to have cognitively powerful agents with any goals, why is value loading likely to be difficult? Don't we just get whatever we programmed?
The answer is that we get what we programmed, but not necessarily what we wanted. The worrisome scenario isn't AIs spontaneously developing emotional resentment for humans. It's that we create an inductive value learning algorithm and show the AI examples of happy smiling humans labeled as high-value events - and in the early days the AI goes around making existing humans smile and it looks like everything is OK and the methodology is being experimentally validated; and then, when the AI is smart enough, it invents molecular nanotechnology and tiles the universe with tiny molecular smiley-faces. Hume's Law, unfortunately, implies that raw cognitive power does not intrinsically prevent this outcome, even though it's not the result we wanted.
For now, the value loading problem is unsolved. There are no proposed full solutions, even in principle. And if that goes on being true over the next decades, I can't promise you that the development of sufficiently advanced AI will be at all a good thing.
- 1. Scaremongering: Fear boosts ad revenues and Nielsen ratings, and many journalists appear incapable of writing an AI-article without a picture of a gun-toting robot.
2. "It's impossible": As a physicist, I know that my brain consists of quarks and electrons arranged to act as a powerful computer, and that there's no law of physics preventing us from building even more intelligent quark blobs.
3. "It won't happen in our lifetime": We don't know what the probability is of machines reaching human-level ability on all cognitive tasks during our lifetime, but most of the AI researchers at a recent conference put the odds above 50 percent, so we'd be foolish to dismiss the possibility as mere science fiction.
4. "Machines can't control humans": Humans control tigers not because we are stronger, but because we are smarter, so if we cede our position as smartest on our planet, we might also cede control.
5. "Machines don't have goals": Many AI systems are programmed to have goals and to attain them as effectively as possible.
6. "AI isn't intrinsically malevolent": Correct - but its goals may one day clash with yours. Humans don't generally hate ants - but if we want to build a hydroelectric dam and there's an anthill there, too bad for the ants.
7. "Humans deserve to be replaced": Ask any parent how they would feel about you replacing their child by a machine, and whether they'd like a say in the decision.
8. "AI worriers don't understand how computers work": This claim was mentioned at the above-mentioned conference, and the assembled AI researchers laughed hard.
- The Singularity - the fateful moment when AI surpasses its creators in intelligence and takes over the world - is a meme worth pondering. It has the earmarks of an urban legend: a certain scientific plausibility ("Well, in principle I guess it's possible!") coupled with a deliciously shudder-inducing punch line ("We'd be ruled by robots!"). Did you know that if you sneeze, belch, and fart all at the same time, you die? Wow. Following in the wake of decades of AI hype, you might think the Singularity would be regarded as a parody, a joke, but it has proven to be a remarkably persuasive escalation. Add a few illustrious converts - Elon Musk, Stephen Hawking, and David Chalmers, among others - and how can we not take it seriously? Whether this stupendous event takes place ten or a hundred or a thousand years in the future, isn't it prudent to start planning now, setting up the necessary barricades and keeping our eyes peeled for harbingers of catastrophe?
I think, on the contrary, that these alarm calls distract us from a more pressing problem...
- I do not believe that machines that think exist, or that they are likely to exist in the foreseeable future. If I am wrong, as I often am, any thoughts I might have about the question are irrelevant.
If I am right, then the whole question is irrelevant.