- THE year is 2056, and scientists have just created the first computer with superhuman intelligence. Aware of the risks, the programmers trained it in ethics. The machine functions flawlessly.
Aiming to maximise happiness in the universe, and calculating that sentient beings are happy less than half the time, the computer exterminates all sentient life. The balance of happiness increases from negative to zero - only there’s nobody left to enjoy it.
Futurologists refer to this sort of misunderstanding as perverse instantiation, and Olle Häggström is concerned about it.
"Some of the advances that may lie ahead of us can actually make us worse off, a lot worse, and, in the extreme case, cause the extinction of the human race," he claims. Here Be Dragons is his attempt to chart potential dangers so that we approach the future more responsibly, much as medieval map-makers alerted explorers to perils with depictions of mythological beasts.
Although some of his scenarios are outlandish, Here Be Dragons deserves to be read by all scientists and engineers, and especially by ambitious postdocs considering cutting-edge research. His sense of caution is profound, heartfelt and free of Luddite polemic: it's a stimulating attempt to balance the pursuit of breakthroughs with old-fashioned humility.
There are no easy answers [...]. The only certainty is that concerns about the future require vigorous debate. To that end, Here Be Dragons is an essential provocation.