New AI paper with James Miller and Roman Yampolskiy

The following words by Alan Turing in 1951, I have quoted before:

My contention is that machines can be constructed which will simulate the behaviour of the human mind very closely. [...] Let us now assume, for the sake of argument, that these machines are a genuine possibility, and look at the consequences of constructing them. [...] It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control.

It is but a small step from that ominous final sentence about machines taking over to the conclusion that when that happens, everything hinges on what they are motivated to do. The academic community's reaction to Turing's suggestion was a half-century of almost entirely ignoring it, and only the last couple of decades have seen attempts to seriously address the issues that it gives rise to. An important result of the early theory-building that has come out of this work is the so-called Omohundro-Bostrom framework for instrumental vs final AI goals. I have discussed it, e.g., in my book Here Be Dragons and in a 2019 paper in the journal Foresight.

Now, in collaboration with economist James Miller and computer scientist Roman Yampolskiy, we have another paper on the same general circle of ideas, this time with emphasis on the aspects of Omohundro-Bostrom theory that require careful scrutiny in light of the game-theoretic considerations that arise for an AI living in an environment where it needs to interact with other agents. The paper, whose title is An AGI modifying its utility function in violation of the strong orthogonality thesis, is published in the latest issue of the journal Philosophies. The abstract reads as follows:

An artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable and that was influenced by how other agents chose to interact with it. This instrumental drive would conflict with the strong orthogonality thesis since the modifications would be influenced by the AGI’s intelligence. AGIs in highly competitive environments might converge to having nearly the same utility function, one optimized to favorably influencing other agents through game theory. Nothing in our analysis weakens arguments concerning the risks of AGI.

Read the full paper here!

My conversation with James Miller on the Swedish covid-19 strategy

There has been much talk the last few days about the interview that Swedish Minister for Foreign Affairs Ann Linde did with German broadcaster Deutsche Welle about the Swedish covid-19 strategy. The combination of arrogance and ignorance she displayed made the interview painful to watch.

Last Thursday American economist James Miller and I recorded an episode of his podcast Future Strategist on the same topic. The episode can now be heard on Soundcloud or Poddtoppen or whatever is your favorite pod platform. I tried to give a fair and level-headed account of the Swedish approach to covid-19 and how it differs from those of other countries, but I'll leave it to the listener to judge whether I've mitigated the damage to Sweden's international reputation done by Linde, or if I exasperated the situation even further. Either way, the sound quality was unfortunately not great.

An essential collection on AI safety and security

The xxix+443-page book Artificial Intelligence Safety and Security, edited by Roman Yampolskiy, has been out for a month or two. Among its 28 independent chapters (plus Yamploskiy's introduction), which have a total of 47 different authors, the first 11 (under the joint heading Concerns of Luminaries) have previously been published, with publication years ranging from 2000 to 2017, while the remaining 17 (dubbed Responses of Scholars) are new. As will be clear below, I have a vested interest in the book, so the reader may want to take my words with a grain of salt when I predict that it will quickly become widely accepted as essential reading in the rapidly expanding and increasingly important fields of AI futurology, AI risk and AI safety; nevertheless, that is what I think. I haven't yet read every single chapter in detail, but have seen enough to confidently assert that while the quality of the chapters is admittedly uneven, the book still offers an amazing amount of key insights and high-quality expositions. For a more systematic account by someone who has read all the chapters, see Michaël Trazzi's book review at Less Wrong.

Most of the texts in the Concerns of Luminaries part of the book are modern classics, and six of them were in fact closely familiar to me even before I had opened the book: Bill Joy's early alarm call Why the future doesn't need us, Ray Kurzweil's The deeply intertwined promise and peril of GNR (from his 2005 book The Singularity is Near), Steve Omohundro's The basic AI drives, Nick Bostrom's and Eliezer Yudkowsky's The ethics of artificial intelligence, Max Tegmark's Friendly artificial intelligence: the physics challenge, and Nick Bostrom's Strategic implications of openness in AI development. (Moreover, the papers by Omohundro and Tegmark provided two of the cornerstones for the arguments in Section 4.5 (The goals of a superintelligent machine) of my 2016 book Here Be Dragons.) Among those that I hadn't previously read, I was struck most of all by the urgent need to handle the near-term nexus of risks connecting AI, chatbots and fake news, outlined in Matt Chessen's The MADCOM future: how artificial intelligence will enhance computational propaganda, reprogram human cultrure, and threaten democracy... and what can be done about it.

The contributions in Responses of Scholars offer an even broader range of perspectives. Somewhat self-centeredly, let me just mention that three of the most interesting chapters were discussed in detail by the authors at the GoCAS guest researcher program on existential risk to humanity organized by Anders Sandberg and myself at Chalmers and the University of Gothenburg in September-October last year: James Miller's A rationally addicted artificial superintelligence, Kaj Sotala's Disjunctive scenarios of catastrophic AI risk, and Phil Torres' provocative and challenging Superintelligence and the future of governance: on prioritizing the control problem at the end of history. Also, there's my own chapter on Strategies for an unfriendly oracle AI with reset button. And much more.

Videos from the existential risk workshop

The public workshop we held on September 7-8 as part of the ongoing GoCAS guest researcher program on existential risk exhibited many interesting talks. The talks were filmed, and we have now posted most of those videos on our own YouTube channel. They can of course be watched in any order, although to maximize the illusion of being present at the event, one might follow the list below, in which they appear in the same order as in the workshop. Enjoy!

• Anders Sandberg: Tipping points, uncertainty and systemic risks: what to do when the whole is worse than its parts?
• Karin Kuhlemann: Complexity, creeping normalcy, and conceit: Why certain catastrophic risks are sexier than others
• Phil Torres: Agential risks: Implications for existential risk reduction
• Karim Jebari: Resetting the tape of history
• Due to a technical mishap, we have no video for David Denkenberger's talk on Cost of non-sunlight dependent food for agricultural catastrophes. Try instead watching his talk Feeding everyone no matter what given at CSER in Cambridge last year, which covers much of the same ground.
• Thore Husfeldt: Plausibility and utility of apocalyptic AI scenarios
• Roman Yampolskiy: Artificial intelligence as an existential risk to humanity
• Stuart Armstrong: Practical methods to make safe AI
• Robin Hanson: Disasters in the Age of Em and after
• Katja Grace: Empirical evidence on the future of AI
• James Miller: Hints from the Fermi paradox for surviving existential risks
• Catherine Rhodes: International governance of existential risk
• Seth Baum: In search of the biggest risk reduction opportunities

Workshop on existential risk to humanity

Come to Gothenburg, Sweden, for the Workshop on existential risk to humanity that I am organizing on September 7-8 this year! The topic is a matter of life and death for the human civilization, but I still expect the workshop to be stimulating and enjoyable. The list of world-class researchers who are confirmed as speakers at the workshop include several who are already known to readers of this blog: Stuart Armstrong, Seth Baum, Robin Hanson, James Miller, Anders Sandberg and Roman Yampolskiy. Plus the following new aquaintances: Milan Cirkovic, David Denkenberger, Karin Kuhlemann, Catherine Rhodes, Susan Schneider and Phil Torres. All of these will in fact stay in town for various durations before and/or after the workshop, at the guest research program on existential risk that runs throughout September and October.

The workshop is open to all interested, and participation is free of charge. Pre-registration, however, is required.

Singulariteten ur ett ekonomiskt perspektiv

Vad händer när datorer och robotar blir lika intelligenta som vi människor, eller ännu intelligentare? Redan datalogins fader Alan Turing såg framför sig en sådan utveckling, som framgår av följande citat ur hans uppsats Intelligent Machinery: A Heretical Theory från 1951:

My contention is that machines can be constructed which will simulate the behaviour of the human mind very closely. [...] To do so would of course meet with great opposition, unless we have advanced greatly in religious toleration from the days of Galileo. There would be great opposition from the intellectuals who were afraid of being put out of a job. It is probable though that the intellectuals would be mistaken about this. There would be plenty to do, trying to understand what the machines were trying to say, i.e. in trying to keep one’s intelligence up to the standard set by the machines, for it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s 'Erewhon'.

Många tänkare har sedan dess varit inne på liknande idéer. Ett särskilt vanligt inslag har varit tanken att utvecklingen, så snart den kommit över en viss kritisk nivå, kan komma att gå extremt snabbt. Matematikern Jack Good¹ var 1965 tidigt ute med den tankegången, vilken senare har kommit att betecknas Singulariteten:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.

Jag anser (som bekant är för trogna läsare av denna blogg) att idén om en eventuell kommande Singularitet är värd att ta på allvar. Den bästa introduktionen till Singulariteten har enligt min mening David Chalmers skrivit, och jag kan vidare rekommendera det temanummer av Journal of Consciousness Studies som jag uppmärksammade för ett år sedan, med reaktioner på Chalmers uppsats. Men Singulariteten är ett kraftigt underutforskat område² som förtjänar att belysas ur många olika synvinklar. Därför är det ytterst glädjande att en amerikansk nationalekonom vid namn James Miller nyligen utkommit med boken Singularity Rising, där han applicerar ett nationalekonomiskt perspektiv och dito metod på Singulariteten och på ett knippe besläktade så kallat transhumanistiska scenarier. Boken är mycket intressant och samtidigt mäkta irriterande. För en närmare beskrivning och några reflektioner hänvisar jag till min recension i nyutkomna Ekonomisk Debatt 2/2013.

Fotnoter

1) Av Wikipedia får vi bl.a. veta att Good arbetade tillsammans med Turing, både under kriget (med knäckandet av tyskarnas Enigma-kod) och senare.

2) En jämförelse med klimatfrågan är slående. Singularitetens potentiella konsekvenser är knappast mindre än klimatförändringarnas, men är inom såväl forskning som publik diskussion mindre uppmärksammat - jag tror knappast att jag överdriver om jag säger att det skiljer en faktor 1000 i hur mycket kraft vi ägnat åt att utforska och diskutera dessa båda ohyggligt betydelsefulla orosmoln på mänsklighetens horisont.