More on OpenAI and the race towards the AI precipice

The following misguided passage, which I've taken the liberty to translate into English, comes from the final chapter of an otherwise fairly reasonable (for its time) book that came out in Swedish way back in 2021. After having outlined, in earlier chapters, various catastrophic AI risk scenarios, along with some tentative suggestions for how they might be prevented, the author writes this:

Some readers might be so overwhelmed by the risk scenarios outlined in this book, that they are tempted to advocate halting AI development altogether [...]. To them, my recommendation would be to drop that idea, not because I am at all certain that the prospects of further AI development outweigh the risks (that still seems highly unclear), but for more pragmatic reasons. Today's incentives [...] for further AI development are so strong that halting it is highly unrealistic (unless it comes to a halt as a result of civilizational collapse). Anyone who still pushes the halting idea will find him- or herself in opposition to an entire world, and it seems to make more sense to accept that AI development will continue, and to look for ways to influence its direction.

The book is Tänkande maskiner, and the author is me. Readers who have been following this blog closely may have noticed, for instance in my lecture on ChatGPT and AI alignment from December 16 last year, that I've begun to have a change of heart on the questions of whether (parts of) AI development can and should be halted or at least slowed down. I still think that doing so by legislative or other means is a precarious undertaking, not least in view of international race dynamics, but I have also come to think that the risk landscape has become so alarming, and that technical solutions to AI safety problems are so likewise precarious, that we can no longer afford to drop the idea of using AI regulation to buy ourselves time to find these solutions. For those of you who read Swedish, the recent debate in Swedish newspaper Göteborgs-Posten on the role of the leading American AI developer OpenAI in the race towards AGI (artificial general intelligence) further underlines this shift in my opinion:

• O. Häggström: Ledande AI-företag spelar rysk roulette med mänsklighetens överlevnad, GP, February 18, 2023.
• D. Dubhashi, M. Johansson and D. Sumpter: Sprid inte skrämselpropaganda om artificiell intelligens, GP, February 24, 2023.
• O. Häggström: Forskareliten är övertygad om att AI snart kan bli smartare än människan, GP, February 27, 2023.

In the middle of this debate, the highly relevant document Planning for AGI and beyond was released by OpenAI on February 24 - a document in which their CEO Sam Altman spelled out the company's view about their participation in the race towards creating AGI and the risks and safety aspects involved. I could have discussed this document in my final rejoinder of the GP debate, but decided against that for reasons of space. But now let me tell you what I think about it. My initial reaction was a sequence of thoughts, roughly captured in the following analogy:

Imagine ExxonMobil releases a statement on climate change. It’s a great statement! They talk about how preventing climate change is their core value. They say that they’ve talked to all the world’s top environmental activists at length, listened to what they had to say, and plan to follow exactly the path they recommend. So (they promise) in the future, when climate change starts to be a real threat, they’ll do everything environmentalists want, in the most careful and responsible way possible. They even put in firm commitments that people can hold them to.

An environmentalist, reading this statement, might have thoughts like:

• Wow, this is so nice, they didn’t have to do this.
• I feel really heard right now!
• They clearly did their homework, talked to leading environmentalists, and absorbed a lot of what they had to say. What a nice gesture!
• And they used all the right phrases and hit all the right beats!
• The commitments seem well thought out, and make this extra trustworthy.
• But what’s this part about “in the future, when climate change starts to be a real threat”?
• Is there really a single, easily-noticed point where climate change “becomes a threat”?
• If so, are we sure that point is still in the future?
• Even if it is, shouldn’t we start being careful now?
• Are they just going to keep doing normal oil company stuff until that point?
• Do they feel bad about having done normal oil company stuff for decades? They don’t seem to be saying anything about that.
• What possible world-model leads to not feeling bad about doing normal oil company stuff in the past, not planning to stop doing normal oil company stuff in the present, but also planning to do an amazing job getting everything right at some indefinite point in the future?
• Are they maybe just lying?
• Even if they’re trying to be honest, will their bottom line bias them towards waiting for some final apocalyptic proof that “now climate change is a crisis”, of a sort that will never happen, so they don’t have to stop pumping oil?
This is how I feel about OpenAI’s new statement, Planning for AGI and beyond.

Well, to be honest, this is not actually my own reaction, but my favorite AI blogger Scott Alexander's. However, since my initial reaction to OpenAI's document was close to identical to his, it seemed fine to me to just copy-and-paste his words. They are taken from his brand new blog post in which he, after laying out the above gut reaction, goes into a well-reasoned, well-informed and careful discussion of what parts of the gut reaction are right and what parts are perhaps less so. I know I have an annoying habit of telling readers of this blog about further stuff that they absolutely need to read, but listen, this time I really mean it: in order to understand the (probably) by far most important and consequential action going on right now in the world, you must read OpenAI's statement followed by Scott Alexander's commentary.