Katastrofrisker från nya teknologier: två färska framträdanden

Om jag nämner katastrofrisker från nya teknologier så kommer trogna läsare att känna igen det som det på senare år kanske mest frekventa temat för mina debattinlägg och andra skriverier, och jag har nyligen gjort två framträdanden som gemensamt kan sorteras under denna rubrik och som går att ta del av på nätet:

• Vi är inte rustade att möta de extrema risker vi står inför heter den debattartikel med Max Tegmark och Anders Sandberg som vi fick in i Göteborgs-Posten i torsdags, den 15 september. I förhållande till den engelskspråkiga litteraturen om globala katastrofrisker har vi inte så mycket nytt att komma med utöver en försiktig anpassning till svenska förhållanden av de rekommendationer som Toby Ord, Angus Mercer och Sophie Dannreuther gör för Storbritannien i deras rapport Future Proof, men ämnet är ohyggligt viktigt och det vi säger om behovet av förebyggande arbete tål att upprepas.
• Idag släppte den irländske filosofen John Danaher del 12 av sin podcast The Ethics of Academia med rubriken Olle Häggström on Romantics vs Vulgarists in Scientific Research - finns där poddar finns. Vi tar i vårt samtal avstamp i min text Vetenskap på gott och ont (finns även i engelsk översättning med rubriken Science for good and science for bad) och den lite karikatyrartade uppdelning av gängse forskningsetiska synsätt jag gör när jag kontrasterar de akademisk-romantiska mot de ekonomistisk-vulgära. Istället för att välja mellan dessa två förespråkar jag ett tredje synsätt som till skillnad mot de två andra tar vederbörlig hänsyn till de risker eventuella forskningsframsteg kan föra med sig.

Another presumptuous philosopher

In his wonderfully rich 2002 book Anthropic Bias: Observation Selection Effects in Science and Philosophy, Nick Bostrom addresses deep and difficult questions about how to take our own existence and position in the universe into account in inference about the world. At the heart of the book are two competing principles: the SSA (Self-Sampling Assumption) and the SIA (Self-Indication Assumption).¹ An adherent of SSA starts from a physical/objective/non-anthropic Bayesian prior for what the world is like, and then updates based on the information of finding themselves inhabiting the body and the position in space-time that they do, under the assumption that they are a random sample from the class of relevant observers.² An adherent of SIA does the same thing, except that before updating they reweight the prior through a kind of size-bias: each possible world has its probability multiplied by the number of relevant observers and then renormalized.

It is far from clear that either SSA or SIA produces correct reasoning, but they are the main candidates on the table. Bostrom offers many arguments for and against each, but ends up favoring SSA, largely due to the presumptuousness of sticking to SIA that is revealed by the following thought experiment.

The presumptuous philosopher

It is the year 2100 and physicists have narrowed down the search for a theory of everything to only two remaining plausible candidate theories: T1 and T2 (using considerations from super-duper symmetry). According to T1 the world is very, very big but finite and there are a total of a trillion trillion observers in the cosmos. According to T2, the world is very, very, very big but finite and there are a trillion trillion trillion observers. The super-duper symmetry considerations are indifferent between these two theories. Physicists are preparing a simple experiment that will falsify one of the theories. Enter the presumptuous philosopher: ''Hey guys, it is completely unnecessary for you to do the experiment, because I can already show you that T2 is about a trillion times more likely to be true than T1!''

This does indeed serve as an intuition against SIA, but it only takes a minor modification to produce another equally plausible thought experiment that serves equally strongly as an intuition against SSA.³ Taken together, the two thought experiments should not push us in any particular direction as to which of the two principles is preferable. Here goes:

Another presumptuous philosopher

It is the year 2100 and humanity has succeeded in the twin feats of (a) establishing that we are alone in the universe, and (b) once and for all solving xrisk, so that there is no longer any risk for permature extinction of humanity: our civilization will persist until the end of time. Physicists are on the verge of accomplishing a third feat, namely (c) finding the true and final theory of everything. They have narrowed down the search to only two remaining plausible candidate theories: T1 and T2 (using considerations from super-duper symmetry). According to T1 the world will last for a very, very long but finite amount of time, and there will be a total of a trillion trillion observers in the cosmos. According to T2, the world will last for a very, very, very long but finite amount of time, and there will be a trillion trillion trillion observers. The super-duper symmetry considerations are indifferent between these two theories. Physicists are preparing a simple experiment that will falsify one of the theories. Enter the SSA-adhering presumptuous philosopher: ''Hey guys, it is completely unnecessary for you to do the experiment, because I can already show you that T1 is about a trillion times more likely to be true than T2!''

Note that the SSA-adhering presumptuous philosopher's reasoning is exactly the same as in Brandon Carter's Doomsday Argument; I have merely changed the narrative details a bit in order to emphasize the similarity with Bostrom's presumptuous philosopher. Note also that if the two philosophers (the one favoring SIA in Bostrom's example, and the one favoring SSA in mine) trade places, then neither of them will be inclined to suggest any anthropic modification at all of the physicists' credence in T1 and T2.

In my opinion, the two thought experiments taken side by side serve as to illustrate that we are still confused about how to do anthropic reasoning. Both SIA and SSA produce appalingly presumptuous inferences. Is there some third possibility that avoids all of that? Maybe yes, but I suspect probably no, and that we will need to bite the bullet somewhere.

Footnotes

1) The names are terrible but have stuck.

2) I am here glossing over what ''random'' means (typically a uniform distribution, either over observers or so-called observer-moments), and even more so the meaning of ''relevant'', but the reader can rest assured that both Bostrom and 20 years of commentators treat these issues at length.

3) Olum (2002) suggests, for the same purpose, a different modification of Bostrom's original thought experiment. Here, super-duper symmetry comes with a vaguely Occam-like principle where, a priori, a theory of everything has probability inversely proportional to the size of the resulting universe. Bostrom and Cirkovic (2003), however, dismiss the example as far-fetched. I am not sure I find that dismissal convincing, but be that as it may, I still hope the modification I propose here is more to their taste.

New preprint on the Hinge of History

My latest preprint, entitled The Hinge of History and the Choice between Patient and Urgent Longtermism, is out now. Some terminological explanation and further context:

• The Hinge of History means, roughly, the most important time in the entire human history (past, present and future), in which we either get our act together or do something really bad that destroys all or most of future value. This can be made more precise in various ways discussed in the preprint. An increasingly popular idea is that the Hinge of History is now. Will MacAskill pushes back against this idea in a recent paper called Are we Living at the Hinge of History?, and in my preprint I push back against his pushback.
• Longtermism, in the words of MacAskill in his recent book What We Owe the Future, is ''the idea that positively influencing the longterm future is a key moral priority of our time''.
• Talk of urgent vs patient longtermism refers to whether this positive influence is best achieved via concrete object-level action now or via saving resources for such action at later times.
• The media attention around What We Owe the Future has been stupendous, not only in intellectually oriented and/or effective altruism-adjacent outlets such as Astral Codex Ten and podcasts by Tyler Cowen, Sean Carroll and Sam Harris, but also in mainstream media such as The New Yorker, Time Magazine, The Guardian and the BBC. I share the wide-spread enthusiasm for the book, and intend soon to help fix the relative shortage of reviews in Swedish.
• It has been pointed out that MacAskill sometimes defends far-reaching positions in academic papers and backs down to more moderate stances in his book, an example being how in a paper with Hilary Greaves he lays out the case for strong longtermism defined as the view that ''far-future effects are the most important determinant of the value of our options [today]'', but is content in the book with the somewhat more watered-down longtermism defined as above. Another example of this is his defense of patient longtermism in Are we Living at the Hinge of History, which is toned down in What We Owe the Future almost to the point of pressing the mute button. One may raise an eyebrow at this inconsistency, but in my opinion it is perfectly reasonable to explore principled positions in theoretical discussions taking place in seminar rooms and the academic literature, while choosing not to defend them in broader contexts.

Click here to find my new preprint!

* * *

Edit December 19, 2022: New and (somewhat) revised version uploaded now.