I am grateful to David Sumpter for his guest post Why "intelligence explosion" and many other futurist arguments are nonsense yesterday. He holds what I believe to be a very common view among bright and educated people who have come across the idea of developments in artificial intelligence eventually leading to what is known as an intelligence explosion or the Singularity (i.e., an extremely rapid escalation towards superintelligence levels, with profound consequences for humanity and the world). I think David articulates his view and his argument very clearly, but I also think the argument is wrong, and in the following I will explain why. Note that my ambition here is not to argue that an intelligence explosion is likely (on that topic I am in fact agnostic). Instead, I will limit the discussion to the more modest task of showing that David's argument fails.
David's conclusions essentially boil down to two claims, namely
- (1) the possibility and likelihood of a future intelligence explosion is not a scientific topic,
- (2) a future intelligence explosion is very unlikely.
Concerning (1), here's how David spells out what is needed to call something science:
- We make models, we make predictions and test them against data, we revise the model and move forward. You can be a Bayesian or a frequentist or Popperian, emphasise deductive or inductive reasoning, or whatever, but this is what you do if you are a scientist. Scientific discoveries are often radical and surprising but they always rely on the loop of reasoning coupled with observation.
The first thing that needs to be stressed in this context is that contemporary thinking about the Singularity is not pure speculation in the sense of being isolated from empirical data. On the contrary, it is fed with data of many different kinds. Examples include (a) the observed exponential growth of hardware performance known as Moore's law, (b) the observation that the laws of nature have given rise to intelligent life at least once, and (c) the growing body of knowledge concerning biases in the human cognitive machinery that David somewhat nonchalantly dismisses as irrelevant. See, e.g., the book by Ray Kurzweil (2005) and the paper by Eliezer Yudkowsky (2013) for this and much more. No single one of these data implies an intelligence explosion on its own, but they all serve as input to the emerging theory on the topic, and they are all subject to refinement and revision, as part of "the loop of reasoning coupled with observation" that David talks about in the demarcation criterion above.
At this point, some readers (including David, I presume) will object that it is not statements about down-to-earth things like current hardware performance growth or biases in human cognition that need to be tested, but high-flying hypotheses like
- C1 = "an intelligent explosion is likely to happen around 2100".
Here, a comparison with the more familiar territory of climate science may be helpful. Climate science deals routinely with hypotheses such as
- C2 = "under the IPCC AR4 emission scenario A2, global average temperature in 2100 will likely exceed the pre-industrial level by at least 3°C".
Let me move on to David's claim (2) about the event of a future intelligence explosion being very unlikely. Let's call this event E1. Here David proceeds by comparing E1 to the event E2 of the future realization of some comparatively more pedestrian technological development such as "a person-sized robot can walk around for two minutes without falling over when there is a small gust of wind".3 David thinks (reasonably enough) that E1 is a more difficult and involved project to achieve than E2. Many times more difficult and involved. From this he concludes - and this is his non sequitur - that (under any probability model that reasonably well models the relevant aspects of the real world) the probability P(E1) is many times smaller than P(E2), whence P(E1) must be very small.
The trick employed by David here is simply invalid. It is just not true that a task T1 that is many times more difficult and involved than another task T2 must have a probability P(T1 is achieved) that is many times smaller than P(T2 is achieved). For a simple counterexample, imagine me on May 17, 2014, standing on the starting line of the Göteborgsvarvet half marathon. I have previously completed this race 16 times out of 16 attempts, and I am (let's assume) about as well-praperade as I usually am, apart from being just slightly older. Let T1 be the task of completing the full race (21,097.5 meters) and let T2 be the task of completing the first 500 meters of the race. T1 is many times more difficult and involved than T2. Yet, the probabilities that I achieve T1 and T2, respectively, do not differ all that dramatically: P(T1 is achieved) is about 0.97, while P(T2 is achieved) is about 0.995. So it is manifestly not the case that P(T1 is achieved) is many times smaller than P(T2 is achieved).
So David's case for (2) is based on a fallacy. His impulse to try to say something nontrivial about P(E1) is laudable, but a simple trick like the one above just won't do. After having thought about it for a few years, I am convinced that estimating P(E1) is a deep and difficult problem.4 If David seriously wants to shed light on it, I don't think he has any choice but to roll up his sleeves and get to work on the substance of the problem. A key part of the problem (but not the only one) seems to be the "return on cognitive investment" issue introduced on the first few pages of Yudkowsky (2013): which is closer to capturing the truth - the k<1 argument (page 3 of Yudkowsky's paper) or the k>>1 argument (page 5)?
1) If there is any direct link between them, I'd say it's this: if claim (1) is true, then claim (2) is unscientific.
2) A key word here is "serious". There are also climate denialists, who are all too happy to brand hypotheses like C2 as mere speculation and therefore unscientific. In case there are any such creatures among my readers, let me give a third example C3, borrowed from Footnote 3 in a 2008 paper of mine, namely
- C3 = "a newborn human who is immersed head-to-toe for 30 minutes in a tank of fuming nitric acid will not survive".
3) Here I'm skipping his lovely detour through Mesopotamia.
4) Forecasting technology is, except in very short time perspectives, an extraordinarily difficult problem. Nassim Nicholas Taleb highlights, on p 172 of his provocative 2007 book The Black Swan, one of the difficulties:
- If you are a Stone Age historical thinker called on to predict the future in a comprehensive report for your chief tribal planner, you must project the invention of the wheel or you will miss pretty much all of the action. Now, if you can prophesy the invention of the wheel, you already know what a wheel looks like, and thus you already know how to build a wheel, so you are already on your way.