A spectacularly uneven AI report

Earlier this week, the EU Parliament's STOA (Science and Technology Options Assessment) committee released the report "Should we fear artificial intelligence?", whose quality is so spectacularly uneven that I don't think I've ever seen anything quite like it. It builds on a seminar in Brussels in October last year, which I've reported on before on this blog. Four authors have contributed one chapter each. Of the four chapters, three are very good two are of very high quality, one is of a quality level that my modesty forbids me to comment on, and one is abysmally bad. Let me list them here, not in the order they appear in the report, but in one that gives a slightly better dramatic effect.

• Miles Brundage: Scaling Up Humanity: The Case for Conditional Optimism about Artificial Intelligence.

  In this chapter, Brundage (a research fellow at the Future of Humanity Institute) is very clear about the distinction between conditional optimism and just plain old optimism. He's not saying that an AI breakthrough will have good consequences (that would be plain old optimism). Rather, he's saying that if it has good consequences, i.e., if it doesn't cause humanity's extinction or throw us permanently into the jaws of Moloch, then there's a chance the outcome will be very, very good (this is conditional optimism).

• Thomas Metzinger: Towards a Global Artificial Intelligence Charter.

  Here the well-know German philosopher Thomas Metzinger lists a number of risks that come with future AI development, ranging from well-known ones concerning technological unemployment or autonomous weapons to more exotic ones arising from the possibility of constructing machines with the capacity to suffer. He emphasizes the urgent need for legislation and other government action.

• Olle Häggström: Remarks on Artificial Intelligence and Rational Optimism.

  This text is already familiar to readers of this blog. It is my humble attempt to sketch, in a balanced way, some of the main arguments for why the wrong kind of AI breakthrough might well be an existential risk to humanity.

• Peter Bentley: The Three Laws of Artificial Intelligence: Dispelling Common Myths.

  Bentley assigns great significance to the fact that he is an AI developer. Thus, he says, he is (unlike us co-contributors to the report) among "the people who understand AI the most: the computer scientists and engineers who spend their days building the smart solutions, applying them to new products, and testing them". Why exactly expertise in developing AI and expertise in AI futurology necessarily coincide in this way (after all, it is rarely claimed that farmers are in a privileged position to make predictions about the future of agriculture) is not explained. In any case, he claims to debunk a number of myths, in order to arrive at the position which is perhaps best expressed in the words he chose to utter at the seminar in October: superhumanly intelligent AI "is not going to emerge, that’s the point! It’s entirely irrational to even conceive that it will emerge" [video from the event, at 12:08:45]. He relies more on naked unsupported claims than on actual arguments, however. In fact, there is hardly any end to the inanity of his chapter. It is very hard to comment on at all without falling into a condescending tone, but let me nevertheless risk listing a few of its very many very weak points:

  1. Bentley pretends to speak on behalf of AI experts - in his narrow sense of what such expertise entails. But it is easy to give examples of leading AI experts who, unlike him, take AI safety and apocalyptic AI scenarios seriously, such as Stuart Russell and Murray Shanahan. AI experts are in fact highly divided in this issue, as demonstrated in surveys. Bentley really should know this, as in his chapter he actually cites one of these surveys (but quotes it in shamelessly misleading fashion).

  2. In his desperate search for arguments to back up his central claim about the impossibility of building a superintelligent AI, Bentley waves at the so-called No Free Lunch theorem. As I explained in my paper Intelligent design and the NFL theorems a decade ago, this result is an utter triviality, which basically says that in a world with no structure at all, no better way than brute force exists if you want to find something. Fortunately, in a world such as ours which has structure, the result does not apply. Basically the only thing that the result has going for it is its cool name, something that creationist charlatan William Dembski exploited energetically to try to give the impression that biological evolution is impossible, and now Peter Bentley is attempting the analogous trick for superintelligent AI.

  3. At one point in his chapter, Bentley proclaims that "even if we could create a super-intelligence, there is no evidence that such a super-intelligent AI would ever wish to harm us". What the hell? Bentley knows about Omohundro-Bostrom theory for instrumental vs final AI goals (see my chapter in the report for a brief introduction) and how it predicts catastrophic consequences in case we fail to equip the superintelligent AI with goals that are well-aligned with human values. He knows it by virtue of having read my book Here Be Dragons (or at least he cites it and quotes it), on top of which he actually heard me present the topic at the Brussels seminar in October. Perhaps he has reasons to believe Omohundro-Bostrom theory to be flawed, in which case he should explain why. Simply stating out of the blue, as he does, that no reason exists for believing that a superintelligent AI might turn agianst us is deeply dishonest.

  4. Bentley spends a large part of his chapter attacking the silly straw man that the mere progress of Moore's law, giving increasing access to computer power, will somehow spontaneously create superintelligent AI. Many serious thinkers speculate about an AI breakthrough, but none of them (not even Ray Kurzweil) think computer power on its own will be enough.

  5. The more advanced an AI gets, the more involved will the testing step of its development be, claims Bentley, and goes on to argue that the amount of testing needed grows exponentially with the complexity of the situation, essentially preventing rapid development of advanced AI. His premise for this is that "partial testing is not sufficient - the intelligence must be tested on all likely permutations of the problem for its designed lifetime otherwise its capabilities may not be trustable", and to illustrate the immensity of this task he points out that if the machine's input consists of a mere 100 variables that each can take 10 values, then there are 10¹⁰⁰ cases to test. And for readers for whom it is not evident that 10¹⁰⁰ is a very large number, he writes it in decimal. Oh please. If Bentley doesn't know that "partial testing" is what all engineering projects need to resort to, then I'm beginning to wonder what planet he comes from. Here's a piece of homework for him: calculate how many cases the developers of the latest version of Microsoft Word would have needed to test, in order not to fall back on "partial testing", and how many pages would be needed for writing that number in decimal.

  6. Among the four contributors to the report, Bentley is alone in claiming to be able to predict the future. He just knows that superintelligent AI will not happen. Funny, then, that not even his claim that "we are terrible at predicting the future, and almost without exception the predictions (even by world experts) are completely wrong" doesn't seem to induce as much as a iota of empistemic humility into his prophecy.

  7. In the final paragraph of his chapter, Bentley reveals his motivation for writing it: "Do not be fearful of AI - marvel at the persistence and skill of those human specialists who are dedicating their lives to help create it. And appreciate that AI is helping to improve our lives every day." He is simply offended! He and his colleagues work so hard on AI, they just want to make the world a better place, and along comes a bunch of other people who have the insolence to come and talk about AI risks. How dare they! Well, I've got news for Bentley: The future development of AI comes with big risks, and to see that we do not even need to invoke the kind of superintelligence breakthrough that is the topic of the present discussion. There are plenty of more down-to-earth reasons to be "fearful" of what may come out of AI. One such example, which I touch upon in my own chapter in the report, is the development of AI technology for autonomous weapons, and how to keep this technology away from the hands of terrorists.

A few days after the report came out, Steven Pinker tweeted that he "especially recommend[s] AI expert Peter Bentley's 'The Three Laws of Artificial Intelligence: Dispelling Common Myths' (I make similar arguments in Enlightenment Now)". I find this astonishing. Is it really possible that Pinker is that blind to the errors and shortcomings in Bentley's chapter? Is there a name for the fallacy "I like the conclusion, therefore I am willing to accept any sort of crap as arguments"?

Litet efterspel till panelen i Bryssel

Sent i lördags eftermiddag (den 9 december) ökade plötsligt trafiken hit till bloggen kraftigt. Den kom mestadels från USA, och landade till större delen på min bloggpost The AI meeting in Brussels last week från förrförra månaden, så till den grad att bloggposten inom loppet av 24 timmar klättrade från typ ingenstans till tredje plats på all-time-high-listan över denna bloggs mest lästa inlägg (slagen endast av de gamla bloggposterna Quickologisk sannolikhetskalkyl och Om statistisk signifikans, epigenetik och de norrbottniska farmödrarna). Givetvis blev jag nyfiken på vad som kunde ha orsakat denna trafikökning, och fann snabbt en Facebookupdatering av den framstående AI-forskaren Yann LeCun vid New York University. LeCun länkar till min Bryssel-bloggpost, och har ett imponerande antal Facebook-följare, vilket förklarar trafikökningen.

Precis som sin AI-forskarkollega Peter Bentley och kognitionsforskaren Steven Pinker - vilka båda deltog i Bryssel-panelen - anser LeCun att alla farhågor om en eventuell framtida AI-apokalyps är obefogade. Hans imponerande meriter som AI-forskare väcker såklart förhoppningar (hos den som läser hans Facebookuppdatering) om att han skall presentera väsentligt bättre argument för denna ståndpunkt än dem som Pinker och Bentley levererade i Bryssel - förhoppningar som dock genast kommer på skam. LeCuns retoriska huvudnummer är följande bisarra jämförelse:

[F]ear mongering now about possible Terminator scenarios is a bit like saying in the mid 19th century that the automobile will destroy humanity because, although we might someday figure out how to build internal combustion engines, we have no idea how to build brakes and safety belts, and we should be very, very worried.

Vad som gör jämförelsen bisarr är att (åtminstone 1900-talets och dagens) bilar totalt saknar de självreproducerande och rekursivt självförbättrande egenskaper som en tillräckligt intelligent framtida AI enligt många bedömare kan väntas få, vilka är grunden för de potentiellt katstrofala intelligensexplosionsscenarier som diskuteras av bland andra Yudkowsky, Bostrom och Tegmark (liksom i min egen bok Here Be Dragons). Till skillnad mot i AI-scenarierna finns inget rimligt bilscenario där vår oförmåga att bygga fungerande bromsar skulle ta kål på mänskligheten (allt som skulle hända om det inte gick att få till en fungerande bromsteknologi vore att ingen skulle vilja köra bil och att biltillverkningen upphörde). Vad som krävs för att LeCuns jämförelse skall få minsta relevans är att han påvisar att fenomenet med en rekursivt självförbättrande AI inte kan bli verklighet, men han redovisar inte tillstymmelse till sådant argument.

LeCuns Facebookuppdatering bjuder också på en direkt självmotsägelse: han hävdar dels att "we have no idea of the basic principles of a purported human-level AI", dels att...

[t]he emergence of human-level AI will not be a singular event (as in many Hollywood scenarios). It will be progressive over many many years. I'd love to believe that there is a single principle and recipe for human-level AI (it would make my research program a lot easier). But the reality is always more complicated. Even if there is a small number of simple principles, it will take decades of work to actually reduce it to practice.

Här frågar sig naturligtvis den vakne läsaren: om nu LeCun har rätt i sitt första påstående, hur i hela glödheta h-e kan han då ha den kunskap han gör anspråk på i det andra? Det kan naturligtvis hända att han har rätt i att utvecklingen kommer att gå långsamt, men hans dogmatiska tvärsäkerhet är (i avsaknad av solida argument för att backa upp ståndpunkten) direkt omdömeslös.

Vad som gör LeCuns Facebookuppdatering den 9 december ännu mer beklämmande är att dess text är kopierad från en kommentar han skrev i en annan Facebooktråd redan den 30 oktober. Uppenbarligen tyckte han, trots att han haft mer än en månad på sig att begrunda saken, att hans slagfärdigheter var av tillräckligt värde för att förtjäna ytterligare spridning.

The AI meeting in Brussels last week

I owe my readers a report from the seminar entitled "Should we fear the future? Is it rational to be optimistic about artificial intelligence?" at the European Parliament's STOA (Science and Technology Options Assessment) committee in Brussels last Thursday. In my opinion, all things considered, the event turned out OK, and it was a pleasure to meet and debate with the event's main speaker Steven Pinker as well as with co-panelists Miles Brundage and Thomas Metzinger.¹ I'd just like to comment on Pinker's arguments for why we should not take seriously or publicly discuss the risk for an existential catastrophe caused by the emergence of superintelligent AGI (artificial general intelligence). His arguments essentially boil down to the follwing four points, which in my view fail to show what he intends.

1. The general public already has the nuclear threat and the climate threat to worry about, and bringing up yet another global risk may overwhelm people and cause them to simply give up on the future. There may be something to this speculation, but to evaluate the argument's merit we need to consider separately the two possibilities of
(a) apocalyptic AI risk being real, and

(b) apocalyptic AI risk being spurious.

In case of (b), of course we should not waste time and effort on discussing such risk, but we didn't need the overwhelming-the-public argument to understand that. Consider instead case (a). Here Pinker's recommendation is that we simply ignore a threat that may kill us all. This does not strike me as a good idea. Surviving the nuclear threat and solving the climate crisis would of course be wonderful things, but their utility is severely hampered in case it just leads us into an AI apocalypse. Keeping quiet about a real risk also seems to fly straight in the face of one of Pinker's most dearly held ideas, namely that of scientific and intellectual openness, and Enlightenment values more generally. The same thing applies to the situation where we are unsure whether (a) or (b) holds - surely the approach best in line with Enlightenment values is then to openly discuss the problem and to try to work out whether the risk is real.

2. Pinker held forth a bunch of concerns that seemed more or less copy-and-pasted from the standard climate denialism discourse. These included the observation that the Millennium bug did not cause global catastrophe, whence (or so the argument goes) a global catastrophe cannot be expected from a superintelligent AGI (analogously to the oft-repeated claim that the old Greek's fear that the skies would fall down turing out to be unfounded shows that greenhouse gas emissions cannot accelerate global warming in any dangerous way), and speculations about the hidden motives of those who discuss AI risk - they are probably just competing for status and research grants. This is not impressive. See also yesterday's blog post by my friend Björn Bengtsson for more on this; it is to him that I owe the (in retrospect obvious) parallel to climate denialism.

At this point, one may wonder why Pinker doesn't do the consistent thing, given these arguments, and join the climate denialism camp. He would probably respond that unlike AI risk, climate risk is backed up by solid scientific evidence. And indeed the two cases are different - the case for climate risk is considerably more solid - but the problem with Pinker's position is that he hasn't even bothered to find out what the science of AI risk says. This brings me to the next point.

3. All the apocalyptic AI scenarios involve the AI having bad goals, which leads Pinker to reflect on why in the world would anyone program the machine with bad goals - let's just not do that! This is essentially the idea of the so-called Friendly AI project (see Yudkowsky, 2008, or Bostrom, 2014), but what Pinker does not seem to appreciate is that the project is extremely difficult. He went on to ask why in the world anyone would be so stupid as to program self-preservation at all costs into the machine, and this in fact annoyed me slightly, because it happened just 20 or so minutes after I had sketched the Omohundro-Bostrom theory for how self-preservation and various other instrumental goals are likely to emerge spontaneously (i.e., without having them explicitly put into it by human programmers) in any sufficiently intelligent AGI.

4. In the debate, Pinker described (as he had done several times before) the superintelligent AGI in apocalyptic scenarios as having a typically male psychology, but pointed out that it can equally well turn out to have more female characteristics (things like compassion and motherhood), in which case everything will be all right. This is just another indication of how utterly unfamiliar he is with the literature on possible superintelligent psychologies. His male-female distinction in the general context of AGIs is just barely more relevant than the question of whether the next exoplanet we discover will turn out to be male or female.

In summary, I don't think that Pinker's arguments for why we should not talk about risks associated with an AI breakthrough hold water. On the contrary, I believe there's an extremely important discussion to be had on that topic, and I wish we had had time to delve a bit deeper into it in Brussels. Here is the video from the event.

(Or watch the video via this link, which may in some cases work better.)

Footnotes

1) My omission here of the third co-panelist Peter Bentley is on purpose; I did not enjoy his presence in the debate. In what appeared to be an attempt to compensate for the hollowness of his arguments,² he reverted to assholery on a level that I rarely encounter in seminars and panel discussions: he expressed as much disdain as he could for opposing views, he interrupted and stole as much microphone time as he could get away with, and he made faces while other panelists were speaking.

2) After spending a disproportionate amount of his allotted time on praising his own credentials, Bentley went on to defend the idea that we can be sure that a breakthrough leading to superintelligent AGI will not happen. For this, he had basically one single argument, namely his and other AI developers' experience that all progress in the area requires hard work, that any new algorithm they invent can only solve one specific problem, and that initial success of the algorithm is always followed by a point of diminishing returns. Hence (he stressed), solving another problem always requires the hard work of inventing and implementing yet another algorithm. This line of argument conveniently overlooks the known fact (exemplified by the software of the human brain) that there do exist algorithms with a more open-ended problem-solving capacity, and is essentially identical to item (B) in Eliezer Yudkowsky's eloquent summary, in his recent essay There is no fire alarm for artificial general intelligence, of the typical arguments held forth for the position that an AGI breakthrough is either impossible or lies far in the future. Quoting from Yudkowsky:

Why do we know that AGI is decades away? In popular articles penned by heads of AI research labs and the like, there are typically three prominent reasons given:

(A) The author does not know how to build AGI using present technology. The author does not know where to start.

(B) The author thinks it is really very hard to do the impressive things that modern AI technology does, they have to slave long hours over a hot GPU farm tweaking hyperparameters to get it done. They think that the public does not appreciate how hard it is to get anything done right now, and is panicking prematurely because the public thinks anyone can just fire up Tensorflow and build a robotic car.

(C) The author spends a lot of time interacting with AI systems and therefore is able to personally appreciate all the ways in which they are still stupid and lack common sense.

The inadequacy of these arguments lies in the observation that the same situation can be expected to hold five years prior to an AGI breakthrough, or one year, or... (as explained by Yudkowsky later in the same essay).

At this point a reader or two may perhaps be tempted to point out that if (A)-(C) are not considered sufficient evidence against a future superintelligence, then how in the world would one falsify such a thing, and doesn't this cast doubt on whether AI apocalyptic risk studies should count as scientific? I advise those readers to consult my earlier blog post Vulgopopperianism.