tisdag 19 februari 2019

I P4 om det allra senaste på AI-fronten

I höstas besökte jag P4 Göteborgs förmiddagsprogram tre gånger inom loppet av en månad. Därefter följde några månaders radiotystnad1 för min del, men idag var jag tillbaka i studion hos Stefan Livh; vårt samtal börjar vid pass 1:44:50 in i programmet och pågår till cirka 2:10:05.

Ämnet för dagen var AI (artificiell intelligens), och mer specifikt den AI-programvara kallad GPT-2 som i torsdags offentliggjordes av det San Francisco-baserade icke-kommersiella företaget OpenAI, vilket drogs igång 2015 med pengar från bl.a. Elon Musk, och har som uttalat mål att driva AI-utvecklingen i en riktning som gynnar mänskligheten som helhet. I huvudsak är GPT-2 ett generellt textgenereringsprogram som, givet en eller ett par korta inledningsfraser från användaren, fortsätter texten på egen hand. Dess häpnadsväckande förmåga att ansluta till genre, tonfall och sakinnehåll framgår av en rad konkreta exempel.

Lite på tvärs mot hur de tidigare brukat göra har OpenAI valt att inte släppa GPT-2 fri, av oro för hur den skulle kunna komma att användas för dåliga syften: den förefaller idealisk för massproduktion av fake news, och det är lätt att föreställa sig hur dess teknik kan kombineras med något i stil med Google Duplex för att skapa allt bedrägligare imitationer av människor. Genom att ändå offentliggöra sitt framsteg säger sig OpenAI skänka samhället åtminstone ett visst mått av förvarning och handlingsutrymme innan det troligtvis oundvikliga sker - att andra AI-utvecklare lanserar liknande program med motsvarande eller bättre prestanda. För min del tror jag att det är viktigt att inte stanna vid att diskutera vad dessa programvaror förmår göra idag. Vi bör försöka blicka lite längre än så, för det är knappast troligt att teknikutvecklingen nu hux flux skulle ha kommit till vägs ände.

Mer om GPT-2 kan läsas bland annat i Wired och The Guardian, men överlag är mediarapporteringen tunnare än väntat. I Sverige är det ännu så vitt jag kunnat finna bara Ny Teknik (och nu som sagt P4 Göteborg) som uppmärksammat genombrottet.

Fontot

1) I P4 alltså. I P1:s Vetandets värld gick det ju att höra mig den 7 januari.

torsdag 7 februari 2019

Samtal i Ängelholm

Rubriken Inspirational People gjorde mig tillräckligt smickrad för att ställa upp, och oavsett om jag egentligen förtjänar att platsa eller inte så är det inte någon dum idé att försöka skänka inspiration och råg i ryggen åt ungdomar genom att visa upp intervjuer med vuxna människor som gjort något intressant av sina liv. Den 19 september förra året begav jag mig till Ängelholm för att samtala framför en videokamera med Daniel Swärd som driver detta projekt. Det 70 minuter långa samtalet blev lite vindlande och ostrukturerat, men till slut kom vi i alla fall in på vad jag trodde var det jag kommit dit för att tala om: existentiell risk.

onsdag 6 februari 2019

I Curie om etik, forskning och framtida teknikutveckling

Vetenskapsrådets nättidning Curie publicerade idag en artikel rubricerad Forskning på gott och ont. Artikeln bygger på intervjuer med mig och med Uppsalafilosofen Jessica Nihlén Fahlquist, och våra svar på de forskningsetiska spörsmålen sammanfaller till ganska stor del med vad som kom fram i den paneldiskussion på Formas i november förra året som vi båda deltog i. För egen del lutar jag mig dessutom till stor del mot den argumentation jag lade fram tidigare under 2018 i min essä Vetenskap på gott och ont - en argumentation som kretsar kring den moralregel jag formulerat och som citeras i Curie-artikeln:
    Det kan aldrig vara acceptabelt att bedriva forskning vars risk att störta mänskligheten i fördärv och utplåning inte uppvägs av dess potential att skapa mänsklig blomstring och välfärd. Ej heller går det an att inleda ett forskningsprojekt utan att noggrant och uppriktigt ha övervägt denna aspekt.

onsdag 30 januari 2019

Some notes on Pinker's response to Phil Torres

The day before yesterday I published my blog post Steven Pinker misleads systematically about existential risk, whose main purpose was to direct the reader to my friend and collaborator Phil Torres' essay Steven Pinker's fake enlightenment. Pinker has now written a response to Phil's essay, and had it published on Jerry Coyne's blog Why Evolution is True. The response is feeble. Let me expand a little bit on that.

After a highly undignified opening paragraph with an uncharitable and unfounded speculation about Phil's motives for writing the essay,1 Pinker goes on throughout most of his response to explain, regarding all of the quotes that he exibits in his book Enlightenment Now and that Phil points out are taken out of context and misrepresent the various authors' intentions, that... well, that it doesn't matter that they are misrepresentations, because what he (Pinker) needed was words to illustrate his ideas, and for that it doesn't matter what the original authors meant. He suggests that "Torres misunderstands the nature of quotation". So why, then, doesn't Pinker use his own words (he is, after all, one of the most eloquent science writers of our time)? Why does he take this cumbersome detour via other authors? If he doesn't actually care what these authors mean, then the only reason I can see for including all these quotes and citations is that Pinker wants to convey to his readers the misleading impression that he is familiar with the existential risk literature and that this literature gives support to his views.

The most interesting case discussed in Phil's essay and Pinker's response concerns AI researcher Stuart Russell. In Enlightenment Now, Pinker places Russell in the category of "AI experts who are publicly skeptical" that "high-level AI pose[s] the threat of 'an existential catastrophe'." Everyone who has actually read Russell knows that this characterization is plain wrong, and that he in fact takes the risk for an existential catastrophe caused by an AI breakthrough extremely seriously. Phil points this out in his essay, but Pinker insists. In his response, Pinker quotes Russell as saying that "there are reasons for optimism", as if that quote were a demonstration of Russell's skepticism. The quote is taken from Russell's answer to the 2015 Edge question - an eight-paragraph answer that, if one reads it from the beginning to the end rather than merely zooming in on the phrase "there are reasons for optimism", makes it abundantly clear that to Russell, existential AI risk is a real concern. What, then, does "there are reasons for optimism" mean? It introduces a list of ideas for things we could do to avert the existential risk that AI poses. Proposing such ideas is not the same thing as denying the risk.

It seems to me that this discussion is driven by two fundamental misunderstandings on Pinker's part. First, he has this straw man image in his head of an existential risk researcher as someone proclaiming "we're doomed", whereas in fact what existential risk researchers say is nearly always more along the lines of "there are risks, and we need to work out ways to avoid them". When Pinker actually notices that Russell says something in line with the latter, it does not fit the straw man, leading him to the erroneous conclusion that Russell is "publicly skeptical" about existential AI risk.

Second, by shielding himself from the AI risk literature, Pinker is able to stick to his intuition that avoiding the type of catastrophe illustrated by Paperclip Armageddon is easy. In his response to Phil, he says that
    if we built a system that was designed only to make paperclips without taking into account that people don’t want to be turned into paperclips, it might wreak havoc, but that’s exactly why no one would ever implement a machine with the single goal of making paperclips,
continuing his light-hearted discourse from our encounter in Brussells 2017 where he said (as quoted on p 24 of the proceedings from the meeting) that
    the way to avoid this is: don’t build such stupid systems!
The literature on AI risk suggests that, on the contrary, the project of aligning the AI's goals with ours to an extent that suffices to avoid catastrophe is a difficult task, filled with subtle obstacles and traps. I could direct Pinker to some basic references such as Yudkowsky (2008, 2011), Bostrom (2014) or Häggström (2016), but given his plateau-shaped learning curve on this topic since 2014, I fear that he would either just ignore the references, or see them as sources to mine for misleading quotes.

Footnote

1) Borrowing from the standard climate denialist's discourse about what actually drives climate scientists, Pinker says this:
    Phil Torres is trying to make a career out of warning people about the existential threat that AI poses to humanity. Since [Enlightenment Now] evaluates and dismisses that threat, it poses an existential threat to Phil Torres’s career. Perhaps not surprisingly, Torres is obsessed with trying to discredit the book [...].

måndag 28 januari 2019

Steven Pinker misleads systematically about existential risk

The main purpose of this blog post is to direct the reader to existential risk scholar Phil Torres' important and brand-new essay Steven Pinker's fake enlightenment.1 First, however, some background.

Steven Pinker has written some of the most enlightening and enjoyable popular science that I've come across in the last couple of decades, and in particular I love his books How the Mind Works (1997) and The Blank Slate (2002) which offer wonderful insights into human psychology and its evolutionary background. Unfortunately, not everything he does is equally good, and in recent years the number of examples I've come across of misleading rhetoric and unacceptably bad scholarship on his part has piled up to a disturbing extent. This is especially clear in his engagement (so to speak) with the intertwined fields of existential risk and AI (artificial intelligence) risk. When commenting on these fields, his judgement is badly tainted by his wish to paint a rosy picture of the world.

As an early example, consider Pinker's assertion at the end of Chapter 1 in his 2011 book The Better Angels of Our Nature, that we "no longer have to worry about [a long list of barbaric kinds of violence ending with] the prospect of a nuclear world war that would put an end to civilization or to human life itseslf". This is simply unfounded. There was ample reason during the cold war to worry about nuclear annihilation, and from about 2014 we have been reminded of those reasons again through Putin's aggressive geopolitical rhetoric and action and (later) the inauguration of a madman as president of the United States, but the fact of the matter is that the reasons for concern never disappeared - they were just a bit less present in our minds during 1990-2014.

A second example is a comment Pinker wrote at Edge.org in 2014 on how a "problem with AI dystopias is that they project a parochial alpha-male psychology onto the concept of intelligence". See p 116-117 of my 2016 book Here Be Dragons for a longer quote from that comment, along with a discussion of how badly misinformed and confused Pinker is about contemporary AI futurology; the same discussion is reproduced in my 2016 blog post Pinker yttrar sig om AI-risk men vet inte vad han talar om.

Pinker has kept on repeating the same misunderstandings he made in 2014. The big shocker to me was to meet Pinker face-to-face in a panel discussion in Brussels in October 2017, and hear him make the same falsehoods and non sequiturs again and to add some more, including one that I had preempted just minutes earlier by explaining the relevant parts of Omohundro-Bostrom theory for instrumental vs final AI goals. For more about this encounter, see the blog post I wrote a few days later, and the paper I wrote for the proceedings of the event.

Soon thereafter, in early 2018, Pinker published his much-praised book Enlightenment Now: The Case for Reason, Science, Humanism, and Progress. Mostly it is an extended argument about how much better the world has become in many respects, economically and otherwise. It also contains a chapter named Existential threats which is jam-packed with bad scholarship and claims ranging from the misleading to outright falsehoods, all of it pointing in the same direction: existential risk research is silly, and we have no reason to pay attention to such concerns. Later that year, Phil Torres wrote a crushing and amazingly detailed but slightly dry rebuttal of that chapter. I've been meaning to blog about that, but other tasks kept coming in the way. Now, however, when Phil's Salon essay... ...is available, is the time. In the essay he presents some of the central themes from the rebuttal in more polished and reader-friendly form. If there is anyone out there who still thinks (as I used to) that Pinker is an honest and trustworthy scholar, Phil's essay is a must-read.

Footnote

1) It is not without a bit of pride that I can inform my readers that Phil took part in the GoCAS guest researcher program on existential risk to humanity that Anders Sandberg and I organized in September-October 2017, and that we are coauthors of the paper Long-term trajectories of human civilization which emanates from that program.

fredag 11 januari 2019

Recognized by the BBC for our work on the long-term future of humanity

Yesterday (January 10), BBC and their features editor Richard Fisher published a long but highly comprehensible and easy-to-read article on The perils of short-termism: Civilisation's greatest threat. It focuses on the central human predicament of needing to weigh short-term versus long-term concerns, and highlights work we did at the GoCAS guest researcher program on existential risk to humanity during September-October 2017 which Anders Sandberg and I organized at Chalmers and the University of Gothenburg. That particular piece of work was turned into the paper Long-term trajectories of human civilization,1 which I've reported on before on this blog.2 Here's an extract from the BBC piece:
    In early September 2017, the world’s attention was focused on various pieces of salient news: Hurricane Irma was brewing in the Caribbean, Donald Trump’s administration had announced plans to dismantle an Obama-era immigration policy, and photographers captured Prince George’s first day at school.

    Around the same time, a small, little-known group of researchers were meeting at a workshop in Gothenburg, Sweden with a goal to look much, much further ahead – far beyond this latest news cycle. Motivated by a moral concern for our descendants, their goal was to discuss the existential risks facing humanity.

    The meeting would lead to an intriguing and readable co-authored paper called Long-term Trajectories of Human Civilisation, which attempts to “formalise a scientific and ethical field of study” for thousands of years hence. As they write: “To restrict attention to near-term decades may be akin to the drunk searching for his keys under the streetlight: it may be where empirical study is more robust, but the important part lies elsewhere.”

    The Trajectories group began with the assumption that, while the future is uncertain, it is not unknown. We can predict many things with reasonable confidence, via observed patterns, repeating events, and established behaviours throughout human history. For example: biology suggests that each mammalian species exists, on average, for roughly 1 million years before it becomes extinct; history shows that humanity has continually colonised new lands and strived to transform our abilities with technology; and the fossil record demonstrates that global extinction events can and do happen.

    Extrapolating these patterns and behaviours into the future allowed them to map out four possible long-term trajectories for our species:

    • Status quo trajectories, in which human civilisation persists in a broadly similar state into the distant future.

    • Catastrophe trajectories, in which one or more events cause significant harm to human civilisation.

    • Technological transformation trajectories, in which radical technological breakthroughs put human civilisation on a fundamentally different course.

    • Astronomical trajectories, in which human civilisation expands beyond its home planet and into the accessible portions of the cosmos.
    Following their discussions in Sweden and afterwards, the Trajectories group concluded that the ‘status quo’ path would be a pretty unlikely scenario once you get to longer-term timescales. “Instead, civilisation is likely to either end catastrophically or expand dramatically,” they write.

    So on the optimistic path, merging with some as-yet-unimagined technology or colonising the stars are two scenarios that are entirely possible with the passage of enough time, they suggest. Both paths could lead to our descendants thriving for millions if not billions of years, spreading out into the Universe or evolving into a more advanced species.

    But we also are almost certain to face serious existential risks along the way. Natural disasters have pruned life on Earth continually – this much we know. What worries the Trajectories researchers more is that in the 20th and early 21st Century we've added a whole host of additional human-made risks into the mix too – from nuclear armageddon to AI apocalypse to anthropogenic climate change.

    In their paper, they lay out a variety of chilling scenarios where civilisation is rewound back to pre-industrial times, or wiped out altogether. [...]

    The researchers can't predict which order any of this will play out. They can predict, though, that it’s our trajectory to shape – for better or worse, the decisions we make this century could shape the next one and far beyond. “The stakes are extremely large, and there may be a lot that people today can do to have a positive impact,” they write.

    The question is: will we?

"The Trajectories group" - well, why not, it has a nice ring to it. Read the entire BBC article here!

Footnotes

1) Seth D. Baum, Stuart Armstrong, Timoteus Ekenstedt, Olle Häggström, Robin Hanson, Karin Kuhlemann, Matthijs M. Maas, James D. Miller, Markus Salmela, Anders Sandberg, Kaj Sotala, Phil Torres, Alexey Turchin and Roman V. Yampolskiy (2019) Long-term trajectories of human civilization, Foresight, to appear.

2) The same paper recently also caught the attention of Swedish public radio.

måndag 7 januari 2019

Om existentiell risk och mänsklighetens framtid i P1:s Vetandets värld idag

Jag vill varmt rekommendera dagens avsnitt av P1:s Vetandets värld, trots att dess rubrik Så räknar man ut mänsklighetens framtid lovar något mer än avsnittet och dess medverkande kan leverera. Jag kommer utförligt till tals, jämte en rad andra forskare: Anders Sandberg, Simon Beard, Luke Kemp och Lauren Holt, samt David Sumpter1 vars korta inhopp på slutet tjänar att illustrera hur kontroversiell det slags radikal och långsiktig framtidsforskning är som avsnittet behandlar.

Det gästforskarprogram om existentiell risk som Anders och jag organiserade på Chalmers och Göteborgs universitet hösten 2017 omnämns, med särskilt fokus på den artikel rubricerad Long-term trajectories of human civilization som en rad av de medverkande gästforskarna, med Seth Baum i spetsen, bidrog till. Även den ökända gemapokalypsen behandlas kort, liksom min artikel Challenges to the Omohundro—Bostrom framework for AI motivations.

Fotnot

1) Jag uppskattar varmt matematikerkollegan David Sumpters och mina återkommande diskussioner om vetenskap och futurologi, trots (eller tack vare!) vår djupa meningsskiljaktighet i synen på vetenskap, som läsaren kan få en partiell inblick i genom Davids gästinlägg här på bloggen i oktober 2013 och mitt svar.