More on the Lemoine affair

My blog post two days ago about Google engineer Blake Lemoine who has been put on paid administrative leave for breaking the company's confidentialty rules was written in a bit of a haste, ignoring what I now think may be the two most important aspects of the whole story. I will make up for that omission here, but will not repeat the background, for which I refer back to that earlier blog post. Here are the two aspects:

First, Lemoine is a whistleblower, and whistleblowing tends to be personally very costly. But we very much need whistleblowers, and due to this externality mismatch we also need society to treat its whistleblowers well - even in cases (such as, I suspect, the one at hand) where the message conveyed turns out ultimately wrong. While I do not have any concrete suggestion for what a law supporting this idea should look like, I do believe we ought to have such laws, and in the meantime it is up to each of us to be supportive of individual whistleblowers. Our need for them is greater in Big Tech than in perhaps any other sector, because by responding disproportionally to their commercial incentives rather than to the common good, these companies risk causing great harm:

• this is already happening (see this recent text by Jonathan Haidt), and
• things can get orders of magnitude worse (see this recent text by Eliezer Yudkowsky).

Second, while (as I've said) Lemoine is probably wrong about the AI system LaMDA having achieved consciousness, it is extremely important that we do not brush the issue of AI consciousness permanently aside, lest we otherwise risk creating atrocities, potentially on a scale that dwarfs present-day meat industry. Therefore, the dogmatic attitude of his high-level manager Jen Gennai (Director of Responsible Innovation at Google) that Lemoine describes is totally unacceptable:

When Jen Gennai told me that she was going to tell Google leadership to ignore the experimental evidence [about LaMDA being sentient] I had collected I asked her what evidence could convince her. She was very succinct and clear in her answer. There does not exist any evidence that could change her mind. She does not believe that computer programs can be people and that’s not something she’s ever going to change her mind on.

The possibility of AI consciousness needs to be taken seriously, and it is an issue that can escelate from the hypothetical and philosophical to actual reality sooner than we think. As I remarked in my previous blogpost, AI futurology and AI safety scholars have tended to ignore this issue (largely, I believe, due to its extreme difficulty), but a notable recent exception is the extraordinarily rich paper Propositions Concerning Digital Minds and Society by Nick Bostrom and Carl Shulman. Among its many gems and deep insights, let me quote a passage of particular relevance to the issue at hand:

• Training procedures currently used on AI would be extremely unethical if used on humans, as they often involve:
  • No informed consent;
  • Frequent killing and replacement;
  • Brainwashing, deception, or manipulation;
  • No provisions for release or change of treatment if the desire for such develops;
  • Routine thwarting of basic desires; for example, agents trained or deployed in challenging environments may possibly be analogous to creatures suffering deprivation of basic needs such as food or love;
  • While it is difficult conceptually to distinguish pain and pleasure in current AI systems, negative reward signals are freely used in training, with behavioral consequences that can resemble the use of electric shocks on animals;
  • No oversight by any competent authority responsible for considering the welfare interests of digital research subjects or workers.
• As AI systems become more comparable to human beings in terms of their capabilities, sentience, and other grounds for moral status, there is a strong moral imperative that this status quo must be changed.
• Before AI systems attain a moral status equivalent to that of human beings, they are likely to attain levels of moral status comparable to nonhuman animals—suggesting that changes to the status quo will be required well before general human-level capabilities are achieved.
  • The interests of nonhuman animals are violated on a massive scale in, for example, factory farms, and there is a strong case that this is morally wrong.
  • Nevertheless, there are some systems in place to limit the harm and suffering inflicted on animals (e.g., minimum standards for cage size, veterinary care, outlawing of various forms of animal abuse, the “three Rs” in animal experimentation, etc.).
  • Digital minds that are morally comparable to certain nonhuman animals should ideally have protections similar to those that ought to be extended to those animals (which are greater than those that are at present actually extended to farmed animals).
• Some research effort should be devoted to better understand the possible moral status, sentience, and welfare interests of contemporary AI systems, and into concrete cost-effective ways to better protect these interests in machine learning research and deployment.

Filterfatalism

Den här bloggposten är i första hand till för de läsare som känner till och hunnit smälta det Stora Filtret - den modell för mänsklighetens plats i universum som Robin Hanson introducerade 1998 och som jag förra året gjorde ett försök att förmedla till bloggens läsekrets. Låt mig ändå helt kort rekapitulera vad det handlar om.

Låt N vara antalet planeter lämpade för liv i det synliga universum, och låt p vara sannolikheten att en på måfå vald sådan planet utvecklar inte bara liv utan rentav en teknologisk civilisation på en nivå motsvarande mänskligheten anno 2013. Låt dessutom q vara sannolikheten att en på måfå vald teknologisk civilisation på en nivå motsvarande mänskligheten anno 2013 lyckas gå vidare mot sådana höjder att de, via t.ex. galaktisk ingenjörskonst, sätter ett sådant avtryck i universum att detta kan observeras av astronomer var dessa än befinner sig i (rätt halva av) deras ljuskon. Då är Np ungefär lika med antalet planeter som givit upphov till civilisationer på vår nivå eller högre, och Npq är ungefär lika med antalet planeter som givit upphov till civilisationer vilka satt enorma avtryck i universum.

Vad vet vi om N, p och q? Först och främst vet vi att Npq inte gärna kan vara något särskilt stort tal, då vi ju inte observerat något som kan tolkas som spår av någon intergalaktisk megacivilisation. Dessutom har vi, inte minst med tanke på senare års framgångar i studiet av exoplaneter, synnerligen goda skäl att tro att N är mycket mycket stort (tusentals miljarder miljarder). Men om vi accepterar att N är mycket mycket stort, och att Npq inte är särskilt stort, då har vi inget annat val än att acceptera att produkten pq är väldigt nära noll. Och har vi väl gått med på det, då måste vi också gå med på att minst en av sannolikheterna p och q är pytteliten. Eller med andra ord: Antingen har vi här på Jorden varit med om något extremt ovanligt i form av uppkomsten av en civilisation på vår nivå (p pyttelitet), eller också är det extremt ovanligt att civilisationer på vår nivå går vidare till nivåer där de börjar sätta kraftiga avtryck i universum (q pyttelitet).

Vilket av talen p och q som är pyttelitet (kanske båda) vet vi däremot inte. För dem som, likt mig, har svårt att föreställa sig att den mänskliga civilisationen skall kunna stabilisera sig till något slags stabil Bullerbytillvaro utan att snabbt (inom på sin höjd några tusen år eller så) inledda en storskalig expansion i och kolonisation av yttre rymden, är det lite läskigt att tänka sig att q är pyttelitet, ty det verkar ju i så fall peka på att vår civilisation troligtvis inom kort är på väg att gå under. Vi bör istället hoppas att det är p som är pyttelitet, så att möjligheten ändå finns att q är hyggligt stort. (Det är därför som Nick Bostrom skriver "Why I hope the search for extraterrestrial life finds nothing".)

Jag har ändå velat tänka mig att vi kan ha stor praktisk nytta av att utröna hur det står till med det Stora Filtret. Om det visar sig att q är pyttelitet, då vet vi att stora faror lurar. Men att ha kunskap om en stor fara är ju ofta första steget mot att bemästra den. Kanske skulle, i ett sådant scenario, vår kunskap om q:s litenhet kunna bättra på våra taskiga odds.

För oss med en böjelse för just den sortens dagdrömmerier kommer det som en mäkta obehaglig överraskning att ta del av vad Carl Shulman har att säga om saken. Hans bloggpost i ämnet, rubricerad Future Filter Fatalism, är så välformulerad att den förtjänar att citeras i sin helhet:

One of the more colorful vignettes in philosophy is Gibbard and Harper’s "Death in Damascus" case:
Consider the story of the man who met Death in Damascus. Death looked surprised, but then recovered his ghastly composure and said, ‘I am coming for you tomorrow’. The terrified man that night bought a camel and rode to Aleppo. The next day, Death knocked on the door of the room where he was hiding, and said ‘I have come for you’.

‘But I thought you would be looking for me in Damascus’, said the man.

‘Not at all’, said Death ‘that is why I was surprised to see you yesterday. I knew that today I was to find you in Aleppo’.

That is, Death’s foresight takes into account any reactions to Death’s activities.

Now suppose you think that a large portion of the Great Filter lies ahead, so that almost all civilizations like ours fail to colonize the stars. This implies that civilizations almost never adopt strategies that effectively avert doom and allow colonization. Thus the mere fact that we adopt any purported Filter-avoiding strategy S is strong evidence that S won’t work, just as the fact that you adopt any particular plan to escape Death indicates that it will fail.

To expect S to work we would have to be very confident that we were highly unusual in adopting S (or any strategy as good as S), in addition to thinking S very good on the merits. This burden might be met if it was only through some bizarre fluke that S became possible, and a strategy might improve our chances even though we would remain almost certain to fail, but common features, such as awareness of the Great Filter, would not suffice to avoid future filters.

Har Shulman fel eller rätt?