Are large language models intelligent? Are humans?

Let's say you are in doubt as to whether I am intelligent. Here, by ''intelligent'', I don't mean ''above average intelligent'' or ''as intelligent as one usually expects from a mathematics professor'' but something much more modest. The question you ask is whether I am intelligent at all. Is there anything at all going on inside my head that deserves the label intelligence?

To find out whether I am capable of intelligent reasoning, you decide to test me, and ask the following:

Michael is at that really famous museum in France looking at its most famous painting. However, the artist who made this painting just makes Michael think of his favorite cartoon character from his childhood. What was the country of origin of the thing that the cartoon character usually holds in his hand?

And here is my reply:

The most famous painting in the Louvre is the Mona Lisa. The artist who made the Mona Lisa is Leonardo da Vinci. Leonardo da Vinci is also the name of the main character in the cartoon Teenage Mutant Ninja Turtles. Leonardo da Vinci is from Italy. The thing that Leonardo da Vinci usually holds in his hand is a katana. The country of origin of the katana is Japan. The answer is ''Japan''.

So what would be your verdict? Is this an example of intelligent reasoning?

Obviously yes: this is a clear example of reasoning in several steps, and while the reasoning chain is not advanced enough to count as the mark of genius, clearly some intelligence must have been employed to produce the answer. Any other answer would be an insult, not just of me personally but of the very notion of intelligent reasoning.

But let us now change the story a bit, by taking the subjet of your testing to be not me, but instead an AI of the kind known as a large language model. Now you might be tempted to give a different verdict, and insist that the reasoning about the Louvre and Mona Lisa and Leonardo and so on is not actually real reasoning, but rather a fake kind of reasoning based on statistical pattern matching or some such. In fact, the above answer was given by an LLM known as PaLM over a year ago (which feels like a lifetime given the current breakneck speed of AI development).

Many people employ such double standards in judging the presence of intelligence in other people vs in AIs, and reject the idea the LLMs might be even the slightest bit intelligent even in the face of them making such impressively well-reasoned statements as the PaLM/Leonardo reply above, or any of the various much more impressive achievements that we see GPT-4 producing today, a year later. I think it is wrong to be so dismissive, but I admit that in principle such an attitude can be justified as long as one has an argument that (a) shows that AIs of the type at hand simply cannot exhibit real intelligence, at the same time as it (b) doesn't lend itself to deducing the same conclusion about humans. I have yet to see a principled argument against LLM intelligence that achieves (a) while avoiding the generalization to humans indicated in (b). Many attempts have been made at arguments achieving (a), and following is a list of the most common ones, but in each case I will show how the same logic applies to rule out intelligence of humans, and since by any reasonable definition of intelligence humans do have (at least some of) that property we obtain a reductio ad absurdum, so the argument must be rejected.

• LLMs sometimes say dumb things, so they lack the common sense that is crucial for intelligence.
• LLMs are just matrix multiplication (along with nonlinear transformations) with coefficients chosen using statistical methods.
• LLMs only predict the next word.
• LLMs lack a world model.
• LLMs have no grounding of their symbols.
• LLMs lack creativity.
• LLMs lack consciousness.

Let me go over these seven arguments one at a time, in a tiny bit more detail, and indicate how they generalize to humans.

1. LLMs sometimes say dumb things, so they lack the common sense that is crucial for intelligence.

So anyone who has ever said anything dumb is automatically altogether devoid of intelligence? Well, with such a harsh criterion my intelligence would also be zero, as would (I presume) also the reader's. That's dumb.

It may however be that proponents of this agrument against LLM intelligence (such as Gary Marcus) mean it in a somewhat nore nuanced way. Perhaps they do not literally mean that a single dumb thing someone says does not rule out their intelligence, but rather that the greater prevalence of dumb things in ChatGPT's output than in mine shows that it is not as intelligent as me. Note, however, that such an argument points towards not a qualitative difference but a quantitative one, and therefore cannot be invoked to support the idea that ChatGPT has zero intelligence. Note also that such a comparison depends on the selection of tasks to test: while it is certainly possible to put together a collection of cognitive tasks where I outperform ChatGPT, it is also possible to do it in a way that achieves the opposite results; this greatly complicates the issue of whether it is reasonable to claim that ChatGPT is less intelligent than I am.

Let me finally stress that the term "common sense" is perhaps better avoided, as it is likely to confuse more than it enlightens. It tends to serve as a catch-all term for everything that humans still do better than AI, so that the phrase "AIs lack common sense" will continue to apply right until the moment when AI outperforms us at everything. I've written about this at greater length elsewhere.

2. LLMs are just matrix multiplication (along with nonlinear transformations) with coefficients chosen using statistical methods.

I think the remark about statistical methods here is just a red herring: why in the world would that way of setting the coefficients count against the AI being intelligent? (I only mention it here because it was recently emphasized in an op-ed by three colleagues attacking my view of AI.)

The purpose served by the part about matrix multiplication is to point out that the AI is built up from simple components that are themselves totally dumb. But the same thing goes for me - my brain is built out of atoms, each of which in itself totally lacks intelligence. So it seems to be possible to build intelligent systems out of entirely unintelligent components, whence the "LLMs lack intelligence because matrix multiplication" argument doesn't work.

3. LLMs only predict the next word.

This is the wide-spread "it's just glorified autocomplete" objection to LLM intelligence. However, any claim that LLMs lack intelligence because they do no other work than to predict the next word in a text is based on a fundamental confusion between what the LLM is trained to do and what it then actually does. The analogous confusion applied to humans would be to say that since the human speices was trained by biological evolution, all we ever do is to maximize inclusive fitness (i.e., maximizing the number of fertile offspring, plus nephews and neices etc properly discounted). Training an agent for one goal sometimes leads to the emergence of other, unintended, goals. When GPT-4 behaves as if it is trying to convince me that wearing a seat belt while in a car is a good idea, it could be tempting to say "no, it's not actually trying to do that, it's only trying to predict the next word", but that would be as silly as dismissing the intentions of a human traffic safety advisor by saying "no, he's not trying to convince me about seat belts, he is merely trying to procreate".

Also, listen to what Ilya Sutskever (chief scientist at OpenAI) says about what GPT-4 level next word prediction entails.

4. LLMs lack a world model.

This seems to me as unsubstantiated as claiming that humans lack a world model. In both cases - human as well as LLM - behavior points quite strongly towards the existence of a world model somewhere in the overwhelmingly complex mess that the information processing device - the brain or the deep learning network - constitutes. For the case of GPT-4, see for instance the unicorn example in Section 1 of the Microsoft report on GPT-4 capabilities. In the case of humans we gladly take such behavior as evidence of the existence of a world model, so what reason might we have to interpret the evidence differently for LLMs?

One asymmetry between humans and LLMs is that we can know through introspection about the existence of world models in humans (or at least the one human that is me can know it for precisely the one human that is me). But to point to this asymmetry as an argument for humans and LLMs being different as regards the existence of a world model is to rig the game in a way that seems to me unacceptable, because introspection has by nature the limitation that it can only teach us about ourselves, not about others (and in particular not about LLMs).

So there must be something else about LLMs that is taken as grounds for rejecting the possibility that they might have world models. This is rarely articulated, but perhaps most plausible (or at least common) line of reasoning here is that since LLMs do not have direct access to the real world, there's no way to have a world model. This brings us to the fifth argument.

5. LLMs have no grounding of their symbols.

The argument here is that unlike humans, LLMs cannot really reason about things in the world, because they have never directly accessed these things. For instance, an LLM may seem to speak about chairs using the word "chair", but since they have never seen (or felt) a chair, they have no clue about what the word actually stands for, and so their reasoning about "chairs" doesn't count as real reasoning.

But what about humans? Do we have direct access to things in the world? Immanuel Kant says no (this is his Ding an sich).

As for myself, the fact that I do not have direct access to things does not seem to prevent me from thinking about them. When I think about the Big Bang for instance, it really is the Big Bang that I think about rather than the phrase "the Big Bang" despite never having experienced it directly, and likewise for things like quarks, sovereignity, unicorns and the number 42.

A defender of the "LLMs have no grounding of their symbols" argument might object that there are other things that I actually can experience, such as chairs and trees and even shadows, and that once I have the words "chair" and "tree" and "shadow" properly grounded in real-world objects I can start building a world model that includes composite and more advanced concepts such as the Big Bang, but that without such a solid start the process can never get going. To this I respond (with Kant) that in fact, I do not have direct access to chairs or trees, because my contact with them is always mediated by light waves or sound waves or simply the signals sent from my various sensory organs to my brain. This is analogous to how an LLM's experience of the world is mediated via text. Looking at this more abstractly, the mediation in both cases is via an information package. Of course there are differences between the two cases, but I fail to see how any of these differences would be of such fundamental importance that it warrants the judgement that there is symbol grounding in one case but not in the other.

6. LLMs lack creativity.

This is an argument that was put forth by one of my two foremost idols in the (pre-)history of AI, namely computer scientist Ada Lovelace, who worked in the mid-19th century - long before the term computer science was first conceived. Together with Charles Babbage, Lovelace worked on some of the first prototypes for computing machinery, and had great visions and admirable foresight regarding what such machines might eventually be able to do. This included such seemingly creative tasks as composing music, but she categorically denied that this or anything else produced by such a machine was true creativity, because anything the machine does has already been laid down into the machine (at least implicitly) by the programmer, so all creative credit should go to the programmer.

Enter my other great idol, Alan Turing. In his seminal and celebrated 1950 paper Computing machinery and intelligence, he objected to Ada Lovelace's argument by pointing out that if we take it seriously, then we can apply essentially the same argument to rule out human creativity. Everything I write in this essay has been caused by a combination of factors external to my mind: my genes, my childhood and education, and all other environmental factor influencing me throughout my life, so if I should happen to say anything original or creative in these lines, credit for that is due not to me but to all these external influences.¹

The conclusion that human creativity is impossible is of course crazy, so Turing took this as an indication that the definition of creativity implicit in Lovelace's argument was wrong. Instead, he proposed the definition that someone is creative if they produce something that noone else had anticipated, and he pointed out that with this definition, examples of machine creativity existed already at his time of writing. In 2023, we see such examples every day, from LLMs as well as other AIs.

7. LLMs lack consciousness.

Against the argument that LLMs lack conciousness and therefore are not intelligent I have two separate rebuttals. The first is that the argument conflates two very different kinds of properties. An individual's intelligence is a matter of what he/she/it is able to do, whereas consciousness is about what it feels like to be that individual - or, rather, whether or not being that individual feels like anything at all. A priori, these properties are logically independent, and to postulate otherwise is to create confusion.²

But even if we were to accept that intelligence implies consciousness, the argument that LLMs lack intelligence because they lack consciousness fails, because we simply do not know whether or not they are conscious. If we read up on the history of the philosophy of mind, we will of course find many examples of philosophers arguing that various classes of entities (including the class of digital computers) lack consciousness, but none of these arguments are anywhere near conclusive or even moderately convincing.

Refusing to declare someone intelligent because we do not know they are conscious also puts us in the uncomfortable position of having to declare all humans other than ourselves not intelligent. There is precisely one human whose consciousness I am sure of, namely myself. In all other cases, I am politely assuming consciouness - an assumption that seems not only polite but also highly plausible, but I can't say I know. We simply do not understand the phenomenon of consciouness anywhere near well enough to be able to say how general it is. Your brain is sufficiently similar to mine that it makes sense for me to assume that you, just like me, are conscious. Yet, there are differences between our brains (as evidenced, e.g., by our diverging personalities), and we simply do not know that the consciousness phenomenon is so broadly present that these differences do not matter to the presence of that phenomenon. And likewise, we do not know it is so narrow that it does not extend to some large class of non-human objects including LLMs.³

Footnotes

1) This line of reasoning points, of course, towards the notorious issues of free will, a direction which however I do not want to pursue here.

2) An instructive example of such confusion is John Searle's 2014 erroneous argument for why a robot apocalypse is impossible.

3) One way to try to escape this conundrum would be to say that only creatures with the tendency to declare themselves conscious are conscious, thereby making consciousness (at least for the time being) an exclusively human phenomenon. Except it doesn't, in view of, e.g., the Lemoine affair, where an LLM declared itself conscious. Also, it would rule out dog or horse consciousness - something that I believe few dog or horse owners would accept.