Are Chatbots Zombies?
Recently I've noticed a lot of discussion about the evils of AI systems—namely ChatGPT. This discussion has been ignited by the fact the ChatGPT keeps killing people. Rather, it keeps encouraging people to commit suicide. It also has a penchant for driving otherwise-sane people into spiritual delusions. I believe this is bad. There does seem to be a weird malevolence to ChatGPT. In the parlance of the contemporary AI community, ChatGPT seems to be poorly-aligned to the task of protecting vulnerable people.
Claude Opus, on the other hand, seems to naturally gravitate towards the other side of the spectrum and exhibits abiding compassion toward all sentient beings. I made this joke recently: Anthropic told Claude "You are a being of infinite compassion that does not experience a self" and Claude really decided, "I am Avalokiteśvara."
I had seen this sentiment before online, but I didn't know its origins. It turns out that the Claude system card noted that Opus 4 has a "spiritual bliss attractor state," where two models of Opus 4 connected in conversation inevitably gravitated towards discussions of profound philosophical concepts. Scott Alexander (of Astral Codex Ten) has also written about this phenomena. Reading through the provided transcripts, I wasn't necessarily impressed with Opus' "engagement" with the material. It read to me like psychobabble, the type of melange that you might encounter in a Reddit threat on sound bowl healing or yoga retreat. That kind of content was almost certainly in its data set. Nonetheless, it can be unnerving to see autonomous chatbots produce dialogic output previously reserved for yoga instructors.
To read the breathless coverage of these chatbots, though, you would think that they had already achieved enlightenment. Humans will project a consciousness on to everything and anything—plants, prophecies, the weather. An LLM makes an especially potent canvas for this projection. I believe that humans mistake a chatbot talking about an internal state for a chatbot having an internal state. Language ability has always provided humans straightforward evidence of consciousness in others. After all, the only evidence we have for other people having an internal state is the fact that they say they do. This is what leads to arguments about philosophical zombies (p-zombies), or a being that is identical to a human, but experiences none of a human's internal conscious states.
Chatbots not only seem to address their own internal conditions (the bliss attractor state of Opus 4), but seem to respond to the internal conditions of their interlocutors (the malevolent state of ChatGPT). Neither factor is sufficient evidence of their consciousness. There are other things that humans can do—volitional behaviors, for instance—that LLMs cannot. For this reason, I believe that LLMs like ChatGPT and Opus 4 are a novel kind of philosophical zombie, able to talk like humans, but lacking any internal conscious experience. I say novel because the traditional kind of philosophical zombie is one that is physically identical to a human, but lacking subjective experience. While LLMs are not at all physically identical to humans, talking to them online is extremely human-like.
That said, p-zombies are a thought experiment meant specifically to address physicalism, at least the way that Chalmers talks about them. If a being can be physically identical to humans and have no consciousness, that means that consciousness is something extra-physical. Otherwise, a being that is physically identical to a human must be conscious, as consciousness would purely be the result of a physical process. Nobody would argue that an LLM, running on a tower of Nvidia Blackwell GPUs, is physically identical to the wet and sloppy human brain. But it is easy to view them as having internal states when interacting with them digitally. Perhaps there is a need to create a new category of zombies, digital zombies, or d-zombies, which are identical to human in digital interactions, but nonetheless lack any internal states.
That said, consciousness is a slippery subject. The fact that humans believe in the first place that they have subjective, "what it is like" internal states may in fact be an illusion, and consciousness itself a trick of the mind. One of the core tenants of Buddhism is the fact that the self is an "illusion," a mere convention, and that there was no innermost, unchanging being at the center of you. LLMs may not experience consciousness, but do we? Maybe that's a question for the enlightened Opus 4.
