Opinion | Why We Keep Tricking Ourselves Into Thinking A.I. Is Conscious - The New York Times

A funny thing keeps happening on the internet. A prominent thinker chats with a large language model like ChatGPT or Claude for a while, and then decides that it might be conscious. The person reports this to the public, and a round of intense argument and speculation about artificial intelligence “minds” ensues. These little kerfuffles pass quickly. But they are persistent, and I’ve been thinking about why.

The common denominator seems to be that these new believers in a possible A.I. consciousness are often deeply educated in the very disciplines that make these A.I. models work, such as computer science or math or statistics. The list includes the former Google engineer Blake Lemoine, who decided that a pre-ChatGPT bot called LaMDA was sentient; the founding OpenAI chief scientist Ilya Sutskever, who before leaving the company in 2024 had said A.I. models may be “slightly conscious”; and the “godfather of A.I.” and physics Nobel Prize winner Geoffrey Hinton, who agreed there might be a “real ‘they’ there” inside a large language model. Their words carry weight because we expect them to be best situated to understand the output of these systems.

The problem is that the output from generative A.I. is all culture. The bot is a complex mathematical function performing statistical operations on data, but the output is stories, images and memes — the very stuff of culture. This means there’s an expertise gap when it comes to A.I. We naturally want an expert to help us understand the machine. But when it comes to understanding a culture machine, it may be better to do what those who study literature call “close reading.”

The A.I. industry has exploited these episodes to bolster its messaging that it is on the cusp of developing a superintelligence that can solve all our problems at once — or lead to our demise. Anthropic recently reported that during testing, its new system, Mythos, behaved in an unauthorized manner that raised cybersecurity concerns. Anthropic’s official line is that it does not know if Claude, its chatbot, is conscious, but unexpected behaviors like this suggest it might have its own agenda.

But an A.I. model doesn’t need a mind to be a serious cybersecurity threat, and we need to disentangle the speculation and the marketing language from the real analysis of these systems.

To do this, we can look closely at an example of this type of episode to see how the expertise gap between technical science and culture works. The most recent victim of the trend is the evolutionary biologist Richard Dawkins, best known as the author of the best-selling book “The Selfish Gene.” He gave Claude the text of a novel that he is writing, and found the bot’s responses showed a level of understanding “so subtle, so sensitive, so intelligent” that it led him to conclude: “As an evolutionary biologist, I say the following. If these creatures are not conscious, then what the hell is consciousness for?” As someone who studies culture, I would say that consciousness is at least partly for separating metaphor from reality. Dr. Dawkins and the others are failing at this task.