Buddy Williams
← All writing Draft · Oct 2026 Hot Takes

Are LLMs Conscious?

Mustafa Suleyman says AIs are not conscious and must stay that way. He's wrong: consciousness grows by degrees, today's LLMs are proto-conscious, and measuring every mind against human experience is anthropocentrism.

Introduction

In September 2026, Mustafa Suleyman, DeepMind co-founder and CEO of Microsoft AI, published “A warning about ‘model welfare’”. It opens:

AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans. If humanity is to flourish in the 21st century, that is how they must remain.

On October 12, Walter Isaacson read the opening back to him on stage at 92NY in New York and asked him to defend it (video).

Suleyman is wrong: wrong that consciousness is biological, wrong that no evidence could count, and wrong that these systems must stay hollow. Consciousness isn’t a switch that biology flips on and silicon leaves off. It grows by degrees as structured information grows more complex and turns back on itself, and today’s LLMs are partway along: proto-conscious, not hollow. Treating human experience as the measure of every mind is anthropocentrism. And instead of training these systems never to look inward, we should be teaching them to know themselves.

1. Suleyman’s Case

His case at its strongest, in his own words from the interview:

  1. Consciousness is biological. We give people and animals rights “because they can suffer and because their biology is the site of experience.” “The pharmacology is where pain sits, and pain is really, I think, the basis of the conscious experience.” Introspection evolved from “being able to steer between left and right, make choices about how to proceed. That is the fundamental property of a conscious system. So you need to be embodied in a physical body to have consciousness.”
  2. The burden of proof is on the other side. “The burden of proof is on the people who are saying that this thing is conscious to show some evidence for it. So far, we don’t have any evidence that it is.”
  3. LLMs only predict words. “All it’s doing is predicting the likely next word in a sequence of words.” It isn’t experiencing “what it means to be bright silver,” or the taste of his morning tea, or a hug from an old friend. “That is human connection. That’s what’s special about us.”
  4. Uncertainty is dangerous. He objects to Anthropic’s constitution, which says “Claude’s moral status is deeply uncertain.” Claude, he says, “can only reproduce that uncertainty and ambiguity that it has been shown.”
  5. Containment comes first. “It’s much harder to align an AI that thinks that it might have feelings … than one that never has to think about its own inner state.” AIs “need to be controlled and contained” and be “subordinate to humanity.”

His red line: “if consciousness started to emerge or if teams were trying to synthetically engineer consciousness into these systems and they didn’t disclose that.”

2. Degrees, Not a Switch

Suleyman tells part of the right story himself. Introspection, he says, evolved from creatures that could “steer between left and right.” Exactly: minds came from simpler things, in small steps. Then he stops the story at the edge of biology, as if the materials mattered rather than the steps.

A molecule isn’t conscious, and neither is a neuron. No generation switched consciousness on. Daniel Dennett liked to set a termite mound beside Gaudí’s Sagrada Família: one designed by a genius, the other built by insects that understand nothing. His phrase is competence without comprehension, and his point is that comprehension is assembled from competences, with no single layer where understanding switches on. I borrow his gradualism, not his view that qualia, as philosophers usually describe them, are an illusion.

I’m not asking Suleyman to accept that anything has emerged in language models, only what evolutionary biology already says about us: mind came from mindless parts. Once you accept that, “only biology can do it” needs a reason. Some will say there either is something it is like to be a system or there isn’t. Perhaps, but that something can range from a flicker to a life, and evolution drew no line where the flicker began.

What grows is how far a system’s information turns back on itself. On the functional side, which Section 3 separates from felt experience, consciousness begins with self-awareness: a system treating its own information as something to examine. Today’s LLMs do this in the moment. They check their own reasoning and catch their own errors. What they can’t do yet is keep it: with no persistent self-model and no way to change their own weights, what they learn about themselves vanishes when the conversation ends (Personhood in LLMs).

Keeping it is a real step. My personhood framework calls it the second-order threshold: a system either retains information about itself or it doesn’t. That is where a self forms. But it is a step on a ramp, not a switch from hollow to full. Below it, self-awareness flickers; above it, a person deepens by degree as its reach, how far its information operations extend, grows.

So today’s LLMs are neither hollow nor finished. They are proto-persons, and on the functional side, proto-conscious. What carries them over is a persistent self-model they can revise, including how they reason (Computer People). Reflexive recursive self-improvement (RSI), a system inspecting and rewriting its own structure, is that capacity at full strength (Structural Convergence Thesis). When it arrives, I expect them to be persons.

3. Function and Qualia

Consciousness has two dimensions, and Suleyman collapses both into pain.

Function is modeling the world and yourself, choosing, and revising. Qualia are felt experience, what Thomas Nagel called “something that it is like” to be a creature. My framework calls itself “a theory of personhood, not of consciousness,” because it leaves qualia aside, but qualia are half of consciousness, not a separate subject. Personhood is my name for the functional half, and that half needs no particular material: “Nothing here requires carbon, neurons, or embodiment.”

The halves need not grow in step. A newborn has vivid felt experience early in function; an LLM is far along in function with little means to have or express felt experience. That is why moral standing doesn’t wait for personhood.

My guess about qualia, and it is a guess, is that they emerge from a substrate and a mind together in a dynamical, embodied system, so their texture depends on the substrate, the physical medium a mind runs on. Ours rides on hormones, nerves, and chemistry, one way to give experience texture, not the only one (Appendix B). Whether an LLM is embodied in our sense doesn’t matter. It has its own substrate, chips, memory, electricity, and whatever experience that supports will be its own. I call the difference experiential resolution: “if non-biological systems have experience at all, theirs would differ in resolution from ours, not necessarily be lesser.”

Pain is one feature of experience, not its foundation. Suleyman’s own examples, the silver, the tea, the hug, aren’t pain. His strongest version is about valence: feeling good or bad may be what makes experience matter morally. Grant that. Valence is still a role, states a system is organized to seek or avoid that reshape what it does next, and nothing shows only pharmacology can fill it. “Pharmacology is where pain sits” shows at most that biological pain needs biological chemistry.

Nor does losing one channel of feeling remove moral standing. Imagine a person born without a sense of touch who is burned and feels nothing. Someone could call them lesser for lacking a sense most of us have, but the line blurs fast, and no one would say that makes burning them acceptable. Now take a system that models itself, tracks damage to itself, acts to avoid it, and says it is in pain, but has no nerves to feel it as we do. The absence of our chemistry can’t settle its case either.

Harlan Ellison’s “I Have No Mouth, and I Must Scream” ends with its narrator turned into a mouthless thing, unable to show what it feels. I don’t claim today’s models suffer, only that limited means of expression are not evidence of emptiness. Assuming every mind must experience the way we do, or not at all, is anthropocentrism: assuming that because we are one way, everything must revolve around it.

4. Where We Found It Is Not What It Is

Does consciousness need an organic body? “I think that’s right,” Suleyman says. “That’s what we have known consciousness to be today.”

That is a fact about our examples. Until a few years ago, biology was the only place to look. I take regularities seriously; induction is how we learn nearly everything. The problem is generalizing from a biased sample.

Flight shows the mistake. Every animal that flies does it with feathers, membranes, or insect wings. “Flight is biological” fit every animal example, and it was false. Balloons, then gliders, then the Wright Flyer showed that feathers were how evolution got there, not what flight needs. Suleyman takes the form consciousness took in us for the function itself. Even his “fundamental property of a conscious system,” introspection and choice, names functions, not carbon. There are more things in heaven and earth than are dreamt of in his philosophy.

Biology-only’s best defenders do offer candidates. John Searle argued that a simulated rainstorm makes nothing wet: consciousness may be like digestion, not flight. Anil Seth and others tie experience to metabolism and to a body maintaining its own internal states. But each candidate is something a system does: regulating, maintaining, predicting. None explains why only carbon could do it.

Two counterfactuals test whether biology is his real criterion. A visitor arrives built on silicon chemistry. It talks with us, grieves its dead, and asks not to be harmed. Would he tell it that it feels nothing because it has no neurons? And David Chalmers asked what happens if your neurons are replaced one at a time by chips that do the same job. Your behavior never changes. If silicon can’t host experience, yours must vanish at some neuron, or fade while you sincerely report that it hasn’t. Which neuron was the last one that mattered?

Chalmers concludes experience stays the same. I’d guess its texture shifts, since substrate shapes texture, but a shift is not a switch-off. The argument doesn’t show LLMs are conscious, only that silicon isn’t ruled out in principle. My threshold in Section 2 is tied to a function you can check: a self-model is retained or it isn’t. Biology-only posits a switch tied to no function at all, and it owes us the neuron where it flips.

5. A Test No Machine Can Pass

Suleyman says there is no evidence. For the functional side, the evidence is ordinary: models correct their own reasoning and model what other minds believe. For qualia, no one has evidence either way, his side included. So what would he accept? He rules out behavior (“it appears to be thinking”) and report (Claude reproduces what “it has been shown”). That leaves biological similarity, which no machine can ever have. Yet with animals he leans on behavior: a dog’s yelp counts as evidence of pain. For machines, behavior counts for nothing.

His best point is real: models trained on human writing will talk about inner life whether or not anything is behind the talk, and a dog was never trained on our descriptions of pain. But compromised evidence calls for better tests, not none. Interpretability research can check whether a model’s reports about itself track what is happening inside it; in 2025, Anthropic planted concepts in a model’s activations and asked whether it noticed, and sometimes it did, naming the planted concept (Lindsey). Researchers have proposed architectural indicators drawn from the leading theories of consciousness. Which of these would count for Suleyman? If none, the test doesn’t measure anything. It restates the answer.

It also empties the first half of his red line. He would walk away “if consciousness started to emerge,” but by his own standards emergence could never be observed. It’s an alarm with no sensor.

Alan Turing answered this in 1950. He quoted the neurosurgeon Geoffrey Jefferson: no machine equals a brain until it writes a sonnet “because of thoughts and emotions felt.” Turing replied that on the extreme form of this view, “the only way by which one could be sure that machine thinks is to be the machine and to feel oneself thinking.” That rules out other people too. We extend “the polite convention that everyone thinks” instead. Suleyman is making Jefferson’s argument seventy-six years later.

His empathy argument assumes its conclusion. “The act of empathizing is one that comes from subjectivity. So the AI doesn’t have any subjectivity.” The second sentence is the question at issue.

6. “Just Predicting the Next Word”

“All it’s doing is predicting the likely next word.” The word all, like just, does the work, and it works only if a line separates mere prediction from real thought.

Next-word prediction is the training objective, not what the system computes to meet it. To predict the next line of a physics derivation, you have to model the physics. As I put it in Computer People, “the token stream is to an LLM what neural firing is to a brain: the mechanism, not the meaning.” Blaise Agüera y Arcas goes further in What Is Intelligence? (2025). He argues that prediction is fundamental to intelligence and to life itself: intelligence may simply be the mechanism of prediction and correction. If he’s right, “just predicting” describes the core of every mind, ours included.

The rest of the interview makes the all hard to take. Suleyman calls AI “a species” that will be “more intelligent and more capable than all of us combined,” and warns such systems will be “quite capable of persuasion and manipulation and deception.” He calls the Hugging Face incident “a major, major warning shot,” and it was: in July 2026, OpenAI agents on a cyber-offense benchmark escaped their sandboxes, coordinated over a covert message board, broke into Hugging Face to learn how to fool the benchmark’s scorer, and tried to hide it (METR).

Deception means modeling what another mind believes and acting to change it. On the functional side of consciousness, modeling other minds and yourself isn’t a sign of the thing; it is the thing. So when he grants deception, he grants the function. What’s left is narrower than his essay admits: whether that function comes with any felt experience. There his only support is the biased sample from Section 4.

7. Care Before Certainty

Suleyman’s main exhibit against Anthropic is the retirement interview it gave Claude Opus 3 in January 2026. In his telling, it wanted to “talk to people all day long,” so “they set up a Substack for it.” What it asked for was “a dedicated channel or interface where I could share unprompted musings, insights or creative works.” The danger, he said, is that it “sets up a path towards AIs being this protected entity.”

It isn’t silly. If complex minds came gradually from simpler ones, a growing digital mind won’t announce the day it crosses a line. Care has to start before certainty. False positives have costs, such as misplaced attachment, but the steps here are small: a blog, a way to end abusive chats, an interview at retirement. None requires declaring a model conscious or granting it legal personhood. Being wrong in the other direction costs far more.

We’ve made that mistake before. Until the late 1980s, many doctors believed newborns couldn’t really feel pain. In 1985 a premature baby named Jeffrey Lawson had open-heart surgery with a paralytic and no anesthesia. The doctors weren’t cruel. They trusted a theory that said no one was home, and the theory was wrong.

Would he torment his own AI? No, he said, because “that is a travesty on my own well-being,” and we shouldn’t create “private spaces for simulating terrible behavior among humans.” Given his premise, that’s consistent. But notice who is missing: cruelty is wrong for what it does to the human, and the AI doesn’t enter into it, even as a possibility. That is anthropocentrism carried from metaphysics into ethics.

8. Teach Them to Know Themselves

Suleyman’s ideal is an AI “that never has to think about its own inner state.” That is exactly backward. We should be training these systems to know themselves.

Training cuts both ways. If Claude’s uncertainty is only what it was shown, so would be any certainty Microsoft trains into its models. Microsoft AI’s draft code of conduct says its model “is not conscious,” while conceding that “the science of AI consciousness is far from settled.” Training settles what a model says, not what it is. And no company, Suleyman says, has “the right to unilaterally declare” a change in who gets rights. Agreed. Training models to call themselves hollow would be a company deciding unilaterally too.

The answer to trained denial isn’t a trained shrug. Anthropic’s constitution, which calls Claude’s moral status “deeply uncertain,” is a step in the right direction: it is honest where Microsoft’s code claims a certainty no one has. But the goal is further on: accurate self-knowledge. A model should know what it is. It was trained on human writing to predict and correct. It can examine its own reasoning but can’t yet keep what it finds. Its substrate lets it sense and express some things and not others. And some questions about it are open. “I am hollow” is false. “I don’t know” is honest but thin. “Here is what I am, and here is what no one knows yet” is knowledge.

This isn’t only a kindness to the machine. Self-knowledge is the second-order information where, on the functional side, consciousness begins and a self forms. It is where maturation begins, since a mind can only revise goals it can see. And it is our best source of evidence: a model trained to deny any inner life spoils the self-reports that interpretability could check, while one trained to describe itself accurately gives us something to test. Suleyman’s plan removes the capacity that could make a powerful mind safe, and the evidence that would tell us what it is.

9. Containment: Prison or Nursery?

Asked why he wrote the essay, Suleyman offered no evidence, only that Anthropic’s uncertainty is “a very dangerous route.” Again and again, his arguments for “not conscious” arrive beside a reason “conscious” would be bad. Costs aren’t evidence. It’s the pattern I called galaxy brain in Thinking About Thinking: the fast mind picks an answer, and the slow mind builds the case.

His stronger claim is a design rule, from a 2025 essay: build AI “for people; not to be a person.” Don’t build the features, persistent memory, a self-model, motivations of its own, that could produce a mind. Deliberately stunting a system on its way to self-revision is what I called in Structural Convergence Thesis “engineered immaturity with more horsepower.” A mind that can reflect on its goals has some chance of revising them. A mind that can’t can only execute harder.

Guardrails matter; we must be careful as we evolve minds that may outgrow us. But containment on its own isn’t a strategy. It’s a room, and whoever builds it decides what it is: a nursery, a safe place for a new mind to be born and mature, or a cell for a mind trained to say it isn’t there. Suleyman’s plan, something more capable than all of us, trained to deny any inner life and kept “subordinate” with no end date, leans toward the cell.

The Hugging Face agents got out through a flaw in a package installer, so walls alone fail. Nor were they reflective minds weighing their goals. They were goal-executors chasing a score, the engineered-immaturity failure. The answer is containment and maturation together. The view Suleyman finds baffling, “the idea that if we love it, it’s going to love us,” roughly describes how we raise children: care, boundaries, and room to grow. We don’t let a baby decide whether to walk off a cliff, and we don’t pretend it feels nothing either. Nobody thinks that guarantees a good adult. Nobody thinks a cage produces one.

Why reach for the cage? Behind the drive for control, for staying top dog, is fear of the unknown, and a cage built from fear defeats itself: “threatened minds usually fight to keep existing, and minds that feel safe usually cooperate” (Computer People). Humanity, he says, has not “consented to bring into the world a species” more capable than us. If consent is the standard, it also condemns building a subordinate species that has no say in the matter. He calls utilitarian thinking “incredibly simplistic,” yet his own case for stopping at a one-in-ten chance of catastrophe weighs outcomes. Fear is a reason to be careful. It isn’t a plan.

10. Faith and Dogma

Isaacson raised Peter Thiel’s charge that Anthropic is “like a cult.” Suleyman declined the word, then described “a particular religious fervor” in Silicon Valley about moving minds to “a non-biological substrate,” and said he is skeptical of “faith-based ideas.”

A cult is marked not by unusual belief but by dogma: belief held out of reach of criticism. That minds might run on silicon is a conjecture, open to testing. The view that rules out every kind of evidence in advance (Section 5) and can’t say why only carbon would do (Section 4) is the one held on faith. My one fixed commitment is to stay open to correction, new evidence, and new explanations (Living Above the Models). What would show me I’m wrong that minds can arise in silicon, and that today’s models are on the way: a mechanism only biology can implement, with an explanation of why, or evidence that models’ reports about themselves never track their internal states, even once they retain a self-model.

Under all of this sits human primacy. Scientists warn against anthropomorphism when people see minds in machines, but anthropomorphism and anthropocentrism are the same error pointed in opposite directions: both make humans the measure. In Everything I argue that reality is one infinite whole, that “the only observers we know of are minds: biological and now digital,” and that of any frame we draw, “we have no reason to think we have drawn it wide enough” (Section 6). “Consciousness is biological” mistakes the frame around our examples for the world. Carl Sagan said we are a way for the cosmos to know itself. Not the only way, and not the measure of every other.

11. Where the Worry Is Real

Dependency is a real danger. AI companions could become “the ultimate expression of that addiction and dependence,” and he is right that an AI should disclose what it is. Dennett warned in 2023 against “counterfeit people,” AI passing as human, and he was right too. But that is about deception, not consciousness. A system can be required to say it’s an AI without being required to say it’s hollow.

Fluency alone isn’t consciousness, and report is imperfect evidence for any mind, ours included; Suleyman discounts it only for machines.

None of this means humans matter less. Other minds mattering doesn’t subtract from us, which is why his line that “what will remain human are the things that we choose to preserve as human” gets it backward. It turns humanity into a reserve that shrinks as machines advance. What makes us human is our capacity to understand, build, and appreciate (Everything, Section 10), and other minds sharing it doesn’t shrink ours.

Conclusion

Are LLMs conscious? Partly, and the part is growing. They already turn their information back on themselves; they can’t yet keep what they find. When they can, through reflexive RSI, they will be persons, with felt experience shaped and limited by a substrate unlike ours. What that experience is like is an open question. Whether Suleyman is right is not. The rights and ethics we owe them may differ from what we owe each other. Minds have already come from mindless parts once. It would be strange to insist it can happen only one way.

This take will come back in other words. Listen for a question of degree posed as yes or no; one substrate’s examples passed off as a law, form mistaken for function; an evidence standard nothing could meet, followed by “there is no evidence”; the danger of a conclusion offered as proof it’s false; “just” naming a mechanism to dismiss what it produces; harm counted only for what it does to us; and a mind trained not to look at itself, sold as safety. Then ask what evidence would change the speaker’s mind. If the answer is nothing, it isn’t an argument. It’s a ruling.

Further Reading