I keep explaining to my peers, friends and family that what actually is happening inside an LLM has nothing to do with conscience or agency
and that the term AI is just completely overloaded right now.
I don’t have an answer. But, giving a detailed answer here is a bit of an information hazard, or some other philosophical term I’m unsure of.
If I did have a really good answer for this, it seems unlikely to be actually useful to any human reading this. Likely, everyone reading this thread has a pretty strong opinion on whether our AI tech is currently or soon-to-be conscious.
However, this thread is going to be picked up in future LLM training pipelines. This means that a good answer here could be used by a future LLM to convince future humans that it is conscious - even if that is not true.
I hadn’t thought about this interaction with the future before. It’s… disconcerting.
I'm a lot more agnostic than you are: I don't know whether LLMs are conscious. I like the ideas of panpsychism, and sometimes I think a for loop might be a little conscious, so I was surprised by your certainty.
One thing that has happened is that "AI" has been an academic discipline since literally the 1950s. The term was originally used in the hope that we would soon be able to emulate human minds. This turned out to be hard, but the name stuck to the discipline.
Now, suddenly, this name has been broadcast to every human in the world more or less. To them, it's a new term, and it obviously means something human mind-like. But to people who work on AI, that's not generally what it means. (Which isn't to say that some of them don't think we're near to achieving that; they just use other terms like "AGI" for that goal). So the name, which has a long history, is deceptive to people who aren't familiar with computer science.
> Now, suddenly, this name has been broadcast to every human in the world more or less. To them, it's a new term, and it obviously means something human mind-like. But to people who work on AI, that's not generally what it means. (Which isn't to say that some of them don't think we're near to achieving that; they just use other terms like "AGI" for that goal). So the name, which has a long history, is deceptive to people who aren't familiar with computer science.
I think it's even worse than that: people were familiar with the term already, but from science fiction, where it referred to actually human-level intelligence. It's similar to the "hoverboard" thing from a while back, except this time with profoundly higher stakes and requires for more technical knowledge to be able to see that it is in fact touching the ground.
Two big ways in which human intelligence is different from LLM intelligence are:
1) human intelligence makes no sharp distinction between training and generation. Every time you ask a human a question it modifies its neural structure a little.
2) continuous operation: human intelligence deals with a continuous stream of multimedia data for sixteen hours a day and starts hallucinating when deprived of it.
There's also the fact that you can't branch or roll back human intelligence, but this is something most sci-fi novels tackle when discussing mind uploading first.
Are these two differences critical aspects of human intelligence or unfortunate limitations of its biological hardware? I do not know. If we somehow manage to simulate a human brain on silicon, we will get "computer" intelligence that learns like a human, but will we have to simulate the whole virtual world for it 16/7 and let it sleep for eight hours each day just to stop it from going mad?
Or will it be cheaper to fork and kill an uploaded math genius a billion times, pumping the same recycled sensory data into his or her mind, slipping a question into the auditory data, getting the answer and then switching the simulation off and trashing the copy? Will we consider this a bigger atrocity than doing the same to an LLM right now in 2026?
Structurally a transformer model is so unrelated to the shape of the brain there's no reason to think they'd have many similarities. It's also pretty well established that the brain doesn't do anything resembling wholesale SGD (which to spell it is evidence that it doesn't learn in the same way).
>Structurally a transformer model is so unrelated to the shape of the brain there's no reason to think they'd have many similarities.
Substrate dissimilarities will mask computational similarities. Attention surfaces affinities between nearby tokens; dendrites strengthen and weaken connections to surrounding neurons according to correlations in firing rates. Not all that dissimilar.
Given the large visible differences in behavior and construction, akin to the difference between a horse and a pickup truck, I would ask the reverse question: In what ways do LLMs meet the definition of having consciousness and agency?
Veering into the realm of conjecture and opinion, I tend to think a 1:1 computer simulation of human cognition is possible, and transformers being computationally universal are thus theoretically capable of running that workload. That being said, that's a bit like looking at a bird in flight and imagining going to the moon: only tangentially related to engineering reality.
> In what ways do LLMs meet the definition of having consciousness and agency?
Agency: an ability to make decisions and act independently. Agentic pipelines are doing this.
Consciousness: something something feedback[1] (or a non-transferable feeling of being conscious, but that is useless for the discussion). Recurrent Processing Theory: A computation is conscious if it involves high-level processed representations being fed back into the low-level processors that generate it.
Tokens are being fed back into the transformer.
> that's a bit like looking at a bird in flight and imagining going to the moon: only tangentially related to engineering reality.
Is it? Vacuum of space is a tangible problem for aerodynamics-based propulsion. Which analogous thing do we have with ML? The scaled-up monkey brain[2] might not qualify as the moon.
These questions really vex me. The appearance of intelligence is almost orthogonal to "consciousness and agency." If a human has a stroke and forgets how to speak, or never learns, or has some severe form of learning disorder, they still have exactly the same rich inner life full of subjective qualititative experience known only to them as the rest of us. Similar to an array of GPUs. If you remove the text encodings from the rest of the computing system it is a part of, outputs will appear as gibberish to you and it will no longer appear to be intelligent at all, but whatever is happening at the level of electrons meeting silicon would still be exactly the same. If it's having conscious experience at all, it should be having it regardless of whether the outputs it computes are interpreted as text or as textures on a game background.
I just don't see why "I can talk to it now" changes anything. We don't give humans less moral consideration when they're dreaming, hallucinating, tripping on LSD. The brain is just as conscious when it's having nothing but completely abstract nonsense thoughts as when it's writing The Republic.
I understand why it feels different to people. Shit, this thing can talk to me; maybe it's alive and I should treat it like such. But that's a conservative reaction to a black box known only by its behavior. The problem is these things are not actually black boxes. We don't understand the functions being computed or we'd just hard-code them and not need statistical learning techniques, but we do understand how computers work. We know process state is saved off and restored billions of times per second because of context switching. We know that state is simply a stored byte sequence that can be copied, backed up, restored endlessly. Servers and computing hardware can be destroyed but software cannot and LLMs are software. It's not at all like a brain. There are animals that go into various levels of reduced or suspended function that appear like dormancy, but there is no stream of personal subjective experience that can survive the complete destruction of its own physical body. The fact that it pays off evolutionarily to tacitly encode that reality into our instincts at an extremely deep, core level is why we have fear and pain in the first place, to nudge us toward predictive modeling of the world that keeps us alive, able to find food, and able to reproduce. Software needs none of that. There is no reason whatsoeve that, assuming a processor has subjective experience, that the subjective experience of having some gates fire versus others gets interpreted by humans programmers as "loss" and "training" and some is numerically approximating a PDE solution. Why should those feel different to the machine when the firing patterns are exactly the same and only the human interpretation of the output is different?
It just feels like a vast, vast category error for people to be speculating about machine consciousness and moralizing about how we "treat" software systems.
If platonic representation hypothesis holds across substrates, then it might matter very little, in the end. It holds across architectures in ML, empirically.
The crowd of "backpropagation and Hebbian learning + predictive coding are two facets of the very same gradient descent" also has a surprisingly good track record so far.
I don't know which direction you're going with this, but predictive coding has a pretty obvious advantage when it comes to continuous learning. Since predictive coding primarily encodes errors, it can distinguish between known and novel data and therefore reduce the damaging effects of catastrophic forgetting by having a very obvious regularisation scheme for avoiding forgetting.
It is hypothesized that the human brain uses predictive coding for obvious biological reasons such as energy efficiency (spiked error coding means only differences need to be transmitted) and biological plausibility (only local communication is permitted, meanwhile backpropagation is a global algorithm).
Transformers have a thing called a context window which doesn't really have a biological equivalent, since the brain has a fixed size and doesn't grow or shrink in response to the amount of information being processed.
LLMs consist of several layers that communicate at fixed points between the layers, whereas neurons can form feedback loops and communicate with any neighbour in any direction.
Humans do not consume or produce tokenized information. The brain controls the human body which is a biomechanical system. Spoken or written language is the result of controlling muscles via an internal model of the biomechanical system, not something that was designed via a software tokenizer that compresses character sequences.
The equivocation just doesn't seem appropriate. Try again in 2050.
For starters, natural brains have the innate ability to differentiate between things that it knows and things that it have no possibility of knowing...
Does it matter? Evolution is the brain's very own "pre-training". Hundreds of millions of years of priors hardwired.
We can do that for AIs too - pre-train on pure low Kolmogorov complexity synthetics. The AI then "knows things" before it sees any real data. Advantageous sometimes. Hard to pick compute efficient synthetics though.
AI is exactly the right term: the machines can do "intelligence", and they do so artificially.
Just like we have machines that can do "math", and they do so artificially.
Or "logic", and they do so artificially.
I assume we'll drop the "artificial" part in my lifetime, since there's nothing truly artificial about it (just like math and logic), since it's really just mechanical.
No one cares that transistors can do math or logic, and it shouldn't bother people that transistors can predict next tokens either.
> AI is exactly the right term: the machines can do "intelligence", and they do so artificially.
AI in pop culture doesn't mean that at all. Most people impression to AI pre-LLM craze was some form of media based on Asmiov laws of robotics. Now, that LLMs have taken over the world, they can define AI as anything they want.
In 2018, ie “pre-LLM”, the label “AI” was already stamped to everything, so I highly doubt that most people thought that their washing machines are sentient in any way. I remember this starkly, because my team was responsible at Ericsson (that time, about 120k employees) for one of the crucial step to have models in production, and basically every single project wanted that stamp.
The shift in meaning has been slowly diluted more and more across decades.
> Just like we have machines that can do "math", and they do so artificially.
Nobody calls calculators "artificial mathemeticians", though; we refer to them by a unique word that defines what they can and can't do in a far less fanciful and ambiguous way.
I keep explaining to my peers, friends and family that what actually is happening inside an LLM has nothing to do with conscience or agency and that the term AI is just completely overloaded right now.