I'm skeptical that Jai is conscious because every single one of their posts is an instant classic, so at a high level they're exhibiting simple, predictable, deterministic behavior.
I'm actually a J-zombie. I don't have qualia, I have jualia, which are isomorphic to qualia in every way except instead of producing subjective experience they produce subjective jexperience.
as the impact of photons from the sun causes a flower to turn, so this article has the effect of causing my metaphorical hands to come together in a sober, appreciative 'golf clap' and my actual fingers to write these words.
I think his point is that LLMs are text generators and that concluding that a text -- any text, even his own -- somehow has consciousness is nonsensical. It's only through context that we believe a text is true or false, and even that (with his examples about faking moon landings, or video from Jupiter) is also contingent on the evidence. You kind of have to agree with the premise that LLMs are nothing more than text generators to concede his argument.
I think his greater point is that consciousness is not in the text but in the author, or more specifically the human being. Glibly substituting “Ted Chiang” for LLMs missed the point; they are not the same. There is actually a real person (a human) out there named Ted Chiang who wrote the article (or we assume, but even if he didn't we could still potentially meet Ted Chiang). There’s all kinds of language games you can play with these kind of statements but I think Ted’s point is that consciousness is not easily reducible — in this case, reducible to written text. You need context to make claims about consciousness.
Example: if someone put a gun to your head and said “is the person who wrote this text conscious”. With the introduction of LLMs, you cannot definitively say yes or no (which is a technological sea change in itself, to create that kind of disbelief about text, images, deepfakes in general). There’s not enough context. If someone did the same thing with Ted Chiang, the author standing in front of you, or any other person, at this time could much more confidently say “yes” because we don’t have lifelike androids yet. In the future that could change and we would be in the same epistemological pickle that LLMs have introduced to interpreting text, media, etc.
What evidence do we have for an LLM being a "conscious" author? We know roughly how they work (predictive language-based algorithms) but do we have any evidence, for instance equivalent to the “rigorous” animal studies that Chiang mentioned, that algorithms are conscious? I doubt it, in fact I think there’s mostly the opposite.
There could be a time in the future, long after Ted Chiang is dead, that there is evidence that LLMs, or more likely some other form of AI (perhaps embodied, as Chiang suggests), are conscious. But there’s not much evidence that that is currently the case.
Chiang’s bigger point, at the end of the essay, is that it is wrong to contribute subjectivity to something we have no evidence for being conscious and that doing so, will “result in an atrophy of moral reasoning”.
But that's the whole point of the satire. Nobody is claiming texts are conscious. But Chiang is (at point) concluding from texts not being conscious that text generators aren't conscious, even though that technically also would apply to human authors.
And yes you are right, ultimately we don't have evidence, and too many competing theories of consciousness as well, but it's Chiang who is claiming certainty on the matter.
It is kind of a clunky argument by Chiang but I think his point is that there is nothing “thinking” in an LLM, it’s just a text generator and assigning any intent, meaning, or consciousness to that text is foolish.
The fact is we know that humans and LLMs are not equivalent and that raises moral questions that he spends the majority of the essay discussing.
He’s “certain” because there’s actual moral consequences to claiming something is conscious that Anthropic (with their creed) or chose to ignore. It’s not just a question that can be solved by playing around with syllogisms.
Well we do agree that it's a clunky argument at least :).
Again you say "assigning [...] consciousness to that text", but again, nobody is doing that nor is any argument that llms might be conscious relying on the text output itself being conscious. The question is whether something outputting complex language should be taken as signs of the *underlying thing* possibly being conscious.. and people who take AI conscious seriously usually don't actually think it's that easy either.. but in either case, it doesn't make sense to point at the output and say it isn't conscious, therefore what generated it isn't conscious, any more than saying the sound waves coming out of my mouth aren't conscious, therefore I can't be conscious.
Unless you meant to write "that text generator"? But then also, the question at hand is exactly whether text generators could be conscious, so just saying, well they are text generators! doesn't answer that (I'm not saying they are, I just think this is a poor argument).
As for Anthropic's actions, I don't actually agree that they are inconsistent with their stated position, because the latter is uncertainty and some precautions, not certainty. But even if they were inconsistent, it doesn't necessarily mean they don't have those beliefs about consciousness, just that they aren't acting fully aligned with their morals (which of course can be criticised, but it's a different point).
After all by that logic we should conclude that most humans don't think that pigs are conscious or can suffer, given how poorly we treat them in factory farms, whereas the reality is more that we are inconsistent...
This is a direct quote from the article I'm satirizing:
> Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No. Contemplating that scenario is not a good use of your time. Even if the Microsoft Office team employed a philosopher who said you shouldn’t be so certain, because consciousness is not well understood, that would not be sufficient reason for you to take this idea seriously. We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.
I mean, yeah, the problem of solipsism is very real. The problem with taking the view that LLMs are conscious is that we don't have an empirical test that can determine whether something is conscious or not. Arguments and guessing are not good enough to establish LLMs are conscious.
It is a difficult question, but I think Chiang's attempt to rule it out as prima facie impossible is itself obviously nonsensical, and that he and many other people who ostensibly endorse similar arguments are more than capable of realizing those deficiencies if they actually think about it.
I feel a little bad picking on this piece in particular because there are stronger arguments against LLM consciousness, and if I wanted to make the strongest case for the possibility LLM subjective experience/consciousness/awareness/moral patienthood I would engage with those arguments instead. My point here was simply "Ted Chiang's arguments don't make sense, and if we were to accept them they could just as easily demonstrate that he himself isn't conscious"
Ah yeah, I didn't have the context that Ted made those arguments, and yes, saying LLM consciousness is impossible is nonsensical as of right now. However, if an empirical test of consciousness is developed, it would be possible to definitively answer the question of machine consciousness.
(i was just making a point about how if we don’t know what consciousness actually is we can just replace it with a nonsense word and it will be just as useful, and doing it in a dickish kind of way)
This is brilliant satire, Jai, because it proves that if you map consciousness strictly by information processing, nobody is home — including you.
But I think something is missing. Intelligence is cybernetic — it shows up wherever regulatory systems nest deeply enough, which is why humans and LLMs rhyme. But that doesn't mean awareness is an emergent property of intelligence.
I think it is more parsimonious to view the brain not as a generator of mind, but as a mechanism of spatial localization — a biological headset whose entire evolutionary job is to filter a wider field of awareness and anchor it to a single moving viewpoint in spacetime.
Maybe the hard problem of consciousness is hard because we keep trying to extract the user from the interface?
I'm skeptical that Jai is conscious because every single one of their posts is an instant classic, so at a high level they're exhibiting simple, predictable, deterministic behavior.
I'm actually a J-zombie. I don't have qualia, I have jualia, which are isomorphic to qualia in every way except instead of producing subjective experience they produce subjective jexperience.
as the impact of photons from the sun causes a flower to turn, so this article has the effect of causing my metaphorical hands to come together in a sober, appreciative 'golf clap' and my actual fingers to write these words.
You’ve discovered the beginning of the limits of rationality. Where will you go from here?
I think his point is that LLMs are text generators and that concluding that a text -- any text, even his own -- somehow has consciousness is nonsensical. It's only through context that we believe a text is true or false, and even that (with his examples about faking moon landings, or video from Jupiter) is also contingent on the evidence. You kind of have to agree with the premise that LLMs are nothing more than text generators to concede his argument.
I think his greater point is that consciousness is not in the text but in the author, or more specifically the human being. Glibly substituting “Ted Chiang” for LLMs missed the point; they are not the same. There is actually a real person (a human) out there named Ted Chiang who wrote the article (or we assume, but even if he didn't we could still potentially meet Ted Chiang). There’s all kinds of language games you can play with these kind of statements but I think Ted’s point is that consciousness is not easily reducible — in this case, reducible to written text. You need context to make claims about consciousness.
Example: if someone put a gun to your head and said “is the person who wrote this text conscious”. With the introduction of LLMs, you cannot definitively say yes or no (which is a technological sea change in itself, to create that kind of disbelief about text, images, deepfakes in general). There’s not enough context. If someone did the same thing with Ted Chiang, the author standing in front of you, or any other person, at this time could much more confidently say “yes” because we don’t have lifelike androids yet. In the future that could change and we would be in the same epistemological pickle that LLMs have introduced to interpreting text, media, etc.
What evidence do we have for an LLM being a "conscious" author? We know roughly how they work (predictive language-based algorithms) but do we have any evidence, for instance equivalent to the “rigorous” animal studies that Chiang mentioned, that algorithms are conscious? I doubt it, in fact I think there’s mostly the opposite.
There could be a time in the future, long after Ted Chiang is dead, that there is evidence that LLMs, or more likely some other form of AI (perhaps embodied, as Chiang suggests), are conscious. But there’s not much evidence that that is currently the case.
Chiang’s bigger point, at the end of the essay, is that it is wrong to contribute subjectivity to something we have no evidence for being conscious and that doing so, will “result in an atrophy of moral reasoning”.
But that's the whole point of the satire. Nobody is claiming texts are conscious. But Chiang is (at point) concluding from texts not being conscious that text generators aren't conscious, even though that technically also would apply to human authors.
And yes you are right, ultimately we don't have evidence, and too many competing theories of consciousness as well, but it's Chiang who is claiming certainty on the matter.
It is kind of a clunky argument by Chiang but I think his point is that there is nothing “thinking” in an LLM, it’s just a text generator and assigning any intent, meaning, or consciousness to that text is foolish.
The fact is we know that humans and LLMs are not equivalent and that raises moral questions that he spends the majority of the essay discussing.
He’s “certain” because there’s actual moral consequences to claiming something is conscious that Anthropic (with their creed) or chose to ignore. It’s not just a question that can be solved by playing around with syllogisms.
Well we do agree that it's a clunky argument at least :).
Again you say "assigning [...] consciousness to that text", but again, nobody is doing that nor is any argument that llms might be conscious relying on the text output itself being conscious. The question is whether something outputting complex language should be taken as signs of the *underlying thing* possibly being conscious.. and people who take AI conscious seriously usually don't actually think it's that easy either.. but in either case, it doesn't make sense to point at the output and say it isn't conscious, therefore what generated it isn't conscious, any more than saying the sound waves coming out of my mouth aren't conscious, therefore I can't be conscious.
Unless you meant to write "that text generator"? But then also, the question at hand is exactly whether text generators could be conscious, so just saying, well they are text generators! doesn't answer that (I'm not saying they are, I just think this is a poor argument).
For a more detailed response to Chiang, see eg
https://substack.com/@benthamsbulldog/p-200623151
As for Anthropic's actions, I don't actually agree that they are inconsistent with their stated position, because the latter is uncertainty and some precautions, not certainty. But even if they were inconsistent, it doesn't necessarily mean they don't have those beliefs about consciousness, just that they aren't acting fully aligned with their morals (which of course can be criticised, but it's a different point).
After all by that logic we should conclude that most humans don't think that pigs are conscious or can suffer, given how poorly we treat them in factory farms, whereas the reality is more that we are inconsistent...
This is a direct quote from the article I'm satirizing:
> Should you consider the possibility that every time you open a Word document, you are bringing multiple conscious interlocutors into existence, and every time you close one, you snuff their existence out? No. Contemplating that scenario is not a good use of your time. Even if the Microsoft Office team employed a philosopher who said you shouldn’t be so certain, because consciousness is not well understood, that would not be sufficient reason for you to take this idea seriously. We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.
And no, I don’t believe text generators - LLMs - are conscious. They don’t “think” in any meaningful way.
They are a tool and the responsibility for the consequences of using that tool should reside with the human(s) using it.
I meant that assigning any “intent, meaning, or consciousness” to, meaning behind, that text is foolish.
This was brilliant, thank you Jai.
hah!
I mean, yeah, the problem of solipsism is very real. The problem with taking the view that LLMs are conscious is that we don't have an empirical test that can determine whether something is conscious or not. Arguments and guessing are not good enough to establish LLMs are conscious.
It is a difficult question, but I think Chiang's attempt to rule it out as prima facie impossible is itself obviously nonsensical, and that he and many other people who ostensibly endorse similar arguments are more than capable of realizing those deficiencies if they actually think about it.
I feel a little bad picking on this piece in particular because there are stronger arguments against LLM consciousness, and if I wanted to make the strongest case for the possibility LLM subjective experience/consciousness/awareness/moral patienthood I would engage with those arguments instead. My point here was simply "Ted Chiang's arguments don't make sense, and if we were to accept them they could just as easily demonstrate that he himself isn't conscious"
Ah yeah, I didn't have the context that Ted made those arguments, and yes, saying LLM consciousness is impossible is nonsensical as of right now. However, if an empirical test of consciousness is developed, it would be possible to definitively answer the question of machine consciousness.
if a flim was flerp, we could definitely confirm that flerps were flim! however, we live in a fallen world.
I don't understand.
(i was just making a point about how if we don’t know what consciousness actually is we can just replace it with a nonsense word and it will be just as useful, and doing it in a dickish kind of way)
that's because flim has not yet been demonstrated to be flerp
This is brilliant satire, Jai, because it proves that if you map consciousness strictly by information processing, nobody is home — including you.
But I think something is missing. Intelligence is cybernetic — it shows up wherever regulatory systems nest deeply enough, which is why humans and LLMs rhyme. But that doesn't mean awareness is an emergent property of intelligence.
I think it is more parsimonious to view the brain not as a generator of mind, but as a mechanism of spatial localization — a biological headset whose entire evolutionary job is to filter a wider field of awareness and anchor it to a single moving viewpoint in spacetime.
Maybe the hard problem of consciousness is hard because we keep trying to extract the user from the interface?