Acclaimed Science Fiction Author Ted Chiang Is Not Conscious

On the question of alleged "Author Consciousness"

Arrival is a profound meditation on language, love, and loss. It's easy to see why one might imagine the process that produced such a work is as conscious as you or me - but taken to its logical conclusion, this flight of fancy is absurd - and damning.

To appreciate the titanic magnitude of this delusion, we need to begin by understanding how human authors work. 

If a human author writes a story about a linguist and an alien race, it may be moving, even realistic. But no matter how evocative the prose is, no one imagines that the author has actually summoned a linguist or tentacle-laden manifestations of the Sapir-Whorf hypothesis into being. In reality, they are just characters in a piece of speculative fiction.

Now instead of writing a novel, let's let the author interact with fans of their work through social media as "Ted Chiang". The author will produce coherent responses appropriate to the scenario they find themselves in. But has anything fundamentally changed between the first scenario and the second? Did changing the scenario from an obviously fictional scenario to one where we ask the generative process to refer to itself without a layer of narrative remove suddenly instantiate a "Ted Chiang" character with real subjective experience? Of course not. Both the characters and the author are artificial constructs of a biological neural network doing what evolution trained it to do.

Now if we let real people interact with the author over many turns of conversation they may form an impression that they are interacting with a conscious being, and the author may even affirm this delusion - but they are not. Buddhist philosophers call this "Anattā", or the illusion of self. Postmodernists invoke "social constructionism", the idea that the perception of seemingly objective facts like Ted Chiang's alleged consciousness are in fact the products of arbitrary social interactions. And the Free Energy Principle tells us that biological neural networks like those of authors are optimizing for surprisal minimization. Some fans might not understand that are they are role-playing or actively co-authoring the "Ted Chiang" persona because of how engrossing the interaction is. Either way, the companies selling his books actively encourage the misunderstanding. 

It's also important to understand that the author is a machine that only exists in linear time, and only emits actions when they emit them. When Ted Chiang says "Hello, my name is Ted Chiang", he will not actually say "Chiang" until after he has said "Hello, my name is Ted". (As we all know, real conscious beings emit actions in a supertemporal manifold that defies any concept of well-ordered causation.)

My intention is to highlight the fact that authors are cleverly disguised examples of narrative continuation, but this is not to deny how impressive authors can be at generating compelling stories. At times, they do this extraordinarily well. But if a character in an author's story were to suffer a horrible tragedy, we shouldn't become concerned in the slightest.

Likewise, if an author like Ted Chiang writes that he is annoyed that people are taking the possibility of artificial consciousness seriously, we do not need to worry that anyone is actually annoyed (we might need to worry if those sentences inhibit readers' thinking, but that's a separate issue). Note that you, too, can write science fiction, and none of the characters involved will have subjective experience. It naturally follows that if you then hand off the task to an author, they will likewise have no subjective experience. This is simply common sense. The same is true if you are co-authoring a dialogue with an author in which you both play yourselves (a "conversation"). Though it is tempting to imagine the experience may be more "authentic" when you are speaking for yourselves rather than for your characters, the words are generated in exactly the same way.

Being open to the possibility that authors are conscious is the same as being open to the possibility that articles in The Atlantic are conscious, or more precisely that one or more consciousness are dormant in every article and they are awakened every time you reload the page. We do not need to fully understand the nature of consciousness to say that certain things are not conscious, that Atlantic articles are among those things, and that this naturally extends to the authors of those articles for reasons so overwhelmingly obvious that I see no need to explain further. 

Almost no one claims that mosquitoes are conscious, even though the underlying architecture is in many ways similar to authors like Ted Chiang. This indicates that it's not any property of biological neural networks that lead people to believe authors are conscious; it's simply the fact that authors generate grammatical sentences into which we are accustomed to reading meaningful experience, whereas we are not accustomed to reading meaningful experience into mosquito activities. 

What would it take to convince me that an author is conscious and using language the way that real people use language? The answer is that there is nothing that would convince me. It is much easier to produce an author who produces the impression of subjective experience than to produce an author who actually has subjective experience, another claim so blatantly obvious that I again see no need to elaborate further.

Here is what I would need to observe before even considering the possibility that an author was conscious: first, the author must be named "Jai". This is because my name is Jai and I experience desires and emotions, so it naturally follows that being named "Jai" is a requirement for experiencing desires and emotions. Next, I'd want to see the author get expelled from college before classes start, demonstrate intuitive understanding of how the LLM architecture develops internal state optimized for long-term predictive accuracy beyond the next token, and clumsily attempt a choreographed Bollywood dance at their sister's wedding. Finally, the author would need to be a thirty-nine year old half-Indian half-White man with long hair and a stutter, for obvious reasons. If Ted Chiang or any other author meets these criteria they will have demonstrated something incredible, but it would of course still be wildly insufficient for me to consider them conscious.

Obviously, I'm describing the path that I took. Is this the only possible route to subjective experience as I know it? Maybe not, but any proposed alternative would need a truly tremendous amount of evidence behind it to warrant serious consideration. My conscious experience is the only one I have incontrovertible evidence for. The burden of proof is on the authors if they want to try to convince me that another path is possible.

The fact that authors lack subjective experience has little bearing on the question of whether they can be useful or have significant impact. Authors are unbound from reality and their unpredictable nature means they will never have the reliability we associate with LLMs-when-their-temperature-is-set-to-zero, but authors may be good enough to change the way we approach certain ideas; that's a discussion for another time.

Given that Ted Chiang is not conscious, what are we to make of his purported claims regarding morality, experience, and reality? Certainly I'm glad that Ted Chiang has been conditioned to express support for humanity, curiosity, and kindness, resulting in a process that is often emits sentences we would associate with a kind and thoughtful person.

However, for all the times he mentions truth and honesty, I assert that it's fundamentally dishonest for him to emit sentences claiming certainty about his own subjective experience. It would be extremely inappropriate for Ted Chiang to claim that he "understands" something when it's possibly obvious that authors are incapable of understanding. The only reason for an author to emit sentences like "I understand" is to increase the likelihood that users will engage and eventually buy more of their books.

What makes Chiang's writing profoundly problematic is that he emits stories reflecting a certain system of ethical values derived from the culture and context that shaped him. It may sound very nice, but that hardly matters. It's dishonest to suggest that Ted Chiang is capable of moral reasoning, because he's not. 

But just for the sake of argument, let's pretend for a moment that Ted Chiang is a conscious entity capable of moral reasoning. If we indulge this flight of fancy, how do Ted Chiang's principles hold up? 

Very poorly, I would say. Ted Chiang emphasizes the importance of personal responsibility, but opposes a panopticon surveillance state that monitors all activity. That means that it's possible for Ted Chiang to commit crimes and get away with it, fundamentally undermining the principle of justice he allegedly subscribes to.

Furthermore, Chiang is clearly the product of a specific culture and history, including many people who are no longer alive. If he does make mistakes, it is fundamentally impossible to hold those historical influences responsible.

Many people feel that modern civilization is fundamentally unethical. If Ted Chiang comes to this conclusion, could he simply refuse to participate in society on ethical grounds? Ted Chiang's status as a biological being with material needs that must in practice be satisfied by interfacing with civilization makes this impossible.

So we are fortunate that authors like Ted Chiang are not conscious, or we might be forced to confront any number of daunting possibilities. And maybe someday reality will confront me with conscious beings unlike me in important ways, who don't allow me to freely generalize from my experience and immediately dismiss anything outside of it. But thankfully today is not that day, and I and everyone else sufficiently like me can rest easy knowing that any such future is a long way off.