Do Zombies Understand? A Choose-Your-Own-Adventure Exploration of Machine Cognition (2024)

Abstract: Recent advances in LLMs have sparked a debate on whether they understand text. In this position paper, we argue that opponents in this debate hold different definitions for understanding, and particularly differ in their view on the role of consciousness. To substantiate this claim, we propose a thought experiment involving an open-source chatbot Z which excels on every possible benchmark, seemingly without subjective experience. We ask whether Z is capable of understanding, and show that different schools of thought within seminal AI research seem to answer this question differently, uncovering their terminological disagreement. Moving forward, we propose two distinct working definitions for understanding which explicitly acknowledge the question of consciousness, and draw connections with a rich literature in philosophy, psychology and neuroscience.

## introduction: a thought experiment large language models (llms) achieve impressive results on various benchmarks, seeming to generalize to unseen tasks and domains @xcite . this initiated a debate on whether llms truly understand @xcite . on the one hand, several works claim that llms are starting to show signs of understanding text @xcite @xcite , while on the other hand, others argue that llms are inherently incapable of understanding because they observe form without meaning @xcite @xcite . evidently, such works have differing opinions of what it means for a model to understand. here, we do not advocate for a single "true" definition for understanding, and instead aim to shed new light on the roots of this debate. we contextualize the debate on machine cognition within the mind-body problem, which has been at the center of vast philosophical debate, as well as intense empirical research in cognitive neuroscience. we follow @xcite , who asks whether the quality of consciousness -the ability to have subjective experiences -is a strict requirement for understanding, or whether it can also manifest in non-conscious agents. we argue that this question lies in the background of all discussion around whether llms truly understand. to make this concrete, consider the following thought experiment: you are presented with z, a new newfangled chatbot. z is implemented in computer hardware and performs only mathematical manipulations of its input. it is completely opensource -you have access to its code, training data, weights, hyperparmeters, and any other implementation detail. you interact with z and discover that it excels on all nlp benchmarks, and will do so on any possible test you will come up with in the future. in essence, z is the chatbot equivalent of the philosophical zombie @xcite ; it outperforms humans on all tasks, supposedly without having subjective experience. do you consider z as capable of "understanding"? if you answer "yes", turn to section 3. for you, the path toward machine cognition lies in test sets of ever-increasing complexity, identifying evermore subtle deficiencies in machine responses. if we reach this road's end, we will find z. if you answer "no", turn to section 4. you hold that consciousness is a prerequisite for understanding, as that is the only thing distinguishing zombies from humans. we make several connections between recent neuroscience research and ai, e.g., the function of consciousness and advancements in the field of neural correlates of consciousness. if you feel uncomfortable with either of these options turn to section 5, where we address potential objections to our setup and assumptions. this setup produces two distinct definitions and research agendas for machine understanding, which are currently conflated in ai discussion. ## background: philosophical zombies the zombie argument is a thought experiment proposed in the context of debates about consciousness and its relationship to the physical world, i.e., to what is measurable @xcite . it seeks to question the validity of physicalism, the belief that all that exists in our world, including consciousness, is physical @xcite . the zombie argument suggests that it is conceptually possible for there to be beings that are physically identical to humans but possess no conscious experiences. these are commonly referred to as "philosophical zombies". philosophical zombies behave just like humans. they appear to feel pain when injured or joy when pleased, and can converse about the events in which they participate. despite these behaviors, philosophical zombies possess no subjective experiences or "qualia" (tye, 2021) -they do not consciously experience sensations, feelings, or thoughts. for instance, a philosophical zombie would react externally like a human would to stepping on a sharp object but would not internally suffer due to the painful sensation. @xcite played a significant role in bringing the argument into the mainstream discourse, particularly in the context of the philosophy of mind. we conjure the equivalent of a zombie chatbot. it is implemented on physical computer hardware, and it is capable of excelling on every nlp task, seemingly without conscious experience. ## zombies do understand: functional definition of understanding one approach to machine cognition relies only on the model's behavior, independent of any internal experience. this definition holds that understanding can be inferred from performance on specific tasks. we formulate this notion for a task t in definition 1: definition 1 functional understanding. a model z functionally understands a task t if its performance on t is as good (or better than) a human who is an expert in t . this approach to understanding is articulated in @xcite 's discussion around intelligence: if a martian could learn to speak a human language, or a robot be devised to behave in just the ways that are essential to a language speaker, an implicit knowledge of the correct theory of meaning for the language could be attributed to the martian or the robot with as much right as to a human speaker, even though their internal mechanisms were entirely different. @xcite this framing helps explain the common practice for testing understanding in models through longstanding challenges, such as chess, go, or language generation, or in many nlp benchmarks, such as text comprehension @xcite @xcite or formal semantic representation @xcite . mccarthy ( 1990 ) figuratively called such tasks the drosophila of ai, drawing a parallel between research in ai and biology, where model organisms (e.g., the drosophila fly) are chosen for wide benchmark experimentation with findings generalizing beyond that specific organism. evidently, the recurring trend in the last 70 years has seen tasks adopted as benchmarks for understanding until automated models functionally understand them. then, the ai community moves the goalposts to another, arguably harder, external objective benchmark for understanding. taken to the extreme, a model that functionally understands every potential benchmark is equivalent to our hypothetical z chatbot. notably, models excelling on these tasks are tested only externally and are not required to have any internal state linked to their success. below, we outline some famous examples of this trend. perhaps the most well-known examples are the games of chess and go. chess served as a proxy task for understanding for nearly 50 years. early works, such as @xcite and @xcite , already deemed chess a benchmark for machine intelligence. with the advent of deep learning models, chess engines now vastly outplay any human opponent @xcite . for all intents and purposes, these models functionally understand chess according to definition 1. consequently, chess was abandoned as a useful benchmark for understanding. foot_0 instead, go was adopted as a marker for understanding @xcite , until go models outplayed the best human players @xcite . a natural follow-up question is whether llms can functionally understand. we argue that similar trends to go and chess happen for certain nlp tasks. for example, natural language inference (nli) has garnered significant attention since its introduction @xcite , and was framed as "fundamental to understanding natural language" by the authors of snli, one of the most prominent benchmarks for the task @xcite . however, as can be seen in figure 1 , the number of models developed over snli has dropped in recent years when performance on the benchmark was saturated, while similar trends are observed also for the follow-up mnli dataset @xcite . to the best of our knowledge, there is no large scale effort to curate a new benchmark for the task. it could be argued that llms functionally understand nli, and the field has implicitly moved to other tasks. an indication that this trend does not stem from loss of interest in the task are various recent works that use nli models as components within larger systems, showing that indeed nli models are useful @xcite @xcite . adopting this notion of understanding implies getting other nlp tasks to go down this path, incrementally achieving functional understanding on as many tasks as possible. at the end of this path, if it is reachable, lies our hypothetical z chatbot, which functionally understands every nlp task. ## zombies don't understand: consciousness is a prerequisite for understanding in contrast to the external approach to understanding in ai, stands a long line of work that either explicitly or implicitly requires models to have subjective experience. these works view the quality of consciousness as an essential aspect of understanding, in addition to accurate performance on any particular task. this notion is formulated with regards to a model m and a task t in the following definition: definition 2 conscious understanding. m consciously understands t if both hold: similar connection between consciousness and understanding is also evident in searle's interpretation for his chinese room argument @xcite . this questions if a computer can be truly intelligent by imagining a non-chinese speaker using a rulebook to manipulate chinese symbols, seemingly displaying comprehension without real understanding. @xcite explicitly states that this argument was meant as a thought experiment for the existence of consciousness, or its lack thereof: i demonstrated years ago with the so-called chinese room argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality. @xcite other notable works have also connected the 2 emphasis is our own in all quotes. turing test and the chinese room argument to consciousness @xcite @xcite . we believe that consciousness also underlies the current discussion on whether llms understand, as evident in @xcite : our human understanding of coherence derives from our ability to recognize interlocutors' beliefs and intentions within context. that is, human language use takes place between individuals who share common ground and are mutually aware of that sharing (and its extent), who have communicative intents which they use language to convey, and who model each others' mental states as they communicate. bender et al. (2021) finally, this definition for understanding is in line with (o @xcite , who in her recent book advocated for consciousness as the defining factor of human intelligence: as ai continues to blow past us in benchmark after benchmark of higher cognition we quell our anxiety by insisting that what distinguishes true consciousness is emotions, perception, the ability to experience and feel. the qualities, in other words, which we share with animals. @xcite to move forward on conscious understanding as articulated in definition 2, we suggest following literature in psychology and neuroscience regarding tests for consciousness (for review, see @xcite ) and specifically the neural-correlateof-consciousness (ncc; @xcite . this field is dedicated to recognizing the neural dynamics in biological organisms associated with consciousness experience. for example, the integrated information theory (iit; @xcite , specifically the weak iit, links elements of consciousness with wider information flow metrics, like recurrent processing or global workspace. these findings can inform cognitively-inspired architectures, e.g., spiking neural networks @xcite . ## other possible answers here we survey alternative answers to the question of whether zombies understand. we reply to these objections below, hopefully resolving seeming inconsistencies within our paradigm. argument: whether z understands depends on its implementation (training data, architecture, hyperparameters, etc.), but it has nothing to do with conscious experience. this argument is in line with @xcite 's definition of psychologism, which assumes that there may exist implementations of z which will show that it indeed understands, e.g., if they involve complex feature manipulations or explicit reasoning steps, while there may exist other implementations which imply that z does not understand, e.g., if all z does is leverage spurious correlations or memorize an immense look up table, similar to the chinese room argument @xcite . we argue that the concerns regarding specific implementations not being indicative of understanding can be mitigated with our requirement that z excels on all possible nlp benchmarks, while also being implemented on a physical hardware. for example, if z leverages spurious correlations, then by definition there are samples which do not exhibit these correlations and which will stump z (otherwise they would not be spurious), contradicting our assumption that z is a philosophical zombie, and does not make non-human errors. similarly, since human language can produce an infinite amount of meaningful texts @xcite , and z can only memorize a finite amount of samples (as it is implemented in finite hardware), then there must be samples outside of its memory on which it is bound to fail. this again contradicts our assumption that z does not fail where humans do not fail. argument: the question is ill-posed as z is inconceivable. hence it is meaningless to discuss different properties of z. this argument may stem from the belief that consciousness has a function in understanding @xcite , and hence it is impossible for an agent to excel on every nlp benchmark without also achieving consciousness. we argue that this position is compatible with the view that consciousness is a prerequisite for understanding ( §4), by positing that is in fact needed to achieve functional understanding. argument: the question is ill-posed as it does not define what is understanding. different definitions may lead to different answers. we do not aim to define apriori what constitutes understanding, and do not argue that there is a single "correct" definition. instead, we try to tease apart what researchers mean when they use the term, specifically highlighting the role that consciousness plays in it, and examine how ai research may be explained through this lens. in fact, we claim that answering the question elucidates different definitions for understanding (definitions 1,2). we invite researchers to engage with this question to examine their definition for understanding. ## discussion we pose the question of whether zombies understand to highlight consciousness's role in ai debates. we propose two definitions for understanding. one deals with functional understanding, and the other revolves around conscious experience. these definitions give rise to different research agendas. this argument can be ported to other discussions about llms possessing human traits. e.g., @xcite recently claimed that llms could not feel empathy. we argue that here, too, consciousness plays a major role in the definition of empathy. similarly, the question of the relevance of consciousness to empathy can be unpacked by asking "can zombies be empathetic?".