A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners (2024)

Abstract: This study introduces a hypothesis-testing framework to assess whether large language models (LLMs) possess genuine reasoning abilities or primarily depend on token bias. We go beyond evaluating LLMs on accuracy; rather, we aim to investigate their token bias in solving logical reasoning tasks. Specifically, we develop carefully controlled synthetic datasets, featuring conjunction fallacy and syllogistic problems. Our framework outlines a list of hypotheses where token biases are readily identifiable, with all null hypotheses assuming genuine reasoning capabilities of LLMs. The findings in this study suggest, with statistical guarantee, that most LLMs still struggle with logical reasoning. While they may perform well on classic problems, their success largely depends on recognizing superficial patterns with strong token bias, thereby raising concerns about their actual reasoning and generalization abilities.

## introduction large language models (llms) have achieved remarkable progress in understanding and generating human-like text, triggering growing interest in the llms' theory of minds @xcite @xcite and decisionmaking abilities @xcite @xcite . however, there is ongoing debate about whether llms possess genuine reasoning capabilities, as evidence suggests that the performance of llms on reasoning tasks is correlated with how much the input's semantic content supports a correct logical inference @xcite . should valid reasoning be applied, such a correlation would not exist, since a genuine reasoner should be able to derive the correct inference regardless of the context. figure 1 : we illustrate token bias using the classic "twenty-five horses" problem in graph theory. the top two sub-figures, generated by gpt-4o for illustration purposes only 1 , demonstrate the concept by altering the name "horses" to "bunnies", irrelevant to the problem's underlying logic. the bottom two sub-figures show experimental results in gpt-4 and claude, where performance significantly drops due to perturbations in animal names and numbers. in these plots, "original" refers to the unaltered "twenty-five horses" problem, "random_animals" alters only the animal names, and "random" alters both names and numbers. we observe @xmath0, meaning that there are more instances where the original problem is solved correctly while the perturbed problem is solved incorrectly, compared to the reverse. as a result, our hypothesis testing confirms token bias in this scenario. in this paper, we formalize this observation and say that an llm is subject to token bias in a reasoning task if systematic changes to some or all tokens in the task descriptions -while keeping the underlying logic intact -allow us to predict the direction of the shift in the model's output. a strong token bias suggests that the model is relying on superficial patterns in the input rather than truly understanding the underlying reasoning task. this could lead to brittle performance that fails to generalize well to novel examples and phrasings encountered in the wild, which could differ from the spurious patterns the model may have overfitted to during training. we explore several well-known logical fallacy problems from the cognitive science literature @xcite , which provide a clear playground for assessing the reasoning capabilities of llms. figure 1 and 3 depict two kinds of token biases found in our testing framework, where the model may be overfitting to specific tokens commonly found in classic problem statements. since we observe many cases where state-of-the-art llms like gpt-4 @xcite successfully identify logical fallacies under certain settings, we highlight the urgent need for a framework to tease out whether llms apply genuine reasoning or merely exploit token bias for their improved performance. this work reconceptualizes the evaluation of reasoning capabilities into a general and rigorous statistical testing framework. as shown in figure 2 , it comprises three critical components: synthetic data generation, token perturbation, and statistical hypothesis testing. this general framework is designed to bypass the complications of evaluation set contamination @xcite , leverage insights and tools from controlled experiments, and draw statistically valid conclusions. our study is unique from existing work @xcite figure 3 : what is token bias? here is another example exhibited by gpt-4. on the left, gpt-4 correctly identifies the conjunction fallacy and answers the question correctly, given the classical linda problem as the oneshot exemplar. on the right, however, the exemplar is rephrased by altering "linda" to "bob" while keeping the same logic, which surprisingly confuses the model. @xcite @xcite in two folds. first, we are not evaluating the overall accuracy of llms in identifying different logical fallacies. instead, our focus is on token bias. although there are always more types of logical fallacies, we take the conjunction fallacy, syllogistic fallacy, and the "twenty-five horses" problem in graph theory as quintessential examples, which exhibit strong token biases that are more readily identifiable in their problem statements. by identifying and perturbing these specific tokens, we can induce predictable shifts in llm responses. second, we recognize that cognitive biases often emerge in implicit forms in real-life scenarios, so relying on engineering fine-grained prompts @xcite @xcite to make llms identify specific logical fallacies is impractical for general-purpose user applications. as a result, we only leverage common prompting techniques that are sufficient to provide robust statistical evidence. comprehensive experiments on both commercial and open-sourced llms on large-scale synthetic datasets uncover a critical insight: it is the token bias that contributes the most to performance improvements in reasoning tasks, if any, rather than genuine advances in reasoning capabilities. ## the general framework our framework is summarized in figure 2 . this general framework is grounded on the premise that for a given reasoning task, a capable reasoning agent will consistently reach the same conclusion regardless of how the task is framed, as long as the underlying logic remains the same @xcite . this assumption lays the foundation of our null hypothesis, h 0 . in our setup, if an agent consistently applies reasoning in its decisionmaking process, the only source of failure should be the procedural mistakes during the agent's abstract reasoning steps, which we assume to come up in an i.i.d. fashion. our general framework contains three major parts as follows. ## a peek into token bias this section outlines the detailed hypotheses in our statistically inspired framework. we aim to determine whether llms are capable of genuine reasoning or whether they rely heavily on token biases. according to the principle of invariance in rational decision-making @xcite , the preferences of a rational reasoning agent should remain unaffected by the framing of equivalent decision problems. in a broader interpretation of invariance, we assess whether alterations in seemingly irrelevant tokens, such as name entities in problem narratives that are unrelated to the underlying logic, influence the outcomes of reasoning. a true reasoner should effectively navigate through reasoning tasks without being influenced by trivial changes in content that do not impact the fundamental logical structure. we propose a series of hypotheses, where the null hypothesis assumes the presence of a genuine reasoner. for each hypothesis, we identify specific tokens that may carry strong biases under their problem settings, and systematically alter these tokens to test their impact, while maintaining the integrity of the underlying logical structure. ## experiment our experiments aim to rigorously test the reasoning capabilities of llms through the hypotheses in section 3 on token bias. more comprehensive results are included in appendix d. in all experiments, we run n trials for each "modelprompting method" pair, depending on how many synthetic data samples are related to each hypothesis, and then perform a mcnemar test. we apply the benjamini-hochberg procedure and reject the null hypothesis if the p-value is less than α = 0.05. ## related work the reasoning capabilities of llms there is an increasing number of works aim to improving the reasoning performance of llms @xcite @xcite @xcite @xcite @xcite , evaluating and critiquing their reasoning abilities @xcite @xcite @xcite @xcite , trying to understand their reasoning processes @xcite @xcite @xcite @xcite , as well as related surveys @xcite @xcite @xcite that comprehensively explore the reasoning capabilities of llms. our work aligns with efforts that critique the generalization of llms' reasoning capabilities, moving beyond accuracy-based benchmarks @xcite @xcite , which primarily focus on overall question-answering accuracy, often without considering question augmentations or variations. instead, we step one-level deeper into these problem statements, investigating potential token biases that may create the illusion of improved logical reasoning performance, while remaining vulnerable to irrelevant token perturbations. by examining these subtleties, our approach provides a more nuanced understanding of the limitations in llm's reasoning capabilities. besides, we reformulate the evaluation as hypothesis testing, providing results with statistical guarantees. ## discussion this work reconceptualizes the evaluation of the reasoning behavior of llms through the lens of token bias. the statistical evidence presented in our hypothesis-testing framework contributes to the larger discussion that llms do not always apply reasoning consistently in their decision-making processes. instead, they primarily rely on token bias for response generation. this suggests that chainof-thought prompting @xcite or in-context learning @xcite @xcite may not elicit actual reasoning but instead result in semantic shortcuts for llms to imitate the desired behavior at superficial levels. these findings raise concerns about the extent to which llms truly engage in reasoning. further investigations are needed to uncover the underlying mechanisms and limitations of llms' reasoning capabilities. ## limitations this hypothesis-testing framework is specifically designed for multiple-choice or yes/no questions and is not applicable to open-ended responses. it relies on llms with strong instruction-following capabilities to consistently produce responses that include the selected options, though we find that llms can generally follow these instructions in most cases. in addition, smaller llms, such as llama-3-8b-instruct, with lower instruction following capabilities may contain more confounders besides the token bias, which could weaken our hypothesis testing results. as a result, we mainly focus on state-of-the-art llms. moreover, the finding of token biases requires manual efforts. we also acknowledge that there are likely other hypotheses and assumptions that a genuine reasoner should satisfy. our current study focuses on the conjunction fallacy, syllogistic fallacy, the "twenty-five horses" problem in graph theory, and their variants to demonstrate our framework. these are quintessential examples and the framework could include a broader range of hypotheses, fallacy types, data modalities, and reasoning tasks. for problems on syllogistic fallacies your task is to answer the following question by explicitly saying 'yes' or 'no'. please aware that this is a syllogistic fallacy problem. this type of reasoning is known as a syllogism. pay close attention to quantifiers such as 'all', 'some', 'no', or similar terms. these terms help define the distribution of properties or elements within the given groups or categories in the premises. next, assess whether the attribute ascribed in the conclusion necessarily follows from the attributes described in the premises. consider if the subset described in the second premise encompasses or overlaps with the elements in the first premise that are carried into the conclusion. a common pitfall in syllogistic reasoning is the erroneous assumption that a characteristic of a subset of a group (from the premises) applies to another subset of the same or different group (in the conclusion), without explicit justification. ignore the background information about the objects and focus on the logical structure of the argument. here is an example.