Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLM s (2024)

Abstract: Evaluating Large Language Models (LLMs) on reasoning benchmarks demonstrates their ability to solve compositional questions. However, little is known of whether these models engage in genuine logical reasoning or simply rely on implicit cues to generate answers. In this paper, we investigate the transitive reasoning capabilities of two distinct LLM architectures, LLaMA 2 and Flan-T5, by manipulating facts within two compositional datasets: QASC and Bamboogle. We controlled for potential cues that might influence the models’ performance, including (a) word/phrase overlaps across sections of test input; (b) models’ inherent knowledge during pre-training or fine-tuning; and (c) Named Entities. Our findings reveal that while both models leverage (a), Flan-T5 shows more resilience to experiments (b and c), having less variance than LLaMA 2. This suggests that models may develop an understanding of transitivity through fine-tuning on knowingly relevant datasets, a hypothesis we leave to future work.

## introduction at a high level, reasoning refers to the process of an agent deriving information about its environment that extends beyond what is directly observable or retrievable from memory. large language models (llms) have shown capabilities of solving complex questions necessitating this very process @xcite . these models can often solve these tasks in few-shot, such as chain-of-thought (cot) reasoning @xcite @xcite , or in a zero-shot manner @xcite . scaling up llms has demonstrated improvements across various multi-step reasoning benchmarks, such as arithmetic, commonsense, and symbolic reasoning (wei et al., 2022b; lewkowycz et al., 2022; wei 1 the code and dataset can be found at our github page. @xcite . nevertheless, the question of what mechanisms underlie reasoning in these models remains an open one @xcite @xcite . perhaps more pressingly, so does the question of whether existing reasoning benchmarks accurately reflect a model's capacity to reason. one key aspect of such capabilities is the model's proficiency in transitive reasoning. this involves the model's ability to integrate and logically deduce conclusions from at least two pertinent facts when addressing a specific question (see figure 1 ). in this paper, we design a set of novel diagnostic experiments using automatic and manual re-annotations of two compositional datasets -qasc @xcite and bamboogle @xcite -to control for the different sources of information the llms, namely llama 2 @xcite and @xcite , might be exploiting in answering compositional questions. specifically, our experiments control for (i) named entities in qa pairs; for example, a model looks for dates in the facts when prompted with "when" in the question, (ii) word/phrase associations or overlaps across sections of the models' input prompt, e.g., removing b, in the reasoning chain of @xmath0, @xmath1, and (iii) the model's exposure to direct answers to the questions during pre-training and/or fine-tuning by using the bamboogle dataset. our initial experiments (section 4) establish that llms perform well with intermediate facts provided (figure 1 ), demonstrating some transitive reasoning capabilities. manipulations such as removing overlapping words between facts and questions or shuffling word order within facts show no significant impact on performance. however, removing answer keywords notably decreases model performance, indicating some reliance on these keywords rather than purely relying on transitive reasoning. experiments controlling for the modthe prompt begins with the instruction "follow the demonstrations below to answer the given question" followed by 3 demonstrations. each demonstration consists of a "context" with a question, optionally a set of multiple-choice (mc) answers for the qasc dataset @xcite , two supporting facts (fact 1, fact 2), and a set of "steps" including a "deduction" and the correct answer. the test query contains only the "context" and the llm needs to generate the "steps". (b) we perform a series of manipulations to either of the facts by shuffling words, removing overlapping keywords, and gibbering named entities to control for different sources of exploitation of cues in the input by the models. ## related work reasoning llms have exhibited certain emergent abilities @xcite that can be triggered by providing a few demonstrations of cot manually @xcite , automatically @xcite , or entirely zero-shot with an instruction, e.g., 'think step-by-step' @xcite , leading to an increase in performance in downstream tasks that require some form of reasoning. infusing code in either the pretraining and/or fine-tuning stages has also been shown to help @xcite @xcite . despite their effectiveness in solving reasoning tasks, models usually fail to explore different deductive paths to reach the final answer @xcite . this can be resolved by oversampling different reasoning paths in generation @xcite . on the models' reasoning analysis, @xcite investigate that cot helps bridge the gap between observations in the pretraining data. @xcite finds that models exhibit better numerical reasoning capabilities when the prompt terms are more commonly encountered in the pretraining data. @xcite and @xcite introduced an iterative prompting method that further improves reasoning beyond cot. finally, although increasing the model size usually helps single-hop qa, it does not affect compositional reasoning much @xcite . reasoning datasets qasc @xcite , one of the datasets we focus on in this work, is an example of compositional deductive reasoning. it contains science-related multiple-choice questions supported by two statements (facts) that need to be composed to deduce the answer. the answers cannot be directly obtained from a single fact. all the facts follow a simple transitivity rule (section 3.1). bamboogle @xcite , the second dataset of interest, also comprises compositional questions that cannot be answered correctly by a popular search engine. crucially, it was released after one of the models we experimented with (flan-t5) was pre-trained. strategyqa @xcite also contains multi-step boolean questions which require deducing from two or more facts to answer. however, we exclude it from our benchmarks as preliminary experiments revealed more than 50% accuracy in answering the questions without the facts, hence making it harder to pinpoint whether the model knows the answers directly, or is required to reason. hotpotqa @xcite ) is a multi-hop qa dataset that comprises questions with supporting passages that need to be ingested in a multistep manner to find the answer. this requires the model to perform both extracting of information and reasoning, again possibly hindering identifying a direct link between the question and the answer purely due to reasoning. finally, gsm8k @xcite , and svamp @xcite are popular mathematical datasets comprising grade school math word problems accompanied by a sequence of deductive steps to solve them. unlike qasc and bamboogle, gsm8k and svamp do not follow the transitive reasoning style that we aim to study in this paper. instead, they target the mathematical reasoning of the models. note that hotpotqa involves finding supporting facts which is not the aim of this paper as we are only interested in the transitive reasoning abilities of the models. in-context learning (icl) plays an important role in the model's reasoning capabilities @xcite . @xcite concludes that specifying both the input distribution and the label space in the input prompt is what matters for icl. @xcite show that the labels provided within icl serve as a reference point for the model during inference. however, @xcite analyse that the correct input-label mappings could have varying impacts depending on the task at hand. @xcite show that model size matters in how llms deal with icl: larger models can overwrite their semantic priors if presented with contradictory examples in the input prompt. @xcite find that training a model on corrupted prompts has similar performance to models trained on informative prompts. diagnosing reasoning via prompting previous works have also manipulated prompts to uncover reasoning abilities. @xcite investigate ablating or substituting the input prompt with wrong values. similarly, @xcite show that incorrect reasoning in the generated cot steps does not significantly impact model performance; the order of the steps though is crucial. we also study the compositional reasoning behaviour of llms in multi-hop questions. we have gone a step further by designing a unique set of experiments aimed at dissecting the model's reliance on linguistic constructs, individual tokens, and their underlying semantics. in contrast to these studies, we are not interested in the effect of icl: the few-shot demonstrations are kept in their original form. instead, our emphasis is on modifying the properties of the test queries used to assess our model, allowing us to evaluate its performance under varied conditions without altering the context provided to it. these experiments are designed to discern whether the models engage in compositional deductive reasoning or whether they identify alternative patterns to formulate answers. ## qasc and transitivity to investigate transitive reasoning (section 3.1) in llms, we designed several experiments to analyse their behaviour. firstly, we explore the performance when provided with factual information and demonstrations of deduction. subsequently, we investigate the extent to which knowledge is inherently present within these models, essentially gauging how many answers are pre-existing due to pretraining. we also aim to examine the significance of deductions within these demonstrations. finally, we inspect the impact of individual facts on the models' ability to deduce the final answer. in all experiments, we used 3-shot icl foot_2 . the prompts comprise three sections, beginning with the instruction "follow the demonstrations below to answer the given question", followed by 3-shot icl demonstrations, and ending with the test query which prompts the model to generate the response. the overall structure of the prompt is depicted in figure 1 . depending on the diagnostic experiments, this prompt is modified accordingly (refer to tables 5 , 6 , and 7 in appendix c). below is the description of prompts for the diagnostic experiments carried out to analyse the models' behaviour dealing with transitivity. full as illustrated in figure 1 , each demonstration contains a "context" that includes the question, and a set of multiple-choice (mc) answers, accompanied by two supporting facts (fact 1, fact 2), and a set of "steps" that crucially comprises a "deduction" before the final answer. the rationale of the full prompt is to encourage the model to deduce from the facts verbatim before reaching the final answer. qa the demonstrations contain only the question, the mc answers, and only the correct answer as part of the "steps". this prompt aims to check the prior knowledge of the model in answering these questions without any extra information. qa (step-by-step) similar to qa, this prompt contains the "think step by step" instruction at the beginning, but similarly does not contain any facts in the "context", or a "deduction" step. inspired by @xcite this diagnostic experiment helps identify whether the model does any internal reasoning without explicitly being shown how to do so, e.g., via a "deduction" step. qaf similar to the full prompt, the model is provided with the facts in the "context" but not the "deduction" step. therefore, it is tasked with predicting the final answer without generating verbatim any form of reasoning from the facts. this prompt highlights the importance of the deduction step in answering the questions. qaf (fact 1/fact 2 only) identical to the qaf prompt, the only difference is that it omits either of the facts. this outlines which fact carries more weight for the model's reasoning to reach the final answer. generally, fact 1 is closer to the question, and fact 2 contains the answer (@xmath2, respectively). ## word/phrase associations and overlaps a prominent pattern observed within the qasc dataset is the overlap of words or phrases between the questions and the corresponding facts, as well as among the facts themselves (e.g., the question "climate is generally described in terms of what?" and the fact "climate is generally described in terms of temperature and moisture"). deliberately omitting connecting words between facts effectively disrupts the basis for transitive reasoning, which is designed to impair the model's performance if it depends on this reasoning process. to understand how models depend on these linking tokens, we designed the following experiments that manipulate the full prompt in systematic ways.: ## discussion our initial experiments on qasc suggest that llms may exhibit a form of reasoning, as shown by strong baseline performances. specifically, the fact that flan-t5 has been explicitly fine-tuned on this dataset, explains its performance without demonstrations, hinting at internal reasoning abilities. however, further experiments reveal that these models primarily rely on spotting answer keywords rather than true reasoning. their correct answers often stem from prior knowledge and the mcq format of qasc. we attempt to overcome some of the previous limitation with the use of bamboogle. when evaluated without supporting facts, the models demonstrate limited prior knowledge, as expected. notably, flan-t5 performs well with only fact 2, indicating dependence on specific cues. like in qasc, shuffling tokens in bamboogle has minimal impact, suggesting that models may exploit named entities as shortcuts. when controlling for this, flan-t5 that has knowingly been fine-tuned on relevant reasoning datasets shows capabilities in transitive reasoning, unlike llama 2-13b, which relies heavily on named entities. finally, these findings highlight that the ability of models to answer correctly with shuffled word orders largely stems from recognising and using named entities, rather than genuine transitive reasoning. ## conclusion and future work in this paper, we set out to better understand the underlying processes of llms' transitive reasoning through a series of experiments involving the re-annotation of available compositional question answering datasets. experiments revealed that: (a) models not fine-tuned on datasets focused on compositional deductive reasoning perform better when demonstrations include example deductions; (b) there is a noticeable dependence on answer keywords within the facts for question answering, suggesting that performance on reasoning benchmarks should not be taken at face value; and (c) while non-fine-tuned models predominantly rely on named entities to answer questions, models finetuned on transitive reasoning tasks demonstrate stronger reasoning capabilities. while we identified potential cues that models might exploit when answering transitive questions, we defer a detailed analysis of how specific datasets and task objectives influence transitive reasoning during pre-training and fine-tuning to future research. given that most models can answer complex questions using few-shot icl, exploring differences between fine-tuning and icl regarding reasoning abilities would be interesting future work. towards understanding chain-of-thought prompting: an empirical study of what matters. in proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 2717-2739, toronto, canada. association for computational linguistics. lean wang, lei li, damai dai, deli chen, hao zhou, fandong meng, jie zhou, and xu sun. 2023b. label words are anchors: an information flow perspective for understanding in-context learning. in proceedings of the 2023 conference on empirical methods in natural language processing, pages 9840-9855, singapore. association for computational linguistics. xuezhi wang, jason wei, dale schuurmans, quoc v le, ed h. chi, sharan narang, aakanksha chowdhery, and denny zhou. 2023c. self-consistency improves chain of thought reasoning in language models. in the eleventh international conference on learning representations. albert webson and ellie pavlick. 2022. do promptbased models really understand the meaning of their prompts? in proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 2300-2344, seattle, united states. association for computational linguistics.