D e C o T : Debiasing Chain-of-Thought for Knowledge-Intensive Tasks in Large Language Models via Causal Intervention (2024)

Abstract: Large language models (LLMs) often require task-relevant knowledge to augment their internal knowledge through prompts. However, simply injecting external knowledge into prompts does not guarantee that LLMs can identify and use relevant information in the prompts to conduct chain-of-thought reasoning, especially when the LLM’s internal knowledge is derived from biased information on the pretraining data. In this paper, we propose a novel causal view to formally explain the internal knowledge bias of LLMs via a Structural Causal Model (SCM). We review the chain-of-thought (CoT) prompting from a causal perspective and discover that the biased information from pretrained models can impair LLMs’ reasoning abilities. When the CoT reasoning paths are misled by irrelevant information from prompts and are logically incorrect, simply editing factual information is insufficient to reach the correct answer. To estimate the confounding effect on CoT reasoning in LLMs, we use external knowledge as an instrumental variable. We further introduce CoT as a mediator to conduct front-door adjustment and generate logically correct CoTs where the spurious correlation between LLMs’ pretrained knowledge and task queries is reduced. With extensive experiments, we validate that our approach enables more accurate CoT reasoning and enhances LLM generation on knowledge-intensive tasks.

## introduction for knowledge-intensive tasks @xcite @xcite , specific knowledge is required to obtain an accurate response, which can be out of the distribution of llms' internal knowledge @xcite . since frequently fine-tuning llms can be highly expensive and inefficient @xcite , the llm's internal knowledge can also be outdated and cause knowledge bias problems in llms @xcite @xcite . large language models figure 1 : llms' internal knowledge bias can trigger the usage of irrelevant information in prompts, generate incoherent reasoning chains, and impair the model's logical reasoning ability. this example is derived from the experiments by gpt3.5 on hotpotqa where the context is the one provided in the dataset. please note that 'the heavy metal band formed in jakarta is kekal' refers to a heavy metal band that is different from biohazard. however, gpt3.5 incorrectly assumes that 'the heavy metal band' refers to kekal, and provides incorrect information in step 3 of the cot. to efficiently incorporate external knowledge (i.e., context), methods are proposed to retrieve taskrelevant language evidence @xcite @xcite @xcite . additionally, external knowledge bases can also directly augment and edit the knowledge-injected prompts @xcite @xcite . however, simply injecting external knowledge in prompts does not guarantee that llms can identify and use relevant information in the prompts @xcite , especially when the llm learns biased information in pretraining data @xcite . the knowledge bias in llms can further cause knowledge conflict or misunderstanding between exter-nal knowledge and the model's internal knowledge @xcite . in such cases, llms may use irrelevant information from prompts and generate incorrect and unexpected responses @xcite . when the llm relies on chain-of-thought (cot) reasoning for complex tasks, the biased information from pretrained models further impairs llms' reasoning abilities. many works propose to verify and post-edit the generated reasoning paths before prompt again @xcite @xcite to eliminate the factual errors in the generated cot paths. however, logical reasoning errors can not be easily detected or corrected, as the effectiveness of factual verification and postediting reasoning chains can be limited to simply injecting more knowledge. for example in figure 1 , given the query (e.g., ""judgment night" was collaborated by onyx and the heavy metal band formed in which city?") and context which provides external knowledge @xcite , the llm may generate logically incorrect cot (e.g., cot 2), in which the last chain deviates from the reasoning paths (e.g., instead of the origin of "biohazard", some arbitrary band mentioned in the context). such logical incoherence can be caused by the spurious correlation between the query (e.g., the concept "the heavy metal band formed in") and the llm's internal knowledge understanding. thus, the spurious correlation can lead the llm to find some arbitrary evidence in the context regardless of its logical connection to the previous chain, as long as it contains the exact phrase. in such cases, factual verification methods cannot detect logical reasoning errors, and the answer can still be incorrect even with the facts verified as correct. in this work, we propose a novel causal view via a structural causal model (scm) @xcite to formally explain the internal knowledge bias of llms. to measure spurious correlation, we propose to use external knowledge as an instrumental variable @xcite to estimate the average causal effect (ace) of cot reasoning paths in llms through causal intervention. based on the measurement of ace, we can further introduce a cot sampling method to find the best cot as a mediator and conduct frontdoor adjustment @xcite . in this approach, the spurious correlation between llms' internal knowledge and task queries can be reduced, which ensures correct cot reasoning and llm generation. we summarize our contributions as follows: ## related work llms in knowledge-intensive tasks. in knowledge-intensive tasks @xcite @xcite the llm is asked to respond based on the provided context and its intrinsic knowledge. retrievalaugmented prompting methods focus on identifying accurate and comprehensive evidence from support documents @xcite , in-context examples @xcite , knowledge bases @xcite @xcite @xcite , knowledge graphs @xcite @xcite and human feedback @xcite . however, extensive knowledge-injected prompts can introduce irrelevant information to distract llms @xcite and cause llms' unpredictable behaviors @xcite . instead of focusing on how to identify the best knowledge evidence, we investigate how to find logically correct cot reasoning paths. ## a causal view to understand the causal relationships in knowledge-intensive tasks, we introduce a structural causal model (scm) @xcite and identify the internal knowledge understanding of the llms (z) as the confounder. in figure 2a and figure 2b , we formulate two types of conventional knowledge injection methods as two scms respectively. with the scms, we explain the effectiveness of conventional knowledge injection methods and their limitations. we further present the scm of our method, the debiasing chain-of-thought (decot), in figure 2c . the formulation of our method, decot is illustrated in figure 2d and detailed in section 5. a q z e (a) scm for external knowledge e c a q z (b) scm for chainof-thought e c a q z (c) scm for de-cot @xmath0 ## conclusion in this paper, we formally examine the llm's internal knowledge bias and identify it as the confounder by a structural causal model (scm). we revisit the irrelevant contexts issue and further discover the spurious correlation between the llm and task queries, which can further impair the llm's cot reasoning abilities. then, we propose decot, a debiasing chain-of-thought prompting method in knowledge-intensive tasks, which alleviates the spurious correlation and enables the llm to find more accurate and logically sound responses. following the previous evaluation setting @xcite @xcite , and using the same inputs as the baselines, extensive experimental results and case studies validate the effectiveness of our method decot. 14080 since decot is an inference-stage causal intervention method, the improvement on llms' reasoning abilities is attributed to alleviating the bias, but can be limited to the upper bound of the llm's capacity. to alleviate the causal effect of knowledge bias on llms' reasoning abilities, future works can incorporate unbiased causal learning methods in the model pretraining or instruction tuning stage, which may enable more robust cot reasoning. it is also interesting to study the theoretical causal foundation of cot prompting's mediator role in llms, which can be beneficial to better interpretability of black-box llms.