How Interpretable are Reasoning Explanations from Prompting Large Language Models? (2024)

Abstract: Prompt Engineering has garnered significant attention for enhancing the performance of large language models across a multitude of tasks. Techniques such as the Chain-of-Thought not only bolster task performance but also delineate a clear trajectory of reasoning steps, offering a tangible form of explanation for the audience. Prior works on interpretability assess the reasoning chains yielded by Chain-of-Thought solely along a singular axis, namely faithfulness. We present a comprehensive and multifaceted evaluation of interpretability, examining not only faithfulness but also robustness and utility across multiple commonsense reasoning benchmarks. Likewise, our investigation is not confined to a single prompting technique; it expansively covers a multitude of prevalent prompting techniques employed in large language models, thereby ensuring a wide-ranging and exhaustive evaluation. In addition, we introduce a simple interpretability alignment technique, termed Self-Entailment-Alignment Chain-of-thought, that yields more than 70% improvements across multiple dimensions of interpretability. Code is available at https://github.com/SenticNet/CoT_interpretability

## introduction in recent trends, large language models (llm) have shown impressive performance across a diverse array of tasks, primarily through extensive scaling of model size @xcite . techniques such as instruct-tuning @xcite applied across diverse tasks have empowered llms to execute inference on previously unseen tasks. one attributing factor lies with the extensive efforts put into innovating new ways of prompting the llm to better exploit their knowledge base. chainof-thought (cot) @xcite has gathered much attention due to its simple setup which allows the llm to generate not only the task output but also the steps undertaken. in addition to its efficacy in enhancing the model's performance, this prompting method concurrently touches on one of the important aspects of utilizing these models for decision-making: interpretability. the assumption is that the reasoning chain preceding the answer illustrates the model's thought process, enabling the audience to understand how the answer is derived. however, such claims though seemingly plausible should be taken lightly as they may not be faithful to the model's reasoning process @xcite . in this context, plausibility refers to the extent to which an explanation resonates with and is deemed acceptable by a human audience. faithfulness, on the other hand, is characterized by the extent to which the explanation accurately reflects the model's decision-making process. there has been a large number of works that seek to introduce modifications to cot, including self-consistency @xcite , least-to-most @xcite , while others specifically focus on establishing faithful reasoning @xcite . we introduce a simple extension to the list of cot variants, but purely with a focus on enhancing interpretability in the reasoning chain. the approach coined self-entailment-alignment cot (sea-cot) operates similarly to self-consistency, but additionally utilizes a form of consistency between the corresponding reasoning steps and supported context. this action is missing in self-consistency, as the focus is only on the resultant output, potentially leading to unfaithful reasoning which may not support the underlying answer. moreover, we conduct an extensive investigation into the reasoning explanations by evaluating under three pivotal axes of interpretability: faithfulness, robustness, and utility on three commonsense reasoning datasets. these assessments are implemented across multiple prompting techniques including cot and various adaptations of it. ## motivation efforts aimed to enhance faithfulness in nlp take various forms. extractive rationalizing model @xcite , designed to be faithful, generally comprises two separate components: explainer and predictor. this design paradigm conditions the predictor exclusively on text spans extracted by the explainer, positing that the resultant output, ŷ is faithfully aligned with the extracted text, ê. however, prior studies @xcite cautions against such beliefs, identifying limitations in adopting the explain-then-predict approach. the authors mentioned that such an approach restricts the focus of the predictor toward the target identified by the explainer, thereby raising questions about what is being explained. conversely, jacovi et al. @xcite highlight concerns relating to the lack of meaningful insights from multiple text spans. in accordance, we hypothesize that besides the limitation of narrowing the predictors' context, generating the explanation and output using separate modules could compromise the quality of the explanation. we set up a simple study, comparing a modular against a single llm setup on two interpretability traits, faithfulness, and utility, covered in deeper detail in section 5. we adopt the pinto framework @xcite , where the explainer, r θ is a frozen pre-trained llm and the predictor, f ϕ is finetuned on the downstream task, conditioned on both the generated explanation and context, ŷ = f ϕ (x ⊕ ê), where ê = r θ (x), x is the given context and ⊕ is the concatenation process. for the single llm setup, we directly train f ϕ to generate both ê and ŷ jointly. we measure faithfulness by computing the drop in performance when swapping êi with another instance within the same batch, êj̸ =i before deriving ŷ|x; ê. we use leakage-adjusted simulatability (las) @xcite , to measure the utility of the rationale, a higher score would indicate that ê is more useful towards learning ŷ. the details of las are covered in a. @xcite we conduct experiments on two commonsense reasoning datasets: commonsense qa (csqa) @xcite and openbookqa (obqa) @xcite . figure 1 shows that the joint approach scores higher on both accounts of faithfulness and utility. we hypothesize that a single model is in better control of aligning its explanation to the resultant outcome. contrarily, a model relying on explanations synthesized by an external model may instead exhibit a diminished correlation between the interdependent variables, explaining the marginal difference in performance despite being given an unrelated stimulus. notably, this observation resonates well with the recognized capability of recent llms to selfgenerate text serving diverse objectives. in particular, llms pre-trained on a large amount of text can elucidate their reasoning processes, assisted with the appropriate prompting format. this preliminary experiment serves as the main motivation to conduct experiments to scrutinize the quality of explanations produced by a singular llm. ## prompt techniques in this section, we systematically review various ways an llm, f ϕ can be prompted. these methods primarily differ in how the language model is queried to derive the final answer. furthermore, we proposed an approach, sea-cot, aimed at improving the interpretability traits of the reasoning chain to serve as the explanation for the resultant output. a high-level overview is shown in figure 2 . ## proposed approach most adaptations on cot are only aimed at maximizing task performance as covered in section 3. our work is instead focused on enhancing the interpretability of the presented reasoning chain preceding the task output. we adapt from sc-cot, by focusing on the n sequences produced, ranked based on specified objectives. instead of picking explanations based on heuristics such as highest cumulative probability, the reasoning is chosen based on the maximization of two objectives: entailment and overlapping score between the supported context (x ⊕ ŷ) and reasoning ê. we posit that a credible explanation should intrinsically align with the given context it aims to elucidate @xcite ; in this scenario, it encompasses both the question being addressed and the prediction label, measured by the level of entailment. we additionally maximize the overlap between two sets of key tokens 1 , which we show in later experiments to be beneficial towards producing higher quality explanations. this simple approach can be regarded as performing a self-alignment step to pick the most suitable explanation with the n sequences. inspired by works that employ the llm itself to do self-correction, we do the same by asking the llm to rate the entailment level between each own generated reasoning, êi and the joint context, x ⊕ ŷ. we prompt the llm with few-shot examples of natural language inference (nli), x e in figure 16 and determine if the hypothesis entails the premise. the final score to be ranked, s t is a combination of both the probability of entailment, s e , and the iou score, s o . the most interpretable explanation is then chosen via maximizing s t . one caveat is that in the event where |ŷ * | = 1, we fall back to sc-cot. however, this can be avoided by trivially setting the number of sequences, n to be higher than the number of possible options. ## interpretability qualities figure 3: the interpretability qualities measured by different perturbation tests, to achieve the corresponding goals of an explanation. goals referenced from @xcite 1 the two sets of tokens are compared after removing any stopwords to minimize noise within the context interpretability is a multifaceted characteristic with multiple desirable traits concerning various goals of interpretability. inspired by existing work on desirable goals of explainable ai @xcite , we assess three aspects of interpretability: faithfulness, robustness, and utility. we propose these traits as we believe they are directly linked to achieving such goals, illustrated in figure 3 . we discuss the connections in further sections. in accordance, we outline the corresponding evaluations sought out to assess each trait, shown in figure 4 . these evaluations are primarily conditioned on both the context and self-generated reasoning chain. ## conclusion this work studied multiple ways of assessing the interpretability of an explanation. our work is centered on assessing different variants of cot and how we can better determine the suitability of the reasoning by-product as an explanation for the underlying prediction. we also propose a modification to the sc-cot framework called sea-cot, designed specifically to yield explanations that better fulfill the objectives of interpretability. our proposed framework surpasses the robustness, faithfulness, and utility dimensions across multiple reasoning benchmarks. in the future, we plan to extend our work towards instilling interpretability and safety in the training stages @xcite , such as safety alignment in llm.