On Measuring Faithfulness or Self-consistency of Natural Language Explanations (2024)

Abstract: Large language models (LLMs) can explain their predictions through post-hoc or Chain-of-Thought (CoT) explanations. But an LLM could make up reasonably sounding explanations that are unfaithful to its underlying reasoning. Recent work has designed tests that aim to judge the faithfulness of post-hoc or CoT explanations. In this work we argue that these faithfulness tests do not measure faithfulness to the models’ inner workings – but rather their self-consistency at output level.Our contributions are three-fold: i) We clarify the status of faithfulness tests in view of model explainability, characterising them as self-consistency tests instead. This assessment we underline by ii) constructing a Comparative Consistency Bank for self-consistency tests that for the first time compares existing tests on a common suite of 11 open LLMs and 5 tasks – including iii) our new self-consistency measure CC-SHAP. CC-SHAP is a fine-grained measure (not a test) of LLM self-consistency. It compares how a model’s input contributes to the predicted answer and to generating the explanation. Our fine-grained CC-SHAP metric allows us iii) to compare LLM behaviour when making predictions and to analyse the effect of other consistency tests at a deeper level, which takes us one step further towards measuring faithfulness by bringing us closer to the internals of the model than strictly surface output-oriented tests.

## introduction large language models (llms) generate answers in various tasks of increasing difficulty, acting as chatbots @xcite , as programming @xcite or scientific writing assistants @xcite . but often enough they behave unintuitively, showing undesirable behaviour: they can endorse a user's misconceptions @xcite , or generate chainof-thought (cot) @xcite explanations that hide their sensitivity to biasing inputs @xcite ; they can be insensitive to label correctness in in-context learning @xcite , and can produce correct predictions with irrelevant or misleading prompts @xcite . especially in cases of unintuitive behaviour, explanations for their way of acting would be helpful. even though llms can provide plausibly sounding explanations for their answers, recent work argues that model generated natural language explanations (nles) are often unfaithful @xcite . obtaining faithful explanations that accurately reflect the reasoning process of a model (jacovi and goldberg, 2020) is important for understanding the reasons behind an llm's answer, and is instrumental for a trustworthy ai. being able to measure nle faithfulness is most critical when models provide answers we are unable to judge -whether it is ai uncovering new scientific facts or chatgpt helping with homework. recent works aim to test the faithfulness of nles that llms produce about their own predictions (cf. §2.2). but the studies are hard to compare, as they use both different models and data (tab. 3). they test for faithfulness by editing model inputs and measuring whether the prediction changes or stays consistent to the original answer. we argue that faithfulness of a nle is more elusive than what existing tests (including ours) can measure, and that what current tests are measuring is self-consistency. we demonstrate this by comparing all tests (including ours) on the same models and data, showing that predictions differ widely. while existing tests compare output changes resulting from input edits on the surface, we propose a measure that does not need input edits and that more closely analyses how model outputs relate to how it processes the input. overall, our paper contributes the following: • we argue ( §3) that current tests that aim to measure nle faithfulness, in reality measure the selfconsistency of model outputs -without giving insight into a model's inner reasoning processes. ## cc-shap: new shap contribution consistency metric as discussed in §2.2, most self-consistency tests have weaknesses: i) they require semantic evaluations to test whether two model-generated explanations are equivalent; ii) their underlying logic can be difficult to adapt to diverse datasets, or iii) they require input edits for which they often rely on trained helper models. due to these weaknesses, rather than relying on self-consistency tests that compare the outputs of models after modifying their inputs, we instead measure self-consistency by analysing how much a model's input contributes to its answer prediction vs. generated explanationsimilar to the rationale of @xcite . notably, we argue that a necessary condition for a generated explanation to be faithful is that the tokens given as input to the model contribute similarly to the model's answer prediction and to the explanation it generates to justify its prediction. on a high level, this method aims to trace what we aim to measure when determining faithfulness: analyse how the model's actions are related to its internal states. so, when a model makes a prediction for an input, we compute how much each input token contributes towards the prediction. also, when the model generates an explanation, we backtrack how much each input token contributes, for each generated token of the explanation. from these separate calculations we compute cc-shap (consistency measure based on shapley values), our new input-level self-consistency metric, by measuring the convergence between the detected input contributions for answer prediction and its explanation -without any need to specially craft input edits. ## discussion and takeaways given that all faithfulness tests are designed very differently and only focus on the self-consistency of outputs ( §3), it is unsurprising that they deliver diverse results across models and datasets. but the tests show some trends: llama2-and mistralchat are more self-consistent than the base models. this adds to the interesting effects of rlhf and instruction tuning (beyond just model performance). prior work on faithfulness tests already showed that llms have inconsistent behaviour, but none could analyse the divergences in a deeper way. our cc-shap metric makes the effect of inputs on model outputs and explanations transparent. we uncovered that strong models, unlike gpt, show significant changes in contributions when generating nles, but not the answer -while other tests (except 'constructing input from explanation') ignore the nle, and only check whether edits are mentioned verbatim or not. our insights, based on ered more and more deeply -relative to the answer. although cc-shap, like prior methods, measures self-consistency -and not faithfulness -, it has, unlike prior tests, the advantages that it does not require input edits, and that it outputs a continuous value per instance -which helps to stabilise results. it combines the input-and output-level to measure how much individual input tokens contribute to model outputs, which is much nearer to the internal workings of a model than recording the softmax output. thus, we argue that our method takes us one step further towards measuring faithfulness -which is important for llms providing plausibly sounding explanations. by adding cc-shap to our new comparative consistency bank, we showed that cc-shap correlates the most with counterfactual editing ( §a.8), and offer deeper insight into the effects of other tests, on input and output contributions for nles vs. answers ( §b.3). the research interest for the topic of llm explanation faithfulness is constantly growing: between the submission for review and publication of this paper, more studies about faithfulness have emerged ( paul ## conclusion in this paper we argue that existing faithfulness tests of post-hoc and cot-driven nles -are not judging faithfulness, as they are not informed by a models' inner workings, but restrict themselves to evaluating a model's self-consistency at the output level. with our unified platform ccb, we evaluate existing self-consistency tests on a common suite of llms and tasks, showing how much their verdicts differ. we proposed a new self-consistency measure cc-shap that works at token-level, but -by recording model contributions -takes a step further towards an interpretable measurement of faithfulness. our analyses show that chat models tend to be more self-consistent than base models, and that model size has no clear effect on self-consistency. importantly, we show that explanations must be analysed in relation to the given answer. we hope that ccb encourages future work to further investigate different types of consistency behaviours of different model types, for specific tasks and sample properties -to eventually better pinpoint elusive indicators of model faithfulness.