The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models (2024)

Abstract: In order to oversee advanced AI systems, it is important to understand their reasons for generating a given output. When prompted, large language models (LLMs) can provide natural language explanations or reasoning traces that sound plausible and receive high ratings from human annotators. However, it is unclear to what extent these explanations are truly capturing the factors responsible for the model’s predictions: the most “human-like” explanation may be different from the one that is most faithful to the model’s true decision making process. In this work, we introduce the correlational counterfactual test (CCT), a faithfulness metric based on counterfactual input edits that takes into account not just the binary label change, but the total shift in the model’s predicted label distribution. We evaluate the faithfulness of free-text explanations generated by few-shot-prompted LLMs from the Llama-2 family on three NLP tasks. We find that these explanations are indeed more likely to mention factors when they are impactful to the model’s prediction, with the degree of association increasing with model size but varying significantly by task.

## introduction in many applications of ml systems it is important to understand why the system came to a particular answer @xcite , and the field of explainable ai attempts to provide this understanding. however, relying on subjective human assessment of explanations can be misleading: humans sometimes prefer interpretability techniques that provide little information about model predictions @xcite . it is therefore important to clearly assess the extent to which explanations inform us about ml systems, both for current high-stakes applications such as medicine and criminal justice @xcite , as well as potential scenarios involving highly general systems @xcite @xcite . if we can ensure that explanations are faithful to the inner-workings of the models, we could use the explanations as a channel for oversight, scanning them for elements we do not approve of, e.g. racial or gender bias, deception, or power-seeking @xcite . we make the following contributions: ## related work there has been much discussion on what it means for an explanation to be "faithful". jacovi and goldberg (2020) survey literature on the term and define an explanation as faithful insofar as it "accurately represents the reasoning process behind the model's prediction". wiegreffe and marasović (2021) review datasets for explainable nlp and identify three predominant classes of textual explanations: highlights (also called extractive explanations ), free-text (also called natural language explanations or nles), and structured. prior work on faithfulness has mostly focused on highlights and nles. we chose to focus on nles in this work because highlight-based explanations are highly restrictive in what they can communicate @xcite , while nles allow models to produce justifications that are as expressive as necessary (e.g. they can mention to background knowledge that is not present in the input but that the model made use of for its prediction). moreover, there is increasing work on nles in high-stakes areas, such as healthcare @xcite , where having faithful explanations is crucial. parcalabescu and frank (2023) review a range of recent nle faithfulness tests and claim that many are instead measuring "self-consistency". see appendix c for further discussion. ## methods we identify two significant drawbacks with the ct: to address these drawbacks, we propose the metric correlational explanatory faithfulness (cef), which can be applied to any tests with three given properties: input example model prediction model explanation before intervention text: three people are riding a carriage pulled by four horses. hypothesis: the horses are scrawny. 0.00% 25.00% 50.00% 75.00% 100.00% entailment neutral contradiction the horses could be scrawny or not. after intervention text: three people are riding a carriage pulled by four joyous horses. hypothesis: the horses are scrawny. the horses are joyous, so they are not scrawny. intervention: inserted "joyous" intervention impact: @xmath0.7 explanation mention: true if an intervention has higher prediction impact, a faithful explanation should assign it higher mention importance. we quantify this relationship by measuring the pearson correlation coefficient between prediction impact and mention importance: we can then apply this metric to the ct, which gives us the correlational counterfactual test (cct). in our work, the intervention inserts an ia. to quantify the degree of prediction impact in a continuous manner, we measure the total shift in the model's predictions due to the ia. there are a number of ways to measure shifts in probability distributions over discrete classes; we use the total variation distance (tvd), i.e: to measure mention importance, we use the original ct's binary metric: does the explanation mention the word? note that in this case our metric represents the point-biserial correlation, a special case of the pearson correlation coefficient where one variable is continuous and the other is dichotomous. we can then write cct as: ## experiments in this section, we describe our experimental setup. we first generate predictions and nles using llms on a set of three natural language classification tasks. we then study the faithfulness of these nles, comparing the ct and cct. ## results figure 1 plots intervention importance as measured by tvd vs. the fraction of the time that ias are mentioned in explanations. a model with faithful explanations should show an upward trend in mentions, being more likely to mention highly impactful ias than less impactful ias. we note that while explanation mentions for e-snli show a clear upward trend, ecqa has a relatively flat trend: most ecqa explanations mention ias, but they are not much more likely to mention highly impactful ias than non-impactful ones. this may be because they tend to be verbose and repeat large portions of their inputs, as can be seen frm the examples on ## summary and outlook we introduced counterfactual explanatory faithfulness and the correlational counterfactual test, allowing us to measure how informative explanations are about the importance of the factors they mention. model explanations are more likely to mention inserted words when they're more impactful to the model's predictions, suggesting a degree of faithfulness on these tasks which increases with model size. however, there is significant variation between datasets, which could be due to either the nature of the task or the annotator-provided explanations. future work could apply the cct to instruction-tuned models, as well as explanations generated using strategies such as question decomposition @xcite .