Faithfulness Tests for Natural Language Explanations (2023)

Abstract: Explanations of neural models aim to reveal a model’s decision-making process for its predictions. However, recent work shows that current methods giving explanations such as saliency maps or counterfactuals can be misleading, as they are prone to present reasons that are unfaithful to the model’s inner workings. This work explores the challenging question of evaluating the faithfulness of natural language explanations (NLEs). To this end, we present two tests. First, we propose a counterfactual input editor for inserting reasons that lead to counterfactual predictions but are not reflected by the NLEs. Second, we reconstruct inputs from the reasons stated in the generated NLEs and check how often they lead to the same predictions. Our tests can evaluate emerging NLE models, proving a fundamental tool in the development of faithful NLEs.

## introduction explanations of neural models aim to uncover the reasons behind model predictions in order to provide evidence on whether the model is trustworthy. to this end, explanations have to be faithful, i.e., reflect the decision-making process of the model, otherwise, they could be harmful @xcite . however, recent studies show that explanations can often be unfaithful, covering flaws and biases of the model. @xcite show that certain widely deployed explainability approaches that provide saliency maps (with importance scores for each part of the input, e.g., words or super-pixels) can even be independent of the training data or of the model parameters. others also question the effectiveness and reliability of counterfactuals @xcite , concept activations, and training point ranking explanations @xcite . in this work, we investigate the degree of faithfulness of natural language explanations (nles), which explain model predictions with free text. nles are not constrained to contain only input segments, thus they provide more expressive @xcite and usually more human-readable explanations than, e.g., saliency maps @xcite . evaluating the faithfulness of explanations is very challenging in general, as the ground-truth reasons used by a model for a prediction are usually unknown. evaluating the faithfulness of nles is further complicated, as they often include words not present in the input. thus, existing tests evaluating other types of explanations, e.g., saliency maps, cannot be directly applied to nles. as a stepping stone towards evaluating how faithful nles are, we design two tests. our first test investigates whether nle models are faithful to reasons for counterfactual predictions. we introduce a counterfactual input editor that makes counterfactual interventions resulting in new instances on which the model prediction changes but the nle does not reflect the intervention leading to the change. our second test reconstructs an input from the reasons stated in a generated nle, and checks whether the new input leads to a different prediction. we apply our tests to four nle models over three datasets. we aim for our tests to be an important tool to assess the faithfulness of existing and upcoming nle models. foot_0 ## the faithfulness tests given a dataset @xmath0, e i , y i ), with an input x i , a gold nle e i , and a gold label y @xmath1, where l is the set of all labels for x, a model f is trained to produce an nle and a task prediction for each input: f (x i ) = ( êi , ŷi ). we also refer to êi as f (x i ) ex and to ŷi as f (x i ) p . ## related work tests for saliency maps. the faithfulness and, more generally, the utility of explanations were predominantly explored for saliency maps. comprehensiveness and sufficiency @xcite were proposed for evaluating the faithfulness of existing saliency maps. they measure the decrease in a model's performance when only the most or the least important tokens are removed from the input. @xcite propose another faithfulness metric for saliency maps, roar, obtained by masking allegedly important tokens and then retraining the model. in addition, @xcite and @xcite evaluate saliency maps through adversarial input manipulations presuming that model predictions should be more sensitive to manipulations of the more important input regions as per the saliency map. @xcite 5 task accuracy and nle quality are given in ## summary and outlook in this work, we introduced two tests to evaluate the faithfulness of nle models. we find that all four high-level setups of nle models are prone to generate unfaithful nles, reinforcing the need for proof of faithfulness. our tests can be used to ensure the faithfulness of emerging nle models and inspire the community to design complementary faithfulness tests.