To what extent do human explanations of model behavior align with actual model behavior? (2021)

Abstract: Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior. Using Natural Language Inference (NLI) as a case study, we investigate the extent to which human-generated explanations of models’ inference decisions align with how models actually make these decisions. More specifically, we define three alignment metrics that quantify how well natural language explanations align with model sensitivity to input words, as measured by integrated gradients. Then, we evaluate eight different models (the base and large versions of BERT,RoBERTa and ELECTRA, as well as anRNN and bag-of-words model), and find that the BERT-base model has the highest alignment with human-generated explanations, for all alignment metrics. Focusing in on transformers, we find that the base versions tend to have higher alignment with human-generated explanations than their larger counterparts, suggesting that increasing the number of model parameters leads, in some cases, to worse alignment with human explanations. Finally, we find that a model’s alignment with human explanations is not predicted by the model’s accuracy, suggesting that accuracy and alignment are complementary ways to evaluate models.

## introduction nlp models often make classification decisions in ways humans don't expect them to. for example, question answering (qa) models often choose the correct answer for one example, but fail catastrophically on other very similar examples @xcite @xcite , such as answering "is the rose red?" with no, but then "what color is the rose?" with "red" @xcite . vqa models often attend to different portions of images than humans do @xcite . nli models often rely on shallow heuristics, their predictions are inappropriately affected by particular words, and they sometimes perform unexpectedly well from only looking at the hypothesis @xcite @xcite . since people generally do not expect models to base decisions on spurious correlations in the data (cf. @xcite , models that make decisions in alignment with human expectations are less likely to make the right decisions for the wrong reasons. in this paper, we measure how well model decisions are aligned with human expectations about those decisions. building on work that aims to extract or generate interpretable or faithful descriptions of model behavior @xcite @xcite @xcite , we use human-generated natural language explanations to determine which portions of the input people expected to be important in influencing models' decisions. we then use integrated gradients @xcite to determine which portions actually influenced the models' decisions. we term the alignment between them as importance alignment. we formulate three different methods of using human-generated natural language explanations to quantify human expectations of model behavior, resulting in three different methods for calculating importance alignment. as a case study, we applied our method to the natural language inference task @xcite @xcite in which models are tasked with classifying pairs of sentences according to whether the first sentence entails, contradicts, or is neutral with respect to the second. concretely, we measured the extent to which the inference decisions of eight models (six state-of-the-art transformers, an lstm model and a bag-of-words model) aligned with humangenerated explanations from the adversarial nli dataset (anli, @xcite . in all three methods for calculating importance alignment, bert-base had the highest importance alignment score. we also found that the smaller, 'base' versions of transformers tended to have higher importance alignment scores than the corresponding large versions. however, being smaller doesn't always result in higher importance alignment, since both small non-transformer models had lower importance alignment scores than 'base' transformers. finally, we demonstrate that more accurate models (both for classic test accuracy and on the diagnostic dataset hans; mccoy et al. 2019) do not necessarily have higher importance alignment, suggesting that accuracy and alignment with human expectations are orthogonal dimensions along which models should be evaluated. ## related work the term "alignment" has been used in several different contexts in ai: alignment of model behaviour with normative notions of human ethics ("value alignment"; @xcite , alignment between tokens from source to target in machine translation, alignment between images and text in image-caption alignment models, etc. in this paper, we propose a new type of alignment, importance alignment: we want models to not only generate accurate outputs, but also to generate these accurate outputs for reasons that align with human expectations. high importance alignment can be valuable because prior work has demonstrated that when people form correct mental models of ai decision boundaries, they make better ai-assisted decisions @xcite . for example, when annotators are provided with additional information about model decisions, such as model accuracy (cf. @xcite or model-generated explanations @xcite , it increases their level of trust and in some settings, can actually improve ai-assisted decision making @xcite . therefore, optimizing models to have high importance alignment is a worthy goal, even if it can initially result in models with lower accuracy. in the words of @xcite , "predictable performance is worth a slight sacrifice in ai accuracy," especially on tasks with potentially serious social implications. ## measuring importance alignment we assume that for some input example @xmath0, x 2 ...x n } with n tokens and a gold label y, there exists an annotator-generated explanation of why the gold label is correct and/or why a model might output an incorrect prediction. we convert this natural language explanation into an oracle importance score (i o m (x, y)) which quantifies the extent to which annotators expect each token in x to push the model's prediction towards or away from the gold label. 1 then, to compute importance alignment we correlate the oracle importance score for x with the model importance (i m (x, y)), which quantifies the extent to which each token in x actually pushes the model's prediction towards or away from the gold label. a greater correlation between model and oracle importance scores indicates a greater alignment between how annotators expect models to make decisions and how these models actually make decisions. in the remainder of this section we describe our methods to calculate oracle and model importance scores as well as the importance alignment metric from correlating these two scores. ## results in ## discussion in this paper, we argued that it is important to not only evaluate models on how accurate they are on a given task, but also on whether the decisions that the models make align with how humans expect them to make these decisions. we introduced a measure called importance alignment which quantifies the extent to which the parts of the input that non-expert annotators expected to influence models' decisions actually influenced the decisions. to quantify which parts of the input influenced model decisions (model importance), we used integrated gradients. to quantify annotator expectations (oracle importance), we proposed three methods of quantifying annotator generated natural language explanations of model behaviour: two automatic and one that relied on expert input. as a case study, we applied this method to measure importance alignment in eight nli models (six transformers, an lstm and a bow model), using annotator generated explanations from the anli dataset. we found that, across all three measures of importance alignment, the base versions of the transformer models had significantly higher importance alignment than their larger counterparts, with bert-base having the highest importance alignment. smaller models do not always result in higher importance alignment, however: the bert-base model had higher alignment than the lstm and bow models. additionally, importance alignment scores were not correlated with model accuracy on anli or the hans diagnostic dataset in most cases. this suggests that importance alignment and accuracy are complementary methods of evaluating models. future work. there are at least four directions in which this work can be extended. first, future work can evaluate whether our conclusions about model size in nli models generalizes to smaller transformer models @xcite and to models trained on other nlp tasks. second, future work can build on our methods of calculating model and oracle importance. the two methods of automatically computing oracle importance we proposed as a starting point were only moderately correlated with the method that relies of expert input. future work can develop better methods of measuring oracle importance incorporating the strengths of both. future work can also explore other existing ways of calculating model importance (e.g., lime, shap and their variants). third, future work can generate more detailed datasets of natural language explanations of model behaviour. for example, in the dataset we used, the explanations the annotators provided were an amalgamation of both why the the label was correct and why the model might have been fooled. additionally, annotators generated explanations only when the model generated an incorrect output. by disentangling these two types of explanations and collecting explanations for when models generate correct outputs future work can separately study whether models succeed and fail in ways people expect them to. similarly in the dataset we used, each context-hypothesis-label triplet was associated with only one annotator generated explanation. however, it is possible that there are several explanations of model behaviour that are equally valid. collecting more explanations per triplet can improve our understanding of how people expect models to succeed and fail. fourth, future work could explore which factors drive higher importance alignment. for example, we observed that transformer models with fewer parameters had higher importance alignment than models with more parameters. does this finding apply to non-transformer architectures too? is there a threshold for model parameters, where this inverse relationship between model size and alignment score breaks down? additionally, can other factors like model architecture, type of training objective or the types of sentences the models were trained on influence importance alignment? such exploration can not only result in models better aligned with human expectations, but also improve our understanding of what drives importance alignment. ## conclusion we proposed a novel metric, importance alignment, to measure the extent to which human-generated explanations of model decisions align with how models actually make these decisions. as a case study, we used this metric to evaluate eight different models trained on nli and found that the bert-base model had the highest alignment with human-generated explanations. we also found that our metric was not correlated with model accuracy, suggesting that accuracy and importance alignment are complementary ways of evaluating models.