Can Explanations Be Useful for Calibrating Black Box Models? (2022)

Abstract: NLP practitioners often want to take existing trained models and apply them to data from new domains. While fine-tuning or few-shot learning can be used to adapt a base model, there is no single recipe for making these techniques work; moreover, one may not have access to the original model weights if it is deployed as a black box. We study how to improve a black box model’s performance on a new domain by leveraging explanations of the model’s behavior. Our approach first extracts a set of features combining human intuition about the task with model attributions generated by black box interpretation techniques, then uses a simple calibrator, in the form of a classifier, to predict whether the base model was correct or not. We experiment with our method on two tasks, extractive question answering and natural language inference, covering adaptation from several pairs of domains with limited target-domain data. The experimental results across all the domain pairs show that explanations are useful for calibrating these models, boosting accuracy when predictions do not have to be returned on every example. We further show that the calibration model transfers to some extent between tasks.

## introduction with recent breakthroughs in pre-training, nlp models are showing increasingly promising performance on real-world tasks, leading to their deployment at scale for translation, sentiment analysis, and question answering. these models are sometimes used as black boxes, especially if they are only available as a service through apis 2 or if end users do not have the resources to fine-tune the example explanation features calibrator prediction is correct / incorrect lower attributions to nnp indicates a prediction is likely to be wrong figure 1 : calibrator pipeline and examples from the squad-adv dataset. a roberta model trained on squad is correct on the first example but incorrect on the second. features that inspect attribution values produced by lime can differentiate these two on the basis of attributions to nnp in the question and v * in the context. a calibrator using these features can predict whether the original model was right or wrong. models themselves. this black-box nature poses a challenge when users try to deploy models on a new domain that diverges from the training domain, usually resulting in performance degradation. we investigate the task of domain adaptation of black box models: given a black box model and a small number of examples from a new domain, how can we improve the model's generalization performance on the new domain? in this setting, note that we are not able to update the model parameters, which makes transfer and few-shot learning techniques inapplicable. however, we can still make the model more effective in practice by learning a calibrator, or a separate model to make a binary decision of whether the black box model is likely to be correct or not on a given instance. while not fully addressing the domain adaptation problem, calibrating the model can make it more useful in practice, as we can recognize when it is likely to make mistakes @xcite @xcite and modify our deployment strategy accordingly. this paper explores how explanations can help address this task. we leverage black box feature attribution techniques @xcite to identify key input features the model is leveraging, even without access to model internal representations. as shown in figure 1 , we perform calibration by connecting model interpretations with hand-crafted heuristics to extract a set of features describing the "reasoning" of the model. for the question answering setting depicted in the figure, answers turn out to be more reliable when the tokens of a particular set of tags (e.g., proper nouns) in the question are strongly considered. we extract a set of features describing the attribution values of different tags. using a small number of examples in the target domain, we can train a simple calibrator for the black box model. our approach is closely related to the recent line of work on model behavior and explanations. @xcite ; @xcite shows explanations can help users predict model decisions in some ways and @xcite show how these explanations can be semi-automatically connected to model behavior based on manually crafted heuristics. our approach goes further by using a model to learn these heuristics, instead of handcrafting them or having a human inspect the explanations. we test whether our method can improve model generalization performance on two tasks: extractive question answering (qa) and natural language inference (nli). we construct generalization settings for 5 pairs of source and target domains across the two tasks. compared to existing baselines @xcite and our own ablations, we find explanations are indeed helpful for this task, successfully improving calibrator performance among all pairs. we even find settings where explanationbased calibrators outperform fine-tuning the model on target domain data, which assumes glass-box access to the model's parameters. our analysis further demonstrates generalization of the calibrator models themselves: our calibrator trained on one domain can transfer to another new domain in some cases. moreover, our calibrator can also substantially improves model performance in the selective qa setting. ## tasks and datasets our task setup involves transferring from a source domain/task a to a target domain/task b. figure 2 shows the data conditions we operate in. our primary experiments focus on using our features to either calibrate or selectively answer in the black box setting (right side in figure 2 ). in this setting, we have a black box model trained on a source domain a and a small amount of data from the target domain b. our task is to train a calibrator using data from domain b to identify instances where the model potentially fails in the large unseen test data in domain b. we contrast this black box setting with glass box settings (left side in figure 2 ), where we directly have access to the model parameters and can fine-tune on domain b or train on b from scratch. ## selective qa setting our results so far have shown that a calibrator can use explanations to help make binary judgments of correctness for a model running in a new domain. we now test our model on the selective qa setting from @xcite (figure 2 ). this experiment allows us to more directly compare with prior work and see performance in a setting where indomain (id) and out-of-domain (ood) examples are mixed together. given a qa model trained on source domain data, the goal of selective qa is to train a calibrator on a mixture of id source data and known ood data, and test the calibrator to work well on a mixture of in-domain and an unknown ood data. we follow the similar experimental setup as in @xcite . the detailed setting is included in the supplementary material. ## related work our approach is inspired by recent work on the simulation test @xcite , i.e., whether humans can simulate a model's prediction on an input example based on the explanations. simulation tests have been carried out in various tasks @xcite @xcite and give positive results in some tasks @xcite . our approach tries to mimic the process that humans would use to judge a model's prediction by combining heuristics with attributions instead of having humans actually do the task. using "meta-features" to judge a model also appears in literature on system combination for tasks like machine translation @xcite , question answering @xcite , constituency parsing @xcite and semantic parsing @xcite . the work of @xcite in vqa is most relevant to ours; they also use heuristic features, but we further conjoin heuristic with model attributions. our meta-feature set is derived from the presence of certain properties, which is similar to the "concepts" used in concept-based explanations @xcite ), but we focus on using them for estimating model performance rather than explaining a prediction. our work addresses the problem of calibration @xcite , which is frequently framed in terms of models' output probabilities. past work has attempted to tackle this problem using temperature scaling @xcite or label smoothing @xcite , which adjust confidence scores for all predictions. in contrast, we approach this issue by applying a classifier leveraging instance-specific explanations. past work on generalizing to out-of-domain distribution in nlp largely focuses on using unlabeled data from the target domain and requires finetuning a model @xcite @xcite , whereas we improve ood performance of strictly black-box models. ## discussion & conclusion limitations despite showing promising results in improving model generalization performance, our attribution-based approach does suffer from intensive computation cost. using either lime or shap to generate attributions requires running inference a fair number of perturbations when the input size is large (see appendix for details), which limits our method's applicability. but this doesn't undermine the main contribution of this paper, answering the question in the title, and our approach is still applicable as-is in the scenarios where we pay for access to the model but not per query.