Combining Feature and Instance Attribution to Detect Artifacts (2022)

Abstract: Training the deep neural networks that dominate NLP requires large datasets. These are often collected automatically or via crowdsourcing, and may exhibit systematic biases or annotation artifacts. By the latter we mean spurious correlations between inputs and outputs that do not represent a generally held causal relationship between features and classes; models that exploit such correlations may appear to perform a given task well, but fail on out of sample data. In this paper, we evaluate use of different attribution methods for aiding identification of training data artifacts. We propose new hybrid approaches that combine saliency maps (which highlight important input features) with instance attribution methods (which retrieve training samples influential to a given prediction). We show that this proposed training-feature attribution can be used to efficiently uncover artifacts in training data when a challenging validation set is available. We also carry out a small user study to evaluate whether these methods are useful to NLP researchers in practice, with promising results. We make code for all methods and experiments in this paper available.

## introduction deep networks dominate nlp applications and are being increasingly deployed in the real-world. but what exactly are such models "learning"? one concern is that they may be exploiting artifacts or spurious correlations between inputs and outputs that are present in the training data, but not reflective of the underlying task that the data is intended to represent. we assess the utility of attribution methods for purposes of aiding practitioners in identifying train-warning: this paper contains examples with texts that might be considered offensive. ing data artifacts, drawing inspiration from prior efforts that have suggested the use of attribution methods for this purpose @xcite . attribution methods are model-centric; our evaluation of them for artifact discovery therefore complements recent work on data-centric approaches @xcite . we consider two families of attribution methods: (1) featureattribution, which highlight constituent input features (e.g., tokens) in proportion to their "importance" for an output @xcite @xcite , and; we also introduce new hybrid attribution methods that surface relevant features within train instances as an additional means to probe what the model has distilled from training data. this addresses inherent limitations of using either feature or instance attribution alone for artifact discovery. the former can only highlight patterns within a given input, and the latter requires one to inspect entire (potentially lengthy) training instances to divine what might have rendered them influential. consider figure 1 . here a model has learned to erroneously associate african american vernacular english (aave) with toxicity @xcite and with certain punctuation marks ("!"). for a hypothetical test instance "yo! that's sick", both input saliency and instance attribution methods may provide some indication of these artifacts. but combining these via training-feature attribution (tfa) can directly surface the punctuation artifact by highlighting "!" within a relevant training example ("shut up!"); this is not readily apparent from either input or instance attribution. our goal in this work is to evaluate tfa and other attribution methods as tools for identifying dataset artifacts. contributions. the main contributions of this paper are as follows. (1) we propose a new hybrid attribution approach, training-feature attribution (tfa), which addresses some limitations of existing attribution methods. (2) we evaluate feature, instance and training-feature attribution for artifact detection on several nlp benchmarks with previously reported artifacts to evaluate whether and to what degree methods successfully recover these, and find that tfa can outperform other methods. we also discover and report previously unknown artifacts on a few datasets. finally, (3) we conduct a small user-study to evaluate tfa for aiding artifact discovery in practice, and again find that combining feature and instance attribution is more effective at detecting artifacts than using either on its own. ## background and notation assume a text classification setting where the aim is to fit a classifier φ that maps inputs x @xmath0. denote the training set by @xmath1 here we define a linear classification layer on top of bert @xcite as φ, fine-tuning this on d to minimize cross-entropy loss l. two types of attribution methods have been used in prior work to characterize the predictive behavior of φ. feature attribution methods highlight important features (tokens) in a test sample x t . examples of feature attribution methods include input gradients @xcite , and model-agnostic approaches such as lime @xcite . in this work, we consider only gradient-based feature attribution. instance attribution methods retrieve training samples z i deemed "influential" to the prediction made for a test sample x t : ŷt = φ(x t ). attribution methods assign scores to train instances z i intended to reflect a measure of importance with respect to ŷt : i(ŷ t , z i ). importance can reflect a formal approximation of the change in ŷt when z i is upweighted @xcite or can be derived via heuristic methods @xcite . while prior work has considered these attribution methods for "train set debugging" @xcite , this relies on the practitioner to abstract away potential patterns within the influential instances. ## a procedure for artifact discovery we now propose a procedure (figure 2 ) one might follow to systematically use the above attribution methods to discover training artifacts. we note that in 3-a, we aggregate the individual token rankings over all instances (for both feature attribution and tfa methods), which does not require thresholding attribution scores per instance. we now follow this procedure on widely used nlp benchmarks (section 5), finding that we can "rediscover" known artifacts and identify new ones within these corpora (section 6; models we follow @xcite for instance attribution methods; this entails only considering the last layer of bert in our gradient-based instance attribution methods (see appendix, section a). for all benchmarks, we achieve an accuracy within ∼1% of performance reported in prior works using bert-based models. attribution methods we consider two instance attribution methods, rif @xcite and euclidean similarity (euc), based on results from @xcite . for feature attribution, we consider gradients (g) and integrated gradients (ig; @xcite . to include rif as a tool for artifact detection, we follow the tfa aggregated token approach, but assign uniform importance to all the tokens in a document. in addition to the model-centered diagnostics we have focused on in this work, we also consider a few dataset-centered approaches for artifact discovery: (1) pmi @xcite , and (2) competency score @xcite . there are a few inherent shortcomings to purely datasetcentered approaches. first, because they are modelindependent, they cannot tell us whether a model is actually exploiting a given artifact. second and relatedly, they are based on simple observed correlations between individual features and labels, so cannot reveal abstract artifacts. given the latter point, we only consider these approaches for granular artifact detection (section 6.1). ## case studies we now compare attribution methods in terms of their ability to highlight dataset artifacts. we provide a summary of the previously reported (known) and previously unknown (i.e., discovered in this work) artifacts we identify in this way (and with which methods) in setup we sample train/validation/test sets comprising 5k/2k/100 examples respectively from the imdb corpus, such that all examples in the test set contain a rating (i.e., exhibit the artifact). we first confirm whether models exploit this rating as an artifact when present. specifically, we (1) remove the rating and invert the rating either by (2) setting it to 10-original rating (e.g., 1 → 9), or (3) by setting the rating to 1 for positive reviews, and 10 for negative reviews. this flips the prediction for 9%, 34% and 38% of test examples following these three modifications, respectively. 5 this suggests the model exploits this artifact. ## user study so far we have argued that using feature, instance, and hybrid tfa methods can reveal artifacts via case studies. we now assess whether and which attribution methods are useful to practitioners in identifying artifacts in a simplified setting. we execute a user study using imdb reviews @xcite . we use the same train/validation sets as in section 6.1. we randomly sample another 500 instances as a test set. we simulate artifacts that effectively determine labels in the train set, but which are unreliable indicators in the test set (mimicking problematic training data). we consider three forms of simulated granular artifacts. (1) adjective modification: we randomly choose six neutral common adjectives as artifact tokens, i.e., common adjectives (found in ∼100 reviews) that appear with the same frequency in positive and negative reviews (see appendix, section b for a full list). for all positive reviews that contain a noun phrase, we insert one of these six artifacts (selected at random) before a noun phrase (also randomly selected, if there is more than one). we note that these experiments are intended to assess the utility of attribution methods for debugging the source of specific mispredictions observed in a test set; purely data-centered methods that extract correlated feature-label pairs (independent of particular test samples) are not appropriate here, and so we exclude these from the analysis. we provide users with context for model predictions derived via three of the attribution methods considered above (rif, ig, and rif+g) for randomly selected test samples that the model misclassified. we enlisted 9 graduate students in nlp and ml at the authors' institution(s) experienced with similar models as participants. users were asked to complete three tasks, each consisting of a distinct attribution method and artifact type (adjectives, first names, and pronouns); methods and acc label-acc #calls time (m) rif 3.7 100.0 6.4 8.0 ig 31.6 100.0 22.1 8.2 rif+g 47.0 94.5 28.6 10.1 based on these examples, we ask users to identify: (1) the most probable artifacts, foot_5 and, (2) the label aligned with each artifact. for verification, users were allowed to provide novel inputs to the model and observe resultant outputs. we recorded the number of model calls and the total engagement time to evaluate efficiency (we provide a screenshot of our interface in the appendix, section b). we report the accuracy with which users were able to correctly determine the artifact in ## related work artifact discovery previous studies approach the concerning affairs of artifacts by introducing datasets to facilitate investigating models' reliance on them @xcite , analyzing existing artifacts and their effects on models @xcite , using instance attribution methods to surface artifacts and reduce model bias @xcite , or use artifact detection as a metric to evaluate interpretability methods @xcite . to the best of our knowledge, only one previous work @xcite set out to provide a methodical approach to artifact detection. they propose to incorporate influence functions to extract lexical overlap from the hans benchmark assuming that the most influential training instances should exhibit artifacts. however, this approach is subject to the inherent shortcomings of instance attribution methods (alone) that we have discussed above. this work also assumed that the artifact sought was known a priori. finally, @xcite investigate artifacts philosophically, theoretically analyzing spurious correlations in features. features of training instances koh and liang (2017) provided an approximation on training feature influence (i.e., the effect of perturbing individual training instance features on a prediction), and used this approximation in adversarial attack/defense scenarios. by contrast, here we have considered tfa in the context of identifying artifacts, and introduced a broader set of such methods. ## conclusions artifacts-here operationally defined as spurious correlations in labeled between features and targets that owe to incidental properties of data collectioncan lead to misleadingly "good" performance on benchmark tasks, and to poor model generalization in practice. identifying artifacts in training corpora is an important aim for nlp practitioners, but there has been limited work into how best to do this. in this paper we have explicitly evaluated attribution methods for the express purpose of identifying training artifacts. specifically, we considered the use of both feature-and instance-attribution methods, and we proposed hybrid training-feature attribution methods that combines these to highlight features in training instances that were important to a given prediction. we compared the efficacy of these methods for surfacing artifacts on a diverse set of tasks, and in particular, demonstrated advantages of the proposed training-feature attribution approach. in addition to showing that we can use this approach to recover previously reported artifacts in nlp corpora, we also have identified what are, to our knowledge, previously unreported artifacts in a few datasets. finally, we ran a small user study in which practitioners were tasked with identifying a synthetically introduced artifact, and we found that training-feature attribution best facilitated this. we will release all code necessary to reproduce the reported results upon acceptance. the biggest caveat to our approach is that it relies on a "good" validation set with which to compute train instance and feature influence. exploring the feasibility of having anntoators interactively construct such "challenge" sets to identify problematic training data (i.e., artifacts) may constitute a promising avenue for future work. all code necessary to reproduce the results reported in this paper is available at: https://github.com/pouyapez/ artifact_detection .