IAE val: A Comprehensive Evaluation of Instance Attribution on Natural Language Understanding (2023)

Abstract: Instance attribution (IA) aims to identify the training instances leading to the prediction of a test example, helping researchers understand the dataset better and optimize data processing. While many IA methods have been proposed recently, how to evaluate them still remains open. Previous evaluations of IA only focus on one or two dimensions and are not comprehensive. In this work, we introduce IAEval for IA methods, a systematic and comprehensive evaluation scheme covering four significant requirements: sufficiency, completeness, stability and plausibility. We elaborately design novel metrics to measure these requirements for the first time. Three representative IA methods are evaluated under IAEval on four natural language understanding datasets. Extensive experiments confirmed the effectiveness of IAEval and exhibited its ability to provide comprehensive comparison among IA methods. With IAEval, researchers can choose the most suitable IA methods for applications like model debugging.

## introduction along with the fast development and wide application of deep neural networks, a lot of interpretability methods have been proposed to explain the models' predictions, which help people understand the reason behind the models' success and limitations @xcite @xcite . such interpretability or attribution methods can be primarily divided into two categories: feature attribution (fa) and instance attribution (ia). while fa highlights important input features or tokens that support the prediction of the input, ia identifies the training instances leading to the predictions of test examples. the evidences obtained by attribution methods can not only help people understand the model's predictions, but also help researchers debug or optimize models, such as finding problematic training data and improving the data quality. plenty of ia methods have been proposed recently @xcite @xcite ). for a test example, these methods calculate a influence score for each training data, reflecting its influence on the prediction of the test example as shown in the left part of figure 1 . the top ranked training instances are extracted as evidences. while all these methods are potentially chosen to identify evidences, how to evaluate them still remains open. @xcite proposed a randomized-test and two heuristics to test whether ia methods satisfy minimal requirements for evidences. @xcite proposed to evaluate by detecting poisoned training examples. besides, @xcite evaluated ia methods by comparing the prediction difference after removing evidences and a randomized-test. however, all these existing methods consider solely one or two perspectives of evidences such as faithfulness, and they ignored other significant requirements that ideal evidences were expected to meet. it still lacks a holistic evaluation for ia methods. in order to address the above problems, we propose iaeval, the first systematic and comprehensive evaluation scheme for ia methods. iaeval consists of our elaborately designed metrics quantifying and measuring four important requirements for ia, including sufficiency, completeness, stability and plausibility. although these requirements have been deeply discussed for fa @xcite , we apply them to ia for the first time by refining their definition and designing metrics for measurement thoroughly. we demonstrate the overall framework of iaeval in the middle part of figure 1 . with iaeval, the evaluation of the ia methods can be conducted in a methodical and holistic manner. furthermore, owing to iaeval's diverse evaluation dimensions, in-depth analyses can be conducted on each ia method. through such analyses, it is possible to unveil potential drawbacks of a method that may not be detected solely through evaluation from a single perspective (seeing section 7.1). we evaluate ia methods with iaeval on four natural language understanding datasets in two languages: sst-2 @xcite , mnli @xcite in english and lcqmc @xcite , chnsenticorp @xcite in chinese. we evaluate three representative ia methods: feature similarity (fs), gradient similarity (gs) @xcite and representer point selection (rps) @xcite . the systematic nature of iaeval enables us make a thorough comparison from four perspectives among these ia methods as in the right part of figure 1 . with iaeval, researchers can refer to its evaluation results and choose better ia methods to acquire higher-quality evidences, helping with specific applications. our key contributions are summarized as follows: instance attribution. recent attribution methods can be primarily divided into two categories: feature attribution (fa) and instance attribution (ia). in this work, we focus on ia methods which calculate influence scores of each training data for test examples. influence functions (if) (koh and liang, 2017) approximate the importance of each training data via the products of gradients and the inverse hessian matrix. barshan et al. (2020) proposes relative influence to mitigate the overlapping problem of if. pruthi et al. (2020) measures the importance by tracing the loss change of test example caused by each training data. similaritybased methods directly measure the training data's importance by calculating the similarity of features @xcite or gradients @xcite between training data and test example. representer point selection (rps) @xcite propose to measuring influence score by linearly decomposing the pre-activation prediction of test example with regard to each training instance. with the influences scores, the training data are ranked accordingly and the top ranked instances are identified as evidences. evaluation of attribution methods. with the advancement of attribution methods, evaluation of these methods gradually becomes increasingly significant. for fa methods, many datasets with human-annotated evidences have been published for evaluation @xcite @xcite @xcite . meanwhile, many studies give their views on the properties that an evidence should satisfy, such as compactness and sufficiency @xcite , comprehensiveness @xcite , plausibility and faithfulness @xcite . but for ia methods, there is no manually annotated evaluate datasets, and there are few evaluation metrics. hanawa et al. (2021) proposes a randomized-test and two heuristics to evaluate the ia methods. karthikeyan and søgaard (2021) points out the illness of previous metrics and propose to evaluate by detecting poisoned training examples. pezeshkpour et al. (2021) analyzes the correlation of different ia methods and makes comparison according to the prediction difference by removing the identified evidences and a randomization test. the above evaluations of ia methods consider solely one or two perspectives. a comprehensive evaluation covering multiple perspectives is an indispensable necessity for ia methods. ## preliminaries firstly, we introduce background knowledge that is used in the later sections. since most of the existing ia methods are conducted on the classification tasks, we mainly focus on text classification tasks. given a training dataset @xmath0, y i } @xmath1,2,...,n for a task, where x i is the input sentence or sentence pair and y i is the label, we train a classification model θ. we denote the learned representation of x i as f i and the classifier linear layer as θ. given test examples {x t } @xmath2,2,...,m , ia methods quantify the influence score of each training data x i on the prediction of each testing example x t , denoted as score i(x i , x t ). for each test example x t , the topk training instances with the greatest influence scores are selected as the evidences {x t,j } @xmath3,2,...,k . ## iaeval iaeval covers four significant requirements for ia methods including sufficiency, completeness, stability and plausibility. while these requirements have been discussed for fa @xcite , we conduct the optimization to tailor them for ia for the first time. we refine their definition and design novel metrics for measurement. we describe in detail each requirement as well as the corresponding metrics as follows. ## experiments with our evaluation scheme iaeval, we make a comprehensive comparison among the aforementioned ia methods from four distinct perspectives. we report the performance of these methods with regard to each important requirements in ## analysis in this section, we first provide an explanation for the poor performance and anomaly of rps on lcqmc. to illustrate the robustness of iaeval, we conduct ablation studies with regard to evidence size. furthermore, we validate the effectiveness of iaeval with an empirical analysis. we investigate the reason for the poor performance and abnormally high overlap rate. first let's recall the influence score of rps: apart from constants, it consists of two components. one is the gradient of the loss function with respect to the pre-activation logits of each training data. the other is the feature dotproduct of training instances and test example. we discover that the gradient part plays a dominant role in the influence score as shown in figure 3 . the majority of feature dot-products have the largest scale: 10 2 , and they are correlated with the test example. however, only a few amount of gradients have the largest scale: 10 -6 , and it is noteworthy that the gradients are independent of the test examples. as a result, when we multiply these two factors, those small amount of training data with high gradients will constantly get a high influence score regardless of the test example. this explains why the identified evidences by rps have a significant high overlap rate, as shown in section 6.1. we examined the top 1000 evidences of rps and discovered that data with gradient of scale 10 -6 dominant the evidences with a proportion of 98%. such observation verifies our explanation. furthermore, we posit high gradient may be attributed to annotation issues. we randomly sampled 100 training data with gradient of scale 10 -6 and examined the annotation. we find that 26% of the sampled data are labeled incorrectly, leading to the poor performance of rps. we also examined the annotation of 100 training data randomly sampled from whole training data. the annotation error rate is 5%. such observation inspires us that rps can be used to extract illness data, helping with annotation checking. the overall explanation is derived from a comprehensive assessment of performance across four perspectives. without such a multitude of evaluation angles, explanations for the issue would be partial and the limitations would not be unveiled. ## conclusion we propose iaeval, the first systematic and comprehensive evaluation scheme for ia methods. we design novel metrics to quantify and measure four important requirements for the first time, including sufficiency, completeness, stability and plausibility. extensive experiments validate the effctiveness of iaeval. with iaeval, researchers can make comprehensive comparison among ia methods, gain a thorough understanding of their strengths and weaknesses and choose the most suitable methods for specific applications.