Class based Influence Functions for Error Detection (2023)

Abstract: Influence functions (IFs) are a powerful tool for detecting anomalous examples in large scale datasets. However, they are unstable when applied to deep networks. In this paper, we provide an explanation for the instability of IFs and develop a solution to this problem. We show that IFs are unreliable when the two data points belong to two different classes. Our solution leverages class information to improve the stability of IFs.Extensive experiments show that our modification significantly improves the performance and stability of IFs while incurring no additional computational cost.

## introduction deep learning models are data hungry. large models such as transformers @xcite , bert @xcite , and gpt-3 @xcite require millions to billions of training data points. however, data labeling is an expensive, time consuming, and error prone process. popular datasets such as the imagenet @xcite contain a significant amount of errors -data points with incorrect or ambiguous labels @xcite . the need for automatic error detection tools is increasing as the sizes of modern datasets grow. influence function (if) (koh and liang, 2017) and its variants @xcite @xcite are a powerful tool for estimating the influence of a data point on another data point. researchers leveraged this capability of ifs to design or detect adversarial @xcite , poisonous @xcite , and erroneous @xcite examples in large scale datasets. the intuition is that these harmful data points usually have a negative influence on other data points and this influence can be estimated with ifs. @xcite empirically observed that ifs are unstable when they are applied to deep neu- * joint first authors ral networks (dnns). the quality of influence estimation deteriorates as networks become more complex. in this paper, we provide empirical and theoretical explanations for the instability of ifs. we show that ifs scores are very noisy when the two data points belong to two different classes but ifs scores are much more stable when the two data points are in the same class (sec. 3). based on that finding, we propose ifs-class, variants of ifs that use class information to improve the stability while introducing no additional computational cost. ifs-class can replace ifs in anomalous data detection algorithms. in sec. 4, we compare ifs-class and ifs on the error detection problem. experiments on various nlp tasks and datasets confirm the advantages of ifs-class over ifs. ## background and related work we define the notations used in this paper. let @xmath0, y) be a data point, where @xmath1, @xmath2; @xmath3; z -@xmath4; f θ : @xmath5; l z,θ = 1 n n @xmath6, y (i) ) = 1 n n @xmath7; θ) be the empirical risk of f θ measured on z, where ℓ : y × @xmath8; θ = arg min θ l z,θ and θ-@xmath9,θ be the optimal parameters of the model f θ trained on z and z -i . in this paper, f θ is a deep network and θ is found by training f θ with gradient descent on the training set z. ## conclusion in this paper, we study influence functions and identify the source of their instability. we give a theoretical explanation for our observations. we introduce a stable variant of ifs and use that to develop a high performance error detection algorithm. our findings shed light of the development of new influence estimators and on the application of ifs in downstream tasks.