First is Better Than Last for Language Data Inﬂuence
Abstract The ability to identify inﬂuential training examples enables us to debug training data and explain model behavior.
Existing techniques to do so are based on the ﬂow of training data inﬂuence through the model parameters (Koh & Liang, 2017;
Yeh et al., 2018; Pruthi et al., 2020).
For large models in NLP applications, it is often computationally infeasible to study this ﬂow through all model parameters, therefore techniques usually pick the last layer of weights.
However, we observe that since the activation connected to the last layer of weights contains “shared logic”, the data inﬂuenced calculated via the last layer weights prone to a “cancellation effect”, where the data inﬂuence of different examples have large magnitude that contradicts each other.
The cancellation effect lowers the discriminative power of the inﬂuence score, and deleting inﬂuential examples according to this measure often does not change the model’s behavior by much.
To mitigate this, we propose a technique called TracIn-WE that modiﬁes a method called TracIn
(Pruthi et al., 2020) to operate on the word embedding layer instead of the last layer, where the cancellation effect is less severe.
One potential concern is that inﬂuence based on the word embedding layer may not encode sufﬁcient high level information.
However, we ﬁnd that gradients (unlike embeddings) do not suffer from this, possibly because they chain through higher layers.
We show that TracIn-WE signiﬁcantly outperforms other data inﬂuence methods applied on the last layer signiﬁcantly on the case deletion evaluation on three language classiﬁcation tasks for different models.
In addition, TracIn-WE can produce scores not just at the level of the overall training input, but also at the level of words within the training input, a further aid in debugging.
Introduction
Training data inﬂuence methods study the inﬂuence of training examples on a model’s weights (learned during the training process), and in turn on the predictions of other test examples.
They enable us to debug predictions by attributing them to the training examples that most inﬂuence them, debug training data by identifying mislabeled examples, and ﬁxing mispredictions via training data curation.
While the idea of training data inﬂuence originally stems from the study of linear regression (Cook & Weisberg, 1982), it has recently been developed for complex machine learning models like deep networks.
Prominent methods for quantifying training data inﬂuence for deep networks include inﬂuence functions (Koh & Liang, 2017), representer point selection (Yeh et al., 2018), and TracIn (Pruthi et al., 2020).
While the details differ, all methods involves computing the gradients (w.r.t.
the loss) of the model parameters at the training and test examples.
Thus, they all face a common computational challenge of dealing with the large number of parameters in modern deep networks.
In practice, this challenge is circumvented by restricting the study of inﬂuence to only the parameters in the last layer of the network.
While this choice may not be explicitly stated, it is often implicit in the implementations of larger neural networks.
In this work, we revisit the choice of restricting inﬂuence computation to the last layer in the context of large-scale Natural Language Processing (NLP) models.
We ﬁrst introduce the phenomenon of “cancellation effect” of training data inﬂuence, which happens when the sum of the inﬂuence magnitude among different training examples is much larger than the inﬂuence sum.
This effect increases the inﬂuence magnitude of most training examples and reduces the discriminative power of data inﬂuence.
We also observe that different weight parameters may have different level of cancellation effects, and the weight parameters of bias parameters and latter layers may have larger cancellation effects.
To mitigate the “cancellation effect” and ﬁnd a scalable algorithm, we propose to operate data inﬂuence on weight parameters with the least cancellation effect – the ﬁrst layer of weight parameter, which is also known as the word embedding layer.
While word embedding representations might have the issue of not capturing any high-level input semantics, we surprisingly ﬁnd that the gradients of the embedding weights do not suffer from this.
Since the gradient chain through the higher layers, it thus takes the high-level information captured in those layers into account.
As a result, the gradients of the embedding weights of a word depend on both the context and importance of the word in the input.
We develop the idea of word embedding based inﬂuence in the context of TracIn due to its computational and resource efﬁciency over other methods.
Our proposed method, TracIn-WE, can be expressed as the sum of word embedding gradient similarity over overlapping words between the training and test examples.
Requiring overlapping words between the training and test sentences helps capture low-level similarity, while the word gradient similarity helps capture the high-level semantic similarity between the sentences.
A key beneﬁt of TracIn-WE is that it affords a natural word-level decomposition, which is not readily offered by existing methods.
This helps us understand which words in the training example drive its inﬂuence on the test example.
We evaluate TracIn-WE on several NLP classiﬁcation tasks, including toxicity, AGnews, and MNLI language inference with transformer models ﬁne-tuned on the task.
We show that TracIn-WE outperforms existing inﬂuence methods on the case deletion evaluation metric by 4  10 ⇥ .
A potential criticism of TracIn-WE is its reliance on word overlap between the training and test examples, which would prevent it from estimating inﬂuence between examples that relate semantically but not syntactically.
To address this, we show that the presence of common tokens in the input, such as a “start” and “end” token (which are commonly found in modern NLP models), allows TracIn-WE to capture inﬂuence between semantically related examples without any overlapping words, and outperform last layer based inﬂuence methods on a restricted set of training examples that barely overlaps with the test example.
Preliminaries
Consider the standard supervised learning setting, with inputs x 2 X , outputs y 2 Y , and training data D
= { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ... ( x n , y n ) } .
Suppose we train a predictor f with parameter ⇥ by minimizing some given loss function ` over the training data, so that ⇥ = arg min ⇥ P ni
{+=
 ` ( f ( x i ) , y i ) .+} In the context of the trained model f , and the training data D , we are interested in the data importance of a training point x to the testing point x 0 , which we generally denote as I ( x, x 0 ) .
2.1 Existing Methods
We ﬁrst brieﬂy introduce the commonly used training data inﬂuence methods: Inﬂuence functions (Koh & Liang, 2017), Representer Point selection (Yeh et al., 2018), and TracIn
(Pruthi et al., 2020).
We demonstrate that each method can be decomposed into a similarity term S ( x, x 0 ) , which measures the similarity between a training point x and the test point x 0 , and loss saliency terms L ( x ) and L ( x 0 ) , that measures the saliency of the model outputs to the model loss.
The decomposition largely derives from an application of chain rule to the parameter gradients.
[Equation]
The decomposition yields the following interpretation.
A training data x has a larger inﬂuence on a test point x 0 if (a) the training point model outputs have high loss saliency, (b) the training point x and the test point x 0 are similar as construed by the model.
In Section 3.3,we show that restricting the inﬂuence method to operate on the weights in the last layer of the model critically affects the similarity term, and in turn the quality of inﬂuence.
We now introduce the form of each method, and the corresponding similarity and loss saliency terms.
Inﬂuence Functions:
[Equation]
where H ⇥ is the hessian P ni {+=
 r 2 ⇥ `+} ( x, ⇥ ) computed over the training examples.
By an application of the chain rule, we can see that Inf ( x, x 0 ) =
L ( x ) S ( x, x 0 ) L ( x 0 ) , with the similarity term S ( x, x 0 ) = @
f ( x, ⇥ ) @ ⇥
T H  1 ⇥ @
f ( x 0 , ⇥ ) @ ⇥ , and the loss saliency terms L ( x ) = @ ` ( x, ⇥ ) @
f ( x, ⇥ ) .
The work by Sui et al.
(2021) is very similar to extending the inﬂuence function to the last layer to satisfy the representer theorem.
Representer Points:
[Equation]
where a ( x, ⇥ ) is the ﬁnal activation layer for the data point x ,  is the strength of the ` 2 regularizer used to optimize ⇥ , and j is the targeted class to explain.
The similarity term is S ( x, x 0 )
= a ( x, ⇥ ) T a ( x 0 , ⇥ ) , and the loss saliency terms are L ( x )
= 12  n @` ( x, ⇥ ) @
f j ( x, ⇥ ) , L ( x 0 )
= 1 .
TracIn:
[Equation]
where ⇥ c is the weight at checkpoint c , and ⌘ c is the learning rate at checkpoint c .
In the remainder of the work, in our notation, we suppress the sum over checkpoints of TracIn for notational simplicity.
(This is not to undermine the importance of summing over past checkpoints, which is a crucial component in the working on TracIn.)
For TracIn, the similarity term is S ( x, x 0 ) =
r ⇥ f ( x, ⇥ ) T r ⇥ f ( x 0 , ⇥ ) , while the loss terms are L ( x )
= @ ` ( x, ⇥ ) @
f ( x, ⇥ ) , L ( x 0 )
= @
` ( x 0 , ⇥ ) @ f ( x 0 , ⇥ ) .
2.2 Evaluation: Case Deletion
We now discuss our primary evaluation metric, called case deletion diagnostics (Cook & Weisberg, 1982), which involves retraining the model after removing inﬂuential training examples and measuring the impact on the model.
This evaluation metric helps validate the efﬁcacy of any data inﬂuence method in detecting training examples to remove or modify for targeted ﬁxing of misclassiﬁcations, which is the primary application we consider in this work.
This evaluation metric was also noted as a key motivation for inﬂuence functions (Koh & Liang, 2017).
Given a test example x 0 , when we remove training examples with positive inﬂuence on x 0 ( proponents ), we expect the prediction value for the ground-truth class of x 0 to decrease.
On the other hand, when we remove training examples with negative inﬂuence on x 0 ( opponents ), we expect the prediction value for the ground-truth class of x 0 to increase.
An alternative evaluation metric is based on detecting mislabeled examples via self-inﬂuence (i.e. inﬂuence of a training sample on that same sample as a test point).
We prefer the case deletion evaluation metric, as it more directly corresponds to the concept of data inﬂuence.
Similar evaluations that measure the change of predictions of the model after a group of points is removed is seen in previous works.
Han et al.
(2020) measures the test point prediction change after 10% training data with the most and least inﬂuence are removed, and Koh et al.
(2019) measures the correlation of the model loss change after a group of trained data is removed and the sum of inﬂuences of samples in the group, where the group can be seen as manually deﬁned clusters of data.
Deletion curve.
Given a test example x 0 and inﬂuence measure I , we deﬁne the metrics
DEL + ( x 0 , k, I ) and DEL  ( x 0 , k, I ) as the impact on the prediction of x 0 (for its groundtruth class) upon removing top- k proponents and opponents of x 0 respectively:
[Equation]
where, ⇥ +; k ( ⇥  ; k ) are the model weights learned when top- k proponents (opponents) according to inﬂuence measure I are removed from the training set, and c is the groundtruth class of x 0 .
The expectation is over the number of retraining runs.
We expect DEL + to have large negative, and
DEL  to have large positive values.
To evaluate the deletion metric at different values of k , we may plot DEL + ( x 0 , k, I ) and DEL  ( x 0 , k, I ) for different values of k , and report the area under the curve (AUC): AUC - DEL + =
P
[Equation]
We note that the case deletion diagnostics is different to the leave-one-out evaluation of Koh & Liang (2017) by two points.
First, leave-one-out evaluation focuses on removing one point, which is more meaningful in the convex regime where the optimization is initialization-invariant.
We consider the leave-k-out evaluation which is closer to actual applications, as one may need to alter more than one training data to ﬁx a prediction.
Second, we consider the expected value of leave-k-out, to hedge the variance caused by speciﬁc model states, which was pointed out by Søgaard et al.
(2021) to be a major issue for leave-one-out evaluation (especially when the objective is no longer convex).
Cancellation Effect of Data Inﬂuence
The goal of a data inﬂuence method is to distribute the test data loss (prediction) across training examples, which can be seen as an attribution problem where each training example is an agent.
We observe cancellation across the data inﬂuence attributions to training examples, i.e., the sign of attributions across training examples disagree and cancels each other out.
This leads to most training examples having a large attribution magnitude, which reduces the discriminatory power of attribution-based explanations.
Our next observation is that the cancellation effect varies across different weight parameters.
In particular, when a weight parameter is used by most of the training examples, the cancellation effect is especially severe.
One such parameter is the bias, whose cancellation effect is illustrated by the following example:
Example 3.1.
Consider an example where the input x 2 R d is sparse, and x i has feature i with value 1 and all other features with value 0.
The prediction function has the form f ( x ) = x · w + b .
It follows that a set of optimal parameters are w i =
y i , b = 0 .
We further assume that the parameter b is initialized to 0 and has never changed during the gradient descent progress.
In this case, it is clear that the bias parameter b is irrelevant to the model (as removing it will not change the model at all).
However, the inﬂuence to individual examples caused by the bias b is still non-zero.
This is because even that the sum of gradient for bias b is 0 , (s.t. P i @
L ( f ( x i ) ,y i ) @ b = 0 ) , each individual term @
L ( f ( x i ) ,y i )
@ b = @ L ( f ( x i ) ,y i ) @
f ( x i ) is non-zero for most x i .
Note that @ L ( f ( x i ) ,y i ) @ b contributes to the inﬂuence to data x
i directly, and thus the bias parameter b will contribute to the inﬂuence of all the training data.
In the contrary, for each weight variable w j , @ L ( f ( x i ) ,y i ) @
w j is only non-zero for x j , and thus the weight variable w j only contributes to the inﬂuence of one training data x j .
Thus, the bias would affect the inﬂuence for more training examples compared to the weights.
The above example illustrates that while the bias parameter is not important for the prediction model (removing the bias can still lead to the same optimal solution), the total gradient that ﬂows through the bias still high.
In fact, we ﬁnd empirically that the total inﬂuence that ﬂows through the bias is larger than that ﬂowing through the weight, since each training example’s gradient will affect the bias but the total contribution will be cancelled out, so the bias will remain 0 .
We also note that even for deep network models that do not have a sparse input, the neurons connected to the weight are often(due to ReLU types of activation functions).
Thus, the gradient of weight parameters is often sparser compared to the gradient of bias parameters, and thus bias parameters would often have stronger cancellations, which we validate empirically.
3.1 Measuring the Cancellation Effect
In the above example, we deﬁned strong cancellation effect when some weight parameters does not change a lot during training (or has saturated in the training process), but the total strength of the gradient of the weight parameters summed over training data is large.
For weights W , we ﬁrst deﬁne two terms  W c and G ( W ) c ,
[Equation]
where  W c measures the norm of weight parameter change between checkpoint c and c + 1 , and
G ( W ) c measures the sum of weight gradient norm times learning rate summed over all training data.
When  W c is small, this means that the weight W may have saturated at checkpoint c , and the weight may not actually affect the model output much (and thus the weight W is not important for this epoch of training).
When G ( W ) c is large, this means that the sum of gradient norm with respect to W c is still large, and the inﬂuence norm caused by @ l ( x i ,y i ) @
W c will also be large.
To measure the cancellation effect, we deﬁne the cancellation ratio of a weight parameter W as:
[Equation]
When G ( W ) c is large and  W c is small, this means that a non-important weight W c greatly inﬂuenced the total inﬂuence norm, which may be only possible if the inﬂuence contributed from W c to different examples cancelled each other out.
Applying this interpretation to the cancellation of bias parameters, the intuition is that the bias parameters are not mainly responsible for the reduction of testing example loss change during the training process (since  W c is small).
However, they dominate the total inﬂuence strength due to their dense nature ( G ( W ) c is large).
Parameters with high cancellation may not be ideal to the calculation of inﬂuences.
3.2 Removing Bias In TracIn Calculation to Reduce Cancellation Effect
To investigate whether removing weights with high cancellation effect really helps improve inﬂuence quality, we conducted an experiment on a CNN text classiﬁcation on Agnews dataset with 87% test accuracy.
The model is deﬁned as follows: ﬁrst a token embedding with dimension 128 , followed by two convolution layers with kernel size 5 and ﬁlter size 10 , one convolution layers with kernel sizeand ﬁlter size 10 , a global max pooling layer, and a fully connected layer; all weights are randomly initialized.
The ﬁrst layer is the token embedding, the second layer is the convolution layer, and the last layer is a fully connected layer.
The model has 21222 parameters in total (excluding the token embedding), in which 102 parameters are bias variables.
We ﬁnd C ( bias ) to be 16789 , and C ( weight ) to be 2555 , which validates that the bias variables have a much stronger cancellation effect than the weight variables.
A closer analysis shows that G ( bias ) is similar to G ( weight ) ( 627206 and 559142 ), but  ( bias ) is much smaller than  ( weight ) ( 0 .
74 and 4 . 37 .)
Even though the bias parameters has a much smaller total change compared to the weight parameters, their impact on the gradient norm (and thus inﬂuence norm) is even higher than the weight parameters.
This veriﬁes the intuition in Example 3.1 that the bias parameter has a stronger cancellation effect since the gradient to bias is almost activated for all examples despite the actual bias change being small.
To further verify that the TracIn score contributed by the bias may lower the overall discriminatory power, we compute AUC - DEL + and AUC - DEL  for TracIn and TracIn-weight on AGnews with our CNN model.
The AUC - DEL + for TracIn and TracIn-weight is  0 .
036 and  0 .
065 respectively, and the AUC - DEL + for TracIn and
TracIn-weight is 0 .
011 and 0 . 046 .
The result shows that by removing the TracIn score contributed by the bias (with only 102 parameters), the overall inﬂuence quality improves signiﬁcantly.
Thus, in all future experiments, we remove the bias in calculation of data inﬂuence if not stated otherwise.
3.3 Inﬂuence of Latter Layers May Suffer from Cancellation
As mentioned in Section 1, for scalability reasons, most inﬂuence methods choose to operate only on the parameters of the last fully-connected layer ⇥ last .
We argue that this is not a great choice, as the inﬂuence scores that stems from the last fully-connected weight layer may suffer from cancellation effect, as different examples “share logics” in the activation representation of this layer, and have a higher gradient similarity for different examples.
Early layers, where examples have unique logic, may suffer less from the cancellation effect.
We ﬁrst measure the gradient similarity for different examples for each layer, which is E x a ,x b COS-SIM [ @ l ( x a ) / @ w, @ l ( x b ) / @ w ] , where COS-SIM is the cosine similarity.
This measures the expected gradient cosine similarity between two examples.
The expected gradient similarity for testing examples between different layers in the CNN classiﬁcation are: ﬁrst 0 .
035 , second 0 .
075 , third 0 .
21 , last 0 .
This veriﬁes that the latter layers in the neural network have more aligned gradients between examples, and thus share more logics between training examples.
We report the cancellation ratio for each of the TracIn layer varaint in Table 1, where
TracIn-ﬁrst, TracIn-second, TracIn-third, TracIn-last, TracIn-All refer to TracIn scores based on weights of the ﬁrst layer, second layer, third layer, last layer, and all layers (the bias is always omitted).
As we suspected, early layers suffers less from cancellation, and latter layers suffers more from cancellation.
To assess the impact on inﬂuence quality, we evaluate the AUC - DEL + and
AUC - DEL  score for TracIn calculated with different layers on the AGnews CNN model in Tab. 1.
We observe that removing examples based on inﬂuence scores calculated using parameters of later layers (with more “shared logic"") leads to worse deletion score compared to removing examples based on inﬂuence scores calculated using parameters of earlier layers (with more “unique logic”).
Interestingly, the performance of TracIn-ﬁrst even outperforms TracIn-all where all parameters are used.
3 We hypothesize that since the TracIn score based on later layers contain too much cancellation,
[Table] [Table]
it is actually harmful to include these weight parameters in the TracIn calculation.
In the following, we develop data inﬂuence methods by only using the ﬁrst layer of the model, which suffers the least from cancellation effect.
Word Embedding Based Inﬂuence
In the previous section
[Table]
we argue that using the latter layers to calculate inﬂuence may lead to the cancellation effect, which over-estimates inﬂuence.
Another option is to calculate inﬂuence on all weight parameters, but may be computational infeasible when larger models with several millions of parameters are used.
To remedy this, we propose operating on the ﬁrst layer of the model, which contains the less cancellation effect since early layers encodes “unique logit”.
The ﬁrst layer for language classiﬁcation models is usually the word embedding layer in the case of NLP models.
However, there are two questions in using the ﬁrst layer to calculate data inﬂuence: 1.
the word (token) embedding contains most of the weight parameters, and may be computational expensive 2.
the word embedding layer may not capture inﬂuential examples through high-level information.
In the rest of this section, we develop the idea of word embedding layer based training-data inﬂuence in the context of TracIn.
We focus on TracIn due to challenges in applying the other methods to the word embedding layer: inﬂuence functions on the word embedding layer are computationally infeasible due to the large size (vocab size ⇥ embedding_dimension) of the embedding layer, and representer is designed to only use the ﬁnal layer.
We show that our proposed inﬂuence score is scalable thanks to the sparse nature of word embedding gradients, and contains both low-level and high-level information since the gradient to the word embedding layer can capture both high-level and low-level information about the input sentence.
4.1 TracIn on Word Embedding Layer
We now apply TracIn on the word embedding weights, obtaining the following expression:
[Equation]
Implementing the above form of TracIn-WE would be computationally infeasible as word embedding layers are typically very large (vocab size ⇥ embedding dimension).
For instance, a BERT-base model has 23 M parameters in the word embedding layer.
To circumvent this, we leverage the sparsity of word embedding gradients @
` ( x, ⇥ ) @ ⇥ WE , which is a sparse vector where only embedding weights associated with words that occur in x have non-zero value.
Thus, the dot product between two word embedding gradients has non-zero values only for words w that occur in both x, x 0 .
With this observation, we can rewrite TracIn-WE as:
[Equation]
where ⇥ w are the weights of the word embedding for word w .
We call the term @` ( x ) @ ⇥ w T · @` ( x 0 ) @ ⇥
w the word gradient similarity between sentences x, x 0 over word w .
[Table]
4.2 Interpreting Word Gradient Similarity
Equation 4 gives the impression that TracIn-WE merely considers a bag-of-words style similarity between the two sentences, and does not take the semantics of the sentences into account.
This is surprisingly not true!
Notice that for overlapping words, TracIn-WE considers the similarity between gradients of word embeddings.
Since gradients are back-propagated through all the intermediate layers in the model
[Table]
they take into account the semantics encoded in the various layers.
This is aligned with the use of word gradient norm k @ f
( x ) @ ⇥ w k as a measure of importance of the word w to the prediction f ( x ) (Wallace et al., 2019; Simonyan et al., 2013).
Thus, word gradient similarity would be larger for words that are deemed important to the predictions of the training and test points.
Word gradient similarity is not solely driven by the importance of the word.
Surprisingly, we ﬁnd that word gradient similarity is also larger for overlapping words that appear in similar contexts in the training and test sentences.
We illustrate this via an example.
Table 2 shows 4 synthetic premise-hypothesis pairs for the Multi-Genre Natural Language Inference (MNLI) task (Williams et al., 2018).
An existing pretrained model (He et al., 2020) predicts these examples correctly with softmax probability between 0 . 65 and 0 .
Notice that all examples contain the word ‘not’ once.
The word gradient importance k @ f ( x ) @ ⇥ w k for “not” is comparable in all 4 sentences.
The value of word gradient similarity for ‘not’ is 0 .
34 for the pair S1-S2, and  0 .
12 for S1-S3, while it is  0 .
for
S1-S4.
This large difference stems from the context in which ‘not’ appears.
The absolute similarity value is larger for S1-S2 and S1-S3, since ‘not’ appears in a negation context in these examples.
(The word gradient similarity of S1-S3 is negative since they have different labels.)
However, in S4, ‘not’ appears in the phrase “not only ... but”, which is not a negation (or can be considered as double negation).
Consequently, word gradient similarity for ‘not’ is small between S1 and S4.
In summary, we expect the absolute value of TracIn-WE score to be large for training and test sentences that have overlapping important words in similar (or strongly opposite) contexts.
On the other hand, overlap of unimportant words like stop words would not affect the TracIn-WE score.
4.3 Word-Level Decomposition for TracIn-WE
An attractive property of TracIn-WE is that it decomposes into word-level contributions for both the testing point x 0 and the training point x .
As shown in (4), word w in x contributes to TracIn-WE ( x, x 0 ) by the amount @` ( x ) @ ⇥ w T · @` ( x 0 ) @ ⇥
w [ w 2 x 0 ] ; a similar word-level decomposition can be obtained for x 0 .
Such a decomposition helps us identify which words in the training point ( x ) drive its inﬂuence towards the test point ( x 0 ).
For instance, consider the example in Table.
3, which contains two test sentences (T1, T2) and a training sentence S1.
We decompose the score TracIn-WE(S1, T1) and
TracIn-WE(S1,T2) into words contributions, and we see that the word “lazy” dominates TracInWE(S1, T1), and the word “end” dominates TracIn
WE(S1, T2).
This example shows that different key words in a training sentence may drive inﬂuence towards different test points.
The featuredecomposition for inﬂuence introduces additional interpretability to why two examples are highly inﬂuenced.
This is demonstrated in a case study where we cluster difﬁcult training examples based on a normalized TracIn-WE score in Sec.
A.
4.4 An approximation for TracIn-WE
As we note in Sec. 4.1, the space complexity of saving training and test point gradients scales with the number of words in the sentence.
This may be intractable for tasks with very long sentences.
We alleviate this by leveraging the fact that the word embedding gradient for a word w is the sum of input word gradients from each position where w is present.
Given this decomposition, we can approximate the word embedding gradients by saving only the top-k largest input word gradients for each sentence.
(An alternative is to save the input word gradients that are above a certain threshold.)
Formally, we deﬁne the approximation @ ` ( x, ⇥ )
[Equation]
where x i is the word at position i , and x top  k is the set of top-k input positions by gradient norm.
We then propose
[Equation]
Computational complexity Let L be the max length of each sentence, d be the word embedding dimension, and o be the average overlap between two sentences.
If the training and test point gradients are precomputed and saved then the average computation complexity for calculating
TracIn-WE for m training points and n testing points is O ( mnod ) .
This can be contrasted with the average computation complexity for inﬂuence functions on the word embedding layer, which takes O ( mnd 2 v 2 + d 3 v 3 ) , where v is the vocabulary size which is typically larger than 10 4 , and o is typically less than 5 .
The approximation for TracIn-We-Topk drops the computational complexity from O ( mnod ) to O ( mno k d ) where o k is the average overlap between the sets of top-k words from the two sentences.
It has the additional beneﬁt of preventing unimportant words (ones with small gradient) from dominating the word similarity by multiple occurrences, as such words may get pruned.
In all our experiments, we set k to 10 for consistency, and do not tune this hyper-parameter.
4.5 Inﬂuence without Word-Overlap
One potential criticism of TracIn-WE is that it may not capture any inﬂuence when there are no overlapping words between x and x 0 .
To address this, we note that modern NLP models often include a “start” and “end” token in all inputs.
We posit that gradients of the embedding weights of these tokens take into account the semantics of the input (as represented in the higher layers), and enable
TracIn-WE to capture inﬂuence between examples that are semantically related but do not have any overlapping words.
We illustrate this in S5-S7 in Tab. 2 via examples for the MNLI task.
Sentence
S5 has no overlapping words with S6 and S7.
However, the word gradient similarity of “start” and “end” tokens for the pair S5-S6 is 1 . 15 , while that for the pair S5-S7 is much lower at  0 . 05 .
Indeed, sentence S5 is more similar to S6 than S7 due to the presence of similar word pairs (e.g., think and thinks, annoying and boring), and the same negation usage.
We further validate that TracIn-WE can capture inﬂuence from examples without word overlap via a controlled experiment in Sec. 5.
Experiments
We evaluate the proposed inﬂuence methods on 3 different NLP classiﬁcation datasets with BERT models.
We choose a transformer-based model as it has shown great success on a series of downstream tasks in NLP, and we choose BERT model as it is one of the most commonly used transformer model.
For the smaller Toxicity and AGnews dataset, we operate on the Bert-Small model, as it already achieves good performance.
For the larger MNLI dataset, we choose the Bert-Base model with 110 M model parameters, which is a decently large model which we believe could represent the effectiveness of our proposed method on large-scale language models.
As discussed in Section 2.2, we use the case deletion evaluation and report the metrics on the deletion curve in Table 4 for various methods and datasets.
The standard deviation for all AUC values all methods is reported in Table 7.
[Table]
One question to ask is whether the good performance of TracIn-WE is a result that it captures the low-level word information well.
To answer this question, we design a synthetic data inﬂuence score as the TF-IDF similarity Salton & Buckley (1988) multiplied by the loss gradient dot product for x and x 0 .
TR-TFIDF can be understood by replacing the embedding similarity of
TracIn-
Last by the TF-IDF similarity, which captures low level similarity.
[Equation]
Toxicity.
We ﬁrst experiment on the toxicity comment classiﬁcation dataset (Kaggle.com, 2018), which contains sentences that are labeled toxic or non-toxic.
We randomly choose 50 , 000 training samples and 20 , 000 validation samples.
We then ﬁne-tune a BERT-small model on our training set, which leads to 96% accuracy.
Out of the 20 , 000 validation samples, we randomly choose 20 toxic and 20 non-toxic samples, for a total of 40 samples as our targeted test set.
For each example x 0 in the test set, we remove top- k proponents and top- k opponents in the training set respectively, and retrain the model to obtain DEL + ( x 0 , k, I ) and DEL  ( x 0 , k, I ) for each inﬂuence method I .
We vary k over { 10 , 20 , . . .
For each k , we retrain the model 10 times and take the average result, and then average over the 40 test points.
We implement the methods Inﬂuence-last, Representer Points,
TracIn-last, TracIn-WE, TracIn-WE-Topk, TracIn-TFIDF (introduced in Sec. G), TracIn-common (which is a variant of TracIn only using the start token and end token to calculate gradient), and abbreviate TracIn with TR in the experiments.
We see that our proposed TracIn-WE method, along with its variants TracIn-WE-Topk outperform other methods by a signiﬁcant margin.
As mentioned in Sec.
3.3, TF-IDF based method beats the existing data inﬂuence methods using last layer weights by a decisive margin as well, but is still much worse compared to TracIn-WE.
Therefore, TracIn-WE did not succeed by solely using low-level information.
Also, we ﬁnd that TracIn-WE performs much better than TracIn-common, which uses the start and end tokens only.
This shows that the keyword overlaps (such as lazy, end in Table 3) is crucial to the great performance of TracIn-WE.
AGnews.
We next experiment on the AG-news-subset (Gulli, 2015; Zhang et al., 2015), which contains a corpus of news with 4 different classes.
We follow our setting in toxicity and choose 50 , 000 training samples, 20 , 000 validation samples, and ﬁne-tune with the same BERT-small model that achieves 90% accuracy on this dataset.
We randomly choose 100 samples with 25 from each class as our targeted test set.
The AUC - DEL + and AUC - DEL  scores for k 2 { 10 , 20 , . .
, 100 } are reported in Table 4.
Again, we see that the variants of TracIn-WE signiﬁcantly outperform other existing methods applied on the last layer.
In both AGnews and Toxicity, removing 10 top-proponents or top-opponents for TracIn-WE has more impact on the test point compared to removingtop-proponents or top-opponents for TracIn-last.
MNLI.
Finally, we test on a larger scale dataset, Multi-Genre Natural Language Inference (MultiNLI)
Williams et al.
(2018), which consists of 433 k sentence pairs with textual entailment information, including entailment, neutral, and contradiction.
In this experiment, we use the full training and validation set, and BERT-base which achieves 84% accuracy on matched-MNLI validation set.
We choose 30 random samples with 10 from each class as our targeted test set.
We only evaluate TracIn-WE-Topk, TracIn-last and TracIn-TFIDF as those were the most efﬁcient methods to run at large scale.
We vary k 2 { 20 , . .
. , 5000 } , and the AUC - DEL + and AUC - DEL  scores for our test set are reported in Table 4.
Unlike previous datasets, here TracIn-TFIDF does not perform better than TracIn-
Last, which may be because input similarity for MNLI cannot be merely captured by overlapping words.
For instance, a single negation would completely change the label of the sentence.
However, we again see TracIn-WE-Topk signiﬁcantly outperforms TracIn-Last and TracIn-TFIDF, demonstrating its efﬁcacy over natural language understanding tasks as well.
This again provides evidence that TracIn-WE can capture both low-level information and high-level information.
The deletion curve of Toxicity, AGnews, MNLI is in shown in Fig.
4.5 and Fig.
Toxicity-Roberta.
To additionally test whether our experiment results apply to more modern models, we repeat our experiments on the toxicity dataset with a Roberta model Liu et al.
(2019), while ﬁxing other settings.
We ﬁnd that the TracIn-WE and TracIn-WE-Topk still signiﬁcantly outperforms other results.
No Word Overlap.
To assess whether TracIn-WE can do well in settings where the training and test examples do not have overlapping words, we construct a controlled experiment on the Toxicity dataset.
We follow all experimental setting for Toxicity classiﬁcation with the Bert model, but making two additional changes – (1) given a test sentence x 0 , we only consider the top- 5000 training sentences (out of 50 , 000 ) with the least word overlap for computing inﬂuence.
We use TF-IDF similarity to rank the number of word overlaps so that stop word overlap will not be over-weighted.
(2) We also ﬁx the token embedding during training (result when word-embedding is not ﬁxed is in the appendix, where removing examples based on any inﬂuence method does not change the prediction), as we ﬁnd sentence with no word overlaps carry more inﬂuence when the token embedding is ﬁxed.
The AUC - DEL + and AUC - DEL  scores are reported in the lower section of Table 4.
We ﬁnd that
TracIn-WE variants can outperform last-layer based inﬂuence methods even in this controlled setting, showing that TracIn-WE can retrieve inﬂuential examples even without non-trivial word overlaps.
In
Section 4.5, we claimed that this gain stems from the presence of common tokens (“start”, “end”, and other frequent words).
To validate this, we compared with a controlled variant, TracIn-WENoCommon (TR-WE-NoC) where the common tokens are removed from TracIn-WE.
As expected, this variant performed much worse on the AUC - DEL + and AUC - DEL  scores, thus conﬁrming our claim.
We also ﬁnd that the result of TracIn-WE is better than TracIn-common (which is TracIn-WE with only “start” and “end” tokens), which shows that the common tokens such as stop words and punctuation may also help ﬁnding inﬂuential examples without meaningful word overlaps.
Related Work
In the ﬁeld of explainable machine learning, our works belongs to training data importance (Koh &
Liang, 2017; Yeh et al., 2018; Jia et al., 2019; Pruthi et al., 2020; Khanna et al., 2018; Sui et al., 2021).
Other forms of explanations include feature importance feature-based explanations, gradient-based explanations (Baehrens et al., 2010;
Simonyan et al., 2013; Zeiler & Fergus, 2014; Bach et al., 2015;
Ancona et al., 2018; Sundararajan et al., 2017; Shrikumar et al., 2017; Ribeiro et al., 2016; Lundberg & Lee, 2017; Yeh et al., 2019; Petsiuk et al., 2018) and perturbation-based explanations (Ribeiro et al., 2016; Lundberg & Lee, 2017; Yeh et al., 2019; Petsiuk et al., 2018), self-explaining models (Wang &
Rudin, 2015; Lee et al., 2019; Chen et al., 2019), counterfactuals to change the outcome of the model (Wachter et al., 2017; Dhurandhar et al., 2018; Hendricks et al., 2018; van der Waa et al., 2018; Goyal et al., 2019), concepts of the model (Kim et al., 2018;
Zhou et al., 2018).
For applications on applying data importance methods on NLP tasks, there have been works identifying data artifacts (Han et al., 2020; Pezeshkpour et al., 2021) and improving models (Han & Tsvetkov, 2020, 2021) based on existing data importance method using the inﬂuence function or TracIn.
In this work, we discussed weight parameter selection to reduce cancellation effect for training data attribution.
There has been works that discuss how to cope with cancellation in the context of feature attribution:
Liu et al.
(2020) discusses how regularization during training reduces cancellation of feature attribution, Kapishnikov et al.
(2021) discusses how to optimize IG paths to minimize cancellation of IG attribution, and
Sundararajan et al.
(2019) discusses improved visualizations to adjust for cancellation.
Conclusion
In this work, we revisit the common practice of computing training data inﬂuence using only last layer parameters.
We show that last layer representations in language classiﬁcation models can suffer from the cancellation effect, which in turn leads to inferior results on inﬂuence.
We instead recommend computing inﬂuence on the word embedding parameters, and apply this idea to propose a variant of
TracIn called TracIn-WE.
We show that TracIn-WE signiﬁcantly outperforms last versions of existing inﬂuence methods on three different language classiﬁcation tasks for several models, and also affords a word-level decomposition of inﬂuence that aids interpretability.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] In Sec.
B (c) Did you discuss any potential negative societal impacts of your work?
[Yes] In Sec.
C (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] In Sec.
D 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes]
In Sec.
E (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] Only used public data.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]