If Influence Functions are the Answer, Then What is the Question?
Anonymous Author(s)
AffiliationAddress email
Abstract
Influence functions efficiently estimate the effect of removing a single training datapoint on a model’s learned parameters.
While influence estimates align well withleave-one-out retraining for linear models, recent works have shown this alignmentis often poor in neural networks.
In this work, we investigate the specific factorsthat cause this discrepancy by decomposing it into five separate terms.
We studythe contributions of each term on a variety of architectures and datasets and howthey vary with factors such as network width and training time.
While practicalinfluence function estimates may be a poor match to leave-one-out retraining fornonlinear networks, we show that they are often a good approximation to a differentobject we term the proximal Bregman response function (PBRF) .
Since the PBRFcan still be used to answer many of the questions motivating influence functionssuch as identifying influential or mislabeled examples, our results suggest thatcurrent algorithms for influence function estimation give more informative resultsthan previous error analyses would suggest.
Introduction
The influence function [21, 13] is a classic technique from robust statistics that estimates the effect ofdeleting a single data example (or a group of data examples) from a training dataset.
Formally, given aneural network with learned parameters θ ⋆ trained on a dataset D , we are interested in the parametersθ
⋆ − z learned by training on a dataset D − { z } constructed by deleting a single training example zfrom D .
By taking the second-order Taylor approximation to the cost function around θ ⋆ , influencefunctions approximate the parameters θ ⋆ − z without the computationally prohibitive cost of retrainingthe model.
Since Koh and Liang [30] first deployed influence functions in machine learning, influencefunctions have been used to solve various tasks such as explaining model’s predictions [30, 22],relabelling harmful training examples [33], carrying out data poisoning attacks [31], and increasingfairness in models’ predictions [11, 47].
When the training objective is strongly convex (e.g., as in logistic regression with L 2 regularization),influence functions are expected to align well with leave-one-out (LOO) or leave- k -out retraining [30,32, 26].
However, Basu et al.
[7] showed that influence functions in neural networks often do notaccurately predict the effect of retraining the model and concluded that influence estimates are oftenerroneous.
Because of the poor match between influence estimates and LOO retraining, influencefunction methods are often evaluated with alternative metrics such as the recovery rate of maliciouslycorrupted examples using influence scores [28, 30, 46, 27].
But such downstream metrics are veryindirect.
For purposes of validation and algorithmic research, it would be far preferable to have awell-defined quantity that influence functions are meant to approximate.
[Figure]
In this work, we investigate the source of the discrepancy between influence functions and LOOretraining in neural networks.
We decompose the discrepancy into five components: (1) the differencebetween cold-start and warm-start response functions (a concept elaborated on below), (2) animplicit proximity regularizer, (3) influence estimation on non-converged parameters, (4) linearization,and (5) approximate solution of a linear system.
We empirically evaluate the contributions ofeach component on binary classification, regression, image classification, and language modelingtasks and show that, across all tasks, components (1–3) are most responsible for the discrepancybetween influence functions and LOO retraining.
We further investigate how the contribution of eachcomponent changes in response to the change in network width and depth, weight decay, trainingtime, damping, and the number of data points being removed.
Moreover, we show that while influence functions for neural networks are often a poor match to
LOO retraining, they are a much better match to what we term the proximal Bregman responsefunction (PBRF) .
Intuitively, the PBRF approximates the effect of removing a data point whileotherwise trying to keep the predictions consistent with those of the (partially) trained model.
Fromthis perspective, we reframe misalignment components (1–3) as simply reflecting the differencebetween LOO retraining and the PBRF.
The gap between the influence function estimate and the
PRBF comes from only sources (4) and (5), which we found empirically to be at least an order ofmagnitude smaller for most neural networks.
As a result, on a wide variety of tasks, we show thatinfluence functions closely align with the PBRF while failing to approximate the effect of retrainingthe model, as shown in Figure 1.
The PBRF can be used for many of the same use cases that have motivated influence functions, suchas finding influential or mislabeled examples and carrying out data poisoning attacks [30, 31], and cantherefore be considered an alternative to LOO retraining as a gold standard for evaluating influencefunctions.
Hence, we conclude that influence functions applied to neural networks are not inherently“fragile” as is often believed [7], but instead can be seen as giving accurate answers to a differentquestion than is normally assumed.
Related Work
Instance-based interpretability methods are a class of techniques that explain a model’s predictions interms of the examples on which the model was trained.
Methods of this type include TracIn [45],
Representer Point Selection [58], Grad-Cos and Grad-Dot [12, 23], MMD-critic [29], unconditionalcounterfactual explanations [55], and of central focus in this paper, influence functions [30, 21, 13].
Since its adoption in machine learning by Koh and Liang [30], multiple extensions and improvementsupon influence functions have also been proposed, such as variants that use Fisher kernels [28],higher-order approximations [8], tricks for faster and scalable inference [20, 46], group influenceformulations [32, 8], and relative local weighting [5].
However, many of these methods rely on thesame strong assumptions made in the original influence function derivation that the objective needsto be strongly convex and influence functions must be computed on the optimal parameters.
Measuring the quality of influence functions is often performed by measuring the correlation between
LOO retraining and influence function estimations [30, 7, 8, 57].
The effectiveness of influencefunctions can also be evaluated by retraining the model without the most influential data points [28, 20]or attempting to identify mislabelled or poisoned data [28, 30, 46, 27].
In general, influence functionsare assumed to approximate the effects of leave-out retraining from scratch.
However, recent empiricalanalyses have demonstrated the fragility of influence functions and a fundamental misalignmentbetween their assumed and actual effect [7, 18, 27].
For example, Basu et al.
[7] argued that theaccuracy of influence functions in deep networks is highly sensitive to network width and depth,weight decay strength, inverse-Hessian vector product estimation methodology, and test query pointby measuring the alignment between influence functions and LOO retraining.
Although these resultswere interpreted as a failure of influence functions, we claim that many of these apparent failuressimply reflect that influence functions answer a different question than is usually assumed.
Background
Consider a prediction task from an input space X to a target space T where we are given a finitetraining dataset D train =
{ ( x ( i ) , t ( i ) ) }
Ni =1 .
Given a data point z
= ( x , t ) , let y
=
f ( θ , x ) be theprediction of the network parameterized by θ ∈ R d and L ( y , t ) be the loss (e.g., squared error orcross-entropy).
We aim to solve the following optimization problem:
[Equation]
where J ( · ) is the cost function.
We summarize the notation used in this paper in Appendix A.
3.1 Downweighting a Training Example
The training objective in Eqn.
1 aims to find the parameters that minimize the average loss on alltraining examples.
Herein, we are interested in studying the change in optimal model parameterswhen a particular training example
z
=
( x , t ) ∈ D train is removed from the training dataset, or moregenerally, when the data point z is downweighted by an amount ϵ ∈ R .
Formally, this corresponds tominimizing the following downweighted objective:
[Equation]
When ϵ = 1 / N , the downweighted objective reduces to the cost over the dataset with the example zremoved, up to a constant factor.
To see how the optimum of the downweighted objective responds tochanges in the downweighting factor ϵ , we define the response function r ⋆ − z : R → R d by:
[Equation]
where we assume that the downweighted objective is strongly convex and the solution to the downweighted objective is unique given some factor ϵ .
Note that r ⋆ − z (0)
=
θ
⋆ and the response functionis differentiable at 0 by the Implicit Function Theorem [34, 19] when the downweighted objective isfurther twice continuously differentiable.
Influence functions approximate the response function byperforming a first-order Taylor expansion around ϵ 0 = 0 :
[Equation]
We refers readers to Van der Vaart [52] and Appendix B for a detailed derivation.
The optimalparameters trained without z can then be approximated by plugging in ϵ = 1 / N to Eqn. 4.
3.2 Influence Function Estimation in Neural Networks
Influence functions face two main challenges when deployed on neural networks.
First, the influence estimation (shown in Eqn. 4) requires computing an inverse Hessian-vector product ( iHVP ).
Unfortunately, storing and inverting the Hessian requires O ( d 3 ) operations and is infeasible to compute for modern neural networks.
Instead, Koh and Liang [30] tractably approximate the iHVP usingtruncated non-linear conjugate gradient ( CG ) [43] or the LiSSA algorithm [1].
Both approaches avoidexplicit computation of the Hessian inverse (see Appendix G for details) and only require O ( Nd )operations to approximate the influence function.
Second, the derivation of influence functions assumes a strongly convex objective, which is often notsatisfied for neural networks.
The Hessian may be singular, especially when the parameters have notfully converged, due to non-positive eigenvalues.
To enforce positive-definiteness of the Hessian, Kohand Liang [30] add a damping term in the iHVP .
Teso et al.
[51] further approximate the Hessian withthe Fisher information matrix (which is equivalent to the Gauss-Newton Hessian [41]) as follows:
[Equation]
where J y θ ⋆ is the parameter-output Jacobian and H y is the Hessian of the cost with respect to thenetwork outputs.
Here, λ > 0 is a damping term to ensure the invertibility of Hessian or GaussNewton Hessian (GNH).
Unlike the Hessian, the GNH is guaranteed to be positive semidefinite aslong as the loss function is convex as a function of the network outputs [43].
Understanding the Discrepancy between Influence Function and LOORetraining in Neural Networks
In this section, we investigate several factors responsible for the misalignment between influencefunctions and LOO retraining.
Specifically, we decompose the misalignment into five separate terms:(1) the warm-start gap, (2) damping gap, (3) non-convergence gap, (4) linearization error, and (5)solver error.
By summing the parameter (or outputs) differences introduced by each term we canbound the parameter (or outputs) difference between LOO retraining and influence estimates.
We usethe term “gap” rather than “error” for the first three terms to emphasize that they reflect differencesbetween solutions to different influence-related questions, rather than actual errors.
For all modelswe investigate, we find that the first three sources dominate the misalignment, indicating that themisalignment reflects not algorithmic errors but rather the fact that influence function estimatorsare answering a different question from what is normally assumed.
All proximal objectives aresummarized in Table 1.
[Figure]
(1) Warm-Start Gap: Non-Strongly Convex Training Objective.
By taking a first-order Taylor approximation of theresponse function at ϵ 0 = 0 (Eqn. 4), influence functions approximate the effect of removing a data point z at a local neighborhood of the optimum θ ⋆ .
Hence, influence approximationhas a more natural connection to the retraining scheme that initializes the network at the current optimum θ ⋆ ( warm-start retraining ) than the scheme that initializes the network randomly( cold-start retraining ).
The warm-start optimum is equivalentto the cold-start optimum when the objective is strongly convex(where the solution to the response function is unique), makingthe influence estimation close to the LOO retraining on logisticregression with L 2 regularization.
However, the equivalence between warm-start and cold-startoptima is not typically guaranteed in neural networks [53].
Particularly, in the overparametrized regime ( N < d ), neuralnetworks exhibit multiple global optima, and their convergedsolutions depend highly on the specifics of the optimizationdynamics [38, 4, 6, 3].
For quadratic cost functions, gradientdescent with initialization θ 0 converges to the optimum that achieves the minimum L 2 distancefrom θ
This phenomenon of the converged parameters being dependent on the initializationhinders influence functions from accurately predicting the effect of retraining the model from scratchas shown in Figure 2.
We denote the discrepancy between cold-start and warm-start optima aswarm-start gap .
(2) Proximity Gap: Addition of Damping Term in iHVP .
In practical settings, we often impose adamping term (Eqn. 5) in influence approximations to ensure that the cost Hessian is positive-definiteand hence invertible.
As adding a damping term in influence estimations is equivalent to adding L 2regularization to the cost function [43], when damping is used, influence functions can be seen aslinearizing the following proximal response function at ϵ 0 = 0 : [Equation]
[Equation]
Note that our use of “proximal” is based on the notion of proximal equilibria [17].
Intuitively, theproximal objective in Eqn.
6 not only minimizes the downweighted objective but also encouragesthe parameters to stay close to the optimal parameters at ϵ 0 = 0 .
Hence, when the damping term isused in the iHVP , influence functions aim at approximating the warm-start retraining scheme with aproximity term that penalizes the L 2 distance between the new estimate and the optimal parameters.
We call the discrepancy between the warm-start and proximal warm-start optima the proximity gap .
Interestingly, past works have observed that for quadratic cost functions, early stopping has a similareffect to L 2 regularization [53, 2].
Therefore, the proximal response function can be thought of ascapturing how gradient descent will respond to a dataset perturbation if it takes only a limited numberof steps starting from the warm-start solution.
[Figure]
(3) Non-Convergence Gap
Influence Estimation
on Non-Converged Parameters.
Thus far, our analysis has assumed that influence functions are computed on fully converged parameters θ ⋆ at which thegradient of the cost is 0 .
However, in neural network training, we often terminate the optimizationprocedure before reaching the exact optimum due toseveral reasons, including having limited computational resources or to avoid overfitting [9].
In suchsituations, much of the change in the parameters from
LOO retraining simply reflects the effect of trainingfor longer, rather than the effect of removing a training example, as illustrated in Figure 3.
What wedesire from influence functions is to understand theeffect of removing the training example; the effect of extended training is simply a nuisance.
Therefore, to the extent that this factor contributes to the misalignment between influence functions and
LOO retraining, influence functions are arguably more useful than LOO retraining.
Since training the network to convergence may be impractical or undesirable, we instead modifythe response function by replacing the original training objective with a similar one for which the(possibly non-converged) final parameters θ s are optimal .
Here, we assume the loss function L ( · , · )is convex as a function of the network outputs; this is true for commonly used loss functions suchas squared error or softmax-cross-entropy.
We replace the training loss with a term that penalizesmismatch to the predictions made by θ s (hence implying that θ s is optimal).
Our proximal Bregmanresponse function (PBRF) is defined as follows:
[Equation]
[Equation]
where D L ( i ) ( · , · ) is the Bregman divergence defined as:
[Equation]
See Appendix B for the derivation.
The PBRF defined in Eqn.
7 is composed of three terms.
Thefirst term measures the functional discrepancy between the current estimate and the parameters θ sin Bregman divergence, and its role is to prevent the new estimate from drastically altering the
[Table]
predictions on the training dataset. One way of understanding this term in the cases of squared erroror softmax-cross-entropy losses is that it is equivalent to the training error on a dataset where theoriginal training labels are replaced with soft targets obtained from the predictions made by θ s .
Thesecond term is the negative loss on the data point z
= ( x , t ) , which aims to respond to the deletion ofa training example.
The final term is simply the proximity term described before.
Rather than computing the LOO retrained parameters by performing K additional optimizationsteps under the original training objective, we can instead perform K optimization steps under theproximal Bregman objective.
The difference between the resulting parameter vectors is what we callthe non-convergence gap .(4)
Linearization Error:
A First-order Taylor Approximation of the Response Function.
Thekey idea behind influence functions is the linearization of the response function.
We define thelinearized PBRF as:
[Equation]
where L quad ( · , · ) is the second-order expansion of the loss around y s and f lin ( · , · ) is the linearizationof the network outputs with respect to the parameters.
The linearized PBRF is equivalent to theinfluence estimation at the parameters θ s with the GNH approximation and a damping term λ .
Asthe linearized PBRF relies on several local approximations, the linearization error increases whenthe downweighting factor magnitude | ϵ | is large or the PBRF is highly non-linear.
We refer to thediscrepancy between the PBRF and linearized PBRF as the linearization error .(5)
Solver Error:
A Crude Approximation of iHVP .
As the precise computation of the iHVP iscomputationally infeasible, in practice, we use truncated CG or LiSSA to efficiently approximateinfluence functions [30].
Unfortunately, these efficient linear solvers introduce additional error bycrudely approximating the iHVP .
Moreover, different linear solvers can introduce specific biases inthe influence estimation.
For example, Vicol et al.
[54] show that the truncated LiSSA algorithmimplicitly adds an additional damping term in the iHVP .
We use solver error to refer to the differencebetween the linearized PBRF and the influence estimation computed by a linear solver.
PBRF: The Question Influence Functions are Really Answering
The PBRF (Eqn. 7) approximates the effect of removing a data point while otherwise trying to keepthe predictions consistent with those of the (partially) trained model.
Since the discrepancy betweenthe PBRF and influence function estimates is only due to the linearization and solver errors, the PBRFcan be thought of as better representing the question that influence functions are trying to answer.
Reframing influence functions in this way means that the PBRF can be regarded as a gold-standardground truth for evaluating methods for influence function approximation.
Existing analyses ofinfluence functions [7] rely on generating LOO retraining ground truth estimates by imposing strong
L 2 regularization or training till convergence without early stopping.
However, these conditions do not accurately reflect the typical way neural networks are trained in practice.
In contrast, our PBRFformulation does not require the addition of any regularizers or modified training regimes and can beeasily optimized.
In addition, although the PBRF may not necessarily align with LOO retraining due to the warm-start,proximity, and non-convergence gaps, the motivating use cases for influence functions typically donot rely on exact LOO retraining.
This means that the PBRF can be used in place of LOO retrainingfor many tasks such as identifying influential or mislabelled examples.
In these cases, influencefunctions are still useful since they provide an efficient way of approximating PBRF estimates.
Experiments
Our experiments investigate the following questions: (1) What factors discussed in Section 4 contribute most to the misalignment between influence functions and LOO retraining?
(2) While influencefunctions fail to approximate the effect of retraining, do they accurately approximate the PBRF?
How do changes in weight decay, damping, the number of total epochs, and the number of removedtraining examples affect each source of misalignment?
In all experiments, we first train the base network with the entire dataset to obtain the parametersθ s .
We repeat the training procedure 20 times with a different random training example deleted.
Thecold-start retraining begins from the same initialization used to train θ s .
All proximal objectives aretrained with initialization θ s for 50% of the epochs used to train the base network.
Lastly, we use the LiSSA algorithm with GNH approximation to compute influence functions.
Since we are primarily interested in the effect of deleting a data point on model’s predictions, wemeasure the discrepancy of each gap and error using the average L 2 distance between networks’outputs E ( x , · ) ∼D train [ ∥ f ( θ , x ) − f ( θ ′ , x ) ∥ ] on the training dataset.
We provide the full experimentalset-up and additional experiments in Appendix C and D, respectively.
6.1 Influence Misalignment Decomposition
We first applied our decomposition to various models trained on a broad range of tasks coveringbinary classification, regression, image reconstruction, image classification, and language modeling.
The summary of our results is provided in Figure 4 and Table 5 (Appendix E).
Across all tasks, wefound that the first three sources dominate the misalignment, indicating influence function estimatorsare answering a different question from what is normally assumed.
Small linearization and solvererrors indicates that influence functions accurately answers the modified question (PBRF).
Logistic Regression.
We analyzed the logistic regression (LR) model trained on the Cancer and
Diabetes classification datasets from the UCI collection [16].
We trained the model using L-BFGS [39]with L 2 regularization of 0.01 and damping term of λ = 0 .
As the training objective is stronglyconvex and the base model parameters were trained till convergence, we observed that each sourceof misalignment is significantly low.
Hence, in case of logistic regression with L 2 regularization,influence functions accurately capture the effect of retraining the model without a data point.
Multilayer Perceptron.
Next, we applied our analysis to the 2-hidden layer Multilayer Perceptron(MLP) with ReLU activations.
We conducted the experiments in two settings: (1) regression onthe Concrete and Energy datasets from the UCI collection and (2) image classification on 10% ofthe MNIST [15] and FashionMNIST [56] datasets, following the set-up from Koh and Liang [30]and Basu et al.
We trained the networks for 1000 epochs using stochastic gradient descent (SGD)with a batch size of 128 and set a damping strength of λ = 0 .
As opposed to linear models, MLPs violate the assumptions in the influence derivation and weobserved an increase in gaps and errors on all five factors.
We observed that warm-start, proximity,and the non-convergence gaps contribute more to the misalignment than linearization and solvererrors.
The average network’s predictions for PBRF were similar to that computed by the LiSSAalgorithm, demonstrating that influence functions are still a good approximation to PBRF.
[Figure]
Autoencoder.
Next, we applied our framework to an 8-layer autoencoder (AE) on the full MNISTdataset.
We followed the experimental set-up from Martens and Grosse [42], where the encoderand decoder each consist of 4 fully-connected layers with sigmoid activation functions.
We trainedthe network for 1000 epochs using SGD with momentum.
We set the batch size to 1024, used L 2regularization of 10 − 5 with a damping factor of λ = 0 .
In accordance with the findings from our
MLP experiments, the warm-start, proximity, and non-convergence gaps were more significant thanthe linearization and solver errors, and influence functions accurately predicted the PBRF.
[Figure]
[Table]
Convolutional Neural Networks.
To investigatethe source of discrepancy on larger-scale networks,we trained a set of convolutional neural networks ofincreasing complexity and size.
Namely, LeNet [37],
AlexNet [36], VGG13 [49], and ResNet-20 [25] weretrained on 10% of the MNIST dataset and the full
CIFAR10 [35] dataset.
We trained the base networkfor 200 epochs on both datasets with a batch sizeof 128.
For MNIST, we kept the learning rate fixedthroughout training, while for CIFAR10, we decayedthe learning rate by a factor of 5 at epochs 60, 120,and 160, following Zagoruyko and Komodakis [59].
We used L 2 regularization with strength 5 · 10 − 4and a damping factor of λ = 0 . 001 .
Consistent with the findings from our MLP and autoencoderexperiments, the first three gaps were more significant than linearization and solver errors.
We further compared influence functions’ approximations on the difference in test loss when arandom training data point is removed with the value obtained from cold-start retraining, warm-startretraining, and PBRF in Table 2.
We used both Pearson [48] and Spearman rank-order correlation [50]to measure the alignment.
While the test loss predicted by influence functions do not align well withthe values obtained by cold-start and warm-start retraining schemes, they show high correlationswhen compared to the estimates given by PBRF.
Transformer.
Finally, we trained 2-layer Transformer language models on the Penn Treebank(PTB)
[40] dataset.
We set the number of hidden dimensions to 256 and the number of attention headsto 2 .
As we observed that model overfits after a few epochs of training, we trained the base network for10 epochs using Adam.
Notably, we observed that the non-convergence gap had the most considerablecontribution to the discrepancy between influence functions and LOO retraining.
Consistent with ourprevious findings, the first tree gaps had more impact on the discrepancy compared to linearizationand solver errors.
6.2 Factors in Influence Misalignment
We further analyzed how the contribution of each component changes in response to changes innetwork width and depth, training time, weight decay, damping, and the percentage of data removed.
We used an MLP trained on 10% of the MNIST dataset and summarized results in Figure 5.
[Figure]
Width and Depth.
We find that as we increase the width of the network, we observe a decrease in thelinearization error.
This is consistent with previous observations that networks behave more linearlyas the width is increased [38].
In contrast to the findings from Basu et al.
[7], we did not observe astrong relationship between the contribution of the components and the depth of the network.
Training Time.
Unsurprisingly, as we increase the number of training epochs, we observe a decreasein the non-convergence gap.
Weight Decay.
The weight decay allows the training objective to be better conditioned.
Consequently,as weight decay increases, the training objective may act more as a strictly convex objective, resultingin a decrease in overall discrepancy for all components.
Basu et al.
[7] also found that the alignmentbetween influence functions and LOO retraining increases as weight decay increases.
Damping.
A higher damping term makes linear systems better conditioned, allowing solvers tofind accurate solutions in fewer iterations [14], thereby reducing the solver error.
Furthermore, thehigher proximity term keeps the parameters close to θ s , reducing the linearization error.
However,increasing the effective proximity penalty directly increases the proximity gap.
Percentage of Training Examples Removed.
As we remove more training examples from thedataset the PBRF becomes more non-linear and we observe a sharp increase in the linearization error.
The cost landscape is also more likely to change as we remove more training examples, and weobserve a corresponding increase in the warm-start gap.
Conclusion
In this paper, we investigate the sources of the discrepancy between influence functions and LOOretraining in neural networks.
We decompose this difference into five distinct components: the warmstart gap, proximity gap, non-convergence gap, linearization error, and solver error.
We empiricallyevaluate the contributions of each of these components on a wide variety of architectures and datasetsand investigate how they change with factors such as network size and regularization.
Our resultsshow that the first three components are most responsible for the discrepancy between influencefunctions and LOO retraining.
We further introduce the proximal Bregman response function (PBRF)to better capture the behavior of influence functions in neural networks.
Compared to LOO retraining,the PBRF is more easily calculated and correlates better with influence functions, meaning it is anattractive alternative gold standard for evaluating influence functions.
Although the PBRF may notnecessarily align with LOO retraining, it can still be applied in many of the motivating use casesfor influence functions.
We conclude that influence functions in neural networks are not necessarily“fragile”, but instead are giving accurate answers to a different question than is normally assumed.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 5.
(b) Did you describe the limitations of your work?
[Yes] See Section 4 for ways in which
the PBRF differs from LOO retraining and differs from influence estimates.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work
is fundamental in nature and does not enable immediate misuse.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See background
in Section 3 and Appendix B.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix B.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We have not
attached the code but we will release in the future.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We provide training details in section 6 and in C.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See full results in Table 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See dataset
citationsin section 6.
(b) Did you mention the license of the assets?
[Yes] All datasets are licensed under MIT
data license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[-We further let ϵ 
-]
[-= 
-]
[-Let ϵ 
-]
[-= 
-]
[-27 is non-negativeand is 
-]
spent on participant compensation?
[N/A]