Original: M ULTI -S TAGE I NFLUENCE F UNCTION
Revised: M ULTI -S TAGE I NFLUENCE F UNCTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multi-stage training and knowledge transfer from a large-scale pretrain task to various ﬁnetune end tasks have revolutionized natural language processing (NLP) and computer vision (CV), with state-of-the-art performances constantly being improved.
Revised: Multi-stage training and knowledge transfer from a large-scale pretraining task to various ﬁnetuning tasks have revolutionized natural language processing (NLP) and computer vision (CV), with state-of-the-art performances constantly being improved.

Original: In this paper, we develop a multi-stage inﬂuence function score to track predictions from a ﬁnetune model all the way back to the pretrain data.
Revised: In this paper, we develop a multi-stage inﬂuence function score to track predictions from a ﬁnetuned model all the way back to the pretraining data.

Original: With this score, we can identify the pretrain examples in the pretrain task that contribute most to a prediction in the ﬁnetune task.
Revised: With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the ﬁnetuning task.

Original: The proposed multi-stage inﬂuence function generalizes the original inﬂuence function for a single model in Koh & Liang (2017), thereby enabling inﬂuence computation through both pretrain and ﬁnetune models.
Revised: The proposed multistage inﬂuence function generalizes the original inﬂuence function for a single model in Koh & Liang (2017), thereby enabling inﬂuence computation through both pretrained and ﬁnetuned models.

Original: We study two different scenarios with the pretrain embeddings ﬁxed or updated in the ﬁnetune tasks.
Revised: We study two different scenarios with the pretrained embeddings ﬁxed or updated in the ﬁnetuning tasks.

Original: We test our proposed method in various experiments to show its effectiveness and potential applications.
Revised: We test our proposed method in various experiments to show its effectiveness and potential applications.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-stage training has become increasingly important and has achieved state-of-the-art results in many tasks.
Revised: Multi-stage training has become increasingly important and has achieved state-of-the-art results in many tasks.

Original: In NLP applications, it is now a common practice to ﬁrst learn word embeddings (e.g., word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014)) or contextual representations (e.g., ELMo (Peters et al., 2018), BERT (Devlin et al., 2018)) from a large unsupervised corpus, and then reﬁne or ﬁnetune the model on supervised end tasks.
Revised: In NLP applications, it is now a common practice to ﬁrst learn word embeddings (e.g., word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014)) or contextual representations (e.g., ELMo (Peters et al., 2018), BERT (Devlin et al., 2018)) from a large unsupervised corpus, and then reﬁne or ﬁnetune the model on supervised end tasks.

Original: In computer vision applications, it is common to use a pretrained CNN as a feature extractor and only ﬁnetune top-layer networks through training on the end task.
Revised: In computer vision applications, it is common to use a pretrained CNN as a feature extractor and only ﬁnetune top-layer networks through training on the end task.

Original: Also, it has been demonstrated that pretraining ResNet (He et al., 2016) with large hashtag data can greatly beneﬁt many end tasks (Mahajan et al., 2018).
Revised: Also, it has been demonstrated that pretraining ResNet (He et al., 2016) with large hashtag data can greatly beneﬁt many end tasks (Mahajan et al., 2018).

Original: Intuitively, the successes of these multi-stage learning paradigms are due to knowledge transfer from pretrain tasks to the end task.
Revised: Intuitively, the successes of these multi-stage learning paradigms are due to knowledge transfer from pretraining tasks to the end task.

Original: However, current approaches using multi-stage learning are usually based on trial-and-error and many fundamental questions remain unanswered.
Revised: However, current approaches using multi-stage learning are usually based on trial-and-error and many fundamental questions remain unanswered.

Original: For example, which part of the pretrain data/task contributes most to the end task?
Revised: For example, which part of the pretraining data/task contributes most to the end task?

Original: How can one detect “false transfer” where some pretrain data/task could be harmful for the end task?
Revised: How can one detect “false transfer” where some pretraining data/task could be harmful for the end task?

Original: If a testing point is wrongly predicted by the ﬁnetune model, can we trace back to the problematic examples in the pretrain data?
Revised: If a testing point is wrongly predicted by the ﬁnetuned model, can we trace back to the problematic examples in the pretraining data?

Original: Answering these questions requires a quantitative measurement of how the data and loss function in the pretrain stage inﬂuence the end model, which has not been studied in the past and will be the main focus of this paper.
Revised: Answering these questions requires a quantitative measurement of how the data and loss function in the pretraining stage inﬂuence the end model, which has not been studied in the past and will be the main focus of this paper.

Original: To ﬁnd the most inﬂuential training data responsible for a model’s prediction, the inﬂuence function was ﬁrst introduced by Cook & Weisberg (1980) from a robust statistics point of view.
Revised: To ﬁnd the most inﬂuential training data responsible for a model’s prediction, the inﬂuence function was ﬁrst introduced by Cook & Weisberg (1980) from a robust statistics point of view.

Original: Recently, as large-scale applications become more challenging for inﬂuence function computation, Koh & Liang (2017) proposed to use a ﬁrst-order approximation to measure the effect of removing one training point on the model’s prediction, to overcome computational challenges.
Revised: Recently, as large-scale applications become more challenging for inﬂuence function computation, Koh & Liang (2017) proposed to use a ﬁrst-order approximation to measure the effect of removing one training point on the model’s prediction, to overcome computational challenges.

Original: More broadly, there are many works using inﬂuence functions to investigate the impact of training data on models in various machine learning applications, such as tracing back the origins of bias in the word embeddings generated by GloVe (Brunet et al., 2019), and understanding and mitigating disparate impact to improve model fairness (Wang et al., 2019).
Revised: More broadly, there are many works using inﬂuence functions to investigate the impact of training data on models in various machine learning applications, such as tracing back the origins of bias in the word embeddings generated by GloVe (Brunet et al., 2019), and understanding and mitigating disparate impact to improve model fairness (Wang et al., 2019).

Original: However, all of the existing inﬂuence score computation algorithms study the case of single-stage training – where there is only one model with one set of training/prediction data in the training process.
Revised: However, all of the existing inﬂuence score computation algorithms study the case of single-stage training – where there is only one model with one set of training/prediction data in the training process.

Original: To the best of our knowledge, the inﬂuence of pretrain data on a subsequent ﬁnetune task and model has not been studied, and it is nontrivial to apply the original inﬂuence function in (Koh & Liang, 2017) to this scenario.
Revised: To the best of our knowledge, the inﬂuence of pretraining data on a subsequent ﬁnetuning task and model has not been studied, and it is nontrivial to apply the original inﬂuence function in (Koh & Liang, 2017) to this scenario.

Original: In this work, we derive the inﬂuence function from pretrain data to the end task in multi-stage training.
Revised: In this work, we derive the inﬂuence function from pretraining data to the end task in multi-stage training.

Original: Since the computation involves several expensive Hessian vector products, we also show how to compute the inﬂuence function efﬁciently in large-scale problems.
Revised: Since the computation involves several expensive Hessian vector products, we also show how to compute the inﬂuence function efﬁciently in large-scale problems.

Original: Based on this technique, we show that
Revised: Based on this technique, we show that

Original: • in real datasets and experiments across various vision and NLP tasks, predictions using the technique and actual inﬂuence for the pretrain data to the ﬁnetune model are highly correlated (Pearson’s r score to be around 0.6).
Revised: • in real datasets and experiments across various vision and NLP tasks, predictions using the technique and actual inﬂuence for the pretraining data to the ﬁnetuned model are highly correlated (Pearson’s r score to be around 0.6).

Original: This shows the effectiveness of our proposed technique for computing the inﬂuence scores in multi-stage models; • the inﬂuence for the pretrain data to the ﬁnetune model can be split into two parts: the inﬂuence of the pretrain data on the pretrain model, and inﬂuence of the pretrain data on the ﬁnetune model.
Revised: This shows the effectiveness of our proposed technique for computing the inﬂuence scores in multi-stage models; • the inﬂuence for the pretraining data to the ﬁnetuned model can be split into two parts: the inﬂuence of the pretraining data on the pretrained model, and inﬂuence of the pretraining data on the ﬁnetuned model.

Original: Therefore the testing data from the ﬁnetune task will be impacted by changes in the pretrain data, which can be quantiﬁed using our proposed technique; • the inﬂuence of the pretrain data on the ﬁnetune task is highly dependent on 1) the similarity of two tasks or stages, 2) and the number of training data in the ﬁnetune task.
Revised: Therefore the testing data from the ﬁnetuning task will be impacted by changes in the pretraining data, which can be quantiﬁed using our proposed technique; • the inﬂuence of the pretraining data on the ﬁnetuning task is highly dependent on 1) the similarity of two tasks or stages, 2) and the number of training data in the ﬁnetuning task.

Original: Thus our proposed technique provides a novel way to measure how the pretrain data helps or beneﬁts the ﬁnetune data.
Revised: Thus our proposed technique provides a novel way to measure how the pretraining data helps or beneﬁts the ﬁnetuning data.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Multi-stage model training that trains models in many stages on different tasks to improve the end-task has been used widely in many machine learning areas, such as in transfer learning (Ando & Zhang, 2005) and zero-shot learning (Larochelle et al., 2008).
Revised: Multi-stage model training that trains models in many stages on different tasks to improve the endtask has been used widely in many machine learning areas, such as in transfer learning (Ando & Zhang, 2005) and zero-shot learning (Larochelle et al., 2008).

Original: Recently, multi-stage model training has achieved state-of-the-art performance by learning large embeddings or data representation as the pretrain step on a very large pretrain dataset, which is then followed by a ﬁnetune step with further training on the end-task.
Revised: Recently, multi-stage model training has achieved state-of-the-art performance by learning large embeddings or data representation as the pretraining step on a very large pretraining dataset, which is then followed by a ﬁnetune step with further training on the end-task.

Original: Examples include the recently proposed BERT (Devlin et al., 2018), which learns contextual embeddings on a large corpus with the pretrain tasks chosen to be predicting the masked words in a sentence and predicting whether one sentence is after another sentence.
Revised: Examples include the recently proposed BERT (Devlin et al., 2018), which learns contextual embeddings on a large corpus with the pretraining tasks chosen to be predicting the masked words in a sentence and predicting whether one sentence is after another sentence.

Original: This contextual embedding is then used in ﬁnetune tasks, such as a question answering task.
Revised: This contextual embedding is then used in ﬁnetuning tasks, such as a question answering task.

Original: ELMo (Peters et al., 2018) is widely used in multi-stage model training as a sentence feature extractor to beneﬁt the end-task.
Revised: ELMo (Peters et al., 2018) is widely used in multi-stage model training as a sentence feature extractor to beneﬁt the end-task.

Original: Similarly, there are some works in computer vision that train an image representation model on a large number of images as the pretrain step, and then use the resulting features to ﬁnetune another task, such as particular image classiﬁcation tasks.
Revised: Similarly, there are some works in computer vision that train an image representation model on a large number of images as the pretraining step, and then use the resulting features to ﬁnetune another task, such as particular image classiﬁcation tasks.

Original: For example, (Mahajan et al., 2018) uses ResNet in the pretrain step, and the ﬁnetune task is based on hashtag data.
Revised: For example, (Mahajan et al., 2018) uses ResNet in the pretraining step, and the ﬁnetuning task is based on hashtag data.

Original: The rationale for this multi-stage model is that the pretrain task can learn some common or latent representation which could beneﬁt the end task.
Revised: The rationale for this multi-stage model is that the pretraining task can learn some common or latent representation which could beneﬁt the end task.

Original: Another related line of research is on understanding machine learning models.
Revised: Another related line of research is on understanding machine learning models.

Original: One category of research is to explain predictions with respect to model variables, and trace back the contribution of variables to the prediction.
Revised: One category of research is to explain predictions with respect to model variables, and trace back the contribution of variables to the prediction.

Original: For example, Oramas et al.
Revised: For example, Oramas et al.

Original: (2019) automatically detects internal features in the set of classes in the pretrain model that are relevant to interpreting the prediction, and shows for various vision tasks that the proposed scheme can produce detailed explanations based on the features that are relevant to the targeted classes.
Revised: (2019) automatically detects internal features in the set of classes in the pretrained model that are relevant to interpreting the prediction, and shows for various vision tasks that the proposed scheme can produce detailed explanations based on the features that are relevant to the targeted classes.

Original: Guo et al.
Revised: Guo et al.

Original: (2019) aims to interpret variable-wise hidden states in an LSTM model to quantify variable importance and variable-wise temporal importance in the model.
Revised: (2019) aims to interpret variable-wise hidden states in an LSTM model to quantify variable importance and variable-wise temporal importance in the model.

Original: Closely related research has sought to connect model prediction and training data, and trace back the most inﬂuential training data that are most responsible for the model prediction.
Revised: Closely related research has sought to connect model prediction and training data, and trace back the most inﬂuential training data that are most responsible for the model prediction.

Original: Among them, the inﬂuence function (Cook & Weisberg, 1980; Koh & Liang, 2017), which aims to model the prediction changes when training data is added/removed, has been shown to be effective in many applications.
Revised: Among them, the inﬂuence function (Cook & Weisberg, 1980; Koh & Liang, 2017), which aims to model the prediction changes when training data is added/removed, has been shown to be effective in many applications.

Original: There is a series of work on inﬂuence functions, including investigating the inﬂuence of a group of data on the prediction (Koh et al., 2019), using inﬂuence functions to detect bias in word embeddings (Brunet et al., 2019), and using it in preventing data poisoning attacks (Steinhardt et al., 2017), etc.
Revised: There is a series of work on inﬂuence functions, including investigating the inﬂuence of a group of data on the prediction (Koh et al., 2019), using inﬂuence functions to detect bias in word embeddings (Brunet et al., 2019), and using it in preventing data poisoning attacks (Steinhardt et al., 2017), etc.

Original: All of these works only consider a single stage training procedure, and it is not straightforward to apply the existing inﬂuence functions to multi-stage models.
Revised: All of these works only consider a single stage training procedure, and it is not straightforward to apply the existing inﬂuence functions to multi-stage models.

Original: In this paper, we propose to analyze the inﬂuence of pretrain data on predictions in the subsequent ﬁnetune model and end task.
Revised: In this paper, we propose to analyze the inﬂuence of pretraining data on predictions in the subsequent ﬁnetuned model and end task.

Original:  A LGORITHMS
Revised:  A LGORITHMS

Original: In this section, we detail the procedure of multi-stage training, show how to compute the inﬂuence score for the multi-stage training, and then discuss how to scale up the computation.
Revised: In this section, we detail the procedure of multi-stage training, show how to compute the inﬂuence score for the multi-stage training, and then discuss how to scale up the computation.

Original: 3. M ULTI -S TAGE M ODEL T RAINING
Revised: 3. M ULTI -S TAGE M ODEL T RAINING

Original: Multi-stage models, which train different models in consecutive stages, have been widely used in various ML tasks.
Revised: Multi-stage models, which train different models in consecutive stages, have been widely used in various ML tasks.

Original: Mathematically, let Z be the training set for pretrain task with data size |Z| = m , and X be the training data for the ﬁnetune task with data size |X| = n .
Revised: Mathematically, let Z be the training set for pretraining task with data size |Z| = m , and X be the training data for the ﬁnetuning task with data size |X| = n .

Original: In pretrain stage, we assume the parameters of the pretrain network have two parts: the parameters W that are shared with the end task, and the task-speciﬁc parameters U that will only be used in the pretrain stage.
Revised: In pretraining stage, we assume the parameters of the pretrained network have two parts: the parameters W that are shared with the end task, and the task-speciﬁc parameters U that will only be used in the pretraining stage.

Original: Note that W could be a word embedding matrix (e.g., in word2vec) or a representation extraction network (e.g., Elmo, BERT, ResNet), while U is usually the last few layers that corresponds to the pretrain task.
Revised: Note that W could be a word embedding matrix (e.g., in word2vec) or a representation extraction network (e.g., Elmo, BERT, ResNet), while U is usually the last few layers that corresponds to the pretraining task.

Original: After training on the pretrain task, we obtain the optimal parameters W ∗ , U ∗ .
Revised: After training on the pretraining task, we obtain the optimal parameters W ∗ , U ∗ .

Original: The pretrain stage can be formulated as
Revised: The pretraining stage can be formulated as

Revised: Pretrain Stage:

Original: [Equation]
Revised: [Equation]

Original: where g ( · ) is the loss function for the pretrain task.
Revised: where g ( · ) is the loss function for the pretrain task.

Original: In the ﬁnetune stage, the network parameters are W, Θ , where W is shared with the pretrain task and Θ is the rest of the parameters speciﬁcally associated with the ﬁnetune task.
Revised: In the ﬁnetuning stage, the network parameters are W, Θ , where W is shared with the pretraining task and Θ is the rest of the parameters speciﬁcally associated with the ﬁnetuning task.

Original: We will initialize the W part by W ∗ .
Revised: We will initialize the W part by W ∗ .

Original: Let f ( · ) denote the ﬁnetune loss, there are two cases when ﬁnetune the end-task:
Revised: Let f ( · ) denote the ﬁnetuning loss, there are two cases when ﬁnetuning the end-task:

Original: • Finetune Case 1: Fixing embedding parameters W = W ∗ , and only ﬁnetune Θ :Θ [Equation] = arg min Θ
Revised: • Finetuning Case 1: Fixing embedding parameters W = W ∗ , and only ﬁnetune Θ :

Original: [Equation]
Revised: [Equation]

Original: • Finetune Case 2: ﬁnetune both the embedding parameters W (initialized from W ∗ ) and Θ .
Revised: • Finetuning Case 2: ﬁnetune both the embedding parameters W (initialized from W ∗ ) and Θ .

Original: Sometimes updating the embedding parameters W in the ﬁnetune stage is necessary, as the embedding parameters from the pretrain model may not be good enough for the ﬁnetune task.
Revised: Sometimes updating the embedding parameters W in the ﬁnetuning stage is necessary, as the embedding parameters from the pretrained model may not be good enough for the ﬁnetuning task.

Original: This corresponds to the following formulation:
Revised: This corresponds to the following formulation:

Original: [Equation]
Revised: [Equation]

Original: 3. I NFLUENCE FUNCTION FOR MULTI - STAGE MODELS
Revised: 3. I NFLUENCE FUNCTION FOR MULTI - STAGE MODELS

Original: We derive the inﬂuence function for the multi-stage model to trace the inﬂuence of pretrain data on the ﬁnetune model.
Revised: We derive the inﬂuence function for the multi-stage model to trace the inﬂuence of pretraining data on the ﬁnetuned model.

Original: In Figure 1 we show the task we are interested in solving in this paper.
Revised: In Figure 1 we show the task we are interested in solving in this paper.

Original: Note that we use the same deﬁnition of inﬂuence function as (Koh & Liang, 2017) and discuss how to compute it in the multi-stage training scenario.
Revised: Note that we use the same deﬁnition of inﬂuence function as (Koh & Liang, 2017) and discuss how to compute it in the multi-stage training scenario.

Original: As discussed at the end of Section 3.1, depending on whether or not we are updating the shared parameters W in the ﬁnetune stage, we will derive the inﬂuence functions under two different scenarios.
Revised: As discussed at the end of Section 3.1, depending on whether or not we are updating the shared parameters W in the ﬁnetuning stage, we will derive the inﬂuence functions under two different scenarios.

Original: 3.2. C ASE 1: EMBEDDING PARAMETERS W ARE FIXED IN FINETUNE
Revised: 3.2. C ASE 1: EMBEDDING PARAMETERS W ARE FIXED IN FINETUNING

Original: To compute the inﬂuence of pretrain data on the ﬁnetune task, the main idea is to perturb one data example in the pretrain data, and study how that impacts the test data.
Revised: To compute the inﬂuence of pretraining data on the ﬁnetuning task, the main idea is to perturb one data example in the pretraining data, and study how that impacts the test data.

Original: Mathematically, if we perturb a pretrain data example z with loss change by a small (cid:15) , the perturbed model can be deﬁned as
Revised: Mathematically, if we perturb a pretraining data example z with loss change by a small (cid:15) , the perturbed model can be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Note that choices of (cid:15) can result in different effects in the loss function from the original solution in (1).
Revised: Note that choices of (cid:15) can result in different effects in the loss function from the original solution in (1).

Original: For instance, if we set (cid:15) = − 1 n , then we are removing the training data z in the pretrain dataset.
Revised: For instance, if we set (cid:15) = − 1 n , then we are removing the training data z in the pretraining dataset.

Original: For the ﬁnetune stage, since we consider Case 1 where the embedding parameters W are ﬁxed in the ﬁnetune stage, the new model for the end-task or ﬁnetune task will thus be
Revised: For the ﬁnetuning stage, since we consider Case 1 where the embedding parameters W are ﬁxed in the ﬁnetuning stage, the new model for the end-task or ﬁnetuning task will thus be

Original: [Equation]
Revised: [Equation]

Original: The inﬂuence function that measures the impact of a small (cid:15) perturbation on z to the ﬁnetune loss on a test sample x t from ﬁnetune task is deﬁned as
Revised: The inﬂuence function that measures the impact of a small (cid:15) perturbation on z to the ﬁnetuning loss on a test sample x t from ﬁnetuning task is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: with I z, Θ := ∂ ˆΘ (cid:15)
Revised: with I z, Θ := ∂ ˆΘ (cid:15)

Original: [Equation]
Revised: [Equation]

Original: I z,W [Equation] =
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where I z, Θ measures the inﬂuence of z on the ﬁnetune task parameters Θ , and I z,W measures how z inﬂuences the pretrain model W .
Revised: where I z, Θ measures the inﬂuence of z on the ﬁnetuning task parameters Θ , and I z,W measures how z inﬂuences the pretrained model W .

Original: Therefore we can split the inﬂuence of z on the test sample into two pieces: one is the impact of z on the pretrain model I z,W , and the other is the impact of z on the ﬁnetune model I z, Θ .
Revised: Therefore we can split the inﬂuence of z on the test sample into two pieces: one is the impact of z on the pretrained model I z,W , and the other is the impact of z on the ﬁnetuned model I z, Θ .

Original: It is worth mentioning that, due to linearity, if we want to estimate a set of test example inﬂuence function scores with respect to a set of pretrain examples, we can simply sum up the pair-wise inﬂuence functions, and so deﬁne
Revised: It is worth mentioning that, due to linearity, if we want to estimate a set of test example inﬂuence function scores with respect to a set of pretraining examples, we can simply sum up the pair-wise inﬂuence functions, and so deﬁne

Original: [Equation]
Revised: [Equation]

Original: ) where { z ( i ) } and { x ( j ) t } contain a set of pretrain training data and ﬁnetune test data.
Revised: where { z ( i ) } and { x ( j ) t } contain a set of pretraining data and ﬁnetuning test data.

Original: Next we will derive these two inﬂuence scores I z, Θ and I z,W (see the detailed derivations in the appendix) in Theorem 1 below.
Revised: Next we will derive these two inﬂuence scores I z, Θ and I z,W (see the detailed derivations in the appendix) in Theorembelow.

Original: Theorem 1.
Revised: Theorem 1.

Original: For the two-stage training procedure in (1) and (2) , we have
Revised: For the two-stage training procedure in (1) and (2) , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where [ · ] W means taking the W part of the vector.
Revised: where [ · ] W means taking the W part of the vector.

Original: By plugging (10) into (7), we ﬁnally obtain the inﬂuence score of pretrain data z on the ﬁnetune task testing point x t , I z,x t as
Revised: By plugging (10) into (7) [Equation] we ﬁnally obtain the inﬂuence score of pretraining data z on the ﬁnetuning task testing point x t , I z,x t as

Original: [Equation]
Revised: [Equation]

Original: Algorithm 1 shows how to compute the inﬂuence score in (11).
Revised: Algorithm 1 shows how to compute the inﬂuence score in (11).

Original: The pseudocode for computing the inﬂuence function in (11) is shown in Algorithm 1. 
Revised: The pseudocode for computing the inﬂuence function in (11) is shown in Algorithm 1. [Equation] [Equation] and

Original: 3.2. C ASE 2: EMBEDDING PARAMETER W IS ALSO UPDATED IN THE FINETUNE STAGE
Revised: [Equation] [Equation] [Equation] ASE 2: EMBEDDING PARAMETER W IS ALSO UPDATED IN THE FINETUNING STAGE

Original: For the second ﬁnetune stage case in (3), we will also further train the embedding parameter W from the pretrain stage.
Revised: For the second ﬁnetuning stage case in (3), we will also further train the embedding parameter W from the pretraining stage.

Original: When W is also updated in the ﬁnetune stage, it is challenging to characterize the inﬂuence since the pretrain embedding W ∗ is only used as an initialization.
Revised: When W is also updated in the ﬁnetuning stage, it is challenging to characterize the inﬂuence since the pretrained embedding W ∗ is only used as an initialization.

Original: In general, the ﬁnal model ( W ∗∗ , Θ ∗ ) may be totally unrelated to W ∗ ; for instance, when the objective function is strongly convex, any initialization of W in (3) will converge to the same solution.[Equation] [Equation] [Equation]
Revised: In general, the ﬁnal model ( W ∗∗ , Θ ∗ ) may be totally unrelated to W ∗ ; for instance, when the objective function is strongly convex, any initialization of W in (3) will converge to the same solution.

Original: [Equation]

Original: [Equation]

Original: and [Equation] [Equation] However, in practice the initialization of W will strongly inﬂuence the ﬁnetune stage in deep learning, since the ﬁnetune objective is usually highly nonconvex and initializing W with W ∗ will converge to a local minimum near W ∗ .
Revised: However, in practice the initialization of W will strongly inﬂuence the ﬁnetuning stage in deep learning, since the ﬁnetuning objective is usually highly non-convex and initializing W with W ∗ will converge to a local minimum near W ∗ .

Original: Therefore [Equation] we propose to approximate the whole training procedure as
Revised: Therefore, we propose to approximate the whole training procedure as

Original: [Equation]
Revised: [Equation]

Original: where ¯ W, ¯ U are optimal for the pretrain stage, W ∗ , Θ ∗ are optimal for the ﬁnetune stage, and 0 ≤ α (cid:28) 1 is a small value.
Revised: where ¯ W, ¯ U are optimal for the pretraining stage, W ∗ , Θ ∗ are optimal for the ﬁnetuning stage, and 0 ≤ α (cid:28) 1 is a small value.

Original: This is to characterize that in the ﬁnetune stage, we are targeting to ﬁnd a solution that minimizes F ( W, Θ) and is close to ¯ W .
Revised: This is to characterize that in the ﬁnetuning stage, we are targeting to ﬁnd a solution that minimizes F ( W, Θ) and is close to ¯ W .

Original: In this way, the pretrain parameters are connected with ﬁnetune task and thus inﬂuence of pretrain data to the ﬁnetune task can be tractable.
Revised: In this way, the pretrained parameters are connected with ﬁnetuning task and thus inﬂuence of pretraining data to the ﬁnetuning task can be tractable.

Original: The results in our experiments show that with this approximation, the computed inﬂuence score can still reﬂect the real inﬂuence quite well.
Revised: The results in our experiments show that with this approximation, the computed inﬂuence score can still reﬂect the real inﬂuence quite well.

Original: Similarly we can have ∂ ˆΘ (cid:15) ∂(cid:15) , ∂ ˆ W (cid:15) ∂(cid:15) , and ∂ ¯ W (cid:15) ∂(cid:15) to measure the difference between their original optimal solutions in (12) and the optimal solutions from (cid:15) perturbation over the pretrain data z .
Revised: Similarly we can have ∂ ˆΘ (cid:15) ∂(cid:15) , ∂ ˆ W (cid:15) ∂(cid:15) , and ∂ ¯ W (cid:15) ∂(cid:15) to measure the difference between their original optimal solutions in (12) and the optimal solutions from (cid:15) perturbation over the pretraining data z .

Original: Similar to (7), the inﬂuence function I z,x t that measures the inﬂuence of (cid:15) perturbation to pretrain data z on test sample x t ’s loss is
Revised: Similar to (7), the inﬂuence function I z,x t that measures the inﬂuence of (cid:15) perturbation to pretraining data z on test sample x t ’s loss is

Original: [Equation]
Revised: [Equation]

Original: The inﬂuence function of small perturbation of G ( W, U ) to ¯ W, W ∗ , Θ ∗ can be computed following the same approach in Subsection 3.2.1 by replacing ¯ W for W ∗ and [Θ ∗ , W ∗ ] for Θ ∗ in (10).
Revised: The inﬂuence function of small perturbation of G ( W, U ) to ¯ W, W ∗ , Θ ∗ can be computed following the same approach in Subsection 3.2.1 by replacing ¯ W for W ∗ and [Θ ∗ , W ∗ ] for Θ ∗ in (10).

Original: This will lead to
Revised: This will lead to

Original: [Equation]
Revised: [Equation]

Original: After plugging (14) into (13)
Revised: U ) )

Original: [Equation]
Revised: (cid:21)

Original:  we will have the inﬂuence function I z,x t .
Revised: W After plugging (14) into (13), we will have the inﬂuence function I z,x t .

Original: Similarly, the algorithm for computing I z,x t for Case 2 can follow Algorithm 1 for Case 1 by replacing gradient computation [Equation]
Revised: Similarly, the algorithm for computing I z,x t for Case 2 can follow Algorithm 1 for Case 1 by replacing gradient computation.

Original: 3. I MPLEMENTATION DETAILS
Revised: 3. I MPLEMENTATION DETAILS

Original: The inﬂuence function computation for multi-stage model is presented in the previous section.
Revised: The inﬂuence function computation for multi-stage model is presented in the previous section.

Original: As we can see in Algorithm 1 that the inﬂuence score computation involves many Hessian matrix operations, which will be very expensive and sometimes unstable for large-scale models.
Revised: As we can see in Algorithm 1 that the inﬂuence score computation involves many Hessian matrix operations, which will be very expensive and sometimes unstable for large-scale models.

Original: We used several strategies to speed up the computation and make the scores more stable.
Revised: We used several strategies to speed up the computation and make the scores more stable.

Original: Large Hessian Matrices
Revised: Large Hessian Matrices

Original: A Hessian matrix H has a size of p × p , where p is the number of parameters in the model.
Revised: A Hessian matrix H has a size of p × p , where p is the number of parameters in the model.

Original: For large deep learning models with thousands or even millions of parameters, it is almost impossible to ﬁt a p × p Hessian into memory.
Revised: For large deep learning models with thousands or even millions of parameters, it is almost impossible to ﬁt a p × p Hessian into memory.

Original: Also, to invert a Hessian requires O ( p 3 ) operations.
Revised: Also, to invert a Hessian requires O ( p 3 ) operations.

Original: Similar to Koh & Liang (2017), we avoid explicitly computing and storing the Hessian matrix and its inverse, and instead compute product of the inverse Hessian with a vector directly.
Revised: Similar to Koh & Liang (2017), we avoid explicitly computing and storing the Hessian matrix and its inverse, and instead compute product of the inverse Hessian with a vector directly.

Original: Every time we need an inverse Hessian vector product v = H − 1 b , we invoke conjugate gradients (CG), which transforms the linear system problem into an quadratic optimization problem
Revised: Every time we need an inverse Hessian vector product v = H − 1 b , we invoke conjugate gradients (CG), which transforms the linear system problem into an quadratic optimization problem

Original: [Equation]
Revised: H − 1 b ≡ arg min x { 12 x T Hx − b T x } .

Original: In each iteration of CG, instead of computing H − 1 b directly, we will compute a Hessian vector product, which can be efﬁciently done by backprop through the model twice with O ( p ) time complexity.
Revised: In each iteration of CG, instead of computing H − 1 b directly, we will compute a Hessian vector product, which can be efﬁciently done by backprop through the model twice with O ( p ) time complexity.

Original: The aforementioned conjugate gradient method requires the Hessian matrix to be positive deﬁnite.
Revised: The aforementioned conjugate gradient method requires the Hessian matrix to be positive deﬁnite.

Original: However, in practice the Hessian may have negative eigenvalues, since we run a SGD and the ﬁnal H may not at a local minimum exactly.
Revised: However, in practice the Hessian may have negative eigenvalues, since we run a SGD and the ﬁnal H may not at a local minimum exactly.

Original: To tackle this issue, we solve arg min x
Revised: To tackle this issue, we solve [Equation]

Original: { 12 x T H 2 x − b T Hx } , whose solution can be shown the same as (15) since the Hessian matrix is symmetric.
Revised: (15) whose solution can be shown the same as arg min x { 12 x T Hx − b T x } since the Hessian matrix is symmetric.

Original: H 2 is guaranteed to be positive deﬁnite as long as H is not singular.
Revised: H 2 is guaranteed to be positive deﬁnite as long as H is invertible, even when H has negative eigenvalues.

Revised: If H 2 is not ill-conditioned, we can solve (15) directly.

Original: The rate of convergence of the conjugate gradient depends on
Revised: The rate of convergence of CG depends on

Original: [Equation]
Revised: [Equation]

Original: κ ( H 2 )+1 , where κ ( H 2 ) is the condition number of H 2 , which can be very large if H  is ill-conditioned.
Revised: 2 )+1 , where κ ( H 2 ) is the condition number of H 2 , which can be very large if H 2 is ill-conditioned.

Original: To encourage faster convergence, we add a small damping term λ on the diagonal and solve arg min x { 12 x T ( H 2 + λI ) x − b T Hx } .
Revised: When H 2 is ill-conditioned, to stabilize the solution and to encourage faster convergence, we add a small damping term λ on the diagonal and solve arg min x { 12 x T ( H 2 + λI ) x − b T Hx } .

Original: Time Complexity As mentioned above, we can get an inverse Hessian vector product in O ( p ) time.
Revised: Time Complexity As mentioned above, we can get an inverse Hessian vector product in O ( p ) time.

Revised: We assume there are p 1 parameters in our pretrained model and p 2 parameters in our ﬁnetuned model.

Original: Since F and G are summation of loss with respect to all pretrain or ﬁnetune examples, it takes O ( mp  ) or O ( np  ) to compute  Hessian vector product, where m is the number of pretrain examples and n is the number of ﬁnetune examples.
Revised: Since F and G are summation of loss with respect to all pretraining or ﬁnetuning examples, it takes O ( mp 1 ) or O ( np 2 ) to compute a Hessian vector product, where m is the number of pretraining examples and n is the number of ﬁnetuning examples.

Original: We may  subsample the pretrain examples to estimate G ( W, U ) when the number of pretrain examples is gigantic such as pretrain using One-Billion-word dataset (Chelba et al., 2013) for ELMo etc.
Revised: We may also subsample the pretraining examples to estimate G ( W, U ) when the number of pretraining examples is gigantic such as pretraining using One-Billion-word dataset (Chelba et al., 2013) for ELMo etc.

Original: For the  inverse Hessian vector product, the time complexity is O ( npr ) , where r is the number of iterations in CG. 
Revised: For the two inverse Hessian vector products, the time complexity is O ( np 2 r ) and O ( mp 1 r ) , where r is the number of iterations in CG. For other operations in computing the inﬂuence score, vector product has a time complexity of O ( p 1 ) or O ( p 1 ) , and computing the gradients of all pretraining examples has a complexity of O ( mp 1 ) . So computing the total time complexity of computing a multi-stage inﬂuence score is O (( mp 1 + np 2 ) r )

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we will conduct experiment with real datasets on both vision and NLP tasks to show the effectiveness of our proposed inﬂuence function.
Revised: In this section, we will conduct experiment with real datasets on both vision and NLP tasks to show the effectiveness of our proposed inﬂuence function.

Original: We will show the results in the main text on the vision tasks, and some qualitative results on NLP task related to ELMo are presented in Section C in
Revised: We will show the results in the main text on the vision tasks, and some qualitative results on NLP task related to ELMo are presented in Section C in

Original: Appendix.
Revised: Appendix.

Original: 4. I NFLUENCE FUNCTION CORRELATION WITH REAL SCORE
Revised: 4. I NFLUENCE FUNCTION CORRELATION WITH REAL SCORE

Original: To show that our proposed inﬂuence score are a good approximation, we evaluate our proposed multi-stage inﬂuence function on two CNN models with CIFAR and MNIST datasets.
Revised: To show that our proposed inﬂuence score are a good approximation, we evaluate our proposed multi-stage inﬂuence function on two CNN models with CIFAR and MNIST datasets.

Original: The model structures are shown in Table A in Appendix.
Revised: The model structures are shown in Table A in Appendix.

Original: We use Tanh for all activations.
Revised: We use Tanh for all activations.

Original: For both MNIST and
Revised: For both MNIST and

Original: CIFAR models, CNN layers are used as embeddings and fully connected layers are task-speciﬁc.
Revised: CIFAR models, CNN layers are used as embeddings and fully connected layers are task-speciﬁc.

Original: At the pretrain stage, we train the models with examples from only two classes (“bird" vs. “frog") for CIFAR and four classes (0, 1, 2, and 3) for MNIST.
Revised: At the pretraining stage, we train the models with examples from only two classes (“bird" vs. “frog") for CIFAR and four classes (0, 1, 2, and 3) for MNIST.

Original: The resulting embedding is used in the ﬁnetune tasks, where we ﬁnetune the model with the examples from the remaining eight classes in CIFAR or the other 6 numbers in MNIST task for classiﬁcation.
Revised: The resulting embedding is used in the ﬁnetuning tasks, where we ﬁnetune the model with the examples from the remaining eight classes in CIFAR or the other 6 numbers in MNIST task for classiﬁcation.

Original: In order to make the experiments closer to typical real ﬁnetune situations, we reduce the size of the training set in the ﬁnetune task by subsampling.
Revised: In order to make the experiments closer to typical real ﬁnetuning situations, we reduce the size of the training set in the ﬁnetuning task by subsampling.

Original: In this experiment we test the correlation between individual pretrain example’s multi-stage inﬂuence function and the real loss difference when the pretrain examples are removed.
Revised: In this experiment we test the correlation between individual pretraining example’s multi-stage inﬂuence function and the real loss difference when the pretraining examples are removed.

Original: We test two cases (as mentioned in Section 3.1) – where the pretrain embedding is ﬁxed, and where it is updated during ﬁnetune.
Revised: We test two cases (as mentioned in Section 3.1) – where the pretrained embedding is ﬁxed, and where it is updated during ﬁnetuning.

Original: For both MNIST and CIFAR, we ﬁrst train the embedding with the binary classiﬁcation for T pretrain steps.
Revised: For both MNIST and CIFAR, we ﬁrst train the embedding with the binary classiﬁcation for T pretrain steps.

Original: The embedding is then used in the ﬁnetune model.
Revised: The embedding is then used in the ﬁnetuned model.

Original: Then we train the ﬁnetune model for T ﬁnetune steps.
Revised: Then we train the ﬁnetuned model for T ﬁnetune steps.

Original: Depending on different scenarios which will be explained below, the embedding may be ﬁxed or updated in the ﬁnetune task training.
Revised: Depending on different scenarios which will be explained below, the embedding may be ﬁxed or updated in the ﬁnetuning task training.

Original: For a given example in the pretrain data, we calculate its inﬂuence function score with respect to each test example in the ﬁnetune task test set using the method presented in Section 3.
Revised: For a given example in the pretraining data, we calculate its inﬂuence function score with respect to each test example in the ﬁnetuning task test set using the method presented in Section 3.

Original: To evaluate this pretrain example’s contribution to the overall performance of the model, we sum up the inﬂuence function scores across the whole test set in the ﬁnetune task.
Revised: To evaluate this pretraining example’s contribution to the overall performance of the model, we sum up the inﬂuence function scores across the whole test set in the ﬁnetuning task.

Original: To validate the score, we remove that pretrain example and go through the aforementioned process again by updating the model.
Revised: To validate the score, we remove that pretraining example and go through the aforementioned process again by updating the model.

Original: In the updating process we further train the pretrain and ﬁnetune models for T (cid:48) pretrain and T (cid:48) ﬁnetune steps from the original model checkpoints.
Revised: In the updating process we further train the pretrained and ﬁnetuned models for T (cid:48) pretrain and T (cid:48) ﬁnetune steps from the original model checkpoints.

Original: Note that in this process the pretrain is conducted with the new leave-one-out pretrain set, while the training set for the ﬁnetune task is intact.
Revised: Note that in this process the pretraining is conducted with the new leave-one-out pretraining training set, while the training set for the ﬁnetuning task is intact.

Original: Due to computation constraints, we only use the top 100 pretrain examples with the largest inﬂuence function absolute values in this experiments to get 100 score-difference pairs.
Revised: Due to computation constraints, we only use the top 100 pretraining examples with the largest inﬂuence function absolute values in this experiments to get 100 scoredifference pairs.

Original: Then we run a linear regression between the true loss difference values obtained and the inﬂuence score computed to show their correlation.
Revised: Then we run a linear regression between the true loss difference values obtained and the inﬂuence score computed to show their correlation.

Original: The detailed hyperparameters used in these experiments are presented in Appendix B.
Revised: The detailed hyperparameters used in these experiments are presented in Appendix B.

Original: 4.1. E MBEDDING IS FIXED
Revised: 4.1. E MBEDDING IS FIXED

Original: In Figure 2 we show the correlation results of CIFAR and MNIST models when the embedding is ﬁxed in ﬁnetune task training.
Revised: In Figure 2 we show the correlation results of CIFAR and MNIST models when the embedding is ﬁxed in ﬁnetuning task training.

Original: Though we make many approximations in our formulation, from
Revised: Though we make many approximations in our formulation, from

Original: Figures 2a and 2b we can see that there is a clear linear correlation between the true loss difference and the inﬂuence function scores obtained.
Revised: Figures 2a and 2b we can see that there is a clear linear correlation between the true loss difference and the inﬂuence function scores obtained.

Original: The correlation is evaluated with Pearson’s r value.
Revised: The correlation is evaluated with Pearson’s r value.

Original: This supports our argument that we can use this score to detect the examples in the pretrain set which contributes most to the model’s performance.
Revised: This supports our argument that we can use this score to detect the examples in the pretraining set which contributes most to the model’s performance.

Original: In Figure 3 we demonstrate the misclassiﬁed test
Revised: In Figure 3 we demonstrate the misclassiﬁed test

Original: [Figure]
Revised: [Figure]

Original: images in the ﬁnetune task and the images with the largest positive inﬂuence function score in the pretrain dataset.
Revised: images in the ﬁnetuning task and the images with the largest positive inﬂuence function score in the pretraining dataset.

Original: Examples with large positive inﬂuence score are expected to have negative effect on the model’s performance since intuitively when they are added to the pretrain dataset, the loss of the test example will increase.
Revised: Examples with large positive inﬂuence score are expected to have negative effect on the model’s performance since intuitively when they are added to the pretraining dataset, the loss of the test example will increase.

Original: From Figure 3 we can indeed see that the identiﬁed examples are with low quality, and they can be easily misclassiﬁed even with human eyes.
Revised: From Figure 3 we can indeed see that the identiﬁed examples are with low quality, and they can be easily misclassiﬁed even with human eyes.

Original: test example pretrain example test example pretrain

Original: [Figure]

Original: One may doubt the effectiveness of the expensive inverse Hessian computation in our formulation.
Revised: One may doubt the effectiveness of the expensive inverse Hessian computation in our formulation.

Original: As a comparison, we replace all inverse Hessians in (11) with identity matrices to compute the inﬂuence function score for the MNIST model.
Revised: As a comparison, we replace all inverse Hessians in (11) with identity matrices to compute the inﬂuence function score for the MNIST model.

Original: The results are shown in Figure 4 with a much smaller
Revised: The results are shown in Figure 4 with a much smaller

Original: Pearson’s r of 0.17.
Revised: Pearson’s r of 0.17.

Original: This again shows effectiveness of our proposed inﬂuence function.[Figure] [Figure]
Revised: This again shows effectiveness of our proposed inﬂuence function.

Original:  E MBEDDING IS UPDATED IN FINETUNE
Revised: 4.1. E MBEDDING IS UPDATED IN FINETUNE

Original: Practically, the embedding can also be updated in the ﬁnetune process.
Revised: Practically, the embedding can also be updated in the ﬁnetuning process.

Original: In Figure 5 we show the correlation between true loss difference and inﬂuence function score values using (13).
Revised: In Figure 5 we show the correlation between true loss difference and inﬂuence function score values using (13).

Original: We can see that even under this challenging condition our multi-stage inﬂuence function from (13) still has a strong correlation with the true loss difference, with a Pearson’s r = 0 .
Revised: We can see that even under this challenging condition, our multi-stage inﬂuence function from (13) still has a strong correlation with the true loss difference, with a Pearson’s r = 0 .

Original: 4.  F INETUNE T ASK  S IMILARITY TO THE P RETRAIN T ASK
Revised: 4. T HE F INETUNING T ASK ’ S S IMILARITY TO THE P RETRAINING T ASK

Original: In this experiment we explore the relationship between inﬂuence function score and ﬁnetune task similarity with the pretrain task.
Revised: In this experiment, we explore the relationship between inﬂuence function score and ﬁnetuning task similarity with the pretraining task.

Original: Speciﬁcally, we study whether the inﬂuence function score will increase in absolute value if the ﬁnetune task is very similar to the pretrain task.
Revised: Speciﬁcally, we study whether the inﬂuence function score will increase in absolute value if the ﬁnetuning task is very similar to the pretraining task.

Original: To do this, we use the CIFAR embedding obtained from a “bird vs. frog" classiﬁcation and test its inﬂuence function core on two ﬁnetune tasks.
Revised: To do this, we use the CIFAR embedding obtained from a “bird vs. frog" classiﬁcation and test its inﬂuence function scores on two ﬁnetuning tasks.

Original: The ﬁnetune task A is exactly the same as the pretrain “bird vs. frog" classiﬁcation, while the ﬁnetune task B is a classiﬁcation on two other classes (“automobile vs. deer").
Revised: The ﬁnetuning task A is exactly the same as the pretraining “bird vs. frog" classiﬁcation, while the ﬁnetuning task B is a classiﬁcation on two other classes (“automobile vs. deer").

Original: All hyperparameters used in the two ﬁnetune tasks are the same.
Revised: All hyperparameters used in the two ﬁnetuning tasks are the same.

Original: In Figure 6, for both tasks we plot the distribution of the inﬂuence function values with respect to each pretrain example.
Revised: In Figure 6, for both tasks we plot the distribution of the inﬂuence function values with respect to each pretraining example.

Original: We sum up the inﬂuence score for all test examples.
Revised: We sum up the inﬂuence score for all test examples.

Original: We can see that, the ﬁrst ﬁnetune task inﬂuence function has much larger absolute values than that of the second task.
Revised: We can see that, the ﬁrst ﬁnetuning task inﬂuence function has much larger absolute values than that of the second task.

Original: The average absolute value of task A inﬂuence function score is 0.055, much larger than that of task B, which is 0.025.
Revised: The average absolute value of task A inﬂuence function score is 0.055, much larger than that of task B, which is 0.025.

Original: This supports the argument that if pretrain task and ﬁnetune task are similar, the pretrain data will have larger inﬂuence on the ﬁnetune task performance.
Revised: This supports the argument that if pretraining task and ﬁnetuning task are similar, the pretraining data will have larger inﬂuence on the ﬁnetuning task performance.

Original: 4. I NFLUENCE F UNCTION S CORE WITH D IFFERENT N UMBER OF F INETUNE E XAMPLES
Revised: 4. I NFLUENCE F UNCTION S CORE WITH D IFFERENT N UMBER OF F INETUNE E XAMPLES

Original: We also study the relationship between the inﬂuence function scores and number of examples used in ﬁnetune.
Revised: We also study the relationship between the inﬂuence function scores and number of examples used in ﬁnetuning.

Original: In this experiment, we update the pretrain embedding in ﬁnetune stage.
Revised: In this experiment, we update the pretrained embedding in ﬁnetuning stage.

Original: We use the same pretrain and ﬁnetune task as in Section 4.1.
Revised: We use the same pretraining and ﬁnetuning task as in Section 4.1.

Original: In Figure 7, model C is the model used in Section 4.1.
Revised: In Figure 7, model C is the model used in test example [Figure] [Figure] [Figure]

Original:  while in model
Revised: [Figure] The average absolute values of inﬂuence function scores for models A and B are 0.055 and 0.025, respectively. [Figure] Section 4.1.2 while in model

Original: D
Revised: D

Original: we triple the number of ﬁnetune examples as well as the number of ﬁnetune steps.
Revised: we triple the number of ﬁnetuning examples as well as the number of ﬁnetuning steps.

Original: Figure 7 demonstrates the distribution of each pretrain examples’ inﬂuence function score with the whole test set.
Revised: Figure 7 demonstrates the distribution of each pretraining examples’ inﬂuence function score with the whole test set.

Original: The average absolute value of inﬂuence function score in model D is 0.15, much less than that of model C.
Revised: The average absolute value of inﬂuence function score in model D is 0.15, much less than that of model C.

Original: This indicates that with more ﬁnetune examples and more ﬁnetune steps, the inﬂuence of pretrain data to the ﬁnetune model’s performance will decrease.
Revised: This indicates that with more ﬁnetuning examples and more ﬁnetuning steps, the inﬂuence of pretraining data to the ﬁnetuning model’s performance will decrease.

Original: This makes sense as if the ﬁnetune data does not have sufﬁcient information for training a good ﬁnetune task, then pretrain data will have more impact on the ﬁnetune task.
Revised: This makes sense as if the ﬁnetuning data does not have sufﬁcient information for training a good ﬁnetuning task, then pretraining data will have more impact on the ﬁnetuning task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce a multi-stage inﬂuence function to evaluate pretrain examples’ contribution to ﬁnetune model’s prediction.
Revised: We introduce a multi-stage inﬂuence function to evaluate pretraining examples’ contribution to ﬁnetuned model’s prediction.

Original: Two different cases are studied: the pretrain embedding is ﬁxed in ﬁnetune or the pretrain embedding is updated in ﬁnetune.
Revised: Two different cases are studied: the pretrained embedding is ﬁxed in ﬁnetuning or the pretrained embedding is updated in ﬁnetuning.

Original: We test our method on both CV and NLP tasks.
Revised: We test our method on both CV and NLP tasks.

Original: Our experimental results show strong correlation between the proposed multi-stage inﬂuence function scores and the true loss difference when an example is removed from the pretrain data.
Revised: Our experimental results show strong correlation between the proposed multistage inﬂuence function scores and the true loss difference when an example is removed from the pretraining data.

Original: We believe this is a promising way to connect ﬁnetune model’s performance with pretrain data directly.[Figure] [Figure]
Revised: We believe this is a promising way to connect ﬁnetuned model’s performance with pretraining data directly.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Since ˆΘ (cid:15) , ˆ U (cid:15) , ˆ W (cid:15) are optimal solutions, and thus satisfy the following optimality conditions:
Revised: Since ˆΘ (cid:15) , ˆ U (cid:15) , ˆ W (cid:15) are optimal solutions, and thus satisfy the following optimality conditions:

Original: [Equation]
Revised: [Equation]

Original: where ∂ ( W, U ) means concatenate the U and W as [ W, U ] and compute the gradient w.r.t [ W, U ] .
Revised: where ∂ ( W, U ) means concatenate the U and W as [ W, U ] and compute the gradient w.r.t [ W, U ] .

Original: We deﬁne the changes of parameters as ∆ W (cid:15) =
Revised: We deﬁne the changes of parameters as ∆ W (cid:15) =

Original: ˆ W (cid:15) − ˆ W , ∆Θ
Revised: ˆ W (cid:15) − ˆ W , ∆Θ

Original: [Equation]
Revised: [Equation]

Original: = ˆΘ (cid:15) − ˆΘ , and ∆ U (cid:15) =
Revised: = ˆΘ (cid:15) − ˆΘ , and ∆ U (cid:15) =

Original: ˆ U (cid:15) − ˆ U .
Revised: ˆ U (cid:15) − ˆ U .

Original: Applying Taylor expansion to the rhs of (17) we get 

Revised: Applying Taylor expansion to the rhs of (17) we get

Original: ≈
Revised: ≈

Original: [Equation]
Revised: [Equation]

Original: Since W ∗ , U ∗ are optimal of unperturbed problem, ∂
Revised: Since W ∗ , U ∗ are optimal of unperturbed problem, ∂

Original: ∂ ( W,U )
Revised: ∂ ( W,U )

Original: G ( W ∗ , U ∗ )
Revised: G ( W ∗ , U ∗ )

Original: = 0 , and we have
Revised: = 0 , and we have

Original: [Equation]
Revised: [Equation]

Original: Since (cid:15) →
Revised: Since (cid:15) →

Original: [Equation]
Revised: [Equation]

Original: we have further approximation
Revised: we have further approximation

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similarly, based on (16) and applying ﬁrst order Taylor expansion to its rhs we have
Revised: Similarly, based on (16) and applying ﬁrst order Taylor expansion to its rhs we have

Original: [Equation]
Revised: [Equation]

Original: Combining (21) with (20) we have
Revised: Combining (21) with (20) we have

Original: [Equation]
Revised: [Equation]

Original: where [ · ] W means taking the W part of the vector.
Revised: where [ · ] W means taking the W part of the vector.

Original: Therefore,
Revised: Therefore,

Original: I
Revised: I

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: B M ODELS AND H YPERPARAMETERS FOR THE E XPERIMENTS IN S ECTIONS 4.1, 4.2 AND 4. 
Revised: B M ODELS AND H YPERPARAMETERS FOR THE E XPERIMENTS IN S ECTIONS 4.1, 4.2 AND 4.

Original: The model structures we used in Sections 4.1, 4.2 and 4.3 are listed in Table A.
Revised: The model structures we used in Sections 4.1, 4.2 and 4.3 are listed in Table A.

Original: As mentioned in the main text, for all models, CNN layers are used as embeddings and fully connected layers are task-speciﬁc.
Revised: As mentioned in the main text, for all models, CNN layers are used as embeddings and fully connected layers are task-speciﬁc.

Original: The number of neurons on the last fully connected layer is determined by the number of classes in the classiﬁcation.
Revised: The number of neurons on the last fully connected layer is determined by the number of classes in the classiﬁcation.

Original: There is no activation at the ﬁnal output layer and all other activations are Tanh.
Revised: There is no activation at the ﬁnal output layer and all other activations are Tanh.

Original: • For MNIST experiments in Section 4.1.1, we train a four-class classiﬁcation (0, 1, 2, and 3) in pretrain.
Revised: • For MNIST experiments in Section 4.1.1, we train a four-class classiﬁcation (0, 1, 2, and 3) in pretraining.

Original: All examples in the original MNIST training set with with these four labels are used in pretrain.
Revised: All examples in the original MNIST training set with with these four labels are used in pretraining.

Original: The ﬁnetune task is to classify the rest six classes, andwe subsample only
Revised: The ﬁnetuning task is to classify the rest six classes, and

Original:  5000 examples to ﬁnetune.
Revised: we subsample only 5000 examples to ﬁnetune.

Original: The pretrained embedding is ﬁxed in ﬁnetune.
Revised: The pretrained embedding is ﬁxed in ﬁnetuning.

Original: We run Adam optimizer to train both pretrain and ﬁne tune model with a batch size of 512.
Revised: We run Adam optimizer in both pretraining and ﬁnetuning with a batch size of 512.

Original: The pretrain and ﬁnetune models are trained to converge.
Revised: The pretrained and ﬁnetuned models are trained to converge.

Original: When validating the inﬂuence function score, we remove an example from pretrain dataset.
Revised: When validating the inﬂuence function score, we remove an example from pretraining dataset.

Original: Then we re-run the pretrain and ﬁnetune process with this leave-one-out pretrain dataset starting from the original models’ weights.
Revised: Then we re-run the pretraining and ﬁnetuning process with this leave-one-out pretraining dataset starting from the original models’ weights.

Original: In this process, we only run 100 steps for pretrain and ﬁnetune as the models converge.
Revised: In this process, we only run 100 steps for pretraining and ﬁnetuning as the models converge.

Original: When computing the inﬂuence function scores, the damping term for the pretrain and ﬁnetune model’s Hessians are 1 × 10 − 2 and 1 × 10 − 8 , respectively.
Revised: When computing the inﬂuence function scores, the damping term for the pretrained and ﬁnetuned model’s Hessians are 1 × 10 − 2 and 1 × 10 − 8 , respectively.

Original: We sample 1000 pretrain examples when computing the pretrain model’s Hessian summation.
Revised: We sample 1000 pretraining examples when computing the pretraind model’s Hessian summation.

Original: • For CIFAR experiments in Section 4.1.1, we train a two-class classiﬁcation (“bird" vs “frog") in pretrain.
Revised: • For CIFAR experiments in Section 4.1.1, we train a two-class classiﬁcation (“bird" vs “frog") in pretraining.

Original: All examples in the original CIFAR training set with with these four labels are used in pretrain.
Revised: All examples in the original CIFAR training set with with these four labels are used in pretraining.

Original: The ﬁnetune task is to classify the rest eight classes, and we subsample only 5000 examples to ﬁnetune.
Revised: The ﬁnetuning task is to classify the rest eight classes, and we subsample only 5000 examples to ﬁnetune.

Original: The pretrained embedding is ﬁxed in ﬁnetune.
Revised: The pretrained embedding is ﬁxed in ﬁnetuning.

Original: We run Adam optimizer to train both pretrain and ﬁne tune model with a batch size of 128.
Revised: We run Adam optimizer to train both pretrained and ﬁnetuned model with a batch size of 128.

Original: The pretrain and ﬁnetune models are trained to converge.
Revised: The pretrained and ﬁnetuned models are trained to converge.

Original: When validating the inﬂuence function score, we remove an example from pretrain dataset.
Revised: When validating the inﬂuence function score, we remove an example from pretraining dataset.

Original: Then we re-run the pretrain and ﬁnetune process with this leave-one-out pretrain dataset starting from the original models’ weights.
Revised: Then we re-run the pretraining and ﬁnetuning process with this leave-one-out pretraining dataset starting from the original models’ weights.

Original: In this process, we only run 6000 steps for pretrain and 3000 steps for ﬁnetune.
Revised: In this process, we only run 6000 steps for pretraining and 3000 steps for ﬁnetuning.

Original: When computing the inﬂuence function scores, the damping term for the pretrain and ﬁnetune model’s Hessians are 1 × 10 − 8 and 1 × 10 − 6 , respectively.
Revised: When computing the inﬂuence function scores, the damping term for the pretrained and ﬁnetuned model’s Hessians are 1 × 10 − 8 and 1 × 10 − 6 , respectively.

Original: Same hyperparameters are used in experiments in Sections 4.2 and 4.3.
Revised: Same hyperparameters are used in experiments in Sections 4.2 and 4.3.

Original: We also use these hyperparameters in Section 4.1.2’s experiments , except that the pretrained embedding is updated in ﬁnetune and the number of ﬁnetune steps is reduced to 1000 in validation.
Revised: We also use these hyperparameters in Section 4.1.2’s experiments, except that the pretrained embedding is updated in ﬁnetuning and the number of ﬁnetuning steps is reduced to 1000 in validation.

Original: The α constant in Equation 14 is chosen as 0.01.
Revised: The α constant in Equation 14 is chosen as 0.01.

Original: We sample 1000 pretrain examples when computing the pretrain model’s Hessian summation.
Revised: We sample 1000 pretraining examples when computing the pretrained model’s Hessian summation.

Original: [Table]
Revised: [Table]

Original: C ELM O E XPERIMENT
Revised: C ELM O E XPERIMENT

Original: In this section we show inﬂuence function score results with ELMo.
Revised: In this section we show inﬂuence function score results with ELMo.

Original: The ﬁnetune task is a binary sentiment classiﬁcation of twitter 1 and the ELMo model is pretrained on a one-billion-word dataset Chelba et al.
Revised: The ﬁnetune task is a binary sentiment classiﬁcation of twitter 1 and the ELMo model is pretrained on a one-billion-word dataset Chelba et al.

Original: For the ﬁnetune model, we add a hidden layer with 64 neurons and an output layer to build the classiﬁer.
Revised: For the ﬁnetuned model, we add a hidden layer with 64 neurons and an output layer to build the classiﬁer.

Original: The activation function is Tanh.
Revised: The activation function is Tanh.

Original: For simplicity, we use the original pretrained ELMo embedding and the embedding is ﬁxed in ﬁnetune.
Revised: For simplicity, we use the original pretrained ELMo embedding and the embedding is ﬁxed in ﬁnetuning.

Original: We randomly sample a subset of 1000 sentences from one-billion-word dataset.
Revised: We randomly sample a subset of 1000 sentences from one-billion-word dataset.

Original: For a test sentence, we list the pretrain sentences with the  and  smallest 
Revised: For a test sentence, we list the pretrain sentences with the largest and the smallest absolute inﬂuence function score values in one-billion-word dataset.

Original: [Table]
Revised: [Table]