Original: Improving Language Model Fine-tuning with Information Gain Filtration ∗Anonymous ACL submission
Revised: Improving Language Model Fine-tuning with Information Gain Filtration ∗

Original: Abstract
Revised: Abstract

Original: Language model fine-tuning is essential formodern natural language processing.
Revised: Language model fine-tuning is essential for modern natural language processing.

Original: The effectiveness of fine-tuning is limited by the inclusion of training examples that negativelyaffect performance.
Revised: The effectiveness of fine-tuning is limited by the inclusion of training examples that negatively affect performance.

Original: Here we present Information Gain Filtration , a general fine-tuningmethod, for improving the overall final performance of a fine-tuned model.
Revised: Here we present Information Gain Filtration , a general fine-tuning method, for improving the overall final performance of a fine-tuned model.

Original: We define
Revised: We define

Original: Information Gain of an example as the improvement on a validation metric after training on that example.
Revised: Information Gain of an example as the improvement on a validation metric after training on that example.

Original: A secondary learner isthen trained to approximate this quantity.
Revised: A secondary learner is then trained to approximate this quantity.

Original: During fine-tuning, this learner filters informative examples from uninformative ones.
Revised: During fine-tuning, this learner filters informative examples from uninformative ones.

Original: Weshow that our method is robust and has consistent improvement across datasets, fine-tuningtasks, and language model architectures.
Revised: We show that our method is robust and has consistent improvement across datasets, fine-tuning tasks, and language model architectures.

Original:  Introduction
Revised:  Introduction

Original: Language modeling is the task where a model predicts the conditional probability of the next token based on the context of previously observedtokens.
Revised: Language modeling is the task where a model predicts the conditional probability of the next token based on the context of previously observed tokens.

Original: Recent advances in transformers-basedmodels (Vaswani et al., 2017), lead to languagemodeling success as a pre-training objective forself-supervised representation learning.
Revised: Recent advances in transformers-based models (Vaswani et al., 2017), lead to language modeling success as a pre-training objective for self-supervised representation learning.

Original: Once pretrained, language models (LMs) can be updatedfor downstream tasks through fine-tuning (Devlinet al., 2019;
Revised: Once pretrained, language models (LMs) can be updated for downstream tasks through fine-tuning (Devlin et al., 2019;

Original: Radford et al., 2019).
Revised: Radford et al., 2019).

Original: Hence, improving fine-tuning leads to higher quality models.
Revised: Hence, improving fine-tuning leads to higher quality models.

Original: Several methods have been proposed to improve
Revised: Several methods have been proposed to improve

Original: LM fine-tuning performance.
Revised: LM fine-tuning performance.

Original: These include regularization techniques Lee et al.
Revised: These include regularization techniques Lee et al.

Original: (2020), supplementary training on supervised tasks Phang et al.(2018), incorporating out of domain data Mooreand Lewis (2010), and using features from intermediate transformer layers Tenney et al.
Revised: , supplementary training on supervised tasks Phang et al. (2018), incorporating out of domain data Moore and Lewis (2010), and using features from intermediate transformer layers Tenney et al.

Original: Liu et al.
Revised: Liu et al.

Original: In addition, its instability of this process has been investigated with rela- tion to insufficiently general training sets (Mos- bach et al., 2021), and to optimization techniques (Zhang et al., 2021).
Revised: In addition, its instability of this process has been investigated with relation to insufficiently general training sets (Mos- bach et al., 2021), and to optimization techniques (Zhang et al., 2021).

Original: Recently, Dodge et al.
Revised: Recently, Dodge et al.

Original: (2020) showed that the fine-tuning process has high variability between runs being sensitive to data ordering.
Revised: (2020) showed that the fine-tuning process has high variability between runs being sensitive to data ordering.

Original: They reduce this variability by fine-tuning models using many random seeds and keeping the best.
Revised: They reduce this variability by fine-tuning models using many random seeds and keeping the best.

Original: While this improves performance, the reasons for the high variability between random seeds are unknown.
Revised: While this improves performance, the reasons for the high variability between random seeds are unknown.

Original: In this work, we propose a novel approach to improving the effectiveness of fine-tuning by carefully selecting “informative” samples.
Revised: In this work, we propose a novel approach to improving the effectiveness of fine-tuning by carefully selecting “informative” samples.

Original: Our approach uses a secondary learner to estimate the usefulness of each example, and then selects only informative examples for fine-tuning.
Revised: Our approach uses a secondary learner to estimate the usefulness of each example, and then selects only informative examples for fine-tuning.

Original: We show that this technique works well and is applicable in a variety of settings.
Revised: We show that this technique works well and is applicable in a variety of settings.

Original: We further analyze the secondary learner capabilities.
Revised: We further analyze the secondary learner capabilities.

Original:  Background
Revised:  Background

Original: A language model L is a function with parameters θ , which, when given an ordered sequence of tokens X = { x 1 , . . .
Revised: A language model L is a function with parameters θ , which, when given an ordered sequence of tokens X = { x 1 , . . .

Original: , x n } as input, outputs a probability distribution over the next token y , L ( X ; θ ) =
Revised: , x n } as input, outputs a probability distribution over the next token y , L ( X ; θ ) =

Original: ˆ p ( y | X ) .
Revised: ˆ p ( y | X ) .

Original: Given a test set T of (sequence, next token) pairs,
Revised: Given a test set T of (sequence, next token) pairs,

Original: T = { ( X 1 , y 1 ) , . . .
Revised: T = { ( X 1 , y 1 ) , . . .

Original: , ( X n , y n ) } , the perplexity Λ( T ; θ ) of the language model L
Revised: , ( X n , y n ) } , the perplexity Λ( T ; θ ) of the language model L

Original: ( X ; θ ) over the set T is defined as Λ( T ; θ ) =
Revised: ( X ; θ ) over the set T is defined as Λ( T ; θ ) =

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: · log 2 L ( X i ; θ ) , where ¯ p ( y i ) denotes the one-hot probability distribution that assigns all of its probability mass to the token y i .
Revised: [Equation] X i [Equation] θ ) , where ¯ p ( y i ) denotes the one-hot probability distribution that assigns all of its probability mass to the token y i .

Original: LMs are trained to minimize perplexity on very large training corpora.
Revised: LMs are trained to minimize perplexity on very large training corpora.

Original: In practice, pre-trained LMs are often fine-tuned using a new corpus or transferred to a new task (Howard and Ruder, 2018).
Revised: In practice, pre-trained LMs are often fine-tuned using a new corpus or transferred to a new task (Howard and Ruder, 2018).

Original: Formally, let F = { ( X i , y i ) } i be a target set.
Revised: Formally, let F = { ( X i , y i ) } i be a target set.

Original: Fine-tuning on the set
Revised: Fine-tuning on the set

Original: F minimizes the expected value of the loss Λ :
Revised: F minimizes the expected value of the loss Λ :

Original: [Equation]
Revised: [Equation]

Original: The initial parameterization ˆ θ 0 of the LM is defined by its pre-trained parameters ˆ θ 

Revised: The initial parameterization ˆ θ 0 of the LM is defined by its pre-trained parameters ˆ θ

Original: = θ .
Revised: = θ .

Original: Thefine-tuning problem in Eq.
Revised: The fine-tuning problem in Eq.

Original: (1) is solved by applying stochastic gradient descent (SGD) on samplesfrom F .
Revised: (1) is solved by applying stochastic gradient descent (SGD) on samples from F .

Original: We refer to methods that randomly sample contexts to update pretrained model parameters as standard fine-tuning .
Revised: We refer to methods that randomly sample contexts to update pretrained model parameters as standard fine-tuning .

Original:  Information Gain Filtration 
Revised:  Information Gain Filtration 3. Informativeness of an Example

Original: 3. Informativeness of an Example

Original: Given a pre-trained language model L ( X ; θ ) and atarget dataset F , we define the informativeness ofan example ( X, y ) ∈ F as the improvement thatit will grant to the model.
Revised: Given a pre-trained language model L ( X ; θ ) and a target dataset F , we define the informativeness of an example ( X, y ) ∈ F as the improvement that it will grant to the model.

Original: Namely, we define theinformation gain (IG) of ( X, y ) over an objectiveset O as the difference in perplexity measured on
Revised: Namely, we define the information gain (IG) of ( X, y ) over an objective set O as the difference in perplexity measured on

Original: O before and after training on ( X, y ) ,
Revised: O before and after training on ( X, y ) ,

Original: [Equation]
Revised: [Equation]

Original: where θ is the initial parameterization of the LMand θ ′ ( X, y ) is the parameterization after training with the example ( X, y ) .
Revised: where θ is the initial parameterization of the LM and θ ′ ( X, y ) is the parameterization after training with the example ( X, y ) .

Original: The objective set
Revised: The objective set

Original: O = { ( X 1 , y 1 ) , . . .
Revised: O = { ( X 1 , y 1 ) , . . .

Original: , ( X n , y n ) } is a held-out subset of training data that informs our decision aboutwhich contexts are informative.
Revised: , ( X n , y n ) } is a held-out subset of training data that informs our decision about which contexts are informative.

Original: For brevity, we denote IG O
Revised: For brevity, we denote IG O

Original: ( X, y ) as IG ( X ) .
Revised: ( X, y ) as IG ( X ) .

Original: In practice, the objective set could be a subset of the set F .
Revised: In practice, the objective set could be a subset of the set F .

Original: 3. Filtering Examples
Revised: 3. Filtering Examples

Original: Next, we propose Information Gain Filtration(IGF), a new method, that exploits IG ( X ) forfine-tuning.
Revised: Next, we propose Information Gain Filtration (IGF), a new method, that exploits IG ( X ) for fine-tuning.

Original: Given a new example ( X, y ) themethod chooses between a) updating the modelparameters θ backpropagating ( X, y ) , and b)skipping it, leaving the model parameters unchanged.
Revised: Given a new example ( X, y ) the method chooses between a) updating the model parameters θ backpropagating ( X, y ) , and b) skipping it, leaving the model parameters unchanged.

Original: For this purpose, we define the functionq ( X, action ) and assign a value to each of the actions above:[Equation] [Equation]
Revised: For this purpose, we define the function q ( X, action ) and assign a value to each of the actions above:

Original: [Equation]
Revised: [Equation]

Original: where T S KIP is a free “threshold” parameter fordeciding which IG ( X ) values are sufficientlyhigh to warrant a model update.
Revised: where T S KIP is a free “threshold” parameter for deciding which IG ( X ) values are sufficiently high to warrant a model update.

Original: Followingthis definition, we apply a greedy policy for filtering examples during fine-tuning:
Revised: Following this definition, we apply a greedy policy for filtering examples during fine-tuning:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: arg max a ∈{ B ACKPROP , S KIP } q ( X, a ) .

Original: By filtering examples in this way, we aim to reduce the variability effect in data order (Dodge et al. , 2020),and improve the generalizability of our training set(Mosbach et al., 2021).
Revised: By filtering examples in this way, we aim to reduce the variability effect in data order (Dodge et al., 2020), and improve the generalizability of our training set (Mosbach et al., 2021).

Original: 3. Approximating Information Gain 
Revised: 3. Approximating Information Gain Computing IG

Original: Computing IG ( X ) in Eq.
Revised: ( X ) in Eq.

Original: (2) entails a backpropagation step, making direct application of q ( X, action ) as expensive as standard fine-tuning.
Revised: (2) entails a backpropagation step, making direct application of q ( X, action ) as expensive as standard fine-tuning.

Original: Thus, we approximate IG ( X ) with a secondary learner model ˆ Q ( X ) .
Revised: Thus, we approximate IG ( X ) with a secondary learner model ˆ Q ( X ) .

Original: First, we construct a training dataset D by drawing a random subset of examples from the fine-tuning set F and measuring IG ( X ) on the objective set O .
Revised: First, we construct a training dataset D by drawing a random subset of examples from the fine-tuning set F and measuring IG ( X ) on the objective set O .

Original: Each entry in D is of the form ( X i , IG ( X i )) .
Revised: Each entry in D is of the form ( X i , IG ( X i )) .

Original: Using D , we train the secondary learner ˆ Q to predict a normalized IG ( X ) .
Revised: Using D , we train the secondary learner ˆ Q to predict a normalized IG ( X ) .

Original: Normalization helps standarize the threshold T S KIP during filtration.
Revised: Normalization helps standarize the threshold T S KIP during filtration.

Original: The resulting ˆ Q is applied to filter examples for fine-tuning.
Revised: The resulting ˆ Q is applied to filter examples for fine-tuning.

Original: The effectiveness of the learner at distinguishing “high quality” from “low quality” examples should degrade as the parameters diverge from their initial values θ 0 used for constructing D .
Revised: The effectiveness of the learner at distinguishing “high quality” from “low quality” examples should degrade as the parameters diverge from their initial values θ 0 used for constructing D .

Original: To ameliorate this problem, we modify the threshold
Revised: To ameliorate this problem, we modify the threshold

Original: T S KIP during the fine-tuning process.
Revised: T S KIP during the fine-tuning process.

Original: Since ˆ Q is most accurate at the first step, we switch from highly selective (a high value) to highly permissive (a low value).
Revised: Since ˆ Q is most accurate at the first step, we switch from highly selective (a high value) to highly permissive (a low value).

Original: This allows the model to take advantage of the accurate predictions for IG ( X ) early in the fine-tuning process.
Revised: This allows the model to take advantage of the accurate predictions for IG ( X ) early in the fine-tuning process.

Original:  Experimental Results
Revised:  Experimental Results

Original: We next analyze IGF’s performance across different choices of datasets, fine-tuning tasks, and models.
Revised: We next analyze IGF’s performance across different choices of datasets, fine-tuning tasks, and models.

Original: We tested these results on a Books dataset (Zhu et al., 2015), a “Mixed” dataset composed from the Books and a corpus of scraped Reddit comments (Huth et al., 2016), and WikiText- (Merity et al., 2017).
Revised: We tested these results on a Books dataset (Zhu et al., 2015), a “Mixed” dataset composed from the Books and a corpus of scraped Reddit comments (Huth et al., 2016), and WikiText- (Merity et al., 2017).

Original: The Books corpus allows us to fairly compare standard fine-tuning against IGF, whereas the Mixed corpus allows us to analyze the effectiveness of the method at separating informative contexts from uninformative ones.
Revised: The Books corpus allows us to fairly compare standard fine-tuning against IGF, whereas the Mixed corpus allows us to analyze the effectiveness of the method at separating informative contexts from uninformative ones.

Original: We implement standard fine-tuning with Adam (Kingma and Ba, 2015).
Revised: We implement standard fine-tuning with Adam (Kingma and Ba, 2015).

Original: Our secondary learner, ˆ Q , represents the input text X by embedding it with 768dimensional byte-pair embeddings (Gage, 1994), followed by a convolution with kernel width 3, a max-pooling operation over the time axis, and a 2-layer network.
Revised: Our secondary learner, ˆ Q , represents the input text X by embedding it with 768dimensional byte-pair embeddings (Gage, 1994), followed by a convolution with kernel width 3, a max-pooling operation over the time axis, and a 2-layer network.

Original: 4. Fine-tuning Performance
Revised: 4. Fine-tuning Performance

Original: We compare the performance of IGF directly to standard fine-tuning on the pre-trained GPTSmall model (Wolf et al., 2020).
Revised: We compare the performance of IGF directly to standard fine-tuning on the pre-trained GPTSmall model (Wolf et al., 2020).

Original: Figure 1 depicts the results of fine-tuning the model on the Mixed
Revised: Figure 1 depicts the results of fine-tuning the model on the Mixed

Original: [Figure]
Revised: [Figure]

Original: T e s t p e r p l e x i t y
Revised: T e s t p e r p l e x i t y

Original: Standard Mixed Standard Books IGF with Constant Threshold
Revised: Standard Mixed [Figure] Books IGF with Constant Threshold

Original: IGF with Shifting Threshold 

Revised: IGF with Shifting Threshold

Original: T e s t p e r p l e x i t y a t dataset (batch size 16, learning rate 5 × 10 − 5 ,
Revised: T e s t p e r p l e x i t y a t dataset (batch size 16, learning rate 5 × 10 − 5 ,

Original: Adam β 1 = 0 .
Revised: Adam β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: As GPT-2 wastrained originally including Reddit data, we expectsome Mixed dataset examples to be uninformative.
Revised: As GPT-2 was trained originally including Reddit data, we expect some Mixed dataset examples to be uninformative.

Original: Hence, we included standard fine-tuning onthe Books corpus as a reference run with more informative data.
Revised: Hence, we included standard fine-tuning on the Books corpus as a reference run with more informative data.

Original: Standard fine-tuning on Books (orange) achieves a median perplexity of 57.3, compared to 56.9 for IGF with a constant threshold and54.0 for IGF with a shifting threshold.
Revised: Standard fine-tuning on Books (orange) achieves a median perplexity of 57.3, compared to 56.9 for IGF with a constant threshold and 54.0 for IGF with a shifting threshold.

Original: We further tested IGF with shifting thresholdacross several choices of dataset, fine-tuning specifications, and model architecture.
Revised: We further tested IGF with shifting threshold across several choices of dataset, fine-tuning specifications, and model architecture.

Original: We tested on
Revised: We tested on

Original: WikiText-103 when fine-tuning GPT-2 Small (Perplexity: IGF 67.8 vs 69.8),
Revised: WikiText-103 when fine-tuning GPT-2 Small (Perplexity: IGF 67.8 vs 69.8),

Original: GPT-2 Medium (Perplexity: IGF 27.1 vs 27.4), BERT (Devlin et al.
Revised: GPT-2 Medium (Perplexity: IGF 27.1 vs 27.4), BERT (Devlin et al., 2019)

Original: (Masked perplexity IGF 4.29 vs 4.33), andon Stanford Sentiment Treebank (SST-2) (Socheret al., 2013) (Accuracy IGF 94.27 vs 94.06).
Revised: (Masked perplexity IGF 4.29 vs 4.33), and on Stanford Sentiment Treebank (SST-2) (Socher et al., 2013) (Accuracy IGF 94.27 vs 94.06).

Original: In every case, IGF exceeds the performance of standardfine-tuning, suggesting that it is a method broadlyapplicable to a variety of modalities and domains.
Revised: In every case, IGF exceeds the performance of standard fine-tuning, suggesting that it is a method broadly applicable to a variety of modalities and domains.

Original: 4. Understanding the Secondary Learner
Revised: 4. Understanding the Secondary Learner

Original: As the secondary learner aims to approximate theinformativeness of a sample, we analyze next thequality of such approximation.
Revised: As the secondary learner aims to approximate the informativeness of a sample, we analyze next the quality of such approximation.

Original: For this purpose,we created a dataset of 10,000 ( X, IG ( X )) pairsfrom the Mixed corpus using an objective set of160 contexts with 32 tokens each drawn solelyfrom the Books corpus.
Revised: For this purpose, we created a dataset of 10,000 ( X, IG ( X )) pairs from the Mixed corpus using an objective set of 160 contexts with 32 tokens each drawn solely from the Books corpus.

Original: Then, we trained the secondary learner on this dataset and tested it on randomly sampled contexts from the Mixed corpus.
Revised: Then, we trained the secondary learner on this dataset and tested it on randomly sampled contexts from the Mixed corpus.

Original: Because the objective set contains only examplesfrom one corpus, we expect the secondary learnerto assign higher IG ( X ) values to other examplesfrom the same corpus.
Revised: Because the objective set contains only examples from one corpus, we expect the secondary learner to assign higher IG ( X ) values to other examples from the same corpus.

Original: Figure 2 shows that thereis a significant difference in the distributions of ˆ Qvalues between the two corpora, demonstrating itsseparating capabilities.
Revised: Figure 2 shows that there is a significant difference in the distributions of ˆ Q values between the two corpora, demonstrating its separating capabilities.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Above, we used a convolutional neural network as ˆ Q .
Revised: Above, we used a convolutional neural network as ˆ Q .

Original: Here we explore how other simpler methods perform.
Revised: Here we explore how other simpler methods perform.

Original: We encoded the contexts both by using the standard
Revised: We encoded the contexts both by using the standard

Original: GPT-2 Small word embedding, and with a one-hot encoding of the token identities.
Revised: GPT-2 Small word embedding, and with a one-hot encoding of the token identities.

Original: Standard linear regression performed on both encoding types (30K parameters for word embeddings and 450K parameters for one-hot encoding) performs nearly as well at approximating IG ( X ) with a convolutional model.
Revised: Standard linear regression performed on both encoding types (30K parameters for word embeddings and 450K parameters for one-hot encoding) performs nearly as well at approximating IG ( X ) with a convolutional model.

Original: We also tested a learner with only 25K parameters that assigned each token a value by averaging the IG ( X ) values for contexts that contained that token.
Revised: We also tested a learner with only 25K parameters that assigned each token a value by averaging the IG ( X ) values for contexts that contained that token.

Original: Figure 3 compares the performance of these architectures across different training data sizes.
Revised: Figure 3 compares the performance of these architectures across different training data sizes.

Original: The convolutional network is the most sample efficient method, as it can effectively learn IG ( X ) with as few as 2K training examples.
Revised: The convolutional network is the most sample efficient method, as it can effectively learn IG ( X ) with as few as 2K training examples.

Original:  Conclusion
Revised:  Conclusion

Original: In the context of LM fine-tuning, we have shown that a secondary learner can efficiently and effectively distinguish between informative and uninformative training examples.
Revised: In the context of LM fine-tuning, we have shown that a secondary learner can efficiently and effectively distinguish between informative and uninformative training examples.

Original: This secondary learner can select useful training examples in a method we call Information Gain Filtration, leading to better model performance than standard fine-tuning.
Revised: This secondary learner can select useful training examples in a method we call Information Gain Filtration, leading to better model performance than standard fine-tuning.