Selecting Informative Contexts Improves Language Model Fine-tuning (2021)

Abstract: Language model fine-tuning is essential for modern natural language processing, but is computationally expensive and time-consuming. Further, the effectiveness of fine-tuning is limited by the inclusion of training examples that negatively affect performance. Here we present a general fine-tuning method that we call information gain filtration for improving the overall training efficiency and final performance of language model fine-tuning. We define the information gain of an example as the improvement on a validation metric after training on that example. A secondary learner is then trained to approximate this quantity. During fine-tuning, this learner selects informative examples and skips uninformative ones. We show that our method has consistent improvement across datasets, fine-tuning tasks, and language model architectures. For example, we achieve a median perplexity of 54.0 on a books dataset compared to 57.3 for standard fine-tuning. We present statistical evidence that offers insight into the improvements of our method over standard fine-tuning. The generality of our method leads us to propose a new paradigm for language model fine-tuning — we encourage researchers to release pretrained secondary learners on common corpora to promote efficient and effective fine-tuning, thereby improving the performance and reducing the overall energy footprint of language model fine-tuning.

## introduction language modeling is the task of generating language from context. this is often framed as an autoregressive task, where a model predicts the conditional probability of the next word based on the sequence of previously observed or generated tokens. language modeling has seen a recent surge in relevance thanks to its success as a pretraining objective for self-supervised representa-tion learning. the most prominent language models today are transformer-based models @xcite such as bert @xcite and gpt-2 @xcite . language models are most commonly trained with backpropagation using traditional nlp loss functions such as cross entropy. these loss functions are designed so that the models are rewarded for assigning high probability to text that appears commonly in the training corpus. the energy and computational costs of training a state-of-the-art language model from scratch are very high, to the point of impracticality for most researchers. one recent estimate suggests that training a single state-of-the-art model with architecture search takes more energy than five cars will use in their entire lifetimes @xcite . in practice, this cost is sidestepped by pretraining, where a language model is trained once and then released publicly. this language model can then be updated for use in other tasks through fine-tuning. for example, a generic language model can be fine-tuned to generate text that matches the style and syntax of any new corpus @xcite . while better than training from scratch, the cost of fine-tuning such large networks is still relatively high. fine-tuning to convergence for a single task can easily take in excess of a day on multiple energy-intensive gpus @xcite . recent work analyzing the fine-tuning process has shown that it has high variability between runs and is particularly sensitive to data ordering @xcite . those authors propose to overcome this variability by training models using many random seeds and then only keeping the best, effectively trading computational efficiency for model performance. while this improves performance, the reasons for the high variability between random seeds have yet to be explored. we hypothesize that much of this variability can be ex-plained by the random selection of highly "informative" training examples, which most effectively capture low-level distributional statistics of the target corpus. if this is the case, then it should be possible to quickly screen for informative training examples, ensuring high performance at reduced cost. in this paper, we suggest replacing the retrospective approach of testing many random seeds @xcite with a prospective approach to improving the effectiveness of language model fine-tuning. our approach uses a secondary learner to estimate the usefulness of each training example, and then selects only informative examples for training. we show that this technique works well and is applicable in a variety of finetuning settings. we examine why it works well, and present evidence that supports our hypothesis about informative examples explaining data ordering effects. in addition to performance gains, this method may mitigate the energy impact of deep neural network language modeling, as we require fewer backpropagation steps than other techniques that trade computational power for fine-tuning performance. ## related work several methods have recently been proposed to improve language model fine-tuning performance. @xcite proposed a technique based on neural network dropout @xcite for regularizing finetuned language models that involved stochastically mixing the parameters of multiple language models for the same domain, and further demonstrated the usefulness of pretrained weight decay over conventional weight decay for improving language model fine-tuning performance. @xcite showed that adding supplementary training to pretrained language models using supervised tasks yielded state of the art results for bert @xcite . @xcite proposed a related technique for increasing the amount of language model training data from out-of-domain data sources that relies on filtering out high cross-entropy contexts as measured by an in-domain language model. @xcite and @xcite have both suggested that language model finetuning is better in transformer-based models when starting when using features from intermediate layers as opposed to later layers. the instability of language model fine-tuning has previously been investigated by others. @xcite suggested that this instability is caused by a combination of insufficiently general training sets and optimization challenges. @xcite investigated how similar factors, such as non-standard optimization techniques and overreliance on a standard number of training iterations hurts the performance of fine-tuned language models. @xcite , whose work we replicate and build on here, showed that language model finetuning is sufficiently stochastic so that even random seed searches are a suitable technique for improving their overall performance. ## background a language model l is a function with parameters θ, which, when given an ordered sequence of tokens @xmath0, . . . , x n } as input, outputs a probability distribution over the next token y: given a test set t of (sequence, next token) pairs, @xmath1, y 1 ), . . . , (x n , y n )}, the perplexity λ(t ; θ) of the language model l(x; θ) over the set t is defined as: in practice, pre-trained language models are often fine-tuned using a new corpus or transferred to a new task @xcite . formally, let @xmath2, y i )} i be a target set. fine-tuning on the set f tries to minimize the expected value of the loss function λ: the initial parameterization θ0 of the language model is defined by its pre-trained parameters θ0 = θ. the fine-tuning problem in eq. ( 1 ) is then solved by applying stochastic gradient descent (sgd) on samples from f. namely, for a given batch b of samples from f, the language model parameters are updated by θk ← θk-1 -α∇λ(b; θk-1 ), where α is the step size. we refer to methods that randomly sample contexts to update pretrained model parameters as standard finetuning. while random sampling methods are useful @xcite , the stochasticity of context sampling suggests an avenue for additional improvement. such methods make no assumption on the informativeness of the examples in f, instead relying on randomness to find useful training samples. it is worth asking ourselves: can we efficiently measure the informativeness of an example? and if so, can we exploit that measurement for additional fine-tuning improvement? ## results here we first provide an empirical analysis suggesting that igf outperforms standard fine-tuning across different choices of datasets, fine-tuning tasks, and neural architectures. we follow this analysis with an examination of why igf works, and an exploration into the statistical properties of standard fine-tuning and of igf. we tested these results on a standard books dataset @xcite , a "mixed" dataset which is composed of training examples from two corpora (the books corpus and a corpus of scraped reddit comments ( huth et al., 2016) ), and the wikitext-103 dataset @xcite . the books corpus allows us to fairly compare standard fine-tuning against igf, whereas the mixed corpus allows us to analyze the effectiveness of the method at separating informative contexts from uninformative ones. in practice, our secondary learner, q, represents the input text x by embedding it with 768dimensional byte-pair embeddings @xcite . we then pass the input representations through a convolution with kernel width 3, followed by maxpooling operation over the time axis and a 2-layer feedforward network. this architecture was refined through coordinate descent, and evaluated on a separate held-out set of measured ig(x) values. the choice of architecture does not strongly affect method performance (see appendix a, figure 11 ). additionally, a neural network is not necessary for the learner, as simpler learning methods are sufficient (see figure 5 ). ## conclusion and future work in the context of language model fine-tuning, we have shown that a secondary learner can efficiently and effectively distinguish between informative and uninformative training examples. this secondary learner can be used to select useful training examples in a technique we call information gain filtration, leading to better model perfor-