A Natural Bias for Language Generation Models (2023)

Abstract: After just a few hundred training updates, a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens. Yet around this point, these models have identified a simple, loss-minimising behaviour: to output the unigram distribution of the target training corpus. The use of such a heuristic raises the question: Can we initialise our models with this behaviour and save precious compute resources and model capacity? Here we show that we can effectively endow standard neural language generation models with a separate module that reflects unigram frequency statistics as prior knowledge, simply by initialising the bias term in a model’s final linear layer with the log-unigram distribution. We use neural machine translation as a test bed for this simple technique and observe that it: (i) improves learning efficiency; (ii) achieves better overall performance; and perhaps most importantly (iii) appears to disentangle strong frequency effects by encouraging the model to specialise in non-frequency-related aspects of language.

## introduction consider the structure of a number of core tasks in natural language processing (nlp): predicting the next word following a given context. what if you did not understand the context -for example, if you did not know the language? in the absence of such knowledge, the optimal prediction would be the language's most frequent word. in fact, optimally one would predict each word according to its (unigram) frequency. 1 this is precisely the strategy that neural language models have been empirically observed to employ during early training stages @xcite ) -before they have learnt a language's syntax or semantics. although this strategy of predicting the unigram distribution emerges early in training, it still takes the model hundreds (or even thousands) of parameter updates to learn it from a cold start (see fig. 1 or chang and bergen, 2022, fig. 5 ). yet a straightforward factorisation of a language model's final linear layer shows that we can in fact encode this frequency-related knowledge prior to any optimisation, foot_0 with the goal of bypassing this early stage of learning: concretely, this is done by setting the bias term in a model's final linear layer to the log-unigram distribution of the training data. mathematically, this setup can be loosely interpreted as a modular "product of experts" @xcite , where the bias term represents a simple unconditional distribution over the vocabulary, thus allowing the input-dependent logits to specialise in capturing contextual information. indeed, we argue that a more modular design that disentangles word-frequency effects from contextual information may be desirable, given the recently-observed negative effects of word frequency statistics on models' generalisation abilities @xcite @xcite . while this initialisation approach has been historically used in language models @xcite @xcite , it has not seen widespread adoption within our current language generation architectures -an observation we attribute to uncertainty around whether the bias term automatically specialises to capture frequency without explicit encouragement to do so. we first observe that this is not the case -in fact, the final-layer bias term rarely changes from its random initialisation (see app. a.6), suggesting frequency is encoded elsewhere in the model parameters. we then empirically explore the impact of this initialisation on various aspects of model behaviour -within the context of current transformer models for machine translation -including overall performance, learning efficiency, and the relationship between model-assigned probability and word frequency. we find this initialisation indeed leads to increased training efficiency: models achieve higher bleu scores earlier on in training. more surprisingly, it also leads to improved overall performance. we discuss several potential reasons for these results, including changes to training dynamics and a mitigation of overfitting to surface statistics. ## experiments we explore the effects of the unigram bias initialisation strategy on neural machine translation systems in comparison to a standard initialisation technique: initialising the bias to all 0s (denoted @xmath0. ## conclusion and future work in this work, we revisit a simple initialisation technique in the context of modern neural language generation models: setting the bias term in the final linear projection layer to the log-unigram distribution of (sub)words within the training corpus. this strategy leads to more efficient training; perhaps more surprisingly, it also leads to better overall performance in our machine translation experiments. we offer analysis and discussion as to the cause of these trends. an interesting direction for future work could be determining the effects that this initialisation procedure has on various model properties, e.g., its embedding space, and its benefits specifically in low-resource settings. furthermore, extensions of this work could explore potential uses of this strategy in the mitigation of problems with lexically infrequent words, e.g., by analysing via the decomposition in eq. ( 4 ) whether a model's probability estimate for a word is being driven by frequency or contextual components. finally, this technique is not limited to models of distributions over strings; it is in fact applicable to any neural classification setting, the exploration of which is left to future work.