Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing (2024)

Abstract: Language models strongly rely on frequency information because they maximize the likelihood of tokens during pre-training. As a consequence, language models tend to not generalize well to tokens that are seldom seen during training. Moreover, maximum likelihood training has been discovered to give rise to anisotropy: representations of tokens in a model tend to cluster tightly in a high-dimensional cone, rather than spreading out over their representational capacity.Our work introduces a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. We then present a method for reducing the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. Our Syntactic Smoothing method adjusts the maximum likelihood objective function to distribute the learning signal to syntactically similar tokens. This approach results in better performance on infrequent English tokens and a decrease in anisotropy. We empirically show that the degree of anisotropy in a model correlates with its frequency bias.

## introduction humans possess a remarkable ability to quickly understand the meaning of unknown words, given contextual cues. consider the sentence, "the golden gate bridge has been obnebulated every morning this week, limiting visibility of the pacific ocean." for many readers, 'obnebulated' is probably not a familiar term, but we are likely to infer that 1) it is almost certainly a verb because of the -ed suffix and occurrence after perfective and passive auxiliaries, and 2) its meaning relates to visibility and climatic conditions. 1 the 1 'obnebulate' is an obsolete word meaning, "to obscure with or as with a mist; to befog" (oxford english dictionary). ability to integrate unknown words based on syntactic and semantic context is essential for robust language understanding and still poses a significant challenge for language models. nevertheless, pre-trained transformer language models (plms) have proven tremendously capable of solving a wide array of language processing tasks @xcite . part of the success of plms can be attributed to the pre-training objective. despite variations in architecture, the vast majority of language models are pre-trained to maximize the log-likelihood of a word, given the surrounding context @xcite @xcite . as language use is characterized by a zipfian distribution @xcite , language models are exposed to frequent tokens exponentially more often than infrequent ones during pre-training. consequently, the representations of these frequent tokens are optimized based on exponentially more learning signals than those of low-frequency tokens. it has been shown that maximum likelihood objectives lead to representation degeneration in english language models because infrequent tokens are pushed into a narrow manifold of the representational space @xcite . this representation degeneration problem is linked to the broader problem of anisotropy: the hidden states of a language model tend to cluster together into a small cone-shaped subspace, rather than over their full representational capacity @xcite @xcite . as language model evaluation is based on cumulative evaluation scores that conceal how well a model processes infrequent words, the disparities in the representational space are difficult to assess. conventional language modeling approaches require large model sizes to effectively capture longtail vocabulary distributions, limiting the scalability of these methods @xcite . in this work, we propose syntactic smoothing: a syntactically-guided label smoothing approach to improve the representation of infrequent tokens in language models without resorting to perpetual increases of model and training data size. we smoothly distribute the backpropagation signal over syntactically similar tokens using a similarity metric based on part-of-speech (pos) tag distributions. using this method, tokens that are seldom seen during training benefit from the more frequent updates of tokens that occur in similar syntactic functions. we evaluate our method using a new metric for quantifying the frequency bias of language models (illustrated in fig. 1 ) and find that syntactic smoothing reduces both the frequency bias and the degree of anisotropy in a small english language model. we further explore the relationship between anisotropy and frequency bias and their effect on downstream performance. ## related literature through maximum likelihood training, language models implicitly learn to encode token frequency statistics. this training process gives rise to a frequency bias in models that constrains their ability to generalize to infrequent tokens. in this section, we begin by reviewing literature that discusses the challenges of generalizing linguistic knowledge to infrequent tokens. we then examine recent work that links the impact of token frequency to anisotropy in the models' representational space. ## frequency bias we investigate frequency effects using a zeroshot test of grammatical capability known as blimp: the benchmark of linguistic minimal pairs @xcite . blimp comprises 67 datasets (or "subtasks"), each consisting of 1,000 pairs of grammatical and ungrammatical sentences that differ only with respect to a specific linguistic characteristic (covering syntax, morphology, and semantics). language models are tasked with assigning a higher likelihood to the grammatical sentence. the grammatical generalization capabilities of a language model are often summarized by averaging the accuracies achieved across the 67 blimp tasks. while random guessing scores 0.5, state-ofthe-art models have achieved scores of 0.87 when trained on large datasets, and models trained on the 10m-word babylm dataset have achieved scores up to 0.80 @xcite . blimp is carefully balanced to ensure individual tokens occur equally in both sentence types. however, within a single pair, there may be an imbalance in average token frequency: for instance, the sentence grace's piano teachers are known has a log frequency of 8.35 while its associated minimal pair grace's piano teachers are replied has a log frequency of 6.20. we hypothesize that despite the minimal difference in blimp pairs, models trained in a typical manner will be biased by token frequency when determining grammatical acceptability. our goal is to quantify how language model performance differs between blimp pairs with large positive frequency differences (where the correct sentence has more frequently occurring tokens) and with large negative frequency differences (where the correct sentence has much less frequently occurring tokens). we do so in two steps. first, for each blimp sentence pair, we calculate the average (natural log) frequency of the differing tokens. frequencies of individual tokens are computed with respect to a model's training data; for instance, in the example above the token known has a log frequency of 8.35 in the training data. sentence pairs are then ranked by the relative difference in these average frequencies, where positive values indicate a higher average frequency for the acceptable sentence. these relative differences form a distribution, as shown in the middle plot of fig. 1 . then, we compute the blimp score using pseudo log-likelihood @xcite for blimp pairs in the upper and lower thirds of the relative frequency difference distribution. we exclude the middle third, as these represent pairs with minimal frequency differences (see the frequency plot for details). we define a model's frequency bias as the difference between the two blimp scores. the entire process is illustrated in fig. 1 . in practice, we find that standard transformer language models, such as opt-125m @xcite , roberta-base @xcite , and t5base @xcite , exhibit a frequency bias as high as 13.7%. our goal is to develop a model that can attain a frequency bias close to zero while attaining a high blimp score: that is, a model that makes determinations on the grammatical acceptability of sentences based solely on relevant linguistic aspects, rather than relying on possibly misleading statistical artifacts of the training data. ## syntactic smoothing we hypothesize that transformer language models exhibit a strong frequency bias due to their maximum likelihood training objective, which limits infrequent tokens from receiving useful learning signals and thus hinders their ability to effectively encode linguistic information. to address this, we propose at each learning step to backpropagate the learning signal of a target token to all other tokens serving similar syntactic roles; this benefits infrequent tokens that appear less often in the training data. syntactic smoothing implements this strategy by distributing a portion of every update signal to all syntactically similar tokens using a syntactic similarity metric (operationalized below). this results in the representation of infrequent tokens approaching the average representation of all tokens that serve a similar syntactic function; e.g., the representation of a niche word like 'obnebulated' would encode its syntactic role as a verb. our method consists of two components; (1) a similarity metric that uses part-of-speech distributions as a coarse proxy for syntactic similarity, and (2) an adjustment to the loss function to smooth the backpropagation signal over syntactically similar tokens during pre-training. ## 5 results our results are summarized in ## conclusion our work studies the phenomenon of frequency bias in language models that degrades the performance of these models on tokens infrequently observed during training. we develop a novel method for quantifying the degree to which a language model prefers grammatically incorrect sentences that contain frequent tokens over grammatically correct sentences containing infrequent tokens. we introduce a new training approach, syntactic smoothing, that distributes the backpropagation signal to syntactically similar tokens. using a coarse approximation of syntactic similarity based on part-of-speech tags, we show that this approach can remove the frequency bias without degrading broader language understanding. we also find that reductions in frequency bias are strongly correlated with reductions in a model's anisotropy. our findings provide a novel angle through which to observe the role of anisotropy in language modeling.