Frequency Balanced Datasets Lead to Better Language Models (2023)

Abstract: This paper reports on the experiments aimed to improve our understanding of the role of the amount of data required for training attention-based transformer language models. Specifically, we investigate the impact of reducing the immense amounts of required pre-training data through sampling strategies that identify and reduce high-frequency tokens as different studies have indicated that the existence of very high-frequency tokens in pre-training data might bias learning, causing undesired effects. In this light, we describe our sampling algorithm that iteratively assesses token frequencies and removes sentences that contain still high-frequency tokens, eventually delivering a balanced, linguistically correct dataset. We evaluate the results in terms of model perplexity and fine-tuning linguistic probing tasks, NLP downstream tasks as well as more semantic SuperGlue tasks. The results show that pre-training with the resulting balanced dataset allows reducing up to three times the pre-training data.

## introduction historically, using more training data has been considered a key to improving the performance of natural language processing (nlp) tools. the same seems to be true for transformer-based large language models (llms) such as bert @xcite and roberta @xcite ) that exhibit their best performance when being trained with really massive quantities of texts. however, the question arises whether more training data is the only factor for improving results or whether specific characteristics of the provided texts have a particular impact on the learning efficiency of these powerful models. recent research @xcite has found evidence about the impact of the absolute frequency of pretraining data tokens in the prediction capacities of the model. besides, high-frequency terms have proved to be behind some phenomena related to the geometry of the representations causing problems for so called semantic tasks that rely on similarity assessments @xcite . in this paper, we report on our experiments to better understand the impact of token frequency in the pre-training data on language model (lm) quality, in terms of perplexity, and on the quality of the representations that these models provide for fine-tuning. our ultimate goal is to study to what extent the quantity of pre-training data could be lessened, since this could be of great importance for training lm for low-resource languages (lrls). texts are known to follow zipf's law with some tokens occurring very frequently, some occurring with medium frequency, and many tokens that rarely occur forming a long tail. different disciplines have addressed the problems of using long-tailed training data where some elements appear on most of the data but most of the elements are under-represented. under-sampling and oversampling techniques to directly adjust the number of examples by removing and adding data respectively have been proposed @xcite . in this context, we propose an under-sampling algorithm whose objective is to balance token frequency by removing the sentences in which high-frequency tokens (and bigrams) occur, thus improving the estimation of model probabilities for low-frequency tokens. by ensuring that tokens have a balanced exposure during training, the model should become more capable of learning more diverse linguistic patterns, resulting in an improved model perplexity and in better representations for nlp tasks. our experiments show that sampling the data to avoid a highly skewed token frequency distribution delivers results equivalent to pre-training the model with average three times more but unbalanced, raw data. in this paper, we report on how our algorithm processes the dataset, how it iteratively assesses token frequencies and removes sentences that contain still high-frequency tokens, eventually delivering a balanced, linguistically correct dataset. we evaluate the results in terms of model perplexity for english general and domain texts and for four other languages of different morphological complexities. furthermore, we evaluate the approach in terms of the quality of the representations for fine-tuning to a range of tasks: linguistic probing tasks, nlp tasks and more semantic superglue tasks. our hypothesis was that if fuster-baggetto and fresno (2022) analysis is correct, reducing frequency biases would also improve the results in semantic tasks. in what follows, section 2 reviews the related work; section 3 describes the algorithm implemented, as well as its impact on the initial and the resulting datasets. section 4 describes the experimental setup for training ten roberta models with different data and in different languages and domains. also in section 4, we describe the fine-tuning experiments for eleven different tasks that were meant to get thorough information about the impact of training the model with a balanced dataset, specially in semantic tasks. section 5 describes the results of the experiments in terms of perplexity of the models and of f1-score of the different tasks. results are analysed and discussed in section 6. finally, section 7 is devoted to sum up the conclusions and contributions. ## related work a number of works have studied the impact of word frequency on different aspects of llm and, in particular, on the quality of the delivered representations. @xcite have studied bert models and possible memorization based on token frequency, demonstrating that if a token appears fewer than 15 times, the model will disregard it, while a token that appears 100 times or more will be predicted more accurately. @xcite demonstrated that high frequency words and low frequency words are represented differently by transformer llm, in particular by bert. @xcite found evidence that bert models struggle to correctly predict a verb when it belongs to a set of word pairs (s-v), or bigrams, that appears less than 10 times or more than 100 times in the training corpus. @xcite also examined the strong impact of the frequency of terms in pre-training data, although in a gpt model, and found a strong correlation of token frequencies with the resulting model performance. @xcite provided evidence directly linking the presence of outlier dimensions @xcite with the frequency of tokens in the pre-training data. outlier dimensions are those dimensions at which parameters with an unusually high magnitude -in terms of standard deviation--are found consistently across the model layers. @xcite results suggest that outliers are due to the highly skewed token frequency distribution in the textual pre-training data. moreover, @xcite demonstrated that the non-fine-tuned bert models contain token frequency biases that distort the embedding representation space. the distortion leads to the already observed poor performance on semantic tasks because of what @xcite already diagnosed as anisotropy in transformer lms or the representation degeneration problem: the embeddings concentrate in a hypercone instead of occupying the whole space. consistently, @xcite found that randomly sampled words tend to be highly similar when measured by cosine similarity. @xcite have also found evidence that different transformer-based lms have similar behavior regarding poor semantic isometry, even when they differ in number of parameters, architecture or learning objective. as for bias-removal techniques, our approach is different from @xcite , that preferred not to remove frequent tokens from sentences because it would be affecting the ability of the lm. instead, they removed the embeddings after processing, before pooling like @xcite and @xcite . in our experiments, we remove the sentences with a high frequency token to guarantee that only grammatical sentences are processed. our approach is, thus, in line with @xcite findings that got a noticeable difference in downstream tasks performance when using the british national corpus of 100m tokens, as a sample of a curated data set that proved to be enough to learn the required basic linguistic skills. ## frequency in this section, we report on our algorithm that processes a raw text corpus and returns a balanced dataset. as explained in this section, which we set to a size 10m tokens -as we will explain later on in more detail. the processing includes, on the one hand, calculating the number of occurrences of content tokens in the whole dataset and assessing to what extent they usually co-occur with other less frequent tokens, thus forming to be kept bigrams. on the other hand, frequency counts are updated after each sentence removal. algorithm 1 removing sentences from dataset require: s, t max , b min , f req, bi ensure: s balanced 1: s balanced ← s 2: deleted_phrases ← true 3: while deleted_phrases do 4: stop_process ← false 5: for each s i in s balanced do 6: w ← split_sentence(s i ) 7: if all w i in w, f req[w i ] > t max and bi[(w i , )] > b min then 8: remove s i from s balanced 9: update w i from f req 10: update (w i , ) from bi 11: stop_process ← true 12: end if 13: end for 14: if @xmath0: deleted_phrases ← false 16: ## experimental setup in this section, we describe the setup of the different experiments. we trained ten roberta models with different processed and non-processed datasets for english, french, german, turkish data and measured their perplexity to assess the impact of balancing token frequency in pre-training data. we also applied the algorithm for assessing how it could be used in a lrls scenario, such as the one of quechua, with only a 6m tokens corpus that got reduced into only 1m. after that, we describe the fine-tuning of the english models for four different typical nlp tasks, for a linguistic probing classifier and for four different superglue tasks. ## results we now describe the results of the different experiments. in all of them, the use of a balanced dataset for training the lms equals or improves its quality in terms of perplexity of the lm or in terms of accuracy in fine-tuned tasks 8 . ## discussion since word frequency shows a highly skewed distribution in raw texts, when using a balanced dataset (especially when pre-training with a small 10m corpus), the potential biases due to highly-frequent tokens should be moderated, and the system should be able to perform a better computation of the probabilities of medium and rare words. the results of our experiments consistently show that perplexity in all our experiments improves when working with a balanced corpus. in particular, we have seen that using balanced datasets for modeling morphological complex languages with more number of types due to inflection like german, french, turkish and quechua also reduces model perplexity, confirming the robustness of the approach with different probability distributions. the improvement of rare tokens modeling after eliminating high-frequency tokens is clearly shown in the blimp results, which show a better performance for a model trained with a balanced 10m corpus than a model trained with a 100m words corpus for tasks related to medium or rare words like negative polarity items (npi) like 'ever' or control/raising verbs like 'oblige' or 'promise'. note that negative sentences are a minority in english written texts ranging from less than 10% to 32% of the sentences in different reference corpora @xcite . also note that filler-gap phenomena mostly occur in question sentences that are less frequent in any corpus of english too (see, for instance, @xcite ). moreover, since high-frequency words seem to be responsible for representations with outlier components distorting the geometry of the created hyperspace, semantic fine-tuned tasks should improve more than others that do not rely on semantic similarity. for instance, we see an important improvement in the relation classification task (rel), which predicts the relation that holds between two entities, and also an improvement that equals the results of pre-training with 100m in the task of word in context (wic) -a word sense disambiguation task. therefore, our results support fuster-baggetto and fresno (2022) because we found that the lack of frequency bias improves the quality of the semantic representation and makes the system perform significantly better especially in semantic tasks. ## conclusions in the current paper, we reported on an algorithm that processes a corpus assessing token frequencies to remove sentences that contain high-frequency tokens, eventually delivering a balanced, linguistically correct data set. we have shown that training with a balanced corpus improves the quality of the model, as shown in a significant reduction in perplexity for english general and domain texts and for five other languages of different morphological complexities. furthermore, the results also show the quality of the representations for fine-tuning tasks: linguistic probing tasks, nlp tasks and su-perglue tasks also improve the results obtained with a non-balanced, raw dataset, in many cases delivering results that equals the results obtained with models trained with 100m tokens of raw text. in particular, such better results in semantic tasks confirm previous researches that signalled highfrequency tokens distorting the semantic space created by the models. summing up, the contributions of our research are the algorithm for balancing textual data, the different balanced corpora that are available with the code at anonymized, and further evidence about the impact of high-frequency tokens for training transformed-based lms. finally, we think that our findings will be of great interest for lrls, since we have defined the characteristics of the data to be gathered or created so that, using less data, the created models can achieve the best results.