Automatic Quality Estimation for Data Selection and Curriculum Learning (2024)

Abstract: The size of neural models within natural language processing has increased at a rapid pace in recent years.With this increase in model size comes an increase in the amount of training data required for training.While these larger models have shown strong performance, their use comes with added training and data costs, can be resource-prohibitive for many researchers, and uses an amount of language data that is not always available for all languages.This work focuses on exploring quality estimation as a method of data selection or filtering.The aim is to provide models with higher quality data as compared to larger amounts of data.This approach was applied to machine translation models with varying data sizes as well as to the BabyLM Challenge.Given the 100M word dataset provided in the BabyLM Challenge, we test out various strategies for selecting 10M words for pretraining and use a curriculum learning approach based on the quality estimation scoring.We find small improvements in certain data settings.

## introduction in recent years, there has been a dramatic rise in the size of neural network models used for natural language processing tasks. to train these larger models, there has been a similar rise in the size of datasets used for training or pretraining. while these models have been quite successful, this trend comes with several downsides including the cost of creating these larger systems which also inhibits the ability of many researchers who lack access to the large scale computing resources required. by contrast, human language development occurs in children with exposure to far fewer words of training data. inspired by this, the babylm challenge @xcite focuses on "sampleefficient pretraining on a developmentally plausible corpus." one approach to improve model performance in data-limited settings is curriculum learning @xcite . just as human language learners are typically exposed to simpler language before building up to more complex utterances, curriculum learning involves increasing the difficulty of training examples over the course of model training. in order to do this, there must be some measure of "difficulty" in order to assign an ordering to training examples. in this work, we apply quality estimation scoring as an estimation of difficulty. these scores are used to train models for the babylm challenge, specifically restricted to 10 million words or less of training data. quality estimation (qe) in machine translation scores the quality of translation output without the need for a reference translation @xcite . through a series of experiments, we explore the effects of using qe to filter data for machine translation systems for both initial model training and fine-tuning, as well as the result of training on different quantities of data for each model (see section 3). prior work has shown that data filtering through qe can increase model performance @xcite . we explore that further in this work for both machine translation and language modeling in data restricted settings. since quality estimation scores the quality of the output of a machine translation system, it is likely that higher qe scores correspond to sentences which the system has an easier time translating. motivated by this, we experiment with using qe scores as an estimation of the difficulty of a given sentence for an nlp system. in particular, we use this for data selection as well as for difficulty scoring for curriculum learning training of a "baby" language models as part of the 2024 babylm challenge (see section 4). ## conclusion this work explored quality estimation for data filtering and curriculum learning on both machine translation systems and language models. as shown in our machine translation experiments (see section 3.2), modest improvements can be obtained through finetuning on filtered data. this benefit largely went away as the data size scaled up to the full iwslt17 dataset, suggesting that this method has more use for certain data limited settings rather than for general model use. furthermore, base model performance went up more noticeably when additional data was added, showing that more data made a larger difference than higher quality data in this setting. for the babylm challenge strict-small track, teams could form datasets consisting of up to 10 million words to train their language models. we explored several options for data selection from the provided 100 million word dataset. each model was then trained using a curriculum learning approach based on quality estimation scoring. overall, data source made a bigger difference to model performance than curriculum choice. in particular, models trained using the project gutenberg data generally had higher scores on downstream tasks. this suggests that while the other data sources are useful for human children learning language, the higher quality data available in the gutenberg dataset produced a better language model.