On the Role of Corpus Ordering in Language Modeling (2021)

Abstract: Language models pretrained on vast corpora of unstructured text using self-supervised learning framework are used in numerous natural language understanding and generation tasks. Many studies show that language acquisition in humans follows a rather structured simple-to-complex pattern and guided by this intuition, curriculum learning, which enables training of computational models in a meaningful order, such as processing easy samples before hard ones, has been shown to potentially reduce training time. The question remains whether curriculum learning can benefit pretraining of language models. In this work, we perform comprehensive experiments involving multiple curricula strategies varying the criteria for complexity and the training schedules. Empirical results of training transformer language models on English corpus and evaluating it intrinsically as well as after fine-tuning across eight tasks from the GLUE benchmark, show consistent improvement gains over conventional vanilla training. Interestingly, in our experiments, when evaluated on one epoch, the best model following a document-level hard-to-easy curriculum, outperforms the vanilla model by 1.7 points (average GLUE score) and it takes the vanilla model twice as many training steps to reach comparable performance.

## introduction pretrained language models are the foundation for achieving impressive results on many natural language processing tasks today @xcite @xcite but they are also prohibitively expensive to train, requiring enormous computing resources to be effective. the exploding demand of computations, together with the resulting massive energy cost @xcite , pose serious obstacles to the development of new pretrained models, thus, leading to a number of recent research efforts aimed towards addressing this problem by proposing approaches for improving model efficiency @xcite or sample efficiency @xcite , to name just a few. the primary method researchers have explored to address this problem is to develop smaller language models @xcite . in this paper we study a complementary approach to simplify pretraining of language models based on corpus ordering via curriculum learning @xcite . the motivation is that curriculum learning has been shown to help with faster convergence @xcite which, in combination with simpler language models, will give us a sustainable platform for future research into language models. although curriculum learning strategies have been successfully employed in many areas of machine learning, on a wide range of tasks @xcite , little is understood about the role of corpus ordering in the context of pretraining language models with the exception of some notable early works on language modeling @xcite . in this paper, we continue this line of investigation by asking the question whether curriculum based pretraining of transformer language models provides any benefits when compared with traditional vanilla training. in order to create a curriculum from an unlabeled text corpus for such self-supervised form of learning, one needs to define a measure of complexity. we explore if metrics of text readability difficulty designed for humans can serve as beneficial metric in creating a curriculum for machine learning models. the intuition behind this approach is to mimic the manner in which humans learn. training samples are organized by levels of difficulty and training proceeds in steps where the model is first trained on a subset of the corpus at a given difficulty level before being trained on another difficulty level, and so on. for our experiments, we pretrain from scratch multiple models of bert base using the wikitext-103 corpus @xcite , a collection of english articles from wikipedia. we explored standard length-based metrics of complexity as well as document-level complexity using flesch reading ease readability index to organize the corpora into a curriculum. we also investigated two different methods of accessing such an ordered training setone where the bins remain disjoint (binned) and another where the bins are incrementally joined together to increase the training data size (stepped) over n epochs. an extensive evaluation is conducted in terms of metrics related to not just pretraining but also fine-tuning on eight downstream tasks, specifically the suite of datasets from the glue benchmark @xcite . our results demonstrate that while sentencelevel and document-level complexity metrics work comparably well, they outperform the vanilla models in all experiments. furthermore, while easy to hard is a good strategy when complexity is computed at sentence-level, the reverse is true when complexity is measured at document-level. finally, the most encouraging results suggest that corpus ordering takes significantly less time (measured in terms of training steps) as compared to vanilla training, while yielding comparable results as measured by the average glue score. ## corpus complexity we begin by describing the unlabeled corpus of text used during training and the metrics adopted for measuring its complexity. corpus. in this work we choose the wikitext-103 corpus @xcite , which has served as a popular choice of text corpus for language modeling in prior works @xcite . it consists of a set of verified good and featured articles foot_0 from english wikipedia, containing a total of around 103.2 million tokens foot_1 . for all experiments, the predefined splits of the corpus are used. complexity. before we can order the unlabeled text instances in wikitext-103, we need to define some criteria for ranking them. designing a ranking criteria in a self-supervised setup is a challenging problem, especially due to the dual pretraining and fine-tuning paradigm. we choose to estimate sample complexity in terms of human-centered notion of 'difficulty', specifically text readability and lexical richness. in other words, we construct a heuristically informed pre-defined curriculum, and adopt metrics that measure the difficulty of text, based on readability and complexity, at two levels of granularity -sentence and document. the corpus is then divided into bins, from which batches of training input are constructed at random. ## corpus ordering for pretraining we first summarize the process of creating the curriculum and then explain the strategy for training over it. as described in the previous section, sentences or documents are sorted by their complexity score, which are then distributed into nonoverlapping bins, essentially subsets of data (also known as shards @xcite ), such that samples in each bin are similar in complexity. the training consists of t sequential phases, where t denotes the different points of time during the training, where training samples are fetched only from a subset of bins. for instance, @xmath0, @xmath1, and so on. in our experiments, t denotes an epoch. a subset consists of one or more bins, and for creating and iterating over these training subsets during training, we explore two different strategies -binned and stepped. the bins themselves can be accessed in order of either increasing difficulty (from easy to medium to hard), an approach that can be intuitively seen as mimicking the way humans learn, or in the reverse order of decreasing difficulty (from hard to medium to easy), a technique shown to benefit machine learning algorithms @xcite . in doing so, the question we ask is whether curriculum or anti-curriculum help in the context of language modeling, if at all. it is worth mentioning that while the bins are accessed in a pre-defined order (i.e., easy to hard or reverse), the samples within the bins are still randomly selected, thus combining a deterministic schedule with the benefits of randomization that serve neural models well. to sum it up, if we set a phase t to be an epoch, in the binned models, all the bins (and by extension, the entire corpus) are/is iterated over just once, whereas in the stepped models, since newer bins are progressively added in addition to existing bins in the training set, a process that not only modifies complexity but also increases data size over time, on average, this is similar to iterating over the entire corpus n 2 times. finally, the vanilla model, where the corpus is not sorted or binned in any way and data from the entire corpus is accessed randomly at any phase, serves as the baseline model. ## experiments and discussion in this section, we perform a series of experiments for evaluating the effects of corpus ordering strategies in the pretraining of language models. ## related work there is a growing awareness of the benefits of green ai @xcite , with an emphasis on not just accuracy, but also efficiency, with the desirable goal, amongst many, of decreasing the environmental carbon footprint of pretrained language models, during training or inference. tremendous progress is being made towards sustainable or 'greener' models with many efforts in improving model efficiency in order to reduce the amount of compute required @xcite @xcite @xcite @xcite . scaling laws @xcite suggest that optimal compute-efficient training involves training very large models on a relatively modest amount of data and indeed, many studies explore the relationship between the volume of training data and consequent model performance @xcite @xcite @xcite @xcite , generally concluding that rapid improvements in performance are observed as the amount of training data increases, at least until a certain point, after which the improvements slow down. in this work, our focus is on studying the role of corpus ordering in language modeling, and although a handful of works explored this idea via length-based metrics @xcite , they did not integrate document-level measures of complexity, nor did they study this in the context of transformer models. furthermore, we perform an extensive evaluation reporting metrics pertaining to both pretraining and fine-tuning stages. prior work has typically measured the complexity of text in terms of the length of a sequence @xcite @xcite @xcite @xcite . other metrics include features such as diversity, simplicity, and prototypicality @xcite , or the norm of a word embedding @xcite . the effects of these prior efforts have mostly been studied at the fine-tuning stage. at the pretraining stage, @xcite studied the effects of preprocessing training corpora in the context of affective tasks, while more recently, @xcite showed that initially training a model on shorter subsequences before moving onto longer ones has benefits in terms of training time and model perplexity, although they did not study the impact on downstream tasks. although in regular settings curriculum learning has often led to mixed results, under limited training budgets, curriculum has shown to improve the performance @xcite , which motivated us to explore the effects of corpus ordering in the task of transformer-based language modeling, and evaluating the metrics related to not just accuracy but also training efficiency as measured in the number of training steps. ## conclusions and future directions in this paper, we investigate several corpus ordering and training schedules as a way of exploring the effectiveness of curriculum learning, specifically with regards to obtaining better models and potentially reducing training time and/or cost, for pretraining transformer language models (english, in this case). between the two notions of complexity, one computed at sentence-level, and the other at document-level, we find them to perform relatively comparably. in the document-level curriculum, our findings suggest that going from hard to easy training samples may be an effective strategy. furthermore, iterating over disjoint bins one at a time seems comparably effective to incrementally increasing the training data size. finally, interestingly, our empirical results on eight down-stream tasks from glue benchmark reveal that an ordered corpus yields competitive performance as compared to vanilla training in almost half the number of training steps, as measured over two epochs of training. many interesting avenues for future work remain such as devising more efficient corpus ordering algorithms or verifying whether these simple yet effective strategies generalize to different training corpora or model architectures, thus enabling development of sustainable language models.