Modeling Easiness for Training Transformers with Curriculum Learning (2023)

Abstract: Directly learning from complex examples is generally problematic for humans and machines. Indeed, a better strategy is exposing learners to examples in a reasonable, pedagogically-motivated order. Curriculum Learning (CL) has been proposed to import this strategy when training machine learning models. In this paper, building on Curriculum Learning, we propose a novel, linguistically motivated measure to determine example complexity for organizing examples during learning. Our complexity measure - LRC- is based on length, rarity, and comprehensibility. Our resulting learning model is CL-LRC, that is, CL with LRC. Experiments on downstream tasks show that CL-LRC outperforms existing CL and non-CL methods for training BERT and RoBERTa from scratch. Furthermore, we analyzed different measures, including perplexity, loss, and learning curve of different models pre-trained from scratch, showing that CL-LRC performs better than the state-of-the-art.

## introduction pre-trained transformers are sweeping away all other methods of natural language understanding. these models outperform all previous methods and sometimes even humans in many nlp tasks @xcite @xcite . pre-training on unlabeled large-scale corpora seems to be the way that increases performance @xcite . for example, bert is pre-trained on an english corpus of 3.300 million words consisting of books @xcite and wikipedia. however, training these models with large corpora is quite expensive in terms of computation time and memory. the problem of optimizing the computational resources that transformers need is tackled in three main ways: by re-modeling pre-training tasks @xcite , by studying techniques to produce lighter architectures @xcite , and by working with data @xcite @xcite . architecture-level and model-level optimization techniques have been extensively studied in the context of pre-training methods for nlp. data-level approaches have yet to be explored. to this end, we adopted a data-level strategy called curriculum learning (cl), which stems from the complexity of training samples so that the model can achieve better performances. starting from the idea for which humans and animals acquire first elemental concepts and then, gradually, more complex ones, @xcite proposed cl and demonstrated its benefits in shape recognition. this approach presents training data in order of difficulty, starting with easy examples and increasing the degree in parallel with learning. the application of cl in pre-trained language models (plms) has limitations. one of the most critical challenges is to find a criterion for measuring the difficulty of training samples. in supervised tasks, sorting training batches by length and repetitiveness of certain patterns paid off @xcite . in the semisupervised plms, word representations are learned by optimizing loss in the masked language modeling tasks using a set of contiguous blocks of fixedlength text. @xcite proposed a cl strategy focused on training the self-attention mechanism from shorter blocks to longer ones. this is because each head of this mechanism seems to be more attentive to local dependencies than global ones @xcite @xcite . in this paper, building on curriculum learning, we propose a novel, linguistically motivated measure to determine example complexity. this measure -lrc-is based on length, rarity, and compre-hensibility and sorts text complexity into blocks that increase in dimensionality gradually during pre-training of bert and its variants. moreover, by exploiting the organization of the example, our method avoids the loss of context common in standard cl methods applied to plms @xcite . using a small-scale corpus, experimental results demonstrated that our approach outperforms the other methods on glue tasks, and it requires fewer examples to achieve the same results. finally, we showed that cl-lrc achieves sustainable performance compared to cl in terms of perplexity, loss, and learning curves of the different models pre-trained from scratch. ## related works the main studies for optimizing computational resources and increasing the learning capabilities of pre-trained language models (plms) are architecture-based, learning model-based, and, finally, data-driven. although previous works have demonstrated the functionality of architecture-level and model-level approaches, they still need to improve. @xcite , have introduced permutation language modeling that allows models to capture bidirectional contexts and has performed well on long-dependency contexts but requires more data and computational resources to train and deploy. @xcite , have reduced computational costs by modifying the traditional mlm with a discriminator that, in turn, could have limitations in tasks that require a deep understanding of long-term dependencies or complex relationships between words and concepts. @xcite ; @xcite have used parameter reduction techniques and have achieved a light version of bert that is faster and more lightweight but is not as effective as bert in tuning parameters on specific tasks. @xcite have improved bert pre-training by introducing dynamic masking in the mlm task and eliminating the nsp task. these structural changes are the key to increasing the model's performance in downstream tasks, but more data are needed to achieve the same results than in the pre-training of bert. the performance achieved by optimization at the architecture and training levels is a difficult point of resistance to overcome. while these topics have been extensively studied in the context of plms, the data-level approach still needs to be explored. although numerous variants of bert succeed in fixing some critical aspects of pre-training, there open up many gaps at the computational and performance level on downstream tasks. many studies have found that the multi-headed self-attention mechanism requires more computational effort. since each head of this mechanism seems to be more attentive to local dependencies than global ones @xcite @xcite , training local selfattention in shorter blocks seems to be less complex than training global self-attention in longer blocks. therefore, using the size of the input text block is key to measuring the difficulty level of the training samples. for these reasons, @xcite have proposed a curriculum learning (cl) strategy focused on hands-on training of the selfattention mechanism. in particular, they applied the strategy directly to bert pre-training, exploiting the input text block size in the context of the self-attention mechanism as a measure of difficulty for bert pre-training. beyond the world of plms, many studies on cl have used sentence length, external resources, or input sequences to measure difficulty in various nlp tasks. @xcite have proposed a clbased method for parsing tasks. @xcite have proposed a text length-based method on no transformer-based models for tasks of neural machine translation. while @xcite also included the rarity of some terms by applying the method for the reading comprehension task. @xcite propose a gradual masking mechanism of concepts for pre-training the language model that obtains impressive results but is tied to the knowledge graph. in this paper, we propose text complexity techniques coupled with input text block size in the context of the self-attention mechanism. the two approaches are used to measure the difficulty of bert pre-training. our proposal adds a further light step where pre-training text complexity is computed to the incremental cl proposed in @xcite . our model achieves higher performance than other methods on downstream tasks. ## methods since language has a structure, organizing examples during pre-training can improve model performance. curriculum learning (cl) is a training method based on the idea that training algorithms can achieve better results when training data are presented in accordance with the model's current skills. we propose cl-lrc that adds to the standard cl, a measure used to determine example complexity during the pre-training (see figure 1 ). the cl-lrc method consists of three phases: (a) sorting the corpus according to our complexity measure, (b) partitioning the corpus according to specific block sizes, and (c) gradual pre-training by increasing block sizes. firstly, we sorted the corpus by complexity measure, starting with the less complex sentences to more complex ones (section 3.2). secondly, we split the sorted corpus into a series of input blocks of predefined length (section 3.3.2). finally, we trained a model by shifting the training samples from the short block-size to the long one, depending on the predefined number of training steps (section 3.3.3). pre-training was done by masking some block tokens randomly, as precedes the masked language modeling (mlm) task @xcite . in this section, we describe the mlm task and the details of the three phases of our cl-lrc approach. ## experimental results and discussion in the experiments, we evaluated our proposed cl-lrc approach in model performance. therefore, we show that performances increase to the proposed state of the art in @xcite . in order to reproduce the results proposed in previous work, we used wikitext-2 @xcite for pre-training bert @xcite and roberta @xcite . for fine-tuning downstream tasks, we used the famous general language understanding evaluation (glue) dataset @xcite . this choice was made to have terms of comparison with state of the art and for ease of retrieval in the huggingface library @xcite . finally, we performed an ablation study, perplexity, loss, and learning curves on different subsets of the dataset. all experiments were performed on two nvidia rtx a6000 with 48 gb of memory. the code and model will be released for further research. ## conclusion in this paper, building on curriculum learning, we propose a novel measure, -lrc -, to determine example complexity. this measure is applied during pre-training to sort the corpus according to complexity. experiments conducted in a low-resource environment have shown that the proposed method leads to better performance in downstream tasks and may be used to reduce the data needed for reasonable performances. furthermore, this approach is straightforward and thus easy to implement. in further research, we will expand the corpus and validate the scalability of our approach. in addition, it is important to continue investigating different complexity metrics that could be modified during pre-training and their impact on model performance.