Pre-training a BERT with Curriculum Learning by Increasing Block-Size of Input Text (2021)

Abstract: Recently, pre-trained language representation models such as BERT and RoBERTa have achieved significant results in a wide range of natural language processing (NLP) tasks, however, it requires extremely high computational cost. Curriculum Learning (CL) is one of the potential solutions to alleviate this problem. CL is a training strategy where training samples are given to models in a meaningful order instead of random sampling. In this work, we propose a new CL method which gradually increases the block-size of input text for training the self-attention mechanism of BERT and its variants using the maximum available batch-size. Experiments in low-resource settings show that our approach outperforms the baseline in terms of convergence speed and final performance on downstream tasks.

## introduction recent years have seen a series of breakthroughs in pre-trained language representation models. the development of pre-training methods like bert @xcite and its variants @xcite have led to large improvements in many down-stream tasks such as paraphrase identification, sentence textual similarity, sentiment analysis, and natural language inference. one of the advantages in training these models is that they can leverage the unlabeled large-scale corpora which are more available compared to the labeled ones. however, training these models with large-scale corpora is pretty expensive in terms of computational time and memory footprint. in the literature, there are three main approaches that have been adopted to address this problem. these are architecture-based approach @xcite @xcite @xcite , task-based approach @xcite and dataset-based approach @xcite @xcite . while the architecture-based and taskbased methods have been extensively studied in the context of pre-training methods for natural language processing (nlp), dataset-based approach is relatively unexplored. to this end, we adopt a dataset-based method called curriculum learning (cl) which controls the order of training samples so that the model might converge faster with better performance. the idea of cl-like approach was originally proposed by @xcite . the idea is based on the actual learning mechanism of humans and animals, where basic concepts are acquired first, then more complex ones are gradually learned. @xcite formalized this concept as cl to train neural networks. through experimental analysis, @xcite showed the benefit of cl on convergence speed and performance in shape recognition and language modeling tasks. one of the most significant challenges when adapting cl to a new task is to figure out a criterion for measuring the difficulty of the training samples. for example, in object recognition task, the size of objects is a good measure of difficulty @xcite , and the presence of low-frequent words in input text is an indicator of difficulty in language modeling @xcite . these criteria vary greatly depending on the task, thus, it is not easy to define a measure of difficulty which is suitable for a particular task. most studies in the field of cl for nlp have proposed variety of difficulty measure by leveraging heuristics of the target tasks with neural networks @xcite @xcite @xcite . on the other hand, it is not clear how to design cl for language representation models such as bert. in pretraining bert, distributed word representations are learned through optimizing masked language modeling (mlm) loss which is computed by predicting a masked word or token in an input text. the input to the model is not a single sentence but an arbitrary-length span of text called a block. this indicates that it is not obvious how to measure the difficulty of the training samples using cl-based approach proposed in the previous studies. the key component of bert is the multi-head self-attention mechanism that learns to compute token embeddings from its context @xcite . the multi-head self-attention mechanism can be thought as a problem of searching for important token-pairs based on the relative magnitude of attention among all the token-pairs in an input text. this process can be served as a clue which leads us to speculate that it might be possible to formulate cl strategy by focusing on the effective training of the self-attention mechanism in bert. although each individual head of the multihead self-attention mechanism can learn any dependency among tokens, most of the heads tend to pay more attention to local dependencies than global ones @xcite @xcite . it could be easier to train local self-attention in shorter blocks of input text than global self-attention in longer ones. therefore, the block-size of input text can be used as the effective criterion to measure the difficulty-level of training samples for bert. in this paper, we introduce a new cl method which gradually increases the block-size of input text for pre-training bert using the maximum available batch-size to accomplish convergence speedup, and also improve performance in the down-stream tasks. since our approach is very simple, it is easy to apply it to bert and its variants with little effort. using a small-scale corpus, the experimental results demonstrated that our proposed approach outperforms the baseline on glue tasks with faster convergence speed. ## related work to reduce the memory footprint and improve the training speed of pre-trained language models, prior works have shown that architecture-based approaches are very useful. @xcite proposed to leverage knowledge distillation to train a smaller version of bert with faster training speed while maintaining comparative performance. @xcite used factorized embedding parametarization and cross-layer parameter sharing, which led to the reduction of parameter size and training time. de wynter and perry (2020) applied neural architecture search to select the optimal architecture of bert and successfully compressed the size of the model. task-based approaches have also been explored for pre-training language models with high training efficiency. @xcite introduced permutation language modeling which retains the benefits of autoregressive models and allows the models to capture bidirectional context. instead of performing pre-training with mlm task, @xcite trained a bert as a discriminator that determines whether each corrupted token was replaced by a generator model. recent studies have shown that cl is a successful approach for a wide range of machine learning applications @xcite , including the fine-tuning of large-scale language models such as bert @xcite . some of large-scale language models like gpt-3 (brown et al., 2020) and t5 @xcite adopted non-uniform mixing strategies which control the amount of training samples from multiple corpora. however, cl strategy has not directly been applied to pre-training large-scale language models. there exists many studies of cl which used the length of sentences or input sequences as a measure of difficulty in nlp tasks including neural machine translation (kocmi and bojar, 2017), sentiment analysis @xcite , parsing @xcite , poem generation @xcite and reading comprehension task @xcite . in this work, we exploit the block-size of input text in the context of selfattention mechanism as a measure of difficulty for pre-training bert. ## method the overview of the proposed cl method is presented in figure 1 . the method is divided into two stages: (a) splitting a corpus based on specific block-sizes and (b) gradual training of bert by increasing the block-size. in the first stage, we split the original corpus into a series of input blocks with the pre-defined length. in the second stage, we train a model by changing the training samples from the short block-size to the long one depending on the pre-defined number of training steps. in training, some tokens in a block are randomly masked to perform the mlm task. we describe the mlm task and the details of the two stages of our cl approach in this section. ## experiments in the experiments, we evaluate our proposed cl approach in terms of the convergence speed and model performance. we use wikitext-2 (merity et al., 2016) for pre-training roberta @xcite , which is a variant of bert. for fine-tuning on down-stream tasks, we use the general language understanding evaluation (glue) dataset @xcite . all the training and finetuning were carried out on a geforce rtx3090 with 24gb memory. ## conclusion in this paper, we proposed a new cl method for pre-training bert, which progressively increase a block-size of input text. our approach is very simple and thus handy to implement. experiments in the low-resource setting have shown that proposed method leads to faster convergence speed and better performances in down-stream tasks. in further research, we expand the corpus and validate the scalability of our approach. in addition, we speculate that it is important to investigate when the difficulty-level should be changed through the training and how it affect model performances.