On the importance of pre-training data volume for compact language models (2020)

Abstract: Recent advances in language modeling have led to computationally intensive and resource-demanding state-of-the-art models. In an effort towards sustainable practices, we study the impact of pre-training data volume on compact language models. Multiple BERT-based models are trained on gradually increasing amounts of French text. Through fine-tuning on the French Question Answering Dataset (FQuAD), we observe that well-performing models are obtained with as little as 100 MB of text. In addition, we show that past critically low amounts of pre-training data, an intermediate pre-training step on the task-specific corpus does not yield substantial improvements.

## introduction over the past year, pre-trained language models have become the norm in natural language processing. these large-scale transformer-based @xcite networks considerably advanced the state-of-the-art in language understanding @xcite via a two-step process: self-supervised learning on a vast text corpus followed by fine-tuning on a specific downstream task. following these advances, the ongoing trend has been to build bigger models with an ever-increasing amount of data @xcite @xcite . however, pre-training models with billions of parameters over hundreds of gigabytes of text requires tremendous computational resources that only a few companies and institutions can afford. besides, many languages and specific corpora (e.g. legal, scientific) are currently under-resourced. hence, our goal is to explore model architectures and data volumes lowering the entry barrier to new research and practical applications. we conduct experiments on french corpora in order to release the first french compact language models and to illustrate the training process in another language than english. furthermore, we consider the question answering task since compact models may find their purpose in low-latency/faulttolerant information retrieval systems. ## problem statement we intend to study the impact of pre-training data volume when training compact bidirectional transformers @xcite . we assume a scarce resources setting, both in terms of data and computing power. two key aspects are explored: we use the french part of the oscar corpora @xcite for pre-training and the fquad dataset 1 (d'hoffschmidt et al., 2020) for machine reading comprehension fine-tuning. moreover, the models under consideration are based on the camembert @xcite ) language model. ## related work a wealth of work has recently been released @xcite on compressing transformerbased models @xcite through the pre-training of compact models @xcite , distillation @xcite @xcite , pruning @xcite @xcite and quantization @xcite . nevertheless, absolute performance is not the end goal of this study. rather, we investigate the training process of compact models in the absence of larger ones to distillate or prune. furthermore, @xcite acknowledge the difficulty of speeding up sparse models due to the absence of specialized hardware. therefore, from an inference speed standpoint, it is currently preferable to train compact models. language models have been successfully pretrained on domain-specific corpora @xcite and outperform their general-purpose counterparts on targetted downstream tasks. still, training these models involved large datasets and computational resources out of reach for most. multilingual models @xcite @xcite have been released to alleviate the need for languagespecific pre-training. while they offer competitive results, they usually lag behind monolingual models and require larger architectures. @xcite observed that large models did not improve on evaluation tasks when increasing the amount of pre-training data from 4 gb to 138 gb. they left as future work to question the need for large scale pre-training corpora with other model architectures and tasks. @xcite . in this work, we only extract a sample of 4 gb of shuffled lines. ## camembert small camembert @xcite ) is a multi-layer bidirectional transformer @xcite with two architectures: base (12 layers, 768 hidden dimensions, 12 attention heads, 110m parameters) and large (24 layers, 1024 hidden dimensions, 16 attention heads, 355m parameters). it is very similar to roberta @xcite . the main differences are the use of whole-word masking and sentencepiece tokenization @xcite instead of subword-masking and byte-level byte-pair encoding @xcite . roberta itself improves upon bert by aggregating several modifications on top of the original architecture such as removing the next sentence prediction task, dynamic masking, and training with larger batches on more data. we introduce camembert small foot_3 , a camembert-based language model with a small architecture (12 layers, 256 hidden dimensions, 4 attention heads, 17m parameters). the main difference with the original camembert lies in the use of subword-masking. indeed, the authors later found out that whole-word masking had apart from inference speed and size considerations, two main factors explain this architectural choice: ## experiments six overlapping subsets are built from the 4 gb os-car sample. they are denoted as osc 10 , osc 100 , osc 500 , osc 1000 , osc 2000 and osc 4000 (the numbers indicating the number of mb). we extract an additional 10 mb sample from the corpus, which serves as a validation set for the self-supervised pre-training task. on the other hand, fquad consists of a train/dev split of 50,741 and 5,668 question/context pairs. for each oscar subset, we pre-train a camembert small model with the standard masked language modeling (mlm) objective. then we fine-tune the pre-trained models on the question answering task with the same span prediction method as bert @xcite . between those two steps, an optional mlm step over the fquad train set is included. the experiments described were implemented using hugging face's transformers library @xcite and were conducted on an nvidia v100 16 gb. ## conclusion we investigated the importance of pre-training data volume when training compact transformer-based models. we made the observation that 100 mb of raw text are sufficient to reach similar performance as with larger datasets on a question answering task, and that corpus-specific self-supervised learning does not bring significant improvements on that particular problem. these preliminary results pave the way for further experiments with other language models, various architectures and new downstream tasks.