Exploring Transformers as Compact, Data-efficient Language Models (2023)

Abstract: Large scale transformer models, trained with massive datasets have become the standard in natural language processing. The huge size of most transformers make research with these models impossible for those with limited computational resources. Additionally, the enormous pretraining data requirements of transformers exclude pretraining them with many smaller datasets that might provide enlightening results. In this study, we show that transformers can be significantly reduced in size, with as few as 5.7 million parameters, and still retain most of their downstream capability. Further we show that transformer models can retain comparable results when trained on human-scale datasets, as few as 5 million words of pretraining data. Overall, the results of our study suggest transformers function well as compact, data efficient language models and that complex model compression methods, such as model distillation are not necessarily superior to pretraining reduced size transformer models from scratch.

## introduction in the space of a few years, transformers have revolutionized natural language processing. their success has been driven by increasingly large models and more training data. sizes of the most powerful language models have ballooned to billions of parameters and are pretrained with (in some cases) trillions of tokens of text @xcite . however, the size and data input requirements of transformers limit their reach as research tools in two key ways: first, training transformers usually requires access to powerful compute resources. for instance, the creators of the palm model @xcite , used 6,144 tpuv3 chips for pretraining. at the time of this writing, the on-demand cost of this much compute would be a little less than $20,000 per hour. 1 even the moderately sized bert @xcite model required 16 tpu chips for pretraining, putting such a task beyond the meager means of many researchers. costs this high make research on end-to-end pretraining impossible for potentially timely and impactful academic research @xcite . second, large models require pretraining with large datasets that can generally only be obtained from data extracted from the internet. bert, for instance, was trained on a 3.3 billion word webbased corpus. in contrast, datasets derived from other sources, human speech for instance, are necessarily much smaller and sometimes contain only a few million words. using data that is not based on internet text can offer insight into how the nature of language data affects language model performance. currently, such efforts to create language models from smaller, alternative data sources are of growing interest in computational linguistics @xcite . 2,3 most research for creating efficient transformers has focused on distillation, which trains a smaller student model using output from a large, pretrained teacher model @xcite @xcite . while these efforts have produced more efficient models, they require the same large datasets and the use of larger teacher models which themselves require ample compute power during training, even though the end goal is a smaller model. remarkably, there has to date been little research into simply reducing the size of transformers, pretraining them from scratch and fine-tuning them on downstream tasks. the process of increasing the size of transformer models and their data inputs are well explored @xcite . however, it is still an open question to what degree the transformer architecture can function as a lightweight, data-friendly research tool. in this paper, we offer a preliminary study toward addressing these issues. in contrast to previous studies that have approached these topics, we forego the use of knowledge distillation and other complex compression techniques. rather we pretrain various configurations of the electra @xcite transformer in search of parameter and data efficient models. we conduct all of our experiments using a single 12gb gpu to demonstrate the computational efficiency of the models we train. the main contributions of our study are: ## related work the excessive compute requirements of transformers has led to the creation of a sizable body of research into reducing their size and memory footprint. the most well explored strategy is knowledge distillation, a process whereby a full-sized teacher network is used to train a smaller student network. distilbert @xcite , tiny-bert @xcite , minilm @xcite and mobilebert @xcite are popular examples of compact transformers distilled using full sized bert models as teachers. these methods produce effective smaller models, however they don't directly address the amount of input data required and the training process still requires using a full-sized teacher model to train the student model. pruning is another popular model compression method in which some fraction of the trained model's parameters are set to zero. @xcite and @xcite use unstructured pruning methods to eliminate a large percentage of weights throughout transformer models with small corresponding reductions in performance. structured pruning methods such as @xcite set the parameter values of entire regions of the model to zero; in this case whole transformer layers are pruned. @xcite showed that a large percentage of bert's attention heads can be entirely removed before testing without a significant decrease in performance. however, these techniques are are all premised on pretraining full-sized models and then reducing the model size prior to inference time, therefore still have the same pretraining data and compute requirements. there has also been some research directly addressing the size of pretraining datasets for transformers. @xcite and @xcite experimented with reducing the absolute amount of training data in french language models. they showed that full sized french language transformer models can perform well on select tasks with significantly less pretraining data. @xcite and @xcite investigated the effect of different pretraining data volumes on the grammatical knowledge of the roberta-base model using probing techniques. huebner et al. (2021) experimented with using aochildes, the 5 million word dataset composed of child directed speech for pretraining and evaluated their results using a grammatical benchmark based on blimp @xcite . this study is notable because the authors used a very small pretraining dataset derived from human speech and opted to use a scaled-down version of the roberta model @xcite to accommodate it. unfortunately, the resulting model was only tested on narrow set of grammatical learning tasks, using a specialized dataset for evaluation. ## language model: electra-small in this section we describe the electra-small model @xcite and the rationale behind using it as the basis for our experiments. in place of masked language modeling, electra pretrains a transformer encoder stack, structurally identical to bert's, by replacing some input tokens with plausible alternative words sampled from a small generator network. a larger discriminator model then predicts whether or not each input token has the electra-small model also has the advantage of beginning with only 13.6 million parameters and performs favorably to similarly sized models. further, it can be pretrained without the use of model distillation. as the purpose of our study is to train transformers in data and resource scarce settings, it is desirable to use a model that doesn't require a teacher model pretrained on a massive text corpus, and can be trained on a single gpu. to ensure that electra-small can produce results on par with other compact models we use the pretrained models from the huggingface transformer hub and test them on our selected downstream tasks. 4 the results are summarized in ## experiment 1: reducing individual model dimensions in a low data setting in the first set of experiments that we conduct, we test varying the size and configuration of the elec-tra model using the 5 million word subset of openwebtext described in section 3.1 as the pretraining dataset for each model variation. we begin by changing only a single dimension of the model's configuration. the goal of this series of experiments is to determine which parts of the model's architecture can be reduced and what effect these reductions have on performance. in the process, we hope to provide some insight into how the size of each dimension of the transformer model contributes to its downstream performance. ## experiment 2: reducing overall model size in a low data setting guided by our results from the previous experiments, we now aim to find an overall configuration of electra-small that has significantly fewer parameters than its default of 13.6 million and retains most of its downstream performance. using the same 5 million word dataset, we train models with any number of their dimensions reduced or modified. in essence, we trained and evaluated a large number of models with various combinations of our most successful modifications from our previous experiments in search of a robust and well functioning model. we found early in our efforts that simply reducing model size while keeping the dimensions proportionate produced poor results. our results from the previous section suggest that this may be due to the models low tolerance for decreases in its hidden size. we did however find several alternative configurations with parameter counts that ranged from 5.7 to 10 million that retained most of the performance of the full-sized electra-small though we trained several dozen model variations, we present only our 5 most succesful. the dimensions and parameter counts of these models are described in procedure as in our previous experiments where we altered only a single model dimension, we pre-train all of our models using the 5 million word subset of openwebtext for 100,000 steps. we evaluate our models using the same metrics and hyperparameters as the previous experiment in order to compare our results. ## experiment 3: reducing model size in a high data setting in this experiment, we pretrain a selection of models using the full openwebtext corpus as the pretraing dataset and training for a million steps. we use the same five models described in because the training times in this experiment are much longer, we will not repeat the exhaustive study of the effects of changing individual model dimensions as we did in the low data setting. rather, we only pretrain and evaluate the 5 models considered in experiment 2. the goal of this experiment is determine to what degree the results of experiments 1 and 2 will hold with a full-size dataset trained for an extended time. given that the models considered contain so few parameters, it is a natural question as to whether or not they can adequately make use of the additional information provided by more data and longer pretraining. the results of this experiment will also be more readily compared to other compact transformers which are also trained on full-sized datasets. we use the same evaluation criteria as that performed in experiments 1 and 2. ## conclusion in this study, we have shown that the transformers, specifically electra, can function as compact data-efficient models. our results suggest that when training with small datasets, the intermediate size, embedding size and number of layers can all be reduced with little ill-effect. additionally, we presented the glue results for 5 model variations that significantly reduce the overall size of the electra-small model. in the final phase of our experiments, we tested the same five models trained with the full openwebtext corpus. we showed that several compact transformer architectures can function on par with larger models trained using complex distillation methods. finally, we present a compact configuration of electra we call electra-tiny with just 5.7 million parameters that performs remarkably well on the glue benchmark given its small size, requires less compute and can be trained end to end on a single 12gb gpu.