Trained on 100 million words and still in shape: BERT meets B ritish N ational C orpus (2023)

Abstract: While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, well-balanced, and publicly available English text source – the British National Corpus. We show that pre-training on this carefully curated corpus can reach better performance than the original BERT model. We argue that this type of corpora has great potential as a language modeling benchmark. To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evaluate several training objectives and model architectures and replicate previous empirical results in a systematic way. We propose an optimized LM architecture called LTG-BERT.

## introduction in the pursuit of state-of-the-art performance, nlp practitioners utilize increasingly larger amounts of data to pre-train language models, making it difficult to disentangle the improvements made by the proposed modeling choices themselves. instead, our aim is to shift the focus towards more efficient language modeling on a small and standardizable pre-training corpus. we study the data efficiency of current language models on an openly available corpus of approximately 100m words -incidentally the estimated amount of words processed by humans before adulthood @xcite . the goal of this paper is not to rival the paradigm of 'massively pre-trained language models'; instead we would in this work like to pursue a complementary direction of language modeling, which will hopefully lead to more interest in data-efficient language models. in particular, our contribution in this paper is twofold -we show that: large language models are notoriously data hungry, requiring hundreds of gigabytes of raw textual data. this becomes a major obstacle for lowresource languages while also putting a limit to the efficiency of any 'efficient' language model. on top of that, the size of web-crawled corpora makes it almost impossible to control their content and to prevent learning from harmful or copyrighted text @xcite . the british national corpus (bnc; @xcite is a 100-million-word reference corpus, manually curated to cover most aspects of 20 th century british english. massive language models are often pre-trained on nonpublic filtered collections of web-crawled text, which makes any reproduction impossible. we pre-train our models on a small and publicly available corpus, which allows for a replicable comparison of different language modeling configurations and which can be easily utilized in future research of novel variants of language models. we also release the pre-processing scripts, training scripts as well as the final model checkpoints. 1 previously, language models have been pretrained on different corpora tokenized by different tokenizers and fine-tuned by increasingly complex learning methods. this makes any comparison of the underlying neural architectures and pre-training objectives unfair. we make the language models in this paper directly comparable by fixing the training corpus, the tokenizer and the evaluation methods, while keeping them as simple as possible. ## related work the data requirements of language models have been growing in orders of magnitude since their early stages @xcite . taking a huge leap towards more recent work, elmo (embeddings from language models; @xcite were the first to introduce deep contextualized embeddings of words. recognizing the need of a large text corpus for this task, elmo was trained on the 1b word benchmark @xcite . later, bert (bidirectional encoder representations from transformers; @xcite further advanced the performance of contextualized embeddings when it based the entire language model on the transformer architecture @xcite . another important aspect of bert is that it was trained on a larger corpus than elmo: about 3.3b words from crawled english wikipedia and bookcorpus @xcite . to our best knowledge, the exact version of neither of the two subcorpora is publicly available. 2 the issue of limited replicability has become even more pronounced with later large language models: xlnet @xcite was trained on 33b words, roberta @xcite on more than 30b words and gpt-3 @xcite on an approximately 400b word corpus. none of these datasets is available; the authors utilize non-trivial filtering algorithms on openly available web crawls but do not release the end product nor the filtering source code. the effect of corpus size has been thoroughly studied in @xcite as well as in @xcite . they test differently sized random subsets of a bert-like corpus (crawled wikipedia and smashwords) and of a massive webcrawled text corpus (massivetext; @xcite , respectively. unlike them, we evaluate the effect of training on a small corpus, which was carefully curated to create a representative sample of english. the british national corpus is arguably more diverse and informative than a random subset of a web crawl -hence we test how the quality of a pre-training corpus influences the downstream performance, not only how the data quantity matters. we believe this aspect is vital for the future research of effective and reliable language models. 2 bookcorpus @xcite is not available anymore and the authors of bert do not specify what version of wikipedia dump they used or how did they preprocess it ( https://github.com/google-research/ bert#pre-training-data ). 3 british national corpus we use the british national corpus (bnc) as a diverse, balanced, compact, and publicly available monolingual english corpus. bnc is comprised of both written and spoken language with a total of 100 million words. the manually curated content contains a wide range of british english from the late 20 th century -newspapers, journals, books (academic and fiction), letters, essays, unscripted informal conversations or transcribed business meetings, radio shows or phone calls. the written part makes up approximately 90% of the corpus and the remaining 10% contains the transcribed speech. the sources are truncated to contain at most 45 000 words to ensure greater diversity within the limited amount of 100 million words. creation. the process of creating the bnc is extensively described in its documentation on the website. 3 it was created by the so called 'bnc consortium' led by oxford university press, and including major dictionary publishers longman and larousse kingfisher chambers; academic research centres at oxford university computing services, the university centre for computer corpus research on language (ucrel) at lancaster university, and the british library's research and innovation centre. the purpose of the british national corpus project was to construct a balanced and representative sample of current british english at the time. it was created over a period of four years and was a result of careful planning and data selection across a number of selection criteria (domain, time, medium, level) with proportions in the corpus designed to reflect the proportions found in real language use. it is widely acknowledged that the bnc has been a major influence on the construction of language corpora @xcite . one downside of the bnc is that it does not re-flect anything occurring to english language and the world in the 21st century, but still no better alternatives of the same size and quality exists. in addition, bnc was used as a model for creating representative corpora for other languages: e.g., turkish @xcite . version. we use the third release of the corpus, bnc xml edition (2007), which is the final revision of the texts compiled from 1991 to 1994 @xcite . the xml edition did not get any additional content on top of the original text samples, but it got some minor corrections, more metadata and it is supplied in a convenient xml format. ## model architecture we slightly depart from the typical post-norm transformer architecture @xcite used by bert @xcite , as illustrated in figure 1 . preliminary experiments with this model showed that it tends to unpredictably diverge in the later stages of training. this behavior has been noted in previous work on large lms @xcite and accordingly, we follow some of the recent improvements of transformer. normformer. pre-norm variation of the transformer has been shown to lead to more stable convergence with slightly degraded performance @xcite . @xcite claimed to mitigate this degradation by introducing an additional layer normalization operation. for these reasons, we decided to use their so-called normformer architecture to stabilize the training. 5 geglu activation function, proposed in shazeer (2020), enhances the expressiveness of the original transformer feed-forward modules by redefining them as ## training objectives the fixed corpus, tokenizer and fine-tuning procedures establish a controlled test bed for a comparative study of training objectives proposed in the past. the original bert model is trained via two self-supervised training objectives -masked language modeling (mlm) and next sentence prediction (nsp). we evaluate five different configurations of these objectives (three for mlm and two for nsp), as further detailed below. ## evaluation metrics we use three conceptually different methods for evaluating the amount of linguistic knowledge acquired by the bnc language models. 1) the (super)glue datasets test the ability of the model to adapt to various nlu tasks by further optimizing the whole pre-trained model, 2) edge probing tasks evaluate how much linguistic information one can extract from a frozen pre-trained model and 3) blimp utilizes the intrinsic ability of the pretrained network to model language and probes its knowledge without any additional training. we further elaborate on each of these below. ## experiments we conduct a number of experiments in this section. first, we compare different training hyperparameters and model configurations described in section 4. then, using the overall best training setting, we make a comparative study of training objectives (section 5). finally, we investigate the sampling efficiency of our proposed language model and we compare bnc with a wikipedia & bookcorpus subset of the same size. these results can then be used as a baseline performance of bnc-bert in future studies. the central model used in the experiments is a base-sized transformer -12 encoder layers with hidden size 768 and 12 attention heads (more details in appendix e). all reported models utilize the same cased wordpiece tokenizer @xcite with a vocabulary size of 2 14 = 16 384 trained with the bnc dataset (appendix g). this goes against the trend of increasing the subword vocabulary in recent work, foot_9 but a larger vocabulary size would lead to a lot of infrequent tokens within our limited corpus -we roughly follow gowda and may (2020) and '. . . use the largest possible bpe vocabulary such that at least 95% of classes have 100 or more examples in training. @xcite since our aim is to train models comparable to bert base , we train for the same amount of sampled tokens. @xcite trained on 1m batches of 128k tokens, we use 31 250 training steps with batch size of 4m tokens to parallelize and accelerate the process. also, similarly to @xcite , we use sequence length of 128 tokens in the first 90% of training and a larger sequence length of 512 only in the last 10% of steps. we deliberately do not compare against more recent models, which are trained for much longer to achieve slightly bet-ter performance: roberta is trained on 16× more training samples, for example. 11 ## conclusion in this paper, we evaluated how data-efficient masked language models can be. in particular, we trained a variety of models with different training objectives on the same training data: british national corpus. although small by modern standards (100m tokens), it is well balanced and carefully crafted to represent british english of the 20 th century. on a variety of benchmarks, our models perform better than bert base trained on a much larger corpus. we believe that this limited data regime is beneficial for the development of efficient and reliable language models. our finding also suggests that 100 million word tokens is enough to learn basic linguistic skills by current language modeling techniques, given that the data is carefully selected and balanced. to conclude, huge amounts of training data are not always necessarywe should focus on more efficient training settings instead. we showed that the next sentence prediction objective does not improve bert-like models, confirming the findings in @xcite . in addition, the standard subword masking from @xcite is consistently outperformed by the span masking method and the linguistic performance can be substantially increased by utilizing better neural architectures and training configurations. we release the code for training and using bert-like models with the optimal architectural choices (according to our experiments) under the name ltg-bert. 13 13 https://github.com/ltgoslo/ltg-bert the presented results serve primarily as the foundation for future research on efficient language modeling. we hope our work shows the value of careful curation of representative corpora and will spark more interest in this area, where bnc can serve as an undemanding, replicable and openlyavailable training corpus.