Bootstrapping Small & High Performance Language Models with Unmasking-Removal Training Policy (2023)

Abstract: BabyBERTa, a language model trained on small-scale child-directed speech while none of the words are unmasked during training, has been shown to achieve a level of grammaticality comparable to that of RoBERTa-base, which is trained on 6,000 times more words and 15 times more parameters. Relying on this promising result, we explore in this paper the performance of BabyBERTa-based models in downstream tasks, focusing on Semantic Role Labeling (SRL) and two Extractive Question Answering tasks, with the aim of building more efficient systems that rely on less data and smaller models. We investigate the influence of these models both alone and as a starting point to larger pre-trained models, separately examining the contribution of the pre-training data, the vocabulary, and the masking policy on the downstream task performance. Our results show that BabyBERTa trained with unmasking-removal policy is a much stronger starting point for downstream tasks compared to the use of RoBERTa masking policy when 10M words are used for training and that this tendency persists, although to a lesser extent, when adding more training data.

## introduction large-scale pre-trained language models (lms) @xcite @xcite have shown promising ability on handling various downstream tasks including textual classification @xcite and question answering (qa, @xcite . previous research @xcite showed that the size of architecture and amount of pretraining data actually affect the linguistics features learned by state-of-the-art (sota) pre-trained lms like roberta @xcite . it also showed lms require much more data to understand com1 our code can be found in https://github.com/ yangy96/babyberta_continual monsense knowledge in order to achieve high performance on natural language understanding tasks, compared to grammatical ability. on the other hand, lms like roberta are costly to train in terms of gpu computation power and time at both pre-training and fine-tuning stages. @xcite proposed babyberta, a smaller roberta architecture that is trained on a 5m child-directed data corpora without using unmasked tokens during the masked language modeling training. baby-berta reaches the same level of grammaticality as roberta but considerably saves training expenses. however, no further evaluation on tasks other than the grammaticality tests were performed for the model. @xcite also focus on the performance of smaller language models but emphasize the relationship between the architecture size and downstream task performance and train on larger data corpora. in this paper, we would like to answer the following questions: (1) what is the performance for lms based on smaller models like babyberta on downstream tasks that require fine-tuning? and (2) what is an efficient way to improve the behavior of small pre-trained lms on downstream tasks? in our work, we first evaluated both babyberta and roberta on three downstream tasks that target sentence structure and are closely associated with grammatical capabilities. additionally, we propose to have various starting points by combining different ingredients in pre-training including the masking policy, the size of the vocabulary, and the type of the pre-training data corpus (childdirected language, online written language). then, we continually pre-train babyberta and its variants on more wikipedia data to improve performance on target tasks. we observe that: (1) although babyberta has a lower performance on the downstream tasks compared to roberta, the use of the unmasking removal policy and of a small vocabulary is still effective after fine-tuning; (2) running thorough experiments to identify which factors are important for small language models when performing continual pre-training, we find that the influence of the unmasking removal policy persists, although to a lesser extent, when adding more training data. ## babyberta on downstream tasks in this section, we evaluate babyberta models on various downstream tasks and experiment with different methods including continually pre-training. to perform question-answering tasks like qamr and qasrl, we train two linear layers on top of the encoder of the language model (lm) to predict the start and end of the answer span within the context. we implement the classifier using huggingface @xcite . for fine-tuning lms for srl tasks, we utilize the implementation provided in @xcite . ## conclusion in this work, we investigate three important factors for improving smaller language models on downstream tasks: vocabulary set, masking policy, and dataset at the starting point. our findings reveal that continuously pre-training a smaller model like babyberta leads to continued improvement in downstream performance. additionally, employing the unremoval masking policy and utilizing a smaller vocabulary prove advantageous for downstream tasks. we provide a comprehensive investigation into the relationship between pre-training procedures and downstream tasks for small models. in future research, we aim to delve deeper into the abilities acquired during the pre-training stage and their impact on downstream task performance.