Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale. (2023)

Abstract: In recent years, language models have drastically grown in size, and the abilities of these models have been shown to improve with scale. The majority of recent scaling laws studies focused on high-compute high-parameter count settings, leaving the question of when these abilities begin to emerge largely unanswered. In this paper, we investigate whether the effects of pre-training can be observed when the problem size is reduced, modeling a smaller, reduced-vocabulary language. We show the benefits of pre-training with masked language modeling (MLM) objective in models as small as 1.25M parameters, and establish a strong correlation between pre-training perplexity and downstream performance (GLUE benchmark). We examine downscaling effects, extending scaling laws to models as small as ~1M parameters. At this scale, we observe a break of the power law for compute-optimal models and show that the MLM loss does not scale smoothly with compute-cost (FLOPs) below 2.2 \times 10^{15} FLOPs. We also find that adding layers does not always benefit downstream performance.Our filtered pre-training data, reduced English vocabulary, and code are available at https://github.com/text-machine-lab/mini_bert github.com/text-machine-lab/mini\_bert

## introduction in the past few years, large language models (llms) have grown ever larger @xcite @xcite , and the emergent abilities of these models improve with scale. while several studies have looked at the relationship between model size, the amount of training, and performance for llms @xcite , the main focus has been on scaling laws for highcompute settings. very few studies have considered the effects of pre-training at a smaller scale @xcite . thus, the faded gray lines are unique model configurations. the green dots are compute-optimal instances i.e., the model instance with a minimum value of mlm loss for a given range of flops. we observe that there exists a discontinuity in the power curve around 2.2 × 10 15 flops question of when exactly model abilities begin to emerge remains largely unanswered. in this study, we were interested in understanding whether the emergent phenomena can be observed at a drastically reduced scale, and what the relationship is between upstream and downstream performance at this scale. we also wanted to examine model shapes, configurations, and other factors that might affect whether we see the benefits of pre-training when downscaling a model. smaller models have been shown to do poorly when trained even on large volumes of data @xcite , which makes studying downscaling non-trivial. however, during language acquisition, humans are exposed to a reduced-size language before gradually expanding their vocabulary, yet they become fluent even when their vocabulary is limited. taking our cue from humans, we explore the hypothesis that reducing language size might allow us to observe the effects of pre-training in small models. there has been one previous attempt to reduce language size @xcite , but it was quite limited: one reduced-size transformer encoder was trained with a non-standard version of masked language modeling (mlm) loss on a relatively small corpus of child-directed speech and evaluated for its ability to pass linguistic tests from a custom grammar test suite. we use a vocabulary of 21,000 words derived from ao-childes (huebner and willits, 2021), a corpus of child-directed speech, to create a filtered corpus containing a subset of standard pre-training corpora: c4 @xcite , wikipedia, book corpus @xcite , and others. we pretrain over 70 transformer encoder models in the 1-100m parameter range, varying model shape, and configuration, including the number of layers, hidden size, the number of attention heads, and the feed-forward layer dimension (intermediate size). we fine-tune and evaluate a series of checkpoints at different flops count on the subset of glue filtered with the same vocabulary. we present evidence that for a realistically downscaled language, the benefits of pre-training are observable even in smaller models with as few as 1.25m parameters. our results also indicate that models with fewer layers achieve better performance on glue tasks. in contrast to @xcite , we find a strong correlation between upstream and downstream performance (here, model perplexity and glue score). however, pre-training compute optimality does not appear to be crucial for downstream results. we also show that for compute-optimal models at this scale, parameter count does not reliably predict mlm loss, suggesting a limitation to scaling laws. we observe a departure from the flops-perplexity law, characterized by a sudden shift in the exponent value in the low-compute region of @xmath0.2 × 10 15 (cf. figure 1 ). this represents a divergence from previous observations regarding scaling laws @xcite . changing pre-training data size via training token volume @xcite , vocabulary size @xcite , or the number of epochs @xcite has also been explored for effects on language acquisition. these studies have generally demonstrated low-level linguistic tasks involving syntax require relatively little data volume compared to more complex tasks like winogrande @xcite . the relationship between model size, input data, and downstream performance remains the subject of much debate as to the nature of scaling laws. @xcite concluded data size and model size should be scaled equally to achieve compute-optimal training for llm's. further credence to reducing llm compute is lent by @xcite , who found input data pruning can improve models to scale exponentially, and @xcite , who showed llm in-context learning derives from the zipfian distribution of pre-training data. most releavant to this work is huebner et al. (2021) who found a small language model trained on child-directed speech can achieve comparable performance to larger lms on a set of probing tasks. in contrast to them, we train multiple models, explore scaling in the low-compute region and evaluate on a filtered version of glue instead of a set of linguistic tests. ## methodology this section discusses the language simplification process, pre-training data, development of the data tokenizer, language model configuration, and pretraining objective in detail. ## experimental setup in our experiments, we explore the relationship between training data size, model size (number of parameters), model shape, cost of training (flops), and performance during pre-training and downstream. in the following subsections, we will discuss our strategy for exploring various model shapes followed by a discussion on hyperparameter settings in detail. ## conclusions & future work in this study, we investigated whether reducing language size allows the benefits of pre-training to be observed in a downscaled setting for models @xmath1. we evaluated a range of model configurations and found that the advantages of pre-training are observable even for models with as few as 1.25m parameters, with a strong correlation between upstream and downstream performance. however, we also observed that compute-optimal training does not appear to be crucial for downstream results and that parameter count does not reliably predict upstream performance. furthermore, we observed a break of the flop-perplexity power law at the 2.2 × 10 15 flop region, which shows the limited applicability of scaling laws. overall, our experiments provide insight into the behavior of small language models in a downscaled language setting. the next logical steps as a follow-on to this work would be to check whether generative models would demonstrate any emergent abilities in a downscaled setting.