Emergent Abilities in Reduced-Scale Generative Language Models (2024)

Abstract: Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primarily seen in large language models with billions of parameters. This study investigates if such emergent properties are strictly tied to model size or can be demonstrated by smaller models trained on reduced-scale data. To explore this, we simplify pre-training data and pre-train 36 causal language models with parameters varying from 1 million to 165 million parameters. We show that models trained on this simplified pre-training data demonstrate enhanced zero-shot capabilities across various tasks in simplified language, achieving performance comparable to that of pre-trained models six times larger on unrestricted language. This suggests that downscaling the language allows zero-shot learning capabilities to emerge in models with limited size.Additionally, we find that these smaller models pre-trained on simplified data demonstrate a power law relationship between the evaluation loss and the three scaling factors: compute, dataset size, and model size.

## introduction recent advancements in deep learning and distributed computing have enabled the pre-training of language models on a massive scale @xcite @xcite , significantly changing the way these models are used. large pre-trained models proved capable of solving various tasks with zero-shot or few-shot learning, eliminating the need for task-specific finetuning @xcite . this is referred to as in-context learning, an ability which allows these models to understand and solve new tasks based on the provided context. it is argued that this ability 1 code and simplified pre-training data are available at github.com/text-machine-lab/mini_gpt "emerges" with a dramatic increase in the size of the model @xcite . efforts to transfer emergent abilities to small models include imitation learning, where a large language model like gpt-4 acts as a "teacher" to create synthetic datasets with additional instructions and explanations. this synthetic data is then used to train smaller "student" models @xcite @xcite . another approach is distillation where the "student" model is trained to mimic the output probabilities of the "teacher" model @xcite . our work takes a different approach; our goal is to determine whether simplifying the pre-training data itself can unlock emergent language abilities in smaller models. this idea is supported by our previous work @xcite , which highlighted the effects of language simplification for smaller models when fine-tuning on downstream tasks. prior work by eldan and li (2023) reports a similar trend, though their approach requires the use of larger models to produce the simplified language. we bypass this step and instead rely on naturally-occurring language restricted via vocabulary filtering. in this study, we leverage this approach to determine whether language simplification can unlock icl abilities in smaller language models. to do so, we pre-train 36 causal language models with sizes varying from 1m to 165m parameters, on both a simplified english dataset and a standard pre-training dataset and conduct zero-shot evaluations on different tasks. through extensive experimentation, we show that language simplification enables icl abilities in smaller language models on a level comparable to larger-size models pre-trained on non-simplified english corpora. specifically, our contributions are as follows: simple vocabulary (child-parent conversations in aochildes dataset) data simplification (span filtering) downstream evaluation regular pt models simplified pt models simplified evaluation regular evaluation ## related work what is icl? icl is the ability of a pre-trained model to solve tasks without task specific finetuning @xcite @xcite . many large models have shown excellent icl capabilities @xcite . this has shifted the research community's focus towards leveraging prompts to elicit zero-shot or few-shot responses from models. in a similar vein, the technique of chain-of-thought (cot) reasoning, as discussed in wei et al. (2022b) , revealed that including a sequence of intermediate reasoning steps can enhance the reasoning skills of large language models. yet, these abilities are emergent, i.e., it is primarily the larger models that exhibit them. however, recent studies question the belief that improvements in icl result exclusively from increasing model sizes @xcite , suggesting that using discontinuous metrics like accuracy merely creates the illusion of emergent abilities, whereas employing continuous metrics shows gradual, predictable changes in model performance. icl in smaller language models. it has been shown that the emergent abilities observed in larger models can be effectively transferred to smaller models through imitation learning or behavior cloning, where a larger language model such as gpt-4 serves as the "teacher", to generate synthetic datasets with instructions and explanations which can be used to train smaller language models, referred to as "student" models @xcite @xcite . this allows smaller models to leverage the capabilities of their larger counterparts. however, the primary drawback of such methods is that most of the knowledge acquired by the model is done in the pre-training stage and the student model copies the style of the teacher model but does not learn the reasoning capabilities employed by these large models @xcite ). an alternative strategy to enhance the capabilities of smaller models is through distillation from larger models, aiming to replicate the output probabilities and thus transfer the larger model's in-context learning or zero-shot abilities to their smaller counterparts @xcite @xcite . this method forfeits one of the primary benefits of smaller language models, namely their reduced computational requirements, by necessitating the training of larger models. prior work has also looked into pre-training small models with simplified data. for instance, huebner et al. (2021) pre-train an encoder language model with corpus that reflects the lexical exposure of children and find that smaller models can approximate the grammatical acquisition performance of larger models. @xcite examined the effects of downscaling the modeled language during pre-training via vocabulary-based filtering, and showed that pre-training encoders as small as 1.25m parameters may demonstrate large benefits for downstream performance. eldan and li (2023) have demonstrated that coherency in text generation can be achieved by pretraining on a synthesized, simplified dataset generated from gpt-4. notably, this dataset largely comprises of stories, presenting less diversity compared to the datasets typically employed for pre-training larger models. similarly @xcite demonstrate improved performance in smaller models trained on a dataset combining filtered coding examples and synthetic textbook content for coding-related benchmarks. however, their ap-proach, primarily focused on coding challenges, utilizes relatively large models and synthetic data. their dataset filtering approach also relies on an auxiliary classifier for text exclusion. ## results our goal is to understand if the absence of emergent abilities in smaller pre-trained models is simply a matter of model capacity and whether simplifying the problem, i.e., downscaling the language, would allow these abilities to emerge in smaller models. to this end, we evaluate models trained on simplified data against both filtered and standard evaluation datasets. we compare our models with pythia @xcite and opt @xcite (models up to 1.3b parameters) to determine if downscaling the language facilitates emergent capabilities to occur much earlier. ## conclusion in our study, we explored the impact of simplifying pre-training data on the performance of small generative models, specifically those with fewer than 165 million parameters. our primary focus was to assess whether these models exhibit emergent abilities, notably zero-shot learning -the capability to have non-random performance on tasks without explicit prior training. to this end, we evaluated a series of models, each varying in hidden size and the number of layers, and measured their zero-shot performance across different tasks. our findings 4 the results are reported on models trained on 2.1b tokens models pre-trained on a vocabulary-simplified dataset produce outputs that are grammatical, creative, and coherent, and are comparable to those from larger models trained on the complete language dataset.