BERT time Stories: Investigating the Role of Synthetic Story Data in Language Pre-training (2024)

Abstract: We describe our contribution to the Strict and Strict-Small tracks of the 2nd iteration of the BabyLM Challenge. The shared task is centered around efficient pre-training given data constraints motivated by human development. In response, we study the effect of synthetic story data in language pre-training using *TinyStories*: a recently introduced dataset of short stories. Initially, we train GPT-Neo models on subsets of *TinyStories*, while varying the amount of available data. We find that, even with access to less than 100M words, the models are able to generate high-quality, original completions to a given story, and acquire substantial linguistic knowledge. To measure the effect of synthetic story data, we train *LTG-BERT* encoder models on a combined dataset of: a subset of *TinyStories*, story completions generated by GPT-Neo, and a subset of the *BabyLM* dataset. Our experimentation reveals that synthetic data can occasionally offer modest gains, but overall have a negative influence on linguistic understanding. Our work offers an initial study on synthesizing story data in low resource settings and underscores their potential for augmentation in data-constrained language modeling. We publicly release our models and implementation on our GitHub.

## introduction as the performance of modern language models (lms) increases, enabling remarkable feats of language understanding and reasoning, so do their demands in computational resources and training data @xcite . for example, the recently released llama 3 @xcite has 405b parameters and was pre-trained on 15.6t tokens, on 6k h100 gpus. in contrast, children are only exposed to no more than 100 million words by age 13 @xcite , demonstrating exceptional learning efficiency compared to stateof-the-art lms. this need for ever-increasing data and compute casts doubts on the cognitive plausibility of the current lm training regimes, and raises ecological and ethical concerns, such as democratic access to research for industry and research groups with modest resources. to address these issues, the babylm challenge @xcite invites participants to work on cognitive modeling and efficient lm pre-training, given data limitations inspired by human development. this year's iteration of the challenge features three experimental tracks: a strict track with a budget of 100m words, a strict-small track with a budget of 10m words, and a multimodal track with a word budget of 100m words and unlimited visual input. a major change compared to last year's challenge is allowing participants to construct their own training data. in the following sections, we present our contributions to the strict and strict-small tracks. our research draws inspiration from recent advancements in small language models (slms) for text generation, as explored in tinystories @xcite . in this influential work, the authors demonstrate that training on a synthetic dataset of simple stories can enable slms to produce creative, high-quality generations, which are novel with respect to the original training dataset. we hypothesize that for the small data regimes of the babylm challenge, augmenting the initial training corpus with synthetic data of high quality can provide models with unseen linguistic contexts, and as a result improve language understanding. to test our hypothesis, we first extend previous work by decoder encoder one day, a little boy named tim went to the park. he saw a big tiger. the tiger was not mean, but very easy to play with. tim and the tiger played all day. they had lots of fun. then, something unexpected happened. the tiger started to shake. tim was scared. he did not know what was going on. but then, the tiger turned into a nice ... we use a subset of the tinystories dataset (d tiny ) @xcite , to train a decoder transformer for data augmentation. we prompt the decoder with the short stories from d tiny and create a dataset of model generations (d gen ): each story (green) is truncated and used as a prompt (yellow), with the model generating an alternate completion (blue). we supplement the two datasets with a subset of the babylm dataset (d baby ), released by @xcite , and train an encoder model on the combined data. finally, we evaluate the linguistic proficiency of the encoder using the challenge benchmarks. @xcite , investigating generative performance with limited training data. we then train encoder transformer models on a diverse dataset, and measure the effect of synthetic data on linguistic proficiency. in technical terms, following @xcite , we propose to train a gpt-neo decoder @xcite on tinystories, and then use it to generate data for the training of a final encoder model. this poses some initial challenges, as the size of the original tinystories dataset exceeds the challenge limits, with around 373m words. as a result, we experiment with training gpt-neo models while varying the amount of available data and evaluating their generative performance, keeping the model architecture fixed in the process. our investigation of generative performance is complementary to the original work, which measures the effects of varying the depth and size of the model architectures. our findings demonstrate that even in low data regimes @xmath0, gpt-neo models can acquire impressive grammatical under-standing, and generate stories of high quality and diversity, comparable to models trained on the full dataset. for our evaluation see section 4.1. next, we investigate the effect of the synthetic story data on language pre-training. figure 1 illustrates our methodology. we select a small subset of tinystories, train a gpt-neo model, and use it to generate alternate completions to the stories in the training set. we then create a combined dataset consisting of: 1 the subset of tinystories used for gpt-neo training, 2 the generated data, 3 a sample of the babylm dataset @xcite . with the combined dataset, we train an ltg-bert @xcite model, choosing this architecture as it demonstrated superior performance in the text-only tracks of last year. we compare the performance of our models against a variety of baselines, trained with and without the use of synthetic data. our results show that a simple application of synthetic story data for language pre-training results in marginal or even negative gains. nevertheless, given the high generative per-formance of the gpt-neo models, we believe that more research is needed to fully explore and exploit their potential. ## related work previous babylm iteration data augmentation techniques were shown to be beneficial in the previous year's challenge @xcite . specifically, chapgpt @xcite uses regex patterns to extract common phrases from glue tasks, and then harnesses these patterns to generate follow-up questions that serve as additional training data. in the contextualizer paper @xcite , extra training samples are created by dynamically combining chunks of texts from different contexts during training. another approach named baby's cothought @xcite utilizes a large language model (llm) to reformat unrelated sentences from the corpus into coherent paragraphs, thereby improving performance, albeit in defiance of data constraints. language models for data augmentation in recent years, llms have been increasingly leveraged for data augmentation in various domains @xcite . notably, @xcite introduced chatgpt as a tool for generating realistic text samples from a combination of real and artificial data, enhancing training datasets. similarly, transformer architectures, including decoder (gpt-2, @xcite , encoder (bert, @xcite , and seq2seq (bart, lewis et al., 2020) models have been explored for augmentation @xcite . in the work of @xcite , gpt-3 @xcite ) was used to mix real and synthetic text samples for robust data augmentation. moreover, decoder models have been successfully employed to generate training data for encoders, yielding significant improvements in zero-shot learning @xcite . small language models the recent study by @xcite highlighted that small language models (slms), can outperform larger ones by leveraging high-quality synthetic training data, demonstrating fluency, coherence, and creativity despite having fewer parameters. this trend is further supported by work in sequential recommendation, where small models are effectively employed for task-specific purposes @xcite . additionally, @xcite utilize a pre-trained llm to encode prompt tokens, using these representations to guide a smaller lm for more efficient response generation. ## methods we describe our data augmentation method using synthetic story data, as illustrated in figure 1 . ## experiments experimental setup we conduct our experiments on a shared gpu cluster of 8 nvidia v100 16 gb gpus, and additionally evaluate our models on an nvidia rtx-3090 24 gb gpu. all our models are trained using the pytorch @xcite and huggingface @xcite libraries. for our evaluations of blimp, ewok, and (super)glue we build upon the official evaluation pipeline released by the challenge organizers @xcite . ## results we present the final evaluation results for the strict-small and strict tracks at strict-small track in the strict-small track, comparing the results of d baby-10m with d tiny-10m reveals, as expected, that the babylm dataset is more beneficial for language pre-training compared to tinystories. the performance metrics for tinystories are consistently lower, except in the case of ewok. interestingly, replacing half of the babylm dataset with data from tinystories only slightly affects the model's performance. however, as we add more instances of the synthetic story data, the positive impact of the babylm dataset begins to wane, leading performance to approach that of d tiny-10m where babylm was not used at all. this suggests that training is over-influenced by the increased amount of tinystories data. to mitigate this effect, we experimented with equally distributing the samples from the two datasets in a batch. this approach positively impacts the model's performance. notably for blimp, this setup slightly surpasses the performance of the model trained solely on d baby-10m , resulting in the best score overall. further, when compared to other data augmentation scenarios, the performance on glue is increased. moreover, an interesting observation concerns the sampling technique used for augmenting the data. changing the sampling strategy from greedy decoding to nucleus sampling positively affects the model's performance on the blimp and blimp supp. benchmarks, while negatively impacting performance on ewok and glue. this discrepancy is likely due to the nature of the datasets themselves. blimp focuses on evaluating grammatical understanding, while the increased diversity from nucleus sampling exposes the model to a wider range of linguistic structures and syntactic variations, thereby improving performance. conversely, ewok and glue require semantic coherence and factual consistency, where the increased diversity from nucleus sampling may introduce noise and less coherent narratives, potentially confusing the model, and degrading performance. therefore, while more diverse stories benefit syntactic evaluation tasks such as those in blimp, they may not be as useful for semantic or knowledge-based tasks such as those included in ewok and glue. strict track interestingly, for the strict track we notice that data augmentation has a positive effect on the blimp and ewok benchmarks. specifically, adding the d gen-greedy dataset, results in increased performance compared to the baselines trained on d tiny-100m and d baby-100m , as well as a mixture of the two (d tiny-50m + d baby-50m ). additionally, the d tiny-50m + d baby-50m combination is outperformed by both the d gen-greedy and d gen-nucleus-1 models, suggesting that synthetic data can offer modest gains in the strict scenario. as with the strict-small track, increasing the size of the tinystories dataset negatively affects the performance of the models, approaching that of the model trained solely on d tiny-100m . however, in this case, balancing the datasets does not improve the model's performance. in the larger 100m word dataset, even with balancing, the sheer volume of tinystories data may overwhelm the influence of the babylm data. the model is exposed to a much larger quantity of tinystories content, which could dominate learning and reduce the effectiveness of balancing. additionally, while the nucleus sampling strategy once again improves performance on the blimp supp. dataset, it does not assist with blimp as it did in the strict-small track. ## conclusion in this work, we explore data augmentation for language pre-training in a limited data setting. using the tinystories dataset we train gpt-neo models and probe the relationship between generative ability and dataset size. to measure the effect of augmentation with synthetic data, we train ltg-bert models on a diverse set of data configurations. our experiments indicate that while synthesizing high quality data is possible in small data regimes, effectively utilizing it for pre-training can be challenging. some modest gains are observed in the strict track, while careful balancing shows promise for the strict-small track. overall, our evaluation highlights the intricate balance required between data quantity, quality, and integration for effective training. future work suggests investigation of different data domains, mixtures, and proportions, while precise calibration of hyperparameters may prove critical in exploiting the full benefit of synthetic data in low data pre-training.