W hat I f: Leveraging Word Vectors for Small-Scale Data Augmentation (2024)

Abstract: We introduce WhatIf, a lightly supervised data augmentation technique that leverages word vectors to enhance training data for small-scale language models. Inspired by reading prediction strategies used in education, WhatIf creates new samples by substituting semantically similar words in the training data. We evaluate WhatIf on multiple datasets, demonstrating small but consistent improvements in downstream evaluation compared to baseline models. Finally, we compare WhatIf to other small-scale data augmentation techniques and find that it provides comparable quantitative results at a potential tradeoff to qualitative evaluation.

## introduction the use of large language models (llms) has exploded in the recent past, with llms becoming the state of the art for most nlp tasks. while statistical models of language have been around for decades @xcite , the introduction of the transformer @xcite set the stage for a new era in language modeling. early transformer-based language models such as bert @xcite and gpt @xcite are very small by today's standards, with a few hundred million parameters each. in the intervening years, models have grown exponentially both in number of parameters and number of training tokens. these increases in size have been accompanied by increases in performance, with abilities emerging as a consequence of model scale @xcite . current state of the art models tend to have tens of billions to hundreds of billions of parameters and are trained on trillions of training tokens. in this paradigm of increasing scale, there has been relatively little focus on small-scale language modeling, which tends to be restricted to domains such as low-resource machine translation. the babylm challenge @xcite seeks to focus researchers on very small-scale language modeling. the challenge involves using either a 10 or 100 million word "developmentally plausible" corpus @xcite , with 100 million words being roughly amount of words a child hears before reaching adulthood. working at this small scale enables researchers to focus on cognitively inspired methods of language modeling as well as to iterate on language modeling experiments, which is impractical at 100-billion parameter scales. while much recent focus on language modeling has involved scaling up parameter and training token counts, these approaches have drawbacks, including environmental concerns and inaccessibility of hardware @xcite . as a consequence, there has been a recent focus on mid-scale language modeling, creating models that can be run locally on devices such as consumer pcs or smartphones. this research has been promising. microsoft's phi models @xcite boast impressive performance on many language modeling benchmarks, in spite of having only a few billion parameters. phi's major innovation is using only "textbook quality data", curated from only high-quality sources rather than semifiltered data of dubious quality scraped from the internet. at a much smaller parameter scale, eldan and li (2023) trained very small transformers on tinystories, a synthetic dataset of children's stories. in spite of parameter counts below 10 million, these tiny models were able to generate coherent text with real world knowledge and logic. the trend towards improving data quality and quantity rather than solely scaling model parameters has also been applied successfully to largerscale language modeling. llama 3 (team, 2024) attributes its significant improvements in performance over llama 2 @xcite not to changes in architecture, but to "improvements in data quality and diversity as well as by increased training scale." using higher quality input, llama 3 was trained on roughly ten times as many tokens as llama 2. our research is motivated by both the language modeling research on training data, as well as children's processes of language acquisition. training on more data has a human analog. one of the strongest predictors of children's linguistic development is the amount and type of language they hear @xcite . children who hear more words tend to have larger vocabularies, which correlates with better educational outcomes later in life @xcite . we introduce whatif, a lightly supervised data augmentation technique that uses word vectors to augment training data. whatif substitutes words in the training corpus for semantically similar words, enabling our baby models to consider novel yet similar text to the training data. whatif is inspired by a method of improving reading comprehension called predicting. with this strategy, teachers instruct students to periodically ask questions about the text. these questions can be predictions about what might occur later in the text or counterfactuals, which usually take the form of "what if?" questions. teaching with prediction strategies improves reading instruction outcomes. both children @xcite and second language learners (ali and razali, 2019) show improved reading comprehension when em-ploying prediction strategies. our method is lightly supervised, and requires a small part-of-speech dictionary, which we count as part of our token budget. this too has an analog in real-world language acquisition. children receive explicit grammatical knowledge. for example, children who produce ungrammatical speech are often corrected by a parent or caregiver. children who receive explicit grammatical instruction and have explicit grammatical awareness tend to develop better linguistic skills @xcite . we perform experiments and show that whatif increases model performance on a variety of evaluation tasks, and performs comparably to other small-scale language model data augmentation techniques, though these quantitative gains come at a cost to text quality. ## results we perform a variety of experiments to probe the efficacy of our data augmentation technique. all experiments are performed on both the tinystories dataset and the babylm strict-small dataset. for our baselines, we train models using a standard language modeling approach. these examples occasionally need to be truncated, but thanks to the data pre-processing, the overwhelming majority of samples do not require truncation. because each pass of the augmentation process results in lower quality data, we experiment with how many passes of augmented data we create, @xmath0. for every n passes of augmented data, we also include one pass of the nonaugmented gold standard data. this means only 1 6 or 1 11 of data seen while training on augmented data is gold-standard data. we also experiment with the percent of content words to replace, either 50%, which leaves the sample recognizable, or 100%, which drastically changes the training example. examples of different degrees of augmentation can be found in the appendix. ## discussion just like children, small-scale language models benefit from additional data. whatif shows mild but consistent benchmark improvement above the baseline across datasets. we expect this to improves performance on benchmarks by exposing the lm to new scenarios during training. in practice, the augmented data is sometimes fairly low-quality. as a consequence, the lm can learn incorrect facts about the world. for example, augmentation may replace the word mom with dad, without replacing gendered pronouns she with he. this does not seem to have a large negative effect on the model's grammatical abilities, since blimp and blimp supplement scores improve with whatif augmentation. however, ewok scores do not improve decreasing slightly when applied to the tinystories dataset. we suspect that the lm earns incorrect information about the world from bad correlations in the augmented data. the fact that both whatif and contextualizerlike provide similar gains suggests that manipulating the training data in some well-informed way provides modest performance gains. since we observe diminishing returns when ensembling both methods, this might mean that both methods are acting on a similar axis to make the training data more useful to the model. ## limitations and future work this work is only a partial realization of the underlying idea that data augmentation with word vectors could improve model performance. we suspect that small changes to the data augmentation algorithm could bear significant fruit. an additional round of validation to improve the coherence of the augmented data would probably help. our analysis is limited to autoregressive language models, and experiments should be repeated with masked language models. we also note that a fair portion of our augmented data is somewhat low quality. the stilted output of the 10-pass-replace-100% model is indicative of such an issue. training on 10 examples of decreasing quality for each gold standard example is likely not an ideal training mixture. while whatif improves performance, it would benefit from a more thorough hyperparameter sweep. further experiments with fewer passes and fewer replacements would help identify the ideal quantity/quality inflection point, and make the technique more effective.