B aby LM Challenge: Exploring the effect of variation sets on language model training efficiency (2024)

Abstract: While current large language models have achieved a remarkable success, their data efficiency remains a challenge to overcome. Recently it has been suggested that child-directed speech (CDS) can improve training data efficiency of modern language models based on Transformer neural networks. However, it is not yet understood which specific properties of CDS are effective for training these models. In the context of the BabyLM Challenge, we focus on Variation Sets (VSs), sets of consecutive utterances expressing a similar intent with slightly different words and structures, which are ubiquitous in CDS. To assess the impact of VSs on training data efficiency, we augment CDS data with different proportions of artificial VSs and use these datasets to train an auto-regressive model, GPT-2. We find that the best proportion of VSs depends on the evaluation benchmark: BLiMP and GLUE scores benefit from the presence of VSs, but EWOK scores do not. Additionally, the results vary depending on multiple factors such as the number of epochs and the order of utterance presentation. Taken together, these findings suggest that VSs can have a beneficial influence on language models, while leaving room for further investigation.

## introduction while current language models (lms) demonstrate outstanding performance in a range of linguistic and reasoning tasks, there is ample scope to enhance their data efficiency. a state-of-the-art lm like chinchilla uses as much as 1.4 trillion words for pretraining, whereas humans master their native language by hearing less than 100m words by the age of 13 @xcite . child language acquisition could provide insights into it, given that children acquire basic grammar by the age of six @xcite , without as varied and abundant linguistic inputs as those given to modern lms. various studies argue that this highly efficient learning is aided by children's limited cognitive abilities and specific types of inputs towards children @xcite @xcite @xcite . inspired by this, the babylm challenge aims at improving data efficiency in language models as well as providing insights into child language acquisition. it is also suggested that cds is a preferable domain for facilitating the acquisition of linguistic knowledge compared to other domains of data. the findings of these studies include efficient pretraining without sacrificing the performance @xcite , enhanced semantic extraction @xcite , and superior induction of hierarchical structure @xcite . while these studies suggest that cds helps lms learn from limited datasets, further research is needed to determine which specific properties of cds provide an advantage to lms. as one of such properties, some studies highlight variation sets (vss), which are sets of (mostly consecutive) utterances expressing a similar intent with slight variations in the use of words and structures @xcite . this specific pattern is ubiquitous in cds, but not in other speech genres. in first and second language acquisition, several studies indicate that vss in cds support learning of syntactic structure @xcite @xcite by maintaining children's attention on the circumscribed topic and promoting comprehension by introducing new information @xcite . these findings suggest that vss are beneficial for language learning in general and thus could enhance the learning process in lms. in this work, we explore this hypothesis by examining the effect of vss on language models' data efficiency. to fully control the impact of vss, we construct artificial vss based on the description by @xcite , mixing it with actual cds at various rates (0%, 20%, 40%, 60%, 80%, 100%). then we compare the models' accuracy on these constructed datasets and shuffled datasets on blimp @xcite , ewok @xcite , and glue @xcite . ## computational studies on cds computational studies further investigate whether cds is beneficial for acquiring grammatical knowledge in models as well as for human language acquisition. @xcite demonstrate that the use of child-directed speech (cds) enables a small-sized roberta @xcite model trained on 5m words to attain similar linguistic competence as a roberta trained on 30b words. @xcite examine that cds has rich semantic information for grasping causal semantics without syntactic structures, finding that cds is effective in learning to extract semantic information. furthermore, @xcite argue that lms can induce hierarchical structures better when trained on cds than on other typical datasets like wikipedia. while these findings demonstrate the positive effect of cds on language learning, we are interested in which specific properties of cds contribute to this effect. one of the reasons why cds can enhance lms' acquisition of syntactic structures could be its lower lexical complexity, i.e., fewer word types @xcite , which stems from the high repetitiveness of items in cds. however, this repetition occurs across multiple utterances, a characteristic unique to cds. we hypothesize that this could be a key factor in the success of lms' language learning. ## method inspired by human studies of language acquisition, we want to examine whether vss can also help a language model recognize sentence structures in the language. to our knowledge, this effect has only been explored in the pilot experiment of @xcite . the experiment consisted of extracting naturally occurring vss from cds data using multiple automatic vs detection methods. the results showed no significant effect of vss on syntactic performance measured on blimp, but this could be due to the difficulty in fully controlling the number of actual vss detected automatically @xcite . to address this difficulty, we opt for the use of synthetic vss, which allows us to fully control the proportion of vss in our training dataset. for this purpose, we use gpt4o-mini foot_2 to generate artificial vss and augment the training data in different proportions. 3 while the use of a large lm seems to contradict the goal of improving data efficiency, we see this as a first step to measuring the importance of vss. in case of successful results, future work could explore less costly methods to generate vss, such as template-or syntactic rule-based. humans hear sentences in a vs sequentially. however, it is not clear how to present vss to a model in a way that is equivalent to human input, nor how to maximize the effect of vss. therefore, we conduct experiments using two methods. the first method, as shown in the example on the left side of figure 1 , involves concatenating the vs into a single instance and providing it to the model. in this configuration, the model is forced to sequentially process the sentences within the vss. we named this method the "sequential concatenation method". the second method, as shown in the example on the right side of figure 1 , involves placing each sentence in the vs in adjacent batches. in this configuration, the model updates its parameters after processing one sentence in the vs before moving on to the next sentence within the same set. we named this method the "adjacent batch method". ## results and discussion in our experiment, we train gpt-2 from scratch using training data that includes artificial vss. we report the results after the model training has converged, specifically the results after 3 epochs. ## conclusion we presented an initial exploration of the effect of cds-inspired variation sets on language model training efficiency. our results suggest that vss can have a beneficial impact on various linguistic competences. they also reveal that this effect is entrenched with several factors like the order of utterance exposure and the number of training epochs, leaving space for more detailed investigations in the future.