Findings of the Second B aby LM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2024)

Abstract: The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model training on a fixed language data budget of 100 million words or less. This year, we released improved text corpora, as well as a vision-and-language corpus to facilitate research into cognitively plausible vision language models. Submissions were compared on evaluation tasks targeting grammatical ability, (visual) question answering, pragmatic abilities, and grounding, among other abilities. Participants could submit to a 10M-word text-only track, a 100M-word text-only track, and/or a 100M-word and image multimodal track. From 31 submissions employing diverse methods, a hybrid causal-masked language model architecture outperformed other approaches. No submissions outperformed the baselines in the multimodal track. In follow-up analyses, we found a strong relationship between training FLOPs and average performance across tasks, and that the best-performing submissions proposed changes to the training data, training objective, and model architecture. This year’s BabyLM Challenge shows that there is still significant room for innovation in this setting, in particular for image-text modeling, but community-driven research can yield actionable insights about effective strategies for small-scale language modeling.

## introduction this paper describes the second babylm challenge and its findings. the broader goals and motivation of the challenge have remained constant since the first iteration last year. at the heart of both this year's and last year's challenge is the observation that children are incredibly data-efficient language learners, whereas artificial neural-networkbased language models are not. on the one hand, children are exposed to less than 100 million word tokens by the age of 13 @xcite , at which point they have mastered their native language(s). on the other hand, today's ann-based language models are trained on trillions of wordsfive to six orders of magnitude more than the typical human language learner. for a more in-depth discussion on the issue of data efficiency, see the findings of last year's challenge @xcite as well as @xcite , a position piece written by many of the challenge organizers. the learning discrepancy between humans and models raises two important questions: first, how is it that humans are able to learn language so efficiently? and second, what insights from human language learning can be used to improve language models? it is our hope that by creating a platform for interested parties to experiment with datalimited and cognitively inspired language modeling, we can continue to make progress on these interrelated questions. in particular, our goal with babylm is to contribute to: the main difference between this year's and last year's challenge is twofold: first, this year we allowed participants to bring their own datasets, as long as they stayed within the 100 million word limit for our strict track, or the 10 million word limit for our strict-small track. the motivation behind this decision is that pretraining data quality has been linked to large improvement gains in at-scale language models @xcite , so this year we allowed participants to improve the training data beyond the provided dataset, which was effectively a dataset baseline. second, this year included a multimodal track, in which participants trained on aligned text-image data, and tested their models in a novel text-image evaluation pipeline. non-linguistic information, such as visual input, potentially plays a large role in child language acquisition. while visual input is not inherently necessary for successful language acquisition (for example, blind children learn language largely without issue), visual grounding has been linked to faster language learning @xcite . furthermore, visual grounding has long been hypothesized to aid word learning: children learn nouns more easily than verbs @xcite , arguably because the former are more easily linked to visual stimuli than the latter. additionally, children learn concrete nouns easier than abstract nouns @xcite . however, visual grounding also presents several challenges: words may be time-delayed with respect to their referents, or one word may be uttered in a context with multiple competing possible referents. with this in mind, our hope was that the multimodal track would help to explore the space of possible computational models for visual grounding during language acquisition. findings and takeaways. this year, we received 31 submissions from 17 different countries making diverse contributions. examples included submissions proposing novel architectures, new training objectives, innovating on knowledge distillation methods, and proposing curriculum learning methods, among others. we conduct a meta-analysis of the results, which yields several concrete recommendations. the best-performing submissions constructed their own training datasets, proposed new model architecture, or new training objectives. performance on the babylm evaluations also correlated strongly with training flops, suggesting that high-compute training regimes still tend to reliably perform better, even in low-data settings. the babylm research community also showed growing attention to tokenization and multilingual language modeling, while maintaining interest in curriculum learning and applying linguistic biases to language models. our data (pretraining corpora and evaluation data; [link]), preprocessing code [link], baselines @xcite and evaluation pipeline [link] are all publicly available. we also release the submitted models of those who agreed to release them, along with their hyperparameters and results @xcite . the leaderboard may be found here @xcite . the strict and strict-small tracks required that submissions be trained on 100m words or less and 10m words or less, respectively. these tracks no longer required that participants use the fixed dataset from last year's challenge, although we still provided an updated version of this dataset, described in section 3. models in these tracks were evaluated on language-only evaluation tasks. in the multimodal track, participants trained multimodal image-text models. participants were allowed to use any model and training procedure they desired, as long as the model could assign (pseudo) log-likelihoods to strings of text, conditioned on an image. again, participants were free to construct their own datasets, including unlimited visual inputs, as long as the text data was within a 100m word budget. to facilitate easier participation in this track, we released a suggested multimodal dataset that consisted of 50% text-only and 50% paired image-text data. submissions to this track were evaluated on both language-only and additional multimodal tasks. ## pretraining corpus this year, we updated the text-only dataset from the previous competition and provided a novel imagetext dataset for the multimodal track. data for both the text-only and multimodal datasets can be downloaded from https://osf.io/ad7qg/ . for the text-only dataset updates, we increased the proportion of child-oriented data (counting both transcribed speech and written data) to 70% up from 39% last year, and we increased transcribed speech data to 58% up from 55% last year. we have eliminated the wikipedia portion of the data (except for simple english wikipedia) due to being the only non-spoken and non-child-level data, and we have eliminated the qed portion due to quality issues. we have also reduced our reliance on opensubtitles, which can include scripted speech, which is arguably less ecologically valid than other spoken sources. childes now comprises a significantly larger portion of the new dataset. we use the entire available english portion of childes including both caregiver and child utterances, increasing the proportion of child-oriented discourse from 5% last year to 29%. 1 we also replaced last year's children's stories and project gutenberg data with a custom children's stories dataset sourced entirely from project gutenberg. we select child-appropriate books using the provided subject metadata, and then select the 1000 most common books, giving us a combined corpus of 26m words. for more details about other data sources, see @xcite . in addition, we provide a novel image-text dataset to facilitate easier participation in the multimodal track. this dataset has two components: first, we provide 50m words of text-only data, drawn from the 100m babylm corpus via stratified sampling (that is, we preserve the relative distribution from the different data sources). second, we provide paired text-image data that includes 50m words of text. this paired data comes from two sources: 27m words from the localized narratives dataset @xcite and 23m words from the conceptual captions 3m (cc3m) dataset @xcite . for the localized narratives dataset, we used the text captions and the images from the ms-coco @xcite and open images @xcite subsets. for the cc3m dataset, we used the image-caption 1 we thank brian macwhinney (personal correspondence) for alerting us to the existence of this additional childes data. ## evaluation and submission as in last year, we distributed a shared evaluation pipeline based on the lm evaluation harness @xcite . for the strict and strict-small tracks, evaluation tasks were largely the same as the previous year: we used blimp @xcite , the blimp supplement @xcite , and a subset of (super)glue tasks @xcite ) as the public evaluation set. blimp measures whether lms prefer grammatical to minimally-differing ungrammatical sentences (i.e., minimal pairs) and spans a range of grammatical phenomena including subject-verb agreement, binding, and control/raising constructions. the blimp supplement is a disjoint subset of minimal pairs designed specifically for last year's babylm challenge to test linguistic knowledge not covered by blimp, such as dialogue and pragmatics. (super)glue is designed to measure natural language understanding across a diverse array of subtasks; its tasks include question answering and natural language inference, among others. for the multimodal track, participants were required to evaluate on the evaluation benchmarks from the text tracks; this was to establish whether training on image data facilitated sample-efficient language modeling. in addition, we included a suite of multimodal evaluation tasks. the public evaluation datasets included visual question answering (vqa; @xcite and winoground @xcite . vqa measures whether vision-language models (vlms) prefer correct answers to questions about visual inputs, and winoground measures whether lms prefer accurate descriptions of images among minimally differing options (e.g., given an image of dirt on top of a light bulb, does the vlm prefer "a lightbulb on top of dirt", or "dirt on top of a light-bulb", and vice versa given another image where the lightbulb is on top of dirt). this year, we used the elements of world knowledge (ewok) dataset @xcite as the hidden task for the text tracks. this task measures pragmatic, commonsense, and discourse knowledge. for the multimodal track, the hidden task was devbench @xcite ; this benchmark contains subtasks targeted at evaluating visual and linguistic abilities that emerge at different stages of children's development, including subtasks where (i) the model must pick the correct image associated with a given word; (ii) the model must pick the correct image corresponding to a sentence; and (iii) the model must assign appropriately higher or lower similarity scores to more or less similar images. the data for these tasks was released two weeks before the model submission deadline. we selected these tasks based on whether they capture distinct phenomena from the public evaluation tasks, such that optimizing only for individual tasks or narrow subsets of linguistic competencies would not be overly rewarded. most of the evaluation tasks were zero-shot. zero-shot evaluation entails comparing the probabilities of different sequences of text. thus, all submitted models were required to assign a (pseudo) log-likelihood to a sequence of tokens. additionally, the (super)glue tasks required fine-tuning a classification head appended to the model. models did not need to generate sequences for any evaluation task; thus, both autoregressive and masked language modeling architectures could be used. ## competition results in this section, we discuss the overall results of the competition ( §5.1), track winners ( §5.2), and this year's outstanding papers ( §5.3). we found that many submissions focused their efforts on similar techniques. to better quantify this, we devised, in figure 1 , a typology of the most common approaches and assigned each submitted model one or more labels. §6.3 provides more detailed descriptions of each approach, as well as results indicating which ones were most effective. all participants are affiliated with universities or independent research institutions. participants' home institutions are located in 16 different countries. the number of participants by country is as follows (multinational submissions are counted more than once): germany (8), united states (6), netherlands (4), italy (2), uk (2), canada (1), china (1), greece (1), hungary (1), iran (1), israel (1), japan (1), norway (1), singapore (1), sweden (1), switzerland (1), and taiwan (1). ## discussion in this section, we discuss several trends in this year's submissions ( §6.1-6.3) and spotlight approaches ( §6.4) which we believe point the way towards novel and interesting work in this area. ## conclusion the second babylm challenge has demonstrated that significant progress can be made in dataefficient language modeling through communitydriven research efforts. with 31 submissions from 17 countries, the challenge revealed several key insights: innovations in model architecture, training objectives, and dataset construction proved particularly effective, with gpt-bert, a hybrid causalmasked language model architecture, emerging as the strongest approach for the strict and strict-small tracks. however, the strong correlation between training flops and performance suggests that computational resources remain a crucial factor even in low-data settings. while this year's challenge added a multimodal track, in an attempt to model grounded language learning environments, no submissions outper-formed the baselines in this track. this suggests that effectively integrating visual information during language learning remains a significant challenge for current architectures. this year's challenge also featured emerging research directions not present in the previous iteration, with participants exploring linguistically-motivated tokenization strategies and revisiting recurrent neural architectures. looking ahead, we envision the babylm challenge continuing to evolve and expand its scope beyond text-only and vision-language tracks. we hope that future iterations will explore additional modalities. such as speech, and extend to more languages, better reflecting the important fact that human language development proceeds equally well in any natural language. by broadening the challenge's focus while maintaining its core emphasis on data efficiency, we aim to inspire novel approaches that bridge the gap between artificial and human language learning. the strong participation and innovative solutions seen in this year's challenge suggest that the babylm community is well-positioned to tackle these ambitious goals, ultimately working toward language models that better reflect the efficiency and adaptability of human language acquisition. a text only datasets childes. the child language data exchange system (childes; macwhinney, 2000) is a multilingual database compiling transcriptions from numerous researchers of adult-child interactions in a range of environments, from structured laboratory activities to the home. @xcite further process childes, selecting only interactions with american english-speaking children ages 0-6, removing all child utterances, and tokenizing the data. the resulting dataset foot_7 contains about 5m words. british national corpus. the bnc @xcite is a 100m word multi-domain corpus of british english from the second half of the 20 th century. we select only the dialogue portion of the corpus, totaling about 10m words. children's book test. cbt is a compilation of over a hundred children's books from project gutenberg by @xcite . the dataset was originally released with a set of questions for testing named entity prediction, which we do not include in the pretraining data. children's stories text corpus. this dataset consists of manually selected children's stories from project gutenberg. it was compiled by @xcite for the development of a story generation system. project gutenberg. the standardized project gutenberg corpus @xcite ) is a curated and preprocessed selection of over 50k literary books in the public domain from project gutenberg totaling over 3b tokens. 11 this distribution comes with extensive metadata that allows us to filter texts by language and date. opensubtitles. this dataset @xcite ) is a compilation of publicly available subtitles from tv and movies on a third-party website. 12 we use only the english portion. wikipedia. wikipedia is a volunteer-authored encyclopedia hosted by the wikimedia foundation. we use only the english portion. simple english wikipedia. simple english is classified as a separate language in wikipedia, thus the texts here are disjoint from those in english wikipedia. the texts use shorter sentences and high-frequency vocabulary and avoid idioms. switchboard corpus. the switchboard corpus @xcite is a collection of transcribed telephone conversations between pairs of strangers. we accessed the text through the switchboard dialog act corpus @xcite .