Extending the B aby LM Initiative : Promoting Diversity in Datasets and Metrics through High-Quality Linguistic Corpora (2024)

Abstract: BabyLM paves the way for a range of experiments aimed at better understanding language models (LMs) and the differences and similarities between human and artificial language learning. However, the current framework is limited to the English language and a narrow but significant range of evaluation metrics, primarily focused on syntax, semantics, and pragmatics. In this paper, we propose some steps towards extending the framework to other languages, specifically Mandarin Chinese and French, leveraging existing linguistic resources for these languages. Additionally, we advocate for greater exploration of genre variations within subcorpora for training LMs, as well as for the adoption of additional evaluation metrics with different underlying principles. Our proposal consists of using high-quality spontaneous speech corpora as a source for extracting production-related variables, which the models are then fine-tuned to predict. We hypothesize that these production-related features offer insights into the language processing mechanisms underlying the data and that cognitively sensitive models should outperform others in predicting these features. Specifically, we propose focusing on the prediction of phenomena such as speech reductions, prosodic prominences, sequences co-occurring with listeners’ backchannels, and disfluencies. To illustrate our approach, we present an example involving the prediction of speech reductions in spontaneous speech in two different languages (French and English), using models trained on 10 million tokens from different data source mixtures. Although the results are preliminary, they suggest that this task can characterize models for predicting human language processing.

## introduction the babylm initiative is built on three interrelated aspects: (i) data sets for training language models, (ii) evaluation metrics designed to capture cognitive and linguistic skills and their @xcite models that are either more cognitively plausible and/or capable of learning efficiently from "small" datasets. this initiative represents a strategic and timely effort to better understand the differences between artificial and human language learners. while the 2023 edition focus was primarily on models, the 2024 call expands the scope to include investigations into both datasets and evaluation metrics-a crucial step, as we will argue in this position paper. specifically, we propose concrete directions for expanding language model training datasets and exploring new evaluation metrics to deepen the linguistic and cognitive relevance of the babylm evaluation framework. regarding evaluation metrics, we advocate for a novel approach that leverages existing high-quality spontaneous speech corpora. one observation about the babylm initiative to date is that the datasets used are in english. while this is a natural starting point, it represents a significant limitation. expanding the scope to include more languages is not only about better representing linguistic communities or potential model users. achieving comparable, contrastive results across different languages within the babylm framework could offer valuable insights into both the learning models and the underlying learning processes. while the original babylm initiative argues convincingly for a mix of data sources including transcripts of child-caregiver conversations, everyday conversations, subtitles, and simple texts, different mixtures can be explored. due to data scarcity, it is still impossible to gather a 100m data set based on real spoken conversational data but the 10m is accessible for a few languages like english, french and mandarin and a few others. conversational speech is the genre within which humans acquire their basic language skills. it is a genre quite distant from the usual written or web content on which lms are trained, increasing the risk of biases for lms produced. moreover, it has been argued that it is a genre of high relevance to language emergence @xcite . how could a purely interactional dataset, including both child-directed and general conversation transcripts, be compared to more balanced mixtures? this opens the door for testing various hypotheses. for instance, does including more encyclopedic knowledge help with higher-level commonsense tasks, while a purely conversational training set provides a model with better communicative and conversational abilities? in this context, current evaluation metrics, while a good starting point, appear biased in two ways: they tend to favor canonical written forms and prioritize syntactic, semantic, and commonsense pragmatics. however, language and communicative competence include many other dimensions. although the initiative clearly emphasizes the importance of using speech transcripts, both child-directed and everyday conversations, as training data, to our knowledge, none of the evaluation metrics employed address explicitly the specificities of spontaneous speech. to summarize, we argue that, in line with the directions proposed in this year's new call, training datasets, and evaluation metrics are just as crucial as models for understanding the computational learning of language structure. we propose evaluation metrics based on spontaneous speech data and demonstrate how we can build such metrics from different aspects of the speech signal and transcripts obtained from high-quality spontaneous speech corpora. ## related work since the emergence of large language models, there has been strong interest from the computational linguistics community in understanding why they are so successful. @xcite explore the conditions (e.g., the amount of training data) under which roberta develops and leverages linguistic features, such as part of speech (pos) and morphology, as opposed to relying on simpler surface-level features like simple positionbased or length-based features. more recently, several studies have probed llms to better characterize their performance across various domains, particularly with regard to their linguistic competence versus commonsense reasoning. these studies have also examined the relationship between model performance and the amount of training data required for different tasks. in particular, @xcite used training sets of varying sizes, 1m, 10m, 100m, and 1b tokens, to show that syntactic and semantic competence becomes robust in the 10m-100m range, whereas larger datasets are needed to achieve strong results in pragmatic and commonsense reasoning tasks. more broadly, there have been proposals for evaluating the performance of llms on diverse linguistic tasks. @xcite leveraged a substantial body of generative syntax-semantics literature to develop benchmarks based on acceptability judgments, coming either the linguistic literature like the cola benchmark further extended by exploiting more sources and data augmentation methods in blimp @xcite . in addition to these binary decision tasks, @xcite combined three other types of evaluation metrics: classifier probing (following @xcite ), which includes tasks from pos tagging to coreference resolution; information-theoretic probing based on the minimum description length (mdl) principle; and fine-tuning on higher-level tasks such as those in the superglue benchmark. most of the benchmarks have been proposed for english. however, @xcite has inspired a series of language-specific benchmarks, such as climp for mandarin chinese @xcite , as well as benchmarks for other languages like japanese @xcite , dutch @xcite , and russian @xcite . these are important additions to the evaluation landscape. while these benchmarks represent important extensions to the general evaluation framework, they all rely on syntax-semantics structures derived from introspection and textbook data, as will be discussed in the next section. in parallel to these efforts, monolingual language models have been developed using large amounts of data @xcite , as well as experiments involving varied data quantities @xcite . in another line of research, several studies have tested the ability of large language models (llms) to perform tasks inspired by cognitive science, particularly in the domains of semantics and pragmatics @xcite . our approach of using actual speech data to extract production-based metrics can be related to studies that use behavioral or neurophysiological data linked with linguistic datasets. specifically, there has been significant work focusing on textual datasets combined with eye-tracking @xcite or neurophysiological @xcite measures. additionally, datasets from passive listening tasks, linked to fmri, have been released for various languages (e.g., french, mandarin, and english) @xcite . these datasets have been used, for instance, to study the impact of training parameters on a language model's ability to predict neurophysiological data @xcite . focusing on spontaneous speech, @xcite examined the predictability of fmri-derived signals from conversational variables, including lexical information. in terms of specialized language models, @xcite ) developed lms based on child-directed speech transcripts and evaluated them on word-sense disambiguation tasks. they concluded that word acquisition trajectories could be better captured by multimodal models that incorporate acoustic features, among other aspects. regarding more specifically tokenizers, @xcite proposed an evaluation paradigm focusing on the cognitive plausibility of subword tokenization. they compared bpe, wordpiece, and unigramlm and revealed a lower "cognitive correlation" for the latter. lastly, in the most recent babylm edition, @xcite introduced an interesting learning curriculum that constrained vocabulary in the early stages to simulate more cognitively plausible learning curves. although this approach did not yield consistent overall results, marginal gains were observed in selected tasks. ## a proposal for a new source of metrics all initiatives mentioned are grounded in text-based and/or handcrafted paradigms, potentially coupled with behavioral and /or physiological lab measures. in contrast, we propose using actual spontaneous conversational transcripts to build complementary benchmarks that test not only the syntax-semantics dimensions but also real-world language use. these metrics will remain fundamentally linguistic in nature rather than focusing on taskspecific or end-to-end evaluation. language is acquired, especially in its early stages, within spontaneous, conversational environments. while conversational language shares grammatical structures with other genres, its unique characteristics suggest that simply listing syntactic "errors" or semantic incongruities does not fully capture linguistic competence. furthermore, in a conversational context, what may be considered a production error from a formal grammatical perspective is often perfectly acceptable and successfully achieves its communicative purpose. therefore, we aim to develop a complementary approach that provides a broader set of metrics for evaluating language models from both cognitive and communicative perspectives when combined with existing benchmarks. specifically, we propose using spontaneous speech corpora, as they offer insights into human language processing through various observable production phenomena. our approach is a kind of classifier probing @xcite @xcite , but rather than focusing on meta-linguistic tasks (e.g., predicting syntactic categories), we aim to predict phenomena that serve as partial indicators of language processing. we propose a preliminary set of potential metrics, which remains open for further development. these metrics include speech reductions, listener's backchannel signals, prosodic prominences, and disfluencies. the common point among these metrics is that they are all grounded in spontaneous speech production, and each has been the subject of extensive research. ## experiments the experiments evaluated different pre-trained models for our set of tasks. 6 more precisely, we fine-tuned the pretrained models separately on a token classification task to predict which tokens were labeled (reduced / prominent / backchannelled) and which were not. a simple cross-validation was conducted across groups of speakers to maximize diversity across the folds. ## conclusion and roadmap in this position paper, we advocate for advancing the babylm initiative in several key areas. first, expanding beyond english is both necessary and feasible, given the initiative's design centered on "small-scale" data sets. here we used french as an example, but we have also built the mandarin equivalent datasets foot_9 , emphasizing the importance of multilingual perspectives. our proposal focuses on using training data composed entirely of spontaneous speech transcripts, which offers insights into language learning processes. it will be crucial to explore more nuanced variations in training data, such as balancing conversational speech, child-directed speech, and simple texts. equally important is the development of complementary evaluation metrics. we propose using spontaneous speech data to benchmark models and assess linguistic phenomena, such as speech reductions, prosodic prominences, and backchannel responses, as key indicators of human language processing. for the time being, we have english, mandarin, and french training datasets with different data mixtures. the next steps involve systematizing the pilot experiments on speech reductions conducted here for the mandarin dataset. then, we will extract all the other proposed metrics for the benchmark datasets. through this expanded set of experiments, we aim to demonstrate the value of the proposed approach and generalize it to other linguistic phenomena. in a broader perspective, we hope to show that benchmarks like blimp that require a significant amount of expert and naive human input to build, can be complemented with benchmarks derived from the numerous existing high-quality linguistic corpora, without additional human efforts.