Towards Data-Efficient Language Models: A Child-Inspired Approach to Language Learning (2024)

Abstract: In this work, we explain our approach employed in the BabyLM Challenge, which uses various methods of training language models (LMs) with significantly less data compared to traditional large language models (LLMs) and are inspired by how human children learn. While a human child is exposed to far less linguistic input than an LLM, they still achieve remarkable language understanding and generation abilities. To this end, we develop a model trained on a curated dataset consisting of 10 million words, primarily sourced from child-directed transcripts. The 2024 BabyLM Challenge initial dataset of 10M words is filtered to 8.5M. Next, it is supplemented with a randomly selected subset of TVR dataset consisting of 1.5M words of television dialogues. The latter dataset ensures that similar to children, the model is also exposed to language through media. Furthermore, we reduce the vocabulary size to 32,000 tokens, aligning it with the limited vocabulary of children in the early stages of language acquisition. We use curriculum learning and is able to match the baseline on certain benchmarks while surpassing the baseline on others. Additionally, incorporating common LLM training datasets, such as MADLAD-400, degrades performance. These findings underscore the importance of dataset selection, vocabulary scaling, and curriculum learning in creating more data-efficient language models that better mimic human learning processes.

## introduction language models (lms) have revolutionized natural language processing, demonstrating remarkable capabilities in understanding and generating human-like text. however, the training of these models typically requires vast amounts of data, often billions of words, which stands in stark contrast to how human children acquire language. the babylm challenge @xcite seeks to bridge this gap by exploring methods to train lms more efficiently, using significantly less data while still achieving high performance. human children develop impressive language skills despite being exposed to far less linguistic input than traditional large language models (llms). this observation raises intriguing questions about the efficiency of human language acquisition and the potential for more data-efficient machine-learning approaches. our research addresses these questions by mimicking the human language acquisition process. in this work, we present our approach to the babylm challenge, focusing on developing a model that can learn effectively from a dataset more closely aligned with the linguistic experiences of a young child. our primary contributions are as follows: ## methodology our approach to the babylm challenge involves careful data preparation followed by the implementation of a curriculum learning strategy. this section details our methods for dataset curation and the subsequent application of curriculum learning. we started with an initial dataset of approximately 10 million words, primarily sourced from child-directed transcripts. this dataset was chosen to closely mimic the linguistic input that young children typically receive during their language acquisition process. to enhance the quality and relevance of our training data, we implemented a rigorous filtering pro-cess as explained next: duplicate removal: similar to (rae et al., 2021), we identified and removed exact duplicate sentences from the dataset. this step helps to prevent overfitting to specific phrases and ensures a more diverse linguistic input. content refinement: after duplicate removal, we further refined the dataset based on relevance and quality criteria. for instance, we excluded data points where the ratio of punctuation marks to total words exceeded 0.33 and removed samples with less than 10 characters, resulting in a reduced dataset of approximately 8.5 million words. recognizing that modern-age children often acquire language partially through media exposure, we supplemented our refined dataset with television dialogue. we carefully selected approximately 1.5 million words of television dialogue, focusing on content appropriate for and often consumed by young children. the tv data from the tvr dataset @xcite was added to our refined 8.5 million word dataset, resulting in a final training corpus of about 10 million words. the inclusion of tv dialogue adds diversity to our dataset and better reflects the varied sources of language input in a child's environment. following the data preparation phase, we implemented a curriculum learning approach to optimize the training process. this method is designed to present the model with progressively more complex linguistic inputs, mimicking the natural progression of language acquisition observed in human learners. we developed a set of scoring functions to assess the complexity of each data point in our dataset. these functions evaluate various linguistic features as briefed next. each data point is passed through these scoring functions, generating a set of individual scores that capture different aspects of linguistic complexity. once the complexity score is calculated for each data point, we sort the entire dataset in ascending order of these scores. the sorted dataset forms the basis of our curriculum learning approach. training begins with the least complex data points (lowest scores). as training progresses, more complex data points are introduced. by the end of training, the model has been exposed to the full range of linguistic complexity present in the dataset. throughout the training process, the model's learning rate decreases. revisiting simpler examples in later epochs with a lower learning rate helps fine-tune the model's understanding of fundamental concepts while reducing the risk of overfitting. this gradual exposure to complexity allows the model to build a foundational understanding of simpler linguistic structures before tackling more complex ones, potentially leading to more robust and efficient learning. ## discussion we hypothesize that data valuation and attribution methods could offer significant advantages over current data selection techniques. while not directly implemented in our study, methods such as influence functions (koh and liang, 2017) , representer point @xcite , and dynamic approaches like tracin @xcite and hydra @xcite , or rl-based methods for data valuation @xcite , show promise as potential tools for more effective data curation. these techniques, originally designed to quantify the impact of individual data points on model performance, could potentially be adapted to filter large datasets into smaller, higher-quality subsets. unlike traditional data selection methods such as number of characters @xcite , frequency @xcite , or using a blocklist @xcite that may rely on simplistic criteria, these advanced techniques could provide a more nuanced understanding of data importance. by identifying the most influential or informative samples, they might enable researchers to create more compact yet equally effective training sets. this approach could lead to reduced com- ## limitations despite the promising results, this study has several limitations. first, our approach relies on the weights used for scoring data during curriculum learning. with a different set of weights, performance may even decline compared to not using curriculum learning. furthermore, these weights may vary across different datasets, and finding their near-optimal values could be computationally expensive. second, the appropriate amount of tv data was selected experimentally and may differ for other datasets. lastly, the effect of training with this procedure on downstream tasks is unclear and may negatively impact model performance in those tasks. future research should aim to address these limitations by developing a reliable and robust method for determining score weights, selecting the appropriate portion of tv data, and assessing the influence of this approach on downstream task performance. ## conclusion this study, conducted as part of the babylm challenge, has yielded several significant insights into the development of data-efficient language models that more closely mimic human language acquisition. our approach, focusing on careful dataset curation, vocabulary scaling, and curriculum learning, has demonstrated promising results in training a language model with substantially less data than traditional large language models. these results have important implications for both cognitive science and practical nlp applications. by demonstrating that effective language models can be trained on significantly smaller datasets, our work contributes to the ongoing discussion about data efficiency in ai and machine learning. furthermore, our findings suggest potential avenues for developing more cognitively plausible models of language acquisition, which could inform both ai research and our understanding of human language learning.