Using Curriculum Masking Based on Child Language Development to Train a Large Language Model with Limited Training Data (2024)

Abstract: In this paper we detail our submissions to the Strict and Strict-Small tracks of the 2024 BabyLM Challenge. We approach this challenge with two methodologies: i) use of a novel dataset, and ii) development of a pre-training technique based on the fusion of child language acquisition with traditional masked language modeling, which we call curriculum masking. The novel dataset used for this task is based on user submissions to the Reddit forum (i.e., subreddit) “Explain Like I’m Five”, which explains diverse concepts using simple language. Curriculum masking works by creating learning phases based on a standard child language development timeline, where the masked words learned by the model start with simple nouns and gradually expand to include more complex parts of speech. We show that using internet-based training data shows a small improvement in evaluation scores as compared to baseline training data. Our proposed pre-training method of curriculum masking is conceptually novel and also shows improved rates of learning over typical masked language modeling pre-training, potentially allowing for good performance with fewer total epochs on smaller training datasets. Code for the curriculum masking implementation is shared at https://github.com/evan-person/curriculumMaskingBabyLM2024.

## introduction children acquire language skills through exposure to an estimated two to seven million words per year. however, contemporary large language models (llms) require training on massive datasets comprising billions to trillions of words to achieve similar linguistic capabilities. the vast disparity between human language acquisition and current machine learning practices can be shown from the 1 https://github.com/evanperson/curriculummaskingbabylm2024 chinchilla model @xcite , which was trained on 1.4 trillion words. to address these disparities, the babylm challenge was established to explore the feasibility of pre-training llms on datasets comparable in size to those encountered during early childhood language development. it continues on this mission, imposing strict limits on the size and composition of training datasets and aims to create models that learn language in a child-like manner. in this paper, we present our submissions to both the strict and strict-small of the 2024 babylm challenge. we leverage a novel dataset, sourced from the reddit forum (i.e. subreddit) explain like i'm five (eli5). we introduce a curriculum masking training strategy that we designed to mimic how children learn language. traditional masked language modeling (mlm) masks random words during training, which does not accurately reflect the structured manner in which children acquire language. our curriculum masking approach organizes the process into a schedule of stages, starting with simpler words such as nouns, then gradually incorporating more complex words like adjectives and verbs. we hypothesized this scheduling method would help the model build a stronger foundation in language before tackling more advanced sentence structures. we do not count the added information of pos tags as additional word count, as it is only an additional categorical variable attached each token. in conversations on the challenge slack channel, this was agreed to not count towards overall word count for this reason, and so although it is additional information, we make the claim that it does not count as increased training words. the following sections of this paper describe our dataset preparation, implementation of curriculum masking, experimental results, and discussion of the effectiveness of our approach. ## related work to understand the state of related work, we reviewed the most relevant papers from the babylm 2023 challenge @xcite and performed searches for similar concepts to what we propose in this work. several works in the prior challenge utilized curriculum learning, though they administered their curricula through an intentional sequencing of the examples in their training set as opposed to the curriculum-based masking approach we use. @xcite investigated curriculum learning strategies for language model pre-training using limited data. similar to our curriculum masking, their approach progressively increased task complexity, structuring model training in stages. although their methods did not consistently outperform non-curriculum baselines, their focus on vocabulary and data pacing offers valuable insights for optimizing training with limited resources. debenedetto (2023) applied a curriculum learning strategy for low-resource settings, where datasets were ranked by difficulty using a bytesper-line metric. simpler datasets, such as spoken transcriptions, were introduced first, then more complex datasets were gradually introduced during training. their approach outperformed baseline models in most downstream tasks, including blimp and superglue. their curriculum learning methods demonstrated consistent improvements in performance, particularly when trained with more epochs. bunzeck and zarrieß (2023) designed a curriculum learning approach based on child-directed speech which showed an improvement for certain tasks like anaphor agreement, irregular forms, and quantifiers. their work, similar to ours, involved using curriculum learning focused on word frequency and sentence structure. however, they used a static data ordering approach where the training data was organized in a fixed sequence. curriculum masking as a concept has been applied successfully in other domains, such as computer vision. @xcite developed a curriculum based masking strategy for vision tasks. they show that by using a curriculum-based masking approach for training vision models they are able to outperform the same model architecture on some common image classification tasks. ## method for our submission, we make two primary contributions: i) a new eli5 dataset, and ii) a method of curriculum learning that modifies the mlm pretraining task to mimic child language development. in this section we review these two contributions as well as the other method choices made. ## conclusions in our submission to the 2024 babylm challenge, we focus on using an internet-based training dataset that mimics language that would be directed at youth as well as utilizing a developmentally plausible pre-training approach that allows the model to learn specific parts of speech on a schedule. we show that by combining both of these approaches, we can outperform the baseline provided by organizers on the strict-small track (10m word limit) of the challenge, although we did not succeed at outperforming the baseline for the small track (100m word limit). due to a lack of hyperparameter optimization, there is probably a lot of improvement that could be made using curriculum masking, especially considering different masking ratios or masking ratio schedules. one other possibility we are interested in is varying the pos acquisition order and experimenting with the use of training the model on a mix of the pos on the schedule as well as some other words. testing the curriculum masking concept on an autoregressive model would be an obvious thing to try as well. our findings help reinforce the idea that using internet-scraped data provides highly useful data for teaching a language model language understanding as well as world knowledge. additionally, our proposed method of curriculum masking introduces a new method of curriculum learning that shows accelerated learning in our tests on a small dataset. linzen, et al. 2023. findings of the babylm challenge: sample-efficient pretraining on developmentally plausible corpora. in proceedings of the babylm challenge at the 27th conference on computational natural language learning. sang michael xie, hieu pham, xuanyi dong, nan du, hanxiao liu, yifeng lu, percy s liang, quoc v le, tengyu ma, and adams wei yu. 2024. doremi: optimizing data mixtures speeds up language model pretraining. advances in neural information processing systems, 36.