Are B aby LM s Second Language Learners? (2024)

Abstract: This paper describes a linguistically-motivated approach to the 2024 edition of the BabyLM Challenge. Rather than pursuing a first language learning (L1) paradigm, we approach the challenge from a second language (L2) learning perspective. In L2 learning, there is a stronger focus on learning explicit linguistic information, such as grammatical notions, definitions of words or different ways of expressing a meaning. This makes L2 learning potentially more efficient and concise. We approximate this using data from Wiktionary, grammar examples either generated by an LLM or sourced from grammar books, and paraphrase data.We find that explicit information about word meaning (in our case, Wiktionary) does not boost model performance, while grammatical information can give a small improvement. The most impactful data ingredient is sentence paraphrases, with our two best models being trained on 1) a mix of paraphrase data and data from the BabyLM pretraining dataset, and 2) exclusively paraphrase data.

## introduction language models (lms) need a lot of data in order to learn to approximate human linguistic behaviour @xcite . the amounts of linguistic data typically used for training recent lms is is significantly larger than what is available for most of languages of the world, and also much more than what children are typically exposed to during their first language acquisition. a 13 year old is typically exposed to less than 100 million words of linguistic input, which is orders of magnitude less than the amount used in lm pretraining. and still, lms fail to be quite as good in language as human learners. can we teach our models to be more data-efficient? if yes, how? there are two potential strategies. one is to study how children acquire language in a natural setting, and use their acquisitional trajectories and patterns as inspiration for lm training. this intuition is one of the motivations for the babylm challenge (hence the name; other low-resource pretraining contexts are, of course, also relevant): the challenge encourages lm pretraining optimization advancements inspired by human linguistic development @xcite . another direction is to embrace the obvious differences between lm pretraining and the ways human learners acquire their native language. the architectures of current lms are dramatically different from human brain anatomy, and training objectives and strategies have only limited psycholinguistic developmental parallels. finally -and most importantly for our contribution -input for first language acquisition by human learners and for lm pretraining is hardly comparable not only when it comes to dataset size. while the amount of strictly linguistic input that children get is small compared to typical lm training data, children get this input in communicative context that lms lack at the pretraining stage, and it is typically paired with crossmodal data, which is not part of the strict-small track we choose for the babylm challenge. at a very high level, taking this second direction means that we look beyond human linguistic and cognitive development for optimization strategiesor at least, we do not need to expect that those will be the ones that necessarily work best. we sharpen this point and contrast language learning in an acquisitionally realistic setting (firstlanguage, or l1, acquisition) -and language learning in a more artificial setting -learning a second language, l2; a human activity that also leads to (different levels of) linguistic proficiency but contrasts dramatically with l1 acquisition by children. almost everything is different: the set-up, the data, typical tasks the learner faces, and very often modality and their combinations. we choose this particular direction mainly be-cause in the current, second, edition, of the babylm challenge participants are allowed to construct their own datasets within the track word budget. a lot of submissions last year, including ours, experimented with curriculum learning -different ways to order the same data (see our submission @xcite as well as the babylm 2023 findings @xcite ). these attempts gave only limited results. this year we instead focus on the effect of choosing different data on lm pretraining. in particular, roughly in line with how people learn foreign languages through explicit linguistic instruction, we divide training data into blocks roughly corresponding to types of linguistic information commonly found in english-as-a-foreign-language courses. we participate in the strict-small track allowing for only 10m words and experiment with four different types of linguistic information: we find that data on paraphrasing brings in the most significant improvements. grammatical information is only marginally useful, even though it does come with some improvement, depending on the training set-up. finally, lexical information does not seem useful for lm pretraining. one cannot be sure what to attribute these results to: the usefulness or lack thereof of particular types of data; the quality of the actual various datasets that we use; or the properties of evaluation used to judge whether a particular type of data is useful. one way or another, our answer to the question of whether babylms are l2 learners is 'only when it comes to certain types of data'. we think babylm data roughly corresponds to unconstrained linguistic input in a language learner's experience (reading materials and practice conversations with language teachers and peers). the rest of the data in the half/half model comes from the dataset we discuss next. ## results we first discuss the results of our experiments with mlm-only models trained on grammar and lexical data, then we move on to discuss the results of the models with additional training schemes. finally, we cover the results of our best-performing models that we submitted to the challenge. ## conclusion in this year's babylm challenge, we attempted to buck the trend of administering strategies based on l1 acquisition, having seen little success from such strategies in last year's challenge. instead, we hypothesized that l2 acquisition, with more explicit information regarding semantics and syntax, might be what a language model needs. to that end, we also saw limited success. our strategy of using wiktionary data did not show any indication of im-proved output quality. using grammar information did have a small positive effect on blimp scores, though it is unclear whether the grammar itself helped or simply the more diverse data domain. nevertheless, our strategy of reducing context size from the previous year was yet again successful at outperforming the baselines, even those with 10× more data used in training. additionally, using data that includes paraphrases and contrastive pairs helped improve the glue scores by a remarkable 8 points. this goes to show that the data chosen for low-resource pretraining can have a profound impact. the study of the exact structure of data that lms efficiently learn from is a productive future direction, as tentatively shown by our results.