Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies (2024)

Abstract: Curriculum Learning has been a popular strategy to improve the cognitive plausibility of Small-Scale Language Models (SSLMs) in the BabyLM Challenge. However, it has not led to considerable improvements over non-curriculum models. We assess whether theoretical linguistic acquisition theories can be used to specify more fine-grained curriculum learning strategies, creating age-ordered corpora of Child-Directed Speech for four typologically distant language families to implement SSLMs and acquisition-inspired curricula cross-lingually. Comparing the success of three objective curricula (Growing, Inwards & MMM) that precisely replicate the predictions of acquisition theories on a standard SSLM architecture, we find fine-grained acquisition-inspired curricula can outperform non-curriculum baselines and performance benefits of curricula strategies in SSLMs can be derived by specifying fine-grained language-specific curricula that precisely replicate language acquisition theories.

## introduction curriculum learning (cl) has emerged as a promising method to improve the cognitive plausibility of small-scale language models (sslms) in the first babylm challenge @xcite , as a way to gradually introduce more complex linguistic phenomena into the model later in training in a manner that is similar to human language acquisition. cognitively-inspired sslms are models trained on corpora that approximate the volume and nature of input that a first-language learner can expect to receive during language acquisition. these have been found to perform competitively against llms in english @xcite . cl strategies implemented in the babylm challenge either specified a static measure of linguistic complexity, such as lexical frequency @xcite , sorted datasets according to difficulty @xcite , or gradually increased vocabulary sizes @xcite . while the majority of these strategies did not yield consistent improvements over non-curriculum learning baselines @xcite , linguistic theory suggests that children naturally focus on input that is neither too simple nor too difficult but at the right level of challenge for learning @xcite . this is known as the "goldilocks effect", which is a form of self-selecting curriculum learning that appears to naturally occur in first language (l1) acquisition. this raises the question of whether acquisition theories can provide insights into more effective curriculum learning strategies for sslms, and lead to more consistent benefits of cl strategies. our work assesses whether language acquisition theories can provide us with better heuristics for good curriculum learning strategies to train sslms. we compare contrastive acquisition theories for their success when informing objective curriculum learning strategies on a standard architecture @xcite . we train sslms with three new objective curricula called growing, inwards and mmm, each replicating the developmental sequences of contemporary acquisition theories that first-language monolingual learners are theorised to follow in the earliest stages of acquisition cross-linguistically. in practice, these curricula modify the standard masked language modelling objective in babyberta-style models by varying the order and the sequence of masking using different tagsets to simulate different language acquisition theories. the acquisition models specify different crosslingual and language-specific developmental sequences that learners appear to follow in first language acquisition, which has not been implemented or evaluated in the context of deep learning. the multilingual focus of the acquisition models is a goal strongly aligned with the spirit of the babylm shared task. we train sslms with these objective curricula for four typologically distant language families: romance (french), germanic (german), japonic (japanese) and sino-tibetan (chinese). we introduce new age-ordered corpora of child-directed speech (cds) for these languages and select languages for pre-training based on the quantity of cds that can be used to train sslms using similar volumes of data that learners can utilise in first language acquisition. we evaluate these sslms on syntactic minimal pair datasets. we find benefits of the cognitively-inspired objective curricula cross-linguistically, however different strategies lead to better performance for certain languages, particularly finer-grained language-specific versions of the mmm objective. acquisitioninspired objective curricula can obtain comparable performance on minimal pair evaluation datasets to llms, despite requiring approximately 25x fewer parameters and 6,000x fewer words. ## background we survey curriculum learning (cl) strategies used in the 1 st babylm challenge section 2.1 and contrastive models of syntactic acquisition that are utilised to replicate cross-lingual developmental sequences for implementing more cognitively plausible pre-training in sslms in section 2.2. ## results the performance of objective curricula and crosslingual sslms on minimal pairs datasets is summarised in ## discussion acquisition-inspired cl strategies represent a novel large-scale application of language acquisition theory in deep learning, aimed at improving the performance of sslms. acquisitioninspired curricula guide sslms, which function as large statistical learners, to generalise over frequent linguistic categories-such as nouns and verbs-early in the training process and attend to language-specific features, such as the germanic v2 word order. this suggests that more finegrained, language-specific curricula may have performance benefits over non-cl strategies in sslms, which is supported by results showing the limited improvements of universal/maturational theories of acquisition that inform the grow-ing and inwards strategies. although both acquisition models predict universal curricula that should lead to consistent benefits cross-lingually, growing/inwards only improve performance in chinese and japanese, while performing comparably to non-curriculum (non-cl) baselines in french/german and worse than non-cl baselines in english. an additional benefit of using finegrained language-specific curricula is that it enables sslms to learn more complex grammatical phenomena that may rely on semantics like anaphora. we notice notable improvements in ellipsis performance ( ## conclusion this paper assesses whether fine-grained curriculum learning strategies based on acquisition theories can provide better heuristics for cl strategies for sslm pre-training cross-lingually, introducing the mao-childes training corpus to train sslms for four typologically distant language families. mixed results of the maturational growing and inwards acquisition theories in curriculum strategies and the implementation of the coarse/universal prediction of mmm (upos) suggest that there is no guaranteed performance benefit just by devising universal cl strategies based on acquisition theories for sslms in a multilingual setting. training sslms using more fine-grained language-specific curricula that precisely replicate cutting-edge linguistic theories is effective for the mmm (sem) objective in english and chinese and mmm (upos) in japanese. curriculum learning can outperform non-curriculum sslms by specifying fine-grained language-specific curricula that precisely replicate language acquisition theories, highlighting how cognitively-inspired techniques can lead to better-performing data-efficient architectures in the spirit of the babylm challenge.