BAMBINO - LM : (Bilingual-)Human-Inspired Continual Pre-training of B aby LM (2024)

Abstract: Children from bilingual backgrounds benefit from interactions with parents and teachers to re-acquire their heritage language. In this paper, we investigate how this insight from behavioral study can be incorporated into the learning of small-scale language models. We introduce BAMBINO-LM, a continual pre-training strategy for BabyLM that uses a novel combination of alternation and PPO-based perplexity reward induced from a parent Italian model. Upon evaluation on zero-shot classification tasks for English and Italian, BAMBINO-LM improves the Italian language capability of a BabyLM baseline. Our ablation analysis demonstrates that employing both the alternation strategy and PPO-based modeling is key to this effectiveness gain. We also show that, as a side effect, the proposed method leads to a similar degradation in L1 effectiveness as human children would have had in an equivalent learning scenario. Through its modeling and findings, BAMBINO-LM makes a focused contribution to the pre-training of small-scale language models by first developing a human-inspired strategy for pre-training and then showing that it results in behaviours similar to that of humans.

## introduction the recently held babylm challenge @xcite explores pretraining of language models using a constrained dataset analogous to the linguistic exposure of a 13-year-old english-speaking child. in this paper, we extend the babylm challenge to a bilingual setting, drawing inspiration from parent-child interactions in heritage language acquisition @xcite . immigrant children in western societies, who may have acquired their home language at a young age, can sometimes need to re-acquire the same language during the school years when the language becomes a minority. these heritage speakers typically benefit from an extended exposure to the minority language at home or in the community, owing largely to feedback and stimuli provided by parents and family members @xcite . this observation about child bilingualism is in line with the behaviorist theory for child language development @xcite . inspired by this line of work, we ask the following research question in the context of computational language modeling: can a small-scale language model trained on the majority language (e.g. english) be continually pre-trained on the minority language, leveraging the feedback of a second model that is fluent in the latter language? to address this question, we introduce 'bilingual language acquisition modeling based on interleaved optimization of language models (bambino-lm)', a novel continual pretraining strategy that uses a combination of alternation and proximal policy optimization (ppo) using a reward from a second model playing the parent role (i.e., a large language model pre-trained in the minority language). we experiment with babylm trained on english, and continually pretrain this model on an assumed second language, italian.in its connection to cognitive processing, our work makes the following contributions: pre-training small-scale language models is an emerging field that has garnered some interest from the language acquisition community. baby-berta @xcite ) is an early adaptation to this scenario. @xcite introduce the babylm challenge to provide an atypically small dataset for benchmarking small-scale language models. this shared task enables research in not only language acquisition but also sampleefficient pre-training. in the case of our paper, we do not focus on sample efficiency but instead describe ways to enhance the ability of a second language via continual pre-training. our work is conducted in a setup similar to @xcite , where a tiered first/second language acquisition process is attempted. @xcite also experiments with a teacher-student setting but only tests the approach on english tasks. @xcite is another closely related work, which investigates the learning trajectory of large-scale language models by probing their syntactic and semantic capabilities at each step. conventionally in language generation for natural language processing, the design of feedback signals is commonly discussed in the context of knowledge distillation @xcite . recently, reinforcement learning from human feedback (rlhf) utilizes human preferences for serving reward signals when dealing with sparse training labels @xcite , and has been shown successful for generative tasks such as dialogues and summarization. this approach is further extended in @xcite by using ai feedback (rlaif) to remove the dependency on human preference data, leading to better scalability and signal availability. the approach we take in this paper mostly falls within the latter camp, but generally departs from all prior efforts in the way the parent model's perplexity is used to signal the conformity of the child model's generation. this is in contrast to sequencelevel knowledge distillation @xcite where teacher's generation is used to guide the learning process. ## methods figure 1 shows the two phases of bambino-lm. the learning phase involves continual pretraining a small-scale language model (baby model b) whose initial pre-training was originally done on english data, while the feedback phase involves interactions with the italian language model (parent model p). during the learning phase, pre-training for b is continued by employing causal language modeling on italian data. causal language modeling (clm), also known as next token prediction, is a standard technique to train a decoder-only model. the objective is defined as follows: there are two architectural innovations in bambino-lm: feedback phase based on ppo we construct prompt x by selecting the first k tokens from the training example and solicit output y @xmath0. we then use proximal policy optimization (ppo) where b's parameters are updated according to a clipped surrogate objective @xcite . this objective moderates the updates to the policy, facilitating stable and efficient learning by incorporating a clipping mechanism. its definition is given as follows: in the autoregressive setting of language modeling, θ controls the generation of tokens based on the given state or context s t . the probability ratio r t (θ) quantifies the change in the likelihood of selecting action a t (the next token), under the updated policy parameters compared to the previous parameters θ old . this ratio provides understanding on the impact of parameter updates on the policy's behavior, ensuring that changes do not excessively deviate from the previous policy, thereby maintaining training stability. the clipping mechanism, defined by clip(r t (θ); ϵ), restricts r t (θ) within the bound [1ϵ, 1ϵ], mitigating the risk of large policy updates that could lead to divergence. the advantage function, ât = r + γv (s t+1 ) -v (s t ), which reflects the relative gains of selecting a t given s t . this function guides the optimization process by favoring actions that lead to better than expected outcomes. the reward r for the advantage function is then calculated using the following function: ## results in removing the interactive feedback mechanism (w/o ppo) and the alternating strategy (w/o alternating) significantly decreases italian performance compared to our primary model. on uinauil tasks, the average score drops from 0.4613 to 0.4000 (w/o ppo) and 0.3513 (w/o alternating). however, we do not observe significant improvements in performance in both the english language task sets (glue and blimp). for glue tasks, the average scores remain consistent, with 0.4373 for bambino-lm, 0.4375 for w/o ppo, and 0.4357 for w/o alternating. on the blimp dataset, the average scores are 0.5503 for bambino-lm and 0.5554 for w/o ppo. these results indicate that ppo modeling and alternating runs are both crucial for improving the bilingual ability of bambino-lm without negatively impacting english performance. furthermore, the lack of significant changes in english scores reinforces that these strategies enhance bilingual capabilities without compromising performance on existing benchmarks. ## conclusion in future work, we aim to explore the effect of alternative metrics and different reward learning mechanisms that better align with human feedback behaviors. this also includes exploring rewards that capture linguistic quality and provide direct, "constructive" corrections to the model output which is commonly known as an effective learning strategy for language development. although bambino-lm was applied for second language learning with italian as an example, the method must be validated for other languages, especially languages that are distant from english or those that use a different set of tokens. the degradation in the performance of the first language, english, points to the potential of alternating with language modeling for the first language.