Training Bilingual LM s with Data Constraints in the Targeted Language (2025)

Abstract: Large language models are trained on massive scrapes of the web, as required by current scaling laws. Most progress is made for English, given its abundance of high-quality pretraining data. For most other languages, however, such high quality pretraining data is unavailable. In this work, we study how to boost pretrained model performance in a target language with insufficient pretraining data for training a high performing language model by enlisting data from an auxiliary language for which high quality data is available. We study this by quantifying the performance gap between training with data in a data-rich auxiliary language compared with training in the target language, exploring the benefits of translation systems, studying the limitations of model scaling when data is limited in the target languages, and proposing new methods for upsampling data from the auxiliary language. Our results show that stronger auxiliary datasets result in performance gains without modification to the model or training objective for close languages, and, in particular, that performance gains due to the development of more information-rich English pretraining datasets can extend to targeted language settings with limited data.

## introduction the abundance of high quality english data has given rise to exceptional language modeling performance in english @xcite @xcite . however, there are many other languages for which a reasonably large amount of data is available, yet not in equally large amounts as for english. taking mc4 @xcite as an example, languages like german, french, chinese, and japanese have ∼10-100 times less data than english. consequently, most non-english progress comes from relatively small bilingual models (e.g., @xcite @xcite @xcite , or larger massively multilingual models (e.g., le scao et al., 2023; intrator * equal contribution et al., 2024; üstün et al., 2024). other llms such as llama-2, gpt-3, and palm-2 that perform well across a variety of languages are trained primarily on english, with less than 20% of data from other languages @xcite . several works study cross-lingual transfer, but are limited to finetuning smaller models (∼100m parameters) with limited data, or classical nlp tasks, such as part-of-speech tagging, named entity recognition, and natural language inference (e.g., @xcite @xcite . they do not evaluate pretrained language models on contemporary knowledge-based downstream tasks. it is still unclear how to optimally make use of bi-or multilingual data, to optimally increase performance in a target language. this is especially relevant in data constrained settings. for example, imagine one has access to only 500k documents in a target language, and wants to pretrain a language model for this language. 500k documents is sufficient to attain a reasonable perplexity score, but this amount of data is insufficient to do well on modern knowledge-based downstream tasks. many languages in the tail of mc4 fall in this category. currently, a systematic analysis of how to use an auxiliary language in this scenario is lacking, leaving practitioners to a trial-and-error based approach, mostly based on intuition. in this work, we provide such a systematic analysis. in particular, we are interested in whether "higher quality" auxiliary data also leads to higher model performance in the target language. we choose english as the main auxiliary language, matching a common practical setting. to investigate the effect of linguistic similarity, we also experiment with chinese as the auxiliary language, although being somewhat bottlenecked by the amount of available data in chinese. to measure the impact of "better" auxiliary data, we explore well-tested filtering techniques for a monolingual, english, setting. we focus on the impact of dataset size, filtering for data quality and style, and data selection for specialized information relevant to downstream evaluations (figure 1c ), matching recent advancements in state-of-the-art open-source english datasets such as finewebedu @xcite , and datacomp @xcite (figure 1 ). currently, it is still unclear how these techniques extend to a bilingual setting; as we confirm in our experiments, not every filtering technique that performs well in a monolingual setting performs well in a bilingual setting. specifically, our key findings are: 1. auxiliary english data that is generated by some of the existing model-based data filtering pipelines for english can be helpful to complement limited data in a target language ( §3.2); 2. findings are not the same across multiple target languages. we hypothesize that for languages that are "far" from english, gains from better english datasets do not help ( §3.3); 3. when training with high quality auxiliary data, there is a marginal gap between upsampling relevant information and filtering for high quality instructional text ( §4.1-4.2); 4. there are limits to the model sizes that are practical to pretrain with limited target data: data size should scale linearly with model size @xcite , and performance in the target language saturates without increasing target data, regardless of increasing auxiliary data ( §5.2). multilingual language models while much of the llm research has focused on english, large-scale transformer-based multilingual language models have been trained on large multilingual corpora including mbert @xcite , xlm (conneau and lample, 2019), mt5 @xcite , polylm @xcite , and bloom @xcite . these works focus on training models that are language balanced and that perform well across multilingual benchmark tasks in over 100 languages. other sota language models such as llama 2 @xcite , falcon @xcite , and palm 2 @xcite have multilingual capabilities, but over 90% the training data is english, and these models perform poorly across a variety of languages, such as south east asian languages @xcite . other works focus on training smaller bilingual language models in french @xcite @xcite , german @xcite , dutch @xcite , or chinese @xcite , but require a substantial amount of bilingual data: english and the respective language for pretraining. other works focus on understanding languages llms reason in @xcite , and languages llms cannot learn @xcite . still, little work has examined how information seen during pretraining in one language can help downstream task performance in another language. cross-lingual transfer @xcite present a comprehensive survey on cross-lingual transfer in multilingual language models. the survey explains that cross-lingual transfer is a well studied topic for classic nlp tasks, such as part-ofspeech tagging, named entity recognition, dependency parsing, machine translation, etc., although findings are not always consistent. cross-lingual transfer is less well studied for the language modeling objective, and for modern downstream evaluation tasks, such as arc @xcite , hellaswag @xcite , piqa @xcite , sciq @xcite , wino-grande @xcite , etc. philippy et al. end with a number of recommendations, one of which is to study cross-lingual transfer in more detail for generative models, given their recent exceptional performance. our work differs from prior work on cross-lingual transfer in three ways: (i) we do not directly study cross-lingual transfer as we do not first train on the auxiliary language, and then on the target language, (ii) we pretrain 1b and 3b parameters models that are 10-30x the size of typical bert models, and (iii) instead of finetuning, we focus on how models learn languages during pretraining. data selection selecting high quality data for pretraining llms remains an active area of research. early research on data selection was based on heuristics. for example, the original gpt-2 model was pretrained on outbound links filtered from reddit, based on heuristic indicators of whether users found the links interesting, educational, or funny @xcite . prior approaches also upsampled documents from high quality sources like wikipedia @xcite , or used combinations of heuristics such as absence of stop words, document length, word length, etc. @xcite . other works select data based on quality filters and time of collection @xcite , model-based quality filtering @xcite , or textbook quality knowledge @xcite @xcite ). an alternative approach to data selection is re-weighting data samples to select the best data mixtures for training @xcite , or importance sampling based on a downstream task @xcite . still a majority of these filtering techniques are applied to englishonly datasets, and multilingual datasets such as mc4 have limited data filtering @xcite . ## using english data selection pipelines to complement limited target data existing data selection pipelines have been shown to be effective in monolingual (english) pretraining. we investigate whether these pipelines are useful in the bilingual setup with limited target data. in §3.2, we report our findings for experiments with english as the auxiliary language and german as the example target language. in §3.3 we discuss how our findings extend across multiple languages. ## the effect of individual data transformations in §3.2 we investigated the effect of using existing data filtering pipelines in the auxiliary language, but this does not isolate which components of the pipeline provide the best downstream improvements. here, we study individual data transformations that comprise the training pipeline, including model based quality filtering and educational content upsampling, as well as transformations on top of auxiliary data, such as translation. for all experiments, we use the same experimental setup and baselines as in §3, and we focus on german as the target and english as the auxiliary language. ## data and model scaling to study the impact of data and model scaling, we investigate two scaling questions: (1) how does the required quantity of additional auxiliary data relate to the amount of data required if one had enough target language data to train on? ( §5.1) (2) how do findings extend to a larger bilingual model? ( §5.2) following our setup in the previous section, we target our investigation to german (target) and english (auxiliary). ## conclusion this work studies how an auxiliary language for which an abundance of training data is available can boost pretraining for a target language for which only limited data is available. we find that adding auxiliary high quality data obtained by data filtering can improve performance in a target language. however, we find that results are inconsistent across target languages. we hypothesize that for languages further from english, better english datasets are not as helpful as information is not shared between them. finally, we find limitations to scaling models for languages that are data constrained. this work takes a step towards pretraining language models in languages with limited data, and can inspire more research into bilingual learning under data constraints.