Assessing the Role of Data Quality in Training Bilingual Language Models (2025)

Abstract: Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more languages can degrade performance for some languages (such as English), while improving others (typically more data constrained languages). In this work, we investigate causes of these inconsistencies by comparing bilingual and monolingual language models. Our analysis reveals that unequal data quality, not just data quantity, is a major driver of performance degradation in bilingual settings. We propose a simple yet effective data filtering strategy to select higher-quality bilingual training data with only high quality English data. Applied to French, German, and Chinese, our approach improves monolingual performance by 2–4% and reduces bilingual model performance gaps to 1%. These results highlight the overlooked importance of data quality in multilingual pretraining and offer a practical recipe for balancing performance.

## introduction language models (lms) exhibit exceptional performance on a number of language understanding and knowledge tasks @xcite @xcite . while much of the effort in training language models is focused solely on english, recent bi-or multilingual models also incorporate other languages @xcite @xcite @xcite @xcite . in many scenarios, it is beneficial to train a multilingual model as (i) maintaining a separate model for each language can be costly in memory and inference constrained settings, (ii) data relevant to some * equal contribution tasks may only be available in specific languages, and (iii) for many languages, the amount of available high quality data is insufficient for pretraining monolingual language models. in contrast to work in the vision domain, which shows scaling data (even noisy or from different domains) improves model performance @xcite , prior work in language modeling shows that multilingual models are prone to degradations relative to monolingual models @xcite @xcite . fundamentally, training multilingual models requires learning the structure and semantics of each language. thus, such models may require training for longer @xcite and on better data to reach the same performance as prior work shows that the sample complexity of learning from multi-distribution data grows with the number of distributions @xcite . critically, there are two main deficits in prior investigations: data quality. while prior work studies how data size impacts performance degradations in multilingual models, they do not study data quality in multilingual models @xcite . previous papers define data quality according to a few core principles: (i) fluent language @xcite , (ii) long form text @xcite , and (iii) informative text with educational content and textbook format @xcite . we discuss data selection in more detail in section 2. data quality has already been shown to be an important factor in training high performing english language models @xcite @xcite . because of this, there is growing interest in model-based filters for curating high quality data in monolingual settings @xcite @xcite . there are many other languages for which training a language model is practical as a reasonably large amount of unfiltered data is available @xcite . however, the importance of data quality for training multilingual models and data quality filtering in multilingual settings has received little attention. there are several challenges that can arise from filtering high quality data for multilingual pretraining as (i) quality filters may work differently across languages, (ii) the density of high quality data and filtered topics may vary, and (iii) the impact of quality filtering may have a small impact across languages in multilingual settings. model and data size prior works that study gaps in multilingual model performance typically aim to study performance gaps from training on a large number of languages (typically on the order of hundreds), small data per language, and smaller encoder-style architectures @xcite @xcite . in this setting, they refer to the gap as the curse of multilinguality. however, studying gaps in performance in these settings greatly impacts the evaluations that are feasible, makes it difficult to control training data at scale, and does not control for the fact that training a multilingual model is simply a more challenging task, and may require longer training time or higher capacity to achieve the same performance with the same amount of data. this work focuses on exploring how these challenges underlie gaps in multilingual performance. we conduct experiments on data quality (measured by information/knowledge) and language in french and chinese, where we control both through training on translated data. we also provide a recipe for obtaining high quality data in other languages that improves bilingual model performance in three languages: french, german, and chinese foot_0 . collectively, our main contributions show that (i) high quality data filtering in multiple languages without access to native high quality data improves performance in the target language, and reduces gaps in monolingual and bilingual performance. (ii) data quality plays an important role in the performance of bilingual language models (rather than only the language or data size). (iii) high quality english data alone is insufficient for training high performing multilingual language modeling in some tasks. multilingual data curated datasets are essential to training language models. early multilingual datasets include ccnet @xcite , mc4 @xcite , and culturax @xcite all support over 100 languages, though the largest sources of data are english and even other high resource languages such as german, french, chinese, and korean contain 10-50× less data. other datasets such as redpajamav2 contain over 2.5t tokens, but are limited to only a few indo-european languages @xcite , and still a factor of 7-10× less than english. recently the fineweb2 dataset was crafted for many languages with the same heuristic filters as the original fineweb supporting many languages with data for pretraining @xcite . still, there is only a handful of high quality datasets large enough for training language models in select languages like chinese @xcite , french, german, and spanish @xcite . data selection high quality data selection remains an important area of research in training language models. early research on data selection was based on heuristics including gpt-2 @xcite , @xcite , c4 @xcite , and refinedweb @xcite 2023). recent works examine model based filters for labeling general high quality data @xcite , or textbook quality data @xcite . while a majority of this work focuses on english data only, a few works have examined filtering in other languages, such as by perplexity @xcite or filtering using models trained on high quality specialized data in select languages @xcite . other forms of data selection include reweighting data @xcite @xcite and are shown to have varying degrees of success when applied in bilingual settings with data constraints, but only for english @xcite . ## data quality and multilinguality we conduct four types of experiments to demonstrate that the performance gap between a bilingual language model and monolingual models is largely due to the data quality and number of training steps -e.g., multilingual models require better training and for longer. we start with demonstrating that there are performance gaps when not controlling for data quality (section 3.2). we then show that training on a translated pretraining corpus in both languages, thereby controlling data quality, yields no gap between monolingual and bilingual performance (section 3.3). next, we find that at smaller number of training steps, there is a gap between multilingual and monolingual models, and models learn faster with higher quality data (section 3.4). for the experiments controlling data quality, we show results for english and french translated data here, and refer to section e for english and german experiments. finally, we show that quality also depends on the information available in the data, and that high quality english data with translations alone is insufficient for training high performing bilingual models (section 3.5). these experiments are done with chinese and english given the availability of high quality chinese data for trainng and downstream evaluations similar to mmlu. ## language-agnostic data filtering section 3 shows that bilingual models may perform as well as monolingual models when the data used to train the models has sufficient information for the target downstream tasks, and is of comparable quality. however, for a large set of languages, high quality data does not exist. prior works have shown that learnable models as quality filters lead to improved performance in downstream tasks for english @xcite @xcite . to learn a model-based filter for selecting high quality data, we assume access to a small set of high quality data d @xmath0, y)|@xmath1, @xmath2, 1}}, where x is some representation of a document, and y is a binary label indicating the quality of the sample. a binary classifier ϕ is trained on d h to estimate the probability that a document from the general pretraining data d g is high quality. the high quality pretraining set d hq is then selected according to the classification rule for some predefined threshold τ . unlike prior works which assume training the classifier and selecting general high quality data @xcite , or specialized data relevant to downstream tasks @xcite in english, we assume the high quality data sample is available only in english, but will be used to select data in other languages. concurrent to our work, @xcite show that a classifier approach in (grangier et al., 2024a) can be applied to languages other than english. they follow a similar setup using specialized datasets, such as translated mmlu @xcite , and include @xcite , for data selection. as such it is still unclear whether a universal language embedding with high quality seed data available only in english can be used to train a language agnostic filter, and whether monolingual filtering improves bilingual model training. we provide preliminary experiments indicating similar distributions of english and translated french data in appendix d. in our experiments, we parameterize ϕ as a logistic regression, and use a lightweight sentenceclassifier is trained on the embeddings of a small amount of high quality data available in english. the classifier is then used to filter data in all supported languages. bert (sbert) multilingual model foot_4 for extracting features @xcite . for d h , we use the same english data used for training the dclm classifier, which compares data from refinedweb (low quality), and openhermes 2.5 or eli5 (high quality). in our ablations, we also explore using the annotations for finewebedu which scores documents from 1 to 5 based on educational content. we do binary classification using a score above 2 as high quality. the value of τ is selected to ensure enough data for pretraining, and is in the order of 10% of the data following @xcite . in this work, we train a classifier as we have limited data within each cluster for training a 1.3b model at the desired scale, and would repeat data significantly if we instead do data selection for specialized data. ## conclusion training a multilingual language model that performs as well as monolingual models is important for building language models that can work for everyone, and facilitate compute efficiency in memory constrained settings where keeping many monolingual models may be infeasible. however it is also more challenging as it necessitates learning multiple distributions of data. this work provides a simple recipe for selecting high quality data, and demonstrates capability of selecting high quality data in other languages with only high quality english data. selecting high quality data with our recipe reduces gaps between monolingual and bilingual models to less than 1%, and improves monolingual performance. our work takes a step towards pretraining language models in languages with limited high quality data, and can help more research into closing the gap between multilingual and english-centric language models.