Multilingual Data Filtering using Synthetic Data from Large Language Models (2025)

Abstract: Filtering data, particularly data scraped from the internet, has long been recognised as a means to improve model performance. Recent studies have shown that effective filters can be created by utilising Large Language Models (LLMs) to synthetically label data, which is then used to train smaller neural models for filtering purposes. However, this approach has been tested mainly in English. Our paper extends this approach to languages beyond English, including languages not officially supported by the LLM. We validate our results on the downstream task of NMT and demonstrate that our approach is effective at both filtering parallel text for translation quality and filtering for domain specificity. For training the filtering model, we experiment with two different objectives for finetuning pre-trained transformers, as well as an efficient approach based on *n*-gram language models.

## introduction increasing model scale and larger pre-training datasets have fuelled recent advances in the world of llms. beyond scale, other characteristics of pre-training data also significantly impact downstream tasks, such as deduplication and removing low-quality examples @xcite ). an interesting approach that has recently been proposed is training filtering models on synthetic labels, which are generated by prompting llms @xcite @xcite . such filtering models can be efficiently run on very large corpora, such as pre-training data, to select the most appropriate examples for training. due to the flexibility of designing prompts, this pipeline is especially appealing, enabling data to be filtered on criteria beyond quality without requiring labelled data and thereby tailoring the selected pre-training data to the eventual downstream task. the fineweb project @xcite observed that by filtering pre-training data towards educational content, they were able not only to obtain a 4% improvement on the mmlu benchmark @xcite but also obtained faster convergence when compared to a non-filtered baseline. the educational content filter was a classifier trained on synthetic llm-labelled data, and the approach was validated via training a 1.71b parameter model on 350 billion tokens; however, the study focussed on english exclusively. although the experiment validates the methodology's effectiveness for english downstream tasks, the technique could also benefit other languages, where data quality is even more crucial given the overall scarcity of resources. this work attempts to unravel one unexplored axis of synthetic filtering: the method's efficacy beyond english. from here on, we refer to this approach as mdfs (multilingual data filtering using synthetic data). we investigate and evaluate mdfs via the neural machine translation (nmt) task. nmt is an excellent downstream task for several reasons. first, it has a history of data filtering, for example the wmt shared tasks (conference on machine translation, @xcite @xcite . secondly, nmt models are reasonably cheap to train compared to llms, allowing us to run a suite of experiments investigating different setups for filtering multilingual data using mdfs, which would be prohibitively expensive if done with llms. furthermore, nmt has a history of neural qe (quality estimation) metrics such as comet-kiwi or bleurt @xcite , which are effective at filtering training data @xcite . hence, we can employ such qe models trained on high-quality human annotations as a robust filtering baseline. we use mdfs as an instance of a synthetic llm-labelled quality estimator and validate the approach under general translation and domain adaptation setups. as we initially stated, the most significant appeal of mdfs is the flexibility to filter data based on any criteria simply by adjusting the prompt, so we assess its efficacy both in filtering for quality and filtering for domain. firstly, we train @xmath0. secondly, we train @xmath1. we summarise our contributions as follows: in nmt low amounts of noise in the training data can lead to erroneous translations @xcite . as such, nmt has a history of data filtering, especially for scraped corpora such as paracrawl @xcite . a series of cleaning tasks for parallel data @xcite @xcite resulted in the development of several cleaning models for nmt, including laser @xcite embedding-based models and bicleaner @xcite . later @xcite released an updated bi-cleaner incorporating a neural model. bi-cleaner is used to filter public corpora such as paracrawl. compared to our work, these models all focus on removing training examples that are not mutual translations of each other, rather than picking the best translations and can only filter for quality. @xcite compare filtering training data using bicleaner @xcite to filtering using comet-kiwi, a qe model for nmt. the authors filter 50% the wmt 23 @xcite training data for three language pairs and show that filtering with comet-kiwi leads to improved comet scores. they highlight that filtering with qe metrics discriminates in a more fine-grained manner. in contrast, our methodology extends beyond quality assessment, enabling filtering based on different criteria, and is equally applicable to monolingual corpora. furthermore, we conducted experiments implementing filtering at varying threshold levels. ## machine translation as a downstream task in all nmt experiments, we translate from english. we train encoder-decoder standard transformer models with ∼63m parameters. all models are trained for 100,000 updates using fairseq @xcite , selecting the best model using bleu on a held out validation set (for full training hyperparameters see appendix f). for the translation quality experiments, we evaluate on the flores-200 (nllb team, 2022; goyal et al., 2022) test set comprising 1,007 sentences. the @xmath2; we use the first 1,000 sentences as the validation set and the remaining 2,701 as the test set. finally, for the @xmath3, we use the himl 6 (health in my language) and wmt18 @xcite ) biomedical test sets. specifically, we combine the 467 cochrane sentences of himl with the 278 wmt18 biomedical sentences as the test data. we train the nmt models using the nmt training data as outlined in section 4.1. for the translation quality experiments, we filter to thresholds of 1%, 10%, 25% and 50% of the original nmt training dataset size. meanwhile, we use a threshold of 1, 2.5, 5, and 10 million sentences for the medical domain experiments. we generate all results using beam search with a beam size of 5. we report chrf++ @xcite , a lexical metric as neural metrics are less sensitive to wrongly named entities, insertions and deletions (amrhein and sen-6 https://www.himl.eu/test-sets nrich, 2022; @xcite . as medical content often focuses on a few technical terms surrounded by more general language, we believe a lexical metric is more appropriate. ## conclusion we trained classification, regression and n-gram based data filtering models from labels generated by llama-3.1 to filter nmt data based on translation quality and medical relevance. we find that all mdfs approaches effectively utilise the data, including the n-gram based approaches. labelling data with llms is computationally expensive, and as mdfs-ngram models are known to work well with small amounts of training data and are cheap to train we suggest that these prove an effective alternative to finetuning pre-trained encoder only models, especially when only filtering for one criteria. continuous ranking of sentences is not effective at selecting the very best sentences, highlighting the inability of the mdfs models to correctly distinguish between "good" and "excellent" sentences. our findings show that the mdfs filtering pipeline extends beyond english languages. for our medical domain experiments, we report comparable nmt results when filtering english or non-english data. furthermore, these findings support that llms can effectively generate labels for languages they do not officially support, even when compared to a model like comet-kiwi, which was trained using manually annotated data. horizon europe funding guarantee 10052546 and 10039436, and by the national science centre, poland 2023/49/n/st6/02691. for the purpose of open access, the authors have applied a creative commons attribution (cc-by) public copyright licence to any author accepted manuscript version arising.