A Framework for Large-Scale Parallel Corpus Evaluation: Ensemble Quality Estimation Models Versus Human Assessment (2025)

Abstract: We developed a methodology and a framework for automatically evaluating and filtering large-scale parallel corpora for neural machine translation (NMT). We applied six modern Quality Estimation (QE) models to score 55 million English-Ukrainian sentence pairs and conducted human evaluation on a stratified sample of 9,755 pairs. Using the obtained data, we ran a thorough statistical analysis to assess the performance of selected QE models and build linear, quadratic and beta regression models on the ensemble to estimate human quality judgments from automatic metrics. Our best ensemble model explained approximately 60% of the variance in expert ratings. We also found a non-linear relationship between automatic metrics and human quality perception, indicating that automatic metrics can be used to predict the human score. Our findings will facilitate further research in parallel corpus filtering and quality estimation and ultimately contribute to higher-quality NMT systems. We are releasing our framework, the evaluated corpus with quality scores, and the human evaluation dataset to support further research in this area.

## introduction according to the scaling law @xcite , three basic ingredients are required to build a successful large language model: the model's size, the amount of compute spent on training, and the size of the dataset. in this paper, we will focus on the latter. indeed, the amount of text available is limited, and the limitation is even more visible for low-to-mid resource languages (see @xcite . one way to tackle that problem is to translate a decent amount of text using neural machine translation models, trading compute spent on inference to the data. recent advances in the nmt models, such as nllb @xcite and madlad @xcite , offer multilingual translation capabilities for hun-dreds of languages, building bridges to the lowresource languages. unfortunately, the measured quality of translation from english for these target languages is lower foot_0 than that for the popular pairs, such as english to german. this gap can be explained by the lack of training data (now for the nmt task) and the quality of the metrics. while metrics such as chrf @xcite and bleu @xcite are mechanistic and might not work well for fusional languages @xcite , others like comet @xcite or metricx @xcite might not have enough knowledge about low-resourced languages, again, because of the underrepresentation. if we look closer into the training of the nmt model, we might find the apparent abundance of sent2sent parallel corpora available online @xcite . for example, when we began our research, the english to ukrainian corpora had 97 million pairs, which now has around 158 million pairs foot_1 . however, a closer manual inspection reveals that at least part of the data is duplicated, garbled, or even obscene. most importantly, one cannot assess the quality of the whole corpus at the scale needed to build an nmt model. these issues might visibly affect the quality of the models trained on this data @xcite . as such, we identified the following research questions: to address these research questions, we created a methodology and a framework to collect parallel corpora at scale, deduplicate them, and score the individual sentence pairs using an ensemble of six quality estimation (qe) models that work in a multilingual setup. additionally, we ran a human annotation of the stratified random sample, scoring 9775 pairs with the help of students of the linguistics faculty who are proficient in english and ukrainian. using the obtained data, we ran a thorough statistical analysis to assess the performance of selected qe models and build linear, quadratic, and beta regression models on the ensemble to predict the human score. today, we are releasing the framework foot_2 , the evaluated and deduplicated dataset of 55 million sentence pairs foot_3 , and the data collected during the human evaluation. all the code, data, and instructions are published under permissive licenses to allow other scholars to reproduce the same workflow for other languages. ## related work the problem of filtering noisy parallel corpora has been addressed through several approaches: hybrid translation model-based filtering, machine learning classification, which frames filtering as a supervised task, multi-criteria heuristics combining statistical and neural techniques, and neural quality estimation models designed specifically for translation quality assessment. ## methodology to evaluate the quality of the english-ukrainian parallel corpus at scale, we are proposing a pipeline which consists of the following stages: ## results our analysis of the english-ukrainian parallel corpus provided some important findings regarding the relationship between automatic quality estimation and human evaluation. ## applications our research findings can be applied to create a similar evaluation and cleaning pipeline for other language pairs or on newly obtained data for the english-ukrainian language pair as the number of publicly available corpora and the volume of the data continues to grow. better filtration of the training data will result in better nmt models, thus bringing us closer to the ultimate task of seamless, high-quality text translation. the insights about the qe models performance might help others reduce the computational complexity of the task by selecting only the best-performing models. the existing methodology for human evaluation is now operationalized into a plugin for a crowdsourcing framework vulyk foot_8 , making it easy to run similar evaluations or create new direct assessment datasets for other languages. the human evaluation dataset can be used to calibrate the qe models further, fit new ensemble models, or assess the quality of other metrics not included in the current research. today, we are releasing our framework vakula 10 , which allows users to download, parse, deduplicate, and evaluate the parallel corpora from the opus open parallel corpora project. we are also releasing a combined and deduplicated corpus of english-ukrainian parallel sentences with all the scores from qe models and our ensemble mod-els 11 . finally, we are publishing the crowdsourcing plugin for human evaluation tasks, the annotator manual, and the raw data obtained from our experiment 12 . ## conclusion and future work in this paper, we developed a methodology for automatically evaluating and filtering large-scale parallel corpora for nmt. we applied six modern qe models to score 55 million english-ukrainian sentence pairs and conducted human evaluation on a stratified sample of over 9,775 pairs. here are some important findings: for future work, we plan to: