Estimating Machine Translation Difficulty (2025)

Abstract: Machine translation quality has steadily improved over the years, achieving near-perfect translations in recent benchmarks.These high-quality outputs make it difficult to distinguish between state-of-the-art models and to identify areas for future improvement.In this context, automatically identifying texts where machine translation systems struggle holds promise for developing more discriminative evaluations and guiding future research.In this work, we address this gap by formalizing the task of translation difficulty estimation, defining a text’s difficulty based on the expected quality of its translations.We introduce a new metric to evaluate difficulty estimators and use it to assess both baselines and novel approaches.Finally, we demonstrate the practical utility of difficulty estimators by using them to construct more challenging benchmarks for machine translation. Our results show that dedicated models outperform both heuristic-based methods and LLM-as-a-judge approaches, with sentinel-src achieving the best performance.Thus, we release two improved models for difficulty estimation, sentinel-src-24 and sentinel-src-25, which can be used to scan large collections of texts and select those most likely to challenge contemporary machine translation systems.

## introduction not all data samples are equal: machine learning models may struggle with some samples more than others. the ability to automatically assess sample difficulty is indispensable at various stages of model development. for example, during training, organizing samples from the easiest to the hardest, known as curriculum learning, improves both performance and training efficiency @xcite @xcite . ## the difficulty estimation task the difficulty of translating a given text can depend on multiple factors. a text may be challenging, for example, due to its length, syntactic complexity, idiomatic language, or the presence of rare or specialized vocabulary. some aspects that affect translation difficulty may even depend on the translation direction, meaning that the same source text might be more difficult to translate into one language than into another. moreover, translation difficulty might not be uniform across translators, as it can vary with the translator's cultural background and linguistic familiarity -in the case of human translators -or based on factors such as the number of parameters, training data, and model architecture -in the case of machine translation models. given these considerations, we avoid defining translation difficulty in absolute terms, as such a definition may not generalize well. instead, we define difficulty relative to a given target language and to the accuracy of a particular translator, whether human or automatic. more specifically, given a text x, a model 2 m, and a target language l, we assign to x a difficulty score d m,l (x) equal to the quality score assigned to a translation of x into language l produced by m. lower scores indicate a lower translation quality and, therefore, greater difficulty associated with the source text. as an example, suppose we have two texts, x 1 and x 2 , and their respective translations t 1 and t 2 into language l, both produced by model m. a human rater evaluates these translations on a scale from 1 to 100, assigning a score of 60 to the first and 90 to the second. then, d m,l (x 1 ) = 60 and d m,l (x 2 ) = 90. since d m,l (x 1 ) < d m,l (x 2 ), then x 1 is more difficult to translate into l than x 2 , for model m. importantly, the lower the score d, the higher the difficulty and vice versa. task definition. given a source text x, a model m, and a target language l, difficulty estimation is the task of automatically predicting d m,l (x). different from quality estimation, difficulty estimation models do not have access to the translations whose quality is being estimated. indeed, difficulty estimation can be seen as the task of estimating the expected quality of a given text's translation. evaluation. we evaluate difficulty estimation methods according to their ability to rank texts based on difficulty. consider a collection of texts @xmath0, x 2 , . . . , x n , a collection of target languages @xmath1, l 2 , . . . , l l , and a collection of models translating into language l: m @xmath2, m 2 , . . . , m m l . let us also define the vector of ground-truth difficulty scores for model m and language l as d m,@xmath3,l (x 1 ), d m,l (x 2 ), . . . , d m,l (x n ) and the corresponding predictions of a difficulty estimation method as dm,@xmath4,l (x 1 ), dm,l (x 2 ), . . . , dm,l (x n ). we measure the translation difficulty estimation correlation (dec) by averaging the kendall's rank correlation coefficients τ b across models and languages: contrasting dec with standard mt metaevaluation strategies. the evaluation approach used in formula 1, termed "group-by-system" by @xcite , makes dec fundamentally different from other meta-evaluation strategies used in mt evaluation and quality estimation, which typically rely on the group-by-item method instead @xcite . group-by-item calculates the correlation between assessments assigned to different translations of the same source text (i.e., the same evaluation item), and then averages these correlations across all source texts. the primary benefit of this method is that it mitigates spurious correlations between source text features (e.g., length) and translation quality judgments @xcite . however, we argue that these features are precisely what define a text's translation difficulty. since our goal is to measure translation difficulty, we define dec using group-by-system. this method computes the correlation between human and metric assessments for translations of different source texts that were produced by the same mt system, and then averages the correlations across mt systems. by holding the mt system constant, this evaluation directly measures a metric's ability to identify which source texts were more challenging for that system to translate. this distinction is crucial. indeed, using groupby-system to meta-evaluate standard mt or quality estimation metrics would favor those that predict source difficulty rather than purely translation quality. conversely, using group-by-item to evaluate difficulty estimators would be inappropriate, as these estimators assign the same score to all translations of a given source. ## methods for difficulty estimation in this section, we describe several difficulty estimation methods. we include both common and novel approaches. specifically, we categorize difficulty estimators into four groups: heuristic-based, learned, llm-as-a-judge, and artificial crowdbased. we refer the reader to appendix c for implementation details regarding all the considered models. ## experiments we benchmark the estimators using the difficulty estimation correlation measure (dec, formula 1). ## creating difficult benchmarks in this section, we use the top-performing difficulty estimators to create difficult machine translation benchmarks. first, we show that the test set employed at the wmt 2024 general mt shared task @xcite models. then, we define the task of selecting a difficult subset of samples from a given dataset, and evaluate our estimators. ## (difficult): alex bregman predicted to betray astros, sign with shocking blue jays 3 (difficult): some folks really do deserve a badge of honour for their pedantry (c8). veronica coyne of springfield claims that "when bemoaning the loss of the express lane at woolies "12 items or less," a friend told me she'd never used it on principle as it should have been "12 items or fewer."" 4 (easy): washington ## conclusion in this work, we formally define the task of translation difficulty estimation and introduce the difficulty estimation correlation (dec), a dedicated measure for evaluating the performance of difficulty estimators. we conduct a comprehensive evaluation of existing and newly proposed estimators, finding that models explicitly trained for the task significantly outperform traditional, heuristicbased methods and llm-as-a-judge approaches. our analysis identifies sentinel-src-24 as the current state-of-the-art in translation difficulty estimation. we further validate the performance of difficulty estimators in the downstream task of creating difficult benchmarks, demonstrating that they successfully identify samples where modern mt models underperform. in this downstream task too, sentinel-src-24 remains the top-performing method. building on these findings, we develop sentinel-src-25 by incorporating additional data into the training pipeline of sentinel-src-24, and release both models publicly. finally, we conduct a qualitative analysis of sentinel-src-25's predictions, offering intuitive insights into the types of texts it deems difficult.