Fine-Grained Evaluation of E nglish- R ussian MT in 2025: Linguistic Challenges Mirroring Human Translator Training (2025)

Abstract: We analyze how English–Russian machine translation (MT) systems submitted to WMT25 perform on linguistically challenging translation tasks, similar to problems used in university professional translator training. We assessed the ten top-performing systems using a fine-grained test suite containing 465 manually devised test items, which cover 55 lexical, grammatical, and discourse phenomena, in 13 categories. By applying pass/fail rules with human adjudication and micro/macro aggregates, we observe three performance tiers. Compared with the official WMT25 ranking, our ranking broadly aligns but reveals notable shifts.Our findings show that in 2025, even top-performing MT systems still struggle with translation problems that require deep understanding and rephrasing, much like human novices do. The best systems exhibit creativity and can be very good at handling such challenges, often producing more natural translations rather than producing word-for-word renditions. However, persistent structural and lexical problems remain: literal word order carry-overs, misused verb forms, and rigid phrase translations were common, mirroring errors typically seen in beginner translator assignments.

## introduction unlike standard test sets, which consist of randomly selected source material, a test suite contains 'extra-credit' problems: the source items, deliberately designed to be challenging in translation, similar to the 'rich points' method @xcite or preselected items method @xcite in translator training. these focused items are often lexical units or grammatical structures requiring non-literal approaches based on deep situational understanding and coherent target-language encoding. successful translations are expected to abstract away from source form, displaying both familiarity with standard translation techniques and creativity in adapting them to individual contexts. while isomorphic translations may be formally grammatical, they are typically judged suboptimal. in translation didactics, such tasks test translators' ability to identify and resolve translation problems. the fine-grained linguistic test suite has been partially in development over the last few years @xcite @xcite . 1 its original purpose was to track mt systems' ability to handle specific source language phenomena, with evaluation strategies focused on translating those targeted items. this approach provides valuable, fine-grained insights into mt systems, highlighting system strengths across a wide range of phenomena and establishing objective grounds for comparison. however, the recent dominance of large language models (llms) in mt has prompted us to reconceptualize the test suite's role. beyond its original function of testing specific linguistic categories, we also view it as a diagnostic tool for identifying overarching translation challenges that persist across current english-russian mt systems. this evolution reflects the need to understand systemic issues in mt. accordingly, we have tightened our quality requirements to match those of a university professional translation training. our evaluation focus has shifted from assessing the handling of specific linguistic categories to evaluating overall translation quality when processing these 'extracredit' problems. we no longer make concessions for 'gist translation quality,' instead expecting the publication-standard quality of human translation as defined by @xcite . this refined approach maintains our ability to compare systems based on the ratio of successfully handled items across the entire test suite, while simultaneously revealing persistent issues and blind spots that transcend formal categories. we supplement our qualitative analysis with statistical results and system rankings, comparing these findings with wmt's official human evaluation. ## discussion: lessons learned the test suite was revised to exclude items with questionable categorization or insufficient contextindependence for translation in isolation. the rules have become less permissive in terms of fluency and accuracy. there are some similarities between the translation patterns produced by nmt and by learner translators (see the detailed analysis in @xcite . these similarities are most visible in source-language-triggered issues, such as the placement of adverbials and prepositional objects, or the preference for analytical (instead of synthetic) forms of future and passive verbs. this points to the phenomenon of shining through (as defined by @xcite , which belongs to the phenomena of translationese @xcite , i.e., specific features of translated language that make it different from non-translated original language production. however, compared to human translators, nmt systems (especially those outside the top tier) more often generate sentences that obscure the message, overcomplicate structure, or introduce redundancy. such output departs from target-language conventions in ways that sound recognizably nonhuman, often failing to produce a coherent text that conveys a clear, plausible situation. we observed that the same systems might pursue different strategies depending on the conditions. faced with the absence of isomorphic structures in the target language, they are capable of impressively creative solutions. in less demanding contexts, however, they tend to revert to routine, nearliteral strategies that overlook the target language's potential for more optimal expression. ## conclusion in 2025, state-of-the-art english-russian mt presented by the latest llm-based systems shows substantial progress in performance, yet it continues to display important weaknesses. our fine-grained evaluation revealed that even top-performing models still falter on translations requiring deep comprehension and nuanced rephrasing, much like human novice translators. at the same time, the best systems exhibit marked improvements on such 'extracredit' items by re-structuring and wording translations in a more natural russian style instead of relying on word-for-word renditions. this shift toward non-literal, context-aware translation indicates that current mt can approximate some of the flexible strategies employed by skilled human translators. notably, general-purpose llms (e.g., gpt-4.1, claude-4) only attained mid-tier accuracy on our test suite, underscoring that even massive generalist models have not fully solved certain linguistic subtleties specialized mt engines thus continue to hold an edge on fine-grained challenges, though the gap is beginning to narrow as new models adopt more human-like problem-solving approaches.