Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems (2025)

Abstract: As machine translation systems approach human-level quality, traditional evaluation methodologies struggle to detect subtle translation errors. We critically examine limitations in current gold-standard approaches (MQM and ESA), including inconsistencies from variable annotator expertise, excessive categorization complexity, coarse severity granularity, accuracy bias over fluency, and time constraints. To address this issue, we introduce a high-quality dataset consisting of human evaluations for English–Russian translations from WMT24, created by professional linguists. We show that expert assessments without time pressure yield substantially different results from standard evaluations. To enable consistent and rich annotation by these experts, we developed the RATE (Refined Assessment for Translation Evaluation) protocol. RATE provides a streamlined error taxonomy, expanded severity ratings, and multidimensional scoring balancing accuracy and fluency, facilitating deeper analysis of MT outputs. Our analysis, powered by this expert dataset, reveals that state-of-the-art MT systems may have surpassed human translations in accuracy while still lagging in fluency – a critical distinction obscured by existing accuracy-biased metrics. Our findings highlight that advancing MT evaluation requires not only better protocols but crucially, high-quality annotations from skilled linguists.

## introduction recent advances in natural language generation have made evaluation increasingly challenging, as modern systems often produce outputs that match or exceed human-written references @xcite . for generative ai applications, human feedback has become the cornerstone of progress, with techniques like reinforcement learning from human feedback driving signifi1 the dataset is available on hugging face. cant improvements @xcite . however, the process of collecting human judgments faces numerous challenges: annotators seek shortcuts to simplify evaluation tasks @xcite , often prioritizing surface-level properties over aspects requiring deeper analysis. despite these known issues, the field continues to rely heavily on human evaluation, particularly for openended generation tasks, where annotators typically provide single overall scores or preference rankings with limited transparency into their decisionmaking processes @xcite . machine translation (mt) represents a particularly well-established field where human evaluation plays a critical role. over decades, the community has developed sophisticated frameworks for translation quality assessment, with the multidimensional quality metrics (mqm) @xcite framework and its simplified derivative error span annotation (esa) @xcite emerging as dominant paradigms. these evaluation standards are reinforced through annual conference on machine translation (wmt) campaigns, which produce benchmark datasets that influence the majority of contemporary research on translation quality assessment. indeed, the field's progress is largely measured through improvements against these benchmarks. despite this established infrastructure, translation quality assessment remains fundamentally challenging. it demands high levels of cognitive effort, linguistic expertise in multiple languages, domain knowledge, and substantial time investment, all of which conflicts with the practical constraints of large-scale evaluation campaigns. our analysis reveals concerning limitations in current gold standard translation evaluation protocols, including excessive complexity of error categorization in comprehensive approaches like mqm, an oversimplified binary (major/minor) error severity classification, significant bias toward accuracy at the expense of fluency, and alarmingly brief annotation times (often under one minute per segment). the contributions of this paper include: (1) a critical analysis of current mt evaluation practices, with empirical evidence of their limitations; ## rate: refined assessment for translation evaluation this section of our study presents an attempt to design a more effective evaluation system that provides informative quality reports while adhering to five core principles: comprehensiveness, consistency, comparability with established metrics, compactness, and convenience for both human annotators and automated analysis. the proposed rate workflow consists of three main stages: error categorization, severity determination, and quality scaling. error categorization our rate protocol preserves error categorization as a critical component for providing actionable translation feedback while significantly streamlining the taxonomy. rather than eliminating categories entirely (as in esa) or maintaining the extensive hierarchy of mqm, we adopt a balanced approach that distinguishes between semantic errors (affecting meaning: mistranslation, undertranslation, overtranslation, omission) and non-semantic errors (affecting form: grammar, fluency, style, inconsistency, named entities, do-not-translate). this simplified classification enables annotators to provide specific, diagnostic information about translation weaknesses without the cognitive burden and redundancy of mqm's extensive category system. for instance, we incorporate terminology errors within mistranslation, as they fundamentally represent meaning distortions, and we reserve an "other" category for exceptional cases. detailed category definitions appear in appendix b. severity assessment rate expands the traditional binary (major/minor) severity classification to a 5-point scale. the scale ranges from 5 (critical errors with severe impact on meaning or readability) down to 1 (minimal issues that even native speakers might overlook). this expanded scale provides the necessary granularity to differentiate between slight inaccuracies and consequential mistranslations. the middle point (3) captures moderate errors that fall between the traditional major and minor categories. ## related work over the past decades, human evaluation of machine translation (mt) has evolved significantly. early assessments relied on holistic scoring @xcite , gradually giving way to more structured approaches. the conference on machine translation (wmt), established in 2006, has become the primary venue for standardizing evaluation methodologies @xcite , shifting from ranking-based assessments to direct assessment (da) @xcite , where annotators assign continuous quality scores. the multidimensional quality metrics (mqm) framework @xcite introduced a comprehensive error typology that classifies issues by type and severity. esa, a simplified version of mqm, focuses on identifying and categorizing spans of text contain-ing errors, classifying them primarily as major or minor, with less emphasis on detailed error typology @xcite . though wmt has been instrumental in collecting human judgments at scale, such evaluations remain expensive and time-consuming. errormarking protocols like mqm, despite being laborintensive, provide actionable insights crucial for system improvement that holistic scoring cannot offer. these practical constraints have motivated automated metrics like comet @xcite , bleurt @xcite , and metricx @xcite , with recent work exploring large language models for evaluation @xcite @xcite . error spans identified through llm annotation @xcite have been incorporated into ai-assisted human evaluation frameworks, effectively reducing costs while maintaining evaluation quality. however, all these automated approaches ultimately depend on human-curated gold standards for validation. currently, high-quality mqm annotations exist at scale primarily for english-german and chinese-english pairs only, creating a bottleneck for progress as top mt systems converge in quality, making fine-grained distinctions increasingly important yet difficult to assess. ## conclusion our study introduces the rate framework for translation evaluation and contributes a highquality annotation dataset for english-to-russian translations. we demonstrate that existing gold standard methods show significant bias toward ac-curacy at the expense of fluency, while rate offers a more balanced single-score metric that weighs both dimensions, complemented by granular statistics for comprehensive system analysis. our findings reveal that annotator qualifications and time investment dramatically impact evaluation outcomes, potentially more than the protocol itself. we also discovered that state-of-the-art mt systems have surpassed human translations in accuracy while still lagging in fluency, a nuance obscured by one-dimensional evaluation approaches. the rate balanced methodology is an important advancement in translation quality assessment, particularly as mt systems continue to improve and require more sophisticated evaluation methods.