H i MATE : A Hierarchical Multi-Agent Framework for Machine Translation Evaluation (2025)

Abstract: The advancement of Large Language Models (LLMs) enables flexible and interpretable automatic evaluations. In the field of machine translation evaluation, utilizing LLMs with translation error annotations based on Multidimensional Quality Metrics (MQM) yields more human-aligned judgments. However, current LLM-based evaluation methods still face challenges in accurately identifying error spans and assessing their severity. In this paper, we propose HiMATE, a Hierarchical Multi-Agent Framework for Machine Translation Evaluation. We argue that existing approaches inadequately exploit the fine-grained structural and semantic information within the MQM hierarchy. To address this, we develop a hierarchical multi-agent system grounded in the MQM error typology, enabling granular evaluation of subtype errors. Two key strategies are incorporated to further mitigate systemic hallucinations within the framework: the utilization of the model’s self-reflective capability and the facilitation of agent discussion involving asymmetric information. Empirically, HiMATE outperforms competitive baselines across different datasets in conducting human-aligned evaluations. Further analyses underscore its significant advantage in error span detection and severity assessment, achieving an average F1-score improvement of 89% over the best-performing baseline. We make our code and data publicly available at https://github.com/nlp2ct-shijie/HiMATE.

## introduction translation capability reflects the cross-lingual comprehension and generation quality of large language models (llms) @xcite @xcite compared intro v3: add m-mad xcomet labeled data multilingual plm target lm -sentence-level score -error span & severity final score feed finetune infer himate mqm hierarchy (category & description) collaborative discussion self-reflection subtype evaluation final score build infer multi-agent system … mqm hierarchy (category & description) m-mad build final score infer multi-agent system … dimension-aware debate & judge gemba-mqm build act mqm hierarchy (category only) single-agent system final score infer all-in-one prompt to traditional machine translation models, applying llms significantly enhances the translation capabilities of machine translation systems and introduces new challenges to machine translation evaluation (mte) @xcite . previously, statistics-based mte metrics such as bleu @xcite and me-teor @xcite are utilized for their efficiency, despite inherent limitations in accurately capturing semantic similarity. to address this gap, learned metrics such as comet-20 @xcite and robleurt @xcite have been developed to enhance evaluations. these neural metrics can further improve performance via test-time adaptation by uncertainty minimization @xcite . in addition, there are methods to extend the evaluation dimensions to translation difficulty @xcite and the fuzziness of reference @xcite . however, these approaches primarily yield numeric scores and cannot identify the specific error spans in translations @xcite . the machine translation data, annotated based on the guidelines of multidimensional quality metrics (mqm), foot_0 includes detailed information regarding error spans, categories, and severity for each translated sentence, which facilitates more precise finetuning and thorough analysis of mte models. the xcomet model @xcite is finetuned on such data for interpretable scores, while it necessitates resource-intensive multi-stage training and shows a lack of generalization ability. the llm-based single-agent evaluation systems, including eaprompt @xcite and gemba-mqm @xcite , utilize mqm-informed prompts to guide llms in error detection. nevertheless, the hallucination and position bias inherent within llm evaluators affect their performance @xcite . in response to these challenges, the recently proposed method m-mad @xcite employs a multi-agent system to decompose complex mte tasks into simpler assessments, showcasing the potential of this paradigm. however, its effectiveness primarily stems from the self-consistency of agents operating within identical contexts, which may limit nuanced, multi-level error analysis. we argue that the rich semantic and hierarchical structure of the mqm framework has been largely underutilized in current methodological designs. by developing a more fine-grained, hierarchical agent-based system explicitly tailored to leverage the untapped potential of mqm's structured hierarchy, error detection accuracy could be significantly enhanced, thereby narrowing the gap with human expert performance. in this paper, we propose himate, a hierarchical multi-agent framework for machine translation evaluation. specifically, we establish the agent topology based on the mqm error typology, facilitating hierarchical agent nodes to exchange information in alignment with human error classification methodology. error descriptions associated with each node are then employed to initialize multi-agent evaluators, allowing them to focus on detailed, simplified discrimination tasks rather than evaluations against broad, comprehensive criteria. this design facilitates fine-grained subtype error evaluations through multi-agent collaboration. furthermore, we propose two post-stages to enhance the accuracy of the system in error detection and severity assessment. concretely, the subtype evaluator first revises the original translations based on its error detection results, then re-evaluates the validity of the initial judgments through comparison. cases with low confidence during this self-reflection phase are forwarded to the next stage, where agents from different tiers collaboratively determine the presence and severity of translation errors through iterative discussion. in contrast to prior methods, the proposed method leverages mqm information more effectively for decomposing complex tasks and constructing hierarchical multi-agent systems. by guiding these agents through self-reflection and collaboration, we achieve more human-aligned automatic evaluations. figure 1 highlights the key differences between hi-mate and other representative mte metrics. we summarize our contributions as follows: ## methodology in this section, we introduce our proposed framework himate for conducting human-aligned machine translation evaluations. adhering to the hierarchical structure of the mqm error typology, himate organizes translation errors into two tiers: high-level error categories (tier-1) and fine-grained error subtypes (tier-2). two distinct types of agents are assigned to manage these tiers, with each agent specializing in assessing specific error categories, as depicted in figure 2 . initially, tier-2 agents individually perform evaluations on their corresponding error subtypes. this preliminary assessment is then refined by harnessing the self-reflection capabilities of llms. finally, results exhibiting low confidence are subjected to collaborative discussion involving tier-1 and tier-2 agents. the entire evaluation process operates without reliance on reference, ensuring broad applicability for evaluation. ## ablation and analysis 5.1 ablation study to verify the contribution of each stage in himate, we conduct an ablation study in this section. in addition to reporting the performance of individual stages, we also include the results of a single-agent evaluation method (denoted as sa) as a baseline ablation. unlike the subtype evaluation stage of himate, where each agent is dedicated to a specific error type, the sa baseline processes all tier-2 error definitions simultaneously and evaluates all types of errors at once. this baseline operates as a single stage and is capable of detecting multiple errors within a sentence. evaluation results of the ablated framework on the mqm22 dataset are shown in the performance improvements from the selfreflection stage are relatively modest; however, this stage helps reduce the computational cost of the subsequent stage by filtering the high-confidence, reliable judgments. the collaborative discussion stage contributes more substantially, achieving the best or second-best results across all four evaluation metrics. it is worth noting that the performance improvements from collaborative discussion vary across different backbone llms, likely influenced by the instruction-following and divergent thinking capability of different models. ## conclusion this paper introduces himate, a hierarchical multi-agent framework for llm-based machine translation evaluation. the framework assembles a three-stage strategy to enhance error detection and severity assessment, leveraging self-reflection and collaborative discussion mechanisms after subtype evaluation. the experimental results of segmentlevel evaluation and system-level meta-evaluation demonstrate the superior performance of himate. ablation study further validates the contribution of each stage in aligning closely with human judgments. detailed analyses highlight that himate exhibits pronounced advantages in error span detection compared to existing llm-based methods. additionally, evaluations across varying sentence lengths indicate robust and consistent performance. himate also achieves stable advancements across various text domains. the achievement of himate emphasizes the significance of effectively utilizing the human evaluation framework in designing multi-agent collaborative evaluation strategies.