TASER : Translation Assessment via Systematic Evaluation and Reasoning (2025)

Abstract: We introduce TASER (Translation Assessment via Systematic Evaluation and Reasoning), a metric that uses Large Reasoning Models (LRMs) for automated translation quality assessment. TASER harnesses the explicit reasoning capabilities of LRMs to conduct systematic, step-by-step evaluation of translation quality. We evaluate TASER on the WMT24 Metrics Shared Task across both reference-based and reference-free scenarios, demonstrating state-of-the-art performance. In system-level evaluation, TASER achieves the highest soft pairwise accuracy in both reference-based and reference-free settings, outperforming all existing metrics. At the segment level, TASER maintains competitive performance with our reference-free variant ranking as the top-performing metric among all reference-free approaches. Our experiments reveal that structured prompting templates yield superior results with LRMs compared to the open-ended approaches that proved optimal for traditional LLMs. We evaluate o3, a large reasoning model from OpenAI, with varying reasoning efforts, providing insights into the relationship between reasoning depth and evaluation quality. The explicit reasoning process in LRMs offers interpretability and visibility, addressing a key limitation of existing automated metrics. Our results demonstrate that Large Reasoning Models show a measurable advancement in translation quality assessment, combining improved accuracy with transparent evaluation across diverse language pairs.

## introduction large language models (llms) have been demonstrated in zero-shot and few-shot translation scenarios, achieving comparable results to dedicated machine translation systems @xcite . previous work by @xcite used large language models (llms) through prompting to assess the quality * work done while interning at apple inc. large reasoning models @xcite @xcite use long chained reasoning to answer input queries. reasoning models have shown abilities in problemsolving, coding, as well as scientific reasoning and multi-step logical inference @xcite . recent findings show that large reasoning models can also be used in translation. @xcite investigated lrms at machine translation tasks. in their position paper, they identified three shifts brought about by lrms: 1) contextual coherence, where lrms resolve ambiguities and preserve discourse structure through explicit reasoning via context clues; 2) cultural intentionality, where models can adapt translations by inferring speaker intent, audience expectations, and socio-linguistic norms, and finally 3) self-reflection, where lrms can iteratively refine translations during inference, correcting errors dynamically. these three shifts contribute to more nuanced translations. in this paper, we present taser. taser uses lrms with zero-shot prompting to arrive at a translation quality estimation. we define and investigate lrms for the assessment of translation quality in both reference based and reference free scenarios. starting with the evaluation of the prompts from earlier works that showed state-of-the-art result on non-reasoning llms, we iterated on the da+sqm template used for the human assessment of the translation quality as implemented in the appraise framework @xcite for wmt22 @xcite and adapted it towards lrms. we posit that the strengths of lrms lead to translation quality estimation that is more aligned with human judgment, as measured in tables 1 and 2 below. the main contributions of this paper are as follows: ## taser metric in this method, we prompt reasoning models from openai with the following attributes: source language, target language, source text segment, translation segment, and optionally, the human reference segment, analogous to @xcite . after iterating and evaluating on different prompts, we observed that simple zero-shot open ended prompting does not result in the best overall assessment. the prompt that we settled on includes the attributes as listed above as well as includes more direction, particularly assessment instructions and details of what to look for during quality assessment. we leave evaluating other reasoning models and additional language pairs for future work. ## experimental setup our experiments involve measuring the performance of taser on the wmt24 metrics shared task @xcite , where automated metrics are evaluated against human gold labels. the goal is to predict a quality score for each segment in a given test set which can be a variant of direct assessment (da) or multidimensional quality metrics (mqm). we evaluate taser across the evaluation set provided by wmt24. similar to ( kocmi and federmann, 2023a ), we compare our method against the best-performing reference-based and reference free metrics of wmt24. ## conclusion in this paper, we introduced taser: translation assessment via systematic evaluation and reasoning, a novel approach that uses large reasoning models (lrms) for automated translation quality assessment. our work demonstrates that lrms can measurably outperform traditional large language models (llms) and existing automated metrics in evaluating translation quality. taser achieves state-of-the-art performance on the wmt24 metrics shared task when evaluated against the mqm24 dataset. taser's performance demonstrates that the explicit reasoning capabilities of lrms provide tangible benefits for translation assessment tasks. in the near future, we plan to focus on exploring the interpretability advantages offered by the taser reasoning process and how they might address the limitations of existing automated metrics. in addition, we plan to investigate taser under open-source reasoning models. in conclusion, our results suggest that the integration of explicit reasoning processes into evaluation metrics will play a crucial role in advancing the field of machine translation evaluation, ultimately contributing to more reliable and trustworthy automated translation systems across diverse languages and applications.