Tagged Span Annotation for Detecting Translation Errors in Reasoning LLM s (2025)

Abstract: We present the AIP team’s submission to the WMT 2025 Unified MT Evaluation SharedTask, focusing on the span-level error detection subtask. Our system emphasizes response format design to better harness the capabilities of OpenAI’s o3, the state-of-the-art reasoning LLM. To this end, we introduce Tagged SpanAnnotation (TSA), an annotation scheme designed to more accurately extract span-level information from the LLM. On our refined version of WMT24 ESA dataset, our reference-free method achieves an F1 score of approximately 27 for character-level label prediction, outperforming the reference-based XCOMET-XXL at approximately 17.

## introduction with the recent widespread use of the llm-asa-judge approach, research on human-like translation quality evaluation using large language models (llms)-such as gemba-da @xcite , mqm @xcite , esa @xcite , eaprompt @xcite , and automqm @xcite -has grown rapidly. fine-grained error detection enables explainable translation evaluation and informs the design of post-editing systems, making it widely applicable across diverse systems that leverage machine translation. nevertheless, research on fine-grained translation error detection remains scarce; only a handful of studies exist despite growing interest. moreover, the llm-based studies mentioned above focused less on fine-grained error detection itself and more on using it to compute ## limitations this study has the following limitations. first, the experiments were conducted primarily on proprietary large language models (llms) such as gpt-4.1, o3 and o4-mini, without providing comparable results for large-scale open-source models. this limits the generalizability of the proposed approach to other model families. second, although per-language results were reported, we did not conduct in-depth analyses of performance variations or error patterns across specific language pairs. in particular, the causes of performance differences for low-resource languages remain underexplored. third, the monetary cost of using proprietary llms can hinder large-scale adoption. using tiktoken (openai, 2023b) to count tokens, our evaluation on six wmt24 esa language pairs (39,684 segments, 56.82m input tokens, 3.26m output tokens) is estimated to have cost about @xmath03.5 per 1,000 segments) under the current o3 pricing (@xmath18/m output as of september 2025), which may hinder large-scale adoption despite the accuracy gains. ## conclusion and future work we introduced tagged span annotation (tsa), a structured-output framework that pairs numbered inline tags with a json schema to enable reliable, fine-grained translation-error detection. across six wmt language pairs, tsa achieved the highest span-level f1 scores, surpassing the strongest gemba variant by +10.55 and xcomet-xxl by +9.65 absolute points on average. reasoning vs. non-reasoning models. we observed performance differences between reasoning and non-reasoning llms. as future work, we will systematically study how prompting style and intermediate reasoning affect span detection by comparing: (i) direct answers vs. chain-of-thought (cot; @xcite ) prompting, (ii) single-pass decoding vs. self-consistency with multiple cot paths, and (iii) unguided cot vs. cot path selection aided by a span-level verifier. we will evaluate not only span f1 (overall and by severity/type), but also invalid-output rate, span-length bias, latency, and cost to characterize accuracy-efficiency trade-offs.