An Evaluation Resource for Grounding Translation Errors (2025)

Abstract: Current fine-grained error analyses by LLMs gain more and more attention in machine translation, but these analyses do not ground the errors to the reasons why the annotated text spans are erroneous. If LLMs do not know such reasons, the corrections or refinements by LLMs will be untrustworthy.In this paper, we check whether LLMs know such reasons in translation error grounding task. We manually build an evaluation resource through a bi-directional grounding scheme. In the forward direction, we annotate the explanation of the reason for each error span. In the backward direction, we annotate the error span given its explanation, in which the error span is masked. If the error spans of both directions are consistent, we deem the explanation is valid. Such grounding process can regulate the explanation so as to avoid the subjective bias. The evaluation results on this resource show that LLMs perform significantly worse than human in both directions. Furthermore, we apply the error grounding for filtering false alarmed errors, and achieve significant improvement in translation error detection.

## introduction with the recent development of neural networks and large language models (llms), machine translation (mt) systems achieve steady progress in translation quality. although they perform well in certain circumstances, there still exist various type of errors that need further study for building trustworthy translator to generate error-free contents. multidimensional quality metrics (mqm) @xcite is the fine-grained schema for translation error analysis. it contains manual annotation of error spans and has been successfully * corresponding author 1 the resource is released in https://github.com/ sujinchen0308/translation-error-grounding applied in mt researches on evaluation metrics @xcite , quality estimation @xcite , and error correction @xcite . despite its success, mqm annotation only includes information such as error type, location, and severity. there is no manual annotation resource for grounding the translation errors, that is, grounding the errors to the reasons why the annotated text spans are erroneous translations. this is a serious problem because we can not evaluate llms grounding ability without such resource. if llms do not have the ability in grounding the errors to the right reasons, the corrections or refinements made by them will be untrustworthy. even llms correct the errors in some circumstances, their corrections will be superficial if they do not know the underlying reasons of the errors. to address this problem, we manually build the first evaluation resource for grounding the translation errors. figure 1 illustrates the building process, which adopts a bi-directional grounding scheme (bgs). in the forward grounding, the errors are grounded to their explanations, which state the reason why they are deemed errors. in the backward grounding, the explanations are inversely grounded to the corresponding errors. with error spans masked, the explanations are used for identifying the errors in the translation results. through bgs, the error and explanation are mutually checked to guarantee their validity, and are adjusted to achieve enhanced consistency. based on our manually annotated resource, we evaluate llms ability in grounding the translation errors. it shows that their grounding ability is much worse than human. it reveals the serious problem that llms do not know the reasons of the errors very well, potentially leading to the untrustworthy correction or refinement. therefore, our annotated resource can be used as the new benchmark for llms to enhance their abilities in the translation er-source: 定制产品非质量问题不予退换货。 target: customized products will not @xmath0. explanation: there is a translation error in the target, "退换" should be translated as "exchange or return"; so, change "returned" to "exchanged or returned". forward grounding source: 定制产品非质量问题不予退换货。 target: customized products will not be returned for nonquality problems. explanation: there is a translation error in the target, "退换" should be translated as "exchange or return"; so, change "[mask]" to "exchanged or returned". target: customized products will not @xmath1. figure 1 : the illustration of bgs for grounding the translation errors. the error spans are annotated @xmath2. in the forward grounding, given the source sentence (source), the translation result (target), and the error span, we annotate the explanation for the error span. in the backward grounding, given the source, the target, and the explanation with the error span masked by '[mask]', we identify the error span in the target according to the explanation. furthermore, we apply translation error grounding for automatically filtering false alarmed errors. specifically, given automatically detected errors @xcite , we filter errors that are not consistent before and after bgs since the false alarmed error may be grounded to a hallucinated explanation, which is in turn grounded to a different text span. only the errors keeping consistent after bgs are saved as true errors, which have the solid ground of reasons. we found that llms with better ability in translation error grounding are more effective for filtering the false alarmed errors. in summary, the contributions of our work are as follows: ## related works grounding translation errors is related to the finegrained error analysis, which is beyond assigning a single sentence-level score for evaluating the translation quality. the fine-grained error analysis focuses on specific error words or phrases, and gradually gains attentions in mt researches. we detail the fine-grained error analysis researches and their relation to the translation error grounding. ## building the evaluation resource for grounding translation errors we build the resource by manually grounding the mqm errors @xcite . these errors are from the benchmark dataset in wmt 2023/2024 quality estimation tasks foot_1 . specifically, we select the mqm manually annotated errors on the submissions in wmt2022 chinese-to-english (zh-en) and english-to-german (en-de) general translation task to ground the errors. this selection contains results of 7 participated teams in zh-en task and 15 participated teams in en-de task. we uniformly select equal number of sentences for each participating team to annotate, and each team do not overlap in the source side. in the end, we have around 2.0k manual grounding instances for each translation direction. detailed statistics are listed in the appendix a.1. ## llms ability in grounding translation errors based on our evaluation resource, we test llms ability in grounding translation errors. ## filtering false alarmed errors by error grounding bgs is an ecosystem that explains the error in the forward direction, then verifies the explanation in the backward direction. through such explanation and verification process, true errors will be solidly grounded, while false alarmed errors will hardly find their grounds since they may be grounded to hallucinated explanations in the forward direction, which in turn result in different errors in the backward direction. so, we filter the false alarmed errors by checking whether the error spans remain consistent after bgs, which is executed by llms. ## conclusion current manual annotation of the fine-grained translation errors does not explain the reason why they are erroneous, resulting in the hardness of checking whether llms trustworthily know the reason when they conduct the fine-grained error analysis or correction. in this paper, we manually build the resource for evaluating llms trustworthiness in grounding the translation errors. the bi-directional grounding scheme is proposed for the building. in the forward direction, the errors are manually grounded to their explanations. in the backward direction, the explanations are verified by checking whether the errors can be manually detected according to the explanations, which have the error spans masked. llms are evaluated on this resource through such explanation and verification process. results show that llms performs significantly worse than human in both directions. there is large room for llms to improve their grounding ability. furthermore, we apply the error grounding for filtering false alarmed errors, and achieve significant accuracy improvement in the error detection.