Towards Precise Localization of Critical Errors in Machine Translation (2024)

Abstract: The advent of large language models has experienced a remarkable improvement in the field of machine translation. However, machine translation is still vulnerable to critical meaning deviations, which may incur catastrophic issues in social or ethical contexts. In particular, existing critical error detection primarily focuses on identifying sentence-level errors, leaving the precise localization of such errors within the sentence unaddressed. In this paper, we introduce a new task, word-level critical error detection (WCED), to detect critical errors at a fine-grained level in machine translation sentences. The task aims to identify the parts of a machine translation that contain catastrophic meaning distortions. We hypothesize that the ability to determine errors at the sentence level will positively influence the detection of more granular errors. We propose a sentence-level error detection module to predict which words in a sentence have critical errors. Experimental results demonstrate that our method outperforms existing methodologies and LLM in En-De, Zh-En, En-Ru, and En-Ko. Our method is helpful for determining the fine-grained location of errors. We hope that such studies will improve the capacity to address critical errors adeptly.

## introduction recent advancements in large language models (llms) have significantly improved the performance of machine translation (mt), leading to an increased demand for such systems @xcite @xcite . concurrently, the necessity for quality estimation (qe), which automatically evaluates the output of mt systems, has also grown. qe measures the quality of mt outputs based on the source and the mt sentence without a human reference @xcite @xcite ). ## word-level critical error detection (wced) in this section, we define the task of word-level critical error detection (wced) and propose methods to enhance the word-level error detection process. to provide appropriate guidance for exploring errors in a granular manner, we hypothesize an ideal sentence-level ced module capable of accurately identifying the presence of errors at the fine-grained level. even in the absence of a perfect ced module, leveraging existing modules will be sufficient to assess their effectiveness in approximating our hypothesis. ## conclusion in this study, we introduced wced to detect fine-grained critical errors in mt. this task aimed to identify catastrophic meaning distortions at the word level effectively. we proposed a method that utilizes a sentence-level error detection module to predict words containing critical errors within sentences. experimental results demonstrated that our proposed method significantly enhanced the accuracy of error detection compared to comparison methods and chatgpt. these findings indicated that a sentence-level detection module could accurately guide the task of detecting word-level errors. our approach contributed to a more nuanced evaluation and understanding of mt quality and was crucial in preventing potential issues caused by critical errors. future research may improve the wced performance and extend its applicability across various languages and domains. also, we hope that our methods can be adopted in developing and evaluating mt systems to improve their ability to cope with critical errors.