Explainable CED : A Dataset for Explainable Critical Error Detection in Machine Translation (2024)

Abstract: Critical error detection (CED) in machine translation is a task that aims to detect errors that significantly distort the intended meaning. However, the existing study of CED lacks explainability due to the absence of content addressing the reasons for catastrophic errors. To address this limitation, we propose Explainable CED, a dataset that introduces the attributes of error explanation and correction regarding critical errors. Considering the advantage of reducing time costs and mitigating human annotation bias, we leverage a large language model in the data construction process. To improve the quality of the dataset and mitigate hallucination, we compare responses from the model and introduce an additional data filtering method through feedback scoring. The experiment demonstrates that the dataset appropriately reflects a consistent explanation and revision for errors, validating the reliability of the dataset.

## introduction critical error detection (ced) is a sub-task of quality estimation (qe) that aims to identify sentences where the intended meaning from the source text is distorted due to catastrophic errors in machine translation (mt) systems @xcite . these distortions potentially lead to offensive interpretations or cause social, legal, or economic issues. while critical errors are infrequent and can be considered a long-tail problem, it is essential to prevent issues caused by them, thereby emphasizing the importance of ced in ensuring the quality of mt systems @xcite . however, the existing binary classification approach of the ced, which detects the presence of fatal errors, merely blocks the erroneous output from the mt system. users, not native speakers of the target language, cannot specify the description for the error and its solutions (sharou and @xcite . these limitations highlight the necessity for a comprehensive approach to address critical errors to provide more precise guidance for non-native users @xcite . in this regard, we propose a novel explain-able ced dataset that includes descriptions of portions significantly mistranslated from the original intention and the corrected text that aligns with the intended meaning. each instance in explain-able ced consists of the source sentence, target sentence, error type, error explanation, and sentence with the error corrected. to develop the dataset, we use a large language model (llm)based method. by leveraging the llm, we can further reduce the time and computational resources in the data collection process. this approach al-leviates issues associated with the inconsistency between human annotators and inherent biases that are uncontrollable @xcite @xcite . additionally, llm not only exhibits exceptional performance across overall mt tasks @xcite @xcite but also demonstrates more efficient capabilities in data labeling compared to humans @xcite . however, as llms still face challenges related to hallucinations @xcite , our focus on data generation is on mitigating hallucinations. when hallucinations are incorporated in the responses of llms, the responses may differ from each other and encompass potentially contradictory information @xcite @xcite . to mitigate hallucinations in llms, we adopt a method that compares responses to various prompts and selects consistent instances to enhance coherence. furthermore, we aim to improve the quality of the data by filtering it based on feedback scores. in the experiment, we introduce supplementary inputs to investigate the mitigation of the hallucination in the dataset. the results reveal that each instance in the dataset is structured to retain mutually similar semantics, indicating that the dataset is constructed to reflect the hallucination mitigation strategy. we hope that this research will offer solutions to critical errors and aid in future studies aimed at improving the reliability of mt. ## related works the conference on machine translation (wmt) in 2021 introduces the ced for qe @xcite . @xcite proposes a classifier that adds sampling to handle unbalanced data to detect critical errors and integrates existing techniques for finding errors. @xcite introduces a system that uses pre-trained xlm-r as a predictor and stacked ffn layers as a binary classifier and uses commercial machine translation tools to help detect errors. @xcite utilizes prompt-based fine-tuning, combining demonstration and commercial machine translation systems to perform the classification task. explainable qe is an explainability subtask following its first edition at eval4nlp 2021 @xcite . the interpretability of qe systems may be compromised due to their reliance on models with numerous parameters. to address this issue and maintain user trust, explainable qe is proposed @xcite . @xcite proposes the sentence-level qe model's predictor as a feature extractor for sentence word embeddings and utilizes the inverse value of maximum similarity between each word in the target and the source as the word translation error risk value. from a different perspective, perturbation-based qe proposes an unsupervised word-level qe approach for evaluating blackbox mt systems (dinh and niehues, 2023). the knowledge-prompted estimator employs the chain of thought prompting method to provide enhanced interpretability for qe @xcite . as evidenced in previous studies, ced primarily focuses on classifying binary labels that indicate the presence or absence of errors. a limitation of this binary classification is that it only enables the mt system to prevent the presentation of translation results. consequently, users fail to receive translated outputs and struggle to understand and recognize the errors that occur correctly. to address this limitation, we propose a task that allows users to comprehend and accept the critical errors that arise and provides them with corrected translations. ## explainable ced in this section, we introduce a detailed description of the components constituting the explainable ced dataset and a methodology for constructing the dataset through a three-phase process, considering consistency and hallucination. the dataset contains three elements to explain translation errors when given a source sentence and an mt sentence containing the error ( error type refers to a categorized label reflecting the characteristics of errors. when multiple errors are present, we prioritize and address only the most severe ones. we adopt the categories defined by @xcite as follows: for readers, as it constitutes a translation error. the errors may occur when content not present in the source sentence is introduced in the translation or when content from the source sentence is omitted. error explanation refers to a description in natural language that details the occurrence of errors in mt sentences. this includes explicit instances indicating which part of the sentence contains the translation error. beyond the labeled instances, the explanation offers a profound insight into the cause and characteristics of the problem, thereby heightening the awareness of the error's severity. error correction refers to the revised sentence where the translation sentence, which distorted the original meaning, is corrected. the correction aims to modify the erroneous parts in the translation sentence with the least amount of editing. ## experiments we investigate the experimental results for three tasks using the explainable ced dataset. we validate the dataset with a focus on whether hallucination is mitigated due to low-quality data being filtered out foot_1 . to verify that the dataset contains consistent content, we conduct experiments incorporating each dataset element as input. this assumes that if adding each dataset component to the input yields a positive impact, it suggests that the dataset is composed of consistent responses. ## conclusion we introduced explainable ced dataset to provide explainability for critical errors in mt. this dataset offered descriptions of errors across error types and fixing them. in constructing the dataset, we proposed a framework for leveraging llm. the objective was to mitigate the hallucination by maintaining consistency in the model's responses and to enhance the quality of the generation by the self-refine. the results indicated that our dataset maintains consistency despite being generated by various model responses.