KG - IQES : An Interpretable Quality Estimation System for Machine Translation Based on Knowledge Graph (2023)

Abstract: The widespread use of machine translation (MT) has driven the need for effective automatic quality estimation (AQE) methods. How to enhance the interpretability of MT output quality estimation is well worth exploring in the industry. From the perspective of the alignment of named entities (NEs) in the source and translated sentences, we construct a multilingual knowledge graph (KG) consisting of domain-specific NEs, and design a KG-based interpretable quality estimation (QE) system for machine translations (KG-IQES). KG-IQES effectively estimates the translation quality without relying on reference translations. Its effectiveness has been verified in our business scenarios.

## introduction qe is an important task in mt research. it directly reflects the quality of mt output in real time during real-world application. existing popular aqe metrics of mt output include: (1) lexicon-based bleu @xcite , which performs n-gram matching between the mt output and reference translations and calculates the matching score; (2) embedding-based bleurt @xcite and bertscore @xcite , which compare the semantic vector distance between the reference translation and mt output. however, both of the metrics need to rely on reference translations, which are scarce in real-world application @xcite and require heavy labor costs to construct. in recent years, various metrics have been proposed for reference-free qe. one of the current state-of-the-art metrics is model-based comet-qe @xcite , which directly calculates the quality score by using the source sentence and the translation through neural network, but leads to poor interpretability. a more interpretable metric is knowledge-based machine translation evaluation (kobe) @xcite , a google-proposed system-level metric based on kg for common domains, which predicts the translation quality by calculating the similarity between bilingual entities in the source and translated sentences and linked entities in the kg. although this metric has good interpretability and good performance in system-level evaluations of common domains, it is ineffective in specific domains. this is because in specific domains, each paragraph or sentence contains a large number of domain-specific entities, and these entities can only be effectively covered by domain-specific kg, not kg for common domains. to address these shortcomings, we design a highly interpretable segment-level qe system based on multilingual kg for the wireless network foot_0 domain, that is, an interpretable quality estimation system for machine translation based on knowledge graph (kg-iqes). kg-iqes identifies nes in the source sentence first, queries the ne translations in the target language from the kg constructed offline, and matches the translated nes in the kg with nes in the translated sentence. in this way, the translation quality can be estimated, and the entity alignment details can be displayed (see example in figure 1 ). overall, our contributions are as follows: ## related work reference-free qe of mt output is a task that predicts quality of the machine translations by scoring the source text and machine-translated text. many metrics have been proposed in this regard, roughly divided into three categories: according to current research results, embedding-based and model-based metrics are more effective, but the interpretability is poor. they are not intuitive, and it is difficult to apply them in real world. in this paper, we follow the lexicon-based quality evaluation direction for mt output, and design a highly interpretable qe system kg-iqes. its effectiveness is verified in the wireless network domain. ## proposed system in this section, we provide a description of the kg-iqes system. as shown in figure 2 , the system consists of two subsystems: offline and online subsystems. the offline subsystem mainly constructs a multilingual kg based on the monolingual ner model and ne alignment model from the parallel corpora. the online subsystem provides an interpretable quality score for the mt output. ## experiments the following describes the detailed experiment process in the chinese-english corpus of the wireless network domain. first, we carry out ner model training and fine-tuning experiment based on the chinese corpus of the wireless network domain. we then compare two typical cross-lingual ne alignment methods, fast-align and xlm-roberta, in the process of building the bilingual kg (chinese and english) in the wireless network domain. finally, we use our kg-iqes system and google's kobe @xcite to estimate the mt output quality of different translation models, and compare the correlation between the two estimation results with direct assessment (da).