Rethinking the Word-level Quality Estimation for Machine Translation from Human Judgement (2023)

Abstract: Word-level Quality Estimation (QE) of Machine Translation (MT) aims to detect potential translation errors in the translated sentence without reference. Typically, conventional works on word-level QE are usually designed to predict the quality of translated words in terms of the post-editing effort, where the word labels in the dataset, i.e., OK or BAD, are automatically generated by comparing words between MT sentences and the post-edited sentences through a Translation Error Rate (TER) toolkit. While the post-editing effort can be used to measure the translation quality to some extent, we find it usually conflicts with human judgment on whether the word is well or poorly translated. To investigate this conflict, we first create a golden benchmark dataset, namely HJQE (Human Judgement on Quality Estimation), where the source and MT sentences are identical to the original TER-based dataset and the expert translators directly annotate the poorly translated words on their judgments. Based on our analysis, we further propose two tag-correcting strategies which can make the TER-based artificial QE corpus closer to HJQE . We conduct substantial experiments based on the publicly available WMT En-De and En-Zh corpora. The results not only show our proposed dataset is more consistent with human judgment but also confirm the effectiveness of the proposed tag-correcting strategies.For reviewers, the corpora and codes can be found in the attached files.

## introduction quality estimation of machine translation aims to automatically estimate the translation quality of the mt systems with no reference available. the sentence-level qe predicts a score indicating the overall translation quality, and the word-level qe needs to predict the quality of each translated word as ok or bad. recently, the word-level qe attracts much attention for its potential ability to directly detect poorly-translated words and alert the user with concrete translation errors. currently, the collection of the word-level qe datasets mainly relies on the translation error rate (ter) toolkit @xcite . specifically, given the machine translations and their corresponding post-edits (pe, generated by human translators or target sentences of the parallel corpus as the pseudo-pe), the rulebased ter toolkit is used to generate the wordlevel alignment between the mt and the pe based on the principle of minimal editing @xcite . all mt words not aligned to pe are annotated as bad (shown in figure 1 ). such annotation is also referred to as post-editing effort @xcite . the post-editing effort measures the translation quality in terms of the efforts the translator needs to spend to transform the mt sentence to the golden reference. however, in our previous experiments and real applications, we find it usually conflicts with human judgments on whether the word is well or poorly translated. two examples in figure 2 show the conflicts between the ter-based annotation and human judgment. in figure 2a , the translated words, namely "我", "很", "高兴" and "发 言", are annotated as bad by ter since they are not exactly in the same order with their counterparts in the pe sentence. however, from human judgment, the reordering of these words does not source: it is happy for me to be asked to speak here. mt: 我 很 高兴 被 要求 在 这里 发言 。 mt back: i am so happy to be asked to speak here. pe: 被 邀请 在 这里 讲话 我 很 高兴 。 pe back: being invited to talk here makes me so happy. ter-based: 我 很 高兴 被 要求 在 这里 发言 。 human: 我 很 高兴 被 要求 在 这里 发言 。 source: the zaporizhian hetman was then dispatched to istanbul, and impaled on hooks. mt: 扎 波罗 齐安海 特曼 号 随后 被 派 往 伊斯坦布尔 ，并 被 撞 在 钩 上 。 mt back: the zaporizhian hetman was then dispatched to istanbul, and was bumped on the hook. pe: zaporizhian hetman 随后 被 派 往 伊斯坦布尔 ， 并 被 钉 在 钩子 上 。 pe back: zaporizhian hetman was then dispatched to istanbul, and was nailed on hooks. hurt the meaning of the translation and even makes the mt sentence polished. and the word "要求" is also regarded as a good translation by human judgment as it is the synonym of the word "邀 请". in figure 2b , the clause "扎波罗齐安海特曼 号" in a very good translation of "the zaporizhian hetman " from human judgment. however, it is annotated as bad by ter since it is not aligned with any words in the pe sentence. in many application scenarios and downstream tasks, it is usually important even necessary to detect whether the word is well or poorly translated from the human judgment @xcite . however, most previous works still use the ter-based dataset for training and evaluation, which makes the models' predictions deviate from human judgment. ## approach this section first introduces the model backbone and the self-supervised pre-training approach based on the large-scale mt parallel corpus. then, we propose two correcting strategies to make the terbased artificial tags closer to human judgment. ## experiments datasets. to verify the effectiveness of the proposed corpus and approach, we conduct experiments on both hjqe and mlqe-pe. note that mlqe-pe and hjqe share the same source and mt sentences, thus they have exactly the same number of samples. we show the detailed statistics in baselines. to confirm the effectiveness of our proposed self-supervised pre-training approach with tag-correcting strategies, we mainly select two baselines for comparison. in the one, we do not use the pre-training, but only fine-tune xlm-r on the training set of hjqe. in the other, we pre-train the model on the ter-based artificial qe dataset and then fine-tune it on the training set of hjqe. ## related work early approaches on qe, such as quest @xcite and quest++ @xcite , mainly pay attention to feature engineering. they aggregate various features and feed them to machine learning algorithms. @xcite first propose the neural-based qe approach, called predictor-estimator. they first pre-train an rnnbased predictor on the large-scale parallel corpus that predicts the target word given its context and the source sentence. then, they extract the features from the pre-trained predictor and use them to train the estimator for the qe task. this model achieves the best performance on the wmt17 qe shard task. after that, many variants of predictor-estimator are proposed @xcite @xcite . among them, bilingual expert @xcite replaces rnn with multi-layer transformers as the architecture of the predictor. it achieves the best performance on wmt18. @xcite release an open-source framework for qe, called openkiwi, that implements the most popular qe models. recently, with the development of pre-trained language models, many works select the cross-lingual language model as the backbone @xcite @xcite @xcite . many works also explore the joint learning or transfer learning of the multilingual qe task @xcite @xcite . @xcite propose a shared task with the newcollected dataset on explainable qe, aiming to provide word-level hints for sentence-level qe score. freitag et al. (2021b) the qe model can be applied to the post-editing process. @xcite and @xcite use the qe model to identify which parts of the mt sentence need to be corrected. @xcite needs the qe model to determine error spans before giving translation suggestions. ## conclusion in this paper, we focus on the task of word-level qe in machine translation and target the inconsistency issues between ter-based annotation and human judgment. we collect and release a benchmark dataset called hjqe which has identical source and mt sentences with the ter-based corpus and reflects the human judgment on the translation errors in mt sentences. besides, we propose two tagcorrecting strategies, which make the ter-based annotations closer to human judgment and improve the final performance on the proposed benchmark dataset hjqe. we conduct thorough experiments and analyses, demonstrating the necessity of our proposed dataset and the effectiveness of our proposed approach. our future directions include improving the performance of phrase-level alignment. we hope our work will provide some help for future research on quality estimation.