Word-level Translation Quality Estimation Based on Optimal Transport (2024)

Abstract: Word-level translation quality estimation (TQE) is the task of identifying erroneous words in a translation with respect to the source. State-of-the-art methods for TQE exploit large quantities of synthetic training data generated from bilingual parallel corpora, where pseudo-quality labels are determined by comparing two independent translations for the same source text, i.e., an output from a machine translation (MT) system and a reference translation in the parallel corpora. However, this process is sorely reliant on the surface forms of words, with acceptable synonyms and interchangeable word orderings regarded as erroneous. This can potentially mislead the pre-training of models. In this paper, we describe a method that integrates a degree of uncertainty in labeling the words in synthetic training data for TQE. To estimate the extent to which each word in the MT output is likely to be correct or erroneous with respect to the reference translation, we propose to use the concept of optimal transport (OT), which exploits contextual word embeddings. Empirical experiments using a public benchmarking dataset for word-level TQE demonstrate that pre-training TQE models with the pseudo-quality labels determined by OT produces better predictions of the word-level quality labels determined by manual post-editing than doing so with surface-based pseudo-quality labels.

## introduction translation quality estimation (tqe) @xcite is the task of predicting quality labels or scores for a given translation, typically generated by machine translation (mt) systems, with respect to the source text, without referring to a reference translations. predictions can be made at different levels of granularity, such as sentence and word levels. sentence-level quality labels help users determine whether to use an mt output as it is or after post-editing (pe). word-level qual-ity labels better guide post-editors in the translation production process (iso/tc37, 2017), i.e., identifying words that require revision. in this paper, we focus on word-level tqe. the data for training and evaluating word-level tqe models consist of tuples of a source text, an mt output, and quality labels for each word in the mt output. in the tqe shared tasks at the workshop on machine translation (wmt) @xcite @xcite , binary labels, i.e., {"ok," "bad"}, are used as the quality labels. as illustrated in the top part of figure 1 , tqe data are pro-duced through manual pe of mt outputs, where revisions indicate that the words in the mt output are erroneous. it is therefore straightforward to determine gold-standard labels using the translation edit rate (ter) toolkit @xcite foot_0 by identifying the minimum edit distance between two sequences of words relying on surface-level matching. to improve the accuracy further, state-of-theart methods for word-level tqe exploit large quantities of synthetic tqe data for pre-training @xcite @xcite @xcite . figure 1 also shows the typical process of generating synthetic tqe data. first, the source side of a given bilingual parallel corpus is translated with an mt system, and then a pseudo-quality label for each word in the mt output is determined by comparing two independent translations for the same source text: the mt output and the target side of the parallel corpus, i.e., reference translation. previous work @xcite @xcite has used the ter toolkit for the comparison. however, surface-level differences between independent translations do not necessarily indicate errors. for instance, as shown in figure 1 , they can differ in the use of synonyms, interchangeable word orderings, and so forth, even if the mt output is error-free. application of the ter toolkit to such pairs inevitably produces incorrect quality labels and consequently misleads the pre-training of tqe models. in this paper, we describe a method that considers the degree of uncertainty in labeling words in synthetic training data for tqe. to estimate the extent in which each word in the mt output is likely to be correct or erroneous with respect to a reference translation, we propose to use the concept of optimal transport (ot). given a pair of an mt output and a reference translation, our method first obtains contextual word embeddings. it then determines the optimal alignments between words in the mt output and the reference translation with their likelihood. following @xcite , we expect this approach to identify negligible semantic differences between synonymous expressions and corresponding position-free grammatical elements, and properly label them as "ok." empirical experiments using a public benchmarking dataset for word-level tqe, i.e., mlqe-pe @xcite , demonstrate that pre-training tqe models with the ot-based quality labels produces better predictions of the word-level quality labels determined by manual pe than models pre-trained on surfacebased quality labels determined by ter. ## standard framework a word-level tqe model is trained on d @xmath0, t ′ k , y k ) n @xmath1, i.e., a set of n triplets of a source text s k , its machine-translated text t ′ k , and a sequence of quality labels y k corresponding to the words in t ′ k . ## determining pseudo-quality labels with optimal transport this paper describes how better pseudo-quality labels can be assigned to the synthetic tqe data. we assume that the triples d @xmath2, t ′ k , t k ) n @xmath3, d @xmath4, t k ) n @xmath5, and determine the pseudo-quality label for each word in t ′ k by comparing t ′ k with the corresponding t k , as in previous work (section 2.1). in the proposed approach, we apply optimal transport (ot), which identifies the optimal way of converting one distribution into another. the application of ot is inspired by its application to monolingual word alignment @xcite . let [t ′ 1 , . . . , t ′ n ] be a sequence of n words in a given machine-translated text t ′ and [t 1 , . . . , t m ] be a sequence of m words in the corresponding reference translation t . the goal of ot is to identify a matrix @xmath6, where p i,j represents the likelihood of the alignment between t ′ i and t j . to solve our problem with ot, we define the following two concepts: mass of each word: this is a probability simplex, i.e., @xmath7. we denote the mass of n words in t ′ as @xmath8. cost for transportation: a cost function for each pair of words, c(t ′ i , t j ) ∈ r + , can be defined as their dissimilarity. a matrix @xmath9, where c i,@xmath10, t j ), represents a summary of the cost for all pairs of words. the cost is typically computed on the basis of contextual word embeddings. in the process of obtaining the embeddings, such as by using a pre-trained multilingual encoder, we can also refer to the source text (see appendix a), which is an advantage of this method over the ter toolkit. a matrix @xmath11: soft label is a real number between 0.0 and 1.0, where 0.0 indicates that nothing is transported from the word, strongly suggesting that the word is erroneous, while 1.0 indicates that the word perfectly aligns with a word in t . hard label is a binary label, {"ok," "bad"}, which is determined by thresholding the soft label. we introduce this merely for a comparison with the conventional binary labels determined by the ter toolkit. finally, we obtain two sets of synthetic data for word-level tqe: ## experiments to confirm the effectiveness of the proposed method, we conducted experiments using a public dataset for word-level tqe, mlqe-pe @xcite . 2 following recent shared tasks on word-level tqe @xcite @xcite and @xcite , we evaluated tqe models using the matthews correlation coefficient (mcc) @xcite . ## analyses we investigated the quality of the synthetic tqe data and the potential utility of ot-based labels for manually post-edited data. we used the non-zeroshot translation directions of test20 because the post-edited texts for the mt outputs are available, enabling contrastive experiments. ## conclusion this paper has described the application of optimal transport (ot) to determine pseudo-quality labels in synthetic data for word-level tqe. through experiments, we confirmed that ot-based labels better guide pre-training on large quantities of synthetic tqe data and result in higher accuracy in word-level tqe tasks, as measured by mcc. our method achieved consistently better results for pseudo-supervised settings and in zero-shot translation directions, encouraging future applications to less-studied translation directions. in future work, we plan to investigate better and finer-grained specifications of the hyper-parameters for ot. while we determined a single value of λ m , the upper bound of the mass to be transported, for each translation direction, we consider it should be possible to approximate this value for each sentence pair. we have only evaluated our method for predicting target labels; doing so for source labels is another avenue for extension (appendix b).