Error Identification for Machine Translation with Metric Embedding and Attention (2021)

Abstract: Quality Estimation (QE) for Machine Translation has been shown to reach relatively high accuracy in predicting sentence-level scores, relying on pretrained contextual embeddings and human-produced quality scores. However, the lack of explanations along with decisions made by end-to-end neural models makes the results difficult to interpret. Furthermore, word-level annotated datasets are rare due to the prohibitive effort required to perform this task, while they could provide interpretable signals in addition to sentence-level QE outputs. In this paper, we propose a novel QE architecture which tackles both the word-level data scarcity and the interpretability limitations of recent approaches. Sentence-level and word-level components are jointly pretrained through an attention mechanism based on synthetic data and a set of MT metrics embedded in a common space. Our approach is evaluated on the Eval4NLP 2021 shared task and our submissions reach the first position in all language pairs. The extraction of metric-to-input attention weights show that different metrics focus on different parts of the source and target text, providing strong rationales in the decision-making process of the QE model.

## introduction quality estimation (qe) for machine translation (mt) @xcite @xcite aims at providing quality scores or labels to mt output when translation references are not available. sentence-level qe is usually conducted using human produced direct assessments (da) @xcite or post-edits. the latter allows to derive token-level quality indicators such as good and bad tags @xcite . token-level qe is particularly useful for applications such as source preediting or focused mt post-editing, but requires high-quality fine-grained annotated data for supervised learning. furthermore, token-level quality indicators can be seen as explanations for sentencelevel scores, whether given by humans or automatically produced. however, explainability of qe models decisions is obscured by contemporary approaches relying on large data-driven neural-based models, making use of pretrained contextual language models (lm) such as bert @xcite and xlm @xcite , albeit showing steady performance increase as reported in the qe shared tasks @xcite ). yet, the qe layers and architectures are rarely investigated, neither for performance nor for interpretability purposes, and the center of attention is mainly on large pretrained models and generating additional (synthetic) training corpora. in this paper, we present a novel qe architecture which encompasses a metric-to-input attention mechanism allowing for several extensions of the habitual qe approach. first, since sentence-level qe scores are usually obtained with surface-level mt metrics computed between translation outputs and human produced references or post-edits such as hter @xcite , we propose to make use of several metrics simultaneously in order to model translation errors at various granularities, i.e. at the character, token, and phrase levels. second, we design a metric embeddings model which represents metrics in their own space through a dedicated set of learnable parameters, allowing for straightforward extensions of the number and type of metrics. third, by employing an attention mechanism between metric embeddings and bilingual input representations, the metric-to-input attention weights indicate where each metric focuses given an input sequence, increasing the interpretability of the qe components. we conduct a set of experiments on the eval4nlp 2021 shared task dataset @xcite using only the training data along with sentence-level scores officially released for the tasks (illustrated in figure 1 ). in addition, we source religioon pakub vaimu puhastamiseks teatud vahendeid . ## background current state-of-the-art qe approaches are commonly based on sentence encoders taking as input source-translation pairs @xcite @xcite . encoders are usually contextual lms pretrained on large amount of multilingual data. existing qe implementations commonly rely on additional layers added on top of a pretrained lm, which enables multi-task learning for word and sentence-level qe. pretraining of contextual lms is done by optimizing a prediction function given input sequences of tokens containing randomly masked tokens, or tokens randomly replaced by other tokens sampled from the vocabulary. formally, given an input sequence z of n tokens z 1:n , corresponding word (or subword) embeddings x 1:n with dimension d (x 1:@xmath0, and output contextual embeddings h l 1:@xmath1, l] ⊂ n of a transformer encoder @xcite . usually based on the output of the last encoder layer, the model optimizes the following loss function: ℓ(s, t) = -e r∼[1,n] log p (z r | zr ), where z r are randomly sampled tokens from z to be masked or replaced and zr are the remaining tokens from z with @xmath2, n] ⊂ n. to perform qe, qe-specific layers are commonly added on top of pretrained contextual lms, being fed with contextual token embeddings from the topmost (i.e., l-th) layer of the lm. for sentence-level qe the specific component is a regression head formalized by y @xmath3:n )• w s + b s ), where y @xmath4, 1] ⊂ r. w s and b s are trainable parameters of the linear output layer, φ is a pooling function, and σ is the sigmoid function. the output y s of this qe component is a score indicating the sentence-level translation quality. for token-level qe a classification head is implemented as y t 1:@xmath5:n • w t + b t ), where y t 1:@xmath6, w t and b t are trainable parameters of the linear output layer. the output y t 1:n of this qe component is a vector of labels indicating the translation quality of corresponding input tokens. source tokens are annotated according to the accuracy of their translation, while annotations of target tokens also take into account their position in the target sequence. multiple losses are then computed, one for the sentence-level and one for each word-level outputs (source and target tokens), based on gold labels to train (or finetune) the contextual lm and the qe model in an end-to-end fashion using backpropagation @xcite @xcite . commonly used losses are cross-entropy and mean-squared error for classification and regression respectively. however, this approach has limitations while the token-level qe implementation makes use of each input token representation in context thanks to the pretrained lm, the sentence-level qe components relies only on the pooled representation of the input sequence. this approach drastically limits the amount of information flowing through the sentence-level specific set of layers and may force the network to focus more on cues and data artifacts which correlate with qe scores, instead of encoding translation-related features from source and target inputs @xcite . these findings corroborate with the empirical observation made by @xcite , where the authors obtained the best word-level qe results using bert and ignoring target language features when predicting source quality labels and vice-versa. additionally, most recent qe approaches do not allow for the interpretability of sentence-level qe predictions at test time and leads to the current state of qe as a set of black-box components. furthermore, token-level error annotations is costly to produce. ## metric embedding and attention motivated by the limitations to contextual lm based qe, we propose a novel architecture employing metric embeddings and attention, which is computed between the contextual embeddings and the embedded qe criteria of the supervised learning task, namely mt automatic metrics or direct assessment scores provided by human annotators. the metric embedding matrix @xmath7, randomly initialized at the beginning of training, is added on top of the pretrained lm to model metrics in their own space, with a predefined set of sentence-level metrics @xmath8, . . . , m g }. each metric is initially represented as a one-hot vector, noted m @xmath9, g] ⊂ n. its corresponding embedding is retrieved with m j • e, forming the query used in the attention mechanism (eqn. 1): keys and values which are the two other components of the attention mechanism, noted k i and v i respectively, are computed based on the output of the topmost layer of the pretrained lm, which is first fed into a position-wise feed-forward layer following (eqn. 2): metrics to tokens attention weights aim to represent the focus made by a given metric on specific parts of the input sequences. these attention weights are computed between the embedding of a metric and the contextually encoded input tokens (eqn. 3): sentence-level scores are obtained for each metric with the weighted sum of value vectors for each attention head (eqn. 4): with w @xmath10. finally, we project y s j from the model dimensionality to a single score through a metric specific linear layer: y s @xmath11, 1] ⊂ r. token-level qe scores are computed by using the attention weights (see eqn. 3) followed by three transformations: attention heads combination through a linear transformation, concatenation of token embeddings and combined attention heads, combination of metrics through a final linear transformation (eqn. 6): the learning process allows for supervised or unsupervised token-level qe. note that all learnable parameters for the sentence-level qe components except w s j are shared between metrics, including the metric embeddings matrix e. we believe that such an approach enables to capture translation errors at different granularities according to the specificity of each metric, e.g., characters, tokens and phrases, while keeping a reasonable total amount of learnable parameters. the loss functions for sentence-level qe are mean-squared architecture of the metric embeddings and attention mechanism. shaded elements, curved arrows and ⊕ are parameters of the model, i and j are the attention head and the metric indexes respectively. supervised learning is conducted by computing losses according to each output of the model and their corresponding gold labels from the training data. thus, for the sentence-level qe layers, we compute one loss per metric (mean squared error), while for the token-level qe layers, if tokenlevel annotations are available, two losses allow to optimize the model for source and target tokens separately (cross-entropy). unsupervised learning is conducted when token-level annotations are not available, which is one of the objectives in the constrained task of eval4nlp 2021. in this case, only sentence-level losses are used to optimize the parameter of the model through backpropagation. following the guidelines of the shared task, we do not use the direct assessment annotations made by humans at the word-level. ## experimental setup this section presents our experimental setup, including the pretrained models, the datasets and the training procedure. all pretrained models and scripts used in our experiments are based on py-torch @xcite and all computations are conducted on nvidia v100 gpus with cuda v10.2. ## results and analysis we present in results show that our baselines and final submissions largely outperform the organizer's baselines on the four language pairs, while our final submissions reach higher source and target tokenlevel performances compared to our baselines. except for the de-zh pair, our final submissions are outperforming our baselines on the sentence-level pearson's ρ evaluation. because the main objec- in order to evaluate the impact of qe pretraining and finetuning, as well as the difference in performance between ensemble and single models trained using language pair specific (bilingual) and multilingual datasets, we present an ablation study conducted on the word-level qe in as an explanation of sentence-level scores predicted by our model, we propose to extract the attention weights computed between the metric embeddings and the contextually encoded input sequences. a few samples extracted from the validation set are presented in figure 3 . we can see on these examples that individual metrics do not correlate with human annotations. however, the multimetric approach, which relies on heads and metrics combination through linear layers, provides a potential error identification method. ## previous work since the shift of most nlp tasks towards using large pretrained contextual lms as basis for taskspecific finetuning, the research community working on qe for mt moved from a classic two-step process of feature engineering followed by machine learning @xcite @xcite to an end-to-end training neural-based paradigm. first attempts in this direction were conducted by @xcite with the predictorestimator, which inspired further work in using various types of encoders @xcite , enriching the model with features extracted from nmt models @xcite or modifying the pretraining objective of contextual lms for qe adaptation @xcite . more recently, due to the costly nature of data acquisition for supervised learning of qe models, unsupervised approaches were proposed by relying mostly on signals given by nmt systems when translating source sentences @xcite , the so-called glass-box features. alternatively, when the nmt systems which produced data to perform qe on are not available (i.e. blackbox setting), relying on large amount of synthetic data for contextual lm continued training prior to finetuning appears to be an effective way to approximate human judgments of translation quality @xcite . ## conclusion this paper presented a novel qe architecture for unsupervised token-level quality prediction providing sentence-level explainable decisions from the model. we implemented a metric embeddings and attention mechanism on top of a widely used pretrained contextual lm, allowing to add metrics during finetuning and enabling high performance qe both at the levels of token and sentence. this extensible framework was shown to produce results on par or outperforming state-of-the-art qe approaches without relying on human-produced token-level annotations, which could be approximated with the use of relatively cost-effective synthetic data and automatic metrics. our pivot-based translation approach also tackled a recurrent issue in mt when parallel data are scarce and final results for zero-shot language pairs validated this method.