Direct Exploitation of Attention Weights for Translation Quality Estimation (2021)

Abstract: The paper presents our submission to the WMT2021 Shared Task on Quality Estimation (QE). We participate in sentence-level predictions of human judgments and post-editing effort. We propose a glass-box approach based on attention weights extracted from machine translation systems. In contrast to the previous works, we directly explore attention weight matrices without replacing them with general metrics (like entropy). We show that some of our models can be trained with a small amount of a high-cost labelled data. In the absence of training data our approach still demonstrates a moderate linear correlation, when trained with synthetic data.

## introduction quality estimation @xcite is an essential part of the machine translation (mt) pipeline, which estimates the quality of the translation output without relying on any reference. unlike the previous year, three qe sentencelevel tasks were presented in the wmt2021 shared task @xcite . the goal of task 1 is to predict direct assessments (da), i.e. human judgments of translation quality @xcite , whereas in task 2, the task is to estimate the post-editing effort required to obtain a correct translation which is measured by the hter metric @xcite . the goal of task 3 is to determine if the translation output contains at least one critical error. we propose a lightweight glass-box approach that can be applied to task 1 and task 2. the approach is based on using the encoder-decoder attention weight matrices as input features for supervised translation quality estimation. next we 1 http://www.statmt.org/wmt21/ quality-estimation-task.html describe our approach (section 2) and evaluate it experimentally (sections 3-5). ## approach there are several qe models based on attention weights of neural mt systems described earlier @xcite . their main idea is to compute the entropy of encoder-decoder attention weights for each target token and then average these entropies to get a sentence-level metric: in this article, we propose another approach of using attention weights obtained from transformer mt models: instead of summarising them into one metric, we feed all encoder-decoder attentions weights into a convolutional neural network (cnn) to get a qe score. we test the approach in a supervised setting, and also show that it can be applied in a zero-shot scenario when training data for the required language pair is not available. ## data this year, three sentence-level tasks are available: predicting human judgments, post-editing effort and critical errors. in this work, we have focused on the first two tasks. task 1 and 2 include eleven language pairs, seven of which have training (7 000 sentences), development (1 000 sentences) and two test sets (wmt2020 and wmt2021, 1 000 sentences each). for the other four languages only test sets (1 000 sentences) are available, which is called the zeroshot subtask. wmt2020 test set includes goldlabels whereas wmt2021 is the usual blind test without labels before submission. to test our approach, we have focused on two language pairs with training data: english-german (en-de) and estonian-english (et-en), as well as one language pair without training data: english-czech (en-cz). besides data provided by the shared task organizers, we used additional parallel corpora to train cnn networks: the opensubtitles @xcite , jrc acquis @xcite , europarl @xcite , dgt and emea @xcite corpora. ## settings to compare the performance of our approach with cnn models to a previous baseline, we also ran experiments with models based on machine learning algorithms with entropies as input features (ml-ent). below we present the experimental settings which we used for training ml-ent and cnn models. ## results below we present the obtained results and discuss the most interesting observations. to assess the performance of sentence-level qe models, pearson correlation coefficient is used. results for zero-shot en-cs are not impressive ( the essential advantage of using cnn-bleurt models is that they might be used for zero-shot settings when a training dataset is not available. however, the building and tuning of the neural network is not an easy task compared to ml-ent models. the benefits of last ones are relatively fast training and fewer parameters that need to be tuned. features of ml-ent models are identical across da-and hter-models as well as cnn-da and cnn-hter share the same input features. computed correlation coefficients of hter-and damodels are comparable for et-en language pair. nevertheless, we see a completely different picture for en-de: coefficients of da-models are noticeably lower compared to hter-models. as discussed in @xcite , the low results of da models for en-de language pair might be caused by highly-skewed distribution of da scores, as most translations have high quality scores. getting da scores as well as hter scores is a time-consuming and expensive task, so the less data you need, the better. to examine how labelled data we need to train models, we ran 10 tests for each examined amount of data (25%, 50%, 75%) and averaged the obtained correlation coefficients. according to our experiments, both discussed approaches, ml-ent and cnn-hter/da, show comparable high performance even with a small amount of training/validation data. as figure 2 shows, the performance of ml-ent models for et-en (top) and en-de (bottom) language pairs slightly worsens with decreased amounts of training data. the performance of cnn-hter models decreases more noticeably, but still remains quite high. especially in case of the en-et language pair, all models demonstrate a moderate linear correlation with post-editing effort even with using 2000 training/validation examples (1750 for training and 250 for validation). @xcite showed that different layers play different roles in the attention mechanism. to examine it from the qe point of view, we compared cnn-hter models with attention weights extracted from the first three layers, the last three layers and all six layers. according to the models with last layers is comparable to the "all layers" models, whereas the difference between models with first layers and "all layers" models is more noticeable. while the performance gap between different models is not so noticeable for et-en, then for en-de the difference is significant and even more, the lower layers do not provide any "useful" information to the model. ## conclusions we presented sentence-level quality estimation models based on attention weights. the proposed models demonstrated a moderate linear correlation with human judgments as well as with required post-editing effort. the described models can be used as a cost-effective and light-weight qe approach in the machine translation pipeline. results of empirical evaluation show a good performance even with a small amount of training data, as well as moderate performance in the absence of training data ("zero-shot" settings).