C omet K iwi: IST -Unbabel 2022 Submission for the Quality Estimation Shared Task (2022)

Abstract: We present the joint contribution of IST and Unbabel to the WMT 2022 Shared Task on Quality Estimation (QE). Our team participated in all three subtasks: (i) Sentence and Word-level Quality Prediction; (ii) Explainable QE; and (iii) Critical Error Detection. For all tasks we build on top of the COMET framework, connecting it with the predictor-estimator architecture of OpenKiwi, and equipping it with a word-level sequence tagger and an explanation extractor. Our results suggest that incorporating references during pretraining improves performance across several language pairs on downstream tasks, and that jointly training with sentence and word-level objectives yields a further boost. Furthermore, combining attention and gradient information proved to be the top strategy for extracting good explanations of sentence-level QE models. Overall, our submissions achieved the best results for all three tasks for almost all language pairs by a considerable margin.

## introduction quality estimation (qe) is the task of automatically assigning a quality score to a machine translation output without depending on reference translations @xcite . in this paper, we describe the joint contribution of instituto superior técnico (ist) and unbabel to the wmt22 quality estimation shared task @xcite , where systems were submitted to three tasks: (i) sentence and word-level quality prediction; (ii) explainable qe; and (iii) critical error detection. this year, we leverage the similarity between the tasks of mt evaluation and qe and bring together the strengths of two frameworks, comet @xcite , which has been originally developed for reference-based mt evaluation, and openkiwi @xcite , which has been developed for word-level and sentence-level qe. namely, we implement some of the features of the latter, as well as other new features, into the comet framework. the result is cometkiwi, which links the predictor-estimator architecture with comet training-style, and incorporates wordlevel sequence tagging. given that some language pairs (lps) in the test set were not present in the training data, we aimed at developing qe systems that achieve good multilingual generalization and that are flexible enough to account for unseen languages through few-shot training. to do so, we start by pretraining our qe models on direct assessments (das) annotations from the previous year's metrics shared task as it was shown to be beneficial in our previous submission @xcite . then we fine-tune our models with the data made available by the shared task. 2 we experimented with different pretrained multilingual transformers as the backbones of our models, and we developed new explainability methods to interpret them. we describe our systems and their training strategies in section 3. overall, our main contributions are: our submitted systems achieve the best multilingual results on all tracks by a considerable margin: for sentence-level da our system achieved a 0.572 spearman correlation (+7% than the second best system); for word-level our system achieved a 0.341 mcc score (+2.4% than the second best system); and for explainable qe our system achieved 0.486 r@k score (+10% than the second best system). the official results for all lps are presented in transformers. the multi-head attention mechanism is the key component in transformers, being responsible for contextualizing the information within and across input sentences @xcite . concretely, given as input a matrix @xmath0, and matrices k, @xmath1, the scaled dot-product attention at a single head is computed as: the π transformation maps rows to distributions, with softmax being the most common choice, π(z) @xmath2. multi-head attention is computed by evoking eq. 1 in parallel for each head h: first piece select. ## implemented systems the overall architecture of our models is shown in figure 1 . the machine translated sentence @xmath3, ..., t n ⟩ and its source sentence counterpart @xmath4, ..., s m ⟩ are concatenated and passed as input to the encoder, which produces d-dimensional hidden state vectors h 0 , ..., h l for each layer 0 ≤ ℓ ≤ l, where h @xmath5, where ℓ = 0 corresponds to the embedding layer. next, all hidden states are fed to a scalar mix module @xcite ) that learns a weighted sum of the hidden states of each layer of the encoder, producing a new sequence of aggregated hidden states h mix as follows: for sentence-level models, the hidden state of the first token (<@xmath6,0 ∈ r d , which, in turn, is passed to a 2-layered feed-forward module in order to get a sentence score prediction ŷ ∈ r. for word-level models, we first retrieve the hidden state vectors associated with the first word piece of each machine translated token, and then pass them to a linear projection to get word-level predictions ŷi ∈ {ok, bad}, ∀ 1≤@xmath7. moreover, attention matrices a 1,1 , ..., a l,h for all layers and heads are also recovered as a by-product of the forward propagation. pretraining on metrics data. every year, the wmt news translation shared task organizers collect human judgments in the form of das. the collective corpora of 2017, 2018, and 2019 contain 24 lps and a total of 657k samples with source, target, reference, and da score. we follow our experiments from last year @xcite and start by pretraining our qe models on this data using the learning objective proposed by unite @xcite , which incorporates reference translations into training and thus acts as data augmentation. ## experimental results as we have seen in section 3, for our experiments we split the provided development sets into two equal size halves creating a new internal devset and an internal testset. the resulting sets contain ≈ 500 segments per language pair for both da and mqm, word and sentence-level. as for baselines we used our submitted systems from previous shared tasks: for task 1 we used the m1m-adapt @xcite , and for task 2 we used the attn × norm explainer @xcite . the official results for task 1 and task 2 are shown in ## official results we present the official results of our submissions alongside the results from other competitors in section b for all three tasks. for sentence-level, our submissions achieved the best results for 6/9 lps. for word-level, we obtained the best results for 5/9 lps. for the explainable qe track, we obtained the best results for all but two lps (km-en and ps-en). although the critical error detection task had no other competitor for the constrained setting, our submission vastly surpassed the organizers' baseline. we also obtained the best results for the multilingual settings (including and excluding en-yo) for all tasks. finally, when averaging the results for all lps, our submissions place on top for all tasks. ## conclusions and future work we presented the joint contribution of ist and unbabel to the wmt 2022 qe shared task. we found that incorporating references during pretraining improves performance across several lps on downstream tasks, and that jointly training with sentence and word-level objectives yields a further boost. for task 1, our final submissions were ensembles of models finetuned with different pretrained language models as encoders, boosting the results when compared to the previous year submission. for task 2, we take inspiration on the literature of explainability and propose to use gradient information in tandem with attention weights, and to further refine the impact of attention heads towards the prediction via the head mix component. besides leading to better explainability performance for some lps, this strategy is potentially useful to identify good attention heads at inference time for zero-shot lps, and deserves more investigation. overall, our submissions achieved the best results for all tasks (including task 3) for almost all lps by a considerable margin. one of the challenges of leveraging big ensembles is the burdensome weight of parameters and inference time. for future work we will extend our recent work, cometinho @xcite and explore how to effectively distill large ensembles into small and more practical qe systems.