IST -Unbabel 2021 Submission for the Explainable Quality Estimation Shared Task (2021)

Abstract: We present the joint contribution of Instituto Superior Técnico (IST) and Unbabel to the Explainable Quality Estimation (QE) shared task, where systems were submitted to two tracks: constrained (without word-level supervision) and unconstrained (with word-level supervision). For the constrained track, we experimented with several explainability methods to extract the relevance of input tokens from sentence-level QE models built on top of multilingual pre-trained transformers. Among the different tested methods, composing explanations in the form of attention weights scaled by the norm of value vectors yielded the best results. When word-level labels are used during training, our best results were obtained by using word-level predicted probabilities. We further improve the performance of our methods on the two tracks by ensembling explanation scores extracted from models trained with different pre-trained transformers, achieving strong results for in-domain and zero-shot language pairs.

## introduction quality estimation (qe) aims at assessing the quality of a translation system without relying on reference translations @xcite . this paper describes the joint contribution of instituto superior técnico (ist) and unbabel to the explainable quality estimation shared task @xcite . the goal of the shared task is to identify translation errors without direct word-level supervision (constrained track) or with access to word-level labels (unconstrained track). recent advances in qe have led to consistent improvements at predicting quality assessments such as direct assessments (das, @xcite . traditional qe systems had to predict human translation error rate (hter, @xcite ), yet with the advent of neural machine translation, we observed a shift from fluency into ade-quacy errors @xcite . for that reason, das started getting used as the groundtruth score for assessing the quality of translations @xcite . however, with das we lose the ability to generate word-level supervision, impacting the interpretability of sentence-level predictions in terms of lower granularity elements such as word-level translation errors. at the same time, state-of-the-art qe systems such as openkiwi @xcite and tran-squest @xcite build on top of multilingual pre-trained models such as bert @xcite and xlm-roberta @xcite , which are largely responsible for the performance boost we have observed in the last two editions of the wmt qe shared task @xcite . due to the usage of such overparametrized black-box models, this performance boost also comes at the cost of efficiency and interpretability. research in explainable nlp uncovered several strategies to interpret models' decisions, either in a post-hoc manner by querying a trained model for extracting perturbation or gradient measures @xcite , or by building models that are inherently interpretable @xcite . recent works have also put transformers under the lens of explainability, aiming at unraveling interpretable patterns that clarify how decisions emerge from attention heads and across hidden states at each layer @xcite @xcite . in this shared task, we experiment with several of these methods to extract the relevance of input tokens from sentence-level qe models built on top of multilingual pre-trained transformers 1 . for the constrained track, where models are unaware of word-level supervision, our best results were de-rived from attention-based explanations. when we used word-level labels during training, the best results were obtained by using word-level predicted probabilities. furthermore, we were able to push the performance further by ensembling explanations for both tracks. transformers. the multi-head attention mechanism is the bedrock on which transformers are built. they are responsible for contextualizing the information within and across input sentences dynamically @xcite . concretely, given as input a matrix @xmath0, and matrices k, @xmath1, the scaled dot-product attention at a single head is computed as: the π transformation maps rows to distributions, with softmax being the most common choice, π(z) @xmath2. multi-head attention is computed by evoking eq. 1 in parallel for each head h: this way, heads have the capability of learning specialized phenomena. transformers with only encoder-blocks, such as bert @xcite and xlm-roberta @xcite , have only the encoder self-attention, and thus @xmath3. ## constrained track the goal of the constrained track is to identify machine translation errors without explicit word-level annotation. more precisely, it aims at performing word-level quality estimation by casting the task as a prediction explainability problem. in the context of qe, explanations can be seen as highlights, representing the relevance of input words w.r.t. the model's prediction via continuous scores. we next describe the datasets, models, and explainability methods that we used for this track. ## unconstrained track in this track, we opted to use word-level annotation by incorporating a word-level loss to our previous models. to do this, we apply a map from word pieces to tokens after the scalar mix layer and pass the hidden vectors of each token through a feed-forward layer with a sigmoid activation to predict scores ŷi ∈ [0, 1]. we weight the word-level loss by λ and sum it with the sentencelevel loss. as baseline, we train a xlm-r large model using openkiwi with the default hyperparameters. for all word-level models, we train with λ ∈ {10 3 , 10 4 , 10 5 } and save the checkpoint with the best performance on the validation set. results. ## experimental results although we can regard the extracted explanations as errors in the translation output, an analogous evaluation of word-level qe is not straightforward since the standard metrics require binary labels rather than continuous scores. therefore, the explanations are evaluated against the ground-truth word-level labels in terms of the area under the curve (auc), average precision (ap), and recall at top-k (r@k) metrics only on the subset of translations that contain errors. furthermore, since all of our models use subword tokenization, to get explanations for an entire word, we tried aggregating the scores of its word pieces by taking the sum, mean, or max, and we found that taking the sum performs better overall. ## official results the official results of the shared task are shown in overall, results for the unconstrained track are superior to those obtained in the constrained track. however, the opposite is true for de-zh, suggesting that extracting rationales from a sentence-level qe model is a promising weak-supervised strategy to identify translation errors. ## conclusion final remarks. we have shown that the multihead mechanism-the bedrock on which transformers are built-is able to learn the importance of tokens associated with bad tags. furthermore, composing explanations in the form of attention probabilities scaled by the norm of value vectors leads to further improvements @xcite . ensembling these explanations yields the best results overall for all tested metrics on all lps, including zero-shot ones. future work. transformers are composed of many parameters across a vast amount of heads and layers. strategies that explore how explanations are formed as we move to upper layers are promising, such as computing attention flows and differentiable binary masks per layer @xcite . moreover, as shown in figure 4 , we noticed that our best explainers suffer on sentences with higher quality, likely due to the low number of translation errors for those sentences. a simple way to circumvent this problem is to force the explainer to "focus" on words associated with lower scores (or to the bad class in a classification setting). thus, strategies such as framing the prediction of da scores as a classification problem or inducing class-wise rationalizers @xcite ) can be helpful. this shared task focused only on the intersection between explainability and quality estimation, yet for future work we plan to apply explainability methods to recent mt metrics such as comet @xcite and bleurt @xcite .