The E val4 NLP Shared Task on Explainable Quality Estimation: Overview and Results (2021)

Abstract: In this paper, we introduce the Eval4NLP-2021 shared task on explainable quality estimation. Given a source-translation pair, this shared task requires not only to provide a sentence-level score indicating the overall quality of the translation, but also to explain this score by identifying the words that negatively impact translation quality. We present the data, annotation guidelines and evaluation setup of the shared task, describe the six participating systems, and analyze the results. To the best of our knowledge, this is the first shared task on explainable NLP evaluation metrics. Datasets and results are available at https://github.com/eval4nlp/SharedTask2021 .

## introduction recent natural language processing (nlp) systems based on pre-trained representations from transformer language models, such as bert @xcite and xlm-roberta @xcite , have achieved outstanding results in a variety of tasks. this boost in performance, however, comes at the cost of efficiency and interpretability. interpretability is a major concern in modern artificial intelligence (ai) and nlp research @xcite , as black-box models undermine users' trust in new technologies @xcite . in the eval4nlp 2021 shared task, we focus on evaluating machine translation (mt) as an example of this problem. specifically, we look at the task of quality estimation (qe), where the aim is to predict the quality of mt output at inference time without access to reference translations @xcite . 1 translation quality can be assessed at different levels of granularity: sentencelevel, i.e. predicting the overall quality of translated sentences, and word-level, i.e. highlighting specific errors in the mt output. those have traditionally been treated as two separate tasks, each one requiring dedicated training data. in this shared task, we propose to address wordlevel translation error identification as an explainability task. 2 explainability is a broad area aimed at explaining predictions of machine learning models. rationale extraction methods achieve this by selecting a portion of the input that justifies model output for a given data point @xcite . a natural way to explain sentencelevel quality assessment is to identify translation errors. hence, we frame error identification as a task of providing explanations for the predictions of sentence-level qe models. we claim that this task represents a challenging new benchmark for testing explainability for nlp and provides a new way of addressing word-level qe. on the one hand, qe is different from other explainable nlp tasks with existing datasets @xcite in various important aspects. first, it is a regression task, as opposed to binary or multiclass text classification explored in previous work. second, it is a multilingual task where the output score captures the relationship between source and target sentences. finally, qe is fundamentally different from e.g. text classification, where clues are typically separate words or phrases @xcite ) that can often be considered independently of the rest of the text. by contrast, translation errors can only be identified given the context of the source and target sentences. thus, this shared task provides a new benchmark for testing explainability methods in nlp. on the other hand, treating word-level qe as an explainability problem offers some advantages compared to the current approaches. first, we can potentially avoid the need for supervised data at word level. second, gold standard test sets can be made less expensive and more reliable. as we will show in section 2, rationalized sentence-level evaluation can be a middle ground between relatively cheap but noisy annotations derived from post-editing @xcite and very informative but expensive explicit error annotation based on error taxonomies, such as the multidimensional quality metrics (mqm) framework @xcite . for this shared task, we build a new test set with manually annotated explanations for sentence-level quality ratings. to the best of our knowledge, this is the first mt evaluation dataset annotated with human rationales. the main objective of the shared task is threefold. first, it aims to explore the plausibility of explainable evaluation metrics @xcite , by proposing a test set with manually annotated rationales. it helps the community better understand how similar the generated explanations are to the human explanations. second, the shared task encourages research on unsupervised or semisupervised methods for error identification, so as to reduce the cost on word-level mt error annotation. last but not least, the shared task sheds light on how current nlp evaluation systems arrive at their predictions and to what extent this process is aligned with human reasoning. ## data for this shared task, we collected a new test set with (i) manual assessment of translation quality at sentence level and (ii) word-level rationales that explain the sentence-level scores (section 2.1). for training and development purposes, the participants were advised to use existing resources, which are briefly discussed in section 2.2. ## task and evaluation the task consisted of building a qe system that (i) predicts the quality score for an input pair of source text and mt hypothesis, (ii) provides word-level evidence for its predictions. an example of the test data used for evaluation is shown in explanations can be obtained either by building inherently interpretable models @xcite or by using post-hoc explanation methods which extract explanations from an existing model @xcite @xcite , for example by analysing the values of the gradient on each input feature. in this shared task, we provide both sentence-level training data and strong sentencelevel models (see the transquest-lime baseline in section 4), and thus encourage the participants to either train their own inherently interpretable models or use post-hoc techniques on top of our existing sentence-level models. we accommodate the evaluation scheme to be suitable both for approaches that return continuous scores, and for supervised approaches that can return binary scores. namely, we use evaluation metrics based on class probabilities that have been previously adapted for assessing the plausibility of rationale extraction methods @xcite . since explainability methods typically proceed on instance-by-instance basis, and the scores produced for different instances are not necessarily comparable, we compute the evaluation metrics for each instance separately and average the results across all instances in the test set. following @xcite , we define the following evaluation metrics to assess the performance of the submissions to the shared task at the word-level: auc score for each instance, we compute the area under the receiver operating characteristic curve (auc score) to compare the continuous attribution scores against binary gold labels. average precision auc scores can be overly optimistic for imbalanced data. therefore, we also use average precision (ap). ap summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight @xcite . recall at top-k in addition, we report the recall-at-top-k metric commonly used in information retrieval. applied to our setting, this metric computes the proportion of words with the team id participating team nict-kyoto national institute of information and communications technology rubino et al. (2021) ist-unbabel ist/university of lisbon & unbabel treviso et al. (2021) clip-umd department of computer science, university of maryland kabir and carpuat (2021) gringham technical university of darmstadt leiter (2021) heytuda technical university of darmstadt eksi et al. (2021) cuni-prague charles university polák et al. (2021) the shared task used codalab as the submission platform. ## baseline systems random baseline is built by sampling scores uniformly at random from a continuous [0..1) range for each source and target token in a given sentence pair as well as for the sentence-level qe score. transquest-lime uses transquest qe models described in @xcite to produce sentence-level scores. transquest follows the current standard practice of building task-specific nlp models by fine-tuning pre-trained multilingual language models, such as xlm-roberta, on taskspecific data. for ro-en and et-en, the ro-en and et-en transquest models are used, whereas for the zero-shot language pairs we use the multilingual variant of transquest, which was trained on a concatenation of mlqe-pe data. the post-hoc lime explanation method @xcite is then applied to generate relevance scores for the source and target words. lime is a simplification-based explanation technique, which fits a linear model in the vicinity of each test instance, to approximate the decision boundary of the complex model. since in our sentence-level gold standard higher scores mean better quality, we invert lime explanations so that higher values correspond to errors. xmover-shap uses the reference-free metric xmoverscore @xcite to rate translations and uses the (likewise post-hoc) shap explainer (lundberg and @xcite to explain the ratings. in particular, given a source-translation pair, xmoverscore provides a real number to indicate the quality of the translation, in terms of its semantic overlapping with the source sentence, using re-mapped multilingual bert embeddings and a target-side language model. 9 to explain the contribution of each word in the rating, shap creates perturbations of the source/translation sentence by masking out some words and estimates the average marginal contribution of each word across all possible perturbations. the source code for all the baseline systems is available at https://github.com/eval4nlp/ sharedtask2021/tree/main/baselines . ## participants for this first edition of the shared task, we had a total of 6 participating teams listed in nict-kyoto use synthetic data to fine-tune the xlm-roberta language model for the qe task. to produce synthetic sentence-level scores, they translate publicly available parallel corpora using sota neural mt systems and compute three referencebased metrics: chrf @xcite , ter @xcite and bleu @xcite . to simulate word-level annotation, they derive wordlevel labels from the alignment between the mt outputs and human reference translations. the qe model is then jointly trained to predict the scores from different metrics as well as word-level tags. a metric embedding component is proposed where each metric is represented with a set of learnable parameters. an attention mechanism between the metric embeddings and the input representations is employed to obtain word-level scores as explanations for the sentence-level predictions. ist-unbabel participated in the constrained and unconstrained tracks of the shared task. for the constrained track ("ist-unbabel" in for the unconstrained track ("ist-unbabel*" in heytuda use the transquest qe models @xcite for sentence-level prediction and a set of explainability techniques to estimate the relevance of each source and target word. specifically, they explore three perturbation-based methods: lime, shap, and occlusion @xcite , as well as three gradient-based methods: deeplift @xcite , layer gradient x activation @xcite and integrated gradients @xcite . they further use an unsupervised ensembling method to combine the different explainability approaches. gringham use the reference-free metrics xbertscore (i.e., bertscore @xcite with cross-lingual embeddings) and xmoverscore and make them inherently interpretable by considering the token alignments produced by the models. the intuition is that words that are not well-aligned are most likely erroneous. specifically, they explore xbertscore and xmoverscore as sentence-level models and use the corresponding similarity (or distance) matrices to produce token-level scores. clip-umd propose an ensemble of two approaches: (1) the lime explanation technique applied to the transquest sentence-level model; (2) divergent mbert @xcite , which is a bert-based model that can detect crosslingual semantic divergences. divergent mbert is trained using synthetic data where semantic divergences are introduced automatically following a set of pre-defined perturbations. to produce a combination of the two methods, the predictions from each approach are averaged. cuni-prague participated in the unconstrained track. they fine-tune the xlm-r model for wordlevel and sentence-level qe. to map sentence piece tokenization from xlm-r to moses tokenization, they ignore all sentence piece tokens corresponding to a given moses token except the first one. ## results approaches overall, the submitted approaches vary a lot in the way they addressed the task. the following trends can be identified: ## conclusions in this paper, we presented the findings of the eval4nlp-2021 shared task on explainable quality estimation (qe), where the goal is to not only produce a sentence-level score for an mt output, given a source sentence, but also highlight erroneous words in the target (and source) sentence explaining the score. we detailed the data annotation, involving two novel non-english language pairs, our baselines (post-hoc explanation techniques on top of state-of-the-art qe models), as well as the participants' approaches to the task. these include supervised approaches, training on synthetic data as well as genuine post-hoc and inherent explainability techniques. the scope for future research is huge: for example, we aim to include new language pairs, especially low-resource ones, address explainability for metrics in other nlp tasks, e.g. semantic textual similarity @xcite and summarization @xcite , and identify error categories of highlighted words, ideally in an unsupervised manner.