The UMD Submission to the Explainable MT Quality Estimation Shared Task: Combining Explanation Models with Sequence Labeling (2021)

Abstract: This paper describes the UMD submission to the Explainable Quality Estimation Shared Task at the EMNLP 2021 Workshop on “Evaluation & Comparison of NLP Systems”. We participated in the word-level and sentence-level MT Quality Estimation (QE) constrained tasks for all language pairs: Estonian-English, Romanian-English, German-Chinese, and Russian-German. Our approach combines the predictions of a word-level explainer model on top of a sentence-level QE model and a sequence labeler trained on synthetic data. These models are based on pre-trained multilingual language models and do not require any word-level annotations for training, making them well suited to zero-shot settings. Our best-performing system improves over the best baseline across all metrics and language pairs, with an average gain of 0.1 in AUC, Average Precision, and Recall at Top-K score.

## introduction quality estimation (qe) is the task of predicting the quality of the machine translation (mt) output without reference translation. predictions can be done at different levels of granularity, such as sentences or words. the explainable qe shared task @xcite proposes to frame the identification of translation errors as an explainable qe task, where sentence-level quality judgments are explained by highlighting the words responsible for errors in the mt hypothesis. given a source sentence and an mt hypothesis, systems are thus asked to provide word-level judgments of translation quality in addition to sentence-level judgments. our submission builds on state-of-theart sentence-level qe models, monotran-squest @xcite . as suggested by the organizers, we rely on the lime explanation model @xcite to obtain word-level prediction from the monotransquest model's sentence-level score. we hypothesize that synthetic examples of translation errors can help improve word-level predictions. as a result, we combine the predictions of monotransquest-lime with those of the divergent mbert model which addresses the related task of detecting semantic divergences in bitext @xcite . divergent mbert model can detect fine-grained differences in bitext by learning to rank synthetic divergence examples of varying granularity. as a result, our approach does not require any word-level labels at training time. both models are based on multilingual language models and are therefore amenable to zero-shot transfer. our submitted system improves over its components and over the official baseline on all tracks and on all language pairs, based on all evaluation metrics (auc, ap, recall at top-k, and pearson's correlation). compared to the best baseline system for target languages, it improves auc by 0.119 for estonian-english (et-en), 0.068 for romanian-english (ro-en), 0.085 for german-chinese (de-zh), and 0.128 for russian-german (ru-de). similarly, for ap score, it has achieved an improvement of 0.095 for et-en, 0.074 for ro-en, 0.064 for de-zh, and 0.13 for ru-de. for recall at top-k score, it has achieved an improvement of 0.103 for et-en, 0.071 for ro-en, 0.045 for de-zh, and 0.13 for ru-de. for source language word-level scores, it achieves an average gain of 0.18 for auc, 0.071 for ap and 0.12 for recall at top-k score over the average of all languages' baseline scores. finally, for sentence-level scores, it has achieved an improvement of 0.36 for et-en, 0.359 for ro-en, 0.271 for de-zh, and 0.06 for ru-de compared to the average of all baseline models for pearson's correlation. ## approach we first describe the two components of our ensemble and then explain how they are combined. ## datasets in this section, we describe the data used for training, development, and evaluation. ## system configuration we used the same set of system configurations for all the language pairs in our experiment to ensure consistency among all language pairs. monotransquest we have used pre-trained monotransquestmodel on the huggingface transformers library @xcite . we used those pre-trained monotranquest models foot_1 . we have not changed any hyperparameter from those models. divergent mbert for training divergent mbert we have used a batch size of 16, adam optimizer with learning rate 2e -5 and a linear rate warmup. the model was trained with only training data. the model was trained for five epochs. we have varied the hyperparameter settings: epoch was varied from 3 to 15 epochs, the margin was varied from 5 to 10, the alpha value was varied from 0.2 to 1. ## evaluation metrics this section describes the official evaluation metrics for the shared task. • auc score: in this shared task, the auc score between the model predicted output and gold explanation mt score is computed using sklearn foot_2 . given a test set of n sentences: here, w n is a vector representing binary gold word-level labels for each sentence n in the test set and a x t n is the vector for the model predicted word-level score for the target words x t in each target sentence in test set with length t . equation 1 computes the auc score to compare the model predicted word-level scores a against binary gold labels @xcite . here, au c n is the area under the curve generated by plotting the true positive against false positive of the word-level scores of the n th sentence at different thresholds. auc ranges in value from 0 to 1. a model whose predictions are 100% wrong has an auc of 0.0; one whose predictions are 100% correct has an auc of 1.0. here, k is the number of errors in the sentence and @xmath0. the final score is the average over test instances, and ranges from 0 to 1. ## results we describe the performance of our models on the development and test sets using the official shared task metrics. ## conclusion we described the university of maryland's contribution to the eval4nlp 2021 shared task on quality estimation. our submission was based on ensembling existing models: (1) the state-of-theart framework monotransquest model followed by the lime explanation model, and (2) an mbert model trained to detect cross-lingual semantic divergences. we show that averaging the prediction of these models outperforms all the baselines and their individual predictions, even though none of the ensemble components are trained with wordlevel supervision. overall, our approach shows the benefits of leveraging pre-trained multilingual lms to port to multiple language pairs, including in zero-shot settings: the divergent mbert component is even trained on a language pair that is not used for any of the test tasks. in the future, training divergent mbert with other language pairs can lead to more promising results. this work also shows the complementarity of explanation models and of sequence labelers trained on synthetic data for word-level predictions. in future work, controlled comparison of these approaches on the same languages and data conditions can lead to further insights on their respective strengths and weaknesses.