Unbabel’s Participation in the WMT 20 Metrics Shared Task (2020)

Abstract: We present the contribution of the Unbabel team to the WMT 2020 Shared Task on Metrics. We intend to participate on the segmentlevel, document-level and system-level tracks on all language pairs, as well as the “QE as a Metric” track. Accordingly, we illustrate results of our models in these tracks with reference to test sets from the previous year. Our submissions build upon the recently proposed COMET framework: we train several estimator models to regress on different humangenerated quality scores and a novel ranking model trained on relative ranks obtained from Direct Assessments. We also propose a simple technique for converting segment-level predictions into a document-level score. Overall, our systems achieve strong results for all language pairs on previous test sets and in many cases set a new state-of-the-art.

## introduction in this paper we describe our submission to the wmt20 metrics shared task. our work is based on the comet 1 framework, as presented in @xcite , and extended here to evaluation of mt output at segment, document and system-level, forming the basis of our submissions to the corresponding task tracks. recently, automatic evaluation of mt has followed most other sub-fields in nlp with a notable interest in leveraging the power of large, pre-trained language models. metrics such as bert regressor @xcite , bertscore @xcite , bleurt @xcite and our more recent comet @xcite , all build upon developments in language modelling to generate automatic metrics with high correlation with human judgement. our 1 crosslingual optimized metric for evaluation of translation hosted at: https://github.com/ unbabel/comet mt evaluation models follow a similar strategy, specifically utilizing the most recent iterations of the xlm-roberta model presented in @xcite . the uniqueness of our approach comes from our inclusion of the source text as input which was demonstrated in @xcite and @xcite to be beneficial to the model. in our contribution to the shared task, we demonstrate methods of further exploiting information in the source text as well as a technique to fully harness the power of pre-trained language models to further improve the prediction accuracy of our evaluation framework when more than one reference translation is available. for the shared task, we utilize two primary types of models built using the comet framework, namely; the estimator models, which regress directly on human scores of mt quality such as direct assessment; and the comet-rank (base) model used to rank mt outputs and systems. in addition to the models themselves, we also make the following research contributions: we demonstrate that our comet framework trained models achieve state-of-the-art results or are competitive on all settings introduced in the wmt19 metrics shared task, outperforming, in some cases, more recently proposed metrics such as bertscore @xcite , bleurt @xcite and prism @xcite . ## the comet framework as outlined in @xcite , the comet framework allows for training of specialized evaluation metrics that correlate well with different types of human-generated quality scores. the general structure of the framework consists of a cross-lingual encoder that produces a series of token-level vector embeddings for source, hypothesis and reference inputs, a pooling layer which converts the various token-level representations into segment-level vectors for each input, and a predictive neural network that generates a quality score. the latter model can either be trained to regress directly on a score to produce predictions of segment-level quality, or can be trained as a ranker to differentiate mt systems. in our contribution to the shared task, we introduce two varieties of models built on the comet framework that are extensions of the models evaluated in @xcite . ## corpora below we provide an outline of the various datasets used to train our models: ## segment-level task at segment-level, we take each of our estimator models trained to predict mqm, hter and da and predict segment-level scores on the darr data from wmt19. we then generate pairwise rankings based on these predicted scores. for each language pair we apply the formulation of kendall's tau (τ ) from the shared task @xcite as follows: concordant here being the number of times a metric assigns a higher score to the "better" hypothesis h + and discordant, the number of times a metric assigns a higher score to the "worse" hypothesis h -, or that the evaluation was otherwise equal. ## document-level task in the wmt2019 news translation the organizers introduced a document-level translation task @xcite for en-de and en-cs. this means that for those language pairs we are able to obtain document-level direct assessments. we can compute a score taking into account an entire document and correlate it with the human evaluation also carried out at document-level. for our document-level submission we propose the generation of a document-level score as a weighted average of the predicted scores for each segment composing that document (hereinafter called micro-average score), where the same is weighted by segment length. to calculate this score at inference time we pass the entire document (divided into segments) through the model as a single batch. this has the added effect of reducing inference time. ## quality estimation as a metric given the clear parallels between the comet framework and modern approaches to quality estimation such as @xcite , we used our framework to participate in the "qe as a metric" track of the shared task by removing the reference at input and proportionately reducing the dimensions of the feed-forward network to accommodate the reduced input. ## multi-reference handling in this year's shared task we are provided with a second human-generated reference for germanto-english (de-en), russian-english (ru-en) and chinese-to-english (zh-en). given that our base framework currently supports the input of only one single reference, we introduce a method of leveraging information from a second reference at inference time. during standard training of our models, we input source, hypothesis and reference in that order, resulting in a concatenation of embeddings as detailed further in @xcite . during training, with a probability of @xmath0.5 we switch the positions of source and reference, such that the system receives the reference as the source and vice versa. this has two primary effects on our model. firstly, during fine-tuning of the underlying language model, the source embeddings are aligned with the target language embedding space resulting in more useful source embeddings. secondly, it forces the model to treat source and reference as in-terchangeable inputs, allowing it to handle switching of inputs at inference time without excessively hindering the model's predictive ability. finally, at inference time we embed source s, hypothesis h, reference r and the alternative reference r. these embeddings are then passed to the feed-forward neural network in the following six permutations: six passes through the feed-forward gives us six predictions. final, aggregated scores are achieved by taking the mean of the six predictions and multiplying it by 1 minus the standard deviation (σ). the intuition being that 1 -σ gives something of an idea of confidence of the model at the segmentlevel and that scaling the mean prediction to penalize lower confidence might align better with human judgement. ## experimental results below we present results of our various comet models on wmt19 evaluation sets as described ## conclusions in this paper we present comet, unbabel's contribution to the wmt 2020 metrics shared task. we leverage the framework outlined in @xcite to demonstrate state-of-the-art or otherwise competitive levels of correlation with human judgements in all tasks and introduce a novel method of making optimal use of alternative references and demonstrate that the quality of the reference used is relevant to the success of our framework. further investigation of the latter, in particular how to better leverage different kinds of references, represent an interesting direction for future work.