COMET : A Neural Framework for MT Evaluation (2020)

Abstract: We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metric. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.

## introduction historically, metrics for evaluating the quality of machine translation (mt) have relied on assessing the similarity between an mt-generated hypothesis and a human-generated reference translation in the target language. traditional metrics have focused on basic, lexical-level features such as counting the number of matching n-grams between the mt hypothesis and the reference translation. metrics such as bleu @xcite and meteor @xcite remain popular as a means of evaluating mt systems due to their light-weight and fast computation. modern neural approaches to mt result in much higher quality of translation that often deviates from monotonic lexical transfer between languages. for this reason, it has become increasingly evident that we can no longer rely on metrics such as bleu to provide an accurate estimate of the quality of mt @xcite . while an increased research interest in neural methods for training mt models and systems has resulted in a recent, dramatic improvement in mt quality, mt evaluation has fallen behind. the mt research community still relies largely on outdated metrics and no new, widely-adopted standard has emerged. in 2019, the wmt news translation shared task received a total of 153 mt system submissions @xcite . the metrics shared task of the same year saw only 24 submissions, almost half of which were entrants to the quality estimation shared task, adapted as metrics @xcite . the findings of the above-mentioned task highlight two major challenges to mt evaluation which we seek to address herein @xcite . namely, that current metrics struggle to accurately correlate with human judgement at segment level and fail to adequately differentiate the highest performing mt systems. in this paper, we present comet 1 , a pytorchbased framework for training highly multilingual and adaptable mt evaluation models that can function as metrics. our framework takes advantage of recent breakthroughs in cross-lingual language modeling @xcite @xcite to generate prediction estimates of human judgments such as direct assessments (da) @xcite , human-mediated translation edit rate (hter) @xcite and metrics compliant with the multidimensional quality metric framework @xcite . inspired by recent work on quality estimation (qe) that demonstrated that it is possible to achieve high levels of correlation with human judgements even without a reference translation @xcite , we propose a novel approach for incorporat-1 crosslingual optimized metric for evaluation of translation. to illustrate the effectiveness and flexibility of the comet framework, we train three models that estimate different types of human judgements and show promising progress towards both better correlation at segment level and robustness to highquality mt. we will release both the comet framework and the trained mt evaluation models described in this paper to the research community upon publication. ## model architectures human judgements of mt quality usually come in the form of segment-level scores, such as da, mqm and hter. for da, it is common practice to convert scores into relative rankings (darr) when the number of annotations per segment is limited @xcite @xcite . this means that, for two mt hypotheses h i and h j of the same source s, if the da score assigned to h i is higher than the score assigned to h j , h i is regarded as a "better" hypothesis. 2 to encompass these differences, our framework supports two distinct architectures: the estimator model and the translation ranking model. the fundamental difference between them is the training objective. while the estimator is trained to regress directly on a quality score, the translation ranking model is trained to minimize the distance between a "better" hypothesis and both its corresponding reference and its original source. both models are composed of a cross-lingual encoder and a pooling layer. ## experiments we train two versions of the estimator model described in section 2.3: one that regresses on hter (comet-hter) trained with the qt21 corpus, and another that regresses on our proprietary implementation of mqm (comet-mqm) trained with our internal mqm corpus. for the translation ranking model, described in section 2.4, we train with the wmt darr corpus from 2017 and 2018 (comet-rank). in this section, we introduce the training setup for these models and corresponding evaluation setup. ## reproducibility we will release both the code-base of the comet framework and the trained mt evaluation models described in this paper to the research community upon publication, along with the detailed scripts required in order to run all reported baselines. 6 all the models reported in this paper were trained on a single tesla t4 (16gb) gpu. moreover, our framework builds on top of pytorch lightning (falcon, 2019), a lightweight pytorch wrapper, that was created for maximal flexibility and reproducibility. ## related work classic mt evaluation metrics are commonly characterized as n-gram matching metrics because, using hand-crafted features, they estimate mt quality by counting the number and fraction of ngrams that appear simultaneous in a candidate translation hypothesis and one or more humanreferences. metrics such as bleu @xcite , meteor @xcite , and chrf @xcite have been widely studied and improved @xcite @xcite , but, by design, they usually fail to recognize and capture semantic similarity beyond the lexical level. in recent years, word embeddings @xcite @xcite have emerged as a commonly used alternative to n-gram matching for capturing word semantics similarity. embeddingbased metrics like meteor-vector @xcite , bleu2vec @xcite , yisi-1 @xcite , moverscore @xcite , and bertscore @xcite create soft-alignments between reference and hypothesis 6 these will be hosted at: https://github.com/ unbabel/comet in an embedding space and then compute a score that reflects the semantic similarity between those segments. however, human judgements such as da and mqm, capture much more than just semantic similarity, resulting in a correlation upperbound between human judgements and the scores produced by such metrics. learnable metrics @xcite @xcite attempt to directly optimize the correlation with human judgments, and have recently shown promising results. bleurt @xcite , a learnable metric based on bert @xcite , claims state-of-the-art performance for the last 3 years of the wmt metrics shared task. because bleurt builds on top of english-bert @xcite , it can only be used when english is the target language which limits its applicability. also, to the best of our knowledge, all the previously proposed learnable metrics have focused on optimizing da which, due to a scarcity of annotators, can prove inherently noisy @xcite . reference-less mt evaluation, also known as quality estimation (qe), has historically often regressed on hter for segment-level evaluation @xcite @xcite @xcite . more recently, mqm has been used for document-level evaluation @xcite . by leveraging highly multilingual pretrained encoders such as multilingual bert @xcite and xlm (conneau and lample, 2019), qe systems have been showing auspicious correlations with human judgements @xcite . concurrently, the openkiwi framework @xcite has made it easier for researchers to push the field forward and build stronger qe models. ## conclusions and future work in this paper we present comet, a novel neural framework for training mt evaluation models that can serve as automatic metrics and easily be adapted and optimized to different types of human judgements of mt quality. to showcase the effectiveness of our framework, we sought to address the challenges reported in the 2019 wmt metrics shared task @xcite . we trained three distinct models which achieve new state-of-the-art results for segment-level correlation with human judgments, and show promising ability to better differentiate high-performing systems. one of the challenges of leveraging the power of pretrained models is the burdensome weight of parameters and inference time. a primary avenue for future work on comet will look at the impact of more compact solutions such as distilbert @xcite . additionally, whilst we outline the potential importance of the source text above, we note that our comet-rank model weighs source and reference differently during inference but equally in its training loss function. future work will investigate the optimality of this formulation and further examine the interdependence of the different inputs.