MS - COMET : More and Better Human Judgements Improve Metric Performance (2022)

Abstract: We develop two new metrics that build on top of the COMET architecture. The main contribution is collecting a ten-times larger corpus of human judgements than COMET and investigating how to filter out problematic human judgements. We propose filtering human judgements where human reference is statistically worse than machine translation. Furthermore, we average scores of all equal segments evaluated multiple times. The results comparing automatic metrics on source-based DA and MQM-style human judgement show state-of-the-art performance on a system-level pair-wise system ranking. We release both of our metrics for public use.

## introduction automatic metrics for machine translation (mt) evaluation are commonly used as the primary tool for comparing the translation quality of mt systems, often without evaluating systems with the human judgement that can be expensive and timeconsuming @xcite . therefore, studying and developing metrics that correlate well with human judgement is critical. there is an increasing effort in the evaluation of automatic mt metrics, leading with the annual evaluation of metrics at the wmt conference @xcite @xcite . most research has focused on comparing segment-level or system-level correlations between absolute metric scores and human judgements. however, @xcite emphasize that this scenario is not identical to the everyday use of metrics, where instead, researchers and practitioners use automatic scores to compare pairs of systems. for example, when claiming a new stateof-the-art, evaluating different model architectures, and deciding whether to publish results or deploy new production systems. in this work, we focus on training automatic metric based on comet architecture @xcite utilizing a large internal trainset of human segment-level judgements. additionally, we evaluate the metrics in a pair-wise system-level evaluation against human judgement. we develop two metrics: ms-comet intended for reference-based evaluating systems, while ms-comet-qe is designed for quality estimation or source-based evaluation. we use the suffix "-22" to differentiate the models from potential future releases. ## related work there are two main categories of automatic mt metrics: (1) string-based metrics and (2) metrics using pretrained models. the former compares the coverage of various substrings between the humangenerated reference and mt translations, this group includes metrics such as chrf @xcite , bleu @xcite , or ter @xcite . string-based methods largely depend on the quality of reference translations. however, their advantage is that their performance is predictable as it can easily diagnose which substrings affect the score the most. the latter category of pretrained methods consists of metrics that usually use pretrained models to evaluate the quality of mt translations given the source sentence, the human reference, or both. evaluation metrics from this category includes comet @xcite , bleurt @xcite , or bertscore @xcite . they are not strictly dependent on the reference quality (for example, they can better evaluate synonyms or paraphrases), and many studies @xcite @xcite showed their superiority over string-based metrics. on the other hand, their performance is influenced by the data on which they have been trained, which may introduce bias, and the pretrained models present a black-box problem where it is challenging to diagnose potential unexpected behavior of the metric. a separate category of automatic metrics is whether they need a human reference for evaluation. automatic metrics that calculate scores without the need for reference (quality estimation) open the possibility of evaluating monolingual testsets that can be tailored for a specific domain without the need to build expensive human references. we build our metric with the architecture of comet @xcite . 2 it uses the estimator model which uses pretrained language models xlm-roberta to encode source, mt hypothesis and reference in the same cross-lingual space. the model is then fine-tuned on human judgement data. we use the identical hyper-parameters as comet. ## human judgement trainset for training our models, we use a mix of public and internal data that we further denoise by filtering out potentially problematic human judgements. we use the same human judgments data used to train the comet model, i.e. wmt 2017-2019 @xcite @xcite . to test the quality of metrics, we use wmt 2020 @xcite , wmt 2021 @xcite and mqm 2021 @xcite . furthermore, we submitted our model to wmt metrics shared task 2022. in addition to publicly available data, we use a set of internal data, described in @xcite plus newer data collected over the last year. all our internal data are collected with the use of expert annotators. we use a mix of human judgement methods: source-based direct assessment (srcda) @xcite , contrastive direct assessment (contrda, which asks users to rate pairs of system outputs), and sqm presented at wmt general mt 2022 (which uses labeled scale). all collected labels are on a scale of 0-100, where the interface structure is the main difference for human annotators. we use internal testsets for human judgements that have been translated with a tandem of two professional translators, following findings of freitag et al. (2020) that high-quality reference plays an essential role in automatic evaluation. in contrast to publicly available data that uses only the news domain, we use a mix of fifteen domains (news, conversation, legal, medical, social, e-commerce, tech, finance, and others). the news domain is the largest domain utilizing at least half of human judgements. our collection of human judgement data covers 113 languages in contrast to 13 on which comet is trained. a complete list of all supported languages and counts of human judgement for the largest translation directions are in appendix a. reference-less metric ms-comet-qe is trained using all training data and removing reference translations. additionally, many human judgments are evaluated on data that are missing human reference, which is the reason for having more training data for ms-comet-qe. ## evaluation evaluation of automatic metrics is a challenging task investigated in a yearly wmt metrics shared task @xcite . however, there is no community-agreed testset or evaluation method for comparing with humans that are considered gold standards. there are different dimensions how to evaluate automatic mt metrics. let's summarize the main differing points: the list is incomplete as there are other nuances, such as removing outlier systems, using only statistically significant pairs of systems, underlying quality of human judgement, etc. evaluating all combinations of approaches is not reasonable. therefore we mainly follow the approach defined by @xcite and also used by wmt metrics 2021 @xcite . here is a list of constraints for the evaluation: ## results the results for the pairwise system-level scenario are in lastly, our metrics win in the wmt21-contr testset. this is the only genuinely pairwise testset where annotators saw systems next to each other while evaluating them. although we focus on a system-level evaluation, we evaluate how metrics perform in a segmentlevel setting for completeness. we use the testset wmt21-contr to calculate accuracies in the same fashion as for system-level scenario, but taking pairs of segment annotations instead of systemlevel scores. the segment-level results in ## conclusion we have investigated the training comet model with a larger corpus of human judgements covering multiple domains and 113 languages. we employed several steps of filtering lowquality or repetitive human judgement. with those data, we trained two metrics: ms-comet-22 and ms-comet-qe-22, that outperform other current mt metrics on a pair-wise system-level decision task. we release the metrics for public use.