COMET -22: Unbabel- IST 2022 Submission for the Metrics Shared Task (2022)

Abstract: In this paper, we present the joint contribution of Unbabel and IST to the WMT 2022 Metrics Shared Task. Our primary submission – dubbed COMET-22 – is an ensemble between a COMET estimator model trained with Direct Assessments and a newly proposed multitask model trained to predict sentence-level scores along with OK/BAD word-level tags derived from Multidimensional Quality Metrics error annotations. These models are ensembled together using a hyper-parameter search that weights different features extracted from both evaluation models and combines them into a single score. For the reference-free evaluation, we present CometKiwi. Similarly to our primary submission, CometKiwi is an ensemble between two models. A traditional predictor-estimator model inspired by OpenKiwi and our new multitask model trained on Multidimensional Quality Metrics which can also be used without references. Both our submissions show improved correlations compared to state-of-the-art metrics from last year as well as increased robustness to critical errors.

## introduction automatic metrics for machine translation (mt) are a fundamental component of mt research and development. while human evaluation is still of great importance, automatic metrics allow the rapid evaluation and comparison of mt systems on large collections of text and facilitate expansion to low resource languages and domains. neural fine-tuned metrics in particular, have shown the ability to leverage large multilingual data during training to better compare and assess the quality of stateof-the-art mt models, outperforming traditional lexical-based metrics. hence, our research is targeted to and guided by the advancements in these metrics. this paper presents the joint contribution of unbabel and instituto superior técnico (ist) to the wmt 2022 metrics shared task @xcite . we participated in the segment-level and system-level tracks, as well as the "qe-as-a-metric". similar to our participation last year @xcite , our models are based on the comet framework foot_0 @xcite . our efforts this year built on findings and observations from our participation in the wmt 2021 metrics shared task @xcite to further improve comet for the metrics task and to increase its robustness to translation errors such as deviation from named entities, reverse polarity and negation, deviation in numbers, etc. these types of fine-grained critical errors have been shown to be challenging for state-of-the-art metrics and qe systems @xcite . for that reason we aim to take advantage of finer-grained information, incorporating word-level supervision from multidimensional quality metrics (mqm) annotations when available. this approach is motivated by the observed improvements in performance in wmt 2021 metrics when fine-tuning on mqm data. additionally, the importance of word-level supervision as an auxiliary task was established via our participation in the wmt 2022 quality estimation task @xcite , where we found that we get a boost of performance across language pairs when we combine word-and sentence-level targets @xcite . overall, our main contributions are: our submitted metrics, compared to two of the best submissions from last year, improve by a considerable margin in terms of correlations with mqm (≈ 4% in kendall-tau at segment level and ≈ 6% on system-level accuracy) and in the ability of detecting critical errors (≈ 30% in accuracy on smaug challenge set @xcite , a newly proposed challenge set built to test the robustness of mt metrics to errors in named entities, numbers, meaning, inserted content and missing content.). ## corpora every year, since 2017, the organisers of wmt news translation tasks collect annotations in the form of direct assessments (da) @xcite . recently, @xcite showed that da annotations, collected by non-professional crowd-source workers, are noisy and unfit to measure the quality of high performing mt systems. for high quality mt evaluation the authors suggest the use of mqm annotations performed by professionals; they released annotations for @xmath0. since then, along with the da annotations coming from the news translation task, the metrics task organizers collect additional mqm data for @xmath1, @xmath2, and @xmath3. with that said, to test the performance of our new systems we will use the mqm annotations from 2021 news domain. for training we will use all da ranging from 2017 to 2020 and the remaining mqm annotations. one of the findings from last years shared task is that metrics struggle to accurately penalize translations with errors in reversing negation @xcite . also, @xcite showed that using comet as a utility function for minimum bayes risk decoding is more likely to lead to errors in named entities and numbers when compared to lexical metrics such as chrf. in order to measure progress on capturing those errors we will test our new metrics on smaug @xcite . ## implemented systems our goal this year is to build more robust crosslingual optimized metrics for evaluation of translations by ensembling systems that model different aspects of mt evaluation. for that purpose we used three different systems: a comet estimator @xcite trained on da, a newly proposed sequence tagger, trained with mqm data, that works with and without references, and a cometkiwi @xcite model trained on da. ## experimental results as we have seen in section 2, for our experiments we use wmt 2021 news mqm annotations from last year shared task @xcite for testing our metrics. as for baselines we used lexical metrics such as chrf @xcite and bleu @xcite and three state-ofthe-art metrics: bleurt @xcite , comet-20 @xcite and comet-21 @xcite 3 . ## conclusions we present the joint contribution of unbabel and ist to the wmt 2022 metrics shared task. we propose a new architecture trained in a multitask setting which takes advantage of sentence-level scores along with supervision from mqm annotation spans. inspired by unite, our new model can be used with and without references showing promising results when references are not available. our primary submission ensembles our new model along with the comet estimator architecture showing both higher correlations and improved robustness to phenomena that was deemed challenging in previous shared task editions. finally, our "qe-as-a-metric" submission yet again has shown that reference-free is competitive to reference-based evaluation not only at segmentlevel but also at system-level and in terms of detecting critical errors.