Are References Really Needed? Unbabel- IST 2021 Submission for the Metrics Shared Task (2021)

Abstract: In this paper, we present the joint contribution of Unbabel and IST to the WMT 2021 Metrics Shared Task. With this year’s focus on Multidimensional Quality Metric (MQM) as the ground-truth human assessment, our aim was to steer COMET towards higher correlations with MQM. We do so by first pre-training on Direct Assessments and then fine-tuning on z-normalized MQM scores. In our experiments we also show that reference-free COMET models are becoming competitive with reference-based models, even outperforming the best COMET model from 2020 on this year’s development data. Additionally, we present COMETinho, a lightweight COMET model that is 19x faster on CPU than the original model, while also achieving state-of-the-art correlations with MQM. Finally, in the “QE as a metric” track, we also participated with a QE model trained using the OpenKiwi framework leveraging MQM scores and word-level annotations.

## introduction in this paper, we present the joint contribution of unbabel and ist to the wmt 2021 shared task on metrics. we participated in the segment-level and system-level tracks, as well as the "qe as a metric" task. similar to our participation last year @xcite , most of the models are based on the comet framework 1 @xcite . in last year's shared task @xcite , comet along with other trainable metrics such as prism @xcite and bleurt @xcite showed superior correlations with the direct assessments (da) collected for the news translation shared task. this 1 crosslingual optimized metric for evaluation of translation hosted at: https://github.com/ unbabel/comet year, we build on top of the models used last year to take into account that human assessments will be carried out using variants of the multidimensional quality metric (mqm) @xcite framework and no longer based on da @xcite . for this reason, we extended our training dataset to include da evaluations from wmt ranging 2015 to 2020, with the exception of en-de and zh-en for which we do not include the 2020 data given that the same is included in the mqm development data @xcite . finally, we fine-tuned these new models on the znormalized mqm scores provided for this year's shared task. one of the remaining redeeming qualities of automated metrics such as bleu @xcite is that they are incredibly light-weight. despite the higher correlation with human judgement, trainable metrics tend to be slower to run. in an effort to close this gap we present cometinho, a light-weight model based on the comet framework that replaces the original xlm-r large encoder with minilmv2 @xcite . this model is approximately 19x faster at inference time compared to the original comet model @xcite and maintains state-of-the-art correlations with mqm in reference-based evaluations. for the "qe as a metric" track, we show that reference-free evaluation models can reach surprisingly high correlations with human judgements and are competitive with their corresponding referencebased models. last year we also participated with a similar model in the metrics shared task, but here we elaborate in more detail on the primary differences between this model architecture and other comet models. finally, and for the first time, we submit and describe a reference-free model that in addition to learning from mqm scores also makes use of word-level error annotations. this is possible this year given the shift in evaluation method from da tags ok ok ok ok ok ok ok ok ok ok ok ok bad bad mt the main purpose of this project is to design a car for blind driving. source: 这个项目的主要目的 是设计一辆盲人驾驶的车。 reference: to mqm. this model uses the openkiwi foot_0 architecture and its word-level tagging feature to predict ok/bad word tags along with a sentence-level quality score. ## the comet framework for a more comprehensive description of the comet architecture we direct the reader to the original paper @xcite . below we will highlight some relevant features that contrast with the comet reference-free model (comet-qe). in comet we encode segment-level representations using the pretrained, cross-lingual, model xlm-roberta @xcite . even though we encode the source, the hypothesis, and the reference (i.e. the human curated translation of the source) separately, their embeddings are mapped into a shared feature space. subsequently, we obtain combined features using the three embeddings (s, h, and r, for the source, hypothesis, and reference, respectively): h s , h r, |h -s|, and |h -r|. these features, concatenated to r and h and the resulting vector is the input to a feedforward regressor. ## the openkiwi framework when using the mqm framework for the calculation of the quality score, human annotators seek to identify and annotate error spans at the wordlevel, as well as the severity of those errors. we leveraged these word-level annotations using the openkiwi framework @xcite , by transforming each word into an ok or bad tag. in the openkiwi architecture, in contrast with comet-qe, source and hypothesis are jointly encoded. a sentence pair representation is then obtained using average pooling over the hypothesis word embeddings and then used as features to a feed-forward regression layer that learns to produce a sentence level score. at the same time, the word embeddings from the hypothesis are used to predict ok/bad tags and therefore, the model is trained in a multitask setting (regression and sequence labelling). ## corpora in this year's shared task the organisers provided a development set with mqm annotations for the ende and zh-en participating systems on wmt20 @xcite . apart from the official development data we used all the direct assessments available from previous years. ## segment-level task the comet framework is highly flexible and easy to adapt to different types of human judgements @xcite . this year we first pre-trained on the da collected from 2015 to 2020 except for ende and zh-en as described above. like in @xcite model (comet-da). during our experiments we tested two ensembling techniques; averaging the different model predictions and averaging the parameters from the 5 models. those two approaches had similar results but in the end we decided to use the later one for performance. subsequently, we fine-tuned each of the 5 models on the mqm data provided as development for another epoch. as before, we performed weight averaging to obtain an ensemble of those models (comet-mqm). in both the pre-training and finetuning we only perform 1 training epoch in order to ensure that the final models are able to generalise to many language pairs and do not overfit to the news domain. this is especially important since the mqm dataset only contains en-de and zh-en. for cometinho, as previously mentioned, we used the distilled version of xlm-r (minilmv2), available through hugging face, and we followed the same training recipe where we pre-train the model using da's for 1 epoch and then we adapt the model to the mqm data for another epoch. ## system-level task for the system-level task we compute the systemlevel score for each system by averaging the segment-level scores obtained. this follows the same approach used to compute system-level scores based on segment-level human annotations such as da's and mqm which means that a met-ric that achieves strong segment-level correlation should also achieve strong system-level performances. ## qe as a metric task we trained a reference-free model (comet-qe) in the same way we did for reference-based comet models described in section 6. as described in section 2.1, the primary difference between the two models is the inclusion or exclusion of the source as input. ## related work classic n-gram matching mt evaluation metrics such as bleu @xcite have been adopted by the mt community as a primary form of mt evaluation, yet, in the recent years of the wmt metrics shared task @xcite @xcite these classic metrics have been outperformed first by embeddingbased alternatives and more recently by trainable metrics based on pre-trained models. with the rise of word embeddings @xcite @xcite , metrics such as bleu2vec @xcite and meant 2.0 @xcite replaced the typical word/n-gram matching by fuzzy matches based on distributional word representations. these metrics appeared for the first time at the wmt metrics task in 2017 with meant 2.0-srl achieving the highest results at segment-level. in 2018 and 2019 yisi-1 @xcite , a successor of meant 2.0 @xcite , was among the winners of the wmt metrics task. yisi-1 @xcite mostly takes advantage of bert embeddings @xcite to create soft alignments between hypothesis and reference. trainable metrics started as simple regressions based on lexical features (e.g blend @xcite ) but nowadays these metrics also use embeddings to extract features that are then used to regress on quality assessments. the first of such metrics were ruse @xcite and esim @xcite which were based on rnn encoders and worked mostly for english. in 2020, bleurt @xcite and comet @xcite were proposed. both metrics used pre-trained transformer based encoders to extract sentence-level features that are then passed to a regression model; the difference is that comet also extracts features for the source segment which was something overlooked by predecessor metrics. in the 2020 metrics shared task both comet and bleurt achieved some of the highest correlations with human judgements and shared the podium with prism @xcite ## conclusions in this paper we present the unbabel-ist's contribution to the wmt 2021 metrics shared task which for the first time, introduced evaluation using mqm. our specific contributions include; the fine-tuning of direct assessment based models on mqm data which yields impressive gains on the described test sets and a new, lightweight comet model which achieves comparable performance to its predecessors. such a light model can provide interesting opportunities for future work into the incorporation of modern metrics into mt training. finally, but perhaps our most important contributions; we further validate the observations in @xcite that qe as a metric is becoming competitive as an alternative to reference-based evaluation, and, we show that a word-level qe system can be successfully trained on mqm annotations and be competitive with current trainable metrics while providing some intuition about "what" is wrong with a specific translation.