Scaling up C omet K iwi: Unbabel- IST 2023 Submission for the Quality Estimation Shared Task (2023)

Abstract: We present the joint contribution of Unbabel and Instituto Superior Técnico to the WMT 2023 Shared Task on Quality Estimation (QE). Our team participated on all tasks: Sentence- and Word-level Quality Prediction and Fine-grained error span detection. For all tasks we build on the CometKiwi model (rei et al. 2022). Our multilingual approaches are ranked first for all tasks, reaching state-of-the-art performance for quality estimation at word-, span- and sentence-level granularity. Compared to the previous state-of-the-art, CometKiwi, we show large improvements in correlation with human judgements (up to 10 Spearman points) and surpassing the second-best multilingual submission with up to 3.8 absolute points.

## introduction quality estimation (qe) is the task of automatically assigning a quality score to a machine translation output without depending on reference translations @xcite . this paper details the collaborative effort of unbabel and instituto superior técnico (ist) in the wmt23 quality estimation shared task, which encompassed two primary tasks: (i) sentence-and word-level quality prediction and (ii) fine-grained error span detection. as of last year, some language pairs in the test set were absent from the training data. to address this, following a similar approach to the previous year, our systems were developed to achieve good multilingual generalization and to accommodate previously unseen languages. to achieve this, we start by leveraging the direct assessments (da) labeled data obtained from the wmt metrics shared task from 2017 to 2020, the mlqe-pe dataset @xcite , and the training data (da) specifically annotated for indian languages in the 2023 shared task edition. in total, these datasets encompass close to 1m annotations covering 38 language pairs. we start by constructing generic models using this corpus. these generic qe models were subsequently fine-tuned for this year's subtasks. for task 1 -sentence-level, we fine-tuned our generic models exclusively with this year's da data. the architecture of these models remains consistent with our submission from the previous year, but we employ xlm-r xl and xxl as pretrained encoders @xcite . for the word-level quality prediction task, we follow the successful approach of combining the sentence-and wordlevel signals into one loss during the finetuning step, which has yielded positive results in previous iterations @xcite . for fine-grained error span detection, we conducted experiments exploring various approaches that build upon our word-level and sentence-level strategies. in terms of contrasting systems, we explored unbabelqi foot_0 and gpt-4 (openai, 2023). for gpt-4, we used a prompt designed to predict both the location and severity of errors in each translation, akin to the approach used in automqm @xcite . overall, our main contributions are: (i) we introduce approaches for multilingual machine translation quality estimation that are consistently firstranked at word-, span-, and sentence-level granularity; (ii) we explore different approaches to predict the span of problematic translations along with their error severities (ok, minor, major); (iii) we publicly release two of our best models for research purposes (cometkiwi -xl foot_1 and -xxl foot_2 ). to the best of our knowledge, these are the largest open-source qe models publicly released. our submitted systems attain the top multilingual results in all tasks: for task 1 sentence-level prediction, our multilingual system achieves 59.4 spearman correlation points, surpassing the second-best system by nearly 4 absolute points. for word-level, our system achieves a 31.7 mcc score, outperforming the second-best system by almost 2 absolute mcc points. for error span prediction, our multilingual system achieves a 22 f1.0 score, beating the second-best system by more than 5 f 1 points. task 1 -sentence-level quality prediction: submissions for this task were evaluated based on their correlation with direct assessment (da) annotations for five language pairs: @xmath0, @xmath1, @xmath2, @xmath3, and @xmath4. furthermore, they were evaluated using multidimensional quality metrics (mqm) annotations for three language pairs: @xmath5, @xmath6, and @xmath7. training data was made available for all language directions except for he-en. task 1 -word-level quality prediction: submissions for this task underwent evaluation based on tags inferred from post-editions for @xmath8. additionally, they were assessed using mqm annotations for en-de, zh-en, and he-en. no addi- task 2 -fine-grained error span detection: submissions were evaluated on error spans obtained via mqm annotations for 3 language pairs (en-de, zh-en and he-en). no training nor development data is available for he-en. ## implemented systems we largely follow the architecture of cometkiwi @xcite ) -see figure 1 for an illustration. we concatenate the machine translated sentence @xmath9, ..., t n ⟩ and its source sentence counterpart @xmath10, ..., s m ⟩ to serve as input to the encoder. this encoder then produces hidden state matrices h 0 , ..., h l for each layer 0 ≤ ℓ ≤ l, where h ℓ ∈ r (n+m)×d , where ℓ = 0 corresponds to the embedding layer and d is the hidden size. following this, all hidden states are fed to a scalar mix module @xcite that learns a weighted sum of the hidden states of each layer of the encoder, producing a sequence of aggregated hidden states h mix as follows: here λ is a scalar trainable parameter, β ∈ △ l is given by β = sparsemax(ϕ) using a sparse transformation @xcite , with ϕ ∈ r l as learnable parameters, and where we denote by △ l := {β ∈ r l : 1 ⊤ β = 1, β ≥ 0} the probability simplex. 4 for sentence-level models, we use the hidden state of @xmath11, which, in turn, is passed to a 2-layered feedforward module in order to get a sentence score prediction ŷ ∈ r. for word-level and error span detection models, we first retrieve the hidden state vectors associated with each each token in t, and then pass them to a linear projection to get word-level predictions ŷi ∈ y wl , ∀ 1≤@xmath12. the output space of the word-level predictions is different depending on whether the models are constructed for word-level quality prediction (y @xmath13, bad}), or error span detection (y @xmath14, minor, major}). pretrained multilingual encoders. similarly to @xcite , we employ infoxlm l @xcite . 5 additionally, we experiment with scaled-up multilingual encoders, including xlm-r xl, foot_5 and xlm-r xxl. 7 infoxlm l comprises 24 encoder blocks with 16 attention heads each, totaling 550m parameters. xlm-r xl and xlm-r xxl have 32 attention heads for each encoder block, 36 and 48 encoder blocks and a total of 3.5b and 10.7b parameters, respectively. generic models for all tasks. we create, for each model size, a generic model that will then be further adapted to each separate task. to train these models, we use the collective corpora from 2017 to 2019 da annotations of the wmt translation shared task, and the mlqe-pe corpus @xcite . we include the human annotations respective to the language pairs of this year's shared task for 7 different language pairs: da annotations for en-mr, en-hi, en-ta, en-te, en-gu, and mqm annotations for en-de and zh-en. overall, the generic models are trained on sentence-level quality prediction with over 940k samples with source, translation and quality score on 38 different language pairs. when presented with multiple da scores for the same sentence pair, we used the z-score of the das for training but we first normalize the das between 0 and 1, where 1 represents a perfect translation and 0 a random one. task adaptation. after having obtained the generic models, we will train models for each separate stream of the shared-task, i.e., sentence-level, word-level or error span prediction. to do so, we consider the multi-task optimization from @xcite wherein sentence scores can be used alongside supervision from word-level tags. formally, using unconstrained models. for error span detection, we evaluate unbabelqi, an unbabel demo qe system, alongside gpt4 (openai, 2023). we prompt gpt4 to produce an mqm annotation for each source-target pair, based on five-shot examples which vary across language pairs but are consistent within segments of the same language pair. we also apply this system in task 1, deriving a sentence-level score from error spans, in alignment with the mqm framework. this approach bears similarity to automqm @xcite . ## experimental results we present the results on the official test set for each of the tasks for multiple model/data configurations. sentence-level submissions were evaluated using the spearman rank correlation. pearson and kendall correlation were also used as secondary metrics, but here we report only spearman since it was the primary metric used to rank systems. word-level submission were evaluated using mcc, f 1 -ok, and f 1 -bad, but we report only mcc as it was considered the main metric. error span detection was evaluated using f 1 score in which the positive labels are all the characters belonging to erroneous spans. furthermore, each true positive is downweighted to half if the system failed to classify the error span's severity (e.g., minor instead of major). the submitted systems were independently evaluated on in-domain and zero-shot lps for direct assessments and mqm. ## final remarks we describe unbabel and ist joint submission to wmt23 qe shared task. our approaches correlate well with human judgements for all the three granularities of translation quality prediction, ranking first in all multilingual tasks and surpassing the pre-vious state-of-the-art model, cometkiwi-22, by up to 10 spearman correlation points. overall, our models follow the same architecture of last year's participation, cometkiwi. however, this year we leverage more data and larger encoder models. our best final systems are ensembles of different models trained on da, post-edits or mqm scores that complement each other. interestingly, our best systems surpass gpt-4 by a large margin for sentencelevel translation quality prediction, and they are comparable to gpt-4 at error span detection.