Unify Word-level and Span-level Tasks: NJUNLP ’s Participation for the WMT 2023 Quality Estimation Shared Task (2023)

Abstract: We introduce the submissions of the NJUNLP team to the WMT 2023 Quality Estimation (QE) shared task. Our team submitted predictions for the English-German language pair on all two sub-tasks: (i) sentence- and word-level quality prediction; and (ii) fine-grained error span detection. This year, we further explore pseudo data methods for QE based on NJUQE framework (https://github.com/NJUNLP/njuqe). We generate pseudo MQM data using parallel data from the WMT translation task. We pre-train the XLMR large model on pseudo QE data, then fine-tune it on real QE data. At both stages, we jointly learn sentence-level scores and word-level tags. Empirically, we conduct experiments to find the key hyper-parameters that improve the performance. Technically, we propose a simple method that covert the word-level outputs to fine-grained error span results. Overall, our models achieved the best results in English-German for both word-level and fine-grained error span detection sub-tasks by a considerable margin.

## introduction quality estimation (qe) of machine translation (mt) is a task to estimate the quality of translations at run-time without access to reference translations @xcite . there are two subtasks in wmt 2023 qe shared task 2 : (i) sentenceand word-level quality prediction; and (ii) finegrained error span detection. we participated in all two sub-tasks for the english-german (en-de) language pair. the annotation of en-de is multidimensional quality metrics (mqm) 3 , aligned with the wmt 2023 metrics shared task. the mqm annotation provides error spans with fine-grained categories and severities by human translators. inspired by directqe @xcite and clqe @xcite , we further explore pseudo data methods for qe based on the njuqe framework. we generate pseudo mqm data using parallel data from the wmt translation task. specifically, we replace the reference tokens with these tokens sampled from translation models. to simulate translation errors with different severities, we sample tokens with lower generation probabilities for worse errors @xcite . we pretrain the xlmr @xcite large model on pseudo mqm data, then fine-tune it on real qe data. at both stages, we jointly learn sentencelevel scores (mse loss and margin ranking loss) and word-level tags (cross-entropy loss). for task (i), the qe model outputs the sentence scores and the "ok" probability of each token. for task (ii), we set different thresholds for the "ok" probability to predict fine-grained severities. we regard consecutive "bad" tokens as a whole span and take the worse severity of each token as the result. we train different models with different parallel data and ensemble their results as the final submission. overall, we summarize our contribution as follows: our system obtains the best results in english-german for both word-level and fine-grained error span detection sub-tasks with an mcc of 29.7 (+4.1 than the second best system) and f1 score of 28.4 (+1.1) respectively. we rank 2nd place on sentencelevel sub-tasks with a spearman score of 47.9 (-0.4 than the best system). ## background given a source language sentence x and a target language translation ŷ = {y 1 , y 2 , . . . , y n } with n tokens, the mqm annotation provides error spans with fine-grained categories and severities (minor, major, and critical) by human translators. the mqm score sums penalties for each error severity and then normalizes the result by translation length: as shown in ## methodology generally, we unite the sub-tasks (i) and (ii) as follows: ## analysis in this section, we show some key hyperparameters that improve the performance. ## conclusion we present njunlp's work to the wmt 2023 shared task on quality estimation. in this work, we generate pseudo mqm data using parallel data. we pre-train the xlmr large model on pseudo mqm data, then fine-tune it on real qe data. at both stages, we jointly learn sentence-level scores and word-level tags. empirically, we conduct experiments to find the key hyper-parameters that improve the performance. technically, we propose a simple method that covert the word-level outputs to fine-grained error span results. overall, our models achieved the best results in english-german for both word-level and fine-grained error span detection sub-tasks by a considerable margin.