A Multi-task Learning Framework for Quality Estimation (2023)

Abstract: Quality Estimation (QE) is the task of evaluating machine translation output in the absence of reference translation. Conventional approaches to QE involve training separate models at different levels of granularity viz., word-level, sentence-level, and document-level, which sometimes lead to inconsistent predictions for the same input. To overcome this limitation, we focus on jointly training a single model for sentence-level and word-level QE tasks in a multi-task learning framework. Using two multi-task learning-based QE approaches, we show that multi-task learning improves the performance of both tasks. We evaluate these approaches by performing experiments in different settings, viz., single-pair, multi-pair, and zero-shot. We compare the multi-task learning-based approach with baseline QE models trained on single tasks and observe an improvement of up to 4.28% in Pearson’s correlation (r) at sentence-level and 8.46% in F1-score at word-level, in the single-pair setting. In the multi-pair setting, we observe improvements of up to 3.04% at sentence-level and 13.74% at word-level; while in the zero-shot setting, we also observe improvements of up to 5.26% and 3.05%, respectively. We make the models proposed in this paper publically available.

## introduction quality estimation (qe) is a sub-task in the machine translation (mt) field. it facilitates the evaluation of mt output without a reference translation by predicting its quality rather than finding its similarity with the reference @xcite . qe is performed at different levels of granularity, viz., word-level qe @xcite , sentencelevel qe @xcite , and documentlevel qe @xcite . in the sentence-level qe task, current models predict the z-standardized direct assessment (da) 1 https://github.com/cfiltnlp/qe_mtl score when a source sentence and its translation are provided as inputs. the da score is a number in the range of 0 to 100, denoting the quality of the translation, obtained from multiple human annotators. these scores are then standardized into z-scores, which are used as labels to train the qe model @xcite . unlike the sentence-level qe task, the wordlevel qe task consists of training a model to predict the 'ok' or 'bad' tag for each token in a source sentence and its translation. these tags are obtained automatically by comparing the translation with its human post-edits using a token-matching approach. each source sentence token is tagged as 'ok' if its translation appears in the output and is tagged as 'bad' otherwise. similarly, a translation token is assigned an 'ok' tag if it is a correct translation of a source sentence token, and 'bad' otherwise. apart from the tokens in the translation, the gaps between the translation tokens are also assigned ok/bad tags. in case of missing tokens, the gap is tagged as 'bad', and 'ok' otherwise @xcite . to perform each of these tasks, various deep learning-based approaches are being used @xcite . while these approaches achieve acceptable performance by focusing on a single task, the learning mechanism ignores information from other qe tasks that might help it do better. by sharing information across related tasks, one can essentially expect the task performance to improve, especially when the tasks are closely related as is the case with the sentence-level and word-level qe. also, having a separate model for each qe task can cause problems in practical scenarios, like having higher memory and computational requirements. in addition, the different models can produce conflicting information e.g. high da score, but many errors at word level. in this paper, we utilize two multi-task learning (mtl)-based @xcite approaches for word-level and sentence-level qe tasks with the help of a single deep neural network-based architecture. we perform experiments with existing qe datasets @xcite with both mtl approaches to combine word-level and sentence-level qe tasks. we test the following scenarios: a) single-pair qe, b) multi-pair qe, and c) zero-shot qe. the code and models are made available to the community via github. to the best of our knowledge, we introduce a novel application of the nash-mtl @xcite ) method to both tasks in quality estimation. our contributions are: we discuss the existing literature in section 2 and the datasets used in section 3. the mtl-based qe approach is presented in section 4. the experimental setup is described in 5. section 6 discusses the results in detail, including a qualitative analysis of a few sample outputs. we conclude this article in section 7, where we also propose future research directions in the area. ## related work during the past decade, there has been tremendous progress in the field of machine translation quality estimation, primarily as a result of the shared tasks organized annually by the conferences on machine translation (wmt), since 2012. these shared tasks have produced benchmark datasets on various aspects of quality estimation, including wordlevel and sentence-level qe. furthermore, these datasets have led to the development and evaluation of many open-source qe systems like quest @xcite , quest++ @xcite , deepquest @xcite , and openkiwi @xcite . before the neural network era, most of the quality estimation systems like quest @xcite , and quest++ @xcite were heavily dependent on linguistic processing and feature engineering to train traditional machinelearning algorithms like support vector regression and randomized decision trees @xcite . in recent years, neural-based qe systems such as deepquest @xcite , and openkiwi @xcite have consistently topped the leaderboards in wmt quality estimation shared tasks @xcite . these architectures revolve around an encoder-decoder recurrent neural network (rnn) (referred to as the 'predictor'), stacked with a bidirectional rnn (the 'estimator') that produces quality estimates. one of the disadvantages of this architecture is they require extensive predictor pre-training, which means it depends on large parallel data and is computationally intensive @xcite . this limitation was addressed by transquest @xcite , which won the wmt 2020 shared task on sentence-level da. transquest eliminated the requirement for predictor by using cross-lingual embeddings @xcite . the authors fine-tuned an xlm-roberta model on a sentencelevel da task and showed that a simple architecture could produce state-of-the-art results. later the transquest framework was extended to the wordlevel qe task @xcite . a significant limitation of transquest is that it trains separate models for word-level and sentencelevel qe tasks. while this approach has produced state-of-the-art results, managing two models requires more computing resources. furthermore, since the two models are not interconnected, they can provide conflicting predictions for the same translation. to overcome these limitations, we propose a multi-task learning approach to qe. multitask architectures have been employed in several problem domains, such as those in computer vision @xcite and natural language processing (nlp). in nlp, tasks such as text classification @xcite , natural language generation @xcite , part-ofspeech tagging and named entity recognition @xcite have benefited from mtl. in qe too, @xcite has developed an mtl architecture using a bilingual bert model. however, the model does not provide results similar to or better than state-of-the-art qe frameworks such as transquest @xcite . some of the recent wmt qe shared task submissions also use mtl to develop qe systems @xcite @xcite . as all these submissions are not evaluated under the same experimental settings and use different techniques along with mtl, the improvements due to mtl alone are difficult to assess. in this paper, we introduce a novel mtl approach for qe that outperforms transquest in both word-level and sentence-level qe tasks, in various experimental settings. ## datasets: wmt 2022 we use data provided in the wmt21 @xcite , and wmt22 @xcite quality estimation shared tasks for our experiments. we choose language pairs for which word-level and sentence-level annotations are available for the same source-translation pairs. the data consists of three low-resource language pairs: english-marathi (en-mr), nepali-english (ne-en), sinhalese-english (si-en); three mediumresource language pairs: estonian-english (et-en), romanian-english (ro-en), russian-english (ru-en); and one high-resource language pair: english-german (en-de). for the english-marathi language pair, the data consists of 20k training instances and 1k instances each for validation and testing foot_0 . the training set consists of 7k instances for all other language pairs, and validation and test sets consist of 1k samples each. each sample in the word-level qe data for any language pair except english-marathi consists of a source sentence, its translation, and a sequence of tags for tokens and gaps. for the english-marathi pair, the wmt22 dataset does not contain tags for gaps in tokens. therefore, we used the qe corpus builder foot_1 to obtain annotations for translations using their post-edited versions. ## approach in this section, we briefly discuss the transquest framework, explain the architecture of our neural network, and then discuss the mtl approaches we used for the experimentation, along with the mathematical modeling. ## experimental setup this section describes the different experiments we perform and the metrics we use to evaluate our approach. we also discuss the training details and mention the computational resources used for training the models. experiments we perform our experiments under three settings: single-pair, multi-pair, and zeroshot. for each setting, we train one sentence-level, one word-level, and two mtl-based qe models. the first two models are the single-task learning (stl)-based qe models (stl qe), and we use their performance as baselines. the transquest framework @xcite ) contains the monotransquest model for the sentence-level qe task and the microtransquest model @xcite for word-level qe task which helped us reproduce baseline results over all the language pairs investigated for this paper. the next two models are the mtl-based qe models (mtl qe) trained using two different mtl approaches explained in section 4. for training ls models, we use the framework for adapting representation models (farm) foot_2 , while for training nash-mtl models, we used implementation foot_3 shared by the authors. all the experiments use all seven language pairs introduced in section 3. in the single-pair setting, we only use the data of one particular language pair for training and evaluation. however, in the multi-pair setting, we combine training data of all the language pairs and evaluate the model using test sets of all language pairs. for the transfer-learning experiments (zeroshot setting), we combine training data of all language pairs except the language pair on which we evaluate the model. evaluation we use the pearson correlation (r) between the predictions and gold-standard annotations for evaluating the sentence-level qe as it is a regression task. similarly, for the word-level qe, which is treated as a token-level classification task, we consider the f1-score as an evaluation metric. we perform a statistical significance test considering primary metrics using william's significance test @xcite . training details to maintain uniformity across all the languages, we used an identical set of settings for all the language pairings examined in this work. for the stl and ls-mtl models, we use a batch size of 16. we start with a learning rate of 2e -5 and use 5% of training data for warm-up. we use early stopping and patience over the 10 steps. the nash-mtl models are trained using the configuration outlined in @xcite . considering the availability of computational resources, the stl qe models are trained using the nvidia rtx a5000 gpus, and the mtl qe models using the nvidia dgx a100 gpus. additional training details are provided in appendix a. ## results and discussion results of the single-pair, multi-pair, and zero-shot settings are presented in this section. the tables referred to in this section report performance of the stl, ls-mtl, and nash-mtl qe models using the pearson correlation (r) and f1-score for sentence-level and word-level qe, respectively. we could not conduct a direct performance comparison between our qe models and winning entries of the recent wmt qe shared tasks due to the following reasons: (1) nature of the word-level qe task, and its evaluation methodology have changed over the years. until last year, gaps between translation tokens were a part of the data, and the 'ok' or 'bad' tags were predicted for them as well. but the wmt22 shared task did not consider these gaps; and (2) not all the language pairs investigated in this paper have been a part of wmt qe tasks in the same year. therefore, we establish a standard baseline using the transformers-based framework, transquest, and show improvements. we also compare pearson correlation coeffi-cients obtained by stl and mtl qe models to assess whether mtl qe model predictions on both tasks for the same inputs are consistent ( ## conclusion and future work in this paper, we showed that jointly training a single, pre-trained cross-lingual transformer over the sentence-level and word-level qe tasks improves performance on both tasks. we evaluated our approach in three different settings: single-pair, multipair, and zero-shot. the results on both the qe tasks show that the mtl-based models outperform their stl-based counterparts for multiple language pairs in the single-pair setting. given the performance in the zero-shot setting, we see promising transfer-learning capabilities in our approach. consistent scores across both qe tasks for the same inputs demonstrate the effectiveness of the mtl method to qe. we release our mtl-based qe models and our code under the cc-by-sa 4.0 license publicly for further research. in future, we wish to extend this work and evaluate the mtl-based qe models in a few-shot setting to assess the effectiveness of transfer learning. fur-ther, we would like to explore the usage of wordlevel qe and sentence-level qe to assist in the task of automatic post-editing for mt. we also wish to explore the use of language-relatedness for building multi-pair mtl-based qe models.