Semi-supervised Learning for Quality Estimation of Machine Translation (2023)

Abstract: We investigate whether using semi-supervised learning (SSL) methods can be beneficial for the task of word-level Quality Estimation of Machine Translation in low resource conditions. We show that the Mean Teacher network can provide equal or significantly better MCC scores (up to +12%) than supervised methods when a limited amount of labeled data is available. Additionally, following previous work on SSL, we investigate Pseudo-Labeling in combination with SSL, which nevertheless does not provide consistent improvements.

## introduction through the recent development of machine translation (mt), quality estimation (qe) has come to serve the need to predict the quality of translation provided by mt systems when no reference translations are available. qe has been mostly treated as a supervised learning problem, where supervised models can be trained on the source and translated text along with their respective quality labels. for example, qe at the word level includes the source and translated sentences as the data and their label sequence includes an ok or bad label for each translated word in the sentence, which can determine if the word is correctly translated or not and potential errors in the translations can be flagged. in order to train supervised models for such problems, a large amount of labeled data is needed. however, such data is expensive to create as it involves human annotators to post-edit or generate labels for the given translations. whereas the unavailability of labeled data is a problem, there is an abundance of unlabeled data for such a task, i.e. source sentences and the corresponding translations generated by mt systems. semi-supervised learning (ssl) methods could be utilized to train qe models with few labeled data available along with unlabeled data that can be generated in abundance. while the prominent ssl approach of mean teacher has shown good performance in computer vision @xcite , there has been little experimentation in nlp. until now, no research has followed ssl to fine-tune pre-trained language models (lms) for the task of qe of mt. this work focuses on implementing the aforementioned ssl strategies for word-level qe and tries to answer the following questions: figure 1 : example of german translation by a mt system and its human post-edited version with labels of gaps and target words for the example translated text. tags for gap tokens are highlighted in yellow and tags for words in sequence are highlighted in blue @xcite 1. can the ssl method of mean teacher perform equal or better than supervised methods on low-resource conditions? one should note that our research does not aim to achieve the highest mcc scores as compared to sota, but to test if ssl techniques can be useful in low resource conditions. hence, our baseline here are the models created with a fully supervised setup under low resource conditions, which are then compared against our proposed models trained with ssl. ## related work there has been few previous works on ssl methods for nlp. @xcite showed improvement on models trained with labeled data through pseudo-labeling for named entity recognition. @xcite suggest a noise-injected consistency training with entropyconstrained pseudo labeling for labeling extractive summarization data. such approaches have not been investigated for other nlp problems involving token level classification. state-of-the art qe methods @xcite employ fine-tuning of pre-trained lms, but they don't take into consideration low-resource conditions. concerning nonsupervised methods, @xcite perform unsupervised qe by utilizing internal decoding features of the mt models. with regards to low-resource conditions, @xcite demonstrate that it is possible to accurately predict word-level quality for any given new language pair from models trained on other language pairs. in an effort to address low resource conditions, @xcite train off-the-shelf architectures for supervised qe using synthetic data from parallel corpora. to the best of our knowledge, none of the related work in qe of mt has used semi-supervised methods to address low resource conditions. ## methods we focus on the task of qe of mt at the word level, as specified at the shared task of qe of wmt @xcite , which aims at flagging potential errors in the translations generated by any mt system. the word-level task requires assigning binary tags of ok/bad to determine the correctness of each word in source and target/translated sentences. the following types of labels are used: source side each word in the source sentence is assigned a label (ok/bad) which determines if the respective word is correctly translated in the target language or not. target side each word in the target sentence is assigned a label (ok/bad) which determines if the word is a correct translation for the respective word in the source sentence. additionally, figure 2 : training flow diagram of supervised fine-tuning methodology gap tokens are also considered in the beginning, at the end, and between two words of the target sentence. each gap token is assigned the label bad if a word or more than a word is missing in the position of the gap token, and it is tagged ok otherwise. an example of the gap tokens can be seen in figure 1 . the proposed methods involve the training of models with supervised and ssl methods. the fine-tuning of a large lm is done by utilizing three strategies i.e. 1) supervised learning, 2) ssl using the mean teacher approach., 3) ssl using a mean teacher with the pseudo-labeling approach. ## conclusion this research focused on the quality estimation of machine translation at the word level. the goal is to generate a binary label of ok/bad for each word and gap in the translations, by predicting if the word is correctly translated or not. we investigated two approaches of semi-supervised learining that have not been explored yet for the given problem: the first utilized the well-known mean teacher approach that involves a student and a teacher model while training, initialized with the default weights of a pretrained lm. the second proposed architecture extends the former, by involving pseudo-labeling and follows a two-stage learning approach. in the first stage, the model is trained with limited labeled data available, through supervised learning. in the second stage, the teacher and student model are initialized with the model learned in the first stage, and are further trained using only unlabeled data. it was experimentally shown that in low-resource settings the mean teacher architecture performed better or (in one case) comparably to the supervised models, achieving an improvement of up to 12%.