Ensemble Fine-tuned m BERT for Translation Quality Estimation (2021)

Abstract: Quality Estimation (QE) is an important component of the machine translation workflow as it assesses the quality of the translated output without consulting reference translations. In this paper, we discuss our submission to the WMT 2021 QE Shared Task. We participate in Task 2 sentence-level sub-task that challenge participants to predict the HTER score for sentence-level post-editing effort. Our proposed system is an ensemble of multilingual BERT (mBERT)-based regression models, which are generated by fine-tuning on different input settings. It demonstrates comparable performance with respect to the Pearson’s correlation, and beat the baseline system in MAE/ RMSE for several language pairs. In addition, we adapt our system for the zero-shot setting by exploiting target language-relevant language pairs and pseudo-reference translations.

## introduction progress in machine translation (mt) has accelerated due to the introduction of deep learning based approaches, dubbed as neural machine translation (nmt) @xcite @xcite . several metrics (e.g., bleu @xcite , meteor @xcite ) are used to automatically evaluate the quality of the translations outputted by the nmt systems. however, these evaluation metrics require comparing the nmt outputs against human-prepared reference translations, which cannot be readily obtained. to tackle this predicament, recently quality estimation (qe) @xcite has emerged as an alternative evaluation approach for nmt systems. qe obviates the need for human judgements and hence can be efficiently integrated into the dynamic translation pipeline in the industry setting. * work done during internship at iqvia qe is performed at different granularity (e.g., word, sentence, document) @xcite ; in this work we focus on the sentence-level postediting effort task, which predicts the quality of the translated sentence as a whole in terms of the number of edit operations that need to be made to yield a post-edited translation, termed as hter @xcite . sentence-level qe using neural approaches is generally treated as a supervised regression problem involving mainly two steps. in the first step, an encoder is used to learn vector representation/s of the source and translation sentences. while in the second step, the learned representations are passed through a sigmoid output layer to estimate the hter score. these two steps can be performed either with a single model in an end-toend fashion (e.g., bi-rnn @xcite ), or using two separate models (e.g., postech @xcite ). the different qe systems vary in their choice of the encoder, which range from rnnbased to transformer-based models. in this work, we leverage the fine-tuning capability of a transformer-based encoder, namely the mbert @xcite pre-trained model. alongside the standard practice of feeding both the source and target (i.e., translation) sentences as the input sequence @xcite , we also explore other input settings based on only the target-side sentences (i.e., monolingual context). to this end, our final qe system is an ensemble of several mbert models 1 , each generated by fine-tuning on a different input combination comprising the source and/or target sentences. we experiment with the following three input settings: (1) both source and target, (2) just target and (3) both target and a randomly-sampled target sentence in the data forming the input se-quence. empirical analysis on 6 language pairs shows that the ensemble model is able to perform better than the individual fine-tuned models. moreover, we provide experimental results for zero-shot qe, where training data for the test language pair is not available. this we tackle by improvising on the available training/dev data that match the target language of the test language pair and also by generating the pseudo-reference translations in that language. ## data we use the wmt21 qe shared task 2 sentencelevel data @xcite for the following 7 language pairs: english-german (en-de), romanian-english (ro-en), estonian-english (et-en), nepalese-english (ne-en), sinhala-english (si-en), russian-english (ru-en) and khmer-english (km-en). source-side data for each language pair includes sentences from wikipedia articles, with part of the data gathered from reddit articles for ru-en. to obtain the translations, state-of-the-art mt models @xcite built using fairseq toolkit @xcite were used. the label for this task is the hter score for the source-translation pair. annotation was performed first at the word-level with the help of ter foot_2 tool. the word-level tags were then aggregated deterministically to obtain the sentence-level hter score. the training, development, test and blind test data sizes for each language pair (except km-en) are 7k, 1k, 1k and 1k instances respectively. as km-en language pair was introduced for zero-shot prediction, only the test data containing 990 source and translation sentences was provided. ## our approach a key innovation in recent neural models lies in learning the contextualized representations by pretraining on a language modeling task. one such model, the multilingual bert (mbert) foot_3 , is a transformer-based masked language model that is pre-trained on monolingual wikipedia corpora of 104 languages with a shared word-piece vocabulary. training the pre-trained mbert model for a supervised downstream task, aka finetuning, has dominated performance across a wide spectrum of nlp tasks @xcite . our proposed approach leverages this fine-tuning capability of mbert so as to form the component models in the ensemble qe system (section 3.3). that is, each component model is a re-purposed mbert that is fine-tuned for the sentence-level hter score prediction task on one of the three input settings discussed in section 3.2. ## baseline the baseline qe system (baseline) set by the wmt21 organizers this year is the transformerbased predictor-estimator model @xcite . xlm-roberta is used as the predictor for feature generation. the baseline system is fine-tuned on the hter scores and word-level tags jointly. ## results ## conclusion in this work, we describe the ensbrt system submission to the wmt21 qe shared task. ensbrt is based on fine-tuning the multilingual bert pretrained model for sentence-level translation quality score prediction. we explore three different input settings for fine-tuning which include either bilingual or monolingual context, and combine the predictions of the three models using ensemble methods as our final system. furthermore, zeroshot qe is facilitated by using labeled data for existing language pairs and pseudo-references that align with the target language of the unseen test data.