IST -Unbabel 2021 Submission for the Quality Estimation Shared Task (2021)

Abstract: We present the joint contribution of IST and Unbabel to the WMT 2021 Shared Task on Quality Estimation. Our team participated on two tasks: Direct Assessment and Post-Editing Effort, encompassing a total of 35 submissions. For all submissions, our efforts focused on training multilingual models on top of OpenKiwi predictor-estimator architecture, using pre-trained multilingual encoders combined with adapters. We further experiment with and uncertainty-related objectives and features as well as training on out-of-domain direct assessment data.

## introduction quality estimation (qe) is the task of evaluating a translation system's quality without access to reference translations @xcite . this paper describes the joint contribution of instituto superior técnico (ist) and unbabel to the wmt21 quality estimation shared task @xcite , where systems were submitted to two tasks: 1) sentence-level direct assessment; 2) word-and sentence-level post-editing effort. this year's submission combines several ideas built on top of the openkiwi framework. motivated by the mixture of blind and seen language pairs in the test sets, we experimented with extensions that would allow us to train multilingual models that maintain good generalization ability and are robust to the presence of epistemic and aleatoric uncertainty. for both tasks we trained and submitted an ensemble of multilingual models. all submitted models follow the predictor-estimator architecture @xcite and use pretrained models for feature extraction. also, we fine-tune all models on the provided qe data using stacked adapter layers @xcite . we show that we can thus achieve comparable performance across language pairs while minimising the number of trainable parameters (see our main contributions are: ## quality estimation tasks in this year's shared task edition we submitted models for the first two tasks: we note that this year, both tasks 1 and 2 provided additional blind test sets with language pairs that were not included in the data made available for training/development, providing an interesting challenge and motivating multilingual and generalisable approaches. ## experimental results we present the performance of the implemented models on the test-20 dataset. ## conclusions we presented a joint contribution of ist and unbabel to the wmt 2021 qe shared task. our submissions are ensembles of multilingual checkpoints extending the openkiwi framework. we found adapter-tuning to be suitable for fine-tuning openkiwi on the qe tasks data and less prone to overfitting. we showed that pre-training on large, out-of-domain annotated data can prove beneficial both for the direct assessment and the postediting qe tasks. we also demonstrated that handling uncertainty-related sources of information improves the performance when integrated into the qe system. for task 2 we do multi-task training based on the models from the previous task and use multiple checkpoints to create the submitted ensemble.