Efficient Transfer Learning for Quality Estimation with Bottleneck Adapter Layer (2020)

Abstract: The Predictor-Estimator framework for quality estimation (QE) is commonly used for its strong performance. Where the predictor and estimator works on feature extraction and quality evaluation, respectively. However, training the predictor from scratch is computationally expensive. In this paper, we propose an efficient transfer learning framework to transfer knowledge from NMT dataset into QE models. A Predictor-Estimator alike model named BAL-QE is also proposed, aiming to extract high quality features with pre-trained NMT model, and make classification with a fine-tuned Bottleneck Adapter Layer (BAL). The experiment shows that BAL-QE achieves 97% of the SOTA performance in WMT19 En-De and En-Ru QE tasks by only training 3% of parameters within 4 hours on 4 Titan XP GPUs. Compared with the commonly used NuQE baseline, BAL-QE achieves 47% (En-Ru) and 75% (En-De) of performance promotions.

## introduction & related work translation quality estimation (qe) has become one of the important research topics in the discipline of machine translation (mt). qe aims to solve the problem of how to evaluate the quality of the translation results and predict the types of errors and locations @xcite , with only source sentences and machine translation rec 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. sults, without the post edited reference. @xcite @xcite . qe tasks can be divided into word level, phrase level and sentence level. in this paper, we only focus on word-level qe tasks. there are two main categories of neural network machine translation quality estimation systems, end-to-end neural network framework and two-stage neural network architecture. a representative architecture of the first one is named neural qe (nuqe) @xcite , which directly predicts sequence labels by passing source and mt results into a unified model composed with several bi-lstm layers. the other one is predictor-estimator architecture @xcite @xcite , which is composed of two subsequent neural models: 1) a word prediction model that predicts each word given the left and right context of the source and target corpus, and 2) a quality estimation model, which estimates word-level labels based on the features generated by the predictor. because the predictor itself can be regarded as a neural machine translation (nmt) system, which can be trained based on a large volume of external parallel corpora and provides high quality semantic features, predictor-estimator framework is much better than nuqe. transfer learning (tl) or fine-tuning large pretrained language models (plms) is an effective method in nlp, which can produce strong performance on many nlp tasks @xcite @xcite . there are two types of transfer learning. the first one is full-parameter fine-tuning with in-domain data, which aims to fit the distribution of indomain data without damaging out-domain performance. the other one is to add additional layers to the original architecture as adapters and only update those newly added layers, resulting in a significant speed-up for fine-tuning. the bottleneck adapter layer (bal) @xcite proposed by google in 2019, shows that bal-based transfer learning could obtain 99.6% of the sota performance by only training 3% of the parameters. the contribution of our paper is as follows: