Continual Quality Estimation with Online B ayesian Meta-Learning (2021)

Abstract: Most current quality estimation (QE) models for machine translation are trained and evaluated in a static setting where training and test data are assumed to be from a fixed distribution. However, in real-life settings, the test data that a deployed QE model would be exposed to may differ from its training data. In particular, training samples are often labelled by one or a small set of annotators, whose perceptions of translation quality and needs may differ substantially from those of end-users, who will employ predictions in practice. To address this challenge, we propose an online Bayesian meta-learning framework for the continuous training of QE models that is able to adapt them to the needs of different users, while being robust to distributional shifts in training and test data. Experiments on data with varying number of users and language characteristics validate the effectiveness of the proposed approach.

## introduction quality estimation (qe) models aim to evaluate the output of machine translation (mt) systems at run-time, when no reference translations are available @xcite . qe models can be applied for instance to improve translation productivity by selecting high-quality translations amongst several candidates. a number of approaches have been proposed for this task @xcite @xcite @xcite , and a shared task yearly benchmarks proposed approaches @xcite . different users of mt output have varying quality needs and standards, depending for instance on the downstream task at hand, or the level of their knowledge of the languages involved, and training for the task. thus, the perception of the quality of mt output can be subjective, and therefore the quality estimates obtained from a model trained on data from one set of users may not serve the needs of a different set users. however, most existing qe models are trained and evaluated in a static setting which assumes a fixed distribution of train and test data. this consequently leads to suboptimal performance when faced with test data from a different set of users in practice. the few previous approaches to develop qe models that are able to learn from a continuous stream of data suffer from the following limitations: they do not have an explicit objective that encourages the model to exploit common structures shared among different users to continually adapt efficiently for new users @xcite , or assume a fixed number of users, and that the identity of each user is known in advance @xcite . in addition, these previous approaches do not explicitly account for the underlying uncertainties in the data in order to improve performance. in contrast, we propose a continual metalearning framework that makes none of the aforementioned assumptions, but instead considers each user as a task and explicitly meta-learns the common structure shared among different users. this approach further exploits the underlying uncertainties in the streaming data through bayesian inference to improve performance. in addition, the proposed approach is applicable even in a setting where no user identities are available, for instance due to privacy concerns, but where we still want to learn and adapt as efficiently as possible from supervision data that arrives incrementally. ## meta-learning for continual learning with adaptive svgd learning continually from a stream of observations with varying underlying distributions involves dealing with various sources of uncertainty, which a model should properly account for in order to enhance its continual learning performance. one source of uncertainty is in the learning rate, that is, how fast learning should proceed on new data in order to both reduce catastrophic forgetting and enhance performance on the current task. another source is the inherent uncertainty in the values of the model's parameters themselves. learning an adaptive learning rate, for instance as proposed in @xcite , can help account for the first source of uncertainty, and bayesian inference can be used to help a model account for the other source of uncertainty. in order to properly model both sources of uncertainty during continual learning, we propose to both perform inference of model parameters with svgd, and meta-learn an adaptive per-parameter learning rate for svgd updates. thus, the svgd update in equation 2 now becomes: we first initialize the parameters of the qe model, and the learning rate (line 1). then for each mini-batch in a task t that arrives, we store its training instances in the buffer with a probability p (lines 2-6). in the inner loop, we perform k svgd updates (using equation 4 ) starting from the initial model parameters w 0 (lines 7-9). in the outer loop, instances in the current mini-batch are augmented with instances sampled from the buffer (line 10). finally, the augmented mini-batch is used to perform a meta-update on the learning rate (line 11), and on the parameters of the qe model (line 12). because this approach can also be considered the online counterpart to the bayesian model agnostic meta-learning approach of @xcite , we refer to it as continual quality estimation with online bayesian meta-learning (cqe-obml). ## experiments and results the qt21 dataset we evaluate our approach with the publicly available qt21 @xcite , a large-scale dataset containing translations from both statistical (smt) and neural (nmt) machine translation systems in multiple language directions. foot_0 this is the largest dataset with annotator information available. we use data from the english-latvian (en-lv) and english-czech (en-cs) language pairs. these languages were chosen as they contain the largest number of annotators. each instance in the dataset is a tuple of source sentence, its machine translation, the corresponding post-edited translation by a professional translator (post-editor), a reference translation and other information such as (anonymized) post-editor identifier. we construct a qe dataset from this corpus by computing the hter @xcite values between each source sentence and its post-edited translation. we thereafter split the data into train, dev and test splits for each post-editor. a breakdown of the number of train, dev and test instances per post-editor is available in benchmark approaches sequential is a baseline trained sequentially over the streaming data of each task. in each round, the model parameters are initialized from that of the previous round; a-gem @xcite is a continual learning method which utilizes the gradients of samples of previous tasks saved in a buffer as an optimization constraint to prevent catastrophic forgetting; oml-er @xcite augments the online-aware meta-learning approach of @xcite with experience replay from a buffer; la-maml @xcite learns per-parameter learning rates using meta-learning; mtl-iid is trained on the concatenated and shuffled data from all users for multiple epochs in multi-task fashion. it assumes i.i.d access to the data from all users, and thus serves as an upper-bound for the performance. ## conclusions we proposed a framework for the continual metalearning of machine translation quality estimation models, which is able to learn continually from the streaming data of multiple quality estimation users. we further incorporate an adaptive learning rate together with online bayesian inference for improved performance. in experiments on quality estimation data from two language directions, we demonstrate improved performance over recent state-of-the-art continual learning methods.