RTM Super Learner Results at Quality Estimation Task (2021)

Abstract: We obtain new results using referential translation machines (RTMs) with predictions mixed to obtain a better mixture of experts prediction. Our super learner results improve the results and provide a robust combination model.

## introduction quality estimation task in wmt21 @xcite (qet21) address machine translation (mt) performance prediction (mtpp), where translation quality is predicted without using reference translations, at the sentence-level (tasks 1, 2, and 3) and with classification of sentences into containing a critical error or not (task 3). task 1 predicts the sentence-level direct assessment (da) in 11 language pairs categorized according to the mt resources available: ## rtm for mtpp we use rtm models for building our prediction models. rtms predict data translation between the instances in the training set and the test set using interpretants, text data selected close to the task instances in bilingual training settings or monolingual lm settings. interpretants are text data that provide context for the prediction task and are used during the derivation of the features measuring the closeness of the test sentences to the training data, the difficulty of translating them, and to identify translation acts between any two data sets for building prediction models. with the enlarging parallel and monolingual corpora made available by wmt foot_1 , the capability of the interpretant datasets selected to provide context for the training and test sets improve with parallel feature weight decay (parfwd) instance selection @xcite . rtms use parfwd for instance selection and for machine translation performance prediction system (mtpps) @xcite to obtain the features, where additional features from word alignment are added. figure 1 depicts rtms and explains the model building process. we treated all of tasks 1, 2, and 3 as bilingual tasks where parallel corpora are obtained from wmt translation task. 3 the related monolingual or bilingual datasets are used during feature extraction. the machine learning models we use include ridge regression (rr), support vector regression (svr) @xcite , gradient tree boosting, extremely randomized trees @xcite , and multi-layer perceptron @xcite in combination with feature selection (fs) @xcite and partial least squares (pls) @xcite where most of these models can be found in scikit-learn. 4 we use rr to estimate the noise level for svr, which obtains accuracy with 5% error compared with estimates obtained with known noise level (cherkassky and ma, 2004) and @xmath0. we use pearson's correlation (r), mean absolute error (mae), root mean squared error (rmse), relative absolute error (rae), relative mae (maer), and mean rae relative (mraer) as evaluation metrics @xcite . our best non-mixed results are in ## mixture of experts models we use prediction averaging @xcite to obtain a combined prediction from various prediction outputs better than the components, where the performance on the training set is used to obtain weighted average of the top k predictions, ŷ with evaluation metrics indexed by @xmath1: super learner @xcite ) is a stacking model on a library of l learning models that are v -fold cross-validated on the training set and constructs an v × l level 1 dataset. theoretical results show that as the number of different predictors in the ensemble increase, the ensemble result gets closer to the oracle result @xcite . the function that minimize the empirical risk on the validation set will achieve lower error than the function that minimize the overall risk: @xcite . model combination (figure 2 ) selects top k combined predictions and adds them to the set of predictions where the next layer can use another model combination step or just pick the best model according to the results on the training set. we use a two layer combination where the second layer is a combination of all of the predictions obtained. the last layer is an arg max. our test set results using super learner are in the baseline deepquest @xcite use bidirectional gated recurrent unit type recurrent neural networks to model qet. rtm + deepquest combination results in task 2 use linear interpolation of rtm and deepquest results with weights 0 ≤ λ ≤ 1 and 1 -λ respectively as well as polynomial function fits to find the best combination model optimized on the development set. the most common function fit found is f (x) = r p mae rmse a x + bx 3 + cx 2 + dx + e ( task 3 results are in ## conclusion referential translation machines pioneer a language independent approach and remove the need to access any task or domain specific information or resource and can achieve good results in automatic, accurate, and language independent prediction of translation scores. we present rtm ensemble results with super learner.