RTM Ensemble Learning Results at Quality Estimation Task (2020)

Abstract: We obtain new results using referential translation machines (RTMs) with predictions mixed and stacked to obtain a better mixture of experts prediction. We are able to achieve better results than the baseline model in Task 1 subtasks. Our stacking results significantly improve the results on the training sets but decrease the test set results. RTMs can achieve to become the 5th among 13 models in ru-en subtask and 5th in the multilingual track of sentence-level Task 1 based on MAE.

## introduction quality estimation task in wmt20 @xcite ) (qet20) address machine translation (mt) performance prediction (mtpp), where translation quality is predicted without using reference translations, at the sentence-(tasks 1 and 2), word-(task 2), and document-levels (task 3). task 1 predicts the sentence-level direct assessment (da) in 7 language pairs categorized according to the mt resources available: ## rtm for mtpp we use rtm models for building our prediction models. rtms predict data translation between the instances in the training set and the test set using interpretants, data selected close to the task instances in bilingual training settings or monolingual language model (lm) settings. interpretants provide context for the prediction task and are used during the derivation of the features measuring the closeness of the test sentences to the training data, the difficulty of translating them, and to identify translation acts between any two data sets for building prediction models. with the enlarging parallel and monolingual corpora made available by wmt, the capability of the interpretant datasets selected to provide context for the training and test sets improve as can be seen in the data statistics of parfwd instance selection, parallel feature weight decay @xcite . rtms use parfwd for instance selection and machine translation performance prediction system (mtpps) @xcite for obtaining the features, which includes additional features from word alignment. figure 1 depicts rtms and explains the model building process. additionally, we included the sum, mean, standard deviation, minimum, and maximum of alignment word log probabilities as features in task 1. in task 2, we included word alignment displacement features including the average of source and target displacements relative to the length of the source or target sentences respectively and absolute displacement relative to the maximum of source and target sentence lengths. instead of resource based discernment, we treated en-de of tasks 1 and 2 and ru-en as bilingual tasks where significant parallel corpora are available from wmt from previous years and the rest as monolingual, using solely english side of the corpora for deriving mtpp features. in accord, we treat en-de and ru-en as parallel mtpp and the rest as monolingual mtpp. rtm benefits from relevant data selection to be used as interpretants in both monolingual and bilingual settings. the related monolingual or bilingual datasets are used during feature extraction for the machine learning models of mt. the machine learning models we use include ridge regression (rr), kernel ridge regression, support vector regression (svr) @xcite , gradient tree boosting, extremely randomized trees @xcite , and multi-layer perceptron @xcite as learning models in combination with feature selection (fs) @xcite and partial least squares (pls) @xcite where most of these models can be found in scikit-learn. 2 we experiment with: we use pearson's correlation (r), mean absolute error (mae), root mean squared error (rmse), relative absolute error (rae), relative mae (maer), and mean rae relative (mraer) as evaluation metrics @xcite . our best non-mix results are in ## mixture of experts models we use prediction averaging @xcite to obtain a combined prediction from various prediction outputs better than the components, where the performance on the training set is used to obtain r p mae rmse we also use generalized ensemble method (gem) as an alternative to mix to combine using weights and correlation of the errors, c i,j , where gem achieves smaller error than the best combined model @xcite : model combination (figure 2 ) selects top k combined predictions and adds them to the set of predictions where the next layer can use another model combination step or just pick the best model according to the results on the training set. we use a two layer combination where the second layer is a combination of all of the predictions obtained. the last layer is an arg max. we also use stacking (stack) to build higher level models using predictions from base prediction models where they can also use the probability associated with the predictions (ting and witten, 1999). the stacking models use the predictions from predictors as features and additional selected features and build second level predictors. stacking with m predictors is depicted in figure 3 where predictions are used as features for the predictors in the next level. martins et al. ( 2017 ) used a hybrid stacking model to combine the word-level predictions from 15 predictors using neural networks with different initializations together with the previous features from a linear model. our stacking results also use top features from the data similar to the pass through feature of the stacking regressor of sklearn. 3 for these features, we con-r p trans gem mix stack before model combination, we further filter prediction results from different machine learning models based on the results on the training set to decrease the number of models combined and improve the results. a criteria that we use is @xmath0.875 since mraer computes the mean relative rae score, which we want to be less than 1. in general, the combined model is better than the r p mae rmse ## conclusion referential translation machines pioneer a language independent approach and remove the need to access any task or domain specific information or resource and can achieve top performance in automatic, accurate, and language independent prediction of translation scores. we present rtm results with ensemble models and stacking.