Towards a post-editing recommendation system for S panish- B asque machine translation (2018)

Abstract: The overall machine translation quality available for professional translators working with the Spanish–Basque pair is rather poor, which is a deterrent for its adoption. This work investigates the plausibility of building a comprehensive recommendation system to speed up decision time between post-editing or translation from scratch using the very limited training data available. First, we build a set of regression models that predict the post-editing effort in terms of overall quality, time and edits. Secondly, we build classification models that recommend the most efficient editing approach using post-editing effort features on top of linguistic features. Results show high correlations between the predictions of the regression models and the expected HTER, time and edit number values. Similarly, the results for the classifiers show that they are able to predict with high accuracy whether it is more efficient to translate or to post-edit a new segment.

## introduction although machine translation (mt) quality is getting better every day, neither regular users nor professional translators can blindly trust the correctness of a translation. therefore, providing them with information about the quality together with the actual translation seems sensible. we could argue that this is relevant for regular users, who c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. in this paper, we specifically focus on the case of spanish-to-basque professional translators. note that mt quality for this language pair can be considered relatively poor @xcite -at least that provided by freely accessible systems such as itzultzailea foot_0 or google translate foot_1 -and as a result, mt in the professional domain is very rarely used @xcite . in this context, we investigate whether we could build estimation models that may prove informative for translators and help with the integration of mt technology in this sector. to that end, we use a small set of data collected in a post-editing workshop, where postediting seems to benefit productivity at times. we first aim at providing professional translators with indicators of estimated work to guide their decision whether to post-edit or translate from scratch. for this, we build a set of regression models to estimate indicators of post-editing effort (overall mt quality, time and edits) which we obtain from data solely consisting of post-editing work. results show high correlations over 0.70 between real and estimated indicators. nevertheless, it is undeniable that a recommendation model that suggests the most efficient editing approach would be a more direct way to help in such process. the recommendation could be used either to opt for the most efficient approach during editing or to filter out mt output before the editing phase starts. thus, we build classification models using linguistic features to recommend the editing approach that increases the productivity the most. however, given the low accuracy of the classifiers, we try to improve them by adding specific post-editing effort features. as this information is only available once the editing is completed, we estimate it using the above-mentioned regression models. results show a large increase in the capacity of the classifiers to provide the correct editing approach even considering the loss of accuracy introduced by the regression models. the remaining of the paper is structured as follows. a short overview of related work is presented in section 2. in section 3 we describe the data sets and features used to train the models while the experimental set-up is outlined in section 4. section 5 and section 6 present the results for the regression and classification models, respectively. finally, section 7 summarizes the main conclusions and possible lines of future work. ## background in this section we present an overview of the quality and post-editing effort indicators studied in the literature. @xcite brought confidence estimation techniques, mainly used in speech recognition until then, to the area of mt as they considered that these could help in filtering translations for post-editing, among other tasks. they built models for sentence-level annotation by training regressors and classifiers to predict nist and wer values. whereas the tasks themselves proved interesting, experiments revealed that estimated automatic metrics did not match human annotations of quality or post-editing effort. similar results were reported by @xcite , who used a number of mt systemindependent and mt system-dependent features to train a regression algorithm to estimate both nist and human scores. the models performed well for human annotations, but once again, correlations with automatic metrics were not as successful. from then on, @xcite tested the use of ter and hter @xcite , which supposedly consider the actual postediting work translators perform more closely, to build the estimation models. this time, the models correlated well with human annotations of postediting effort. for that reason, hter was established as the global quality indicator in quality estimation (qe) tasks and remains so today, despite attempts at looking for alternative ways of measuring quality @xcite . since then, a number of authors have worked on building models to provide translators with useful information. some have tried to describe postediting time @xcite whereas others have focused on selecting the best mt output from a pool of candidates @xcite , or on recommending whether a source segment should be tackled using a mt candidate or a translation memory candidate @xcite . however, it could be argued that the main bulk of research in quality estimation has been shaped by the yearly qe shared task, in place since 2012. in its first year, participants focused on correlating estimation models with manual annotations of quality defined as 5 levels of post-editing effort @xcite . in 2013 and 2014, the goals were broadened and tasks involved predicting hter and post-editing time and ranking mt candidates @xcite . since then, however, efforts have mainly addressed hter and even if submissions for other indicators such as post-editing time and keystrokes have been welcome, no results have been published on these aspects. in order to provide professionals with a wider set of pointers that guides the translation task, in this paper we expand the post-editing effort indicators. specifically, we propose to create a recommendation system that (1) estimates the quality of the mt output as defined by hter, (2) predicts post-editing effort according to time, and the type and number of edits, and (3) recommends the editing approach for a particular segment by classifying it for either post-editing or translation from scratch. ## data collection and processing unlike for other mainstream language pairs, no readily-available data exists to train quality estimation models for the spanish-basque pair. for this purpose, therefore, we adapted post-editing data collected in a workshop for professional translators run in 2015. in this section we describe the data and the linguistic features that we used to build the estimation models. ## experimental set-up in this section we explain the experiments carried out to predict the mt quality and the post-editing effort required to transform the mt output into the desired quality standard. ## results of regression models for quality and post-editing work in this section we present the regression models that aim to predict the post-editing effort. we report the results for each indicator, namely, overall quality, time and edits, separately using both the pe-17 and pe-107 data sets. ## results of classifiers for editing approach in this section we present the results for the classifiers that aim to predict the editing approach, postediting or translation, a translator should follow when addressing a new segment. we report results with and without additional linguistic features and also analyse the impact of using post-editing ef-fort indicators as features. we predict label-sets of varying numbers of classes. ## conclusions and future work in this paper we tested the feasibility of training a number of estimation models that go beyond the usual mt quality level to build a recommendation system that helps speed up the decision time of professional translators to decide whether to postedit or translate. in particular, we studied if reasonable results could be obtained for the spanish-basque pair, for which mt quality is low, and thus not widely used within the professional sphere. we trained regression models to predict hter, time, types and total number of edits as indicators of post-editing effort using a limited data set. we show that relatively high correlation coefficients can be achieved for almost all indicators. the total edit number seems the easiest to predict whereas accuracy is lower for each of the individual types, particularly for deletions. results also reveal that adding pos and dependency relation frequencies as features did not generally improve the majority of our models. k-nn was the best performing algorithm, with the best results for hter and all the models involving edits and with no contribution from the new linguistic features. for the time model, lr and svr performed best, obtaining marginal gains with the new features. besides providing post-editing effort indicators, we also trained a classification model that would recommend translators the editing approach to take. whereas the baseline models performed poorly, we showed that including post-editing effort indicators as features largely improves the results. as this information is not available for new segments, we successfully used previously trained regression models to add these features in new test sentences. k-nn, cart and mlp consistently show the best performance across all the data sets. given the good results achieved, the next step would involve tuning and testing the models in further data sets. our aim is to investigate to what extent hter, post-editing time and edit types are valuable indicators for professionals translators.