PE effort and neural-based automatic MT metrics: do they correlate? (2023)

Abstract: Neural machine translation (NMT) has shown overwhelmingly good results in recent times. This improvement in quality has boosted the presence of NMT in nearly all fields of translation. Most current translation industry workflows include postediting (PE) of MT as part of their process. For many domains and language combinations, translators post-edit raw machine translation (MT) to produce the final document. However, this process can only work properly if the quality of the raw MT output can be assured. MT is usually evaluated using automatic scores, as they are much faster and cheaper. However, traditional automatic scores have not been good quality indicators and do not correlate with PE effort. We analyze the correlation of each of the three dimensions of PE effort (temporal, technical and cognitive) with COMET, a neural framework which has obtained outstanding results in recent MT evaluation campaigns.

## introduction in the last decade, mt has steadily increased its presence in all fields of translation. this is mainly due to the improvements in quality following the advances in nmt. results of a recent language survey identify post-editing as the second most demanded task among language providers and the activity with the highest growth potential, 64% @xcite . for many language combinations, translators edit, modify and correct the raw mt output to produce a final version. however, this process can only work properly if the quality of the raw mt output can be assured. to assess the quality of the mt output both manual and automatic metrics are currently used. on the one hand, manual evaluations include sentence ranking, fluency and adequacy, direct assessment (da) @xcite , and explicit error analysis, such as the ones based on the multidimensional quality metrics (mqm) framework @xcite . even though most of these evaluations produce quite reliable metrics, they have a high cost in time and resources @xcite , which makes it complicated to use in a daily basis to assess the quality of mt systems. they also suffer from low inter-and intraannotator agreements @xcite . on the other hand, automatic evaluations produce quick results. even though these metrics were originally conceived as a way to compare two systems, in most scenarios they are used as the only means to assess the quality of an mt engine. automatic scores usually show correlation with human judgments of translation @xcite , even though they have been frequently questioned as a way to assess mt output @xcite , especially when they are used to compare high-quality systems @xcite . the most usual automatic metrics currently used, such as bleu @xcite , or ter @xcite are useful but present clear limitations and do not correlate with pe effort @xcite . since the seminal work by @xcite , pe effort includes three dimensions: temporal effort (time spent translating), technical effort (keystrokes and all editing actions) and cognitive effort (mental processes taking place while translating). even though all three are related, there is not a single measure which includes them all @xcite . in recent times, new automatic metrics based on neural networks, such as bleurt @xcite , bertscore @xcite and comet @xcite have shown outstanding results in recent evaluation campaigns @xcite @xcite based on mqm evaluations. we analyse if comet, one of the best-performing metrics in recent campaigns, correlates better with the three dimensions of pe effort and, thus, could be used as a way to predict pe effort. to do so, we collect pe information from ten translators who post-edited a news article from english into spanish translated with two different mt engines. then we study the correlation of each of the pe effort dimensions with comet using pearson product-moment correlation. ## experimental set-up 3.1 mt engines to compare the pe effort measures and automatic scores, we decided to collect information from two different mt engines to avoid any bias produced by the mt model. we use a known commercial mt engine (deepl) foot_0 and an mt engine trained by the authors to translate from english into spanish two different fragments from a news article. for the nmt engine trained by the authors, we first compiled a parallel corpus originated from global voices. in order to do so, we downloaded all the news articles written in english which had a known translated version into spanish from 2004 until 2022. in order to align all the texts, we used mtuoc-aligner foot_1 , which is based on the sbert strategy. that is, we segment and align all the texts written in english and spanish for a specific year without taking into account the news article in which they appear. thus, the task is a search of translated segments in comparable corpora. the next step includes a cleaning process to produce a parallel corpus of 791,959 unique parallel segments. since this number of segments is not enough to train a neural mt system, we selected 20,000,000 million segments from the paracrawl v9 english-spanish corpus using mtuoc-corpuscombination foot_2 . this selection is based on a language model computed from the source segments of the compiled global voices corpus, so the selected segments are expected to be similar segments to those found in the news domain. using this combination, we produced a final training corpus of a total of 20,781,959 segments. from the corpus, we reserved 5,000 segments for validation and 5,000 segments for evaluation. in this way, the training was performed using a combination of the global voices corpus and selected segments from paracrawl, but the validation and the evaluation was carried out using segments from the global voices corpus. we used sentencepiece @xcite to process the corpus using the following parameters: joining languages: true; model type: bpe; vocabulary size 64,000; vocabulary threshold: 50. the (sub)word alignments of the training corpus have been calculated using eflomal @xcite in order to use guidedalignment in the training. the nmt system was trained using the mariannmt toolkit (junczys-dowmunt et al., 2018) with a transformer configuration. two validation metrics were used: bleu-detok and cross-entropy. the early-stopping criterion was set to 5 on any of the metrics, and the validation frequency was set to 5,000. we assessed the quality of the two nmt systems using some of the most frequently-used automatic metrics. for the evaluation, we used mtuoc-eval foot_3 , a tool offering a wide range of automatic evaluation metrics. in ## results we used all the data collected while each of the translators post-edited using posedion to calculate the pe effort indicators. for each segment of the post-edited texts, we calculated the three dimensions of pe effort. for cognitive effort calculated with long pauses, the best values are obtained by hbleu for the marian set, and hter for the deepl set. both measures show a moderate correlation with a high standard error. for the temporal effort calculated with the normalized time, the same two metrics yield again the best results, even though they show a low and moderate correlation with a much lower steyx value. for the technical effort calculated with normalized keystrokes, the best values are obtained by hter, which show a high and moderate correlation with a very low standard error. it is important to note that neither hcomet nor comet perform well in terms of correlation with effort indicators when calculated segment by segment. we must keep in mind, however, that the values of comet related measures are dependent on the models used, and different models can score differently on the same data. furthermore, results differ for each of the pe effort indicators but also for the two mt engines used. even so, measures which take into account the pe version as hypothesis seem to show a moderate correlation, which could suggest they can give an approximate indication of the pe effort necessary. ## conclusions and future work in this paper we have presented the results of an experiment aiming to assess the correlation of several automatic metrics with the three dimensions of post-editing effort: temporal effort, technical effort and cognitive effort. the main goal was to check whether a relatively new neural-based metric, comet, correlates better than other widely used metrics, such as hbleu and hter, and could be used as a predictor for pe effort. the limitations of this paper include the length of the text post-edited and the total number of translators who have participated in the pe task. however, the results obtained from this small sample show that comet does not correlate for any of the pe effort indicators. hbleu and hter show a moderate to strong correlation for some of the indicators, but low for others. this would confirm the results of previous research stating the lack of correlation between all three dimensions of effort. the variability depending on the mt model could point to the types of errors produced by the mt engines and the different pe effort implied in correcting them. for future experiments, we will collect data from a larger number of translators and larger texts, and we will train comet models which can correlate better with some or all the pe effort indicators. the final goal would be to obtain a measure which could predict better pe effort than the current automatic measures used in the translation industry.