Post-editors as Gatekeepers of Lexical and Syntactic Diversity: Comparative Analysis of Human Translation and Post-editing in Professional Settings (2024)

Abstract: This paper presents a comparative analysis between human translation (HT) and post-edited machine translation (PEMT) from a lexical and syntactic perspective to verify whether the tendency of neural machine translation (NMT) systems to produce lexically and syntactically poorer translations shines through after post-editing (PE). The analysis focuses on three datasets collected in professional contexts containing translations from English into French and German into French. Through a comparison of word translation entropy (HTRa) scores, we observe a lower degree of lexical diversity in PEMT compared to HT. Additionally, metrics of syntactic equivalence indicate that PEMT is more likely to mirror the syntactic structure of the source text in contrast to HT. By incorporating raw machine translation (MT) output into our analysis, we underline the important role post-editors play in adding lexical and syntactic diversity to MT output. Our findings provide relevant input for MT users and decision-makers in language services as well as for MT and PE trainers and advisers.

## introduction post-editing (pe) has now largely proved to be a good alternative to purely human translation (ht) in professional contexts. by allowing certain productivity gains without negatively affecting the quality of the final translation @xcite , mt and pe have found their place in professional translation workflows. nevertheless, translators often express mixed feelings towards mt. on one hand, the tool is appreciated for its help when dealing with high workloads and time constraints, but on the other hand, it is perceived as a threat to translation's creativity, originality and naturalness @xcite . these concerns are legitimate: numerous studies have revealed the nmt tendency to produce an output that is less lexically varied and syntactically closer to source text than ht @xcite @xcite @xcite @xcite . furthermore, some studies have found measurable differences on parallel corpora of ht and post-edited machine translation (pemt) in terms of lexical diversity, lexical density and sentence length, among others, suggesting the existence of a post-editese phenomenon @xcite @xcite . however, @xcite , demonstrated the difficulty of generalising these findings over different corpora, domains and language pairs, particularly when analysing authentic comparable corpora. such corpora, in which ht and pemt are the translations of different source texts, necessitate analysis with metrics that encompass the attributes of the source. although demanding, the study of authentic ht and pemt corpora is crucial for developing a detailed understanding of the distinct characteristics of pemt output in professional contexts. in this study, we compare the lexical and syntactic characteristics of authentic ht and pemt output produced in professional contexts relying on metrics suitable for the study of comparable corpora. whenever possible, raw mt output is added to the analysis to confirm initial premises on raw mt, as well as findings by previous studies. we measure the variety of translation solutions by automatically computing average word translation entropy (htra) @xcite and the syntactic equivalence between source and target based on three syntactic equivalence metrics from the as-tred library @xcite to answer the following research question: is the nmt tendency toward lexical impoverishment and source sentence structure mirroring discernible in the pemt final product? throughout our analysis, we make the following observations: ## datasets our experiment is based on three authentic datasets containing professional translations collected from in-house language services 1 . each dataset contains a balanced amount of ht and pemt segments with their respective source. dataset enfr1 was compiled from the same data as in @xcite and contains translations from english into french extracted from documents of the european investment bank (eib). dataset enfr2 and dataset defr are derived from the dataset described in @xcite . dataset enfr2 contains translations from english into french shared with us by a sports organisation based in switzerland, while dataset defr contains translations from german into french collected from an insurance company. for all datasets, raw mt used for pe came from various state-of-the-art nmt systems (generic and/or customised). ## examples loss of translation solution variety: to illustrate what a lower htra score concretely means, we present examples showing the translation solutions distribution for particular lemmas in figures 1 and 2 in appendix a. figure 1 shows the distribution of translation solutions for the noun "impact" in ht, pemt and raw mt within the enfr2 dataset. while pemt and ht exhibit an equal number of translation solutions, their frequencies are more evenly dispersed in ht, resulting in a htra score of 2.84 for ht compared to 2.79 for pemt. in contrast, raw mt yields only two distinct translation solutions (with the absence of translation considered as a translation "choice"), where one solution overwhelmingly dominates, ## conclusion this paper compares authentic sets of ht and pemt produced in three different professional contexts for the language directions english into french and german into french, with additional analysis incorporating raw mt output from deepl for two of the datasets. the objective is to compare ht and pemt in terms of lexical and syntactic variety to verify whether the general tendency of nmt systems to produce lexically and syntactically less varied output still shines through after a pe step performed by professional translators. using htra for the lexical aspects and astred, sacr and dependency label changes @xcite for the syntactic aspects, we note the strong tendency of raw nmt (in this case deepl) to produce lexically less varied translations that tend to mirror the source sentence structure. this tendency is strongly attenuated by the pe step, with pemt output being generally closer to ht than to raw mt on both aspects. this indicates that posteditors presumably add significant levels of lexical and syntactic variety to the mt output ("presumably", because raw mt under analysis is not the one originally used for pemt, but we assume it reflects the general level of lexical and syntactic variation of nmt systems). still, the final pemt output does not systematically reach the same level of variety as ht, especially on the lexical level. for all datasets pemt exhibits lower levels of translation solution variety. a tendency towards more syntactic equivalence between source and target in pemt is clear for one dataset but more nuanced for the two others. these findings are particularly relevant in contexts where lexical and syntactic variety are regarded as criteria for assessing translation quality. furthermore, our work highlights the crucial importance of pe, not only in ensuring the accuracy of the target text, but also in maintaining an adequate level of lexical diversity and syntactic naturalness in the final translation. while this aspect may seem unimportant for certain types of texts, it holds significant relevance for others. in many cases, (human) translation is not only about overcoming language barriers but also about producing "texts that satisfy the linguistic norms of a target culture and are adapted to the assumed knowledge of its reader" @xcite . it is also of utmost importance considering the fact that pemt output is likely to be re-used to train nmt systems, and therefore to amplify over and over the already existing biases. finally, we emphasise the relevance of our findings for the improvement of post-editing training programs and guidelines. while translators are still today often advised to stick to the taus pe guidelines @xcite and to not intervene on the stylistic level, we are convinced that adding lexical and syntactic diversity (even when not strictly necessary from micro-level perspective) to mt output is essential to preserve the quality of translated text at the macro-level.