Studying Post-Editese in a Professional Context: A Pilot Study (2022)

Abstract: The past few years have seen the multiplication of studies on post-editese, following the massive adoption of post-editing in professional translation workflows. These studies mainly rely on the comparison of post-edited machine translation and human translation on artificial parallel corpora. By contrast, we investigate here post-editese on comparable corpora of authentic translation jobs for the language direction English into French. We explore commonly used scores and also proposes the use of a novel metric. Our analysis shows that post-edited machine translation is not only lexically poorer than human translation, but also less dense and less varied in terms of translation solutions. It also tends to be more prolific than human translation for our language direction. Finally, our study highlights some of the challenges of working with comparable corpora in post-editese research.

## introduction much progress has been made since the seminal paper by @xcite introduced the notion of translation universals and suggesting "to capture" the differences between original and translated language using comparable electronic corpora. corpora of translated texts have been widely studied since then and research by @xcite , @xcite and @xcite , among others, have revealed the existence of translationese features. following this, a new type of corpus-based translation studies has recently emerged together with the boom of neural machine translation (nmt) systems and their large integration into professional translation workflows. those new studies are interested in the phenomenon of machine translationese and posteditese, the latter being defined as "the expected unique characteristics of a post-edited text that set it apart from a [human] translated text" @xcite . our study falls within this area of research and focuses of post-editese in professional context. first, we provide a short literature review of previous work on post-editese that will allow us to highlight the novel aspects of our research, as well as the common components that could constitute the basis for the development of a consistent methodology for the study of post-editese. subsequently, we present the main goals of our study, as well as our research questions. we then describe the comparable corpus used for our pilot study and discuss the main advantages and drawbacks of such a corpus for the study of post-editese. following this, we describe the experiments conducted and results obtained. finally, we provide a summary of our findings and some perspectives for the future continuation of this work. ## related work this section presents some of the recent studies investigating the differences between human and raw and/or post-edited machine translation output. @xcite conducted a study on terminological variation and cognate translations in human translation (ht) and post-edited machine translation (pemt) produced by students on a text of approximately 150 words. they observed less variation in pemt than in ht and a priming effect of machine translation (mt) in pemt on the terminological level. they also found that pemt tends to contain more cognate translations. similar results were observed by @xcite in their study comparing two different corpora (each approximately 500 000 words) of medical summaries translated from english into french with or without statistical machine translation (smt). they noted differences between ht and pemt regarding the frequencies of certain words or phrases, as well as a tendency towards standardization of the translations in pemt, as indicated by an over-representation of the most frequent translation solutions. they also observed a higher number of cognate translation or formal equivalences in pemt. finally, they pointed out that ht had a greater expanding ratio than pemt, meaning that ht tends to produce longer translations. @xcite attempted to investigate if ht and pemt could be identified as such by human evaluators as well as by a classifier, which would indicate the existence of a post-editese phenomenon. neither the human evaluators, nor the classifier were able to accurately distinguish ht from pemt. however, the methodology applied to build the classifier brought to light some features that might be useful to discriminate ht and pemt, such as type-token ratio, average word length, ratio of long words or the percentage of frequent words. in his study conducted with translation students between @xcite @xcite compared ht and pemt of wikipedia abstracts from english into italian. while analyzing a set of 41 source n-grams, he noted that the most frequent ht solutions tend to be over-represented in pemt showing "an apparent normalization and homogenization of the choices made by post-editors" compared to ht. in consecutive studies, @xcite and @xcite investigated posteditese features on a news corpus and two literary excerpts (approximately 5000 to 6000 tokens each) by comparing the source, mt, ht and pemt versions for the language direction english into brazilian portuguese. three translation universals (simplification, explicitation and convergence) were investigated through features such as lexical richness, lexical density, mean sentence length, length ratio, number of pronouns and vari-ance scores for the different features. some significant differences between ht and pemt were observed for certain features, but the results were not homogeneous across the different datasets. for the variance scores, they observed that mt and pemt tended to converge for the scores investigated, meaning that they are more similar to each other than they are to the source or ht. although they are good indicators of the existence of a form of post-editese, these mixed results demonstrate that the candidate features of post-editese can be highly influenced by the corpus under investigation. toral 2019 also investigated the simplification translation universal, together with normalization and interference, using lexical richness, lexical density, length ratio and comparison of part-ofspeech (pos) sequences. the experiment was conducted on three different datasets (ranging from 100 to 1000 sentence pairs), five language directions (involving en, de, es, fr, zh) and three types of mt architectures (rule-based, smt and nmt). he observed that pemt texts tended to be lexically simpler, to have a lower lexical density and to have sentences closer to the source text in terms of length. pos sequences also tended to be more similar to the typical pos sequences of the source language. according to the author, these results are evidences of the existence of the posteditese that is a form of exacerbated translationese. the above-mentioned studies present a certain number of similarities both in terms of the corpora or the features under investigation. for instance, it can be noted that they are all, except for one, based on parallel target corpora, i.e., translations of the same source text produced with different translation modes (mt/pemt and ht). as for the features under investigation, we remark a strong representation of features related to lexical richness and diversity (i.e., type/token ratio or the variation of translation solutions), as well as to target text length (i.e., word length, sentence length ratio, text length). ## goals and research questions the aim of our study is to investigate whether some of the findings of previous studies on posteditese can be confirmed on a corpora of authentic ht and pemt translation projects for the language direction english into french. we intend to apply some of the metrics that have proven to be good indicators of post-editese so far and compare our results with the existing hypothesis on posteditese. we also propose the use of a novel metric borrowed from translation process research to study post-editese through the lens of translation variation between ht and pemt. with this work, we hope to contribute to the development of a consistent and reliable methodology for the study of post-editese and to encourage additional work on authentic data in this domain. the following research questions have guided our work: does the use of pemt instead of ht affect the final translation in terms of: ## conclusion and future work in this study, we applied some of the metrics commonly used in post-editese research to comparable corpora of authentic ht and pemt jobs for the language direction english into french. the aim of our study was to investigate if findings of previous studies could be confirmed on such a corpora. we studied the effect of the translation mode (ht or pemt) on lexical richness, lexical density, expanding ratio and adverb translation entropy. below is a summary of our main findings: lexical richness: pemt exhibits lower lexical richness than ht. this difference can partly be explained by the difference in lexical richness observed in the source corpora. however, the amplitude of these differences suggests an effect of the translation mode on lexical richness, with pemt producing lexically poorer translations. those results are coherent with previous finding on machine translationese and post-editese (see for instance @xcite lexical density: our results indicate a tendency toward a lower lexical density in pemt compared to ht. this is in line with the findings of @xcite , but, once again, the differences between target corpora are difficult to interpret due to the differences already existing in the source corpora. expanding ratio: the expanding ratio is much higher for pemt than ht, which means that for a given source sentence length, pemt tends to produce longer target sentences. further investigation with access to raw mt output is needed to uncover the reasons behind this target length discrepancy between ht and pemt. adverb word translation entropy: the htra computed for the list of selected adverbs reveals that pemt presents less variation in the translation solutions of adverbs, supporting the conclusion made by @xcite or @xcite that pemt leads to more uniform translations. this pilot study shows that some of the previously identified post-editese features can be found in authentic pemt jobs and proposes the use of a novel metric for measuring the translation variation in pemt. in addition, our study highlights the complexity of investigating post-editese on parallel corpora. apart from the difficulty of gaining access to authentic data (including raw mt), the question of the comparability of the corpora represents a major challenge. the fact that ht and pemt are not obtained from the same source corpus complicates the interpretation and the generalization of the results. increasing the size and the diversity of the corpora, as well as developing techniques to increase corpus comparability, might be interesting options to overcome these challenges. access to raw mt output could also be very helpful to facilitate the interpretation of the results. despite the challenges faced, we are still convinced that the study of post-editese on authentic data is essential to fully understand the implications and potential consequences on the language use of the currently massive adoption of nmt in the translation industry. in the next stage of our research, we will increase the size of our corpora by adding data from other language services and other domains. we also plan to investigate the htra metric more in depth by calculating scores for other categories and by checking their correlation with human judgement.