D iv EMT : Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages (2022)

Abstract: We introduce DivEMT, the first publicly available post-editing study of Neural Machine Translation (NMT) over a typologically diverse set of target languages. Using a strictly controlled setup, 18 professional translators were instructed to translate or post-edit the same set of English documents into Arabic, Dutch, Italian, Turkish, Ukrainian, and Vietnamese. During the process, their edits, keystrokes, editing times and pauses were recorded, enabling an in-depth, cross-lingual evaluation of NMT quality and post-editing effectiveness. Using this new dataset, we assess the impact of two state-of-the-art NMT systems, Google Translate and the multilingual mBART-50 model, on translation productivity. We find that post-editing is consistently faster than translation from scratch. However, the magnitude of productivity gains varies widely across systems and languages, highlighting major disparities in post-editing effectiveness for languages at different degrees of typological relatedness to English, even when controlling for system architecture and training data size. We publicly release the complete dataset including all collected behavioral data, to foster new research on the translation capabilities of NMT systems for typologically diverse languages.

## introduction recent advances in neural language modeling and multilingual training have prompted a widespread adoption of machine translation (mt) technologies across an unprecedented range of world languages. while the benefits of state-of-the-art mt for crosslingual information access are undisputed @xcite , its usefulness as an aid to professional translators varies considerably across domains, subjects and language combinations @xcite . in the last decade, the mt community has been including an increasing number of languages in its automatic and human evaluation efforts @xcite . however, the results of these evaluations are typically not directly comparable across different language pairs for various reasons. first, referencebased automatic quality metrics are hardly comparable across different target languages @xcite . secondly, human judgments are collected independently for different language pairs, making their cross-lingual comparison vulnerable to confounding factors such as tested domains and training data sizes. similarly, recent work on nmt post-editing efficiency has focused on specific language pairs such as english-czech @xcite , german-italian, german-french @xcite and english-hindi @xcite , but a controlled comparison across a set of typologically diverse languages is still lacking. in this work, we assess the usefulness of stateof-the-art nmt in professional translation with a strictly controlled cross-language setup (figure 1 ). specifically, professionals were asked to translate the same english documents into six typologically different languages (arabic, dutch, italian, turkish, ukrainian, and vietnamese) using the same platform and guidelines. three translation modalities were adopted: human translation from scratch (ht), post-editing of google translate's translation (pe 1 ), and post-editing of mbart-50's translation (pe 2 ), the latter being a state-of-the-art opensource, multilingual nmt system. in addition to post-editing results, subjects' fine-grained editing behavior -including keystrokes and time information -was logged to measure productivity and effort across languages, systems and translation modalities. finally, translators were asked to complete a qualitative assessment regarding their perceptions of mt quality and post-editing effort. the resulting divemt dataset is to our best knowledge the first public resource allowing a direct comparison of professional translators' productivity and fine-grained editing information across a set of typologically-diverse languages. divemt is publicly released alongside this paper as a unique resource to study the language-and system-dependent nature of nmt advances in realworld translation scenarios. post-editing nmt measuring post-editing effort across its temporal, cognitive, and technical dimensions (krings, 2001) is a well-established way to assess the effectiveness and efficiency of mt as a component of specialized translation workflows. seminal post-editing studies highlighted an increase in translators' productivity following mt adoption @xcite @xcite @xcite . however, they also struggled to identify generalizable findings due to confounding factors like output quality, content domains, and high variance across language pairs and human subjects. with the advent of nmt, productivity gains of the new approach were extensively compared to those of smt, the highly-customized dominant paradigm at the time @xcite @xcite . initial results were promising for nmt due to its better fluency and overall results. moreover, translators were shown to prefer nmt over smt for postediting, although a pronounced productivity increase was not always present. more recent work highlighted the productivity gains driven by nmt post-editing in a wider array of languages that were previously challenging for mt, such as english-dutch @xcite , english-hindi @xcite , english-greek (stasimioti and sosoni, 2020), english-finnish and english-swedish (koponen et al., 2020), all showing a considerable variance among language pairs and subjects. interestingly, @xcite found nmt postediting speed to be comparable to translation from scratch in english-czech, and highlighted a disconnect between moderate increases in automatic mt quality metrics and better post-editing productivity. in sum, research on post-editing nmt generally reports increased fluency and output quality, but productivity gains are hardly generalizable across language pairs and domains. importantly, to our knowledge, no previous work has studied nmt post-editing over a set of typologically different languages while controlling for the effects of content types and domains, nmt engines, and translation interfaces. ## the divemt dataset divemt's main purpose is to assess the usefulness of state-of-the-art nmt for professional translators and to study how this usefulness varies across target languages with different typological properties. we present below our data collection setup, which strikes a balance between simulating a realistic professional translation workflow and maximizing the comparability of results across languages. ## conclusions in this work we introduced divemt, the outcome of a post-editing study spanning two state-of-theart nmt systems, 18 professional translators and six typologically diverse target languages under a unified setup. we leveraged divemt's behavioral data to perform a controlled cross-language analysis of nmt post-editing effort along its temporal and editing effort dimensions. the analysis reveals that nmt drives significant improvements in productivity across all the evaluated languages, but the magnitude of these improvements depends heavily on the language and the underlying nmt system. in this setting, productivity measurements across modalities were found to be generally consistent with the recorded editing patterns. our results indicate that translators working on language pairs with significant post-editing productivity gains, on average, perform fewer edits and accept more machinegenerated translations without any editing. we also observed a disconnect between post-editing productivity gains and mt quality metrics collected for the same nmt systems. finally, low source-language relatedness and target morphological complexity seem to hinder productivity when nmt is adopted, even in settings where system architecture and training data are controlled for. in our qualitative analysis, translators' perception of post-editing usefulness was found to be strongly shaped by problematic mistranslations. languages showing large productivity gains for both nmt systems were the only ones associated with a positive perception of pe-mediated gains, as opposed to mixed or negative opinions for other translation directions. in future work, a more fine-grained analysis of the types of edits conducted by the translators, and their differences across languages, could shed more light on our current findings.