Understanding Pre-Editing for Black-Box Neural Machine Translation (2021)

Abstract: Pre-editing is the process of modifying the source text (ST) so that it can be translated by machine translation (MT) in a better quality. Despite the unpredictability of black-box neural MT (NMT), pre-editing has been deployed in various practical MT use cases. Although many studies have demonstrated the effectiveness of pre-editing methods for particular settings, thus far, a deep understanding of what pre-editing is and how it works for black-box NMT is lacking. To elicit such understanding, we extensively investigated human pre-editing practices. We first implemented a protocol to incrementally record the minimum edits for each ST and collected 6,652 instances of pre-editing across three translation directions, two MT systems, and four text domains. We then analysed the instances from three perspectives: the characteristics of the pre-edited ST, the diversity of pre-editing operations, and the impact of the pre-editing operations on NMT outputs. Our findings include the following: (1) enhancing the explicitness of the meaning of an ST and its syntactic structure is more important for obtaining better translations than making the ST shorter and simpler, and (2) although the impact of pre-editing on NMT is generally unpredictable, there are some tendencies of changes in the NMT outputs depending on the editing operation types.

## introduction recent advances in machine translation (mt) have greatly facilitated its practical use in various settings from business documentation to personal communication. in many practical cases, mt systems are used as black-box and one well-tested approach to make use of a black-box mt is preediting, i.e., modifying the source text (st) to make it suitable for the intended mt system. the effectiveness of pre-editing has so far been demonstrated in many studies @xcite @xcite . a study focusing on statistical mt (smt) has also shown that more than 90% of an st can be rewritten into a text that can be machine-translated with sufficient quality @xcite , exhibiting the potential of the pre-editing approach. however, the feasibility and possibility of preediting for neural mt (nmt) has not been examined extensively. while efforts have recently been invested in the implementation of pre-editing strategies for black-box nmt settings, achieving improved mt quality (e.g., @xcite , the potential gains of preediting remain unexplored. notably, the impact of pre-editing on black-box mt is unpredictable in nature. in particular, nmt models trained in an end-to-end manner can be sensitive to minor modifications of the st @xcite , which may affect the feasibility of pre-editing. in short, while pre-editing has been implemented in practical mt use cases, what pre-editing is and how it works with black-box nmt systems remain open questions. to explore the possibility of preediting and its automation, in this study, we provide fine-grained analyses of human pre-editing practices and their impact on nmt. we systematically collected pre-editing instances in various conditions, i.e., translation directions, nmt systems, and text domains ( §3). we then conducted in-depth analyses of the collected instances from the following three perspectives: the characteristics of the pre-edited st ( §4), the diversity of pre-editing operations ( §5), and the impact of pre-editing operations on the nmt outputs ( §6). the findings of these analyses provide useful insights into the effective and efficient implementation of pre-editing for the better use of black-box nmt systems in the future, as well as the robustness of current nmt systems when sts are manually perturbed. ## related work pre-editing is the process of rewriting the source text (st) to be translated in order to obtain better translations by mt. though the scope of effective pre-editing operations depends on the downstream mt system and there is no deterministic relation between pre-editing operations and the quality of mt output, its effectiveness has been demonstrated for various translation directions, mt architectures, and text domains. manual pre-editing has long been implemented in combination with controlled languages @xcite @xcite . in the period of rule-based mt (rbmt), pre-editing was considered as a promising approach since the behaviour of rbmt is more predictable and controllable. for example, @xcite examined the impact of english controlled language rules on two different mt engines, revealing the rules of high effectiveness. the pre-editing approach with controlled languages has also been tested for statistical mt (smt) @xcite @xcite . these studies developed or utilised a set of controlled language rules for rewriting st. while these rule sets are optimised for particular mt systems and differ from each other, we can observe some shared characteristics among them. in particular, rules that prohibit long sentences (e.g., of more than 25 words) are widely adopted in the existing rule sets @xcite . automation of pre-editing is also an important research field in natural language processing. semi-automatic tools such as controlled language checkers @xcite and interactive rewriting assistants @xcite were developed to facilitate manual pre-editing activities. fully automatic pre-editing has long been explored (e.g., @xcite @xcite . in particular, many researchers have examined methods of reordering the source-side word order as a pretranslation processing @xcite @xcite . while the reordering approach has generally proven effective for smt, its effectiveness for nmt is not obvious; negative effects have even be reported @xcite . in recent years, techniques of automatic text simplification have been applied to improve nmt outputs @xcite . the underlying assumption of these studies is that simpler sentences are more machine translatable. previous studies have investigated various preediting methods from different perspectives, focusing on different linguistic phenomena. indeed, individual research has led to improved mt results. however, what is crucially needed is a broad understanding of what pre-editing is and how it works. for example, @xcite addressed this issue by collecting instances of bilingual preediting, i.e., pre-editing st while referring to its mt output, done by human editors and analysing them in detail. they demonstrated the maximum gain of pre-editing for an smt and provided a comprehensive typology of editing operations. nevertheless, their study has two major limitations: (1) recent nmt was not examined, and (2) practical insights for better practices of pre-editing were not sufficiently presented. nmt models trained in an end-to-end manner behave very differently from smt and rbmt, which, in turn, affects pre-editing practices. as reported in several studies, despite their rapid improvement, nmt models are still vulnerable to input noise @xcite @xcite . the pre-editing operations identified in previous studies are not necessarily effective for current blackbox nmt systems. 1 for example, @xcite adopted nine controlled language rules 2 and evaluated their impact on the mt output for german-to-english translation in the technical domain. the human evaluation results revealed that these rules improved the performance of the rbmt, smt, and hybrid systems, but did not have positive effects on the nmt system. hiraoka and yamada (2019) demonstrated the effectiveness of the following three pre-editing rules in improving japanese-to-english ted talk subtitle 5. perfect information in the original text has been completely translated. there are no grammatical errors in the translation. the word choice and phrasing are natural even from a native speaker's point of view. ## characteristics of pre-edited sentences to understand the differences between the original and pre-edited sts, in this section, we describe their general linguistic characteristics. here, we compare the org-st and the best-st that achieved a satisfactory mt result in order to elicit the features of machine translatable st. ## impact of pre-editing on neural machine translation this section investigates how pre-editing operations affect the nmt output. as indicated in §2, nmt systems still lack robustness, and minor modifications of the input would drastically change the output. from the practical viewpoint of deploying pre-editing, predictability is an important object to pursue. here, we examine the impacts of minimum edits of the st on the nmt output. to measure the amount of text editing, hereafter, we use the translation edit rate (ter), which is calculated by dividing the number of edits (insertion, deletion, substitution, and shift) required to change a string into the reference string by the average number of reference words @xcite . for any consecutive pair of sts or their corresponding mt outputs, we used the chronologically later version as the reference. for word-level tokenisation, we used mecab for japanese, nltk foot_4 for english, jieba 13 for chinese, and konlpy foot_6 for korean. ## conclusion and outlook towards a better understanding of pre-editing for black-box nmt settings, in this study, we collected instances of manual pre-editing in various conditions and conducted in-depth analyses of the instances. we implemented a human-in-the-loop protocol to incrementally record minimum edits of st for all combinations of three translation directions, two nmt systems, and four text domains, and obtained a total of 6,652 instances of manual pre-editing. since more than 95% of the sts were successfully pre-edited into one that led to a satisfactory mt quality, our collected instances contain empirical, tacit human knowledge on the effective use of black-box nmt systems. we also investigated the collected data from three perspectives: the characteristics of the pre-edited sts, the diversity of pre-editing operations, and the impact of pre-editing operations on the nmt output. the remarkable findings can be summarised as follows: in future work, we plan to explore the effective implementation of pre-editing. the findings of this study provide a broad overview of the range of pre-editing operations and their expected benefits, which enables us to find feasible pre-editing solutions in practical use cases of black-box nmt systems. to develop automatic pre-editing tools using a collection of pre-editing instances, we need to handle the data insufficiency issue in machine learning, filling the gap between the training data and targeted black-box mt systems. moreover, as our pre-editing instances contain a wide variety of perturbations in the st, they can also be used to evaluate the robustness of mt systems, which can lead to advances in mt research. we aim to jointly improve the two wheels of translation technology: pre-editing and mt.