A User Study of the Incremental Learning in NMT (2020)

Abstract: In the translation industry, human experts usually supervise and post-edit machine translation hypotheses. Adaptive neural machine translation systems, able to incrementally update the underlying models under an online learning regime, have been proven to be useful to improve the efficiency of this workflow. However, this incremental adaptation is somewhat unstable, and it may lead to undesirable side effects. One of them is the sporadic appearance of made-up words, as a byproduct of an erroneous application of subword segmentation techniques. In this work, we extend previous studies on on-the-fly adaptation of neural machine translation systems. We perform a user study involving professional, experienced post-editors, delving deeper on the aforementioned problems. Results show that adaptive systems were able to learn how to generate the correct translation for task-specific terms, resulting in an improvement of the user’s productivity. We also observed a close similitude, in terms of morphology, between made-up words and the words that were expected.

## introduction despite its improvements and obtaining admissible results in many tasks, machine translation (mt) is still very far from obtaining automatic high-quality translations @xcite . thus, a human agent needs to supervise and correct the outputs generated by an mt system. this process is known as post-editing and is a common use case of mt in the industrial environment. as mt systems are continuously improving their capabilities, it has acquired major relevance in the translation market @xcite @xcite . throughout the post-editing process, new data are continuously generated. these new data have valuable properties-they are domain-specific training samples. thus, it can be leveraged to continuously adapt the sysc ⃝ 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. ## related work post-editing mt hypotheses is a practice that was adopted in the translation industry a long time ago (e.g., @xcite . its relevance grew as mt technology advanced and improved. the capabilities of mt post-editing have been demonstrated through many user studies @xcite @xcite . parallel to the rise of the post-editing protocol, using user post-edits to adapt mt systems has also attracted the attention of researches and industry. this was studied in the casmacat @xcite and mate-cat @xcite projects and phrase-based statistical mt systems based on online learning were developed @xcite . with the breakthrough in neural mt (nmt) technology @xcite @xcite , research shifted towards constructing adaptive systems via online learning in this post-editing scenario. the use of online learning to adapt an nmt system to a new domain with post-edited samples was proposed by @xcite and @xcite . other works refined these adaptation techniques and applied them to new use cases @xcite @xcite . the evaluation of mt post-edits is a hard topic that is currently being actively researched (e.g., @xcite @xcite . several works conducted user studies for mt post-editing systems, either phrase-based @xcite @xcite or nmt @xcite @xcite . moreover, two studies showed improvements in terms of productivity time and translation quality with the application of an online learning protocol @xcite . this latter study is tightly related to ours. we extend it by performing a finer-grained evaluation of the outputs of the adaptive systems. ## experimental framework as we extended the work of @xcite , we used their same data and systems. the task at hand consisted of a small medico-technical (description of medical equipment) corpus from their production scenario. it contains specific vocabulary from a very closed domain. it was conformed by two documents of 150 sentences, which contained 1.7 and 2.7 thousand words respectively. the translation direction was english to spanish. the system was trained using the data from wmt'13's translation task @xcite and samples selected by the feature decay selection technique @xcite . the data features are summarized in the system was built with opennmt-py @xcite , using a long short-term memory @xcite recurrent encoder-decoder with attention @xcite . all model dimensions were 512. the system was trained using adam @xcite with a fixed learning rate of 0.0002 @xcite and a batch size of 60. a label smoothing of 0.1 @xcite was applied. at inference time, we used beam search with size 6. the adaptation process followed the findings from @xcite . we tuned the hyperparameters for the adaptation process on our development set, under simulated conditions. for each new postedited sample, we applied two plain sgd updates, with a fixed learning rate of 0.05. as translation environment we used the one designed by @xcite . it connects our adaptive nmt engine with the sdl trados studio interface, which is used by the post-editors in our production workflow. in addition, it also allowed us to trace the productivity metrics and user behavior. ## user study in our study, we focus on the differences between static and adaptive systems based on three main aspects: the productivity of post-editors, the quality of post-edits and the generation differences. ## generation of made-up words on their feedback, the users reported that, in some cases, the system's hypothesis contained words which were not real words (e.g., "absolvido"). this phe-nomenon, although infrequent, was a bit cumbersome. most likely, it is caused by an incorrect segmentation of a word via the byte pair encoding process which, according to their frequency, splits words into multiple tokens. in order to assess its impact, we start by quantifying the issue. although it could be expected for the adaptive system-which has to deal with a higher number of outof-vocabularies, introduced by the user-to generate made-up words with a higher frequency, both systems behave similarly: on document 1 case, the static system generated 0.1% more made-up words and, in the other case (document 2), it was the adaptive system which generated 0.1% more made-up words. furthermore, when comparing both documents, we observe that, despite document 2 having a bigger vocabulary, both static systems generated the same percentage of made-up words. however, document 2's adaptive systems generated 0.2% of made-up words on average. most likely, since we did not have an in-domain corpus for training the systems (see section 3), the bigger the document's vocabulary is, the easier an out-ofvocabulary word may results in an incorrect subword segmentation. ## conclusions and future work in this work, we extended a previous user study of an adaptive nmt system. we conducted new experiments with the help of professional translators, and observed significant improvements of the translation quality-measured in terms of hter and hbleu-and significant improvements of the user's productivity-measured in terms of post-editing time and number of words generated. we also conducted, with the help of two additional professional translators, a human evaluation that verified the quality of the postedits generated during the user study. the users were pleased with the system. they noticed that corrections applied on a given segment generally were reflected on the successive ones, making the post-editing process more effective and less tedious. when comparing the translations generated by both kind of systems, we identified that adaptive systems were able to generate the correct translation of acronyms, entities relating a particular device and specific task terminology. an undesirable side effect mentioned by the users was the sporadic apparition of made-up words. we studied this phenomenon and reached the conclusion that due to the increase in the number of out-ofvocabularies as part of the post-editing process, adaptive systems suffer this problem more than static systems. furthermore, sometimes these made-up words are very similar, in morphological terms, to the correct words-such as a feminine word converted into its non-existent masculine equivalent-which made them harder to detect. however, the cognitive impact in the post-editors will need to be assesed before reaching categorical conclusions. in regards to future work, we should try to assess the cognitive impact of the made-up words phenomenon. we would also like to study the degradation of domainspecific terms, and analyze the impact on the amount of work required to post-edit subsequent sentences as the user provides corrected examples. additionally, we will integrate our adaptive systems together with other translation tools, such as translation memories or terminological dictionaries, with the aim of fostering the productivity of the post-editing process. with this feature-rich system, we would like to conduct additional experiments involving more diverse languages and domains, using domain-specialized nmt systems, testing other models (e.g., transformer, @xcite and involving a larger number of professional post-editors. finally, we also intend to implement the interactive-predictive machine translation protocol @xcite in our translation environment, and compare it with the regular post-editing process.