MMPE : A M ulti- M odal I nterface using H andwriting, T ouch R eordering, and S peech C ommands for P ost- E diting M achine T ranslation (2020)

Abstract: The shift from traditional translation to post-editing (PE) of machine-translated (MT) text can save time and reduce errors, but it also affects the design of translation interfaces, as the task changes from mainly generating text to correcting errors within otherwise helpful translation proposals. Since this paradigm shift offers potential for modalities other than mouse and keyboard, we present MMPE, the first prototype to combine traditional input modes with pen, touch, and speech modalities for PE of MT. Users can directly cross out or hand-write new text, drag and drop words for reordering, or use spoken commands to update the text in place. All text manipulations are logged in an easily interpretable format to simplify subsequent translation process research. The results of an evaluation with professional translators suggest that pen and touch interaction are suitable for deletion and reordering tasks, while speech and multi-modal combinations of select & speech are considered suitable for replacements and insertions. Overall, experiment participants were enthusiastic about the new modalities and saw them as useful extensions to mouse & keyboard, but not as a complete substitute.

## introduction & related work as machine translation (mt) has been making substantial improvements in recent years foot_0 , more and more professional translators are integrating this technology into their translation workflows @xcite . the process of using a pre-translated text as a basis and improving it to create the final translation is called post-editing (pe). while translation memory (tm) is still often valued higher than mt @xcite , a recent study by @xcite shows that professional translators chose pe of mt over pe of tm and translation from scratch in 80% of the cases. regarding the time savings achieved through pe, @xcite find that pe was on average 28% faster for technical translations, @xcite report productivity gains of 36% when using modern neural mt, and @xcite show that pe increases translation throughput for both professionals and lay users. furthermore, it has been shown that pe not only leads to reduced time but also reduces errors @xcite . switching from traditional translation to pe results in major changes in translation workflows @xcite , including the interaction pattern @xcite , yielding a significantly reduced amount of mouse and keyboard events @xcite . this requires thorough investigation in terms of interface design, since the task changes from mostly text production to comparing and adapting mt and tm proposals, or put differently, from control to supervision. while most computer-aided translation (cat) tools focus on traditional translation and incorporate only mouse & keyboard, previous research investigated other input modalities: automatic speech recognition (asr) for dictating translations has already been explored in the 90s @xcite and the more recent investigation of asr for pe @xcite even argues that a combination with typing could boost productivity. mesa-lao (2014) finds that pe trainees have a positive attitude towards speech input and would consider adopting it, and @xcite found that asr for pe was faster than asr for translation from scratch. due to these benefits, commercial cat tools like memoq and matecat are also beginning to integrate asr. the casmacat tool @xcite allows the user to input text by writing with e-pens in a special area. a vision paper @xcite proposes to instead use e-pens for pe sentences with few errors in place and provides examples of symbols that could be used for this. studies on mobile pe via touch and speech @xcite show that participants especially liked reordering words through touch drag and drop, and preferred voice when translating from scratch, but used the iphone keyboard for small changes. teixeira et al. (2019) also explore a combination of touch and speech; however, their touch input received poor feedback since (a) their tile view (where each word is a tile that can be dragged around) made reading more complicated, and (b) touch insertions were rather complex to achieve within their implementation. in contrast, dictation functionality was shown to be quite good and even preferred to mouse and keyboard by half of the participants. the results of an elicitation study by @xcite indicate that pen, touch, and speech interaction should be combined with mouse and keyboard to improve pe of mt. in contrast, other modalities like eye tracking or gestures were seen as less promising. this paper presents mmpe, the first translation environment combining standard mouse & keyboard input with touch, pen, and speech interactions for pe of mt. it allows users to directly cross out or hand-write new text, drag and drop words for reordering, or use spoken commands to update the text in place. all text manipulations are logged in an easily interpretable format (e.g., replaceword with the old and new word) to facilitate translation process research. the results of a study with 11 professional translators show that participants are enthusiastic about having these alternatives, and suggest that pen and touch are well suited for deletion and reordering operations, whereas speech and multi-modal interaction are suitable for insertions and replacements. ## the mmpe prototype this section presents the mmpe prototype (see figure 1 ), which combines pen, touch, and speech input with a traditional mouse and keyboard approach for pe of mt. the prototype is designed for professional translators in an office setting. a video demonstration is available at https://youtu.be/ tkj9owmdd0s . ## evaluation we evaluated the prototype with 11 professional translators 8 . since our participants were german natives, we chose a en-de translation task to avoid asr recognition errors occurring in non-native commands @xcite . in the following, "modalities" refers to touch (t), pen (p), speech (s), mouse & keyboard (mk), and multi-modal combinations (mm, see section 2.6), while "operations" refers to insertions, deletions, replacements, and reorderings. more details on the evaluation are presented in @xcite . ## conclusion while more and more professional translators are switching to the use of pe to increase productivity and reduce errors, current cat interfaces still heavily focus on traditional mouse and keyboard input. this paper therefore presents mmpe, a cat prototype combining pen, touch, speech, and multimodal interaction together with common mouse and keyboard input possibilities. users can directly cross out or hand-write new text, drag and drop words for reordering, or use spoken commands to update the text in place. our study with professional translators shows a high level of interest and enthusiasm about using these new modalities. for deletions and reorderings, pen and touch both received high subjective ratings, with pen being even better than mouse & keyboard. for insertions and replacements, speech and multi-modal interaction were seen as suitable interaction modes; however, mouse & keyboard were still favored and faster. as a next step, we will improve the prototype based on the participants' valuable feedback. furthermore, an eye tracker will be integrated into the prototype that can be used in combination with speech for cursor placement, thereby simplifying multi-modal pe. last, we will investigate whether using the different modalities has an impact on cognitive load during pe @xcite .