Mid-Air Hand Gestures for Post-Editing of Machine Translation (2021)

Abstract: To translate large volumes of text in a globally connected world, more and more translators are integrating machine translation (MT) and post-editing (PE) into their translation workflows to generate publishable quality translations. While this process has been shown to save time and reduce errors, the task of translation is changing from mostly text production from scratch to fixing errors within useful but partly incorrect MT output. This is affecting the interface design of translation tools, where better support for text editing tasks is required. Here, we present the first study that investigates the usefulness of mid-air hand gestures in combination with the keyboard (GK) for text editing in PE of MT. Guided by a gesture elicitation study with 14 freelance translators, we develop a prototype supporting mid-air hand gestures for cursor placement, text selection, deletion, and reordering. These gestures combined with the keyboard facilitate all editing types required for PE. An evaluation of the prototype shows that the average editing duration of GK is only slightly slower than the standard mouse and keyboard (MK), even though participants are very familiar with the latter, and relative novices to the former. Furthermore, the qualitative analysis shows positive attitudes towards hand gestures for PE, especially when manipulating single words.

## introduction in a well-connected world, translation is of everincreasing importance @xcite . to meet translation demands, machine translation (mt) is often employed as a cheaper and faster alternative to human translation (ht) @xcite . even though mt has improved drastically over the last 5 years, discussions about reaching human parity are still ongoing @xcite and limited to a small set of language pairs and domains for which ample training data is available. for most application scenarios, however, mt quality is far from reaching the quality of highly trained professionals. in an attempt to combine the best of both worlds, post-editing (pe) is becoming common practice, where human translators use raw mt output and make the necessary changes to produce an acceptable level of quality @xcite . although translators have approached pe with fear and skepticism @xcite , more recent studies found that nowadays translators are more open to it and that much of the original dislike was attributed to outdated perceptions of mt quality @xcite . independent of translators' perceptions, studies found that pe increases productivity and decreases errors compared to translation from scratch @xcite . pe changes the translation task from mostly text generation to text editing, which involves an increased usage of navigation and deletion keys @xcite . as a result, translators need better support with text editing operations, which raises the question whether interaction modalities other than mouse and keyboard can be beneficial for pe. an interaction modality that has gained attention in other research areas (koutsabasis and vogiatzidakis, 2019) but so far remains unexplored for pe is mid-air hand gestures. in this paper, we (i) investigate which mid-air gestures combined with the keyboard (gk) are suitable for which text-editing operations in pe, (ii) build a prototype supporting pe using gk, and (iii) analyze editing times and subjective feedback on mid-air hand gestures compared to mouse and keyboard (mk) for specific pe operations. to address these goals, we conducted a gesture elicitation study (ges) with professional translators, resulting in a set of gestures for different editing tasks, which were then implemented in a prototype. our experiment shows that, surprisingly, editing durations for most pe tasks were very similar in the conditions gk and mk, even though participants were much more experienced with the latter. furthermore, participants prefer manipulating single items 1 using gestures, while manipulating a group of items, which involves more complex text selection, received poorer subjective feedback. ## related work in this section, we present related research on translation environments, multi-modal approaches to pe, and mid-air gestures for text editing tasks. ## gesture elicitation study a ges is a form of participatory design @xcite where users are incorporated in the design process to inform an appropriate gesture set for a given application. important aspects include leading participants away from technical thinking @xcite , making them assume that gesture recognition is perfect, and considering their behavior as always acceptable @xcite . they should only be informed about the essential details of the task to avoid bias towards particular approaches @xcite . we conduct a ges for three reasons. firstly, there is no universal gesture set suitable for all applications @xcite . secondly, users prefer gestures designed through elicitation studies, because professional designers tends to generate more physically and conceptually complex gestures @xcite . thirdly, to the best of our knowledge, there is no other ges for text editing using gk which we could rely on. in our ges, we employed the guessability approach @xcite which is intended to increase immediate usage of interfaces. it consists of three phases: (1) defining so-called referents (i.e. common operations) that should be achievable through the system, (2) asking participants to propose a gesture for each referent, and (3) analyzing the collected data to generate the final gesture set. ## prototype we used the ges results to define our final gesture set and implement a prototype. for this, the frequently proposed gestures were explored in terms of implementation feasibility given the technology we are using. if two gestures were conflicting, we dropped the less popular one; otherwise we slightly modified it to resolve the conflict. for group selection, we found that the proposed index + thumb gesture practically fails upon selection across multiple lines; thus, we dropped it. in contrast, using both indices can perform this kind of selection, so we implemented it as depicted in figure 2 . note that in contrast to the mouse, the group selection using both index fingers allows the user to manipulate both ends of the selection continuously instead of having one side fixed. for single item/position selection, we only implemented pointing with the right index finger, as it already entails the double tap gesture. for multi-line text, both single and group selection allow pointing with the index finger vertically and horizontally. insertion can also be easily achieved by placing the cursor through pointing followed by typing. for deletion, d s and d g received similar gesture proposals. looking at the proposals in detail, we found that two participants also wanted to delete with a hand down movement. thus, we implemented hand or finger movement down and up to offer consistent deletion possibilities (figure 3 ). for d s , it is sufficient if the cursor is placed somewhere on the word; there is no need to define the start and end of the word through a group selection. replacement can be achieved by either performing a group selection and typing directly, or by selecting a single item or group of items, deleting, and then typing. note that rp s can thus also be achieved without group selection. the most complicated gestures were proposed for reordering; the gestures are a compound of several sub-gestures. since reordering using the right index conflicts with cursor movement, we dropped it. moving both hands while in the selection position turned out to be difficult to perform, as maintaining the same distance between the hands at all times is challenging. therefore, we decided to merge it with the grab proposal; thus, after selection, a grab with the left hand indicates the start of the reordering process. then moving both hands or just the right index finger reorders the text (figure 4 ). once the required position is reached, closing the right hand ends the reordering process and drops the text in the target position. for single item reordering, it is again sufficient to place the cursor on the item without selecting the whole text. the prototype was implemented as an extension to our open-source mmpe cat interface (herbig et al., 2020b,c) foot_2 . mmpe allows translators to use input modalities such as speech, touch, pen, and eye tracking in combination with the standard mouse and keyboard. however, it previously did not support mid-air gestures. the main interface shows the source on the left, and the target on the right, with the currently edited segment enlarged. this additional space turned out to be useful for hand gestures as it simplifies pointing. in addition, all user interactions are logged. mmpe uses angular for the front-end, and node.js for the back-end, with websockets and rest apis for the communication between them. our gesture detection relies on the leap motion controller foot_3 , which is small in size (8cm * 3cm) and can be placed on the top of the keyboard (figure 2 ). the device provides frames of detected hands with 3d positions of finger joints, as well as some basic detection such as whether the fingers are extended or not. based on this information our gesture detection algorithm determines if one of the above gestures is being performed. if only the right hand is detected with the index fingers extended, then the cursor will be updated based on hand movement. moving both index fingers selects the corresponding text in the interface (figure 2 ). when a deletion gesture is detected, the selected text (for group selection), or the word that the cursor is currently positioned on, is removed (figure 3 ). a grab with the left hand puts the currently selected text/word containing the cursor in a reordering visualization. then, movements of the right index are tracked and move the highlighted text as well as an arrow indicator visualizing the currently calculated drop position. releasing the grab then places the text back into the input field at the indicated position (figure 4 ). to avoid unintended gestures while moving the hands back to the keyboard, the user can form a grab in both hands after executing a gesture. since people move their hands at different speeds, we further added sensitivity settings for gestures, similar to the standard mouse settings. a video showing the interactions in practice can be found under: https://youtu.be/qiryeojkfvc . ## prototype evaluation in contrast to the web-based elicitation study, we had to evaluate the prototype in-situ due to the hardware setup. given the covid-19 situation, it was impossible to invite professional translators. therefore we had to conduct a study with our colleagues. to mitigate the difference between nontranslation professional subjects (computer scientists) and translation professionals, we ensured that similar to professional translators, (i) all our participants have academic training (computing degrees instead of translation degrees), (ii) that they are also highly familiar with traditional mouse and keyboard interfaces and use them in their day-today work, (iii) all subjects have relevant language proficiency (source en, target de), and (iv) all work in a multilingual en-de environment. furthermore, as the evaluation required participants only to perform pre-specified text editing operations, without involving any linguistic translation decisions, we hope to minimise the effect of not having translators as participants. ## conclusion the use of mt and pe changes the task of translation from mostly text production to fixing errors within useful but partly incorrect mt output. this affects the interface design of cat tools, where translators need more support for text editing tasks. the literature suggests that other interaction modalities than mk, or combinations thereof, could better support pe operations. to the best of our knowledge, this is the first study that investigates the usefulness of mid-air hand gestures for pe of mt. our ges with 14 freelance translators yielded a set of gestures to manipulate both single items and groups of items, which we further refined by considering conflicting gestures and exploring them practically. the resulting prototype allows users to (i) place the cursor by pointing with the index finger, (ii) select ranges of text by pointing with both index fingers, (iii) moving the hand or index finger up or down for deletion, and (iv) reorder by selecting text, forming a grab with the left hand, pointing with the right index finger to the desired position, and releasing the grab to drop the text. these gestures, combined with the keyboard, support all text manipulations required for pe. due to covid-19, only a small-scale prototype evaluation with non-translator participants was possible. nonetheless, as the prototype design was guided by an elicitation study with translation professionals which usually leads to well-perceived interfaces and since we designed the study to mitigate bias induced by a sub-optimal participant sample, we expect that professional translators would have given us comparable feedback. the findings overall suggest that gk could be a suitable interaction modality for pe and thus merits further research: even though participants had years of experience with mk, our quantitative analysis of editing time showed that gk was only slightly slower for most operations, especially when manipulating single items. similarly, qualitative data shows that manipulating single items was rated higher than operations working on groups of items, as participants found the group selection gesture "cumbersome" to perform. this finding indicates that further effort should be invested in improving group operations, which are also common in pe (e.g. by exploring if a different placement of the detection device could increase detection accuracy). however, the appealing results on single item operations and the satisfactory results on group operations bode well and warrant further exploration with professional translators in a realistic pe scenario. we do expect that after using the interface for a longer period of time, users will become more effective, as is common with other interfaces: the new interface is competing with decades of mk musclememory training. however, only future long-term studies can show if editing times with gk will become as low as or even lower than with mk approaches. apart from efficiency, participants in our previous studies @xcite argued for having multiple suitable options to interact with text, instead of performing the same movements all day long. therefore, it is not just a question of speed but also user satisfaction and health: additional modalities may help guard against carpaltunnel syndrome and provide exercise alternatives in a seated environment. to conclude, this new interaction modality, which so far was overlooked by research on cat tools and post-editing, performs better than expected and therefore warrants further investigation. overall, we hope that future research will pick up the insights from the first and second study and help advance the state-of-the-art in pe.