Backtranslation Feedback Improves User Confidence in MT , Not Quality (2021)

Abstract: Translating text into a language unknown to the text’s author, dubbed outbound translation, is a modern need for which the user experience has significant room for improvement, beyond the basic machine translation facility. We demonstrate this by showing three ways in which user confidence in the outbound translation, as well as its overall final quality, can be affected: backward translation, quality estimation (with alignment) and source paraphrasing. In this paper, we describe an experiment on outbound translation from English to Czech and Estonian. We examine the effects of each proposed feedback module and further focus on how the quality of machine translation systems influence these findings and the user perception of success. We show that backward translation feedback has a mixed effect on the whole process: it increases user confidence in the produced translation, but not the objective quality.

## introduction when dealing with machine translation (mt) on the web, most of the attention of the research community is paid to inbound translation. in this scenario, the recipients are aware of the mt process, and thus it is their responsibility to interpret and understand the translated content correctly. for an mt system, it is sufficient to achieve such quality that allows a recipient to get the gist of the meaning of texts on webpages. for outbound translation, it is the other way round: the responsibility to create the content in the way that it is correctly interpreted by a recipient lies on the authors of the message. the main issue is that the target language might be entirely unknown to them. prototypically it is communication by email, filling in foreign language forms, or involving some other kind of interactive medium. the focus in this scenario is placed not only on producing high-quality translations but also on reassuring the author that the mt output is correct. one of the approaches to improving both quality and authors' confidence, first employed in this scenario by @xcite , is to provide cues that indicate the quality of mt output as well as suggest possible rephrasing of the source. they may include backward translation to the source language, highlighting of the potentially problematic parts of the input, or suggesting paraphrases. except for preliminary work by @xcite , the impact of individual cues has not yet been properly explored. in this paper, we present the results of a new experiment on outbound translation. building on the previous works, the focus was expanded to investigate the influence of different levels of performance of the underlying mt systems, as well as utilizing a much greater range and diversity of participants and evaluation methods. native english speakers were tasked to produce text either in czech or in estonian with an outbound translation system in an e-commerce context. every user also reported a confidence score upon finishing each stimulus trial. a native czech or estonian speaker later evaluated each final translation for fluency and adequacy. the set of available cues varied for each participant from stimuli to stimuli, following a controlled experimental design, in order to determine the impact of specific combinations of cues on the self-reported confidence and the final translation quality. for our study, we made use of the ptakopět system @xcite . this bespoke software was specifically developed to examine user behavior when testing machine translation user interfaces, especially in the context of outbound translation. 1 the structure of the paper is as follows. after an overview of the related work in section 2, we present the environment for the outbound translation we used for the experiment, including the mt systems and modules that provided cues to the users, in section 3. section 4 describes the data that we collected during the experiment, and in section 5 we further analyze them to reveal and discuss various aspects of our approach to outbound translation. we conclude with the main findings in section 6. ## related work despite recent advances in neural machine translation (nmt) quality, resulting in output comparable to human professionals in specific settings @xcite , it is far from reasonable to blindly believe that the output of mt systems is perfectly accurate. it should thus not be simply included in an email or another message without some means of verification. feedback in this scenario is needed, which would tell users if the translation is correct and ideally even give instructions on how to improve it. a related area of interactive machine translation (imt) focuses mainly on either post-editor scenarios @xcite @xcite or generally scenarios in which users are able to produce the translation themselves and the system only aims to speed it up or improve it @xcite . outbound translation differs from common imt scenarios by the fact that the user does not speak the target language, and hence operates on the mt result only in a limited way. the first work to deal with this task by zouhar and bojar (2020) focused on working with czech-german mt in context of asking and reformulating questions. a preliminary experiment on the effect of translation cues has been carried out by @xcite , but it was conducted on a much smaller scale both in terms of participants and annotators and with non-native speakers of english. this may have affected the results that differ in some aspects, especially in the usefulness of the word-level quality estimation. ## environment for testing outbound translation in order to test the effect of different cues, we utilized ptakopět, a web-based tool for outbound translation. the tool provides machine translation together with cues in the form of backward translation, quality estimation and paraphrasing. these cues are intended to help the user arrive at a better translation and increase their confidence in the produced output. the tool is modular, allowing the modules for mt and cues to be either replaced with others or turned on and off. by linking a collection of sample stimuli to the tool it can also be used to conduct experiments. participants are asked to react to stimuli by formulating texts in a language known to them and producing and editing translations in a language they do not know. the set of cues they are pre-sented with may vary. the users are also asked to report their confidence in the produced output. in this experiment, each participant was presented with a sequence of scenes, interacting with the outbound translation system in each of them. figure 1 shows an example of a scene and user interaction. in the following sections, we describe the main components of the experiment. ## data collected in the experiment during a single scene, the participant saw a stimulus, worked on it and then finished it either by rating their confidence or by describing the reason for skipping. the participant was continuously presented with the translation output and the cues. we logged all incoming data as well as requests to the modules and their responses together with timestamps. in total, 52 english speaking participants joined our experiment, out of whom 49 were native 7 quoradata.quora.com/first-quora-dataset-release-question-pairs ## evaluation and results having recorded the essential interactions of participants with ptakopět, we further analyzed the collected data, especially user inputs and their translations. viable inputs. unless a participant skipped a scene, it was concluded by confirming the final input and its translation. we were also interested in examining intermediate complete sentences which users considered and later abandoned. we call these viable intermediate inputs. the collection of such inputs was possible because the ptakopět tool continually records user's interaction. we set the minimum time without any edit for an input to be sent to the forward translation module to 1000 ms. despite this relatively long period, still many incomplete or erroneous inputs were recorded, perhaps while the user was deliberating. we thus used a simple heuristic to extract the viable ones. for an input to be considered viable, it had to end with a full stop, an exclamation mark, or the same token as the final input ended. furthermore, its length had to be within a 25% margin around the length of the final input without whitespaces. 9 whereas each confirmed scene by design resulted in 1 final input and its translation, the number of intermediate viable inputs (non-final) was 0.62. their average length was 98.43% of the final input. evaluation of translation quality. the extracted viable inputs and their translations were rated for quality and adequacy by 12 czech and 3 estonian native speakers. for each viable input, the annotators were shown the source, its translation and the corresponding stimulus. they were asked to rate on the scale from 1 (least) to 5 (most) 9 this rule discredits inputs meant to be viable, where the very last token was later edited, though. manual examination of the data verified the efficacy of the heuristic. the following statements: • src-sti: the meaning of user input corresponds to what is entered in the form shown in the image. ## conclusion in this paper, we demonstrated through an experiment the effect of three translation cues on user confidence and translation quality. the backward translation cue proves to be a powerful means to enhance user confidence in mt. at the same time, it neither increase nor decrease significantly the translation quality. the fact that backtranslation feedback has a marginal effect to objective quality but greatly increases user confidence is surprising because it is the most intuitive low-effort approach to outbound translation scenarios which can be done even with publicly available mt systems. the paraphraser seems to increase user confi-dence less (compared to not being present), with no or slightly negative impact on the translation quality. without a better method to generate diverse and still adequate paraphrases, employing this cue is questionable. the effect of word-level quality estimation appears to be even more questionable. we attribute it mainly to the underlying word-level models, which may not be mature enough for user-facing applications. despite the loss in objective translation quality, the cpu-optimized student mt model either managed to maintain its teacher's high trustworthiness or compensated for it by its speed. future work. scores in both user confidence and overall translation quality annotation cluster together. having the distribution less concentrated by changing the underlying task with stimuli or by working with more low resource languages could reveal stronger dependencies between individual variables. we limited ourselves to only three baseline solutions to help in outbound translation. in the future work, inspiration could be drawn from the approaches of interactive machine translation systems and these could be adapted for the purposes of outbound translation.