A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translation (2018)

Abstract: We present an approach to interactivepredictive neural machine translation that attempts to reduce human effort from three directions: Firstly, instead of requiring humans to select, correct, or delete segments, we employ the idea of learning from human reinforcements in form of judgments on the quality of partial translations. Secondly, human effort is further reduced by using the entropy of word predictions as uncertainty criterion to trigger feedback requests. Lastly, online updates of the model parameters after every interaction allow the model to adapt quickly. We show in simulation experiments that reward signals on partial translations significantly improve character F-score and BLEU compared to feedback on full translations only, while human effort can be reduced to an average number of 5 feedback requests for every input.

## introduction interactive-predictive machine translation aims at obtaining high-quality machine translation by involving humans in a loop of user validations of partial translations suggested by the machine translation system. this interaction protocol can easily be fit to neural machine translation (nmt) @xcite by conditioning the model's word predictions on the user-validated prefix @xcite . user studies conducted by @xcite the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. the goal of our work is to further reduce human effort in interactive-predictive nmt by combining the advantages of the interactive-predictive protocol with the advantages of learning from weak feedback. for the latter we rely on techniques from reinforcement learning @xcite , a.k.a. bandit structured prediction @xcite @xcite in the context of sequence-to-sequence learning. our approach attacks the problem of reducing human effort from three innovative directions. frequent updates improve the adaptability of our system, resulting in a further reduction of human effort. the rest of this paper is structured as follows. in section 2, we will situate our approach in the context of interactive machine translation and analyze our contribution related to reinforcement learning for sequence prediction problems. details of our algorithm are given in section 3. we evaluate our approach in a simulation study where bandit feedback is computed by evaluating partial translations against references under a character f-score metric @xcite without revealing the reference translation to the learning system (section 4). we show that segment-wise reward signals improve translation quality over reinforcement learning with sparse sentence-wise rewards, measured by character f-score and corpusbased bleu against references. furthermore, we show that human effort, measured by the number of feedback requests, can be reduced to an average number of 5 requests per input. these implications of our new paradigm are discussed in section 5. ## related work the interactive-predictive translation paradigm reaches back to early approaches for ibm-type @xcite and phrase-based machine translation @xcite . @xcite and @xcite presented neural interactive translation prediction -a translation scenario where translators interact with an nmt system by accepting or correcting subsequent target tokens sug-gested by the nmt system in an auto-complete style. nmt is naturally suited for this incremental production of outputs, since it models the probability of target tokens given a history of target tokens sequentially from left to right. in standard supervised training with teacher forcing, this history comes from the ground truth, while in interactivepredictive translation it is provided by the prefix accepted or entered by the user. both approaches use references to simulate an interaction with a translator and compare their approach to phrasebased prefix-search. they find that nmt is more accurate in word and letter prediction and recovers better from failures. similar to their work, we will experiment in a simulated environment with references mimicking the translator. however, we do not use the reference directly for teacher forcing, but only to derive weak feedback from it. furthermore, our approach employs techniques to reduce the number of interactions, and to update the model more frequently than after each sentence. our work is also closely related to approaches for interactive pre-post-editing @xcite . the core idea is to ask the translator to mark good segments and use these for a more informed re-decoding. both studies could show a reduction in human effort for post-editing in simulation experiments. we share the goal of using human feedback more effectively by targeting it towards essential translation segments, however, our approach does adhere to the left-to-right navigation through translation hypotheses. in difference to these approaches, we try to reduce human effort even further by minimizing the number of feedback requests and by frequent model updates. reinforcing/penalizing a targeted set of actions can also be found in recent approaches to reinforcement learning from human feedback. for example, @xcite presented a scenario where users interactively label freely chosen good and bad parts of a policy's trajectory. the policy is directly trained with this reinforcement signal to play a real-time strategy game. simulations of nmt systems interacting with human feedback have been presented firstly by @xcite , @xcite , or @xcite who apply different policy gradient algorithms, william's reinforce @xcite or actorcritic methods @xcite @xcite , respectively. while @xcite approach operates in a fully supervised learning scenario, where rewards are simulated in comparison to references with smoothed and length-rescaled bleu, @xcite and @xcite limit the setup to sentence-level bandit feedback, i.e. only one feedback is obtained for one completed translation per input. in this paper, we use actor-critic update strategies, but we receive simulated bandit feedback on the sub-sentence level. we adopt techniques from active learning to reduce the number of feedbacks requested from a user. @xcite apply active learning for interactive machine translation, where a user interactively finishes the translation of an smt system. the active learning component decides which sentences to sample for translation (i.e. receive full supervision for) and the smt system is updated online @xcite . in our algorithm the active learning component decides which prefixes to be rated (i.e. receive weak feedback for) based on their average entropy. entropy is a popular measure for uncertainty in active learning: the rationale is to feed the learning algorithm with labeled instances where it is least confident about its own predictions. this uncertainty sampling algorithm @xcite is a popular choice for active learning for nlp tasks with expensive gold labeling, such as text classification @xcite , word-sense disambiguation @xcite and statistical parsing @xcite . our method falls into the category of stream-based online active learning (as opposed to pool-based active learning, selecting instances from a large pool of unlabeled data), since the algorithm decides on the fly (online) which translation prefixes of the stream of source tokens to request feedback for. instead of receiving gold annotations, as in the studies mentioned above, our algorithm receives weaker, bandit feedback -but the motivation of minimizing human labeling effort is the same. ## reinforcement learning for interactive-predictive translation in the following, we will introduce the key ideas of our approach, formalize them, and present an algorithm for reinforcement learning for interactivepredictive nmt. ## experiments we simulate a scenario where the learning nmt system requests online bandit feedback for partial translations from a human in the loop. the following experiments will give an initial practical assessment of our proposed interactive learning algorithm. our analysis of the interactions between actor, critic and simulated human will provide further insights into the learning behavior of bip-nmt. ## conclusion we presented a novel algorithm, coined bip-nmt, for bandit interactive-predictive nmt using reinforcement learning techniques. our algorithm builds on advantage actor-critic learning @xcite for an interactive translation process with a human in the loop. the advantage over previously presented algorithms for interactive-predictive nmt is the low human effort for producing feedback (a translation quality judgment instead of a correction of a translatioin), even further reduced by an active learning strategy to request feedback only for situations where the actor is uncertain. we showcased the success of bip-nmt with simulated feedback, with the aim of moving to real human feedback in future work. before deploying this algorithm in the wild, suitable interfaces for giving real-valued feedback have to be explored to create a pleasant user experience. furthermore, in order to increase the level of human control, a combination with the standard paradigm that allows user edits might be considered in future work. finally, our algorithm is in principle not limited to the application of nmt, but can furthermorethanks to the broad adoption of neural sequenceto-sequence learning in nlp -be extended to other structured prediction or sequence generation tasks.