Learning Coupled Policies for Simultaneous Machine Translation using Imitation Learning (2021)

Abstract: We present a novel approach to efficiently learn a simultaneous translation model with coupled programmer-interpreter policies. First, we present an algorithmic oracle to produce oracle READ/WRITE actions for training bilingual sentence-pairs using the notion of word alignments. This oracle actions are designed to capture enough information from the partial input before writing the output. Next, we perform a coupled scheduled sampling to effectively mitigate the exposure bias when learning both policies jointly with imitation learning. Experiments on six language-pairs show our method outperforms strong baselines in terms of translation quality quality while keeping the delay low.

## introduction simultaneous machine translation (simt) is a setting where the translator needs to incrementally generate the translation while the source utterance is being received. this is a challenging translation scenario as the simt model needs to trade off delaying translation output and the quality of the generated translation. recent research on simt relies on a strategy to decide when to read a word from the input or write a word to the output @xcite . this is based on a sequential decision making formulation of simt, where the decision making about the next read/write action is made by an agent, interacting with the neural machine translation (nmt) environment. current approaches are sub-optimal as they either fix the agent's policy to focus learning the nmt model @xcite or learn adaptive agent policies while the nmt model is fixed @xcite . we argue that the interpreter should also learn to generate correct translation from incomplete input information. this is challenging as we need to optimize both programmer's and interpreter's policies to balance the tradeoff between quality and delay in the reward. previous research has considered the use of imitation learning (il) to train the agent's policy @xcite , which is generally superior to reinforcement learning (rl) in terms of the stability and sample complexity. however, the bottleneck of il in simt is the unavailability of the oracle sequence of actions. designing algorithmic oracles to compute sequence of read/write actions with low translation latency and high translation quality is under-explored. we present an il approach to efficiently learn effective coupled programmer-interpreter policies in simt, based on the following contributions. first, we present a simple, fast, and effective algorithmic oracle to produce oracle actions from the training bilingual sentence-pairs based on statistical word alignments @xcite . next, we design a framework that uses scheduled sampling on both programmer and interpreter. this is different from the typical il scenarios, where there is only one policy to learn. as the two policies collaborate, their learning needs to be robust not only to their own incorrect predictions, but also to incorrect predictions of the other policy to mitigate this coupled exposure bias. experiments on six language pairs (translating to english from arabic, czech, german, romanian, hungarian, and bulgarian) show the policies trained using our approach compares favorably with strong policies from the previous work. we attribute the effectiveness of the learned coupled policies to (i) the scheduled sampling, which handles the coupled exposure bias, resulting in up to 5-8 bleu score improvements, and (ii) the quality of oracle actions generated by our algorithmic oracle, which balances translation quality and delay. algorithm 1 generation in npi-simt 1: i, j ← 0 2: while a stopping condition is not met do 3: ## npi approach to simt we describe generation in our neural programmerinterpreter (npi) approach to simultaneous machine translation (simt) in algorithm 1. at each time step t, the programmer needs to decide whether to read the next source word or to write the next target word in the translation. the interpreter then immediately executes the action generated by the programmer. both programmer and interpreter are modeled using markov decision process (mdp) where prediction at particular timestep depends on the history of previous predictions. the indices i t and j t are the number of read and write actions in the program up to time step t. the programmer needs to sequentially decide about the next action, given the previous actions @xmath0. that is, our programmer is modeled as p prog (a|@xmath1, @xmath2, @xmath3. the interpreter needs to execute the action generated by the programmer. at time step t, if the generated action a t is read, we reveal the next input token. otherwise, if write, then we generate the next target word according to p intp (y|@xmath4, @xmath5, @xmath6. 1 the probabilistic model. the probability of simultaneously generating the translation y and the 1 the counter i and j are also incremented according to the respective actions at time t. training the model. in simt, we are interested in not only producing a high quality translation, but also reducing the delay between the times of receiving the source words and generating their translations. training of the model based on this hybrid training objective can be done by reinforcement learning (rl) or imitation learning (il). the rl approach has been attempted by @xcite @xcite for training the programmer; however, it is unstable due to sparsity of the reward function and these works also assumed a fixed interpreter. we thus take the il approach for a sample efficient, effective, and stable learning of policies in npi-simt. ## deep coupled imitation learning our goal is to learn a pair of policies for the programmer and interpreter using il. §3.1 describes the method of learning of both policies where their learning inter-dependency needs to be taken into account. §3.2 describes our novel oracle program actions for each sentence pair in the training set, i.e., the program a which has been responsible for generating the translation y for a source utterance x with as low delay as possible. our overall training algorithm is depicted in algorithm 2. @xmath7, ..., x|x| ] is the encoding of the input sequence and ŷ = [ ŷ1 , ..., ŷ|y| ] is a list of interpreter hidden states for all predictions. during training, these values are calculated before calculating the loss of the programmer. ## experiments our experiments aim to measure the effectiveness of our proposed method versus a strong wait-k baseline, over a range of languages of varying difficulty, syntactic complexity, and lexical complexity. ## conclusion this paper proposes a simple and effective way to train a simultaneous translation system to produce low delay translations. our central contribution is to determine a sufficient, if not minimum, amount of inputs to translate each target token. this is achieved using word-alignment to create an oracle, which is then used as part of a training algorithm based on imitation learning to learn coupled policies, for a "programmer" which decides when to wait for more input producing translation tokens, and an "interpreter" which generates the translation. we show the importance of scheduled sampling during learning, which is crucial to combat exposure bias. overall we show improvements in bleu score over naively trained systems with modest translation delays. future work is needed to better understand the effect of various alignment models and symmetrization methods on the generated oracle. beyond this, other opportunities include applying the model to the real speech input and applying more sophisticated imitation learning techniques that involves the generated trajectories of both "interpreter" and "programmer".