Better Simultaneous Translation with Monotonic Knowledge Distillation (2023)

Abstract: Simultaneous machine translation (SiMT) presents a unique challenge as it requires generating target tokens before the source sentence is fully consumed. This can lead to the hallucination problem, where target tokens are generated without support from the source sentence. The prefix-to-prefix training data used to train SiMT models are not always parallel, due to divergent word order between the source and target languages, and can contribute to the problem. In this paper, we propose a novel approach that leverages traditional translation models as teachers and employs a two-stage beam search algorithm to generate monotonic yet accurate reference translations for sequence-level knowledge distillation. Experimental results demonstrate the significant improvements achieved by our approach over multiple strong SiMT baselines, leading to new state-of-the-art performance across various language pairs. Notably, when evaluated on a monotonic version of the WMT15 De-En test set, which includes references generated in a more monotonic style by professional translators, our approach achieves even more substantial improvement over the baselines. The source code and data are publicly available for further exploration.

## introduction simultaneous machine translation (simt) starts to translate with only a partial observation of the source sentence and can present unique challenges compared to full-sentence translation, particularly when employing offline nmt models. prefix-toprefix (p2p) methods such as the wait-k policy @xcite have been developed to narrow the gap between training and inference. however, these methods inherently rely on parallelism at the prefix level, which may not always be present in conventional parallel text. ## related works simt policy there are two types of simt policies: fixed and adaptive. fixed policies, such as wait-k in @xcite , first read k source tokens and then alternately read/write one token. @xcite proposed an efficient multipath training for the wait-k policy to randomly sample k during training. adaptive policies make read/write decisions dynamically. @xcite decides read/write actions via reinforcement learning. milk @xcite predicts a bernoulli variable to determine read/write actions, which is further implemented into transformer architecture mma @xcite . @xcite developed adaptive wait-k through heuristic ensemble of multiple wait-k mod-els. @xcite proposed a generative framework to generate read/write decisions. @xcite applies connectionist temporal classification (ctc) by treating the blank symbol as the wait action. @xcite develops a read/write policy by modeling the translation process as information transport and taking the received information as the evidence for read/write decisions. ## monotonic translation construction we propose two approaches for creating monotonic pseudo-targets for source sentences in traditional parallel data. this new data is then used to train simt models through knowledge distillation (kd). ## discussions the first beam search stage of our monotonic kd method is equivalent to test-time wait-k inference described in @xcite . this stage, however, may fail to produce accurate rankings of partial hypotheses, given that it relies on offline models for translating partial inputs. information, is capable of more accurately scoring and ranking these partial hypotheses. we conducted an analysis on the wmt15 @xmath0.e., monotonic one-stage beam search) and our monotonic two-stage beam search. as shown in ## conclusion long-distance reorderings in conventional parallel data can negatively impact the training of simultaneous translation models. to address this problem, we propose a novel two-stage beam search algorithm to generate monotonic yet accurate pseudo translations that are then used to train simt models through sequence-level knowledge distillation. experiments on three language pairs demonstrate that this method can consistently improve multiple simt models and achieve new state of the art performance for simultaneous translation.