Improving Simultaneous Translation by Incorporating Pseudo-References with Fewer Reorderings (2021)

Abstract: Simultaneous translation is vastly different from full-sentence translation, in the sense that it starts translation before the source sentence ends, with only a few words delay. However, due to the lack of large-scale, high-quality simultaneous translation datasets, most such systems are still trained on conventional full-sentence bitexts. This is far from ideal for the simultaneous scenario due to the abundance of unnecessary long-distance reorderings in those bitexts. We propose a novel method that rewrites the target side of existing full-sentence corpora into simultaneous-style translation. Experiments on Zh \rightarrow En and Ja \rightarrow En simultaneous translation show substantial improvements (up to +2.7 BLEU) with the addition of these generated pseudo-references.

## introduction simultaneous translation, which starts translation before the source sentence ends, is substantially more challenging than full-sentence translation due to partial observation of the (incrementally revealed) source sentence. recently, it has witnessed great progress thanks to fixed-latency policies (such as wait-k) @xcite and adaptive policies @xcite . however, all state-of-the-art simultaneous translation models are trained on conventional parallel text which involve many unnecessary long-distance reorderings @xcite ; see fig. 1 for an example. the simultaneous translation models trained using these parallel sentences will learn to either make bold hallucinations (for fixed-latency policies) or introduce long delays (for adaptive ones). alternatively, one may want to use transcribed corpora from professional simultaneous interpretation @xcite @xcite . these data are more monotonic in word-order, but they are all very ⇤ ⇤ equal contribution. † currently at columbia university. small in size due to the high cost of data collection (e.g., the naist one @xcite has only 387k target words). more importantly, simultaneous interpreters tend to summarize and inevitably make many mistakes @xcite @xcite due to the high cognitive load and intense time pressure during interpretation @xcite . how can we combine the merits of both types of data, and obtain a large-scale, more monotonic parallel corpora for simultaneous translation? we propose a simple and effective technique to generate pseudo-references with fewer reorderings; see the "pseudo-refs" in fig. 1 . while previous work @xcite addresses this problem via languagespecific hand-written rules, our technique can be easily adopted to any language pairs without using extra data or expert linguistic knowledge. training with these generated pseudo references can reduce anticipations during training and result in fewer hallucinations in decoding and lower latency. we make the following contributions: ## preliminaries we briefly review full-sentence neural translation and the wait-k policy in simultaneous translation. full-sentence nmt uses a seq2seq framework (fig. 2 ) where the encoder processes the source sentence @xmath0, x 2 , ..., x m ) into a sequence of hidden states. a decoder sequentially generates a target sentence @xmath1, y 2 , ..., y n ) conditioned on those hidden states and previous predictions: the model is trained as follows: simultaneous translation translates concurrently with the (growing) source sentence, so @xcite propose the wait-k policy (fig. 2 ) following a simple, fixed schedule that commits one target word on receiving each new source word, after an initial wait of k source words. formally, the prediction of y for a trained wait-k model is this way, the model learns to implicitly anticipate at testing time, though not always correct (e.g., in fig. 2 , after seeing x 1 x 2 ="-˝ñ" (china 's), output y 1 ="there") . the decoder generates the target sentence ŷ with k words behind source sentence x: ## pseudo-reference generation since the wait-k models are trained on conventional full-sentence bitexts, their performance is hurt by unnecessary long-distance reorderings between the source and target sentences. for example, the training sentence pair in fig. 2 , a wait-2 model learns to output y 1 ="there" after observing to solve this problem, we propose to generate pseudo-references which are non-anticipatory under a specific simultaneous translation policy by the method introduced in section 3.1. meanwhile, we also propose to use bleu score to filter the generated pseudo-references to guarantee that they are semantic preserving in section 3.2. ## conclusions we have proposed a simple but effective method to generate more monotonic pseudo references for simultaneous translation. these pseudo references cause fewer anticipations and can substantially improve simultaneous translation quality.