Non-autoregressive Streaming Transformer for Simultaneous Translation (2023)

Abstract: Simultaneous machine translation (SiMT) models are trained to strike a balance between latency and translation quality. However, training these models to achieve high quality while maintaining low latency often leads to a tendency for aggressive anticipation. We argue that such issue stems from the autoregressive architecture upon which most existing SiMT models are built. To address those issues, we propose non-autoregressive streaming Transformer (NAST) which comprises a unidirectional encoder and a non-autoregressive decoder with intra-chunk parallelism. We enable NAST to generate the blank token or repetitive tokens to adjust its READ/WRITE strategy flexibly, and train it to maximize the non-monotonic latent alignment with an alignment-based latency loss. Experiments on various SiMT benchmarks demonstrate that NAST outperforms previous strong autoregressive SiMT baselines.

## introduction simultaneous machine translation (simt; @xcite @xcite @xcite , also known as real-time machine translation, is commonly used in various practical scenarios such as live broadcasting, video subtitles and international conferences. simt models are required to start translation when the source sentence is incomplete, ensuring that listeners stay synchronized with the speaker. nevertheless, translating partial source content poses significant challenges and increases the risk of translation errors. to this end, simt models are trained to strike a balance between latency and translation quality by dynamically determining when to generate tokens (i.e., write action) and when to wait for additional source information (i.e., read action). unidirectional encoder 与 (and) 沙龙 (sharon) 举行 (hold) 了 会谈 (talk) autoregressive simt @xmath0 figure 1 : illustration of the non-monotonicity problem and the source-info leakage bias in the training of autoregressive simt models. in this case, the ar simt model learns to predict at the third time step based on the source contexts "布什 (bush)", "与 (and)", "沙龙 (sharon)", and the ground truth contexts "bush", "held". although the source token "举行 (hold)" has not been read yet, it is exposed to the ar simt model through its corresponding token "held" in the ground truth context. simt models that generate target tokens in a nonautoregressive (nar) manner @xcite by removing the target-side token dependency. we argue that an nar decoder is better suited for streaming translation tasks. firstly, the target tokens are modeled independently in nar models, which facilitates the development of a non-monotonic alignment algorithm between generation and reference, alleviating the non-monotonicity problem. additionally, the conditional independence assumption of the nar structure liberates the model from the need for teacher forcing in training, thereby eliminating the risk of source-side information leakage. these advantageous properties of the nar structure enable simt models to avoid aggressive anticipation and encourage the generation of monotonic translations with fewer reorderings that align with the output of professional human interpreters. in this work, we propose non-autoregressive streaming transformer (nast). nast processes streaming input and performs unidirectional encoding. translations are generated in a chunk-bychunk manner, with tokens within each chunk being generated in parallel. we enable nast to generate blank token ϵ or repetitive tokens to build read/write paths adaptively, and train it to maximize non-monotonic latent alignment @xcite with a further developed alignment-based latency loss. in this way, nast effectively learns to generate translations that are properly aligned with the source in a monotonic manner, achieving high-quality translation while maintaining low latency. extensive experiments on wmt15 @xmath1. ## related work simt simultaneous machine translation requires a read/write policy to balance latency and translation quality, involving fixed and adaptive strategies. for the fixed policy, @xcite proposed wait-k, which first reads k source tokens and then alternates between read/write action. @xcite introduced an efficient training method for the wait-k policy, which randomly samples k during training. @xcite proposed a mixture-of-experts wait-k to learn a set of wait-k policies through multiple experts. for the adaptive policy, @xcite trained an agent to decide read/write via reinforcement learning. @xcite introduced milk, which incorporates a bernoulli variable to indicate the read/write action. @xcite proposed mma to implement milk on transformer. @xcite introduced caat, which leverages rnn-t and employs a blank token to signify the read action. @xcite proposed gsimt to generate the read/write actions. @xcite proposed to train a casual ctc encoder with gumbel-sinkhorn network @xcite to reorder the states. @xcite proposed hmt to learn when to start translating in the form of hmm, achieving the current state-of-theart simt performance. nar generation non-autoregressive models generate tokens parallel to the sacrifice of target-side dependency @xcite . this property eliminates the need for teacher forcing, motivating researchers to explore flexible training objectives that alleviate strict position-wise alignment imposed by the naive mle loss. @xcite proposed latent alignment model with ctc loss @xcite , and @xcite further explored non-monotonic latent alignments. @xcite introduced sequence-level training objectives with reinforcement learning and bag-of-ngrams difference. @xcite trained nat model using the best monotonic alignment and @xcite further extended it to order-agnostic cross-entropy loss. in addition, some researchers are focusing on strengthening the expression power to capture the token dependency. huang et al. (2022) apart from text translation, the nar model also demonstrated impressive performance in diverse areas such as speech-to-text translation @xcite , speech-to-speech translation @xcite ) and text-to-speech synthesis @xcite . ## conclusion in this paper, we propose non-autoregressive streaming transformer (nast) to address the nonmonotonicity problem and the source-info leakage bias in existing autoregressive simt models. comprehensive experiments demonstrate its effectiveness.