Decoder-only Streaming Transformer for Simultaneous Translation (2024)

Abstract: Simultaneous Machine Translation (SiMT) generates translation while reading source tokens, essentially producing the target prefix based on the source prefix. To achieve good performance, it leverages the relationship between source and target prefixes to exact a policy to guide the generation of translations. Although existing SiMT methods primarily focus on the Encoder-Decoder architecture, we explore the potential of Decoder-only architecture, owing to its superior performance in various tasks and its inherent compatibility with SiMT. However, directly applying the Decoder-only architecture to SiMT poses challenges in terms of training and inference. To alleviate the above problems, we propose the first Decoder-only SiMT model, named Decoder-only Streaming Transformer (DST). Specifically, DST separately encodes the positions of the source and target prefixes, ensuring that the position of the target prefix remains unaffected by the expansion of the source prefix. Furthermore, we propose a Streaming Self-Attention (SSA) mechanism tailored for the Decoder-only architecture. It is capable of obtaining translation policy by assessing the sufficiency of input source information and integrating with the soft-attention mechanism to generate translations. Experiments demonstrate that our approach achieves state-of-the-art performance on three translation tasks.

## introduction simultaneous machine translation (simt) @xcite is designed for generating translations in real-time scenarios such as online conferences and real-time subtitles. it predicts the target tokens (i.e., target prefix) based on the already read source tokens (i.e., source prefix), aiming to achieve good tradeoffs between latency and translation quality. during training, simt models need to learn the correspondence between source and target prefixes, crucial for extracting policies that ensure superior performance during inference @xcite . existing research on simt primarily focuses on the encoder-decoder architecture and is categorized into fixed and adaptive policies. for fixed policy @xcite @xcite , the model utilizes heuristic rules to determine the source prefix used for generating translations, which ignores the correspondence between the source and target prefixes. this may lead to redundant or missing source information during translation, resulting in inferior performance @xcite . for adaptive policy @xcite , the model dynamically decides whether to read or output tokens based on the relationship between the source and target prefixes. this dynamic adjustment of policy in response to the translation status allows for improved tradeoffs @xcite . however, there is a lack of exploration in simt regarding the decoder-only architecture. with the rise of language models, the decoderonly architecture has exhibited superior performance across diverse tasks @xcite . as illustrated in figure 1 , the decoder-only architecture, compared to the encoder-decoder architecture with an equivalent number of parameters, can support more layers, thereby offering parameter efficiency and better scalability @xcite . importantly, simt relies on unidirectional encoding @xcite , and the decoder-only architecture seamlessly accommodates this requirement. therefore, we explore the capability of decoder-only architecture in simt. however, directly applying the decoder-only architecture to the simt task poses challenges in both training and inference. during inference, with the arrival of each source token, there is a position increase of the generated target prefix, necessitating the model to re-encode the target prefix. this exacerbates the inference cost, particularly at low latency @xcite . during training, the model learns to predict the corresponding target prefix based on a given source prefix. consequently, it is necessary to construct corresponding target prefixes for all possible source prefixes in a sentence pair. this limitation hinders the model from learning the translation policy and leads to an increase in training costs compared to the encoder-decoder architecture. to overcome the above limitations, we propose the first simt model based on decoder-only architecture, named the decoder-only streaming transformer (dst). to alleviate the issue of reencoding, dst encodes the positional information of the source prefix and the target prefix separately. this ensures that the expansion of the source prefix does not impact the position of the generated target prefix, thereby reducing the inference costs. to assess the contribution of partial source information to generating target tokens, dst uses the proposed streaming self-attention (ssa) in replace of the conventional masked self-attention in the decoder layer to decrease training costs and derive a translation policy. during training, ssa can consider all possible source prefixes for the target prefixes in a sentence pair. specifically, ssa predicts attention allocation for different source prefixes and combines it with the soft-attention mechanism to obtain expected attention for all source tokens and tokens in the target prefix. this expected attention is then utilized to derive the context vector. by leveraging ssa, the model learns the importance of all source prefixes in translating the target prefix, thereby reducing training costs. during inference, ssa accumulates the allocated attention from all prefixes of the input source tokens, enabling an assessment of the sufficiency of input source information for generating translation. the model utilizes this assessment to determine whether to read or generate tokens, thereby acquiring the translation policy. experiments demonstrate that dst achieves state-of-theart performance on three tasks. ## background simultaneous machine translation the simt model @xcite dynamically reads the source sentence @xmath0, ..., x j ) with length j and generates translation @xmath1, ..., y i ) with length i based on a policy. to articulate the policy, we introduce g i , representing the number of source tokens involved in translating y i . thus, the translation policy from x to y can be denoted as @xmath2, ..., g i ). during training, the simt model undergoes optimization by minimizing the cross-entropy loss: ## method in this section, we introduce the decoder-only streaming transformer (dst). dst adopts the decoder-only architecture and employs the proposed streaming self-attention (ssa) in place of masked self-attention at each layer. during inference, ssa accumulates the attention assigned to the input source tokens to assess the sufficiency of source information and obtain the translation policy accordingly. during training, dst leverages our proposed constraints to ensure the learning of the ssa mechanism. the details of dst are introduced as follows. ## analysis to deepen the understanding of our method, we conduct multiple analyses. the experiments are all performed on the @xmath3. ## related work simultaneous machine translation (simt) begins generating translation before reading the entire source sentence. to ensure the quality of generated translations and meet the latency requirements, it is necessary to determine the suitable source prefixes for translation. the process of determining the source prefixes for target tokens is referred to as the policy @xcite . depending on whether the model decides the policy utilizing the correspondence between source and target prefixes, simt methods can be broadly categorized into fixed policy and adaptive policy. for fixed policy, the simt model reads or generates tokens according to the heuristic rules. @xcite proposes wait-k policy, which reads k source tokens first, and then writes and reads one token alternately. @xcite introduces the unidirectional encoder, where the preceding source tokens cannot attend to subsequent words. @xcite proposes moe wait-k, which assigns a wait-k policy to each head and combines the results from multiple heads to generate translations. there are some methods that enhance the simt capability by modifying the reference @xcite . however, the fixed policy requires the simt model to generate tokens even when source information is insufficient, leading to a decline in performance of simt. unlike fixed policy, adaptive policy can dynamically decide whether to read or generate tokens based on the correspondence between source prefix and target prefix. @xcite implements the adaptive policy through a composition of several fixed policies. @xcite proposes a generative framework, which uses a reparameterized poisson prior to regularise the policy. @xcite models the contribution of each source token to the target tokens through optimal transport and determines the policy by accumulating the contributions of input source tokens. @xcite utilizes the hidden markov model to model the translation policy as the hidden events and the translation as the observed events. @xcite first proposes to conduct simt with non-autoregressive architecture and achieves good results. however, previous simt methods all employ the encoder-decoder architecture. with the rise of language models, the decoderonly architecture has demonstrated superior performance across various tasks @xcite @xcite . the decoderonly architecture has the advantages of good scalability and parameter efficiency. more importantly, its unidirectional encoding nature aligns greatly with streaming input. therefore, we explore the application of the decoder-only architecture in simt and propose the first decoder-only simt model. ## conclusion in this paper, we propose the first decoder-only simt model, which leverages the characteristics of the decoder-only architecture to implement the adaptive policy. experiments show that our method achieves state-of-the-art performance and exhibits excellent scalability and model efficiency.