Attention as a Guide for Simultaneous Speech Translation (2023)

Abstract: In simultaneous speech translation (SimulST), effective policies that determine when to write partial translations are crucial to reach high output quality with low latency. Towards this objective, we propose EDAtt (Encoder-Decoder Attention), an adaptive policy that exploits the attention patterns between audio source and target textual translation to guide an offline-trained ST model during simultaneous inference. EDAtt exploits the attention scores modeling the audio-translation relation to decide whether to emit a partial hypothesis or wait for more audio input. This is done under the assumption that, if attention is focused towards the most recently received speech segments, the information they provide can be insufficient to generate the hypothesis (indicating that the system has to wait for additional audio input). Results on en->de, es show that EDAtt yields better results compared to the SimulST state of the art, with gains respectively up to 7 and 4 BLEU points for the two languages, and with a reduction in computational-aware latency up to 1.4s and 0.7s compared to existing SimulST policies applied to offline-trained models.

## introduction in simultaneous speech translation (simulst), systems have to generate translations incrementally while concurrently receiving audio input. this requirement poses a significant challenge since the need of generating high-quality outputs has to be balanced with the need to minimize their latency, i.e. the time elapsed (lagging) between when a word is uttered and when it is actually translated by the system. in direct simulst systems @xcite , foot_0 the balance between output quality and latency is managed by a decision policy, which is the strategy for determining, at each time step, whether to emit a partial translation or to wait for additional audio input. decision policies can be divided into two categories: fixed and adaptive. fixed policies are usually based on simple heuristics @xcite , while adaptive policies take into account the actual input content to make the decisions @xcite . recent works @xcite @xcite proved the superiority of adaptive policies over fixed ones. however, a major limitation of these policies is that they require training ad-hoc and complex simulst architectures, which results in high computational costs. computational costs are also inflated by the common practice of simulating the simultaneous test conditions by providing partial input during training to avoid the quality drops caused by the mismatch between training and test conditions @xcite @xcite @xcite @xcite . this practice is independent of the decision policy adopted, and typically requires dedicated trainings for each latency regime. to mitigate this issue, offline-trained st systems have been employed for simultaneous inference @xcite @xcite and, along this direction, @xcite demonstrated that dedicated trainings simulating the inference conditions are not necessary since offline-trained systems outperform those specifically trained for simulst. the effectiveness of using offline-trained st models for simultaneous inference has been also confirmed by the last iwslt 2022 evaluation campaign @xcite , where the winning submission to the simulst task @xcite ) is an offline model exploiting the local agreement policy by @xcite . however, despite its good results, this policy relies on a strategy (the generation of two consecutive hypotheses prior to starting the emission) that has a significant impact on latency. this raises the need for effective policies that i) are adaptive, ii) are directly applicable to offline st models, and iii) achieve low latency at low computational costs. towards these objectives, we propose edatt (encoder-decoder attention), foot_2 a novel adaptive policy for simulst that leverages the encoderdecoder attention patterns of an offline-trained st model to decide when to emit partial translations. in a nutshell, our idea is that the next word of the partial hypothesis at a given time step is safely emitted only if the system does not attend to the most recent audio frames, meaning that the information received up to that time step is sufficient to generate that word. building on this idea, our contributions are summarized as follows: ## background in terms of architectural choices, transformer @xcite and its derivatives @xcite @xcite @xcite are the de-facto standard both in offline and simultaneous @xcite @xcite . a generic transformer model is composed of an encoder, whose role is to map the input speech sequence @xmath0, ..., x n ] into an internal representation, and a decoder, whose role is to generate the output textual sequence @xmath1, ..., y m ] by exploiting the internal representation in an autoregressive manner @xcite , that is by consuming the previously generated output as additional input when generating the next one. the encoder and the decoder are composed of a stack of identical blocks, whose components may vary depending on the particular transformerbased architecture, although they all share the same dot-product attention mechanism @xcite . in general, the attention is a function that maps a query matrix q and a pair of key-value matrices (k, v ) to an output matrix @xcite . the output is obtained as a weighted sum of v , whose weights are computed through a compatibility function between q and k that, in the case of the scaled dot-product attention used in the original transformer formulation, is: in the encoder layers, q, k, and v are computed from the same speech input sequence x, realizing the so-called self -attention a self (x). differently, in the decoder layer, two types of attention are computed sequentially: self-attention, and encoder-decoder (or cross) attention. in the encoder-decoder attention, q comes from the previous decoder layer (or directly from the previously generated output y, in the case of the first decoder layer) while k and v come from the output of the encoder, hence the matrix can be expressed as a cross (x, y). in this work, we only exploit the encoder-decoder attention matrix to guide the model during simultaneous inference. therefore, we use the notation a instead of a cross for simplicity, and henceforth refer to this matrix as the encoder-decoder representation of a specific decoder layer d considering the attention head h. ## edatt policy we propose to exploit the information contained in the encoder-decoder attention matrix of an offline st model during inference to determine whether to wait for additional audio input or emit a partial translation. the use of attention as the core mechanism of our policy is motivated by related works in machine translation (mt) and language modeling, which prove that attention scores can encode syntactic dependencies @xcite and language representations @xcite , as well as align source and target tokens @xcite @xcite . we posit (and demonstrate in section 5) that this encoderdecoder attention relationship between source audio and target tokens also exists in offline st models, and can be used to guide them during simultaneous inference. our approach builds on the following hypothesis (see figure 1 ): at each time step, if the attention is focused towards the end of the input audio sequence (1), the system will probably need more information to correctly produce the current output candidate. on the contrary (2), if the attention concentrates on early audio frames (far enough from the last received ones), the current output candidate can be safely emitted because the early encoded information is sufficient. accordingly, the model will continue to emit the next token of the partial hypothesis until the above condition is verified, that is until its encoder-decoder attention scores do not focus towards the end of the received speech segment. the rationale is that if the encoder-decoder attention of the predicted token points to the most recent speech information -i.e. attention scores are higher towards the last audio frames receivedthis information could be incomplete and therefore still insufficient to generate that token. more formally, at each time step t, edatt determines whether to emit the next token y j , given the previously generated tokens y j-1 = [y 1 , ..., y j-1 ] and the partial audio input sequence x t , by looking at the sum of the last λ encoder-decoder attention weights of the vector a j (x t , y j-1 ). specifically, y j is emitted if: ## attention analysis to validate our hypothesis and study the feasibility of our method, we start by exploring the encoderdecoder attention matrices of the offline trained models. we proceed as follows: first, by visualizing the attention weights, we check for the existence of patterns that could be exploited during simultaneous inference. then, we analyze the performance of the edatt policy to discover the best value of λ, the decoder layer d, and the attention head h from which to extract the attention scores that better balance the quality-latency trade-off. ## related works the first policy for simulst was proposed by @xcite and is derived from the wait-k policy @xcite developed for simultaneous text-to-text translation. most of subsequent studies have also adopted the wait-k policy @xcite @xcite @xcite @xcite . in parallel, several strategies have been developed to directly learn the best policy during training by means of ad-hoc architectures @xcite @xcite and training procedures aimed at 9 despite the benefits in terms of quality-latency tradeoff, the significantly higher costs of the a40 gpu over the k80 gpu (4.1 vs 0.9 usd/h in amazon web services, https://aws.amazon.com/it/ec2/ pricing/on-demand/ ) makes unlikely that such a gpu will soon be of widespread use for simultaneous inference. reducing latency @xcite @xcite @xcite . the latter adaptive policies obtained better performance according to the most recent results observed in @xcite . we define our policy as adaptive as well, as it relies on the encoder-decoder attention mechanism, whose dynamics are influenced by the audio input that increases incrementally over time. however, edatt completely differs from prior works on adaptive policies that exploit attention @xcite @xcite because is the first policy that does not require influencing the behaviour of the attention weights through dedicated training strategies, therefore being directly applicable to offline-trained st models. by doing so, we realize i) an adaptive policy, ii) directly applicable to offline-trained st models, iii) which achieves low latency at low computational costs. ## conclusions after investigating the encoder-decoder attention behavior of offline st models, we presented edatt, a novel adaptive decision policy for simulst that guides an offline st model to wait or to emit a partial hypothesis by looking at its encoder-decoder attention weights. comparisons with state-of-the-art simulst architectures and decision policies reveal that, at lower computational costs, edatt outperforms the others at almost every latency, with translation quality gains of up to 7.0 bleu for @xmath2.0 bleu for @xmath3. moreover, it is also capable of achieving a computational-aware latency of less than 2s with a reduction of 0.7-1.4s compared to existing decision policies applied to the same offline st systems.