S tream A tt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection (2024)

Abstract: Streaming speech-to-text translation (StreamST) is the task of automatically translating speech while incrementally receiving an audio stream. Unlike simultaneous ST (SimulST), which deals with pre-segmented speech, StreamST faces the challenges of handling continuous and unbounded audio streams. This requires additional decisions about what to retain of the previous history, which is impractical to keep entirely due to latency and computational constraints. Despite the real-world demand for real-time ST, research on streaming translation remains limited, with existing works solely focusing on SimulST. To fill this gap, we introduce StreamAtt, the first StreamST policy, and propose StreamLAAL, the first StreamST latency metric designed to be comparable with existing metrics for SimulST. Extensive experiments across all 8 languages of MuST-C v1.0 show the effectiveness of StreamAtt compared to a naive streaming baseline and the related state-of-the-art SimulST policy, providing a first step in StreamST research.

## introduction streaming speech-to-text translation (streamst) is the task of automatically translating spoken content from the source language into the target language in real-time, while continuously receiving an input audio stream. by processing longer, unsegmented audio, streamst adds another layer of complexity to the difficulties of simultaneous st (simulst) which, instead, operates on -often manually -presegmented speech segments @xcite @xcite @xcite . in simulst, the primary objective revolves around finding a balance between producing highquality translations and minimizing latency. this balance is managed by a simultaneous policy, which is the strategy for determining, at each time step, whether to emit a partial translation hypothesis or to wait for additional audio input. this hypothesis, together with the processed audio, is temporarily stored in memory to provide context for subsequent generations and is automatically removed from memory at the end of each audio segment @xcite . however, when the input is a continuous, unbounded stream, the memory retained as useful context can indefinitely grow, rendering the direct application of conventional simulst approaches to streamst impractical due to latency and computational constraints. 1 despite representing the real-world scenario for providing real-time st in many applications, such as interpreting @xcite and lectures @xcite , and garnering increasing market interests, 2 research on streaming translation remains limited, with existing works solely focusing on text-to-text machine translation (mt) @xcite . moreover, as these works focus on (unbounded) text streams as input, there is currently no metric in the literature suitable to evaluate the streamst task, where the input is an audio stream. to fill these gaps, in this paper we delve into the unexplored domain of streamst and its associated challenges. first, we define the concept of streaming policy for st by dividing the decision-making process into two steps: hypothesis selection, to determine which part of the translation hypothesis should be emitted (akin to the simultaneous policy), and history selection, to identify which part of past audio and generated partial translations should be retained in memory. then, motivated by the success of direct st models @xcite in overcoming the high latency of cascade architectures in the related field of simulst @xcite @xcite @xcite , we propose streamatt foot_2 (section 3), the first streamst policy designed for direct st systems. to enable the evaluation of our streamst solution, we also introduce streamlaal 3 (section 4), the first latency metric for streamst. streamlaal is designed to facilitate a direct comparison with simulst solutions, which provide upper-bound results as they operate on pre-segmented audio. lastly, we demonstrate the effectiveness of streamatt through extensive experiments across all 8 languages of must-c v1.0. we show that our policy significantly outperforms a naive streaming baseline (section 6.1) that relies on a fixed number of past words and audio frames as memory, and is even competitive with the related state-of-the-art simulst policy at low latency (section 6.2), providing a first promising step in streamst research. ## related works while the terms "streaming" and "simultaneous" translation have often been used interchangeably in the literature, we adhere to the definition of streaming by @xcite , which refers to the handling of unbounded and continuous streams that, in our case, are audio streams. consequently, all works that assume to operate on pre-segmented audio input, and only focus on effective methods to determine when and what to emit @xcite @xcite @xcite , are hereinafter categorized as related to simulst. some of these works explore how to automatically detect word boundaries in the audio with dedicated modules integrated into the st system @xcite @xcite . however, they still use this information only to determine when and what to emit, lacking a mechanism to determine which portion of the memory has to be retained and which can be discarded, hence relying on pre-segmented audio for the simultaneous inference @xcite . it follows that all these works are limited to the hypothesis selection step (i.e., the simultaneous policy) and, differently from this paper, they all ignore the history selection step, which is necessary to deal with continuous audio inputs. in the context of text-to-text machine translation, the streaming scenario has instead been addressed in @xcite . these works rely on an mt system trained with the wait-k simultaneous policy @xcite , which consists in waiting for a predefined number of source tokens before starting the translation. the same policy is applied at inference time as the hypothesis selection strategy, while the history selection step consists of keeping a fixed number of textual segments (given by a segmenter model in @xcite or a memory mechanism integrated into the mt system in @xcite ) as a context for the current translation generation. as these works are tailored for textual inputs, the evaluation metrics they rely on do not directly apply to streamst, disregarding the crucial aspect of latency measurement from audio streams, which we instead also address in this work. ## streaming policy the generic decision steps of a streamst policy (figure 1 ) can be schematized as follows: hypothesis selection: given both audio and textual history and the newly received chunk of audio, the hypothesis selection step determines whether and how many of the newly predicted words to emit. this can be easily traced back to the role of a simulst policy. ## streaming latency metric for evaluating the performance of streamst, the standard metrics adopted in simulst cannot be applied as is. in fact, they are not designed to evaluate outputs obtained from entire audio streams but, instead, refer to (manually) segmented audio and their corresponding translations for the computation. we hence adapt them to the streaming scenario to define our streamst latency metric. specifically, we opt to use the family of latency metrics based on average lagging @xcite for speech @xcite , given their widespread adoption in simulst @xcite . among these metrics, we select the length-adaptive average lagging or laal @xcite , which corrects the standard al formulation to avoid the underestimation of the latency when predictions are longer than the reference translation. given @xmath0, ..., x |x| ] as the speech segment, where each element x j has duration t j , as the reference words, and @xmath1, ..., y |y| ] as the hypothesis words, laal for simulst is formulated as: t j } is the index of the target token when the end of the source sentence is reached, and d * i represents the delay of an oracle policy that starts to emit words as soon as the speech starts and is perfectly in sync with the speaker. we adapt laal by considering the entire (unsegmented) stream of audio @xmath2, ..., x |s| ] instead of the single speech segment x, for which we have a continuous stream of predicted translation words y s . since we have reference transla- with their corresponding delays by applying the mwersegmenter tool @xcite between each reference sentence y * i and the entire stream of predicted translation y s , in a similar fashion to what has been done for streaming @xcite . consequently, we obtain the laal for the entire audio stream (streamlaal), by computing: the laal metric is calculated for every speech segment x i of the stream s and its corresponding reference y * x i with the automatically aligned prediction y x i and then averaged over all the speech segments of the stream x 1 , ..., x |s| to obtain streamlaal. as this formulation builds upon the original laal metric, it enables direct comparisons between the results obtained in streamst and those reported in related works on simulst. in this way, we can measure the gap between streamst systems and their simulst counterparts, which provide upper-bound results as they operate on pre-segmented audio. ## results in this section, we first compare our proposed strea-matt policy for streamst with the streaming baseline (section 6.1) and then with the state-of-the-art simulst policy (section 6.2). this is followed by an analysis of our approach (section 6.3). ## conclusions our work addressed the underexplored domain of streamst, which tackles the challenge of translating spoken content from the source language to the target language while incrementally receiving an audio input stream. unlike simulst, which deals with pre-segmented speech chunks, streamst grapples with the inability to retain the entire growing history in memory due to latency and computational constraints. despite growing interest in its applications, research on streaming translation remains limited, with existing studies solely focusing on text-to-text translation, leaving the domain of streamst and its challenges, including the absence of a suitable evaluation metric, still unaddressed. to fill these gaps, in this paper, we delved into the domain of streamst by first defining the concept of streaming policy for st. then, building on insights from simulst research underscoring the efficacy of direct st systems in overcoming the latency issues of cascade architectures, we proposed streamatt, the first streamst policy tailored for direct st models. to enable the evaluation of streamst solutions, we also introduced stream-laal, the first streamst latency metric designed to facilitate direct comparisons with simulst models. through empirical evaluation on all 8 languages of must-c v1.0, we showed that streamatt significantly outperforms a naive streaming baseline, and is competitive with the simulst state-ofthe-art alignatt policy at lower latency, providing a first promising step in streamst research.