Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inference (2023)

Abstract: A popular approach to streaming speech translation is to employ a single offline model with a wait-k policy to support different latency requirements, which is simpler than training multiple online models with different latency constraints. However, there is a mismatch problem in using a model trained with complete utterances for streaming inference with partial input. We demonstrate that speech representations extracted at the end of a streaming input are significantly different from those extracted from a complete utterance. To address this issue, we propose a new approach called Future-Aware Streaming Translation (FAST) that adapts an offline ST model for streaming input. FAST includes a Future-Aware Inference (FAI) strategy that incorporates future context through a trainable masked embedding, and a Future-Aware Distillation (FAD) framework that transfers future context from an approximation of full speech to streaming input. Our experiments on the MuST-C EnDe, EnEs, and EnFr benchmarks show that FAST achieves better trade-offs between translation quality and latency than strong baselines. Extensive analyses suggest that our methods effectively alleviate the aforementioned mismatch problem between offline training and online inference.

## introduction streaming speech translation (st) systems generate real-time translations by incrementally processing audio frames, unlike their offline counterparts that have access to complete utterances before translating. typically, streaming st models use unidirectional encoders @xcite @xcite @xcite et @xcite and are trained with a read/write policy that determines whether to wait for more speech frames or emit target tokens. however, it can be expensive to maintain multiple models to satisfy different latency requirements @xcite in real-world applications. recently, some works @xcite have shown that a single offline model with bidirectional encoders (such as wav2vec2.0 @xcite ) can be adapted to streaming scenarios with a wait-k policy @xcite to meet different latency requirements and achieve comparable or better performance. however, there is an inherent mismatch in using a model bidirectionally trained with complete utterances on incomplete streaming speech during online inference. intuitively, speech representations extracted from streaming inputs (figure 1(b) ) are less informative than those from full speech encoding (fig-ure 1(a)) due to limited future context, especially toward the end of the streaming inputs, which can be exacerbated by the aforementioned mismatch problem. this raises a natural question: how much do the speech representations differ between the two inference modes? we analyze the gap in speech representations, measured by cosine similarity, at different positions in the streaming input compared to using the full speech (section 3). we observe a significantly greater gap for representations closer to the end of a streaming segment, with an average similarity score as low as 0.2 for the last frame, and the gap quickly narrows for earlier frames (figure 2 ). additionally, we observe more degradation in translation quality for utterances with the greatest gap in speech representations between online and offline inference (see appendix b.2). we conjecture that the lack of future contexts at the end of streaming inputs can be detrimental to streaming speech translation when using an offline model. to this end, we propose a novel future-aware inference (fai) strategy. this approach is inspired by masked language models' ability @xcite to construct representations for masked tokens from their context. specifically, we append a few mask embeddings to the end of the streaming input and leverage the acoustic encoder (wav2vec2.0)'s ability to implicitly construct representations for future contexts, which can lead to more accurate representations for the other frames in the streaming input. furthermore, we propose a future-aware distillation (fad) framework that adapts the offline model to extract representations from streaming inputs that more closely resemble those from full speech encoding. we expand the original streaming input with two types of future contexts: one with m oracle speech tokens for the teacher model, and another with m mask tokens for the student model, which is initialized from the teacher model. we minimize several distillation losses between the output of the teacher and student models. by incorporating additional oracle future contexts, the speech representations for the frames in the original streaming input extracted by the teacher model resemble those when the full speech is available. fad aims to adjust the offline model to extract similar representations for streaming input as it would for full speech. in combination with fai, we improve the model's ability to extract quality representations during both training and inference, alleviating the aforementioned mismatch problem. we refer to our approach as fast, which stands for future-aware streaming translation. we conducted experiments on the must-c ende, enes, and enfr benchmarks. the results show that our methods outperform several strong baselines in terms of the trade-off between translation quality and latency. particularly, in the lower latency range (when al is less than 1000ms), our approach achieved bleu improvements of 12 in ende, 16 in enes, and 14 in enfr over baseline. extensive analyses demonstrate that our futureaware approach significantly reduces the representation gap between partial streaming encoding and full speech encoding. ## background and related work speech translation systems can be roughly categorized into non-streaming (offline) and streaming (online) depending on the inference mode. regardless of the inference mode, speech translation models typically employ the encoder-decoder architecture and are trained on an st corpus @xmath0, z, y)}, where @xmath1, . . . , x t ) denotes an audio sequence, @xmath2, . . . , z i ) and @xmath3, . . . , y j ) the corresponding source transcription and target translation respectively. ## preliminary analysis in this section, we examine the mismatch problem in transformer-based @xcite st architecture between offline training and online decoding. in offline full-sentence st, the speech representation of each frame is obtained by attending to all frames, including future frames, in the transformer encoder layers. recently, a common approach in speech translation is to stack a pre-trained wav2vec2.0 @xcite as the acoustic encoder with a semantic mt encoder-decoder, resulting in state-of-the-art performance in the st task @xcite @xcite . this approach leverages the ability of wav2vec2.0 pre-training to learn better speech representations. when applying an offline model to streaming inference, the lack of future frames causes an apparent mismatch problem, which can lead to a deterioration in the extracted speech representations. to quantify this effect, we examine three offline st models trained on the must-c ende dataset using the chimera @xcite , stemm @xcite , and mosst @xcite architectures, with a trainable acoustic encoder initialized from wav2vec2.0. we conduct analysis on the tst-common set with a duration between 2s and 10s by removing outliers and noisy data, resulting 1829 examples. for an input sequence of audio frames @xmath4, . . . , x t ), the convolutional subsampler of wav2vec2.0 shrinks the length of the raw audio by a factor 320 and outputs the full speech representation sequence a. for readability reasons, we uniformly use the notation t to denote the sequence length of @xmath5, . . . , a t ). this simplified notation does not undermine any of our conclusions while making the equations more readable. for streaming input ∀@xmath6, @xmath7, . . . , x t ), wav2vec2.0 will output the representation ât = (â t,1 , . . . , ât,t ). to quantify the difference in speech representa-tions between offline and online inputs, we compute the cosine similarity s t,t ′ between the speech representation at the t ′ -th (t ′ ≤ t) position in the streaming audio input xt and at the same position with full-sentence encoding. we then calculate the statistics s-τ by averaging the cosine similarity over both the testset b and the time dimension with a reverse index -τ corresponding to a position τ -1 frames before the end of the streaming input. figure 2 displays the s-τ curve for the last 100 positions in streaming inputs. for τ > 10, the averaged cosine similarity s-τ is greater than 0.8, indicating that the representations at those positions in a streaming input are similar to those with the full speech. however, the curve shows a sharp decline in the averaged cosine similarity s-τ for the ending positions, particularly for the last one (τ = 1), suggesting that the mismatch problem can significantly affect the quality of speech representation for these positions. we provide additional analysis in appendix b. ## method to address the mismatch problem between offline training and online inference, we propose a novel methodology called future-aware streaming translation (fast). this approach adapts an offline st model for streaming scenarios by using a future-aware inference (fai) strategy during inference and a future-aware distillation (fad) strategy during training. an overview of our proposed method is depicted in figure 3 . ## conclusion in this paper, we examine streaming speech translation from a new perspective. we investigate the effects of the input mismatch between offlinetraining and online-decoding. we find that the rep-resentations at the ending positions in the streaming input are particularly poor, directly impacting the translation quality. we propose fast, which introduces future contexts to improve these representations during training and testing via fad and fai, respectively. experiments and analysis demonstrate their effectiveness in bridging the representation gap between full speech encoding and partial streaming encoding. furthermore, our methods can be generally beneficial to streaming speech translation models that are based on wav2vec2.0. in the future, we will explore the relevant method independent on wav2vec2.0.