Training Simultaneous Speech Translation with Robust and Random Wait-k-Tokens Strategy (2023)

Abstract: Simultaneous Speech Translation (SimulST) is a task focused on ensuring high-quality translation of speech in low-latency situations. Despite this, the modality gap ( e.g. , unknown word boundaries) between audio and text presents a challenge. This gap hinders the effective application of policies from simultaneous text translation (SimulMT) and compromises the performance of offline speech translation. To address this issue, we first leverage the Montreal Forced Aligner (MFA) and utilize audio transcription pairs in pre-training the acoustic encoder, and introduce a token-level cross-modal alignment that allows the wait- k policy from SimulMT to better adapt to SimulST. This token-level boundary alignment simplifies the decision-making process for predicting read/write actions, as if the decoder were directly processing text tokens. Subsequently, to optimize the SimulST task, we propose a robust and random wait- k -tokens strategy. This strategy allows a single model to meet various latency requirements and minimizes error accumulation of boundary alignment during inference. Our experiments on the MuST-C dataset show that our method achieves better trade-off between translation quality and latency.

## introduction simultaneous speech translation (simulst) is a task designed to generate real-time translations by incrementally consuming audio frames. a common practice is to cascade a streaming automatic speech recognition (asr) system and a simultaneous text machine translation (simulmt) model @xcite . the latter has been significantly improved earlier, as evidenced by @xcite . then, the prefix-to-prefix (p2p) framework (such as the wait-k policy) @xcite has been developed to reduce the discrepancy between training and inference. an efficient wait-k method was proposed by @xcite to train multiple paths by randomly sampling k values, instead of training multiple models w.r.t. different k values. the progress in simulmt has greatly influenced advancements in simulst. in particular, transformer-based endto-end neural architectures have achieved performance levels comparable to cascade systems in both offline and simultaneous st tasks @xcite @xcite @xcite . typically, an end-to-end simulst model requires a high-quality speech translation model that can take as input partial audio and a decision policy for controlling read/write actions. one simple but effective approach is to borrow ideas from simulmt. however, applying simulmt strategies directly to simulst poses a challenge because speech signals are continuous, while text tokens are discrete and have inherent meanings. therefore, the strategies of simulmt, which defines input granularity at the token level, cannot be directly applied to simulst. integration of a segmentation module into simulst has become a common practice, as it serves as a prerequisite for resembling simulmt policies. notable examples include the connectionist temporal classification (ctc)-based approach @xcite and the fixed-sized chunk method @xcite . however, ctc was not designed for simultaneous scenarios and the fixed-sized chunk method lacks flexibility in representing semantic meanings. the continuous integrate-and-fire (cif) method @xcite , which is ideally suited for streaming input, has recently attracted more attention in the streaming speech-to-text area @xcite @xcite . our work builds upon the research line of the cif module. we propose a fine-grained cif module as the traditional cif lacks clear and interpretable training supervision, as shown in figure 1 . we use the montreal forced aligner (mfa) @xcite to acquire frame-word or -phoneme alignments, and use them to pre-train an acoustic encoder with a fine-grained cif loss. by aligning speech and transcription, we can readily adapt the read and write policy of simulmt to simulst. an issue similar to the exposure bias @xcite , whereby the prediction error of segmentation may cause a gap between training and inference, is rarely discussed in previous works. we propose a robust-alignment training strategy to minimize this gap. moreover, previous simulst works based on wav2vec2.0 @xcite often overlook its bidirectional attention, which isn't inherently a streaming fashion. we discuss this by suggesting a random wait-k policy, making our model compatible with a bidirectional encoder. our robust strategy solely involves frame-transcription alignment and is independent of the read/write policy, thus making it feasible to combine with other adaptive policies. our main contributions can be summarized as follows. ## related works speech translation can be roughly classified into offline and simultaneous scenarios, with our discussion primarily focusing on end-to-end models. offline speech translation generates each target token based on the full audio representation. following the advent of the first end-to-end neural network model for st @xcite , a significant portion of end-to-end st works emphasized offline scenarios. @xcite ; @xcite ; @xcite ; alinejad and sarkar (2020); @xcite demonstrated the effectiveness of pre-training in improving performance. consequently, this has become the standard paradigm for current end-to-end models. other prevalent research areas include multitask learning @xcite @xcite @xcite , knowledge distillation @xcite @xcite , and curriculum learning @xcite . simultaneous speech translation was first realized through a pipelined system featuring streaming asr and a simulmt model. asr played a natural role in segmenting speech frames and aligning with transcriptions, albeit at the cost of increased latency. recently, end-to-end simulst has attracted more research attention. thus, a simulst model requires a policy that determines whether to wait for more speech frames or generate new target tokens. previous works often used a fixed size speech chunk @xcite @xcite . adaptive size policy has been thoroughly explored in simulmt @xcite @xcite . as to simulst, the works of @xcite are most relevant to our research. the former first applied cif to simulst, and the latter determined the read/write policy by learning to segment audio frames into meaningful units. our model achieves cross-modal token-level alignment through pre-training a novel fine-grained cif mod-ule, thereby facilitating downstream simulst. considering the frame length of each segmentation, our model also triggers an adaptive policy. ## main method end-to-end simultaneous speech translation models typically employ a sequence-to-sequence learning framework built on an encoder-decoder architecture. for a standard st training corpus, @xmath0, x, y)} represents a triplet dataset that includes audio, transcription, and translation sequences. the simulst model is generally defined as a probabilistic model: in our work, the aim of the pre-training stage is to embed audio features into the text representation space (i.e., source embedding space) and to identify the candidate ending timestamp for the function g(j). the fine-tuning stage directly optimizes the simulst model as per eq. ( 1 ). to achieve these objectives, we initially utilize the transcription-translation pairs in d to pre-train an offline machine translation model (nmt). in practice, additional mt corpus is more readily available, allowing us to pre-train the nmt using even larger data sets. the overall model architecture, comprising both stages, is displayed in figure 2 . ## conclusion we have proposed a two-step training method for simultaneous speech translation. the first step involves performing a token-level cross-modal alignment between the audio and the transcription in terms of both sequence length and semantic rep-resentation. the second step entails end-to-end training of the st model, leveraging a robust and random wait-k tokens policy. remarkably, our model manages to satisfy the objectives of offline and streaming speech translation within a single checkpoint. experimental results substantiate that our approach achieves a commendable trade-off between translation quality and latency. given that our semantic alignment pre-training is independent of the downstream policy in simultaneous translation, we aim to explore opportunities to better accommodate an adaptive read/write policy in future work.