A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation (2024)

Abstract: Simultaneous translation models play a crucial role in facilitating communication. However, existing research primarily focuses on text-to-text or speech-to-text models, necessitating additional cascade components to achieve speech-to-speech translation. These pipeline methods suffer from error propagation and accumulate delays in each cascade component, resulting in reduced synchronization between the speaker and listener. To overcome these challenges, we propose a novel non-autoregressive generation framework for simultaneous speech translation (NAST-S2 x ), which integrates speech-to-text and speech-to-speech tasks into a unified end-to-end framework.We develop a non-autoregressive decoder capable of concurrently generating multiple text or acoustic unit tokens upon receiving fixed-length speech chunks. The decoder can generate blank or repeated tokens and employ CTC decoding to dynamically adjust its latency. Experimental results show that NAST-S2 x outperforms state-of-the-art models in both speech-to-text and speech-to-speech tasks. It achieves high-quality simultaneous interpretation within a delay of less than 3 seconds and provides a 28× decoding speedup in offline generation.

## introduction simultaneous machine translation @xcite @xcite @xcite models are widely applied in communication scenarios, eliminating barriers between individuals with different linguistic backgrounds. in practice, simultaneous translation systems can be broadly categorized into speech-to-text (simul-s2t; @xcite and speech-to-speech (simul-s2s; @xcite variants. regardless of the modality of output, simultaneous translation models initiate generation ## approach we provide a detailed introduction to our nonautoregressive generation framework for end-toend simultaneous speech-to-any translation in this section. ## related work researches in simultaneous speech translation can be roughly categorized into simul-s2t @xcite and simul-s2s @xcite variants. simul-s2t with the rise of neural networks, simul-s2t models no longer rely on the transcription as a bridge @xcite . given the difference between speech and text input, some researchers focus on how to divide speech chunks and then execute strategies. ma et al. (2020c) employed fixed-length segmentation and implemented wait-k (ma et al., 2019) and mma (ma et al., 2020d) based on that; ren et al. (2020); zeng et al. (2021); chen et al. (2021) utilized asr results to partition and execute wait-k or its variants. zhang et al. ( 2022 ) trained a segmentation model to detect semantic units. @xcite trained a model to dynamically segment with differentiable approach, then extending it to a segment-to-segment framework @xcite . additionally, some researchers have also attempted to use transducer @xcite and incorporate attention mechanisms to enhance its performance @xcite . besides, some researchers are leveraging offline models for simultaneous inference. liu et al. (2020) considered the agreeing prefixes of two consecutive chunks as stable hypotheses. papi et al. (2023b,c) used attention as guidance, allowing the model to generate output for the current step if its attention is not focused on the most recently received frames. simul-s2s there have been limited prior studies exploring simul-s2s. zheng et al. (2020) and sudoh et al. (2020) both developed cascade models by integrating streaming asr, simul-t2t, and incremental tts components. additionally, liu et al. (2021b) proposed latency reduction strategies for incremental tts in simul-s2s. moreover, ma et al. (2022) introduced a variational version of mma to s2ut (lee et al., 2022) and constructed the first end-to-end simul-s2s model. ## conclusion in this paper, we present a non-autoregressive streaming generation framework for simultaneous speech-to-any translation, which integrates both simul-s2t and simul-s2s tasks into a unified framework. experimental results on various benchmarks showcase the superiority of our model.