S tream S peech: Simultaneous Speech-to-Speech Translation with Multi-task Learning (2024)

Abstract: Simultaneous speech-to-speech translation (Simul-S2ST, a.k.a streaming speech translation) outputs target speech while receiving streaming speech inputs, which is critical for real-time communication. Beyond accomplishing translation between speech, Simul-S2ST requires a policy to control the model to generate corresponding target speech at the opportune moment within speech inputs, thereby posing a double challenge of translation and policy. In this paper, we propose StreamSpeech, a direct Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning. Adhering to a multi-task learning approach, StreamSpeech can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an “All-in-One” seamless model. Experiments on CVSS benchmark demonstrate that StreamSpeech achieves state-of-the-art performance in both offline S2ST and Simul-S2ST tasks. Besides, StreamSpeech is able to present high-quality intermediate results (i.e., ASR or translation results) during simultaneous translation process, offering a more comprehensive real-time communication experience.

## introduction simultaneous speech-to-speech translation (simul-s2st), which involves generating target speech while concurrently receiving streaming speech inputs @xcite , has become an indispensable technology for low-latency communication in various scenarios, such as international conferences, live broadcasts and online subtitles. to produce high-quality translated speech under low latency, simul-s2st requires a policy to determine the optimal moments to start translating within the streaming speech inputs (i.e., read action) and subsequently generate coherent target speech outputs (i.e., write action) @xcite . existing simultaneous translation methods focus on text-to-text (simul-t2tt) @xcite @xcite and speech-to-text translation (simul-s2tt) @xcite @xcite . such methods typically require cascading external modules such as speech recognition (asr) and text-to-speech synthesis (tts) to accomplish simul-s2st. however, this cascaded approach tends to amplify inference errors progressively between modules @xcite , and also impedes the joint optimization of various modules @xcite . to address these issues, developing a direct simul-s2st model is imperative, particularly given the promising potential exhibited by end-to-end models such as @xcite and @xcite . direct speech-to-speech translation (s2st) is already highly challenging, and the goal of accomplishing it simultaneously (simul-s2st) further streaming speech encoder simultaneous text decoder text-to-unit encoder unit ctc decoder : _hello _every one : 63 16 3 23 23 54 … source ctc decoder target ctc decoder : _hello _every one : _bonjour _à _tous asr nar-s2tt ar-s2tt s2ut … … … <@xmath0. for translation, speech involves more diverse representation due to additional features such as timbre and intonation @xcite , which renders directly translating source speech to target speech challenging. in simultaneous scenarios, beyond translation, the model additionally requires a policy to identify the appropriate translating moments, which is nontrivial to directly accomplish due to the continuous nature and uncertain duration of speech @xcite . therefore, simul-s2st faces the double challenges of translation and policy. to address the challenges of translation and policy, we aim to introduce textual information of both source and target speech to guide simul-s2st, which can not only provide intermediate supervision for translation but also guide the policy by establishing an alignment between source and target speech with text as a bridge. specifically, a reasonable policy should control the model to wait until recognizing the presence of text in the received speech (read), facilitated by the alignment between source speech and source text. subsequently, the model should generate target speech corresponding to inputs (write), which can be guided by the alignments from the source speech to target text and from the target text to target speech. given the pivotal role of text in both translation and alignment-guided policy, we propose stream-speech, a direct simul-s2st model that jointly learns translation and policy in a unified framework of multi-task learning. streamspeech employs the advanced two-pass architecture @xcite , which first translates source speech into target text hidden states, and then converts the text hidden states into target speech. furthermore, we introduce multiple connectionist temporal classification (ctc) @xcite 2 background speech-to-speech translation (s2st) the corpus we used for speech-to-speech translation (s2st) task is denoted as quadruple: the current mainstream methods for s2st @xcite extract a discrete unit sequence @xmath1, • • • , u |u | from the target speech, and employ a two-pass architecture, where both the first and second passes use autoregressive encoderdecoder. the first pass transforms the source speech to target text hidden states, and the second pass generates the discrete unit sequence based on the text hidden states, followed by a pre-trained unit-based hifi-gan vocoder @xcite for target speech synthesis. in addition to the primary speech-to-unit translation (s2ut, @xmath2, an auxiliary speech-to-text translation task (s2tt, @xmath3. connectionist temporal classification (ctc) @xcite ctc is a technique used to model alignment between two sequences of unequal lengths. for a longer input sequence x , ctc decoder generates a same-length sequence z containing repeated and blank tokens ϕ, which is subsequently shortened by merging consecutively repeated tokens and removing blank tokens ϕ via collapsing function π (•). during training, given the ground-truth sequence y, ctc loss is calculated on all sequences z that can be reduced to y via the collapsing function: ## related work recent research often focuses on simultaneous textto-text (simul-t2tt) and speech-to-text (simul-s2tt) translation. simul-t2tt simul-t2tt methods fall into fixed and adaptive. for fixed methods, @xcite proposed wait-k policy, which waits for k tokens before alternately read/write one token @xcite @xcite . for adaptive methods, monotonic attention @xcite @xcite , alignments @xcite , nonautoregressive architecture @xcite or language models @xcite are employed to dynamically perform 7 conclusion in this paper, we propose streamspeech, an "all in one" seamless model that handles streaming asr, simultaneous translation and real-time speech synthesis via a unified model. experiments show the superiority of streamspeech on offline s2st, streaming asr, simultaneous s2tt and simultaneous s2st. moreover, intermediate products such as asr or s2tt results can also be presented to user during translation process as a reference, offering a better low-latency communication experience. linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 4252-4261, online. association for computational linguistics. yun tang, juan pino, changhan wang, xutai ma, and dmitriy genzel. 2021b. a general multi-task learning framework to leverage text data for speech to text tasks. in icassp 2021 -2021 ieee international conference on acoustics, speech and signal processing (icassp), pages 6209-6213. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in i. guyon, u. v. luxburg, s. bengio, h. wallach, r. fergus, s. vishwanathan, and r. garnett, editors, advances in neural information processing systems 30, pages 5998-6008. curran associates, inc. changhan wang, anne wu, and juan pino. 2020. covost 2: a massively multilingual speech-to-text translation corpus. xingshan zeng, liangyou li, and qun liu. 2021. real-trans: end-to-end simultaneous speech translation with convolutional weighted-shrinking transformer. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 2461-2474, online. association for computational linguistics. ruiqing zhang, zhongjun he, hua wu, and haifeng wang. 2022a. learning adaptive segmentation policy for end-to-end simultaneous translation. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 7862-7874, dublin, ireland. association for computational linguistics. shaolei zhang, qingkai fang, zhuocheng zhang, zhengrui ma, yan zhou, langlin huang, mengyu bu, shangtong gui, yunji chen, xilin chen, and yang feng. 2023. bayling: bridging cross-lingual alignment and instruction following through interactive translation for large language models. shaolei zhang and yang feng. 2021a. ict's system for autosimtrans 2021: robust char-level simultaneous translation. in proceedings of the second workshop on automatic simultaneous translation, pages 1-11, online. association for computational linguistics. shaolei zhang and yang feng. 2021b. universal simultaneous machine translation with mixture-of-experts wait-k policy. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 7306-7317, online and punta cana, dominican republic. association for computational linguistics. shaolei zhang and yang feng. 2022a. gaussian multihead attention for simultaneous machine translation. in findings of the association for computational linguistics: acl 2022, pages 3019-3030, dublin, ireland. association for computational linguistics. shaolei zhang and yang feng. 2022b. informationtransport-based policy for simultaneous translation. in proceedings of the 2022 conference on empirical methods in natural language processing, pages 992-1013, abu dhabi, united arab emirates. association for computational linguistics. shaolei zhang and yang feng. 2022c. modeling dual read/write paths for simultaneous machine translation. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2461-2477, dublin, ireland. association for computational linguistics. shaolei zhang and yang feng. 2022d. reducing position bias in simultaneous machine translation with length-aware framework. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 6775-6788, dublin, ireland. association for computational linguistics. shaolei zhang and yang feng. 2023a. end-to-end simultaneous speech translation with differentiable segmentation. in findings of the association for computational linguistics: acl 2023, pages 7659-7680, toronto, canada. association for computational linguistics. shaolei zhang and yang feng. 2023b. hidden markov transformer for simultaneous machine translation. in the eleventh international conference on learning representations. shaolei zhang and yang feng. 2023c. unified segmentto-segment framework for simultaneous sequence generation. in advances in neural information processing systems, volume 36, pages 45235-45258. curran associates, inc. shaolei zhang, yang feng, and liangyou li. 2021. future-guided incremental transformer for simultaneous translation. proceedings of the aaai conference on artificial intelligence, 35(16):14428-14436. shaolei zhang, shoutao guo, and yang feng. 2022b. wait-info policy: balancing source and target at information level for simultaneous machine translation. in findings of the association for computational linguistics: emnlp 2022, pages 2249-2263, abu dhabi, united arab emirates. association for computational linguistics.