D iff S 2 UT : A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech Translation (2023)

Abstract: While Diffusion Generative Models have achieved great success on image generation tasks, how to efficiently and effectively incorporate them into speech generation especially translation tasks remains a non-trivial problem. Specifically, due to the low information density of speech data, the transformed discrete speech unit sequence is much longer than the corresponding text transcription, posing significant challenges to existing auto-regressive models. Furthermore, it is not optimal to brutally apply discrete diffusion on the speech unit sequence while disregarding the continuous space structure, which will degrade the generation performance significantly. In this paper, we propose a novel diffusion model by applying the diffusion forward process in the continuous speech representation space, while employing the diffusion backward process in the discrete speech unit space. In this way, we preserve the semantic structure of the continuous speech representation space in the diffusion process and integrate the continuous and discrete diffusion models. We conduct extensive experiments on the textless direct speech-to-speech translation task, where the proposed method achieves comparable results to the computationally intensive auto-regressive baselines (500 steps on average) with significantly fewer decoding steps (50 steps).

## introduction speech-to-speech translation (s2st) aims at translating speech of one language to speech of another language, breaking the communication barriers between people around the world who speak different languages. conventional cascaded systems @xcite for s2st typically consist of automatic speech recognition (asr), machine translation (mt) or end-to-end speech-to-text translation (s2t), followed by textto-speech synthesis (tts). however, integrating multiple separated modules into a single system would cause the problem of error propagation and incur expensive computational costs. on one hand, text-based cascaded systems face challenges when dealing with low-resource languages that lack text annotations or even without written systems @xcite . on the other hand, the speech-totext process disregards acoustic information such as accentuation, attitude, and emotional nuances, which are crucial for effective human spoken communication. recent works of textless direct s2st models @xcite @xcite @xcite @xcite @xcite directly translate speech to speech in an end to end manner without the intermediate generation steps relying on text transcripts. among them, s2ut model @xcite takes the advantage of recent progress in spoken language modeling @xcite to obtain the discrete speech representations, or discrete units, to build the textless s2st systems. the entire system consists of a sequence-to-sequence transformer @xcite with a speech encoder and an auto-regressive unit decoder, followed by a unit hifi-gan vocoder @xcite trained separately for unit-towaveform conversion. experiments @xcite have demonstrated notable improvements in translation for unwritten languages. despite the promising translation capabilities of the aforementioned auto-regressive-based s2st models, they suffer from o(n ) decoding time complexity for predicting one token per step. furthermore, the learned discrete speech unit sequences are typically much longer than the corresponding text transcripts due to the low information density of speech data, with a rate of 50 units per second (sequence length up to 500 on average), where the generation latency bottleneck is exposed. however, speech is a unique modality that lies between audio and text. on one hand, speech is a continuous form of acoustic data represented as a waveform. on the other hand, speech encompasses high-level linguistic information similar to discrete text data and can be represented as discrete units with ssl models, which is crucial for the translation task. it is challenging to directly apply the diffusion forward process like images or texts for s2st. for example, translatoron @xcite directly generates the continuous spectrogram for speech translation but the performance is significantly inferior to the discrete unit based model s2ut @xcite . we conduct experiments similar to diffsound @xcite that uses naive discrete diffusion models to generate discrete units for s2st. our primary results (see to tackle these challenges, in this work, we propose a new paradigm called diffs2ut for the s2st task, a novel diffusion generative model that bridges the continuous representation space and discrete unit space of speech, while simultaneously preserving the semantic structure of the continuous speech representation space as well. specifically, as depicted in figure 1(c) , there exists an one-toone map between the discrete units and the continuous k-means centroid vectors. therefore, in the diffusion forward process, we first map a unit to the continuous vector derived from the learned k-means space. next, we gradually add gaussian noise to corrupt it with the continuous diffusion process. subsequently, the noisy embedding vector is converted into the discrete unit index through the nearest neighbor search. as a consequence, we obtain the perturbed discrete unit sequences which are derived from the semantically structured continuous space, rather than being uniformly sampled from the vocabulary as in @xcite . the perturbed unit sequences are semantically close to the unperturbed ones when the added noise is small, which cannot be achieved in the discrete diffusion process. in the diffusion backward process, we train the model to predict the uncorrupted discrete unit sequences with cross-entropy in parallel. we evaluate our framework diffs2ut on the real-world s2st datasets @xcite . the model achieves 14.8/15.2/14.5/13.6 bleu score on the eurapal-st es-en/fr-en/en-es/en-fr test sets, significantly surpassing the vanilla diffusion models, and achieving comparable performance to auto-regressive models while requiring much fewer generation steps. to summarize, the main contributions of this paper are: textless direct speech to speech translation textless direct speech-to-speech translation (s2st) aims at translating the source speech to the target speech directly without any text intermediates. translatoron @xcite is the first s2st model to translate speeches by generating spectrograms directly. tjandra et al. (2019); zhang et al. (2021) build the s2st system by pre-training a vq-vae model to convert target speeches into discrete codes and learn a speech-to-code translation model. s2ut model @xcite utilizes the self-supervised speech encoder hubert @xcite pre-trained on large corpus of unlabeled speeches to convert speeches into discrete units with k-means clustering, which outperforms the vq-vae based approach in @xcite . @xcite extend the s2ut model to the real-word s2st data voxpopuli-s2s @xcite and propose a speech normalizer to obtain the speaker-invariant representations. @xcite propose a large-scale multilingual s2st corpus with the speech mining method @xcite . by introducing the pre-trained models including the wav2vec @xcite encoder and mbart @xcite decoder to the s2ut model, the translation quality is further boosted in @xcite . transpeech @xcite alleviates the acoustic multi-modal problem with bilateral perturbation. speech2s @xcite performs joint pre-training on speech and text datasets to align the acoustic and textual modalities. although the above methods achieve good performance in the s2st task, they are built on the auto-regressive generation models and suffer from the slow generation speed due to the excessive lengths of speech sequences. our work leverages diffusion generative models to speed up the translation process without compromising the translation quality. ## background in this section, we start with the formulations of both continuous and discrete diffusion generative models. then we introduce some preliminaries for the proposed diffs2ut system, such as how to convert a speech to discrete units and units to a waveform. ## diffs2ut in this section, we elaborate our proposed model diffs2ut in detail. we first describe the model architecture for s2st, followed by the novel training and sampling algorithms for diffusion. ## conclusion we present a diffusion generative model for the s2st task named diffs2ut that integrates the continuous and discrete diffusion models by decoupling the diffusion forward and backward processes. the model applies the diffusion forward process in the continuous speech representation space, while employing the diffusion backward process in the discrete speech unit space. in this way, we effectively preserve the semantic structure of the continuous speech representation space in the diffusion process. we evaluate the proposed diffs2ut model on the real-world speech-to-speech datasets and demonstrate the significantly boosted generation quality compared to the conventional discrete diffusion models. furthermore, the model can achieve comparable results to auto-regressive models but with much fewer decoding steps.