Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? (2024)

Abstract: Recently proposed two-pass direct speech-to-speech translation (S2ST) models decompose the task into speech-to-text translation (S2TT) and text-to-speech (TTS) within an end-to-end model, yielding promising results. However, the training of these models still relies on parallel speech data, which is extremely challenging to collect. In contrast, S2TT and TTS have accumulated a large amount of data and pretrained models, which have not been fully utilized in the development of S2ST models. Inspired by this, in this paper, we first introduce a composite S2ST model named ComSpeech, which can seamlessly integrate any pretrained S2TT and TTS models into a direct S2ST model. Furthermore, to eliminate the reliance on parallel speech data, we propose a novel training method ComSpeech-ZS that solely utilizes S2TT and TTS data. It aligns representations in the latent space through contrastive learning, enabling the speech synthesis capability learned from the TTS data to generalize to S2ST in a zero-shot manner. Experimental results on the CVSS dataset show that when the parallel speech data is available, ComSpeech surpasses previous two-pass models like UnitY and Translatotron 2 in both translation quality and decoding speed. When there is no parallel speech data, ComSpeech-ZS lags behind by only 0.7 ASR-BLEU and outperforms the cascaded models.

## introduction direct speech-to-speech translation (s2st) refers to translating source language speech directly into target language speech using an end-to-end model @xcite . in contrast to traditional cascaded s2st @xcite , which generates target speech through a pipeline of automatic speech recognition (asr), machine translation (mt), and textto-speech (tts), direct s2st not only can reduce error accumulation along modules but is easier to deploy @xcite , so that it has attracted considerable interest from researchers in recent years @xcite . speech data usually employs smaller semantic units than text data, where a long span of speech units corresponds to a word. as a result, s2st has to carry out the conversion between longer sequences, which involves implicitly segmenting the speech units into dependent semantic groups and mapping between the source and target group sequences, making s2st extremely challenging. to reduce the complexity of directly converting speech input to speech output, some two-pass models @xcite introduce intermediate target text representations and guide their learning by predicting the ground truth target text during training. in this way, the direct speech-to-speech conversion is decomposed into two passes of speech-to-text conversion and textto-speech conversion. the generation of target text helps organize the long speech units into words at the semantic level, thereby facilitating training. despite the two-pass architecture, the models are still trained end-to-end by sequentially generating text and speech via multi-task learning. this modeling mechanism reduces learning complexity while retaining the benefits of end-to-end training, surpassing the performance of cascaded s2st. despite the success of two-pass s2st models, they still face a challenge: the training of these models still requires large-scale parallel speech data composed @xmath0, target text, target @xmath1, which is difficult to collect. on the other hand, s2tt and tts have accumulated a large amount of training data and numerous sophisticated models. leveraging these existing resources would definitely reduce the cost of constructing an s2st model. what fits perfectly is that both the output of s2tt and the input of tts are target text sequences. if s2tt and tts can share the representations of the text sequence, they can be connected to form a two-pass s2st model. in this way, the existing resources of s2tt and tts can be utilized and only a small amount of parallel speech data is needed to finetune the two-pass model. however, the s2tt and tts models have their own vocabularies. for example, s2tt models commonly use subword vocabularies, whereas tts models often employ phoneme or character vocabularies. this distinction implies that the target text will be segmented into different vocabulary sequences by s2tt and tts models. consequently, s2tt and tts cannot share the text representations and hence cannot be directly connected. to solve the above problems, we propose a composite speech-to-speech translation model, named comspeech, which connects s2tt and tts models with a vocabulary adaptor based on connectionist temporal classification (ctc; @xcite . the vocabulary adaptor models the conversion from the output vocabulary sequence of s2tt to the input vocabulary sequence of tts as a non-autoregressive sequence-to-sequence generation problem. with the help of ctc, it can explore all the possible generated sequences that can be organized into the input sequence of tts. finally, with only a small set of parallel speech data, com-speech can be trained end-to-end with gradient backpropagation through the model, thereby finetuning existing s2tt and tts models into a twopass s2st model. furthermore, we propose a zeroshot training strategy comspeech-zs, with which comspeech can be trained solely based on s2tt and tts training data. it aims to align the text representation space of s2tt and tts models via contrastive learning, enabling seamless connection between s2tt and tts models without the need for finetuning with parallel speech data. experimental results on the cvss dataset demonstrate that com-speech surpasses previous two-pass models like unity and translatotron 2 in both translation quality and decoding speed. in the zero-shot learning scenario, the translation quality of comspeech-zs is only 0.7 asr-bleu lower than comspeech and surpasses cascaded systems. the s2st dataset usually contains three-way parallel data d @xmath2, y, t )}, where s denotes the source speech, y is the target text, and t is the target speech. due to the challenges of directly generating target speech, recent developments have introduced two-pass s2st models, which enhance translation quality by incorporating the target text into the generation process. next, we will introduce the model structure of unity @xcite , as it is one of the most representative twopass s2st model. specifically, it consists of four sub-modules: the source speech encoder f enc , the target text decoder f dec , the text-to-speech encoder g enc , and the target speech decoder g dec . firstly, f enc and f dec perform the s2tt task, trained with the cross-entropy loss: ## proposed model: comspeech in this section, we introduce a more general twopass s2st model architecture: composite speechto-speech translation (comspeech) model. com-speech comprises three modules: f, a, and g, where f denotes arbitrary s2tt model, g denotes arbitrary tts model for the target language, and a represents the vocabulary adaptor inserted between f and g. the inclusion of the vocabulary adaptor a is necessitated by the fact that s2tt and tts models typically employ different vocabularies for the target language. it can facilitate the conversion of target text representation sequences between different vocabularies. figure 1 illustrates the model architecture of comspeech. formally, we use s to denote the filter bank features of the source speech, and use t to denote the target speech representation, which can be either mel-spectrograms or discrete units, both of which can be synthesized into the waveform using a vocoder. we use w and v to denote the vocabularies of the s2tt model and the tts model, respectively. the tokenized sequences of the target text y under these two vocabularies are represented as y @xmath3, ..., y w n ) and y @xmath4, ..., y v m ), respectively. next, we will introduce the detailed implementations of each module in comspeech. ## comspeech-zs: training comspeech without parallel speech data as mentioned earlier, collecting s2st data is extremely challenging. in this section, we explore the possibility of training comspeech exclusively on s2tt and tts data, achieving zero-shot s2st without the need for parallel speech data. the idea is to train distinct modules using s2tt and tts data concurrently, while achieving representation alignment in the output space of the tts encoder. this enables the speech synthesis capabilities learned from the tts data to generalize to s2st during inference. figure 2 illustrates the process of training and inference. specifically, we use d @xmath5, y )} and d @xmath6, t )} to denote the s2tt and tts datasets, respectively, and train the model concurrently using both datasets. for tts data batches, we train the tts model g with l tts , as illustrated in figure 2 @xcite . for s2tt data batches, firstly, we train the s2tt model f and vocabulary adaptor a with l s2tt and l ctc . secondly, since the tts encoder receives the target text embedding emb(y v ) during tts training but vocabulary adaptor's output h v during inference, we aim for the output representations of the tts encoder with these two inputs to be identical, thereby achieving zero-shot generalization. therefore, we introduce a representation alignment loss after the tts encoder during s2tt training, as illustrated in figure 2(b) . representation alignment formally, we use c @xmath7, ..., c h m ) and c @xmath8, ..., c e m ) to denote the output representations of tts encoder with these two types of input: firstly, we align representations with the mse loss: additionally, we incorporate a contrastive learning objective to align representations. this objective maximizes the similarity between representations from the same input pairs and minimizes the similarity between representations from different input pairs. the contrastive loss is formulated as: two-stage finetuning in the zero-shot learning scenario, we adopt a two-stage finetuning strategy to facilitate the learning process. in the first stage, we only utilize the s2tt dataset d s2tt to train the s2tt model f and vocabulary adaptor a through the objectives l s2tt and l ctc . in the second stage, we conduct training using both the s2tt and tts datasets, with the following training objectives: throughout the entire training process, the s2st data is not used. during inference, we can achieve zero-shot s2st as illustrated in figure 2(c ). ## conclusion in this paper, we first introduce a novel s2st model architecture, named comspeech, which incorporates a ctc-based vocabulary adaptor capable of connecting arbitrary s2tt and tts models to obtain an s2st model. furthermore, we propose a training method using only s2tt and tts data. by aligning in the tts encoder's representation space using contrastive learning, we achieve zeroshot s2st without relying on parallel speech data. experimental results on the cvss dataset demonstrate that comspeech surpasses previous s2st models in both translation quality and decoding efficiency. moreover, in the zero-shot learning scenario, our model achieves performance close to the supervised learning scenario and surpasses the cascaded system. in the future, we will explore building direct s2st models based on more powerful s2tt and tts models.