Direct Speech-to-Speech Translation With Discrete Units (2022)

Abstract: We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We tackle the problem by first applying a self-supervised discrete speech encoder on the target speech and then training a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the discrete representations of the target speech. When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass. Experiments on the Fisher Spanish-English dataset show that the proposed framework yields improvement of 6.7 BLEU compared with a baseline direct S2ST model that predicts spectrogram features. When trained without any text transcripts, our model performance is comparable to models that predict spectrograms and are trained with text supervision, showing the potential of our system for translation between unwritten languages.

## introduction speech translation aims at converting speech from one language into speech or text in another language. the technology helps bridge the communication barriers between people speaking different languages and can provide access to multimedia content in different languages. conventional speech-to-text translation (s2t) systems take a cascaded approach by concatenating automatic speech recognition (asr) and machine translation (mt). in recent years, end-to-end s2t @xcite is proposed to alleviate the error propagation issue between asr and mt. these s2t models can be further combined with text-to-speech (tts) 1 audio samples are available at https: //facebookresearch.github.io/speech_ translation/direct_s2st_units/index.html synthesis to provide both speech and text translation, which allows the technology to be adopted in a wider range of applications. more recently, researchers have started exploring building direct speech-to-speech translation (s2st) models without relying on text generation as an intermediate step @xcite . direct s2st has the benefits of lower computational costs and inference latency as fewer decoding steps are needed compared to cascaded systems. in addition, direct s2st is a natural approach for supporting translation for languages without a writing system @xcite . @xcite first addresses the problem by training an attention-based sequence-to-sequence model that maps source speech spectrograms into target spectrograms. model training is challenging as it requires the model to learn not only the alignment between two languages but also the acoustic and linguistic characteristics of both languages. as a result, there is a performance gap between the direct s2st system and an s2t+tts cascaded system. the recent success in self-supervised learning for speech has demonstrated that speech representations learned from a large unlabelled speech corpus can lead to impressive performance on a variety of downstream tasks @xcite including asr @xcite , speaker and language identification @xcite , etc. moreover, discretized speech units obtained from the clustering of self-supervised speech representations allow researchers to take advantage of existing nlp modeling techniques on speech, such as spoken generative language modeling @xcite . in this work, we tackle the challenge of modeling target speech in direct s2st by predicting selfsupervised discrete representations of the target speech instead of mel-spectrogram features. compared with spectrogram features, self-supervised discrete units can disentangle linguistic content from speaker identify or prosodic information in speech @xcite . with the use of discrete units, we can also apply common practice such as beam search during inference. we investigate direct s2st with discrete units in the scenarios where the source and target transcripts may or may not be available, the latter case being representative of unwritten languages. for the written languages, we present a framework that jointly generates speech and text output by combining s2st and s2t tasks through a shared encoder and a partially shared decoder. we resolve the length mismatch issue between the speech and text output during decoding with connectionist temporal classification (ctc) @xcite . experiments show that with the combination of discrete units prediction, speech and text joint training and beam search, our direct s2st system matches the performance of a cascaded s2t+tts system. for the unwritten target languages, we first extend the use of discrete units to text-to-speech translation @xcite when there are source text transcripts available. then we show that with multitask learning using both discrete representations for the source and the target speech, it is possible to train a direct s2st system without the use of any text transcripts. in addition, we measure the system runtime and memory usage during inference and empirically show that the proposed framework is the most efficient compared to the direct s2st system that predicts spectrogram features or other cascaded systems. the rest of this paper is organized as follows. after introducing background and related work in the next section, we describe our system in detail in sec. 3. following this, we present experimental results including objective evaluation on translation quality, subjective evaluation on speech quality, and system benchmark in sec. 4. finally sec. 5 concludes with a discussion of potential future work. ## related work conventional s2st systems are built by combining either cascaded or end-to-end s2t models with tts @xcite . the majority of the speech translation research has focused on the s2t setup. studies on asr+mt systems explore better ways to integrate asr output lattice to mt models @xcite in order to alleviate the error propagation issue between the two. end-to-end s2t @xcite has the potential to resolve the issue, as long as it is properly trained with multitask learning @xcite , model pre-training @xcite or data augmentation @xcite to overcome the data scarcity problem. studies on tts for s2st focus more on synthesizing the paralinguistic information transferred from the source speech, such as prosody @xcite and word-level emphasis @xcite . on the other hand, translatotron @xcite ) is an attention-based sequence-to-sequence framework that directly translates mel-spectrogram of the source speech into spectrogram features of the target speech. multitask learning is essential in facilitating the model to converge, though there is still a performance gap towards a s2t+tts cascaded system. the follow-up and concurrent work, translatotron 2 @xcite , addresses the over-generation issue by conditioning the spectrogram synthesizer directly on the output from the auxiliary target phoneme decoder. @xcite propose to build a single deep-learning framework step-by-step by pre-training asr, mt and tts models separately and connecting them with transcoder layers. however, the inference process requires the asr and mt decoders to complete decoding a full sequence, and thus it loses the latency advantage of a direct s2st system. @xcite ; @xcite both investigate direct s2st models under the unwritten language setup by transforming the target speech into discrete representations through variational auto-encoder (vae), training a sequence-to-sequence model for translation into target discrete units, and an inverter for converting the units to speech. in this work, we propose to train a transformerbased speech-to-discrete unit model for direct s2st. we design a text decoding task conditioned on the intermediate representation of the discrete unit decoder in addition to the auxiliary tasks proposed in @xcite . we choose to use hubert @xcite to generate the target self-supervised discrete units, since @xcite ; @xcite ; @xcite have shown its superior performance across asr, spoken language modeling and speech synthesis, compared to other unsupervised representations, including vae-based representations used in @xcite . overall, there exists little work on direct s2st due to the lack of parallel s2st training data. while jia et al. (2019b) performs one set of experiments on in-house real-world s2st data, jia et al. (2019b, 2021); tjandra et al. (2019); zhang et al. (2020); kano et al. (2021) all take advantage of tts services to produce synthetic target speech for model training. we follow the same approach and conduct our experiments with single-speaker synthetic target speech. ## conclusion we investigate training direct s2st models with the use of self-supervised discrete representations as targets. we examine model training under both the written and unwritten language scenarios. for the former, we propose a framework with joint speech and text training that performs on par with an s2t+tts baseline, yet it can run more efficiently. we demonstrate the possibility of translating between two unwritten languages by taking advantage of discrete representations of both the source and the target speech for model training. our empirical benchmark shows that the proposed direct s2st system with discrete units is the most efficient during inference compared with a direct s2st model that predicts spectrogram features or other cascaded systems. with the recent release of large-scale s2s dataset @xcite , we plan to investigate the proposed framework with real data in the future. another important aspect in generating speech output is the voice and prosody. in our work, we focus on content translation and leave the para-linguistic aspect of speech translation to future work. we use an open-sourced asr model for evaluation, so the results should be comparable with all future research in the field that follows the same evaluation protocol. we will also release the code for reproducing the experiments.