Textless Speech-to-Speech Translation on Real Data (2022)

Abstract: We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data. The key to our approach is a self-supervised unit-based speech normalization technique, which finetunes a pre-trained speech encoder with paired audios from multiple speakers and a single reference speaker to reduce the variations due to accents, while preserving the lexical content. With only 10 minutes of paired data for speech normalization, we obtain on average 3.2 BLEU gain when training the S2ST model on the VoxPopuli S2ST dataset, compared to a baseline trained on un-normalized speech target. We also incorporate automatically mined S2ST data and show an additional 2.0 BLEU gain. To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.

## introduction speech-to-speech translation (s2st) technology can help bridge the communication gap between people speaking different languages. conventional s2st systems @xcite usually rely on a cascaded approach by first translating speech into text in the target language, either with automatic speech recognition (asr) followed by machine tranlsation (mt), or an end-toend speech-to-text translation (s2t) model @xcite , and then applying text-to-speech (tts) synthesis to generate speech output. on the other hand, researchers have started exploring direct s2st @xcite @xcite @xcite , which aims at translating speech in the source language to speech in the target language without the need of text generation as an intermediate step. however, text transcriptions or phoneme annotations of the speech data is often still needed during model training for multitask learning @xcite or for learning a decoder that generates intermediate representations @xcite to facilitate the generation of speech output. more than 40% of the languages in the world are without text writing systems foot_1 , while very limited work exist to tackle the challenge of training direct s2st systems without the use of any text data @xcite . moreover, due to the lack of s2st training data, previous work on direct s2st mainly rely on tts to generate synthetic target speech for model training. the recent release of the large-scale s2st data from voxpopuli @xcite has opened up the possibility of conducting s2st research on real data. in addition, @xcite have demonstrated the first proof of concept of direct s2s mining without using asr or mt systems. the approach may potentially mitigate the data scarcity issue, but the authors had not evaluated the usefulness of such data for s2st frameworks. most recently, @xcite have proposed to take advantage of self-supervised discrete representations @xcite , or discrete units, learned from unlabeled speech data as the target for building a direct s2st model. experiments conducted with synthetic target speech data have shown significant improvement for translation between unwritten languages. in this work, we extend the textless s2st setup in @xcite , i.e. training an s2st system without the use of any text or phoneme data, and conduct experiments on real s2st datasets, including voxpop-uli @xcite and automatically mined s2st data @xcite . to tackle the challenge of modeling real target speech where there are multiple speakers with various accents, speaking styles and recording conditions, etc., we propose a speech normalization technique that finetunes a self-supervised pre-trained model for speech with a limited amount of parallel multipleto-single speaker speech. experiments on four language pairs show that when trained with the normalized target speech obtained from a speech normalizer trained with 10-min parallel data, the performance of a textless s2st model can be improved by 3.2 bleu points on average compared with a baseline with un-normalized target speech. the main contributions of this work include: ## related work direct @xcite propose a sequence-to-sequence model with a speech encoder and a spectrogram decoder that directly translates speech from one language into another language without generating text translation first. the model can be trained end-to-end, while phoneme data is required in model training. on the other hand, tjandra et al. (2019); zhang et al. (2020) build direct s2st systems for languages without text writing systems by adopting vector-quantized variational auto-encoder (vq-vae) (van den oord et al., 2017) to convert target speech into discrete codes and learn a speech-to-code translation model. most recently, lee et al. (2021) propose a direct s2st system that predicts self-supervised discrete representations of the target speech. the system, when trained without text data, outperforms vq-vae-based approach in zhang et al. (2020). as a result, in this work, we follow the design in lee et al. (2021) and focus on training direct s2st systems with real data. s2st data voxpopuli @xcite provides 17.3k hours of s2st data from european parliament plenary sessions and the simultaneous interpretations for more than 200 language directions, the largest to-date. there exists few s2st corpora as the creation process requires transcribing multilingual speech @xcite @xcite or highquality asr models @xcite . on the other hand, duquenne et al. (2021) extend distancebased bitext mining @xcite to the audio domain by first learning a joint embedding space for text and audio, where sentences with similar meaning are close, independent of the modality or language. the technique was applied to mine for speech-to-speech alignment in librivox foot_2 , creating 1.4k hours of mined s2st data for six language pairs. the usefulness of the s2st datasets is often showcased indirectly through a speech retrieval task @xcite or human evaluation of the data quality @xcite , since existing direct s2st systems are mostly trained with synthetic target speech @xcite @xcite @xcite . in this work, we develop an s2st system that can be trained on real target speech to mitigate the discrepancy between the s2st system and corpus development process. speech normalization speech normalization reduces the variation of factors not specified at the input when building tts systems. one manual approach is to use clean data from a single speaker with minimal non-textual variation @xcite @xcite . for automatic methods, silence removal with voice activity detection (vad) is a fundamental approach @xcite @xcite . speech enhancement can remove the acoustic condition variation when building tts models with noisy data @xcite . speaker normalization through voice conversion, which maps target speech into the same speaker as the source speech in the context of s2st @xcite , can be considered as another speech normalization method. in this work, we propose a novel speech normalization technique based on self-supervised discrete units, which maps speech with diverse variation to units with little non-textual variation. ## system we follow @xcite to use hubert @xcite to discretize target speech and build a sequence-to-sequence speech-to-unit translation (s2ut) model. we describe the proposed speech normalization method and the s2ut system below. ## experimental setup we examine four language pairs: spanish-english (es-en), french-english (fr-en), english-spanish (en-es), and english-french (en-fr). all experiments are conducted using fairseq @xcite @xcite 4 . ## conclusion we present a textless s2st system that can be trained with real target speech data. the key to the success is a self-supervised unit-based speech normalization process, which reduces variations in the multi-speaker target speech while retaining the lexical content. to achieve this, we take advantage of self-supervised discrete representations of a reference speaker speech and perform ctc finetuning with a pre-trained speech encoder. the speech normalizer can be trained with one hour of parallel speech data without the need of any human annotations and works for speech in different recording conditions and in different languages. we conduct experiments on the voxpopuli s2st dataset and the mined speech data to empirically demonstrate its usefulness in improving s2st system translation quality for the first time. in the future, we plan to investigate more textless approaches to improve model performance such as self-supervised pretraining. all the experiments and asr evaluation are conducted with public datasets or open-sourced models.