Textless Speech-to-Speech Translation With Limited Parallel Data (2024)

Abstract: Existing speech-to-speech translation (S2ST) models fall into two camps: they either leverage text as an intermediate step or require hundreds of hours of parallel speech data. Both approaches are incompatible with textless languages or language pairs with limited parallel data. We present PFB, a framework for training textless S2ST models that require just dozens of hours of parallel speech data. We first pretrain a model on large-scale monolingual speech data, finetune it with a small amount of parallel speech data (20-60 hours), and lastly train with an unsupervised backtranslation objective. We train and evaluate our models for English-to-German, German-to-English and Marathi-to-English translation on three different domains (European Parliament, Common Voice, and All India Radio) with single-speaker synthesized speech. Evaluated using the ASR-BLEU metric, our models achieve reasonable performance on all three domains, with some being within 1-2 points of our higher-resourced topline.

## introduction speech-to-speech translation (s2st) systems map input speech in the source language to output speech in the target language. in many ways, s2st represents the holy grail of translation as it enables natural, real-time, spoken communication. s2st has a rich history, from cascaded systems combining automatic speech recognition (asr), machine translation (mt), and text to speech (tts) technologies @xcite to recently proposed neural end-to-end systems @xcite ) that directly map from input source language speech to output target language speech. these systems @xcite @xcite @xcite have benefited from model and data scaling, leveraging increasing amounts of parallel speech and/or text data across languages. yet, this is feasible only for a fraction of the world's 7000 languages @xcite ; the majority of world languages have low-resource or no parallel translation data available @xcite . furthermore, thousands of languages are primarily spoken without standardized writing systems (about 3000 languages in ethnologue @xcite have no reported writing system), necessitating textless language processing. recent work on textless speech translation @xcite train end-to-end models on large amounts of parallel speech data, which is expensive to collect and makes these approaches difficult to adapt for low-resource speech translation. sentence embedding-based modular approaches @xcite do not require any parallel speech data, but still require parallel text data to learn cross-lingual sentence embedding spaces. on the other hand, unsupervised s2st approaches @xcite @xcite do not need any parallel speech or text data at all, instead relying on unsupervised cross-lingual learning using large amounts of monolingual speech and text datasets. however, they either train cascaded models that have intermediate text outputs or end-to-end models that use text supervision during training. as a result, they are difficult to adapt for textless languages that are spoken, have non-standard orthographies or poor asr systems. in this work, we adapt the unsupervised s2st pipeline to work in a fully textless manner for the first time. we formulate textless s2st as a unit-to-unit machine translation problem that requires a modest amount (dozens of hours) of parallel speech training data. we begin by pretraining an encoder-decoder unit language model over selfsupervised speech units using monolingual speech data, followed by finetuning it for s2st on a lowresource parallel dataset and finally performing unsupervised backtranslation to further improve per- textless low-resource s2st (ours) speech-to-unit encoder (hubert) mel-spectrogram feature extractor unsupervised @xmath0... <@xmath1... speech vocoder unit vocoder (hifigan) unsupervised tts parallel training data thousands of hours target speech none 😵💫 🥳 20-60 hours 🥳 is it textless? ✅ ❌ ✅ unit vocoder (hifigan) figure 1 : overview of speech-to-speech translation systems. we compare our formulation to two relevant lines of work. we present the first textless speech-to-speech system that does not require a large parallel training dataset. ## methods unsupervised s2st @xcite tackles the problem of text-based lowresource s2st by representing input and output speech as text sequences and training a cascade of models consisting of unsupervised speech recognition @xcite (uasr), unsupervised machine translation @xcite and unsupervised text-to-speech @xcite (utts). to adapt this for textless languages, we represent the input and output speech utterances as self-supervised discrete unit sequences rather than text sequences. instead of uasr, we use a speechto-unit encoder (s2u) and instead of utts, we use a unit-to-speech vocoder (u2s), both largely based on prior work @xcite . instead of text-based umt, we train a unit encoderdecoder (u2u) s2st model using our three-step pretrain-finetune-backtranslate (pfb) approach illustrated in figure 2 adapted from the unsupervised mt literature @xcite . we now describe each of these components below. ## conclusion we present the first textless low-resource speechto-speech translation system, capable of learning from dozens of hours of parallel translation data, built by pretraining, finetuning, and backtranslating a language model over self-supervised speech unit sequences rather than text. we demonstrate its efficacy on 2 language pairs (english-german and english-marathi) across 3 different domains. while our models achieve a decent translation performance, close to supervised toplines in some cases, they still underperform models trained on far more data or models that make use of text data, implying that several challenges still remain to make these models highly performant. however, our approach holds great promise for modelling lowresource, primarily spoken languages. we hypothesize, based on similar findings for text machine translation, that scaling our approach to a larger unit lm pretrained on more data will improve performance and may unlock unsupervised textless s2st akin to unsupervised text mt @xcite . future work can investigate use of better s2u unit encoders for training better unit lms, and training unit lms on a larger set of languages.