Simple and Effective Unsupervised Speech Translation (2023)

Abstract: The amount of labeled data to train models for speech tasks is limited for most languages, however, the data scarcity is exacerbated for speech translation which requires labeled data covering two different languages. To address this issue, we study a simple and effective approach to build speech translation systems without labeled data by leveraging recent advances in unsupervised speech recognition, machine translation and speech synthesis, either in a pipeline approach, or to generate pseudo-labels for training end-to-end speech translation models. Furthermore, we present an unsupervised domain adaptation technique for pre-trained speech models which improves the performance of downstream unsupervised speech recognition, especially for low-resource settings. Experiments show that unsupervised speech-to-text translation outperforms the previous unsupervised state of the art by 3.2 BLEU on the Libri-Trans benchmark, on CoVoST 2, our best systems outperform the best supervised end-to-end models (without pre-training) from only two years ago by an average of 5.0 BLEU over five X-En directions. We also report competitive results on MuST-C and CVSS benchmarks.

## introduction training supervised speech systems requires large amounts of labeled data which is often not available for all but a small fraction of the over 7,000 languages spoken around the world @xcite . despite much recent effort in creating speech translation corpora @xcite , only a few dozen language directions are covered. the lack of labeled training data is even more acute for speech translation because it requires aligned labeled data in two languages which increases the effort to create such datasets. this poses the question of whether speech translation systems can be built using less labeled data or no labeled data at all. recent work on unsupervised speech recognition has achieved performance that can enable useful systems using no labeled data @xcite @xcite @xcite , enabled in large part by the advances in self-supervised speech representation learning @xcite . these techniques were also used to build unsupervised text-to-speech systems @xcite . similarly, unsupervised text-to-text machine translation has shown great promise for certain language directions @xcite @xcite . in this paper, we study a method to build endto-end unsupervised speech-to-text and speech-tospeech translation systems trained on synthetic training data obtained by cascading existing unsupervised techniques: we first transcribe speech utterances in the source language using unsupervised speech recognition @xcite , then translate the resulting transcription using unsupervised machine translation @xcite @xcite , and finally synthesize the translation into a target language speech utterance using unsupervised speech synthesis @xcite . we also consider applying the pipeline directly at inference time. our approach benefits from the use of self-supervised speech models @xcite and to further improve performance, we present a technique to adapt existing self-supervised models to the target domain. step 1: unsupervised adaptation of speech pre-trained model out-of-domain/lang wav2vec 2.0 in-domain src. speech adapted w2v 2.0 training data inference model step 2: unsupervised cascaded pseudo-labeling out-of-domain tgt. speech speech pseudolabels text pseudolabels unsup. asr text de-norm unsup. tts unsup. mt multi-domain src. text multi-domain tgt. text input speech unsup. tgt. asr in-domain src. speech adapted w2v 2.0 step 3: end-to-end model training with pseudo-labels speech pseudolabels text pseudolabels in-domain src. speech end-to-end speech-to-text translation end-to-end speech-to-speech translation input speech translated text translated speech vised systems trained on large amounts of labeled data. wav2vec-u 2.0 @xcite simplifies audio-side pre-processing and improves accuracy through better architecture as well as better training objective. @xcite shows that out-ofdomain speech pre-training or out-of-domain text data hurts the training robustness of wav2vec-u models, especially under low-resource settings. ## methods figure 1 provides an overview of our proposed approach to unsupervised speech-to-text translation (s2tt) and speech-to-speech translation (s2st). we leverage a cascade of unsupervised models to produce pseudo-labels for end-to-end s2tt and s2st model training. to mitigate language and domain mismatch in speech pre-training (wav2vec 2.0), we finetune wav2vec 2.0 models using unlabeled in-domain speech data, and then use the adapted models to build downstream speech recognition models. ## experimental setup we evaluate our translation models on 5 directions into english (fr-en, es-en, ru-en, et-en and lv-en) and 3 directions out of english (en-es, en-ru and en-fr). the 5 non-english languages are from 4 different indo-european language family subgroups: romance (fr and es), slavic (ru), uralic (et) and baltic (lv). for the x-en directions, we evaluate s2tt models on covost 2 (wang et al., 2021b) and evaluate s2st models on cvss-c (jia et al., 2022b), which adds synthetic target speech to covost 2 with a single canonical speaker voice. for the en-x directions, we only evaluate s2tt models. we use must-c (di gangi et al., 2019a) for en-es and en-ru, as well as libri-trans (kocabiyikoglu et al., 2018) for en-fr. for libri-trans, we follow chung et al. (2019) to combine validation set and test set for evaluation. speech pre-training. we use robust wav2vec 2.0 @xcite for english speech, which is trained on datasets from multiple domains. for non-english speech, we adapt open-source vox-populi foot_0 @xcite ) models by ctc finetuning with 1024 discrete labels (fr, es and ru) or 128 discrete labels (et and lv). we use monolingual voxpopuli models for fr and es, and multilingual models of similar languages for ru, et and lv (slavic, uralic and baltic languages, respectively). we extract speech representations from the 15-th layer of the original wav2vec 2.0 models for computing discrete labels. speech recognition. for wav2vec-u 2.0 models, we extract speech representations from the 19-th (15-th) layer of the adapted (original) wav2vec 2.0 models. we increase the dropout on the batch normalized input features to 0.2. we set σ = 0.1 for input gaussian noise and α = 1.0 for r-drop regularization. for wav2vec-u 2.0 loss weights, we set η = 3 and choose λ, γ and δ from 1.0 / 1.5, 1.5 / 2.5 and 0.3 / 0.5, respectively. for text data, we use open web crawled corpus, cc-100 @xcite , which is created with little curation and has large language coverage. for supervised baselines, we fine-tune adapted wav2vec 2.0 models with ctc objective on labeled data, which we denote as "w2v2-ctc". machine translation. we use cc-100 @xcite to train bilingual mbart large models for each language pair. for bidirectional online back-translation, we use the same cc100 data and follow @xcite to apply 99% vocabulary masking for the first 500 updates. for supervised baselines, we fine-tune mbart models with labeled data, which we denote as "mbart-ft". ## conclusion in this paper, we present a simple and effective approach to unsupervised speech-to-text translation (s2tt) and speech-to-speech translation (s2st). our s2tt systems outperform the previous state of the art on libri-trans by 3.2 bleu as well as the best supervised end-to-end models (without pretraining) on covost 2 from only two years ago by an average of 5.0 bleu over five translation directions into english. our s2tt and s2st systems also perform competitively on the must-c and cvss-c benchmarks. our unsupervised speech recognition approach requires tools to phonemize text for the language of interest. phonemizers are not available for all languages and this presents a bottleneck. to address this, future work may develop phonemizers for more languages, explore phonemization approaches that generalize across languages, or wav2vec-u 2.0 model training with graphemic text units such as letters. we train bilingual unsupervised machine translation models with 2.1b english sentences and at least 46m sentences for the non-english language. for extremely low-resource languages, collecting millions of sentences for model training can be challenging. the feasibility of mbart-based online back-translation approach in this setup remains to be validated.