S peech M atrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2023)

Abstract: We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models will be publicly released

## introduction research has progressed in the area of speech-tospeech translation with the goal of seamless communication among people who speak different languages. direct s2st models attract increasing research interest, e.g. @xcite . compared to conventional cascaded models, direct models do not rely on intermediate text representations which make them applicable to the translation of languages without a well-defined writing script. moreover, direct s2st have the advantage of higher training and inference efficiency @xcite . despite the benefits of direct approaches, their training is faced with the major issue of data scarcity in parallel speech. human labeled speech data is expensive to create, there are very few data resources providing speech alignments, and the data amount is quite limited. to mitigate the data scarcity, some works have leveraged multitask learning @xcite , data augmentation with speech variations @xcite , or with synthesized speech @xcite @xcite . it is also shown useful to leverage knowledge transferred from pre-trained models @xcite such as hubert @xcite , wav2vec 2.0 @xcite and mbart @xcite . recently, @xcite is the first work to make speech mining efforts by learning a shared multilingual speech and text embedding space. speech content is encoded by speech encoders into fixed-size representations which are then used for aligning speech and text across different languages. it demonstrates good empirical gains to train direct speech-to-text and speech-tospeech translation systems with the mined data @xcite . in this work, we trained speech encoders for 17 languages foot_0 and mined speech-to-speech alignments for all possible language pairs from voxpopuli @xcite , a collection of european parliament recordings. to the best of our knowledge, speechmatrix is by far the largest freely available speech-to-speech translation corpus, with 136 language directions and an average of 1,537 hours of source speech in each direction for a total of 418 thousand hours. we demonstrate that strong s2st models can be trained with these mined data and validate the good quality of the speech alignments across languages. we are open-sourcing the mined data and the speech encoders used for mining, which could pave the way for future research on s2st. moreover, for reproducibility, we will release model components including multilingual hubert models in four language families for target unit generation, language-specific vocoders for speech synthesis from discrete units, and s2s models trained and presented in this work. speech-to-speech translation (s2st). s2st started from cascaded systems consisting of automatic speech recognition (asr), machine translation (mt) and text-to-speech synthesis (tts) @xcite . the reliance on intermediate text outputs poses limitations on cascaded models to support efficient inference and unwritten languages. given these challenges, there has been a recent surge of research interest in direct approaches to speech translation without the need of texts. translatotron @xcite and translatotron2 @xcite train end-to-end s2st to generate target spectrograms with multitask learning. another line of research replaces the target spectrograms in s2st modeling with discrete units which are learned from a large amount of unlabeled speech @xcite . discrete units have shown to better capture linguistic content than spectrograms. despite these progress on direct s2st, it is faced with the challenge of limited parallel speech. ## speech-to-speech mining the mining approach of this work is built upon the idea of encoding multilingual speech utterances into a shared embedding space. speech encoders project utterances with similar semantic content to fixed-size representations which are close in the embedding space regardless of their languages. the closeness of embeddings reflects the similarity of speech content, and is used as the alignment score in the mining process. in this section, we discuss speech encoders and speech mining. ## experiments & results to evaluate the quality of the mined data, we trained s2st models on speechmatrix data and report the translation performance. we hope that these results will serve as baselines for future studies in speech-to-speech translation. ## bilingual speech-to-speech baselines in this part, we discuss the bilingual s2s models trained in each of 272 language directions in speechmatrix. the architecture of textless model is used for bilingual translation in our experiments @xcite . a textless model consists of a speech encoder, transformer encoder and decoder. training. for a given direction, we extract units for source and target speech with their corresponding hubert models @xcite . taking source speech, the model is trained to predict target unit sequence with cross-entropy loss as well as source unit reconstruction as an auxiliary task. for the training efficiency of extensive s2st experiments, we use a subset of mined data as the train set. mined samples are selected if their alignment scores are above a preset threshold. we performed an analysis of the threshold selection in subsection b.4. comparison with existing results. since we adopt the same model as the previous work @xcite and the only difference lies in the train set, it is straightforward to compare with existing results. ## multilingual speech-to-speech translation multilingual modeling has been explored in tasks of language understanding and machine translation, demonstrating knowledge transfer among languages. however, to our best knowledge, there are few studies of multilingual s2st on real speech, partially due to the lack of multilingual speech-tospeech resources. with the massively multilingual data we have mined, we are able to explore multilingual s2st training. in this work, we focus on many-to-english translation, studying the translation from 6 slavic languages to english in subsection 6.1 and the translation from all 16 languages in speechmatrix to english in subsection 6.2. english-to-many or manyto-many translation are left to future work. we present here multilingual models used in our experiments (more details can be found in appendix c: gshard. gshard is a sparse scaling technique proposed in @xcite . we replace every other transformer layer with an moe layer. ffn modules in an moe transformer layer are shared across experts. a learnable gating function routes input tokens to different experts @xcite . we apply gshard architecture on the decoder of xm transformer, and expert weights are all initialized with the pretrained unit mbart. ## limitations and risks limitations. the hubert model quality is critical to speech-to-speech translation performance, as its extracted units are used by both speech-tounit model and vocoder. we have not explored the optimal strategy of multilingual hubert training. one research question is how to choose a group of languages so that a multilingual hubert model could be well trained. for example, it is arguable whether lithuanian (lt) should be included in slavic or uralic family. other questions could be whether a larger hubert with more model capacity should be used and how we should deal with language imbalance in multilingual training. we provide benchmark results of bilingual speech translation with mined data selected by heuristics. one of our future directions is to come up with a better strategy of mined data selection to improve translation performance and training efficiency. as mentioned in our results analysis, the reported bleu scores are heavily dependent on the asr quality, which may not reflect the speech translation performance accurately. future directions could be improving asr quality or exploring other evaluation metrics without reliance on asr models. potential risks. as a technology used for speech generation, the presented speech translation models or the translation models that will be trained with speechmatrix dataset might have systemic bias or produce inappropriate outputs. ## conclusion in this paper, we introduce a large-scale multilingual speech-to-speech corpus mined from voxpopuli. it is the largest resource of speech alignments with a coverage of 17 languages. we perform an extensive evaluation of the mined parallel speech, showing good quality of the speech alignments. multilingual speech-to-speech models can be efficiently trained on this corpus and we suggest different methods, such as sparse scaling using mixture-of-experts, to further boost translation performance in the multilingual setting.