Towards Robust Speech Representation Learning for Thousands of Languages (2024)

Abstract: Self-supervised learning (SSL) has helped extend speech technologies to more languages by reducing the need for labeled data. However, models are still far from supporting the world’s 7000+ languages. We propose XEUS, a Cross-lingual Encoder for Universal Speech, trained on over 1 million hours of data across 4057 languages, extending the language coverage of SSL models 4-fold. We combine 1 million hours of speech from existing publicly accessible corpora with a newly created corpus of 7400+ hours from 4057 languages, which will be publicly released. To handle the diverse conditions of multilingual speech data, we augment the typical SSL masked prediction approach with a novel dereverberation objective, increasing robustness. We evaluate XEUS on several benchmarks, and show that it consistently outperforms or achieves comparable results to state-of-the-art (SOTA) SSL models across a variety of tasks. XEUS sets a new SOTA on the ML-SUPERB benchmark: it outperforms MMS 1B and w2v-BERT 2.0 v2 by 0.8% and 4.4% respectively, despite having less parameters or pre-training data. Checkpoints, code, and data are found in https://www.wavlab.org/activities/2024/xeus/.

## introduction our planet is home to over 7000 languages @xcite ), yet most speech processing models are only capable of serving at most 100-150 of them @xcite . the biggest constraint in supporting more of languages is the lack of transcribed speech: only around half of the world's languages have a formal writing system @xcite , and even fewer of them have the resources to support the scale of annotated data required for training neural models. a common approach to address this limitation is self-supervised learning (ssl) on large amounts of unlabeled multilingual speech @xcite @xcite , which allows for strong downstream performance even when access to annotated data is limited. while ssl models have more relaxed data requirements relative to supervised models, few works have fully exploited this aspect to scale models to more languages. in fact, most multilingual ssl models remain in the 50-150 language range of coverage @xcite @xcite , reducing the benefits of this advantage. the mms project @xcite sought to address this issue by directly crawling data for more languages, scaling ssl pre-training to 1,000+ languages. the authors collected speech across 1,406 languages and used it to train an ssl model, showing state-of-the-art (sota) results after fine-tuning on multilingual automatic speech recognition (asr) and 3,900-way language identification (lid). while the mms models were publicly released, the crawled data was not, preventing it from being used in future work and thus the models from being reproduced ( an additional issue that is relatively unexplored in ssl research is robustness to noisy data. this aspect is important for multilingual models, since the available recordings of low-resource languages tend to be particularly noisy @xcite . the issue is exacerbated by the fact that existing multilingual ssl corpora lack diversity not only in languages but also in speaking style and recording conditions. wavlm @xcite and wavlablm @xcite tackled this issue by simulating noisy conditions during training, overlapping utterances or adding acoustic noise to simulate multi-speaker and noisy environments respectively. while effective, we believe that this technique can be improved to cover an even wider variety of recording conditions. our goal is to thus build a universal speech encoder that can handle both linguistically and acoustically diverse speech. to achieve this, we propose xeus (pronounced zeus) -a crosslingual encoder for universal speech. xeus is an e-branchformer @xcite encoder pretrained on over 1 million hours of publicly available data across a wide variety of recording conditions. we first curate the data from 37 existing corpora to ensure a diverse selection of speech and recording conditions not often found in standard asr datasets, including but not limited to spontaneous speech, accented speech, code-switching, indigenous languages, and singing voices. we expand the language coverage of xeus by introducing a new ssl corpus that uses data sources previously unseen in speech literature. this corpus, which will be publicly released, contains 7,413 hours of unlabeled audio across 4,057 languages, the widest coverage of any speech processing dataset. to enhance the model's robustness, xeus is also pre-trained with a novel ssl objective of acoustic dereverberation, which requires the model to predict clean discrete phonetic pseudo-labels from simulated reverberant audio. by combining this objective with hubert-style masked prediction @xcite and wavlm-style denoising @xcite , xeus is designed to be the next step towards a truly universal speech encoder for any language or recording condition. in our downstream evaluations, we find that xeus consistently improves over sota ssl models across a wide variety of tasks. xeus sets a new sota on the ml-superb multilingual asr benchmark, outperforming ssl models such as mms @xcite and w2v-bert 2.0 v2 @xcite while having fewer parameters or less training data. our speech translation (st) results show the effectiveness of xeus' wide language coverage, even for languages with less than 10 hours of data in the pre-training corpus. we also explore xeus' potential in generative tasks and show its superiority on speech resynthesis when compared to other ssl encoders. finally, we evaluate xeus' representations on a variety of tasks through the english-only superb benchmark, where it sets a new sota on 4 tasks despite xeus' focus on multilingual performance. to conduct ssl pre-training at such scale, we had to make significant optimizations to existing speech processing toolkits. to encourage further ssl research and reproducibility, we will publicly release this code, along with the training configurations and checkpoints for xeus. we also release all 200+ intermediate checkpoints and training logs obtained throughout the pre-training for further research in the training dynamics of large-scale multilingual ssl models. to summarize, our main contributions are as follows: ## downstream evaluation we examine the capabilities of xeus in various downstream applications. section 5.1 evaluates the multilingual capabilities of xeus in different settings. section 5.2 evaluates the universality of xeus' representations for a broad range of speech information, such as emotion and speaker content. finally, section 5.3 tests the acoustic representations of xeus via speech resynthesis. we provide an overview of each downstream setup in its respective subsection. detailed experimental settings can be found in appendix section a.6. 1 ). we use the ml-superb benchmark @xcite as our main evaluation, as it tests each models across a diverse range of tasks and languages. we complement these experiments with additional analyses on fleurs asr+lid and low-resource st. ## conclusion this work presents xeus, an ssl speech encoder trained on over 1 million hours of data across 4,057 languages. as a community contribution, we release a new dataset with 7,413 hours of unlabeled speech data across those 4,057 languages. we also introduce a novel joint dereverberation task for ssl pre-training, which we use to increase the robustness of xeus. we show that xeus can achieve comparable performance if not outperform other sota ssl models on various benchmarks, while having much stronger performance on long-tail languages. to make xeus reproducible, we will release all training code and configurations, along with model weights. in the future, we hope to extend the downstream use of xeus to a larger scale.