V ox P opuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021)

Abstract: We introduce VoxPopuli, a large-scale multilingual corpus providing 400K hours of unlabeled speech data in 23 languages. It is the largest open data to date for unsupervised representation learning as well as semi-supervised learning. VoxPopuli also contains 1.8K hours of transcribed speeches in 15 languages and their aligned oral interpretations into 15 target languages totaling 17.3K hours. We provide speech recognition (ASR) baselines and validate the versatility of VoxPopuli unlabeled data in semi-supervised ASR and speech-to-text translation under challenging out-of-domain settings. The corpus is available at https://github.com/facebookresearch/voxpopuli .

## introduction recent progress in speech-to-text tasks such as automatic speech recognition (asr) and speech translation (st) has been achieved by the development and application of unsupervised speech pre-training methods @xcite @xcite @xcite , with semi-supervised learning (self-training) @xcite @xcite or a combination of both methods @xcite . this line of research leverages large amounts of unlabeled english speech data @xcite ) that enable improvements in english asr or out-of-english st. large amounts of multilingual audio data are needed in order to achieve similar progress for multilingual asr and st. similarly, most asr and st research is currently conducted on the librispeech @xcite and must-c benchmarks @xcite . as a result, the equal contribution. simultaneous speech translation (interpretation) has witnessed a resurgence with the applications of end-to-end encoder-decoder models. most of the recent studies focus on text output and leverage st corpora that are translated offline in the written form. there are differences, however, between translationese and interpretese @xcite , where interpreters develop a variety of strategies to improve simultaneity. models trained on translation corpora are unlikely to learn from these interpretation skills to achieve better quality-latency trade-offs. finally, there has been little research @xcite @xcite into speech output due to the lack of open data. existing corpora @xcite are either of limited size or no longer publicly available. in this paper, we introduce voxpopuli, a largescale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. it contains the largest open unlabeled speech data to date, totaling 400k hours in 23 languages: bulgarian (bg), czech (cs), croatian (hr), danish (da), dutch (nl), english (en), estonian (et), finnish (fi), french (fr), german (de), greek (el), hungarian (hu), italian (it), latvian (lv), lithuanian (lt), maltese (mt), polish (pl), portuguese (pt), romanian (ro), slovak (sk), slovene (sl), spanish (es) and swedish @xcite . voxpopuli also provides a total of 1.8k hours of transcribed speech in 16 languages @xcite and their aligned oral interpretations into 15 target languages (en, de, fr, es, pl, it, ro, hu, cs, nl, fi, sk, sl, lt and da) totaling 17.3k hours. we describe our corpus creation methodology in section 2 and analyze the created corpus in section 3. we provide asr baselines and demonstrate the value of our multilingual unlabeled data as well as weakly labeled data on several non-english languages in section 4. ## data analysis unlabeled speech as we can see from transcribed speech the voxpopuli transcribed data contains 16 languages totaling 1.8k hours and 4.3k speakers, whose detailed statistics can be found in ## experiments & results we provide voxpopuli asr baselines and validate the versatility of voxpopuli unlabeled data in unsupervised representation learning and semisupervised learning for asr as well as st. we also evaluate the quality of speech-to-speech alignment indirectly via the weakly labeled st data it produces. ## related work multilingual speech corpora librilight @xcite currently represents the largest scale unlabeled speech corpus but it is limited to english. mls @xcite is a recently released large-scale multilingual corpus of read speech in 8 languages, derived from librivox. mailabs 1 is also derived from librivox and has about 1000 hours available in 9 languages. while mls and mailabs are derived from audiobooks, vox-forge 1 and common voice @xcite gather data via crowd-sourcing. voxforge collected data in about 15 different languages with about 300 hours of speech in total; common voice currently supports 60 languages for a total of 7327 validated hours available. the cmu wilderness dataset @xcite collects readings from the new testament, with 700 different languages avail-1001 able. iarpa babel program foot_1 collected data for 24 languages, mostly from conversational telephone speech. the dataset is however not released and under an open license, and focused on low-resource languages, with labeled data ranging between 25 to 65 hours per language. speech-to-text and speech-to-speech translation apart from machine translation @xcite , the european parliament open data has fostered the development of corpora for speech-totext translation and for simultaneous interpretation. europarl-st (iranzo-sánchez et al., 2020) is a multilingual speech-to-text translation corpus with translations between 6 european languages (en, fr, de, es, it and pt). similarly, epic @xcite is derived from the european parliament with simultaneous interpretation speeches in italian, english and spanish. ciair @xcite and stc @xcite are simultaneous interpretation corpora between english and japanese with a total of about 180 hours for the former, while the latter is currently unavailable for download. the mass dataset @xcite also provides speech to speech alignments for about 8k utterances across 8 languages, for a total of about 23h of speech. ## conclusion in this paper, we introduce a large-scale multilingual speech corpus, voxpopuli, for representation learning, semi-supervised learning and interpretation. voxpopuli provides the largest open unlabeled speech data to date, which has broad applications including unsupervised pre-training and selftraining. voxpopuli is also the first corpus for large amounts of open speech-to-speech interpretation data. we provide voxpopuli asr baselines and validate the versatility of voxpopuli unlabeled data in semi-supervised learning under challenging out-ofdomain settings. the corpus is available at https: //github.com/facebookresearch/voxpopuli.