MOSEL : 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages (2024)

Abstract: The rise of foundation models (FMs), coupled with regulatory efforts addressing their risks and impacts, has sparked significant interest in open-source models. However, existing speech FMs (SFMs) fall short of full compliance with the open-source principles, even if claimed otherwise, as no existing SFM has model weights, code, and training data publicly available under open-source terms. In this work, we take the first step toward filling this gap by focusing on the 24 official languages of the European Union (EU). We collect suitable training data by surveying automatic speech recognition datasets and unlabeled speech corpora under open-source compliant licenses, for a total of 950k hours. Additionally, we release automatic transcripts for 441k hours of unlabeled data under the permissive CC-BY license, thereby facilitating the creation of open-source SFMs for the EU languages.

## introduction the introduction of foundation models trained on large datasets is revolutionizing the landscape of many nlp fields @xcite , particularly with the release of large language models (llms) that demonstrated impressive abilities on various tasks @xcite . the interest attracted by such models has come together with concerns about their risks and impact, as well as requests for a better understanding of their inner workings. on the one hand, this has led to regulatory efforts @xcite , while on the other hand, it has sparked a growing interest in open-source models @xcite that can equal contribution. in line with the open source definition and its principles, foot_0 the open source initiative defines as open source ai a "system made available under terms that grant the freedoms to: use the system for any purpose without having to ask for permission", "study", "modify [...] for any purpose", and "share [...] with or without modifications, for any purpose". 2 specifically, it requires that the model and the code "used to train and run the system" are available under an os license, 3 and that the training data is available under an os-compliant license @xcite . this means that an os model should not be trained on data released under licenses that restrict any of the four essential rights -use, study, modify, and share -for any purpose, including commercial use. examples of os-compliant licenses include cdla-permissive-2.0 and cc-by-4.0, which only requires attribution (i.e., acknowledging the source or resource used). instead, data released under licenses like cc-nc-4.0, which prohibits commercial use, or cc-sa-4.0, which mandates that derivative works have to be distributed under the same terms (thereby limiting the freedom to modify and share for any purpose), are not os compliant. focusing on speech foundation models (sfms), none of the existing ones complies with this definition. for instance, seamlessm4t's model @xcite is released under a license that is not os compliant, while whisper's model and inference code @xcite are, but the training code and data are not public. lastly, owsm @xcite , although fulfilling most of the requirements, has been trained using datasets such as @xcite and spgispeech (o'neill et al., 2021), which have licenses that do not permit derivative works or commercial use. as a consequence, to the best of our knowledge, no current sfm satisfies the open source ai definition and can hence claim to be an open source sfm (ossfm). considering the 24 official languages of the european union (eu), foot_3 we take the first step towards filling this gap and, in particular, toward the creation of an eu-ossfm: the collection of oscompliant training data. with this aim, we survey the automatic speech recognition (asr) datasets and the unlabeled speech corpora available for eu languages and list those that can be used to train an eu-ossfm, for a total of 950k hours. this inventory of os-compliant data, which will be continuously updated, is called mosel (massive opensource compliant speech data for the european languages) and is publicly available as a github repository at: ## open source compliant speech data this section surveys the available corpora that are admissible for developing an ossfm for all 24 official eu languages. accordingly, we include datasets that are freely accessible (i.e., excluding paid datasets) and whose data is released under an os-compliant license (i.e., without restrictions on creating and redistributing derivative artifacts, including ai models). 5 this means that, in the case of the widespread creative commons (cc) licenses, we cannot include data released with nonderivative (nd), non-commercial (nc), or share-alike (sa) 6 restrictions. we also exclude datasets whose license is os compliant but containing data released under a non-os-compliant license. in fact, cc licenses "allow licensed material to be included in collections [...], however this does not change the license applicable to the original material". 7 in line with this indication, in cases where the transcripts are os compliant (e.g., cc-by where only attribution is required) but the corresponding speech (or part of it) is not, we document the dataset under the most restrictive license. for instance, gigaspeech @xcite , which is released under apache 2.0, 8 is categorized as non-os compliant since it contains youtube videos under restrictive cc licenses. 9 similarly, mass (zanon boito et al., 2020) and cmu wilderness @xcite are regarded as non-os compliant since they are derived from the bible.is data of the faith comes by hearing organization with nc and nd terms of use. 10 the entire youtube videos). therefore, further checks and processing work would be needed to effectively exploit the dataset for ossfm training. the total speech content (both labeled and unlabeled) amounts to 950,2k hours, which significantly exceeds the total data used to train most of the current sfms (e.g., 680k hours for whisper v2, 180k for owsm), with the only exception of whisper v3 whose training data comprises 5 million of hours. even excluding the 446k hours of youtube-commons, the amount of data remains comparable, especially since whisper v2 and owsm target 99 and 151 languages respectively, instead of the 24 required for an eu-ossfm. looking at language coverage, in light of the high computational resources demanded by this process for large-scale sfm training data, avoiding duplicated efforts across different institutions can significantly reduce the overall environmental impact and costs @xcite , in line with green ai principles @xcite . for this reason, we complement our inventory by providing practitioners with automatic transcripts for 441k hours of unlabeled speech coming from voxpopuli and librilight. 14 the resulting pseudo-labeled data, whose statistics per language are presented in §a.2, covers nearly half of the total data available for training an eu-ossfm and 23 of the 24 eu languages. in line with the spirit of this work, the transcripts are released under the os-compliant cc-by license at hf.co/datasets/fbk-mt/mosel. the data is transcribed using whisper large v3 15 , which is released under the os apache 2.0 license that allows the generated content to be released under any license. in appendix d, we report the asr quality of whisper across the eu languages. the inference is realized by feeding whisper with the corresponding language id and 14 youtube-commons was excluded due to the issues described in §2. ## proof of concept on maltese to showcase that the datasets collected in our survey ( §2) and the generated transcripts ( §3) constitute suitable training data for an eu-ossfm, we conduct a proof-of-concept experiment on maltese. maltese was chosen because it is i) one of the lowest-resourced languages, and ii) the one for which whisper achieves the worst results, as shown in appendix d. 18 for our experiments, we first attempted to train an asr model using only supervised data, but it failed to converge due to its limited size (16 hours). therefore, we trained a model using the few labeled data together with the pseudo-labeled data. 19 as an additional investigation, we also applied to the pseudo-labeled data simple filtering methods to remove audios containing other languages and automatic transcripts containing hallucinations (see appendix b.2). results presented in ## conclusions in response to the urgent need for truly open-source foundation models, this work takes the first step toward an eu open-source speech foundation model, which is the collection of suitable training data called mosel . to this end, we first surveyed the labeled and unlabeled speech datasets for automatic speech recognition that feature at least one of the 24 official eu languages and are available under a license compliant with the open-source terms. we then complemented this effort with the creation and release of automatic transcripts for the available unlabeled data. overall, we collected more than 950k hours of speech content suitable for the training of an eu open-source speech foundation model, also demonstrating its usefulness in maltese, one of the lowest-resourced languages.