Speech Data from Radio Broadcasts for Low Resource Languages (2024)

Abstract: We created a collection of speech data for 48 low resource languages. The corpus is extracted from radio broadcasts and processed with novel speech detection and language identification models based on a manually vetted subset of the audio for 10 languages. The data is made publicly available.

## introduction while automatic speech recognition systems have seen great gains in recognition accuracy, even under challenging acoustic conditions, this success is highly uneven across the languages in the world. for many languages in the world, even reliable audio training data is not easily available. motivated by this, we set out to collect and make publicly available speech data for languages that fall below the top one hundred languages, broadly measured by number of speakers and commercial relevance. we present a novel audio data set for 48 low resource languages. we report on manual efforts to vet collected audio data as well as automatic methods to extract speech from mixed audio data (especially discarding music) and language identification. we collected this data mostly from radio broadcasts by recording audio streams available at radio garden foot_1 . these audio broadcasts are identified by location which gives us some guidance to which broadcasts are likely to contain audio in a desired language. we record audio snippets of 10-60 seconds in length. since much of the audio data contains music, we developed a speech detection model to automatically identify audio files that consist of speech data and not music or other non-speech data. since there are no reliable speech language identification models or even identified speech data for a subset of these languages, we manually vetted audio data for 10 languages to create a corpus of about 5 hours of audio per language that has been verified by native speakers to be speech in each of the targeted languages. with these tools in place (speech crawling, speech detection, speech language identification), we scaled up the effort to 48 languages. the resulting corpus of speech data consists of about 3000 hours of clean raw speech suspected to be in these low-resourced languages. upon further filtering with language identification (lid) systems, this results in about 450 hours of clean speech. 2023 ) introduce a massively multilingual dataset for over 1000+ languages based on recordings of publicly available religious texts. they further train self-supervised, automatic speech recognition, text-to-speech synthesis, and language identification models on this dataset. @xcite introduce a largescale multilingual weakly supervised dataset consisting of about 680k hours of audio for speech recognition. they showed that scaling the amount of data greatly improves the performance and robustness of speech recognition systems. ## related work unlabeled speech data has many uses in building speech applications. representation learning methods like hubert @xcite and w2v-bert @xcite use raw speech data to distill semantic speech tokens from audio. large-scale models such as whisper @xcite , mms @xcite , or seamless @xcite rely partly on raw speech data to scale to hundreds of languages. ## corpus collection the sources of our data are radio broadcasts that are transmitted freely over the internet. we use radio garden to discover and identify stations that broadcast in languages we target. radio garden identifies radio stations with the location from which they broadcast -which provides a pool of candidate stations for each language, based on the region where the language is spoken. the broadcasts are accessible through an api call. we filter this pool of candidate stations by checking manually if they likely broadcast in the targeted language (opposed to, say, english) or exclusively broadcast music. since this effort often relies on researchers that are not familiar with the languages, the process is necessarily imperfect. another obstacle is that some radio station broadcasts are not reliably delivered over the radio garden platform, leading to gaps in the data collection. we break up the audio signal into segments of different lengths, ranging from 10 to 60 seconds. the raw audio is also converted to the flac files and re-sampled to 16khz. we collected this data throughout 2023 and early 2024. ## speech detection to filter out audio files containing music, we use a convolutional recurrent neural network (crnn) @xcite which was trained on a highquality dataset @xcite of speech and music activity labels. the crnn model predicts the probability of music and speech for each audio frame. we also use a feature-based model that calculates the average energy in each chunk of the audio spectrogram. this energy level indicates the intensity of the audio within that chunk. chunks with energy levels higher than 0.5 are classified as music. we set the detection threshold of the crnn model to 0.9 and that of the feature-based model to 0.5. audio files classified as not having music in them by both models are kept and the rest are discarded. ## manual vetting we are addressing several languages for which we do not have reliable language identification methods, or even any speech data that is verified to be in the presumed language. hence, we engaged speakers of these languages to verify that speech audio that we presumed to be in their language was indeed in their language. we carried out this manual vetting for igbo, luo (a.k.a. dholuo), ganda (a.k.a. luganda), nyanja, maithili, marwari, santali, meitei (a.k.a. manipuri), yue chinese, and central kurdish. we recruited native speakers of these languages through language service providers. we carried out this vetting process through three phases, with increasingly larger quantities and more detailed questions. phase 1 since we collected audio from only a few radio stations, our first question was to know which of them are reliable sources of speech data in the targeted languages. we sampled about a hundred 30-second speech segments per language and asked the language experts to assess whether those were indeed in their language. we also encouraged them to identify other language(s) that may be present in utterances, as well as the presence of non-speech or incomprehensible audio. for several languages, the experts also reported code-mixing with other languages, especially for maithili, marwari, meitei, and santali. phase 2 in the second phase, we scaled up the experiment to more audio samples. here, the audio samples were of different lengths (10s, 20s, 30s, and 60 seconds). we also asked detailed questions about music being present in the background, speech being spontaneous or scripted, and about the presence of multiple speakers. phase 3 since our goal was to collect at least 5 hours of vetted audio, we repeated the phase 2 study on additional audio samples using the same vetting protocol. given the feedback from the second phase, we were able to identify generally cleaner audio sources to be vetted, resulting in a much larger ratio of them assessed to be good and without background music. for logistical reasons, we were not able to do this for luo and marwari. we will release the audio with meta data from the annotation effort publicly. ## language identification the lid system follows @xcite . essentially, our lid uses log-mel-filter banks with 64 filters as feature extractor. the features were short-time mean normalized with a 3-second window. silence portions (frames) were removed using an energy voice activity detector (vad) based on kaldi. this vad classifies each frame as speech or non-speech based on the average log-energy in a window. the language embedding architecture follows the x-vector process @xcite as described by @xcite . it consists of an encoder that extracts frame-level discriminant embeddings, a pooling mechanism, and a classification head. we used the res2net architecture as the encoder. the system uses the datasets in the training open condition for training the language embedding. for the backend, the system employs a linear gaussian classifier with a single gaussian per target language, and a shared-covariance across languages. the system is trained on about 30 hours of audio in 10 languages. as shown in luo's severe performance drop on fleurs is due to the difference in the dialects in fleurs and radio broadcast test sets. the poor performance of igbo on both test sets is due to the small amount of data in igbo used in training the lid system. for most languages, our lid model outperforms the with all the tools in place, we scaled up the effort to collect audio speech data for all the targeted 48 languages. for the 10 targeted languages (bold in the table), we collected substantial amounts of data, ranging from 12.43 hours (marwari) to 178.55 hours (maithili) after music detection and language id filtering. scaling up to 48 language was challenging as we could not repeat the expensive first stage of annotations to identify radio stations which broadcast in the languages of interest. we randomly pick radio stations within locations we believe speak the languages of interest and collect data from them. since we did not run annotations for the new languages we did not have ground-truth data to train lid models for those languages. we rely on the mms lid model for these languages. specifically, we use the variant trained on 4017 languages. the amount of data collected per language varies due to the number of radio stations we collected data from at each time. for some languages, we identified many radio stations that broadcast in the language of interest, enabling us to collect hundreds of hours of data. also, we aggressively filtered the corpus for music, which greatly affected the amount of data we collected for some languages. we could not report on the amount of data after lid for egyptian, morrocan, and pashto as the mms model does not support them. other languages with no data after lid had none of the top predictions of the audio files to be in the language. this data was collected from early 2023 to early 2024. ## conclusion we collected a large corpus of speech audio for 48 languages from audio sources. we focused special attention to 10 languages for which we built language identification models based on manually vetted audio data. we will release all audio data (manually vetted and automatically filtered) open source with a liberal license for research and commercial use. we hope that this data fosters research