A Large N orwegian Dataset for Weak Supervision ASR (2023)

Abstract: With the advent of weakly supervised ASR systems like Whisper, it is possible to train ASR systems on non-verbatim transcriptions. This paper describes an effort to create a large Norwegian dataset for weakly supervised ASR from parliamentary recordings. Audio from Stortinget, the Norwegian parliament, is segmented and transcribed with an existing ASR system. An algorithm retrieves transcripts of these segments from Stortinget’s official proceedings using the Levenshtein edit distance between the ASR output and the proceedings text. In that way, a dataset of more than 5000 hours of transcribed speech is produced with limited human effort. Since parliamentary data is public domain, the dataset can be shared freely without any restrictions.

## introduction until recently, automatic speech recognition (asr) models needed to be trained on speech recordings and verbatim transcriptions of such recordings. transcribing speech manually wordby-word takes time, and is often difficult. therefore, there was a limited amount of suitable training data for smaller languages. at the time of writing, the asr datasets with an open license available for norwegian amount to less than 1000 hours (solberg and ortiz, 2022). with the advent of transformer-based asr systems, the possibility has opened up for using non-dedicated datasets as training data for asr (baevski et al., 2020; radford et al., 2022). in 2022, openai released whisper, a multilingual asr model trained on 680 000 hours of audio and subtitles collected from videos online, which performs well across many languages (radford et al., 2022). weakly supervised asr training, training asr models on non-verbatim transcriptions such as subtitles, makes it possible to harvest large quantities of pairs of audio and transcription, which is a great advantage for under-resourced languages like norwegian. as whisper is not trained on verbatim transcriptions, it often does not produce verbatim transcriptions either: it tends to write easily readable text even when the speaker stutters, repeats herself etc. to some degree one might say that the transcriptions represent what the speaker means rather than what the speaker says. for many downstream tasks such as transcribing videos and interviews, this is the desirable outcome. this paper describes an ongoing effort to create a dataset for weakly supervised asr training with data from stortinget, the norwegian parliament. using a method described in (ljubešić et al., 2022), we have aligned segments of audio from meetings at stortinget with corresponding text segments in the official proceedings. this results in a 5000+ hours dataset, 1245 of which may be considered near-verbatim. an advantage of using parliamentary data is that there are no copyright or privacy restrictions on the audio data or the textual data. the dataset will therefore be made available with an open license on huggingface. ## background until 2021, the only large, open dataset for norwegian asr training was a 540 hour dataset with manuscript-read speech made by the firm nordisk språkteknologi (nst) at the beginning of the millenium. 1 this is a reasonable dataset for asr of scripted speech, but it is insufficient as training data for unscripted, spontaneous speech, as it lacks dialectal phenomena as well as hesitations, stuttering, repetitions and other features typical of unplanned speech. moreover, the dataset has almost exclusively transcriptions in bokmål, the more used of the two written standards of norwegian. there were no appropriate datasets for asr training with transcriptions in the other standard, nynorsk, which is closer in spelling to many rural dialects, particularly on the west coast, and is used by a sizeable minority of the population. in 2021, the language bank at the national library of norway released the norwegian parliamentary speech corpus (npsc), consisting of 126 hours of transcribed speech from meetings at stortinget in 2017 and 2018. @xcite the npsc is transcribed verbatim by linguists and philologists, and the transcriptions are tailor-made for asr training and testing. while the members of parliament often read from a manuscript, a large amount of the npsc is unplanned speech and dialectal speech, and solberg and ortiz (2022) showed that training asr models on the npsc has a beneficial effect on the word error rate (wer) across norwegian dialects compared to models trained on the nst dataset only. moreover, using the npsc as training data decreased the differences in wer between dialects. about 13% of the transcriptions in the npsc are in norwegian nynorsk. the ai lab at the national library of norway fine-tuned separate, unsupervised wav2vec2 models on the bokmål and nynorsk transcriptions of the npsc (de la rosa et al., 2023). the 1b parameter bokmål model foot_2 obtains a wer of 6.4% on the bokmål part of the npsc test set, and the 300m parameter nynorsk model foot_3 obtains a wer of 12.7% on the nynorsk part of the test set. in informal tests, the models also give decent results on non-parliamentary data, although not quite as good as on recordings from stortinget. while the npsc is a valuable resource for speech recognition of norwegian, more data is needed. however, manual, verbatim transcription is very time-consuming, expensive and difficult to do in a systematic way. there are often both recordings and official non-verbatim transcripts of meetings of parliaments, and there have been several successful efforts of extracting transcription from parliamentary data (helgadóttir et al., 2017; iranzo-sánchez et al., 2020; kirkedal et al., 2020; virkkunen et al., 2022). as the parliamentary transcripts are not verbatim and older asr systems need verbatim transcriptions as training data, the extraction process often involves significant data cleaning to retain only fragments which are verbatim or close to verbatim (see e.g. kirkedal et al., 2020). ljubešić et al. (2022) describe the creation of a 1860 hour asr dataset from croatian parliamentary data, parlaspeech-hr. the transcriptions are automatically extracted and are not necessarily verbatim, but a score indicates how closely the asr output and the extracted transcription match. they fine-tuned multilingual and slavic wav2vec2 models on a 300 hour subset of the data. the bestperforming model has a wer of 4.3% on the test set of parlaspeech-hr. radford et al. (2022) show that it is possible to obtain good asr results with weak supervision, using subtitles as training data. audio segments of parliamentary speech aligned with official proceedings may be a good data source for weakly supervised asr: there is a large amount of accessible speech data, and there are transcriptions created by humans. furthermore, like subtitles, the parliament proceedings transcribe what the speakers mean instead of transcribing word by word what the speakers say, and they correspond more closely to the desirable output in many downstream tasks. @xcite 3 producing the dataset ## results by using this method, we extracted transcriptions for 724 783 segments, which amounts to 5190 hours, i.e. 84% of the duration of the identified speech segments (6182 hours, cf. section 3.1). the average segment word length for the extracted transcriptions is 59.15. ## future work for now, we have only extracted texts from the parlamint-no corpus. however, parlamint-no contains rich metadata. the complete release of the dataset will include parlamint speaker identifiers as well as identifiers of the relevant sections in parlamint-no. with this information, it is possible to couple the segments of the dataset with the metadata in parlamint-no and enrich them with, e.g. the gender and age of the speakers. we can also extract language codes directly from the metadata, which is likely a more accurate method for identifying bokmål and nynorsk segments than the asr-based method used here. the best way to evaluate the quality of this dataset is to use it to train a weakly supervised asr system. we plan to use it to train whisper models for norwegian and compare the performance of these models to existing whisper models and other norwegian asr models. once this training is complete, we will release the scores, which will give more insights into how useful this dataset may be. ## conclusion this paper describes the creation of a large, transcribed audio dataset for asr training with relatively small human effort. when it is released, the dataset will be several times larger than all openly available, transcribed audio data in norwegian today. we believe that the dataset is the largest open dataset of its kind that can be made for norwegian, as there do not exist, to our knowledge, any other sources of speech recordings and corresponding transcriptions that are free of copyright and privacy restrictions. parliamentary speech is also of a quite different genre from most subtitled videos, which currently constitute the primary source of data for weakly supervised asr, so we expect the dataset to add valuable variation. moreover, the dataset contains 623 hours of speech transcribed to nynorsk, 176 of which have a match ratio above 0.8. we expect this data to be a particularly valuable addition, as there is currently very limited training data for nynorsk asr and few systems that support nynorsk.