Improving Generalization of N orwegian ASR with Limited Linguistic Resources (2023)

Abstract: With large amounts of training data, it is possible to train ASR models that generalize well across speakers and domains. But how do you train robust models when there is a limited amount of available training data? In the experiments reported here, we fine-tuned a pre-trained wav2vec2 ASR model on two transcribed, Norwegian speech datasets, one with parliamentary speech and one with radio recordings, as well as on combinations of the two datasets. We subsequently tested these models on different test sets with planned and unplanned speech and with speakers of various dialects. Our results show that models trained on combinations of the two datasets generalize better to new data than the single-dataset models, even when the length of the training data is the same. Our lexical analysis sheds light on the type of mistakes made by the models and on the importance of consistent standardization when training combined models of this kind.

## introduction automatic speech recognition has experienced tremendous development in the past decades. new models have improved in sophistication as well as complexity, see e.g., (chorowski et al., 2015; chan et al., 2015; amodei et al., 2016; synnaeve et al., 2019; j. li et al., 2020; baevski et al., 2020; b. li et al., 2021; radford et al., 2022) and references therein. however, these models are still at best as good as the data they are trained on. spoken language presents many dimensions, such as degree of spontaneity, dialectal variation, task domain, and age of speaker, that affect the performance of a speech recognizer. models trained on a specific combination of those factors will perform poorly in other conditions. with sufficient computing capacity and data, this problem can be mitigated by training large models on even larger sets of data spanning several conditions, both in terms of the spoken content as well as transcription standards (e.g. whisper, radford et al., 2022). however, when computing power and quantity of data are limited, it is necessary to provide a dataset with a common transcription standard and enough variety of acoustic and linguistic features to fully encapsulate the target language. such a dataset can then be used to finetune a large pre-trained model such as wav2vec2 (baevski et al., 2020); a process less computationally expensive than training from scratch. until recently, freely available speech datasets suitable for training norwegian asr models only contained manuscript-read speech. however, in 2021, the norwegian language bank released the norwegian parliamentary speech corpus (npsc), containing 126 hours of transcribed speech from stortinget, the norwegian parliament. the dataset contains a decent amount of spontaneous speech, as well as dialectal variation. solberg and ortiz (2022) showed that asr models trained on the npsc were better at transcribing spontaneous speech and dialects than models trained on manuscript-read speech only. researchers at the national library of norway have fine-tuned wav2vec2 asr models on the npsc training split. their 1b parameter model 1 obtained a word error rate (wer) of 6.4% on the npsc test split (de la rosa et al., 2023). @xcite however, when we apply asr systems trained on the npsc to speech data such as broadcasts and informal speech, they tend to perform less well than on parliamentary speech. this is understandable since parliamentary speech is rather formal and contains few interruptions and instances of real conversation. moreover, the topics discussed in parliament are relatively restricted, and words which occur frequently in everyday conversations may be rare or nonexistent in such a dataset. finally, the recording conditions are very homogeneous. in this paper, we investigate the effect of finetuning wav2vec2 models on more varied norwegian training data. two models are trained on single training sets with unplanned speech, the npsc and the broadcasting dataset rundkast, while two others are trained on combinations of these datasets. we show that the models trained on combined data sources generalize better to new data than the models trained on single datasets and that this effect cannot be attributed to the length of the training data. our analyses also show that standardization, the process of making transcriptions as uniform as possible across datasets, is key when combining training data in this way. the outline of the rest of the paper is as follows. in section 2, we describe the datasets used in the experiments. section 3 reports on how the models were trained, as well as the experimental setup. the results of the experiments are described and discussed in section 4, and we also use a technique from corpus linguistics called keyword analysis to get a deeper understanding of the differences between the models. section 5 concludes the paper and suggests some avenues for further development and research, based on our results and the state of norwegian asr. @xcite ## datasets for norwegian asr there are a number of datasets that can be used for training and testing norwegian asr systems. this section describes the most important, freely accessible norwegian speech datasets, as well as one which is not freely accessible, rundkast. all of these datasets have transcriptions in norwegian bokmål, the most commonly used of the two written standards of norwegian, while some also have transcriptions in nynorsk. in the experiments of this paper, we only use the bokmål transcriptions. ## experiments the goal of our experiments is to verify to what extent we get improvements in performance when training on more varied spontaneous data than the npsc data alone. to this aim we fine-tuned wav2vec2 models on both rundkast and the npsc individually, as well as on combinations of the two datasets. we then test the models on test sets from the same domain as the training set, as well as from different domains using the nb tale and the nst corpora. ## conclusion and future work in this paper, we have shown that training asr models on a combination of parliament speech data from the npsc and broadcast data from rundkast results in better wer across different test sets than models trained on these datasets individually. this effect persisted even when we control for dataset length. while stortinget is slightly better on the npsc test set than the combined model of similar length, combined short outperforms both single dataset models on all other test sets. in other words, the combined models generalize better to out-of-domain speech data, which makes them more suitable for downstream transcription tasks where different kinds of speech data may be encountered, such as meeting transcriptions and subtitling. the study also highlights that it is important to standardize the training and test data when combining datasets in this way. this standardization may require an intimate knowledge of the transcription guidelines of the different datasets. even though we had standardized the datasets prior to training, as described in section 2.5, we did not discover the differences in the treatment of acronyms and compounds until we investigated the asr outputs in detail. to be able to train on combinations of datasets, one needs to have access to asr dataset of different types and genres. before the release of the npsc in 2021, there were no large, open datasets for asr training with norwegian unplanned speech. the npsc may be released openly because parliamentary recordings are in the public domain. due to copyright and privacy issues, it is more difficult to make a dataset with broadcast data such as rundkast freely available. a recent report from the norwegian board of technology points out that there are not enough open asr datasets for norwegian, and the datasets that exist are not sufficiently varied. it suggests different ways to increase the amount of open speech data, such as a major crowdsourcing initiative (tennøe and wettre, 2022). while we wait for more open data, it may be possible to train asr models on a combination of open and non-open datasets and release the resulting models openly. finally, the results we obtained with whisper are not comparable to ours using wer. this is because the model is trained to produce transcrip-tions of different standards. this emphasizes the importance of developing new metrics that assess the semantic content of the transcriptions rather than the word accuracy.