Boosting N orwegian Automatic Speech Recognition (2023)

Abstract: In this paper, we present several baselines for automatic speech recognition (ASR) models for the two official written languages in Norway: Bokmål and Nynorsk. We compare the performance of models of varying sizes and pre-training approaches on multiple Norwegian speech datasets. Additionally, we measure the performance of these models against previous state-of-the-art ASR models, as well as on out-of-domain datasets. We improve the state of the art on the Norwegian Parliamentary Speech Corpus (NPSC) from a word error rate (WER) of 17.10% to 7.60%, with models achieving 5.81% for Bokmål and 11.54% for Nynorsk. We also discuss the challenges and potential solutions for further improving ASR models for Norwegian.

## introduction automatic speech recognition (asr) is the task of converting speech into text. asr systems are used in a wide range of applications, such as voice assistants, transcription services, and speech-totext translation. it is also increasingly becoming a tool for research in spoken language as the accuracy of the more recent neural-based models is approaching that of humans for certain metrics. in a study by @xcite , the authors estimated that the word error rate (wer) in humanproduced transcriptions on the librispeech benchmark @xcite is roughly 5.83%, while their end-to-end asr model, deepspeech 2, achieved a wer of 5.33% on a clean test set, although it was outperformed by humans on noisy data. since the introduction of deepspeech 2, the field of asr has progressed even further, with the current leaderboard of the benchmark containing over ten models with a wer below 2%. despite the high accuracy in resource-rich languages, asr models are currently unavailable for the vast majority of the world's languages due to the lack of gold annotated data to train such models. recent advances in unsupervised learning of acoustic models have decreased the need for transcribed speech. in this paper, we focus on developing and evaluating a new set of baselines asr models for norwegian based on the wav2vec 2.0 architecture @xcite . we make use of existing pre-trained models and combine them with other language resources for the norwegian languages to further improve the accuracy of the resulting asr systems. our models seem to perform notably better than previous work on newly established datasets. ## norwegian asr the norwegian language has many spoken dialects, which differ lexically, grammatically, and phonologically. additionally, there are two official written standards of norwegian, bokmål and nynorsk, which have somewhat different inflection, vocabulary, and spelling. consequently, high-quality datasets for acoustic modeling of norwegian require speech data in different dialects and should ideally include transcriptions in both written standards. early work on norwegian speech recognition was mostly focused on very limited vocabularies and numbers, tailored for telephone applications and menu navigation @xcite @xcite . compound words are more frequent in norwegian than english, but using traditional pronunciation dictionaries seemed sufficient in controlled lexicons. in norwegian, natural numbers between 20 and 99 can be pronounced differently (e.g. "twentyfour" and "four-and-twenty"), which poses a chal-lenge for natural number recognition. by the year 2000, and under the umbrella of a few eu-funded projects, research focused mostly on overcoming these limitations and extending the use cases to dates, times, nouns, and the spelling out of words, which yielded several important datasets (e.g., speechdat, speechdat-ii, tabu.0) and technical improvements over a short period of time @xcite @xcite @xcite . most approaches were based on hidden markov models and some relied on mel frequency cepstral coefficients (mfcc), commonly by using the hidden markov model toolkit (htk) @xcite . however, these approaches were not designed for open-ended recognition and often struggled with out-of-vocabulary words or real conversations. it was not until the introduction of newer datasets in the last decade that systems with reasonable performance started to appear. ## methods in this work, we evaluate these models, referred to as npsc-bokmål and npsc-nynorsk, and finetune new xls-r 1 billion (1b) parameters and voxrex 300 million (300m) parameters models using the same hyperparameters foot_3 . we train the models on npsc and ablate on different data supplementing strategies derived from the nst dataset. the nst dataset was modernized and reorganized by the national library of norway, and is now available in a reader-friendly format (nordisk språkteknologi, 2020). we omitted the second channel of audio recorded with a distant microphone due to no noticeable differences between the audio recorded with the close microphone. the dataset is representative of the major regions and the language variety spoken in that region, although the representation of the dialectal varieties of the scandinavian languages in the dataset is debatable (see appendix c, we trained all models for 40 epochs on a single nvidia rtx a6000 gpu with an effective batch size of 24 by accumulating gradients every 2 steps on a batch size of 12. the learning rate was set to 2 • 10 - foot_4 , with 2,000 steps of warmup and linear decaying using an adam optimizer with β 1 = 0.9, β 2 = 0.999, and ϵ = 10 - foot_7 . we used the pytorch models available in the huggingface hub. after fine-tuning, separate bokmål and nynorsk 5-gram kneser-ney language model were added where appropriate 5 . two versions of the nst-npsc model were also created, one with the bokmål 5-gram language model, and another one with the nynorsk language model, as we evaluate the nst-npsc model on both subsets of npsc. these language models were created using a combination of the training and validation sets of npsc plus a few thousand random documents from the norwegian colossal corpus @xcite . we processed a total of 78 million words by lowercasing, normalizing, and filtering out the characters that were outside the 28 norwegian letters used for fine-tuning. we used the implementation of kneser-ney models @xcite available in the kenlm library @xcite . the estimation of the ctc α and β values was done by grid search over {0.001, 0.01, 0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 3} on the validation set of the bokmål subset of npsc; we established α = 0.5 and β = 0.001. ## results and discussion we evaluate the performance of the models grouping their scores by the written language of the test sets in npsc and nst. we report word error rates as percentages foot_5 . for comparison purposes, we include the figures obtained in the original npsc paper by @xcite , as well as the work by @xcite who also briefly evaluated asr on npsc. models. in both cases, it can be seen that models trained on the bokmål subset of npsc perform not too well on the test set of nst. similary, models trained only on nst underperform on the test set of the bokmål subset of npsc. adding a 5gram language model yields significant improvements across the board, ranging from a 5 points increase on the worst performing pairs of model and dataset, to a 1 point increase for the best performing pairs. however, the biggest gain in performance is the addition of extra data. the models fine-tuned on combinations of npsc and nst produce significantly better results. on the whole npsc, the 300m nst-npsc model outperform solberg and ortiz (2022) by 9.5 points and the previous state of the art npsc-bokmål model by 4.16 points. for the other datasets, the 1 billion parameters model nst-npsc-bokmål outperformed the rest of models, yielding increases over the npsc-bokmål model of 0.6 points on npsc (bokmål) subset and of 14.89 points on nst. interestingly, the performance of the best 300m and 1b models was very close. an evaluation of the models for each region in the test set of nst can also be found in appendix c with somewhat similar results and trends. we found that there is virtually no difference in the per region performance of the models, even for the unbalanced (in terms of hours of speech in test set) regions of oslo and sør-vestlandet. it is important to notice that the regions identified in nst do not reflect the diversity of spoken dialects in norway. for nynorsk, as shown in in order to evaluate the generalization capabilities of our models, we use the norwegian test set of fleurs. transcriptions on fleurs were normalized as closely as possible to those present in the nst and npsc, with numbers and times written out in text form. we compare the performance of our models against the whisper models @xcite ## future work despite the improved performance of our models compared to the other baselines, asr models for norwegian still face several challenges. one major challenge is the complex phonetics and morphology of the different dialects, which makes it difficult for models to accurately transcribe the phonemes in the input speech to the correct spelling. another challenge is the limited availability of high-quality datasets for norwegian speech, which limits the amount of training data for asr models. to address these challenges, one possible to solution is to combine multiple datasets and sources of training data, such as transcribed speech and synthetic speech, to increase the amount of pretraining data for asr models. with enough transcribed speech, even other more data-hungry architectures could be tested, such as whisper. finally, the prospect of training wav2vec 2.0 directly on non-normalized text is an interesting avenue for research, as it would make the models directly usable without having to transform the output of the models to make them more readable. ## conclusion in this paper, we presented several new models for automatic speech recognition of norwegian. we evaluated these models on several datasets of norwegian speech and compared their performance to previous work, outperforming the previous state of the art. given that we used almost the same settings than the wav2vec 2.0 models released last year, with the addition of extra training time and data there are some interesting findings. first, adding over 400 hours of extra planned speech to the semi-improvised speech part of npsc, performance does not plummet, but actually increases from 6.41 to 5.81 wer for bokmål in the 1b settings. the 300m model seems more sensitive in this regard and the wer decreases from 7.14 to 7.96 wer. for nst, the trend is exactly the same, although the differences are smaller. interestingly, the out of domain performance of the models is also greatly improved by adding the planned speech in nst to npsc. models on both sizes increase their wer scores from 12.98 to 9.88 for the 300m model, and from 13.03 to 9.87 for the 1b model. we are releasing our best performing models and evaluation code for replicability, and hope to contribute to the advance of asr for norwegian.