Multilingual Automatic Speech Recognition for S candinavian Languages (2023)

Abstract: We investigate the effectiveness of multilingual automatic speech recognition models for Scandinavian languages by further fine-tuning a Swedish model on Swedish, Danish, and Norwegian. We first explore zero-shot models, which perform poorly across the three languages. However, we show that a multilingual model based on a strong Swedish model, further fine-tuned on all three languages, performs well for Norwegian and Danish, with a relatively low decrease in the performance for Swedish. With a language classification module, we improve the performance of the multilingual model even further.

## introduction automatic speech recognition (asr) is the task of transforming speech into text, often referred to as transcription. multilingual asr tackles the task in multiple languages with the same model or pipeline. modern asr architectures such as deepspeech @xcite , wav2vec @xcite , and whisper @xcite are capable of transcribing speech with word error rates (wers) well below 10 percent. to achieve this, models require copious amounts of data, which is unavailable for the vast majority of languages. for low-resource languages, multilingual models as means of bootstrapping the performance are often the only solution. @xcite demonstrate that a multilingual setting can be beneficial even for highresource languages. @xcite , however, suggest that limiting models to smaller, typologically related languages is more productive than training on all languages at once. as such, it can be argued that scandinavian languages are a great fit for multilingual nlp models. swedish, danish, and norwegian all originate from old norse and share numerous similarities, such as largely overlapping lexicons and similar grammar. as noted by @xcite , the similarities across the three languages are not linear, since swedish and norwegian are most similar in speech, whereas danish and norwegian are most similar in writing. nevertheless, @xcite argue that scandinavian languages are so similar that large text-based language models for these languages should be trained jointly. it has also been shown that utilizing the similarities between the scandinavian languages can improve text-based tasks such as machine translation (e.g. @xcite and parsing (e.g. @xcite . however, to the best of our knowledge, there is no work where the usefulness of combining the scandinavian languages is reported for speech-based tasks, such as asr. we focus on identifying whether a multilingual asr model for swedish, danish, and norwegian can be trained to utilize an existing highquality monolingual model, as we fine-tune a strong swedish end-to-end model to also handle the danish and norwegian languages. in addition, we analyze how well the monolingual asr models transfer across the scandinavian languages in a zero-shot setting. we also evaluate how the multilingual setting affects the quality of transcription as opposed to monolingual settings. lastly, we show that a language classification module can be used for selecting a language model in the multilingual setting. we conduct all experiments on the wav2vec 2.0 @xcite based asr models. additional experiments, as well as more in-depth analysis, can be found in černiavski (2022). ## previous work language models for asr the usage of a language model in speech recognition has contin-uously proven to boost the quality of transcription. positive results have been observed with both statistical n-gram language models @xcite and transformer-based models, such as bert @xcite . most considerable improvements seem to result from domain-specific language models, as contextualization and biasing of models have repeatedly improved the quality of transcription @xcite . multilingual asr transcription of multiple languages via a single model or pipeline has been made possible through a variety of architectures. approaches range from an assemble of monolingual models connected through a preceding language classification component @xcite @xcite , to models sharing the phone models @xcite or hidden layers of acoustic models @xcite across two or more languages, to being conjunct on all levels, sharing all components and treating all input languages as one @xcite . as a general rule, the effects of a multilingual setting on the quality of transcription are twofold. low-resource languages tend to reap the benefits, as models seemingly generalize from the patterns learned in higher-resource languages @xcite . highresource languages, however, tend to suffer @xcite , likely due to the noise introduced through the exposure of models to data in (a) foreign language(s). nevertheless, @xcite demonstrated that there appear to be ways of mitigating the toll of a multilingual setting on the resource-rich languages by means of a typologically motivated choice of languages in a cluster as well as cluster-specific rather than one-for-all decoders. ## methodology we first evaluate the performance of monolingual swedish, danish, and norwegian models on the test sets of each language (i.e. the swedish model was evaluated on swedish, danish, and norwegian test sets). for comparison, we also evaluate the performance of an english asr model on the three scandinavian languages. we do so first to obtain comparable word error rates of each model for their intended language, except for english; second, to explore a zero-shot setting, where we explore whether the typological similarity of scandinavian languages enables the asr models trained on one of the languages to transcribe the data in the other two languages. we add english, a more distant germanic language, for comparison. in a second set of experiments, we fine-tune trial multilingual asr models for swedish, danish, and norwegian. we aim to utilize the high quality of the already fine-tuned swedish model @xcite to bootstrap the transcription in danish and norwegian as opposed to training a model on the three languages from scratch. as such, we attempt fine-tuning the swedish model in the following settings: for comparison, we also train a model on all three languages (15,000 entries per language, 45,000 total) on the pre-trained, but not fine-tuned swedish model foot_0 . we train these models for 5 epochs and evaluate on the trilingual development set every 1,000 updates. in order to investigate the effect of adding a language model for the multilingual models, we train a language classifier, see section 5 for details. in our final experiment, we select the best-performing trial model (retraining da+no+se full) to train a multilingual model for 20 epochs. we evaluate the model in two settings. in the first, we use no language model in the decoding. in the second, we use the language classifier to predict the language, in order to select a 4-gram language model for the predicted language. we train the 4-gram language models on the entirety of the original nst training sets, except for swedish, where we exclude the sample used as test set. we report word error rate as our main evaluation metric and perform a brief qualitative analysis of the most common errors. we use the cv dataset to construct a validation set for the multilingual models. for swedish and danish, we randomly sampled 2,000 validated entries per language. no validated data were available for norwegian bokmål; we, therefore, used a held-out sample of 2,000 entries from the norwegian nst training dataset. in total, we used roughly one hour of speech per language for validation. we processed each subset by downsampling the audio to 16 khz and normalizing the transcriptions. the normalization involved lower-casing all characters and removing non-alphanumeric characters, such as punctuation markers. models for monolingual baselines in swedish, norwegian, and english, we used wav2vec 2.0 models publicly available on huggingface 3 . due to the lack of an existing fine-tuned danish model at the time, we fine-tuned one ourselves: we used the publicly available pre-trained danish wav2vec 2.0 model 4 , which we then fine-tuned on one gpu for 10 epochs on our danish nst train subset. in the encoder, we retain the original parameters of the pre-trained model, whereas in the decoder, we set the batch size to 10, gradient accumulation steps to 3, learning rate to 1e-4, and weight decay to 0.005. ## language classification module the language classification module is initialized on top of the same pre-trained swedish wav2vec 2.0 model we used for asr. we train it on 15,000 entries per language, randomly sampled from the train sets (45,000 entries in total). we set the batch size to 4, learning rate to 1e-4, and gradient accumulation steps to 2, and use mean pooling. we evaluate the classification module on a concatenation of the test sets from all three languages, with results in figure 1 . the classifier reached an overall accuracy of 98% across the three languages, with very few confusions between danish and swedish. it is also noticeable that most errors occur for short segments, often containing a single word. for segments of at least five seconds, the accuracy is near perfect. ## results and discussion wers for swedish, danish, norwegian, and english asr models on the three scandinavian languages are shown in even though the results are poor, we note that the results for the scandinavian languages largely follow the patterns for mutual intelligibility between human speakers @xcite ; the swedish asr model is better at transcribing norwegian than danish, the danish model is better for norwegian than swedish, and the norwegian model is somewhat better for swedish than danish. the latter difference is more pronounced for character error rates, see @xcite . the scores confirm that the addition of a simple n-gram language model leads to stable improvements of the quality in transcription, even in a cross-lingual setting. the swedish model is an exception, though, likely due to the overall high quality of the model, which is only limited by such a language model. lastly, qualitative analysis of the outputs reveals that some of the predictions considered to be errors due to a deviation from the ground truth are grammatically correct alternative spellings that can have the same pronunciation. for instance, in the output of the monolingual swedish model, some of the most common substitution errors are skall instead of ska, i stället instead of istället, and i dag instead of idag. due to the usage of wer as an evaluation metric, the latter two examples are treated as 2 errors each. this is because wer considers istället to be substituted with stället and treats the preposition i to be an insertion error. similar patterns can be observed in the outputs for the other two languages, which leads us to believe that wer might not be the most suitable evaluation metric for scandinavian, and possibly other, languages. wer for the trial multilingual models are shown in figure 2 . the results indicate that the initialization of the multilingual model from a monolingual model is only effective in low-resource settings. this is because a model trained from scratch on all three languages reaches comparable wer within roughly 5,000 steps. nevertheless, despite the subtle difference, the average wers on all three languages indicate that the model initialized from a fine-tuned swedish model and further fine-tuned on complete training sets (retraining da+no+se full) is second only to monolingual baselines. analogous patterns can be seen in terms of character error rates @xcite . hence, we choose this setting for training our final multilingual model. the scores of the final multilingual asr model able to transcribe swedish, danish, and norwegian, as opposed to the monolingual baselines are shown in we observe that the multilingual model performs significantly better in norwegian than it does in danish, which can also be seen from the progression in the wers of the trial models shown in figure 2 . this is likely because the development and test sets for norwegian we used were from the same domain, which was not the case for danish, but it may also be influenced by norweigan pronunciation being closer to swedish than danish to swedish. černiavski (2022) presents a more detailed qualitative analysis of the transcription in the monolingual versus multilingual setting, as well as with and without lm settings. we observe that cross-lingual errors (e.g. when a swedish word is transcribed with a norwegian spelling) are very rare in a multilingual setting even when lms are not used. ## conclusions