Multilingual self-supervised speech representations improve the speech recognition of low-resource A frican languages with codeswitching (2023)

Abstract: While many speakers of low-resource languages regularly code-switch between their languages and other regional languages or English, datasets of codeswitched speech are too small to train bespoke acoustic models from scratch or do language model rescoring. Here we propose finetuning self-supervised speech representations such as wav2vec 2.0 XLSR to recognize code-switched data. We find that finetuning self-supervised multilingual representations and augmenting them with n-gram language models trained from transcripts reduces absolute word error rates by up to 20% compared to baselines of hybrid models trained from scratch on code-switched data. Our findings suggest that in circumstances with limited training data finetuning self-supervised representations is a better performing and viable solution.

## introduction over half of the world's population uses at least two languages regularly @xcite . despite this common occurrence, automatic speech recognition (asr) models don't work well with speech that includes code-switching: when a speaker alternates between two or more languages or varieties within utterances @xcite . for lowresource languages, we encounter two issues when attempting to address this problem: insufficient data for end-to-end-training and insufficient data for language modelling. recently, self-supervised pre-training of speech such as wav2vec 2.0 @xcite have proven to give very low error rates for english asr. although very costly to pre-train, the english models and cross-lingual (xlsr) representations @xcite are available for finetuning to efficiently make speech recognisers for many languages. in this work we ask: does fine-tuning xlsr improve recognition of code-switched data over traditional training on code-switched data? to test this phenomenon, we look at four african languages (isizulu, isixhosa, sesotho, setswana) code-switched with english. we also explore three questions about how to go about this fine-tuning process. we first experiment with different types of data to add to the codeswitched dataset in order to improve asr performance, asking 1. should we add monolingual data? many other methods incorporate language identification (language id) into models, so we ask: 2. does it help to add language identification in our pipeline (either explicitly or implictly)? . we test this by augmenting utterances to implicitly identify the language and use a multi-task learning setup to learn frame-level language id and asr simultaneously. finally, we ask: 3. does a simple n-gram language model trained on the code-switched data improve performance despite the tiny amount of data?. we use the codeswitched corpus to train bigram and trigram models which we use when decoding the models. we find that finetuning multilingual pretrained models, augmented with a simple trigram language model, works well for recognizing code-switched data in low-resource languages, significantly better than prior methods of training bespoke models (cnn-tdnn-f acoustic model + lstm language model) from scratch. we find that neither language id nor adding monolingual data adds further performance gains and perhaps surprisingly, that adding monolingual data worsened model performance. our findings suggest that in circumstances with limited training data, finetuning self-supervised representations are likely a better performing and viable solution. ## related work in speech processing, work on code-switching can be divided into code-switching detection @xcite @xcite using language identification @xcite and end-to-end recognition @xcite . in this work, we look at both methods via finetuning of self-supervised representations, namely wav2vec 2.0 @xcite . language identification methods either identify the language before doing the asr on the speech or have language id trained in tandem with the acoustic model of representations. end-toend recognition splits into two main approaches: a multilingual modelling with cross lingual representations @xcite @xcite and parallel modelling generating multiple transcriptions which are interpolated to result in one transcription with the highest likelihood @xcite . for low-resource languages, we encounter two issues when attempting to apply these methods: a lack of sufficient data for end-to-end training and a lack of sufficient data for neural language modelling in the low-resource language or the codeswitched language pair. the absence of a language model for the codeswitched pair leads to prior less computationally expensive methods to fail and the lack of sufficient data for the model to generalise, resulting in poor performance of models. in our work, we focus on leveraging a pretrained self-supervised acoustic model, wav2vec 2.0 @xcite to finetune an existing multilingual acoustic model for our chosen language pairs. we incorporate language identification to see if this additional signal can improve performance given the small datasets. ## conclusion in this work, we have finetuned wav2vec 2.0 xlsr with codeswitched data of south african languages and english. we found that this system augmented with a simple bigram or trigram language model beats baseline models trained with lstm language models. we also found that it helps to add data from other languages, albeit very related languages and in the exact the same genre/domain. we were not able to improve the model with various kinds of language id information; these methods may see more success for languages with character sets that overlap less, or when there is enough data to train an end-to-end model from scratch. this work demonstrates a method to train asr models on codeswitching data with relatively minimal computation and a very basic n-gram language model, suggesting a direction for addressing an important task in the low-resource settings that characterise many of the world's languages.