A character-based analysis of impacts of dialects on end-to-end N orwegian ASR (2023)

Abstract: We present a method for analyzing character errors for use with character-based, end-to-end ASR systems, as used herein for investigating dialectal speech. As end-to-end systems are able to produce novel spellings, there exists a possibility that the spelling variants produced by these systems can capture phonological information beyond the intended target word. We therefore first introduce a way of guaranteeing that similar words and characters are paired during alignment, thus ensuring that any resulting analysis of character errors is founded on sound substitutions. Then, from such a careful character alignment, we find trends in system-generated spellings that align with known phonological features of Norwegian dialects, in particular, “r” and “l” confusability and voiceless stop lenition. Through this analysis, we demonstrate that cues from acoustic dialectal features can influence the output of an end-to-end ASR systems.

## introduction automatic speech recognition (asr) has, like all machine learning tasks, struggled with generalization. that is, a model will perform well on the task and data it was trained on but when presented with new examples, especially examples that differ in some dimension from the training data, the model will perform markedly less well. in the task of asr, this means that models often struggle with generating correct transcriptions for speakers whose age, gender, or dialect differs from that of the speakers on which the model was originally trained. of specific focus in this paper is the impact of dialect on a modern asr system. dialect information has been used in different ways in asr. in some applications, such as dialect identification (did), the goal is to correctly identify the dialect for a given sample of speech. @xcite , for example, used a combination of speech and text features to perform did. in other cases, did is combined with asr systems to improve transcription accuracy. for example, @xcite used bottleneck features extracted via unsupervised deep learning to perform did for both chinese and arabic. similarly, @xcite used a multitask model for both did and asr. this multitask approach outperformed the single task systems on both did and asr. beyond did, the behavior of asr systems has been analyzed with respect to dialectal speech (as we do in this paper). this in order to explore phonetic phenomena, as well as to gain insights into the way those complex systems work. in these studies, even when dialectal information is not an explicit target, there is still an interest to understand what phonetic and dialectal information has been captured in asr models. with traditional asr models, this investigation has been fairly straightforward as these models have consisted of three semi-independent components: the acoustic model, the language model, and the lexicon. because of the separate acoustic models within these multi-component models, one could, for example, perform clustering on the model parameters themselves such as @xcite . in this work, salvi performed clustering on the acoustic model features and correlated the resulting clusters with known dialectally realized phonemes. instead of directly using an acoustic model from an asr system, @xcite adapted the concept of an hmm acoustic model to automatically discover dialect-specific phonetic rules. unlike multi-component asr systems, investigating modern, end-to-end models for phonetic and dialectal information is quite different. whereas parameters from an acoustic model may be extracted and used independently, the acoustic information in an end-to-end model cannot be so easily excised. this design makes it more challenging, but not impossible, to investigate what acoustic information is captured where in the network. belinkov and collaborators used the output from each layer of an end-to-end system to train phonetic, grapheme, and articulatory classifiers @xcite . @xcite investigated dialectal information captured by an end-to-end system using not only layer-wise classification but also gradient and information-theoric analysis. all of these works are focused on analyzing the networkinternal representations detached from actual network output. the output from asr models is constrained by the model architecture. traditional asr models with lexicons are bound to output only words contained within that lexicon. this means that all transcripts generated by these models contained only real, known words even if the transcribed output did not necessarily match the word that was spoken. additionally, these models do not allow for acceptable variation in spelling. for example, the word, "favorite," would always be spelled "favorite" never "favourite," even if the latter might better reflect the preference of a british english speaker. conversely, these newer end-to-end architectures, trained using connectionist temporal classification (ctc) loss, produce output at the character instead of word level. this permits the model to create novel words and spellings, potentially better reflecting the phonetic realization of the spoken word. given that ctc models are allowed to generate novel spellings, there exists the potential that dialectal information will be captured by the model output itself via non-standard spellings. the goal of this paper is to investigate whether dialectal acoustic information can impact spellings with an end-to-end model. in order to test this, we used wav2vec 2.0 @xcite to generate transcriptions of norwegian speech. we then performed an analysis of the resulting transcripts for captured dialectal knowledge via a dialectalregion based evaluation of character error patterns. from this analysis we are able to see known norwegian dialectally-based phonological patterns, specifically around "r" and "l" confus-ability and stop consonant voicing. thus we illustrate that strong enough acoustic dialectal cues can effect the character output of an end-to-end asr system. ## norwegian language and dialects in this paper, we focus our analysis on the norwegian language. though spoken by a relatively small population of a little over 5 million speakers, norwegian contains many dialects differentiated in phonology, syntax, and lexicon. in addition to dialectal variation, norwegian also maintains two official written standards: bokmål and nynorsk; though neither written standard directly corresponds with a spoken variant. furthermore, norway does not recognize any official language standard. indeed, people are encouraged to use their preferred written standard and native dialect in all aspects of work and life. the variety in dialects stems from norway's challenging and rugged topography that has historically forced the populace to organize into many, smaller communities. over time, the diversity we see in norwegian dialects developed in these small, isolated communities. as described by phoneticians, there now exist large dialectal phonetic variations ranging from infinitive verb endings to palatalization of consonants, to /r/ and /l/ realizations, to the various pronunciations for the personal pronoun for "i", jeg -ranging from @xcite to @xcite to [i] and more @xcite . while the number of specific norwegian dialects is quite large, we can group these dialects into larger dialect groups for the purpose of this investigation. these grouping could be either into the regional names used by skjekkeland or into the even larger, cardinal regions of "east," "west," "north," "south," and "mid." the analysis outlined in this paper relies on these cardinal regions. ## discussion due to the fact that we have been able to largely see acoustic dialectal features surfacing through our analysis, we find that this method of carefully aligning text and aggregating results has promise. furthermore, we infer that the models have learned enough about norwegian to understand standard spellings and apply these generalizations to broader contexts. additionally, the phonetic information in the dialects is strong enough to cause the models to utilize this general spelling knowledge and create more acoustically aligned outputs. however, going so far as to say that the models have internalized some knowledge about the dialects themselves (e.g., phonetic features) is perhaps more than can be reasonably asserted from this analysis. through this paper we have explored a couple of known dialectally-motivated phonological realizations. there still, however, exist more that could be explored. as mentioned in section 4.2, there exists a pattern of retroflexting of alveolar consonants for certain norwegian dialects. this analysis could certainly be extended to those environments. however, there are also phonological changes that are hard, or potentially impossible to see in spelling changes. for example, alveolars are palatalized (most strongly) in the "mid" region as well as in certain phonological environments in the "north" and the northern parts of the "west" and "east" regions. this palatalization would be hard to see in spellings since there is no standard way in norwegian orthography of representing a palatalized sound. additional norwegian phonological features that have no written representation (such as toneme) would also be invisible to the analysis performed in this paper. as the npsc is derived from parliamentary speeches, the distribution of parliament speakers emulates the population distribution of the country. thus our models, all of which were trained on npsc, have the same speaker representation. that is, the "east" region would be the most represented in the training data. given this, and the results in ## conclusion through this paper, we demonstrate how an analysis of character errors in transcriptions generated by an end-to-end asr system can contain dialectal trends mirroring those known through linguistic descriptions. we showed increased confusability between "r" and "l" in regions where those phonemes are realized similarly. we also showed increased incidences of voiceless stop lenition in a region known for that phenomena. these errors indicate that the end-to-end system has successfully learnt to spell in norwegian, going so far so as to slightly spell in dialect.