Language Complexity and Speech Recognition Accuracy: Orthographic Complexity Hurts, Phonological Complexity Doesn’t (2024)

Abstract: We investigate what linguistic factors affect the performance of Automatic Speech Recognition (ASR) models. We hypothesize that orthographic and phonological complexities both degrade accuracy. To examine this, we fine-tune the multilingual self-supervised pretrained model Wav2Vec2-XLSR-53 on 25 languages with 15 writing systems, and we compare their ASR accuracy, number of graphemes, unigram grapheme entropy, logographicity (how much word/morpheme-level information is encoded in the writing system), and number of phonemes. The results demonstrate that a high logographicity correlates with low ASR accuracy, while phonological complexity has no significant effect.

## introduction when a human learns a second language, a complex writing system and a complex phonological system can both be obstacles to language learning. for example, learners of a language like chinese may spend years learning thousands of characters, while japanese learners of english commonly struggle with mastering the two liquid phonemes /r/ and /l/ that are not distinguished in japanese. by analogy with these observations, we may ask: do computational models of language, like humans, also struggle with these linguistic complexities? in this paper, we investigate the relationship between the accuracy of automatic speech recognition (asr) and linguistic complexity, specifically orthographic and phonological complexity. to answer this question, this study proposes three hypotheses about factors that may make learning asr hard. ## experimental setup and methods this section describes the setup and the methods used in the experiments. foot_0 ## results this section reports the results of the experiments to test the three hypotheses. we also ran experiments on additional languages with phonographic scripts (i.e., alphabetic or abugida writing systems), for which we were unable to measure logographicity due to lack of reliable grapheme-to-phoneme tools. these results are summarized in thus, there is a clear positive correlation of asr accuracy of fine-tuned wav2vec 2.0 models with orthographic complexities but not with phonological complexities. in other words, logographic writing systems and large character inventories can harm asr performance, supporting the first and second hypotheses, but the self-supervised pretrained model is robust to different phonological complexities, rejecting the third hypothesis. in addition to the numerical results, there were also differences in the learning curve of fine-tuning to different writing systems. as figure 5 shows for the three writing systems of japanese, the model struggles to learn to transcribe in a writing system with a larger inventory of graphemes. for the mixed orthography of kanji and kana, the validation cer never goes under 100% until 5800 steps, while the kana-only and romaji-only models start to grasp transcription at much earlier steps (1800 steps and 1300 steps, respectively). furthermore, the curves of validation cers over the steps is less smooth in complex orthographies (kanji, hangul syllables, and hanzi) than the phonographic scripts (kana, romaji, hangul jamo, zhuyin, and pinyin). this demonstrates that complex orthographies result not only in poorer asr performance but also slower learning speed and more required training data to achieve the desired performance. ## discussion in this section, we discuss the implications of the main results from the previous section. ## conclusion this study investigated what linguistic factors can confuse asr performance of fine-tuned selfsupervised models (wav2vec2-xlsr-53) focusing on orthographic and phonological complexities. the experiments trained a fine-tuned model for each language and writing system, covering 25 languages and 11 writing systems in total. the results demonstrated that speech recognition accuracy, in particular cer, strongly correlates with orthographic complexities, that is, with the size of the grapheme inventory and the degree of logographicity of a language's writing system. on the other hand, the results showed that cer has no significant correlation with the size of the phonological inventory of the target language. in addition, more complex orthographies turned out to make the model learn less accurately, more slowly, and less stably than phonographic writing systems. these results confirm the robustness of the self-supervised pretrained model fine-tuned on languages with unseen phonology and the negative effect of orthographic complexities on asr performance.