Linguistic Knowledge in Multilingual Grapheme-to-Phoneme Conversion (2021)

Abstract: This paper documents the UBC Linguistics team’s approach to the SIGMORPHON 2021 Grapheme-to-Phoneme Shared Task, concentrating on the low-resource setting. Our systems expand the baseline model with simple modifications informed by syllable structure and error analysis. In-depth investigation of test-set predictions shows that our best model rectifies a significant number of mistakes compared to the baseline prediction, besting all other submissions. Our results validate the view that careful error analysis in conjunction with linguistic knowledge can lead to more effective computational modeling.

## introduction with speech technologies becoming ever more prevalent, grapheme-to-phoneme (g2p) conversion is an important part of the pipeline. g2p conversion refers to mapping a sequence of orthographic representations in some language to a sequence of phonetic symbols, often transcribed in the international phonetic alphabet (ipa). this is often an early step in tasks such as text-to-speech, where the pronunciation must be determined before any speech is produced. an example of such a g2p conversion, in amharic, is illustrated below: €≈r{ → [amar1ï:a] 'amharic' for the second year, one of sigmorphon shared tasks concentrates on g2p. this year, the task is further broken into three subtasks of varying data levels: high-resource ( 33k training instances), medium-resource (8k training instances), and lowresource (800 training instances). our focus is on the low-resource subtask. the language data and associated constraints in the low-resource setting will be summarized in section 3.1; the reader interested in the other two subtasks is referred to @xcite for an overview. in this paper, we describe our methodology and approaches to the low-resource setting, including insights that informed our methods. we conclude with an extensive error analysis of the effectiveness of our approach. this paper is structured as follows: section 2 overviews previous work on g2p conversion. section 3 gives a description of the data in the lowresource subtask, evaluation metric, and baseline results, along with the baseline model architecture. section 4 introduces our approaches as well as the motivation behind them. we present our results in section 5 and associated error analyses in section 6. finally, section 7 concludes our paper. ## previous work on g2p conversion the techniques for performing g2p conversion have long been coupled with contemporary machine learning advances. early paradigms utilize joint sequence models that rely on the alignment between grapheme and phoneme, usually with variants of the expectation-maximization (em) algorithm @xcite . the resulting sequences of graphones (i.e., joint graphemephoneme tokens) are then modeled with n-gram models or hidden markov models (e.g., @xcite @xcite . a variant of this paradigm includes weighted finite-state transducers trained on such graphone sequences @xcite . with the rise of various neural network techniques, neural-based methods have dominated the scene ever since. for example, bidirectional long short-term memory-based (lstm) networks using a connectionist temporal classification layer produce comparable results to earlier n-gram models @xcite . by incorporating alignment information into the model, the ceiling set by n-gram models has since been broken @xcite . attention further improved the performance, as attentional encoder-decoders @xcite learned to focus on specific input sequences. as attention became "all that was needed" @xcite , transformer-based architectures have begun looming large (e.g., @xcite . recent years have also seen works that capitalize on multilingual data to train a single model with grapheme-phoneme pairs from multiple languages. for example, various systems from last year's shared task submissions learned from a multilingual signal (e.g., elsaadany and suter, 2020; @xcite . ## the low-resource subtask this section provides relevant information concerning the low-resource subtask. ## our approaches this section lays out our attempted approaches. we investigate two alternatives, both linguistic in nature. the first is inspired by a universal linguistic structure-the syllable-and the other by the error patterns discerned from the baseline predictions on the development data. ## results the performances of our systems, measured in wer, are juxtaposed with the official baseline results in ## error analyses in this section, we provide detailed error analyses on the test-set predictions from our best system. the goals of these analyses are twofold: (i) to examine the aspects in which this model outperforms the baseline and to what extent, and (ii) to get a better understanding of the nature of errors made by the system-we believe that insights and improvements can be derived from a good grasp of error patterns. we analyzed the mismatches between predicted sequences and ground-truth sequences at the segmental level. for this purpose, we again utilized many-to-many alignment @xcite , but this time between a predicted sequence and the corresponding ground-truth sequence. 6 for each error along the aligned sequence, we classified it into one of the three kinds: the frequency of each error type made by the baseline model and our systems for each individual language is plotted in figure 2 . some patterns are immediately clear. first, both systems have a similar pattern in terms of the distribution of error types across language, albeit that ours makes fewer errors on average. second, both systems err on different elements, depending on the language. for instance, while adyghe (ady) and khmer (khm) have a more balanced distribution between consonant and vowel errors, slovene (slv) and welsh (wel_sw) are dominated by vowel errors. third, the improvements gained in our system seem to come mostly from reduction in vowel errors, as is evident in the case of khmer, latvian (lav), and, to a lesser extent, slovene. the final observation is backed up if we zoom in on the errors in these three languages, which we visualize in figure 3 . many incorrect vowels generated by the baseline model are now correctly predicted. we note that there are also cases, though less common, where the baseline model gives the right prediction, but ours does not. it should be pointed out that, although our system shows improvement over the baseline, there is still plenty of room for improvement in many languages, and our system still produces incorrect vowels in many here we only visualize the cases where either the baseline model gives the right vowel but our system does not, or vice versa. we do not include cases where both the baseline model and our system predict the correct vowel, or both predict an incorrect vowel, to avoid cluttering the view. each baseline-ground-truth-ours line represents a set of aligned vowels in the same word; the horizontal line segment between a system and the ground-truth means that the prediction from the system agrees with the ground-truth. color hues are used to distinguish cases where the prediction from the baseline is correct versus those where the prediction from our second system is correct. shaded areas on the plots enclose vowels of similar vowel quality. finally, we look at several languages which still resulted in high wer on the test set-ady, gre, ita, khm, lav, and slv. we analyze the confusion matrix analysis to identify clusters of commonly-confused phonemes. this analysis again relies on the alignment between the groundtruth sequence and the corresponding predicted sequence to characterize error distributions. the results from this analysis are shown in figure 4 , and some interesting patterns are discussed below. figure 2 suggests that khmer has an equal share of consonant and vowel errors, and the heat maps in figure 4 reveal that these errors do not seem to follow a certain pattern. however, a different picture emerges with latvian and slovene. for both languages, figure 2 indicates the dominance of errors tied to vowels; consonant errors account for a relatively small proportion of errors. this observation is borne out in figure 4 , with the consonant heat maps for the two languages displaying a clear diagonal stripe, and the vowel heat maps showing much more off-diagonal signals. what is more interesting is that the vowel errors in fact form clusters, as highlighted by white squares on the heat maps. the general pattern is that confusion only arises within a cluster where vowels are of similar quality but differ in terms of length or pitch accent. for example, while [i:] might be incorrectly-predicted as [i], our model does not confuse it with, say, @xcite . the challenges these languages present to the mod-els are therefore largely suprasegmental-vowel length and pitch accent, both of which are lexicalized and not explicitly marked in the orthography. for the other three languages, their errors also show distinct patterns: for adyghe, consonants differing only in secondary features can get confused; in greek, many errors can be attributed to the mixing of [r] and [r]; in italian, front and back mid vowels can trick our model. we hope that our detailed error analyses show not only that these errors "make linguistic sense"and therefore attest to the power of the modelbut also point out a pathway along which future modeling can be improved. ## conclusion this paper presented the approaches adopted by the ubc linguistics team to tackle the sigmor-phon 2021 grapheme-to-phoneme conversion challenge in the low-resource setting. our submissions build upon the baseline model with modifications inspired by syllable structure and vowel error patterns. while the first modification does not result in more accurate predictions, the second modification does lead to sizable improvements over the baseline results. subsequent error analyses reveal that the modified model indeed reduces erroneous vowel predictions for languages whose errors are dominated by vowel mismatches. our approaches also demonstrate that patterns uncovpredicted vowels slovene vowels χʷ χ ʔʷ ʔ ʒʷ ʒ ʑ ʐʷ ʐ z x w tʼ t͡ ʃʼ t͡ ʃ t͡ ʂ t͡ sʼ t͡ s t ʃʼ ʃʷʼ ʃʷ ʃ ʂʷ ʂ s ʁʷ ʁ r qʷ q pʼ pʷʼ p n m ɮ ɬʼ ɬ l kʼ kʷ kʲʼ kʲ j ħ ɣ ɡʷ ɡʲ f d͡ ʒ d͡ z d ɕ b b ɕ dd͡ z d͡ ʒf ɡʲ ɡʷɣ ħ j kʲkʲʼ kʷkʼ l ɬ ɬʼ ɮmn ppʷʼ pʼ qqʷr ʁʁʷs ʂʂʷ ʃ ʃʷʃʷʼʃʼ t t͡ st͡ sʼt͡ ʂ t͡ ʃ t͡ ʃʼ tʼ w x z ʐʐʷʑ ʒʒʷʔʔʷχχʷ ered from careful error analyses can inform the directions for potential improvements.