Zero-shot Learning for Grapheme to Phoneme Conversion with Language Ensemble (2022)

Abstract: Grapheme-to-Phoneme (G2P) has many applications in NLP and speech fields. Most existing work focuses heavily on languages with abundant training datasets, which limits the scope of target languages to less than 100 languages. This work attempts to apply zero-shot learning to approximate G2P models for all low-resource and endangered languages in Glottolog (about 8k languages). For any unseen target language, we first build the phylogenetic tree (i.e. language family tree) to identify top- k nearest languages for which we have training sets. Then we run models of those languages to obtain a hypothesis set, which we combine into a confusion network to propose a most likely hypothesis as an approximation to the target language. We test our approach on over 600 unseen languages and demonstrate it significantly outperforms baselines.

## introduction grapheme-to-phoneme (g2p) plays a crucial role in many nlp tasks. in particular, it is used heavily in many speech-related tasks such as speech recognition and speech synthesis @xcite . even in the latest end-to-end systems, it still has a strong impact on the speech performance @xcite . typically, the g2p task is language-dependent-many languagespecific factors affect the g2p process such as the general characteristics of scripts @xcite , phonotactic constraints @xcite and other orthography factors @xcite . for example, in in our experiments, we build a large dataset from wiktionary in which we use 260 languages as the training languages and test our approach on 600 unseen languages. we apply our approach to 3 different architectures: a joint-sequence n-gram model @xcite , an lstm sequence-to-sequence model @xcite , and a transformer-based sequence-to-sequence model @xcite . using any of the architectures, our approach outperforms all baselines by more than 5% per (phoneme error rate). the main contributions of this work are as follows: ## related work traditionally, a g2p component is built using rulebased models. for example, the phonological constraints can be incorporated into context-sensitive grammars and implemented using finite-state transducers @xcite . however, designing the rules requires many hours from linguists and can be prohibitive for low-resource languages if they have deep orthographies 2 . statistical models overcome this problem by learning the rules automatically. typically, there are two steps in building such a model: first, the 2 orthographies in which the relationship between graphemes and phonemes has been obscured by history or is otherwise complicated. sequence of phonemes and graphemes are aligned to each other, then another prediction model is built on top of the alignment. the alignment model is typically done using expectation and maximization @xcite . the prediction model can be done using neural networks @xcite , decision trees @xcite , joint-sequence models @xcite and wfst-based n-gram models @xcite . more recently, deep neural networks have been applied to the g2p task. various architectures have been explored, for example, rnns @xcite @xcite , cnns @xcite and transformers @xcite . traditionally, each g2p model was typically built for one high-resource language. recently, many researchers have started to focus on lowresource g2p models. one related work adapts high-resource language models to low-resource language models by measuring similarity between languages and phonemes @xcite . this previous work creates a new training set for every low-resource language by adapting the training set from the top-3 nearest languages. however, there are several issues with this approach. first, it has to prepare separate training sets and n-gram models for every testing language, which is quite computationally expensive. it also suffers from the limited training set problem even after merging top-resource languages. next, we define the language similarity and language families. finally, we explain how to ensemble nearest languages models to predict g2p for an unseen language. ## experiments in this section, we show the experiment results on our g2p models. first, we introduce the main datasets we used to build our model, next we describe our baseline models and g2p architectures we use in our experiments. finally, we demonstrate that the proposed ensemble approach outperforms algorithm 1: g2p algorithm data: input, lang (grapheme sequence and its language) result: output (ensembled phoneme sequence) klangs ← kn earestlanguage(lang) hyps ← [] for @xmath0, klang) ; / * generate hypothesis for every nearest language * / hyps.append(hyp) end x ← conf usionn etwork() for @xmath1, n) x ← composite(x, n, a) end output ← [] for @xmath2; / * vote 1 phoneme per confusion set * / output.append(p) end those baseline models in different architectures. ## limitations while we get reasonable performance in our testing languages, we acknowledge that there are several limitations in our approach: first, both of our training languages and testing languages are limited to languages available in wiktionary. the full glottolog phylogenetic tree has 110 top-level branches in total, however, our dataset only spans 40 branches. therefore if we want to apply our approach to unseen languages in the remaining 70 branches, we have to depend on unrelated languages to build our ensemble model, which might lead to worse performance. second, as our approach heavily depends on glottolog and wiktionary, if the language is not available in the glottolog database or the vocabulary quality in wiktionary is not good enough, then our approach cannot be applied to it. finally, many of the 8k languages do not have orthographies, therefore it might be difficult or meaningless to evaluate the g2p performance for them. ## conclusion in this work, we propose a zero-shot learning method to approximate g2p models for 8k languages in the world. we use the phylogenetic tree to measure the distance between languages and combine multilingual outputs. we test our approach on 600 unseen languages and demonstrate it significantly outperforms baselines. we hope the proposed model can be used in many speech tasks such as phone recognition for low resource languages @xcite . we will release our datasets and models for 8k languages to allow more researchers to explore this direction. 8