CLUZH at SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion (2020)

Abstract: This paper describes the submission by the team from the Institute of Computational Linguistics, Zurich University, to the Multilingual Grapheme-to-Phoneme Conversion (G2P) Task of the SIGMORPHON 2020 challenge. The submission adapts our system from the 2018 edition of the SIGMORPHON shared task. Our system is a neural transducer that operates over explicit edit actions and is trained with imitation learning. It is well-suited for morphological string transduction partly because it exploits the fact that the input and output character alphabets overlap. The challenge posed by G2P has been to adapt the model and the training procedure to work with disjoint alphabets. We adapt the model to use substitution edits and train it with a weighted finite-state transducer acting as the expert policy. An ensemble of such models produces competitive results on G2P. Our submission ranks second out of 23 submissions by a total of nine teams.

## introduction g2p requires mapping a sequence of characters in some language into a sequence of international phonetic alphabet (ipa) symbols, which represent the pronunciation of this input character sequence in some abstract way (not necessarily phonemic, despite the name of the task) (figure 1 ). multilingual g2p is task i of this year's sig-morphon challenge. it features fifteen languages from various phylogenetic families and written in different scripts. we refer the reader to @xcite for an overview of the language data. each language comes with 3,600 training and 450 development set examples. it is permitted to use external resources as well as to build a single multilingual model. we participate in this shared task with an adaptation of our sigmorphon 2018 system (makarov @xmath0:/ ("giants") irish of cois fhairrge @xcite figure 1 : example of g2p. and clematide, 2018b), which was particularly successful in type-level morphological inflection generation. our system is a neural transducer that operates over explicit edit actions and is trained with imitation learning @xcite @xcite . it has a number of useful inductive biases, one of which is the familiar bias towards copying the input (implemented as the traditional copy edit). this is particularly useful for morphological string transduction problems, which typically involve small and local edits and where most of the input is preserved in the output. this contrasts with models that rely purely on generating characters such as generic encoder-decoder models, which as a result suffer, particularly on smaller-sized datasets. copying requires that the input and output character alphabets overlap, preferably substantially. this also allows our il training to leverage a simple-to-implement expert policy (which during training provides demonstrations to the learner of how to optimally solve the task). the optimal completion of the target given the prediction generated so far during training requires finding edits that would extend the prediction so that the levenshtein distance @xcite between the target and the partial prediction + the future suffix is minimized. unfortunately, this objective alone would not discriminate between multiple edit action sequences that relate the input and the partial prediction + the future suffix. to address this spurious ambiguity, our il training adds edit sequence scores, computed using traditional costs, 1 into the objective. this naturally encourages the system to copy, however this would fail on any editing problem with disjoint alphabets. g2p poses an interesting challenge for a system like ours. on the one hand, g2p shares many similarities with morphological string transduction: the changes are mostly local, it would suffice to perform traditional left-to-right transduction, and a substantial part of the work is arguably applying equivalence rules (e.g. the german letter "g" most often converts to /g/, "a" to /a/ or /a:/), which is similar to copying. yet, a general solution to g2p cannot rely on overlapping alphabets since many scripts do not share many symbols, if any at all, with ipa (e.g. korean or georgian). our solution adapts the model to use substitution edits and trains it with a weighted finite-state transducer acting as the expert policy. ## model description the underlying model is a neural transducer introduced in @xcite . it defines a conditional distribution over traditional edits p θ (y, a | x) = |a| @xmath1, x), where x is an input sequence of graphemes and @xmath2. . . a |a| is an edit action sequence. (the output sequence of ipa symbols y is deterministically computed from x and a.) the model is equipped with a long short-term memory (lstm) decoder and a bidirectional lstm encoder @xcite . the challenge is training this model: due to the recurrent decoder, it cannot be trained with exact marginal likelihood unlike the more familiar weighted finite-state transducer @xcite or its neuralizations @xcite . for a more detailed description of the model, we refer the reader to makarov and clematide (2018a). foot_1 il training makarov and clematide (2018a) propose training the model using il, a general model fitting framework for sequential problems over exponentially sized output spaces. il has been applied successfully to natural language processing (nlp) problems, e.g. transition-based parsing @xcite and language generation @xcite . il relies on the availability of demonstrations of how the task can optimally p(#) σ : / p(del(σ)) the second part is crucial for training accurate models especially in the limited resource setting, as it reduces spurious ambiguity arising under the first part of the objective alone. it is also the second part of the training objective that hinges on the overlap of the input and output alphabets, as this permits minimization using the edit distance dynamic program with traditional costs. ## submission details we train separate models for each language on the official training data and use the development set for model selection. 4 our submission does not use any additional lexical resources. for most of the models, we employ unicode decomposition normalization (nfkd) foot_4 as a data preprocessing step. importantly, this helps decomposing unicode syllable blocks used e.g. in hangul. the size of the development set is rather small (450 examples), and having examined the data, we suspect that overly relying on the development set for model selection might hurt generalization. for example, the french development set contains three exceptions to the "ill"-/j/ equivalence; thus, a single model that achieves a high score on the development set might, in fact, be overfitting. to counter this, we build an eleven-model-strong majorityvote ensemble. fortunately, training a neural transducer is fast as one epoch takes just about four minutes on average on a single cpu, due to the relatively small number of model parameters. 4 results and discussion our system ranks second among 23 submissions by a total of nine teams ( error analysis we look in some detail at the errors on the korean test data that all or almost all of the individual models of the ensemble make. as expected, lexicalized phenomena contribute most of the errors: vowel length (which is neither phonemic nor phonetically realized in the speech of all except elderly speakers @xcite ) and tensification. vowel length is not indicated in korean orthography, and neither is tensification (with some exceptions). knowing whether a word is an english borrowing (e.g. 섹스 seksȗ foot_5 (sex)) or whether a word is a compound and where the morpheme boundary lies (초승달 ch'osȗng-tal (new moon)) could help predict non-automatic tensification correctly in a small number of cases ([s "" e fl ks "" w] vs * [s h ...] and [ > t c h o fl s h wnt "" aí] vs * [...daí]). computed using the utf-8 aware version of the isri analytic tools for ocr evaluation. foot_6 how good is sed policy? somewhat surprisingly, using sed as part of the expert policy results in competitive performance. yet, sed is a very crude model (e.g. because of the lack of context, when used as a conditional model, sed assigns less probability to any edit sequence containing insertions than the same sequence but with all the insertions removed; this e.g. makes it unusable as a standalone model for g2p). on top of this, we also do not use learned roll-out, which would be recommended when training with a sub-optimal expert @xcite . we leave it for future work to examine whether the neural transducer's performance on g2p would improve from replacing sed with a more powerful model. ## conclusion this presents the approach taken by the cluzh team to solving the sigmorphon 2020 multilingual grapheme-to-morpheme conversion challenge. our submission is based on our successful sigmorphon 2018 system, which is a majorityvote ensemble of neural transducers trained with imitation learning. we adapt the 2018 system to work on transduction problems with disjoint input and output alphabets. we add substitution actions (not available in previous versions of the system) and employ a memoryless probabilistic finite-state transducer to define the expert policy for the imitation learning. we use majority-vote ensembling to counter the overfitting to the small development sets. these simple modifications result in a highly competitive performance even without the use of any exernal resources or learning a single multilingual model. our ensemble ranks second out of 23 submissions by a total of nine teams. our error analysis indicates that addressing many of the errors requires additional information such as knowing the word's lexical category, morphological segmentation, or etymology. we will make our code publicly available.