CLUZH at SIGMORPHON 2021 Shared Task on Multilingual Grapheme-to-Phoneme Conversion: Variations on a Baseline (2021)

Abstract: This paper describes the submission by the team from the Department of Computational Linguistics, Zurich University, to the Multilingual Grapheme-to-Phoneme Conversion (G2P) Task 1 of the SIGMORPHON 2021 challenge in the low and medium settings. The submission is a variation of our 2020 G2P system, which serves as the baseline for this year’s challenge. The system is a neural transducer that operates over explicit edit actions and is trained with imitation learning. For this challenge, we experimented with the following changes: a) emitting phoneme segments instead of single character phonemes, b) input character dropout, c) a mogrifier LSTM decoder (Melis et al., 2019), d) enriching the decoder input with the currently attended input character, e) parallel BiLSTM encoders, and f) an adaptive batch size scheduler. In the low setting, our best ensemble improved over the baseline, however, in the medium setting, the baseline was stronger on average, although for certain languages improvements could be observed.

## introduction the sigmorphon grapheme-to-phoneme conversion task consists of mapping a sequence of characters in some language into a sequence of whitespace delimited international phonetic alphabet (ipa) symbols, which represent the pronunciation of this input character sequence (not necessarily a phonemic transcription, despite the name of the task) according to the language-specific conventions used in the english wiktionary. 1 the data was collected and post-processed by the wikipron project @xcite . post-processing removes stress and syllable markers and applies ipa segmentation for combining and modifier diacritics as 1 https://en.wiktionary.org/ lang. grapheme phoneme wiktionary ice persóna pʰ ɛ r̥ s o uː n a /ˈpʰɛr̥ .souːna/ fra williams w i l j a m z /wi.ljamz/ bul засичайки z ɐ s t͡ ʃ ʃ ə j kʲ i /zɐˈsitʃəjkʲi/ kor 검출 k ɘː m t͡ ɕʰ u ɭ [ˈkʌ̹ (ː)mt͡ ɕʰuɭ] well as contour information. see figure 1 for the post-processed shared task entries and the original entries from the wiktionary pronunciation section. for more information, we refer the reader to the shared task overview paper @xcite . in the low and medium data setting, the 2021 sigmorphon multilingual g2p challenge features ten different languages from various phylogenetic families and written in different scripts. the low setting comes with 800 training, 100 development and 100 test examples. in the medium setting, the data splits are 10 times larger. although it is permitted to use external resources for the medium setting, all our models used exclusively the official training material. our system is a neural transducer with pointer network-like monotonic hard attention @xcite that operates over explicit character edit actions and is trained with imitation learning @xcite @xcite . it is an adaptation of our type-level morphological inflection generation system that proved its data efficiency and performance in the sigmorphon 2018 shared task @xcite . g2p shares many similarities with traditional morphological string transduction: the changes are mostly local and often simple depending on how close the spelling of a language reflects pronunciation. for most languages, a substantial part of the work is actually p(#) σ : / p(del(σ)) ## model description the model defines a conditional distribution over substitution, insertion and deletion edits p θ (a | x) = |a| @xmath0, x), where @xmath1. . . x |x| is an input sequence of graphemes and @xmath2. . . a |a| is an edit action sequence. the output sequence of ipa symbols y is deterministically computed from x and a. the model is equipped with an lstm decoder and a bidirectional lstm encoder @xcite . at each decoding step j, the model attends to a single grapheme x i . the attention steps monotonically through the input sequence, steered by the edits that consume input (e.g. a deletion shifts the attention to the next grapheme x i+1 ). the imitation learning algorithm relies on an expert policy for suggesting intuitive and appropriate character substitution, insertion and deletion actions. for instance, for the data sample кит → /k j it/ (russian: "whale"), we would like the following most natural edit sequence to attain the lowest cost: subs @xcite on the training data, which is a memoryless weighted finite-state transducer shown in figure 2 . the resulting sed model is integrated into the expert policy, the sed policy, that uses viterbi decoding to compute optimal edit action sequences for any point in the action search space: given a transducer configuration of partially processed input, find the best edit actions to generate the remaining target sequence suffix. during training, an aggressive exploration schedule p sampling (i) = 1 1+exp(i) where i is the training epoch number, exposes the model to configurations sampled by executing edit actions from the model. for an extended description of the sed policy and il training, we refer the reader to the last year's system description paper @xcite . ## submission details modifications such as mogrifier lstms, additional input character skip connections, or parallel encoders increase the number of model parameters and make it difficult to compare the baseline system directly with its variants. additionally, we did not have enough time before the submission to systematically explore and fine-tune for the best combination of model modifications and hyper-parameters. in the end, after some light experimentation we had to stick to settings that might not be optimal. we train separate models for each language on the official training data and use the development set exclusively for model selection. as beam decoding for mogrifier models sometimes suffered compared to greedy decoding, we built all ensembles from greedy model prediction. like the baseline system (b), we train the sed model for 10 epochs, use one-layer lstms, hidden state dimension 200 for the decoder lstms and action embedding dimension 100. for the low (l) and medium (m) setting, we have the following specific hyperparameters: we submit 3 ensemble runs for the low setting: cluzh-1: 15 models with char input, cluzh-2: 15 models with seg input, cluzh-3: 30 models with char or seg input. we submit 4 ensemble runs for the medium setting: cluzh-4: 5 models with char input, cluzh-5: 10 models with seg input, cluzh-6: 5 models with seg input, cluzh-7: 15 models with char or seg input. due to a configuration error, medium results were actually computed without two add-ons: mogrifier lstms and the additional input character. in postsubmission experiments, we computed runs that enabled these features and report their results as well (cluzh-4m/5m). ## results and discussion in a post-submission experiment on the high setting, we built a large 10 5-strong seg-based ensem-9 three parallel encoders with 200 hidden units each; character embedding dimension of 200; no mogrifier; no input character added to the decoder. 10 character embedding dimension: 200; action embedding dimension: 100; 10 parallel encoders with hidden state dimen-ble. it achieves an impressive low word error rate of 38.7 compared to the official baseline (41.94) and the best other submission (37.43). future work: performance variance between different runs of our lstm-based architecture makes it difficult to reliably assess the actual usefulness of the small architectural changes; extensive experimentation, e.g. in the spirit of @xcite , is needed for that. one should also investigate the impact of the official data set splits: the observed differences between the development set and test set performance in the low sion 100; decoder hidden state dimension: 500; minimal batch size: 5; maximal batch size: 20; epochs: 200 (subsampled to 3,000 items); patience: 24; no mogrifier; no input character added to the decoder. setting for slovene or greek are extreme. crossvalidation experiments might help assess the true difficulty of the wikipron datasets. ## conclusion this paper presents the approach taken by the cluzh team to solving the sigmorphon 2021 multilingual grapheme-to-phoneme conversion challenge. our submission for the low and medium settings is based on our successful sigmorphon 2020 system, which is a majority-vote ensemble of neural transducers trained with imitation learning. we add several modifications to the existing lstm architecture and experiment with ipa segment vs. ipa character action predictions. for the low setting languages, our ipa character-based run outperforms the baseline and ranks second overall. the average performance of segment-based action edits suffers from performance outliers for certain languages. for the medium setting languages, we note small improvements on some languages, but the overall performance is lower than the baseline. using a mogrifier lstm decoder and enriching the encoder input with the currently attended input character did not improve performance in the medium setting. post-submission experiments suggest that network capacity for the submitted systems was too small. a post-submission run for the high-setting shows considerable improvement over the baseline.