SIGMORPHON 2022 Shared Task on Grapheme-to-Phoneme Conversion Submission Description: Sequence Labelling for G 2 P (2023)

Abstract: This paper describes our participation in the Third SIGMORPHON Shared Task on Grapheme-to-Phoneme Conversion (Low-Resource and Cross-Lingual) (McCarthy et al.,2022). Our models rely on different sequence labelling methods. The main model predicts multiple phonemes from each grapheme and is trained using CTC loss (Graves et al., 2006). We find that sequence labelling methods yield worse performance than the baseline when enough data is available, but can still be used when very little data is available. Furthermore, we demonstrate that alignments learned by the sequence labelling models can be easily inspected.

## introduction this paper describes our participation in the third sigmorphon shared task on grapheme-to-phoneme conversion (low-resource and cross-lingual) @xcite . we evaluate 3 sequence labelling methods for graphemeto-phoneme conversion (henceforth: g2p). we approach the challenge of different lengths of grapheme and phoneme sequences by allowing to predict multiple phonemes from each grapheme. the shared task consists of 3 tracks and includes 10 languages. the 3 tracks are high resource, low resource, and transfer. for the high resource track, grapheme-phoneme pairs are given for 1000 words. for the low resource track, grapheme-phoneme pairs are given for 100 words. for the transfer track, grapheme-phoneme pairs are given for 100 words in the target language and additionally graphemephoneme pairs are given for 1000 words in a transfer language (that is related to the target language, e.g. @xmath0. the test set is the same for each track and contains 100 words of the target language. additionally, a development set is provided for each target language. the development set also is the same for each track. all of our models are applicable to all languages and tracks. sequence labelling approaches can claim several advantages over the main alternative, namely (neural) encoder-decoder approaches: sequence labelling does not require beam search for inference, may allow for smaller models, and defines a direct alignment between the input and predictions. the latter property may make models more interpretable and help with error analysis. however, sequence labelling is less flexible than encoderdecoder approaches and requires special handling of cases where the input and target sequences are of different length. ## related work common approaches to g2p are joint-ngram models @xcite , encoder-decoder models @xcite @xcite , and sequence labelling @xcite @xcite . in previous iterations of this shared task on g2p, encoder-decoder models were dominant both in terms of performance and in terms of number of submissions @xcite . while this shows that neural encoder-decoder models yield superior performance compared to joint-ngram models, little work has been done to evaluate the performance of neural sequence labelling models. therefore, two of our three proposed methods (explained in section 3) directly use or build on existing approaches, namely work by @xcite and @xcite . our third method has so far, to our knowledge, not been proposed for string transduction. it is however close to the approaches by @xcite and @xcite : both propose to augment the grapheme sequence by extra symbols, so that phoneme sequences that are longer than the grapheme sequence can be predicted. we propose to turn their approach upside-down and allow each grapheme to predict multiple phonemes, instead of optionally deleting unnecessarily added input symbols. however, in our current implementation, we predict a constant number of phonemes (which includes blank symbols) from each grapheme which is less flexible than the method proposed by @xcite , but avoids error propagation due to incorrectly predicted number of insertions. generally, no pure sequence labelling method can achieve the same flexibility as sequence-to-sequence models, but for some problems with strong local relationship between the input sequence and the target sequence, like g2p, sequence labelling may be sufficient. ## method we propose and evaluate 3 different sequence labelling methods. to refer to the different methods, we term them by their main inspiration: "supervised" (cf. @xcite ; @xcite ), "gram-ctc" (cf. @xcite ), and "inverse-scatter-ctc" (cf. @xcite ; @xcite ). the main challenge when applying sequence labelling methods to sequence transduction problems is finding a way to handle different lengths of the grapheme sequence and the phoneme sequence. the solution common to all our proposed methods is allowing to predict phoneme ngrams from grapheme unigrams and allowing to delete graphemes. since in g2p the length of phoneme sequences cannot differ arbitrarily from the respective grapheme sequences, predicting ngrams, which imposes a strict bound on the length of the predicted phoneme sequence, is still a realistic approach. in the following, we describe each method in more detail. ## models for sequence labelling, we always use a plain 1layer bilstm model. models are trained using the different approaches described in section 3. for the transfer track, we do not entirely mix the target language and the transfer language, but we share the same embeddings and lstm encoder for both languages and use separate classification layers, since we found this to yield better performance. our intuition is that transfer data stabilises training and mitigates overfitting of embeddings and the lstm encoder, but the target phonemes differ to a degree that makes separate decoding necessary. for each language and track, we train 10 models and keep the best 5 performing models in terms of wer on the development set. we use these 5 models to compute word-level majority-voted ensemble predictions. we resolve ties by choosing the prediction from the model with lowest wer on the development set among all predictions with most votes. the different approaches also require different hyperparameters. however, common to all setups are embedding size 64, no dropout or weight decay, vanilla sgd optimizer with one-cycle learning rate scheduler @xcite , and only keeping the best checkpoint from each training run based on wer on development set evaluated after every epoch. hyperparameters that differ are training for 100 epochs with batch size 2, max. learning rate 0.01, clipping gradients with absolute value greater than 1 and the lstm encoder having 128 hidden units for supervised and gram-ctc, whereas inverse-scatter-ctc models are trained for 80 epochs with batch size 16, max. learning rate 0.1, no gradient clipping, and 256 hidden units for the lstm encoder. all models and training routines are implemented in pytorch @xcite . ## results in however, for the transfer task, we make different observations: while gram-ctc performs worse in the transfer setting than in the low resource setting, supervised training is able to use the additional transfer data in order to improve upon its performance in the low resource setting. the improvement amounts to approximately 7 points in wer (macro-average). therefore, transfer data can indeed be very helpful when used with the right kind of model. this being said, gram-ctc still outperforms supervised training in 6 out of 10 languages (transfer track). the fact that the macro-average wer of the supervised model is eventually lower is mainly due to the much better performance of supervised training on tagalog (24 vs 50). if we ignore tagalog when calculating macro-average scores, gram-ctc (both low and transfer tracks) and supervised training (transfer track) perform almost equally well, with a slight advantage for gram-ctc in transfer setting. in depending on τ , performance of inverse-scatter-ctc can be slightly better than performance of gram-ctc, but we do not observe any decisive advantages. we demonstrate this in figure 1 : the numbers in the heatmap show for how many languages the model on the x-axis achieves strictly lower wer than the model on the y-axis. despite having the lowest macro-average, supervised training actually is not superior to any model for more than 50% of the languages. contrarily, inverse-scatter-ctc with τ =4 achieves better performance than most models for more than 50% of the languages, but has second-worst macroaveraged wer. overall, figure 1 shows that which model is best is language dependent and there is no clear winner among the models evaluated in this paper. similar results can be found also for the high and the low resource tracks. compared to the baseline 1 , our models generally perform worse in the high-resource tack, but better in the low resource and transfer tracks. this suggests that sequence-to-sequence models may be superior to sequence labelling models when enough data is available, while it is still possible to train neural (sequence labelling) models in the ultra-low resource settings. ## model inspection in the introduction, we claim that one advantage of sequence labelling methods is that they define a direct alignment between graphemes and phonemes, which can be inspected. therefore, in in german, this is a rather rare phenomenon, occurring only for the grapheme "x", which is pronounced as "k s", and also for the glottal stop p when words start with a vowel. for example, the pronunciation of german "axt" (english: "axe, ax") is predicted as "a k t", while "a k s t" is correct. one possibility of how to make use of these direct alignments is using predefined mappings of graphemes to phonemes to restrict which phonemes may be predicted. another advantage, of course, is error analysis. for the german model, for example, we would recommend adding more examples containing "x" to the training set. in fact, there is only one such example in the high resource training data, namely "existieren" (english: "to exist"). ## conclusion we presented and evaluated 3 sequence labelling methods for g2p: supervised, gram-ctc, and inverse-scatter-ctc. we show that all 3 methods can be applied to all 3 tracks, but no method seems clearly superior to the other methods. in the high resource setting, the baseline sequence-tosequence model seems to yield better performance than sequence labelling methods. however, sequence labelling methods seem to perform better in the very low resource settings. finally, we show that inverse-scatter-ctc models learn reasonable alignments of graphemes and phonemes, thereby validating the claim that sequence labelling models allow for comparatively easy model inspection.