The SIGMORPHON 2022 Shared Task on Cross-lingual and Low-Resource Grapheme-to-Phoneme Conversion (2023)

Abstract: Grapheme-to-phoneme conversion is an important component in many speech technologies, but until recently there were no multilingual benchmarks for this task. The third iteration of the SIGMORPHON shared task on multilingual grapheme-to-phoneme conversion features many improvements from the previous year’s task (Ashby et al., 2021), including additional languages, three subtasks varying the amount of available resources, extensive quality assurance procedures, and automated error analyses. Three teams submitted a total of fifteen systems, at best achieving relative reductions of word error rate of 14% in the crosslingual subtask and 14% in the very-low resource subtask. The generally consistent result is that cross-lingual transfer substantially helps grapheme-to-phoneme modeling, but not to the same degree as in-language examples.

## introduction many speech technologies demand mappings between written words and their pronunciations. in open-vocabulary systems, as well as certain resource-constrained embedded systems, it is insufficient to simply list all possible pronunciations; these mappings must generalize to rare or unseen words as well. therefore, the mapping must be expressed as a mapping from a sequence of orthographic characters-graphemes-to a sequence of sounds-phones or phonemes. 1 grapheme-to-phoneme (g2p) datasets vary in size across languages @xcite . in low-resource scenarios, an effective way of "breaking the resource bottleneck" @xcite is cross-lingual transfer of information from a highresource language, either by annotation projection 1 we note that referring to elements of transcriptions as phonemes implies an ontological commitment which may or may not be justified; see @xcite ) for discussion. therefore, we use the term phone to refer to symbols used to transcribe pronunciations. ( @xcite or adapting a model to a new language @xcite @xcite @xcite . the intent is that either the data or the learned representations and parameters carry across languages. cross-lingual transfer shows promise for grapheme-to-phoneme conversion @xcite . since this shared task began, zeroshot grapheme-to-phoneme procedures have been proposed, using no examples in the language of interest @xcite . sigmorphon in 2020 and 2021 hosted shared tasks on grapheme-to-phoneme conversion @xcite . the tasks have drawn wide participation, and in both years the participants outperformed the baseline systems by respectable margins. a major finding of the most recent iteration @xcite is that the largest improvements came from data augmentation, rather than alterations of the core model. consequently, we have proposed a third edition of the shared task that explores data efficiency and language relatedness through cross-linguality. this year's subtasks are designed so that, by contrasting these two aspects, we can answer two questions about data efficiency: this year, we study 10 language pairs, including two surprise pairs which were not released to the participants until close to the deadline as a challenge. each pair of languages shares a script and some other relationship (e.g., phylogeny or hegemony). we investigate three data settings: foot_0 cross-lingual transfer a small amount of data (100 words) in the language of interest (the "target language") and a large amount of data (1000 words) in a nearby language (the "transfer language"). very-low resource a small amount of data ( 100 words) in the target language and no data in the transfer language. low resource a large amount of data ( 1000 words) in the target language and no data in the transfer language. in every case, we use the same 100-word test set, providing only the graphemes to the participants. because the language pairs are consistent across the subtasks, we can draw meaningful contrasts. altogether, 15 systems were submitted, which allow substantial insights into our questions about data efficiency and g2p modelability. this third iteration of the sigmorphon shared task on graphme-to-phoneme conversion introduces transfer languages, new target languages, surprise languages, and stringent quality assurance, as the subtask structure which enables comparison. ## data as in the two previous years, all pronunciation data was drawn from wikipron @xcite , a massively multilingual pronunciation database extracted from the online dictionary wiktionary. depending on the language and script, wiktionary pronunciations are either manually entered by human volunteers working from language-specific pronunciation guidelines or generated from the graphemic form via language-specific server-side scripting. wikipron scrapes these pronunciations from wiktionary, optionally applying casefolding to the graphemic form, removing any target language transfer language swedish norwegian nynorsk german dutch italian romanian ukrainian belarusian tagalog cebuano bengali assamese persian pashto thai eastern lawa irish welsh burmese shan morphological information from the unimorph morphological lexicons @xcite were again provided to participants; however, no participant made use of these, just like last year. language selection while the 2021 shared task considered both high-resource and low-resource settings, we did not control for the language itself. it was hard to extrapolate from the scores to claims about the resource requirements and difficulties of particular languages. this year, we use the same languages in all settings. this makes it reasonable and appropriate for the results to be directly compared, answering the two questions from section 1. these languages were chosen to avoid particularly pathological languages noted in previous years (english, croatian) and those with unique and hard-to-predict phenomena, like stød in danish. ## task definition in this task, participants were provided with a collection of words and their pronunciations, and then scored on their ability to predict the pronunciation of a set of unseen words. ## evaluation the primary metric for this task was word error rate (wer), the percentage of words for which the hypothesized transcription sequence is not identical to the gold reference transcription. as all three subtasks involve multiple languages, macro-averaged wer was used for system ranking. participants were provided with two evaluation scripts: one which computes wer for a single language, and one which also computes macroaveraged wer across two or more languages. the 2020 shared task also reported another metric, phone error rate (per), but this was found in the 2021 shared task to be highly correlated with wer and was not reported. ## baseline the baseline system from 2021, the monotonic hard attention system from cluzh @xcite , remained the baseline architecture in 2022. it is is a neural transducer system using an imitation learning paradigm @xcite . all models were tuned to minimize perlanguage development-set wer. we reuse the best hyperparameter settings from last year. alignments are computed using ten iterations of expectation maximization, and the imitation learning policy is trained for up to sixty epochs (with a patience of twelve) using the adadelta optimizer. a beam of size of four is used for prediction. final predictions are produced by a majority-vote tencomponent ensemble. internal processing uses the decomposed unicode normalization form (nfd), but predictions are converted back to the composed form (nfc). an implementation of the baseline was provided during the task and participating teams were encouraged to adapt it for their submissions. in many cases, the baseline's loss did not improve over the course of training. we indicate this with a '-' in tables 2 to 4 . ## submissions the shared task received 15 submissions from 3 teams. below we provide brief descriptions of sub-missions to the shared task; more detailed descriptions of the first two-as well as various exploratory analyses and post-submission experiments-can be found in the system papers later in this volume. tü-g2p girrbach (2022) evaluated three sequence labeling approaches to grapheme-tophoneme conversion. in the supervised case, girrbach trained a bilstm model to predict phoneme n-grams. the labels are derived from external alignments calculated by a custom neural aligner. second, girrbach trained a gram-ctc model @xcite to jointly predict and realign phoneme n-grams. finally, the main approach is to use a standard bilstm sequence labeling model, but predict multiple (τ ∈ {3, 4, 5}) phoneme unigrams from each grapheme. girrbach uses standard ctc @xcite to train the model, which is possible because predicting multiple phonemes from each grapheme causes the number of predicted symbols to always be greater than the number of target phonemes. note that using ctc avoids relying on external alignments in any way. for the transfer task, girrbach shares the same grapheme embeddings and bilstm encoder between target and transfer language, but uses different prediction layers. hammond hammond (2022) submitted one system. he initially built a transformer-based system, but because data are so minimal, it performed poorly. he switched to an hmm-based system @xcite . for the transfer condition, which was his priority, he used the provided transfer data and augmented the system in two ways. first, he used a simplified version of the splicing augmentation scheme developed by @xcite for the core data. second, for the transfer languages, he only used data where the phonologies overlapped at the bigram level; in other words, he only included transfer training pairs that only included phonetic bigrams that occurred in the target languages. they took an explicitly multilingual approach to modeling the g2p tasks, fine-tuning and evaluating a single model that covered all languages in the task. having a single model for all languages made it necessary to tell the model, for each input, which language it was generating the pronunciation for, which was accomplished by prefixing each input string with the language's three-letter code (followed by a single space). @xcite ## results overall, teams were able to outperform the baseline in the cross-lingual and very-low resource settings, at best achieving relative reductions of word error rate of 14% in the cross-lingual subtask and 14% in the very-low resource subtask. the best results for each setting are given in tables 2 to 4 . non-neural approaches like hmms with data augmentation were particularly successful in regimes where transformer models often founder, mirroring findings in machine translation and morphological inflection @xcite . ## discussion we once again see an enormous difference in language difficulty. in particular, hammond (2022) provides examples from the welsh/irish language pair to suggest that phylogenic or hegemonic similarity of languages does not entail similarity of orthography and phonology. moreover, phoneme oovs were a problem in the very-low resource setting: many phonemes and phenomena were simply not observed in 100 randomly sampled examples. this suggest room for typological information to improve modeling. as mentioned above, the data here are a mixture of broad and narrow transcriptions. at first glance, this might explain some of the variation in language difficulty; for example, it is easy to imagine that the additional details in narrow transcriptions make them more difficult to predict. however, for many languages, only one of the two levels of transcription is available at scale, and other languages, divergence between broad and narrow transcriptions is impressionistically quite minor, as asserted in @xcite . however, this impression ought to be quantified. the inclusion of the very-low resource subtask is intended to be a challenging case for participants; however, we did not anticipate the degree to which it would be challenging. in many cases, the baseline and participants' systems achieve a word error rate of zero or one. clearly, there is room for improvement in minimally supervised graphemeto-phoneme conversion. participants were permitted in all three subtasks to make use of lemmas and morphological tags from unimorph as additional features. however, no team made use of these resources. some prior work (e.g., @xcite has found morphological tags highly useful, and ashby et al. (2021) suggests this information would make an impact in french. the results of the shared task suggest several next steps for carrying out a g2p shared task: ## conclusion the third iteration of the shared task on multilingual grapheme-to-phoneme conversion is structured to provide answers to questions about the value of cross-lingual transfer and data availability. three teams submitted fifteen systems, achieving substantial reductions in both absolute and relative error over the baseline in two of three subtasks. we hope the code and data, released under permissive licenses, foot_3 will be used to benchmark grapheme-to-phoneme conversion and sequence-to-sequence modeling techniques more generally-especially in challenging low-resource scenarios.