The SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion (2020)

Abstract: We describe the design and findings of the SIGMORPHON 2020 shared task on multilingual grapheme-to-phoneme conversion. Participants were asked to submit systems which take in a sequence of graphemes in a given language as input, then output a sequence of phonemes representing the pronunciation of that grapheme sequence. Nine teams submitted a total of 23 systems, at best achieving a 18% relative reduction in word error rate (macro-averaged over languages), versus strong neural sequence-to-sequence baselines. To facilitate error analysis, we publicly release the complete outputs for all systems—a first for the SIGMORPHON workshop.

## introduction speech technologies such as automatic speech recognition and text-to-speech synthesis require mappings between written words and their pronunciations. even recent attempts to do away with explicit pronunciation models via "end-to-end" systems (e.g., @xcite @xcite @xcite ) must induce an implicit mapping of this sort. for open-vocabulary applications, these mappings must generalize to unseen words, and so must be expressed as mappings between sequences of graphemes-i.e., glyphsand phonemes or phones-i.e., sounds. 1 for some languages, this mapping is sufficiently consistent that a literate, linguisticallysophisticated speaker can simply enumerate the necessary rules; this sequence of rules can then be compiled into a finite-state transducer (e.g., @xcite . however, rulebased systems require linguistic expertise to develop and maintain, and may be brittle or inaccurate. therefore, modern speech engines usually treat grapheme-to-phoneme conversion as a machine learning problem, either using generative models expressed as weighted finite-state transducers (e.g., @xcite @xcite or discriminative models based on conditional random fields @xcite , recurrent neural networks (e.g., @xcite @xcite or transformers @xcite . while the grapheme-to-phoneme conversion (or g2p) task is crucial to speech technology, the vast majority of published research focuses on english or a few other highly-resourced, globally hegemonic languages for which free pronunciation dictionaries are available. one exception, a recent study by @xcite , compares naïve rule-based systems and neural network-based sequence-to-sequence models for 20 languages; unfortunately, the data used in this study is proprietary. like many other types of language resources, pronunciation dictionaries are expensive to create and maintain, and until recently, free high-quality dictionaries were only available for a small number of languages. this limitation to a handful of languages is unfortunate because, as we discuss below, writing systems are almost as diverse as languages themselves. therefore, we present a multilingual grapheme-to-phoneme conversion task with data sets, evaluation metrics, and strong baselines. in this we are aided by the recent release of wikipron @xcite , a freely available collection of pronunciation dictionaries. the resulting task, the first of its kind, included data from fifteen languages and scripts, and received 23 submissions from nine teams. ## data fifteen language/script pairs were chosen to cover a wide variety of script types. ten of the scripts are alphabetic systems known to descend from phoenician (and ultimately from egyptian hieroglyphs); of these, seven are variants of the latin script. two others, the armenian aybuben and the georgian mkhedruli, are alphabetic scripts of unknown origin, but may ultimately be modeled on greek @xcite . the devanāgarī script used to write hindi, is an alphasyllabary, in which most glyphs-known traditionally as akṣara-denote consonant and consonant-vowel sequences. vowels (or their absence) are primarily indicated with diacritics. it too is thought to ultimately descend from phoenician. hiragana, one of several scripts used to write japanese, is a syllabary, in which most glyphs denote entire syllables the glyphs themselves are derived from chinese characters. like hiragana, the korean hangul script is also a syllabary it may have been have been inspired by 'phags-pa, a tibetan alphabet which is itself a distant cousin of devanāgarī @xcite . it is important to note that languages-and the scripts used to write them-differ enormously in their affordances for grapheme-to-phoneme conversion. writing systems are, at their core, linguistic analyses, albeit sometimes quite naïve, and (as argued in defrancis 1989) explicitly encode details of the phonological and phonetic structure of the language they are used to write. still, the exact details of these mappings can vary greatly between even closely related languages and/or scripts. whereas related languages may retain telltale grammatical features across millennia, dozens of languages have abruptly switched from one script to another in just the last century, usually in response to political-rather than linguisticconcerns. it is thus unsurprising that @xcite find grapheme embeddings induced by training g2p systems are poorly correlated with gross phonological typology, and experiments with "polyglot" g2p models (e.g., @xcite have produced equivocal results. while we did not pay particular attention to language families when selecting language family, we note that nine of the languages are indo-european (though no two are closely related) and none of the remaining six-adyghe, georgian, hungarian, japanese, korean, and vietnameseare known to be genetically related to each other. ## methods the primary data for the shared task is derived from wikipron @xcite ), a massively multilingual resource of grapheme-phoneme pairs extracted from wiktionary, an online multilingual dictionary. depending on language and script, these pronunciations may be manually entered by human volunteers-usually working from language-specific pronunciation guidelines-or generated using server-side scripting routines; some languages (e.g., bulgarian and french) use a mixture of the two approaches. wikipron is configured to apply case-folding where appropriate. it removes stress and syllable boundary markers and segments pronunciation strings-encoded in the international phonetic alphabet-using the segments library @xcite . for this task, words with multiple pronunciations-both homographs and free pronunciation variants-were excluded, since pronunciations for such words are often selected by a rather different procedure: they are chosen from a small, predetermined set of possible pronunciations using classifiers conditioned on local context (e.g., @xcite . training and development data for ten languages-the "development" languages-was released at the start of the task; equivalent data for the five "surprise" languages was released one week before the start the evaluation phase. as there is considerable variation in the number of available examples for any given language, each languages' data was downsampled to 4,500 examples. we regard as a "medium-resource" setting for this task; these data sets are, for instance, several orders of magnitude smaller than the proprietary g2p data used by @xcite . following similar procedures in other shared tasks (e.g., @xcite , words were sampled according to their frequency in the largest available wortschatz @xcite corpus for that language. these frequencies were smoothed by adding a 0. was not available for adyghe, so uniform sampling was used for this language. the downsampled data was then randomly split into training (80%; 3,600 examples), development (10%; 450 examples), and testing (10%; 450 examples) shards. for some languages, wiktionary contains pronunciations for both lemmas (i.e., headwords, citations forms) and inflection variants; for others, pronunciations are only available for lemmas. we hypothesized that cases where one inflectional variant of a lemma is present in the training data and another in the test data-as might occur if the data was split totally at randomwould make the overall task somewhat easier. to forestall this possibility, the splitting procedure was constrained so that all inflectional variants of any given lemma-according to the unimorph 2 @xcite ) paradigm tables, also extracted from wiktionary-are limited to a single shard. for example, since the french word acteur 'actor' occurs in the training shard, so must its plural form acteurs. this additional constraint was applied to all languages but japanese and vietnamese, for which no unimorph data was available. we note that wiktionary does not generally provide pronunciations for inflectional variants in japanese, and that vietnamese is a highly isolating language with no discernable system of inflection @xcite , so this is unlikely to have introduced bias. ## evaluation the primary metric for this task was word error rate (wer); we also report phone error rate (per). wer this is the percentage of words for which the hypothesized transcription sequence is not identical to the gold reference transcription; lower wer indicates better performance. following common practice in speech research, we multiply the wer by 100 and display it as a percentage. we choose this as the primary metric for the shared task because we hypothesize that any g2p error, no matter how small, will result in a substantial degradation in subjective quality for downstream speech applications. per this is a more forgiving measure measuring the normalized distance (i.e., in number of insertions, deletions, and substitutions) between the predicted and reference transcriptions. it is computed by summing the minimum edit distancecomputed with the @xcite algorithm-between prediction and reference transcriptions, and dividing by the sum of the reference transcription lengths. that is, per := 100 × ∑ n i edits(p, r) ∑ n i |r| where p is the predicted pronunciation sequence, r is the reference sequence, and edits(p, r) is the levenshtein distance between the two. once again, we multiply it by 100, though strictly speaking it is not a true percentage because it can hypothetically exceed 100. as with wer, lower per indicates better performance. participants were provided with two evaluation scripts: one which computes the two metrics for a single language, and another which macroaverages the metrics across all languages. ## baselines three baselines were made available at the start of the task. to aid reproducibility, participants were also provided with a conda "environment", a schematic that allows users to reconstruct the exact software environment used to train and evaluate the baselines. several submissions made use of the baselines for data augmentation or ensemble construction. we make these baseline implementations available under the task1/baselines subdirectory of the shared task repository. foot_2 pair n-gram model the first baseline consists of a pair n-gram model, which be can thought of as a finite-state approximation of a hidden markov model with states representing graphemes and emissions representing output phones. the model is quite similar to the phonetisaurus toolkit @xcite ), but here is implemented using the opengrm toolkit @xcite ; see @xcite for a full description. the sole hyperparameter for this model, markov model order, is tuned separately for each language using the development set. encoder-decoder lstm the second baseline is a neural network sequence-to-sequence model consisting of a single-layer bidirectional lstm encoder and a single-layer unidirectional lstm decoder connected using an attention mechanism @xcite . it is implemented using the fairseq library @xcite . lstm-based encoder-decoder models have been claimed to outperform pair n-gram g2p models, both in monolingual (e.g., @xcite and multilingual (e.g., @xcite ) evaluations, though these prior studies use substantially more training data than is available in this task. during training, we perform 4,000 updates to minimize label-smoothed crossentropy @xcite ) with a smoothing rate of .1. we use the adam optimizer (kingma and ba 2015) with a learning rate of α = .001 and weight decay coefficients of β = (.9, .98), and clip norms exceeding 1.0. we use the development set to tune-for each language-batch size (256, 512, 1024), dropout (.1, .2, .3), and the size of the encoder and decoder modules. a module is said to be "small" when it has a 128-dimension embedding layer and a 512-unit hidden layer, and "large" when it has a 256-dimension embedding layer and a 1024-unit hidden layer. in both cases, the decoder shares a single embedding layer for both inputs and outputs. altogether, this defines a 36element hyperparameter grid. during tuning, we employ a form of early stopping; we save a checkpoint every 5 epochs, and then use the checkpoint that achieves the lowest wer on the development set. we use a beam of size 5 for decoding. ## system descriptions below we provide brief descriptions of submissions to the shared task. ## results we now review baseline and submission results. ## discussion when this task was initially proposed, there was some concern that the submissions-if not the baselines themselves-would easily achieve perfect or near-perfect performance on some languages. this was not the case. even on the "easiest" language, the best submission has .89% wer, and for three languages, no submission achieves an error rate below 20%. at the same time, we observe a large range of error rates across languages. it is tempting to speculate that word and/or phone error rates actually represent differences in difficulty. insofar as this is correct, we can begin to ask what makes a language "hard to pronounce", much like how @xcite ask what makes a language "hard to language-model". one thing that may make a language hard to pronounce is data sparsity. consider the case of korean, which has by far the highest baseline error rate of all fifteen languages. three features of korean and of hangul conspire to make this task particularly challenging. first, hangul is a syllabary, and therefore necessarily has a much larger graphemic inventory than an alphabet or alphasyllabary. a whopping 889 unique hangul characters appear across the 4,500 words used for this task. 4 secondly, hangul is a relatively deep or abstract orthography (in the sense of @xcite ; it operates at a roughly-morphophonemic level whereas lithuanian and hungarian, for example, are is roughly phonemic. third, korean has many phonological processes that operate across syllable boundaries. since the effect of these processes is not indicated by the highly abstract, morphophonemic orthography, they can only be learned by observing the targeted syllable bigrams during training. @xcite to the lstm baseline and observe errors caused by underapplication of these coda-onset cluster rules. it is unsurprising then that several submissions achieved substantial gains by either romanizing hangul or decomposing it into its constituent jamo during preprocessing, since both techniques reduce the size of the input vocabulary. the results suggest that g2p technologies are not yet language-agnostic (in the sense of bender 2009). however, some caution is in order here: inter-language differences in word error rate may also reflect inconsistencies in the wikipron data itself. during the task, participants reported apparent transcription inconsistencies in the bulgarian, georgian, and lithuanian wiktionary data. if these inconsistencies are due to overly-narrow allophonic transcriptions, one might suspect that they can be learned by sufficiently sophisticated sequence-to-sequence models. however, if they represent free variation, inconsistent application of the transcription guidelines, or even typographical errors, they inflate error rates and increase the risk of overfitting. in response to this, we have begun development of quality assurance software for wikipron, including a phone-based whitelisting approach. we anticipate that manual error analysis will reveal errors in the wiktionary data, similar to the large number of test data er-rors identified by @xcite for the 2017 conll-sigmorphon morphological inflection task. to encourage this sort of error analysis, for the first time in the history of the sigmor-phon workshop, we publicly release the predictions made by all 23 submissions. 5 finally, we plan to apply large-scale consistency-enforcing edits upstream, i.e., to wiktionary itself. while the baselines are somewhat naïve and lack the sophisticated data augmentation and ensembling techniques used by the top submissions, we were pleasantly surprised by the substantial reductions in error achieved by the participating teams. as mentioned above, the best submissions handily outperforms the baselines for all languages. interestingly, this is true for the most challenging languages-like korean, where the best submission achieves a 45% relative word error reduction over the baseline-but also for vietnamese, the language with the lowest baseline wer; there, the best submission achieves an impressive 81% relative word error reduction. as mentioned above, top submissions make use of techniques such as preprocessing, data augmentation, ensembling, multi-task learning (e.g., phoneme-to-grapheme conversion), and self nor do any of the submissions make use of morphological analyzers or lexicons, which were found to be helpful in earlier work (e.g., @xcite . we speculate that such resources might further improve performance. finally we note that submissions make use of unsupervised tokenization techniques such as byte-pair encoding @xcite . finally, we note that several participants expressed interest in a low-resource version of this challenge, and two teams simulated a lowresource setting. we leave the design of a lowresource task for future work. ## conclusion sigmorphon, under whose auspices this task was conducted, was once known as sigphon and was primarily focused on computational phonetics and phonology. the shared task on multilingual grapheme-to-phoneme conversion, a uniquely phonological problem, thus represents something of a return to the roots of this special interest group. in this task, nine teams submitted 23 g2p systems for fifteen languages and achieved substantial improvements over the provided baselines. the results suggest many directions for improving g2p systems and the pronunciation dictionaries used to train them.