Results of the Second SIGMORPHON Shared Task on Multilingual Grapheme-to-Phoneme Conversion (2021)

Abstract: Grapheme-to-phoneme conversion is an important component in many speech technologies, but until recently there were no multilingual benchmarks for this task. The second iteration of the SIGMORPHON shared task on multilingual grapheme-to-phoneme conversion features many improvements from the previous year’s task (Gorman et al. 2020), including additional languages, a stronger baseline, three subtasks varying the amount of available resources, extensive quality assurance procedures, and automated error analyses. Four teams submitted a total of thirteen systems, at best achieving relative reductions of word error rate of 11% in the high-resource subtask and 4% in the low-resource subtask.

## introduction many speech technologies demand mappings between written words and their pronunciations. in open-vocabulary systems-as well as certain resource-constrained embedded systems-it is insufficient to simply list all possible pronunciations; these mappings must generalize to rare or unseen words as well. therefore, the mapping must be expressed as a mapping from a sequence of orthographic characters-graphemes-to a sequence of sounds-phones or phonemes. 1 the earliest work on grapheme-to-phoneme conversion (g2p), as this task is known, used ordered rewrite rules. however, such systems are often brittle and the linguistic expertise needed to build, test, and maintain rule-based systems is often in short supply. furthermore, rulebased systems are outperformed by modern neu1 we note that referring to elements of transcriptions as phonemes implies an ontological commitment which may or may not be justified; see @xcite ) for discussion. therefore, we use the term phone to refer to symbols used to transcribe pronunciations. with the possible exception of @xcite , who evaluate against a proprietary database of 20 languages and dialects, virtually all of the prior published research on graphemeto-phoneme conversion evaluates only on english, for which several free and low-cost pronunciation dictionaries are available. the 2020 sigmor-phon shared task on multilingual grapheme-to-phoneme conversion @xcite ) represented a first attempt to construct a multilingual benchmark for grapheme-to-phoneme conversion. the 2020 shared task targeted fifteen languages and received 23 submissions from nine teams. the second iteration of this shared task attempts to further refine this benchmark by introducing additional languages, a much stronger baseline model, new quality assurance procedures for the data, and automated error analysis techniques. furthermore, in response to suggestions from participants in the 2020 shared task, the task has been divided into high-, medium-, and low-resource subtasks. ## data as in the previous year's shared task, all data was drawn from wikipron @xcite , a massively multilingual pronunciation database extracted from the online dictionary wiktionary. depending on the language and script, wiktionary pronunciations are either manually entered by human volunteers working from language-specific pronunciation guidelines and/or generated from the graphemic form via language-specific serverside scripting. wikipron scrapes these pronunciatons from wiktionary, optionally applying case-folding to the graphemic form, removing any stress and syllable boundaries, and segmenting the pronunciation-encoded in the interna-tional phonetic alphabet-using the python library segments @xcite . in all, 21 wikipron languages were selected for the three subtasks, including seven new languages and fourteen of the fifteen languages used in the 2020 shared task. 2 in several cases, multiple scripts or dialects are available for a given language. for instance, wikipron has both latin and cyrillic entries for serbo-croatian, and three different dialects of vietnamese. in such case, the largest data set of the available scripts and/or dialects is chosen. furthermore, wikipron distinguishes between "broad" transcriptions delimited by forward slash (/) and "narrow" transcriptions delimited by square brackets @xcite ). 3 once again, the larger of the two data sets is the one used for this task. ## quality assurance during the previous year's shared task we became aware of several consistency issues with the shared task data. this lead us to develop quality assurance procedures for wikipron and the "upstream" wiktionary data. for a few languages, we worked with wiktionary editors who automatically enforced upstream consistency via "bots", i.e., scripts which automatically edit wiktionary entries. we also improved wikipron's routines for extracting pronunciation data from wiktionary. in some cases (e.g., vietnamese), this required the creation of language-specific extraction routines. in early versions of wikipron, users had limited means to separate out entries for languages written in multiple scripts. we therefore added an automated script detection system which ensures that entries for the many languages written with multiple scripts-including shared task languages maltese, japanese, and serbo-croatian-are sorted according to script. we noticed that the wikipron data includes many hyper-foreign pronunciations with nonnative phones. for example, the english data includes a broad pronunciation of bach (the surname of a family of composers) as /bɑːx/ with a velar fricative /x/, a segment which is common in german but absent in modern english. furthermore, unexpected phones may represent simple human error. therefore, we wished to exclude pronunciations which include any nonnative segments. this was accomplished by creating phonelists which enumerate native phones for a given language. separate phonelists may be provided for broad and narrow transcriptions of the same language. during data ingestion, if a pronunciation contains any segment not present on the phonelist, the entry was discarded. phonelist filtration was used for all languages in the medium-and low-resource subtasks, described below. ## task definition in this task, participants were provided with a collection of words and their pronunciations, and then scored on their ability to predict the pronunciation of a set of unseen words. ## evaluation the primary metric for this task was word error rate (wer), the percentage of words for which the hypothesized transcription sequence is not identical to the gold reference transcription. as the medium-and low-resource subtasks involve multiple languages, macro-averaged wer was used for system ranking. participants were provided with two evaluation scripts: one which computes wer for a single language, and one which also computes macro-averaged wer across two or more languages. the 2020 shared task also reported another metric, phone error rate (per), but this was found to be highly correlated with wer and therefore has been omitted here. ## baseline the 2020 shared task included three baselines: a wfst-based pair n-gram model, a bidirectional lstm encoder-decoder network, and a transformer. all models were tuned to minimize perlanguage development-set wer using a limitedbudget grid search. best results overall were obtained by the bidirectional lstm. despite the extensive gpu resources required to execute a ## submissions below we provide brief descriptions of submissions to the shared task; more detailed descriptions-as well as various exploratory analyses and post-submission experiments-can be found in the system papers later in this volume. @xcite produced a single submission to the low-resource subtask. the model is inspired by the previous year's bidirectional lstm baseline but also employs several data augmentation strategies. first, much of the development data is used for training rather than for validation. secondly, new training examples are generated using substrings of other training examples. finally, the az model is trained simultaneously on all languages, a method used in some of the previous year's shared task submissions (e.g., @xcite . @xcite produced four submissions to the medium-resource subtask and three to the low-resource subtask. all seven submissions are variations on the imitation learning baseline model (section 6). they experiment with processing individual ipa unicode characters instead of entire ipa "segments" (e.g., cluzh-1, cluzh-5, and cluzh-6), and larger ensembles (e.g., cluzh-3). they also experiment with input dropout, mogrifier lstms, and adaptive batch sizes, among other features. ## results multiple submissions to the high-and lowresource subtasks outperformed the baseline; however, no submission to the medium-resource subtask exceeded the baseline. the best results for each language are shown in ## discussion we once again see an enormous difference in language difficulty. one of the languages with the highest amount of data, english, also has one of the highest wers. in contrast, the baseline and all four submissions to the medium-resource subtask achieve perfect performance on georgian. this is a substantial change from the previous year's shared task: with a sample roughly half the size of this year's task, the best system @xcite as mentioned above, the data here are a mixture of broad and narrow transcriptions. at first glance, this might explain some of the variation in language difficulty; for example, it is easy to imagine that the additional details in narrow transcriptions make them more difficult to predict. however, for many languages, only one of the two levels of transcription is available at scale, and other languages, divergence between broad and narrow transcriptions is impressionistically quite minor. however, this impression ought to be quantified. while we responded to community demand for lower-and higher-resource subtasks, only one team submitted to the high-and medium-resource subtasks, respectively. it was surprising that none of the medium-resource submissions were able to consistently outperform the baseline model across the ten target languages. clearly, this year's baseline is much stronger than the previous year's. participants in the high-and medium-resource subtasks were permitted to make use of lemmas and morphological tags from unimorph as additional features. however, no team made use of 8 https://github.com/cuny-cl/wikipron/ issues/138 resources. some prior work (e.g., @xcite ) has found morphological tags highly useful, and error analysis ( §8.2) suggests this information would make an impact in french. there is a large performance gap between the medium-resource and low-resource subtasks. for instance, the baseline achieves a wer of 10.6 in the medium-resource scenario and a wer of 25.1 in the low-resource scenario. it seems that current models are unable to reach peak performance with the 800 training examples provided in the lowresource subtask. further work is needed to develop more efficient models and data augmentation strategies for low-resource scenarios. in our opinion, this scenario is the most important one for speech technology, since speech resourcesincluding pronunciation data-are scarce for the vast majority of the world's written languages. ## conclusions the second iteration of the shared task on multilingual grapheme-to-phoneme conversion features many improvements on the previous year's task, most of all data quality. four teams submitted thirteen systems, achieving substantial reductions in both absolute and relative error over the baseline in two of three subtasks. we hope the code and data, released under permissive licenses, foot_6 will be used to benchmark grapheme-to-phoneme conversion and sequence-to-sequence modeling techniques more generally.