Avengers, Ensemble! Benefits of ensembling in grapheme-to-phoneme prediction (2021)

Abstract: We describe three baseline beating systems for the high-resource English-only sub-task of the SIGMORPHON 2021 Shared Task 1: a small ensemble that Dialpad’s speech recognition team uses internally, a well-known off-the-shelf model, and a larger ensemble model comprising these and others. We additionally discuss the challenges related to the provided data, along with the processing steps we took.

## introduction the transduction of sequences of graphemes to phones or phonemes, 2 that is from characters used in orthographic representations to characters used to represent minimal units of speech, is a core component of many tasks in speech science & technology. this graphemeto-phoneme conversion (or g2p) may be used, e.g., to automate or scale the creation of digital lexicons or pronunciation dictionaries, which are crucial to fst-based approaches to automatic speech recognition (asr) and synthesis @xcite . the sigmorphon 2021 workshop included a shared task on g2p conversion, comprising 3 sub-tasks. 3 the low-and mediumresource tasks were multilingual, while the high-resource task was english-only. this paper provides an overview of the three baseline-beating systems submitted by the dialpad team for the high-resource sub-task, ## sub-task 1: high-resource, english-only the organizers provided 41,680 lines of data in total; 33,344 for training, and 4,168 each for development and test. the data consists of word/pronunciation pairs (word-pron pairs, henceforth), where words are sequences of graphemes and pronunciations are sequences of characters from the international phonetic alphabet @xcite . the data was derived from the english portion of the wikipron database @xcite , a massively multilingual resource of word-pron pairs extracted from wiktionary foot_0 and subject to some manual qa and postprocessing. 5 the baseline model provided was the 2nd place finisher from the 2020 g2p shared task @xcite . it is an ensembled neural transition model that operates over edit actions and is trained via imitation learning @xcite . evaluation scripts were provided to compute word error rate (wer), the percentage of words for which the output sequence does not match the gold label. notwithstanding the baseline's strong prior performance and the amount of data available, the task proved to be challenging; the baseline system achieved development and test set wers of 45.13 and 41.94, respectively. we discuss possible reasons for this below. ## models we trained and evaluated several models for this task, both publicly available, in-house, and custom developed, along with various ensembling permutations. in the end, we submitted three sets of baseline beating results. the organizers assigned sequential identifiers to multiple submissions (e.g. dialpad-n); we include these in the discussion of our entries below, for ease of subsequent reference. ## results our main results are shown in ## additional experiments we experimented with different ensembles and found that incorporating models with different architectures generally improves overall performance. in the standalone results, only the top three models beat the baseline wer, but adding additional models with higher wer than the baseline continues to reduce overall wer. ## error analysis we conducted some basic analyses of the dialpad-1 submission's patterns of errors, to better understand its performance and identify potential areas of improvement. 13 ## conclusion we presented and discussed three g2p systems submitted for the sigmorphon2021 english-only shared sub-task. in addition to finding a strong off-the-shelf contender, we show that naive ensembling remains a strong strategy in supervised learning, of which g2p is a sub-domain, and that simple majority-voting schemes in classification can often leverage the respective strengths of sub-optimal component models, especially when diverse architectures are combined. additionally, we provided more evidence for the usefulness of linguistically-informed subword modeling as an input transformation on speech-related tasks. we also discussed additional experiments whose results were not submitted, indicating the benefit of exploring top-n model vs ensemble trade-offs, and demonstrating the potential benefit of an edit-distance based tiebreaking method for ensemble voting. future work includes further search for the optimal trade-off between ensemble size and performance, as well as additional exploration of the edit-distance voting scheme, and more sophisticated ensembling/voting methods, e.g. majority voting at the phone level on aligned outputs.