Data Augmentation for Transformer-based G 2 P (2020)

Abstract: The Transformer model has been shown to outperform other neural seq2seq models in several character-level tasks. It is unclear, however, if the Transformer would benefit as much as other seq2seq models from data augmentation strategies in the low-resource setting. In this paper we explore strategies for data augmentation in the g2p task together with the Transformer model. Our results show that a relatively simple alignment-based strategy of identifying consistent input-output subsequences in grapheme-phoneme data coupled together with a subsequent splicing together of such pieces to generate hallucinated data works well in the low-resource setting, often delivering substantial performance improvement over a standard Transformer model.

## introduction the transformer model @xcite has recently been shown to be robust for character-level translation tasks, outperforming other recurrent sequence-to-sequence (seq2seq) models in a wide range of tasks, including morphological inflection, grapheme-to-phoneme (g2p), and text normalization @xcite . a transformer-based model also served as the baseline system for both the sigmorphon 2020 shared tasks on graphemeto-phoneme conversion @xcite and low-resource morphological inflection @xcite , delivering substantially better performance than other models. 1 a common thread in research with characterlevel seq2seq has been that, for situations where few training examples are available, alternative strategies to produce more robust performance must be taken. for morphology tasks, this has included strategies such as instructing the model 1 our code is available at https://github.com/ lonelyrider-cs/sig_shared_tasks . ## data augmentation we experimented with two strategies of data augmentation: our first strategy was to identify in the training data grapheme sequences in the beginning of words and at the ends of words that (almost) always map to the same phoneme sequence, such as a word-initial c consistently mapping to k. subsequently we generated new training data by swapping such sequences across words, generating new words. this initial strategy failed to provide improvements on the development set, and we moved to a more refined version of this idea, discussed in more detail below. ## results the main results are shown in ## related work recurrent neural networks in a variety of models have been applied to the g2p problem, including lstms and bidirectional lstms @xcite , as well as convolutional networks @xcite . the transformer for g2p is investigated in @xcite and @xcite , showing improvements over previous models, at least in high-resource settings. low-resource settings for g2p in general are examined in @xcite , and a number of papers have experimented with high-resource to lowresource transfer learning @xcite ; ## conclusion we have developed a method for data augmentation for the g2p task based on a 1-to-1 alignment of input/output strings together with a confidence calculation of what parts of the aligned strings can be used to splice together an augmented dataset. used together with the popular transformer seq2seq model, we see significant and consistent improvements on very small datasets of 100 examples, moderate improvements on medium-size datasets (500 examples), with the advantage tapering off and mostly disappearing completely with the shared tasks' datasets of 3,600 examples.