Making a Point: Pointer-Generator Transformers for Disjoint Vocabularies (2020)

Abstract: Explicit mechanisms for copying have improved the performance of neural models for sequence-to-sequence tasks in the low-resource setting. However, they rely on an overlap between source and target vocabularies. Here, we propose a model that does not: a pointer-generator transformer for disjoint vocabularies. We apply our model to a low-resource version of the grapheme-to-phoneme conversion (G2P) task, and show that it outperforms a standard transformer by an average of 5.1 WER over 15 languages. While our model does not beat the the best performing baseline, we demonstrate that it provides complementary information to it: an oracle that combines the best outputs of the two models improves over the strongest baseline by 7.7 WER on average in the low-resource setting. In the high-resource setting, our model performs comparably to a standard transformer.

## introduction deep learning models define the state of the art on the majority of sequence-to-sequence tasks in natural language processing (nlp). even when training data is limited, neural networks outperform many alternative approaches, e.g., on machine translation @xcite or on morphological generation tasks @xcite . for the second group, the use of mechanisms for copying has drastically improved performance when training sets are small @xcite . our work builds on the insight that the ability to copy elements from the input over to the outputas done by a pointer network @xcite or a pointer-generator network @xcite can increase model performance on sequence-tosequence tasks in the low-resource setting, as it simplifies the learning problem. however, existing neural models require that inputs and outputs consist of elements from overlapping sets. here, we propose a pointer-generator transformer model for disjoint source and target vocabularies. our model, shown in figure 1 , is a hybrid of an lstm pointer-generator model @xcite and a transformer model @xcite . additionally, we integrate a mapping function, which defines a correspondence between elements in the source and target vocabularies. we apply our model to the task of graphemeto-phoneme conversion: mapping the spelling of a word to a representation of its pronunciation @xcite . this task is a great testing ground for our approach: input and output vocabularies are disjoint for many languages (cf. we experiment with our model on varying training set sizes. in the low-resource setting, averaged over 15 languages, our architecture improves performance by 5.1 wer over a standard transformer. further, it outperforms both the transformer and a copy baseline for up to 1000 training examples. while it underperforms our best performing baseline, we show that it provides valuable complementary information to it. in the high-resource setting, it performs comparably to a vanilla transformer. ## conclusion we introduced a pointer-generator transformer for sequence-to-sequence tasks with disjoint vocabularies, and evaluated it on g2p in 15 different languages. while our model did not perform as well as our strongest baseline, we showed that our model learns complementary information to the latter, thus succeeding on examples the baseline fails on. we further demonstrated that combining the best outputs of our pointer-generator transformer and the baseline results in a lower wer on lowresource g2p than each individual model. we leave the development of a system that can combine the two g2p systems for future work.