Applying the Transformer to Character-level Transduction (2021)

Abstract: The transformer has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks. Yet for character-level transduction tasks, e.g. morphological inflection generation and historical text normalization, there are few works that outperform recurrent models using the transformer. In an empirical study, we uncover that, in contrast to recurrent sequence-to-sequence models, the batch size plays a crucial role in the performance of the transformer on character-level tasks, and we show that with a large enough batch size, the transformer does indeed outperform recurrent models. We also introduce a simple technique to handle feature-guided character-level transduction that further improves performance. With these insights, we achieve state-of-the-art performance on morphological inflection and historical text normalization. We also show that the transformer outperforms a strong baseline on two other character-level transduction tasks: grapheme-to-phoneme conversion and transliteration.

## introduction the transformer @xcite has become a popular architecture for sequence-to-sequence transduction in nlp. it has achieved state-of-theart performance on a range of common word-level transduction tasks: neural machine translation @xcite , question answering @xcite and abstractive summarization @xcite . in addition, the transformer forms the backbone of the widely-used bert @xcite . yet for character-level transduction tasks like morphological inflection, the dominant model has remained a recurrent neural network-based sequence-code will be available at https://github.com/ shijie-wu/neural-transducer . we evince our two primary contributions: (1) we set the new state of the art morphological inflection using the transformer and (2) we demonstrate the transformer's dependence on the batch size. gehring et al. (2017) and vaswani et al. (2017) suggest that transformers (and convolutional models in general) should be better at remembering long-range dependencies. in the case of morphology, none of these considerations seem relevant: inflecting a word (a) requires little capacity to model long-distance dependencies and is largely a monotonic transduction; (b) it involves no semantic disambiguation, the tokens in question being letters; (c) it is not a task for which parallelization during training appears to help, since training time has never been an issue in morphology tasks. foot_1 in this work, we provide state-of-the-art numbers for morphological inflection and historical text normalization, a novel result in the literature. we also show the transformer outperforms a strong recurrent baseline on two other characterlevel tasks: grapheme-to-phoneme (g2p) conversion and transliteration. we find that a single hyperparameter, batch size, is largely responsible for the previous poor results. despite having fewer parameters, the transformer outperforms the recurrent sequence-to-sequence baselines on all four tasks. we conduct a short error analysis on the task of morphological inflection to round out the paper. ## the transformer for characters the transformer. the transformer, originally described by @xcite , is a selfattention-based encoder-decoder model. the encoder has n layers, consisting of a multi-head selfattention layer and a two-layer feed-forward layer with relu activation, both equipped with a skip connection. the decoder has a similar structure as the encoder except that, in each decoder layer between the self-attention layer and feed-forward layer, a multi-head attention layer attends to the output of the encoder. layer normalization @xcite is applied to the output of each skip connection. sinusoidal positional embeddings are used to incorporate positional information without the need for recurrence or convolution. here, we describe two modifications we make to the transformer for character-level tasks. a smaller transformer. as the dataset sizes in character-level transduction tasks are significantly smaller than in machine translation, we employ a smaller transformer with @xmath0. we use 4 self-attention heads. the embedding size is d @xmath1. in the preliminary experiments, we found that using layer normalization before self-attention and the feed-forward layer performed slightly better than the original model. it is also the default setting of a popular implementation of the transformer @xcite . the transformer alone has around 7.37m parameters, excluding character embeddings and the linear mapping before the softmax layer. we decode the model left to right in a greedy fashion. feature invariance. some character-level transduction is guided by features. for example, in the case of morphological reinflection, the task requires a set of morphological attributes that control what form a citation form is inflected into (see fig. 2 for an example). however, the order of the features is irrelevant. in a recurrent neural network, features are input in some predefined order as special characters and pre-or postpended to the input character sequence representing the citation form. the same is true for a vanilla transformer model, as shown on the left-hand side of fig. 2 . this leads to optimization. we use adam (kingma and ba, 2014) with a learning rate of 0.001 and an inverse square root learning rate scheduler @xcite with 4k steps during the warm-up. we train the model for 20k gradient updates and save and evaluate the model every 400 gradient updates. we select the best model out of 50 checkpoints based on development set accuracy. the number of gradient updates and checkpoints are roughly the same as @xcite , the single model state of the art on the 2017 sigmorphon dataset. we use their model as a baseline model. for all experiments, we use a single predefined random seed. ## related work character-level transduction is largely dominated by attention-based lstm sequence-to-sequence @xcite models @xcite . character-level transduction tasks usually involve input-output pairs that share large substrings and alignments between these are often monotonic. models that address the task tend to focus on exploiting such structural bias. instead of learning the alignments, @xcite use external monotonic alignments from the sigmor-phon 2016 shared task baseline @xcite . @xcite use this approach to win the conll-sigmorphon 2017 shared task on morphological inflection @xcite . @xcite shows that explicitly modeling alignment (hard attention) between source and target characters outperforms soft attention. @xcite further shows that enforcing monotonicity in a hard attention model improves performance. ## conclusion using a large batch size and feature invariant input allows the transformer to achieve strong performance on character-level tasks. however, it is unclear what linguistic errors the transformer makes compared to recurrent models on these tasks. future work should analyze the errors in detail as @xcite does for recurrent models. while wu and cotterell shows that the monotonicity bias benefits character-level tasks, it is not evident how to enforce monotonicity on multi-headed self-attention. future work should consider how to best incorporate monotonicity into the model, either by enforcing it strictly @xcite or by pretraining the model to copy (anastasopoulos and neubig, 2019).