Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems (2020)

Abstract: Applying the Transformer architecture on the character level usually requires very deep architectures that are difficult and slow to train. These problems can be partially overcome by incorporating a segmentation into tokens in the model. We show that by initially training a subword model and then finetuning it on characters, we can obtain a neural machine translation model that works at the character level without requiring token segmentation. We use only the vanilla 6-layer Transformer Base architecture. Our character-level models better capture morphological phenomena and show more robustness to noise at the expense of somewhat worse overall translation quality. Our study is a significant step towards high-performance and easy to train character-based models that are not extremely large.

## introduction state-of-the-art neural machine translation (nmt) models operate almost end-to-end except for input and output text segmentation. the segmentation is done by first employing rule-based tokenization and then splitting into subword units using statistical heuristics such as byte-pair encoding (bpe; @xcite or sentencepiece @xcite . recurrent sequence-to-sequence (s2s) models can learn translation end-to-end (at the character level) without changes in the architecture @xcite , given sufficient model depth. training character-level transformer s2s models @xcite is more complicated because the selfattention size is quadratic in the sequence length. in this paper, we empirically evaluate transformer s2s models. we observe that training a character-level model directly from random initialization suffers from instabilities, often preventing it from converging. instead, we propose finetuning subword-based models to get a model without explicit segmentation. our character-level models show slightly worse translation quality, but have better robustness towards input noise and better capture morphological phenomena. our approach is important because previous approaches have relied on very large transformers, which are out of reach for much of the research community. ## related work character-level decoding seemed to be relatively easy with recurrent s2s models @xcite . but early attempts at achieving segmentation-free nmt with recurrent networks used input hidden states covering a constant character span @xcite . @xcite showed that with a sufficiently deep recurrent model, no changes in the model are necessary, and they can still reach translation quality that is on par with subword models. @xcite and @xcite can leverage characterlevel information but they require tokenized text as an input and only have access to the character-level embeddings of predefined tokens. training character-level transformers is more challenging. @xcite successfully trained a character-level left-to-right transformer language model that performs on par with a subword-level model. however, they needed a large model with 40 layers trained on a billion-word corpus, with prohibitive computational costs. in the most related work to ours, @xcite managed to train a character-level nmt with the transformer model using transparent attention @xcite . transparent attention attends to all encoder layers simultaneously, making the model more densely connected but also more computationally expensive. during training, this improves the gradient flow from the decoder to the encoder. @xcite claim that trans-tokenization the cat sleeps on a mat. the cat sleeps on a mat . ## our method we train our character-level models by finetuning subword models, which does not increase the number of model parameters. similar to the transfer learning experiments of @xcite , we start with a fully trained subword model and continue training with the same data segmented using only a subset of the original vocabulary. to stop the initial subword models from relying on sophisticated tokenization rules, we opt for the loss-less tokenization algorithm from senten-cepiece @xcite . first, we replace all spaces with the sign and do splits before all non-alphanumerical characters (first line of we use bpe @xcite for subword segmentation because it generates the merge operations in a deterministic order. therefore, a vocabulary based on a smaller number of merges is a subset of a vocabulary based on more merges estimated from the same training data. examples we experiment both with deterministic segmentation and stochastic segmentation using bpe dropout @xcite . at training time, bpe dropout randomly discards bpe merges with probability p, a hyperparameter of the method. as a result of this, the text gets stochastically segmented into smaller units. bpe dropout increases translation robustness on the source side but typically has a negative effect when used on the target side. in our experiments, we use bpe dropout both on the source and target side. in this way, the charactersegmented inputs will appear already at training time, making the transfer learning easier. we test two methods for finetuning subword models to reach character-level models: first, direct finetuning of subword models, and second, iteratively removing bpe merges in several steps in a curriculum learning setup @xcite . in both cases we always finetune models until they are fully converged, using early stopping. ## experiments to cover target languages of various morphological complexity, we conduct our main experiments on two resource-rich language pairs, english-german and english-czech; and on a low-resource pair, english-turkish. rich inflection in czech, compounding in german, and agglutination in turkish are examples of interesting phenomena for character models. we train and evaluate the english-german translation using the 4.5m parallel sentences of the wmt14 data @xcite . czech-english is trained on 15.8m sentence pairs of the czeng 1.7 corpus @xcite and tested on wmt18 data @xcite . englishto-turkish translation is trained on 207k sentences of the setimes2 corpus (tiedemann, 2012) and evaluated on the wmt18 test set. we follow the original hyperparameters for the transformer base model @xcite , including the learning rate schedule. for finetuning, we use adam (kingma and ba, 2015) with a constant learning rate 10 -5 . all models are trained using marian @xcite . we also present results for character-level english-german models having about the same number of parameters as the best-performing subword models. in experiments with bpe dropout, we set dropout probability @xmath0.1. we evaluate the translation quality using bleu @xcite , chrf @xcite , and meteor 1.5 @xcite . following @xcite , we also conduct a noisesensitivity evaluation to natural noise as introduced by @xcite . with probability p words are replaced with their variants from a misspelling corpus. following @xcite , we assume the bleu scores measured with input can be explained by a linear approximation with intercept α and slope β using the noise probability p: bleu ≈ βp + α. however, unlike them, we report the relative translation quality degradation β/α instead of only β. parameter β corresponds to absolute bleu score degradation and is thus higher given lower-quality systems, making them seemingly more robust. to look at morphological generalization, we evaluate translation into czech and german using morpheval @xcite . morpheval consists of 13k sentence pairs that differ in exactly one morphological category. the score is the percentage of pairs where the correct sentence is preferred. ## results the results of the experiments are presented in with the exception of turkish, models trained by finetuning reach by a large margin better translation quality than character-level models trained from scratch. in accordance with @xcite , we found that bpe dropout applied both on the source and target side leads to slightly worse translation quality, presumably because the stochastic segmentation leads to multimodal target distributions. the detailed results are presented in appendix a. however, for most language pairs, we found a small positive effect of bpe dropout on the finetuned systems (see for english-to-czech translation, we observe a large drop in bleu score with the decreasing vocabulary size, but almost no drop in terms of meteor score, whereas for other language pairs, all metrics are in agreement. the differences between the subword and character-level models are less pronounced in the low-resourced english-to-turkish translation. whereas the number of parameters in transformer layers in all models is constant at 35 million, the number of parameters in the embeddings decreases 30× from over 15m to only slightly over 0.5m, with overall a 30% parameter count reduction. however, matching the number of parameters by increasing the model capacity narrows close the performance gap, as shown in in our first set of experiments, we finetuned the model using the character-level input directly. experiments with parent models of various vocabulary sizes (column "direct finetuning" in with our character-level models, we achieved higher robustness towards source-side noise (figure 1 ). models trained with a smaller vocabulary tend to be more robust towards source-side noise. character-level models tend to perform slightly better in the morpheval benchmark. detailed results are shown in in czech, models with a smaller vocabulary better cover agreement in gender and number in pronouns, probably due to direct access to inflective endings. unlike german, character-level models capture worse agreement in coordinations, presumably due to there being a longer distance in characters. training and inference times are shown in ## conclusions we presented a simple approach for training character-level models by finetuning subword models. our approach does not require computationally expensive architecture changes and does not require dramatically increased model depth. subword-based models can be finetuned to work on the character level without explicit segmentation with somewhat of a drop in translation quality. the models are robust to input noise and better capture some morphological phenomena. this is important for research groups that need to train and deploy character transformer models without access to very large computational resources. with subword units. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1715-1725, berlin, germany. association for computational linguistics. jörg tiedemann. 2012. parallel data, tools and interfaces in opus. in proceedings of the eighth international conference on language resources and evaluation (lrec-2012), pages 2214-2218, istanbul, turkey. european languages resources association (elra). ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n. gomez, lukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems 30, pages 5998-6008, long beach, ca, usa.