Character-Level Translation with Self-attention (2020)

Abstract: We explore the suitability of self-attention models for character-level neural machine translation. We test the standard transformer model, as well as a novel variant in which the encoder block combines information from nearby characters using convolutions. We perform extensive experiments on WMT and UN datasets, testing both bilingual and multilingual translation to English using up to three input languages (French, Spanish, and Chinese). Our transformer variant consistently outperforms the standard transformer at the character-level and converges faster while learning more robust character-level alignments.

## introduction most existing neural machine translation (nmt) models operate on the word or subword-level, which tends to make these models memory inefficient because of large vocabulary sizes. characterlevel models @xcite instead work directly on raw characters, resulting in a more compact language representation, while mitigating out-of-vocabulary (oov) problems @xcite . characterlevel models are also very suitable for multilingual translation since multiple languages can be modeled using the same character vocabulary. multilingual training can lead to improvements in the overall performance without an increase in model complexity @xcite , while also circumventing the need to train separate models for each language pair. models based on self-attention have achieved excellent performance on a number of tasks, including machine translation @xcite and representation learning @xcite ; multi-head attention add & norm feed forward add & norm conv 1d width 3 conv 1d width 5 conv 1d width 7 concat. conv 1d width 3 parallel conv 1d layer parallel conv 1d layer multi-head attention add & norm feed forward add & norm a) b) figure 1 : a comparison of the encoder blocks in the standard transformer @xcite and our novel modification, the convtransformer (b), which uses 1d convolutions to facilitate character interactions. @xcite . despite the success of these models, their suitability for character-level translation remains largely unexplored, with most efforts having focused on recurrent models (e.g., @xcite ; @xcite ). in this work, we perform an in-depth investigation of the suitability of self-attention models for character-level translation. we consider two models: the standard transformer from @xcite and a novel variant that we call the convtransformer (figure 1 , section 3). the convtransformer uses convolutions to facilitate interactions among nearby character representations. we evaluate these models on both bilingual and multilingual translation to english, using up to three input languages: french (fr), spanish (es), and chinese (zh). we compare the performance when translating from close (e.g., fr and es) and on distant (e.g., fr and zh) input languages (section 5.1) and we analyze the learned character alignments (section 5.2). we find that self-attention models work surprisingly well for character-level translation, achieving competitive performance to equivalent subword-level models while requiring up to 60% fewer parameters (under the same model configuration). at the character-level, the convtransformer outperforms the standard transformer, converges faster, and produces more robust alignments. ## convolutional transformer to facilitate character-level interactions in the transformer, we propose a modification of the standard architecture, which we call the convtransformer. in this architecture, we use the same decoder as the standard transformer, but we adapt each encoder block to include an additional subblock. the sub-block (figure 1 , b ), inspired from @xcite , is applied to the input representations m , before applying self-attention. the sub-block consists of three 1d convolutional layers, c w , with different context window sizes w. in order to maintain the temporal resolution of convolutions, the padding is set to w-1 2 . we apply three separate convolutional layers, c 3 , c 5 and c 7 , in parallel, using context window sizes of 3, 5 and 7, respectively. the different context window sizes aim to resemble characterlevel interactions of different levels of granularity, such as on the subword-or word-level. to compute the final output of the convolutional subblock, the outputs of the three layers are concatenated and passed through an additional 1d convolutional layer with context window size 3, c 3 , which fuses the representations: for all convolutional layers, we set the number of filters to be equal to the embedding dimension size d model , which results in an output of equal dimension as the input m . therefore, in contrast to @xcite , who use max-pooling to compress the input character sequence into segments of characters, here we leave the resolution unchanged, for both transformer and convtransformer models. finally, for additional flexibility, we add a residual connection @xcite from the input to the output of the convolutional block. ## experimental set-up datasets. we conduct experiments on two datasets. first, we use the wmt15 @xmath0, on which we test different model configurations and compare our results to previous work on character-level translation. we follow the preprocessing in @xcite and use the newstest-2014 dataset for testing. second, we conduct our main experiments using the united nations parallel corporus (un) @xcite , for two reasons: (i) un contains a large number of parallel sentences from six languages, allowing us to conduct multilingual experiments; tasks. our experiments are designed as follows: (i) bilingual scenario, in which we train a model with a single input language; (ii) multilingual scenario, in which we input two or three languages at the same time without providing any language identifiers to the models and without increasing the number of parameters. we test combining input languages that can be considered as more similar in terms of syntax and vocabulary (e.g. fr and es) as well as more distant (e.g., es and zh). ## conclusion we performed a detailed investigation of the utility of self-attention models for character-level translation. we test the standard transformer architecture, as well as introduce a novel variant which augments the transformer encoder with convolutions, to facilitate information propagation across nearby characters. our experiments show that self-attention performs very well on characterlevel translation, with character-level architectures performing competitively when compared to equivalent subword-level architectures while requiring fewer parameters. training on multiple input languages is also effective and leads to improvements across all languages when the source and target languages are similar. when the languages are different, we observe a drop in performance, in particular for the distant language. in future work, we will extend our analysis to include additional source and target languages from different language families, such as more asian languages. we will also work towards improving the training efficiency of character-level models, which is one of their main bottlenecks, as well as towards improving their effectiveness in multilingual training. ment appears to be better preserved. this is another indication that the convtransformer is more robust for multilingual translation of distant languages. (iv) for multilingual translation with three inputs, where two of the three languages are close (fr+es+@xmath1, figure 7 ), we observe a similar pattern, with word alignments being better preserved by the convtransformer. source pour que ce cadre institutionnel soit efficace, il devra remédier aux lacunes en matière de réglementation et de mise en oeuvre qui caractérisent à ce jour la gouvernance dans le domaine du développement durable. reference for this institutional framework to be effective, it will need to fill the regulatory and implementation deficit that has thus far characterized governance in the area of sustainable development. @xmath2 to ensure that this institutional framework is effective, it will need to address regulatory and implementation gaps that characterize governance in sustainable development. convtransformer in order to ensure that this institutional framework is effective, it will have to address regulatory and implementation gaps that characterize governance in the area of sustainable development.