Compositional Source Word Representations for Neural Machine Translation (2018)

Abstract: The requirement for neural machine translation (NMT) models to use fixed-size input and output vocabularies plays an important role for their accuracy and generalization capability. The conventional approach to cope with this limitation is performing translation based on a vocabulary of sub-word units that are predicted using statistical word segmentation methods. However, these methods have recently shown to be prone to morphological errors, which lead to inaccurate translations. In this paper, we extend the source-language embedding layer of the NMT model with a bi-directional recurrent neural network that generates compositional representations of the source words from embeddings of character n-grams. Our model consistently outperforms conventional NMT with sub-word units on four translation directions with varying degrees of morphological complexity and data sparseness on the source side.

## introduction neural machine translation (nmt) has improved the state-of-the-art performance in machine translation of many languages @xcite . however, current nmt systems still suffer from poor performance in translating infrequent or unseen words, limiting their deployment for translating low-resource and morphologically-rich languages. this problem is c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. mainly caused by the fundamental design of the model, which requires observing many examples of a word until its input representation (i.e. embedding) becomes effective. moreover, the convention of limiting the input and output vocabularies to few tens of thousands of words to control the computational complexity of the model leads to coverage issues. in fact, a word can be translated only if an exact match of it is found in the vocabulary. to cope with this well-known problem, several studies have suggested to redefine a new model vocabulary in terms of the interior orthographic units compounding the words, such as character n-grams (costa-jussa and fonollosa, 2016; @xcite or statistically-learned sub-word units @xcite @xcite . in spite of providing an ideal open vocabulary solution, the former set of approaches mostly failed to achieve competitive results. this might be related to the semantic ambiguity caused by solely relying on embeddings of character n-grams which are generally learned by disregarding any lexical context, hence, morphology. in fact, building a vocabulary of sub-word units for training the nmt model and performing translation based on subword embeddings has now become the prominent approach. however, many studies have shown that statistical word segmentation methods can break the morphological structure of words, leading to loss of semantic and syntactic information in the sentence and, consequently, inaccurate translations @xcite @xcite @xcite . principally, these solutions are unsupervised methods and can never reach the accuracy of morphological analyzers, which, on the other hand, are not available in every language and can-not provide sufficiently compact vocabularies for the large training sets typically used in nmt. in order to increase the accuracy in translating rare and unseen words with nmt, in this paper, we propose to learn information about the source language morphology directly from the bilingual lexical context and use this information to compose word representations from a minimal set of input symbols. in addition to improving the quality of input word representations, our approach also aims at eliminating the necessity of using a separate and sub-optimal word segmentation step on the source language. the approach of learning word embeddings compositionally has recently been applied in language modeling and has found to be promising @xcite . in this study, which extends (ataman and federico, 2018b) foot_0 , we present and evaluate an approach for improving the source language input representations in nmt by augmenting the embedding layer with a bi-directional recurrent neural network (bi-rnn), which can learn compositional input word representations from embeddings of character n-grams. we compare our approach against conventional embedding-based representations of sub-word units learned from statistical word segmentation methods in official evaluation benchmarks, under low to medium resource conditions, by pairing english with four languages: czech, german, italian and turkish, where each language represents a distinct morphological typology. the experimental findings show that our compositional input representations provide significantly and consistently better translation quality for rare and unknown words than the prominent sub-word embedding based nmt approaches in all language directions. ## neural machine translation the nmt model we use in this paper @xcite is based on the idea of predicting the conditional probability of translating a source sentence @xmath0, x 2 , . . . x m ) of length m, into a target sentence @xmath1, y 2 , . . . y j . . . y l ) of length l, using the decomposition the model is trained by maximizing the loglikelihood of a training dataset consisting of parallel sentence pairs in two languages using stochastic gradient descent methods @xcite and the backpropagation through time @xcite algorithm . the inputs of the model are one-hot vectors, which have a single bit set to 1 to identify a given word in the vocabulary. each word vector is mapped to an embedding, a continuous representation of the word in a lower-dimensional but more dense space. then, the encoder, a stacked bi-rnn, learns a distributed representation of the source sentence x in the form of m dense vectors corresponding to its hidden states. the output states of a stacked rnn encoder with l layers is computed using the following equations: the number of parameters used by the model are mainly defined by the sizes of the source and target vocabularies, which requires to use fixed-size vocabularies in order to control the computational complexity. however, this limitation creates an important bottleneck when translating from and to low-resource and morphologically-rich languages, due to the sparseness of the lexical distribution. ## related work in order to improve the translation accuracy of rare words in nmt, previous studies have pro-posed several approaches which share the representations of word pieces among different words. these approaches include either engineering new nmt models that efficiently work at the character level, or performing a pre-processing step where words are segmented into smaller units using supervised or statistical tools before computing the nmt vocabulary. ## learning compositional input representations via bi-rnns one drawback of using statistical word segmentation methods for vocabulary prediction in nmt is that these methods constitute a pre-processing step to nmt, and hence they are not optimized for the translation task. moreover, as given in figure 1a , transforming sentences into sequences of sub-words leads to distributing the probability of a source word among multiple tokens, thus, increases the complexity of the alignment task performed by the attention model. in order to improve the accuracy in translating rare words in nmt, instead, we propose to perform nmt using word representations learned compositionally from smaller orthographic symbols inside the words, such as character n-grams, that can easily fit in the model vocabulary. this composition is essentially a function which can establish a mapping between combinations of orthographic units and lexical meaning, that is learned using the bilingual context, so that it can produce representations that are optimized for machine translation. in our model (figure 1b ), the one-hot vectors retrieve the corresponding source embeddings for every word and feed them to an additional composition layer, which computes the final representations that are input to the encoder. for learning the mapping between the sublexical units and the lexical context, we employ a bi-rnn. hence, by encoding the context of each interior unit inside the word, we believe that the network be able to capture important cues about their functional role, i.e. semantic or syntactic contribution to the word meaning. we implement the network using grus @xcite , which have shown comparable performance to lstm units @xcite while performing faster computation. as a minimal set of input symbols required to cope with contextual ambiguities, and at the same time optimize the size of the nmt vocabulary, we opt to use intersecting sequences of character trigrams, as recently suggested by vania and lopez (2017). our preliminary experiments @xcite ) also confirmed the stand-alone sufficiency of character trigrams as fundamental units in the compositional nmt model. given a bi-rnn with a forward (f ) and backward (b) layer, the input representation w of a token of t characters is computed from the hidden states h f t and h 0 b , i.e. the final outputs of the forward and backward rnns, as follows: ## experiments in order to evaluate our approach in nmt, we set up an evaluation benchmark which models nmt from four languages: czech (cs), german (de), italian (it) and turkish (tr) into english (en), where each input language represents a different lexical distribution reflected by its morphological characteristics, simulating conditions ranging from the low-resource and high sparseness (turkish) to the high-resource and low sparseness (italian) cases. for training the czech-english and german-english nmt models, we use the available data sets from the wmt 2 @xcite shared task on machine translation of news, which consist of europarl @xcite , commoncrawl 2 the first conference on machine translation and news commentary @xcite . for achieving a comparable size of training data, we reduce the training set in german-english using the invitation model @xcite . we evaluate these models on the official test sets from 2016. due to the lack of sufficient amount of news domain data, for the italian-english and turkish-english directions, we build generic nmt systems using data collected from ted talks @xcite , eu bookshop (skadins et al., 2014), global voices, gnome, tatoeba, ubuntu @xcite , kde4 @xcite , open subtitles @xcite and setimes @xcite , and reduce the size of the training data for having comparable numbers of tokens (italian) and types (turkish) with the other languages. these models are evaluated on the official test sets from the evaluation campaign of iwslt 3 @xcite . the morphological characteristics of the languages used in our study are presented in we perform nmt by keeping the segmentation 3 the international workshop on spoken language translation with shared tasks organized between 2003-2017. ## results and discussion italian to english translation direction, the performance of the simple model is higher than the compositional model by 0.36 bleu, 0.005 ter and 0.0035 chrf points. the better performance of the compositional model in translating german, czech and turkish suggests that our approach is beneficial in eliminating the morphological errors caused by segmentation in languages with different morphological typologies. the improvements are highest for czech and german, both of which have a fusional morphology of medium to high complexity, and the source language vocabulary of the training data ranges from around 400,000 to 500,000 types of words, indicating a high level of lexical sparseness. at a comparable vocabulary size, the improvements are generally lower in turkish to english translation direction, where the input language has an agglutinative morphology with a much higher level of data sparseness. this might be due to the efficient performance of lmvr in generating morphologically-consistent sub-word units in the low-resource setting of agglutinative languages. nevertheless, the results suggest that our compositional model can learn a higher level of morphological knowledge than lmvr, which was previously found to provide comparable performance to morphological analyzers in turkish-english nmt using the embedding-based input representations @xcite . moreover, it can also generalize over different types of morphology in both low and high resource settings. in the italian to english translation direction, despite the comparable size of training data with czech and german in the high-resource setting, the source word vocabulary is around 150,000 words, which represents the low level of sparse-ness. the higher overall performance of the nmt model which uses bpe for vocabulary reduction compared to the compositional model suggests that the embedding based sub-word representations are sufficient in reducing this vocabulary to fit into a space of 30,000 units. nevertheless, in order to observe the actual accuracy in translating rare words, we carry out a focused analysis where we sample from the test sets only the sentences that contain singletons (i.e. words that are observed once in the training corpus) in the source side and evaluate the translation accuracy obtained with each nmt model on these sentences. this sampling results in 190 sentences in italian, 470 sentences in turkish, 562 sentences in german and 611 sentences in czech to english directions. the results of this analysis, which can be found in we extend our analysis in order to also evaluate the performance of different approaches in translating out-of-vocabulary (oov) words. similarly, we sample from the test sets only the sentences which contain oovs, resulting in relatively larger test sets of 443 italian, 1096 turkish, 1396 czech and 1449 german sentences. the evaluation of each nmt model on these sets, results of which are also given in model in all languages, where the improvements range from 0.90 to 2.46 bleu points. these findings suggest that our compositional nmt approach provides a higher generalization capability compared to conventional approaches to open vocabulary nmt. ## conclusion in this paper, we have addressed the problem of translating rare words in nmt and proposed to solve it by replacing the conventional sub-word embeddings with input representations compositionally learned from character n-grams using a bi-rnn. our approach showed significant and consistent improvements over a variety of languages with different morphological typologies, making it a competitive approach for nmt of low-resource and morphologically-rich languages. in the future, we plan to extend our approach in order to improve also the target side representations used by the nmt decoder and to evaluate it under similar morphological and data sparseness conditions on the target side. finally, our benchmark and implementation are available for public use.