Improving Target-side Lexical Transfer in Multilingual Neural Machine Translation (2020)

Abstract: To improve the performance of Neural Machine Translation (NMT) for low-resource languages (LRL), one effective strategy is to leverage parallel data from a related high-resource language (HRL). However, multilingual data has been found more beneficial for NMT models that translate from the LRL to a target language than the ones that translate into the LRLs. In this paper, we aim to improve the effectiveness of multilingual transfer for NMT models that translate into the LRL, by designing a better decoder word embedding. Extending upon a general-purpose multilingual encoding method Soft Decoupled Encoding (Wang et al., 2019), we propose DecSDE, an efficient character n-gram based embedding specifically designed for the NMT decoder. Our experiments show that DecSDE leads to consistent gains of up to 1.8 BLEU on translation from English to four different languages.

## introduction the performance of neural machine translation (nmt; @xcite ) tends to degrade on low-resource languages (lrl) due to a paucity of parallel data @xcite . one effective strategy to improve translation in lrls is through multilingual training using parallel data from related high-resource languages (hrl) @xcite . the assumption underlying cross-lingual transfer is that by sharing parameters between multiple languages the lrl can benefit from the extra training signal from data in other languages. one of the most popular strategies for multilingual training is to train a single nmt model that translates in many directions by simply appending a flag to each source sentence to indicate which 1 open-source code is available at https://github. com/luyug/decsde target language to translate into @xcite . many works focus on using multilingual training to improve many-to-one nmt models that translate from both an hrl and an lrl to a single target language @xcite @xcite . in this situation, sentences from the hrl-target corpus provide an extra training signal for the decoder language model, on top of cross-lingual transfer on the source side. when training an nmt model that translates into an lrl, however, multilingual data tends to lead to smaller improvements @xcite @xcite . in this paper, we aim to improve the effectiveness of multilingual training for nmt models that translate into lrls. prior work has found vocabulary overlap to be an important indicator of whether data from other languages will be effective in improving nmt accuracy @xcite . therefore, we hypothesize that one of the main problems limiting multilingual transfer on the target side is that the lrl and the hrl may have limited vocabulary overlap, and standard methods for embedding target words via lookup tables would map corresponding vocabulary from these languages to different representations. to overcome this problem, we design a target word embedding method for multilingual nmt that encourages similar words from the hrls and the lrls to have similar representations, facilitating positive transfer to the lrls. while there are many methods to embed words from characters @xcite @xcite , we build our model upon soft decoupled encoding (sde; @xcite ), a recently-proposed generalpurpose multilingual word embedding method that has demonstrated superior performance to other alternatives. sde represents a word by combin-ing a character-based representation of its spelling and a lookup-based representation of its meaning. we propose decsde, an efficient adaptation of sde to nmt decoders. decsde uses a low-rank transformation to assist multilingual transfer, and it precomputes the embeddings for a fixed vocabulary to speedup training and inference. we test our method on translation from english to 4 different low-resource languages, and decsde brings consistent gains of up to 1.8 bleus. ## translating into low-resource languages standard nmt training is performed solely on parallel corpora from a source language s to a target language t . however, in the case that t is an lrl, we can use parallel data from s and a related hrl t to assist learning. the standard look-up embedding in nmt turns words from both the lrl and the hrl into vectors by mapping their indices in the vocabulary to the corresponding entry in the embedding matrix. this is harmful for positive transfer, because different words with similar spellings from the lrl and the hrl are mapped to independent embeddings. for example, "ola" in galician and "olá" in portuguese both mean "hello", but they would have separate representations through the look-up embedding. we give a demonstration of this embedding (mis-)alignment in § 5.2. since the target side data is essential for training the decoder's language model, representing lexicons from the lrl and hrl into shared space is especially important to improve positive transfer for nmt models that translate into lrls. ## soft decoupled encoding to address the limitation of the standard word representation for target side multilingual transfer, we turn to soft decoupled encoding (sde; wang et al. (2019) ), a word embedding method designed for multilingual data. sde decomposes a word embedding into two components: a character n-gram embedding with a language-specific transformation that represents its spelling, and a semantic embedding that represents its meaning. given a word w from the target language l i , sde embeds the words in three steps. character aware embeddings are first used to calculate the lexical representation of w. we extract a bag of n-grams frequency vector from w, denoted as bon(w), where each row corresponds to the number of times a character n-gram in the vocabulary appears in w. the character aware embedding of the w is then computed as language-specific transformation is then applied to lexical embedding c(w) to account for the divergence between the hrl and the lrl: latent semantic embeddings of w are calculated using an embedding matrix w @xmath0, which is shared between the languages. we use c i (w) as the query vector to perform attention @xcite ) over the embeddings in this section, we build upon the previously described sde, and design a new method for multilingual word representation on the target side. there are two aspects to consider when incorporating character-based representations like sde in decoders: 1) the embedding method should be efficient during both training and inference time, as it needs to be calculated over the entire vocabulary; 2) it should support popular decoder design decisions, such as weight tying (press and wolf, 2017), which allows the decoder to share the parameters of the target embedding matrix and the decoder projection before the softmax operation. with these considerations in mind, we introduce decsde, a multilingual target word embedding method based on sde for nmt decoders. fixed vocabulary and weight tying the standard sde is designed to encode words directly without segmenting them into subwords @xcite . this design choice works well for encoding words on the source side, but it can cause problems for the decoder, which requires a finite vocabulary to generate words for each time step. therefore, we choose to segment the target sentences into subwords @xcite , and encode each subword using decsde. the use of a fixed vocabulary also allows us to perform weight tying. specifically, we construct an embedding matrix for the decoder by precomputing the decsde embedding for each subword in the target vocabulary. this embedding matrix can then be used both as the encoder lookup table and as the projection matrix before the decoder softmax. efficient training and inference one drawback of the standard sde is that it requires more computation than standard look-up table embeddings because the lexical embedding requires one to extract and embed all character n-grams for each word. this problem is especially important for the decoder, since it needs to embed all target words in the vocabulary for each time step to calculate the probability distribution over the vocabulary. to make training more efficient, we extract the character n-grams for all words in the target vocabulary, and use an optimized embedding bag layer 2 to parallelize the calculation of lexical embeddings for all words in a batch. for inference, we precompute the decsde embedding for all subwords, effectively making inference as fast as the regular look-up table embedding. an analysis of training and inference speed can be found in § 5.2. ## implications and future work in this paper, we have demonstrated that decsde, a multilingual character-sensitive embedding method, improves translation accuracy into low resource languages. this implies, on a higher level, that looking into the character-level structure of the target-side vocabulary when creating word or subword embeddings is a promising way to improve cross-lingual transfer. while ablations have shown that the proposed design decisions (such as lowrank language-specific transformation, weight tying, etc.) are reasonable ones, this is just a first step in this direction. future work could examine even more effective methods for target-side lexical sharing in mt or other language generation tasks.