M ULTILINGUAL N EURAL M ACHINE T RANSLATION WITH S OFT D ECOUPLED E NCODING
A BSTRACT
Multilingual training of neural machine translation (NMT) systems has led to impressive accuracy improvements on low-resource languages.
However, there are still signiﬁcant challenges in efﬁciently learning word representations in the face of paucity of data.
In this paper, we propose Soft Decoupled Encoding (SDE), a multilingual lexicon encoding framework speciﬁcally designed to share lexicallevel information intelligently without requiring heuristic preprocessing such as pre-segmenting the data.
SDE represents a word by its spelling through a character encoding, and its semantic meaning through a latent embedding space shared by all languages.
Experiments on a standard dataset of four low-resource languages show consistent improvements over strong multilingual NMT baselines, with gains of up to 2 BLEU on one of the tested languages, achieving the new state-of-the-art on all four language pairs 1 .
I NTRODUCTION
Multilingual Neural Machine Translation (NMT) has shown great potential both in creating parameter-efﬁcient MT systems for many languages (Johnson et al., 2016), and in improving translation quality of low-resource languages (Zoph et al., 2016; Firat et al., 2016;
Gu et al., 2018; Neubig &
Hu, 2018; Nguyen & Chiang, 2018).
Despite the success of multilingual NMT, it remains a research question how to represent the words from multiple languages in a way that is both parameter efﬁcient and conducive to cross-lingual generalization.
The standard sequence-to-sequence (seq2seq) NMT model (Sutskever et al., 2014) represents each lexical unit by a vector from a look-up table, making it difﬁcult to share across different languages with limited lexicon overlap.
This problem is particularly salient when translating low-resource languages, where there is not sufﬁcient data to fully train the word embeddings.
Several methods have been proposed to alleviate this data sparsity problem in multilingual lexical representation.
The current de-facto standard method is to use subword units (Sennrich et al., 2016; Kudo, 2018), which split up longer words into shorter subwords to allow for generalization across morphological variants or compounds (e.g. “un/decide/d” or “sub/word”).
These can be applied to the concatenated multilingual data, producing a shared vocabulary for different languages, resulting in sharing some but not all subwords of similarly spelled words (such as “traducci´on” in Spanish and “traduc¸˜ao” in Portuguese, which share the root “tradu-”).
However, subword-based preprocessing can produce sub-optimal segmentations for multilingual data, with semantically identical and similarly spelled languages being split into different granularities (e.g. “traducci´on” and “tradu/c¸˜ao”)
leading to disconnect in the resulting representations.
This problem is especially salient when the high-resource language dominates the training data (see empirical results in Section 4).
In this paper, we propose Soft Decoupled Encoding (SDE), a multilingual lexicon representation framework that obviates the need for segmentation by representing words on a full-word level, but can nonetheless share parameters intelligently, aiding generalization.
Speciﬁcally, SDE softly decouples the traditional word embedding into two interacting components: one component represents how the word is spelled, and the other component represents the word’s latent meaning, which is shared over all languages present at training time.
We can view this representation as a decomposition of language-speciﬁc realization of the word’s form (i.e. its spelling) and its language-agnostic semantic function.
More importantly, our decoupling is done in a soft manner to preserve the interaction between these two components.
SDE has three key components: 1) an encoding of a word using character n -grams
(Wieting et al., 2016); 2) a language speciﬁc transform for the character encoding; 3) a latent word embedding constructed by using the character encoding to attend to a shared word embedding space, inspired by Gu et al.
Our method can enhance lexical-level transfer through the shared latent word embedding while preserving the model’s capacity to learn speciﬁc features for each language.
Moreover, it eliminates unknown words without any external preprocessing step such as subword segmentation.
We test SDE on four low-resource languages from a multilingual TED corpus (Qi et al., 2018).
Our method shows consistent improvements over multilingual NMT baselines for all four languages, and importantly outperforms previous methods for multilingual NMT that allow for more intelligent parameter sharing but do not use a two-step process of character-level representation and latent meaning representation (Gu et al., 2018). Our method outperforms the best baseline by about BLEU for one of the low-resource languages, achieving new state-of-the-art results on all four language pairs compared to strong multi-lingually trained and adapted baselines (Neubig & Hu, 2018).
L EXICAL R EPRESENTATION FOR M ULTILINGUAL NMT
In this section, we ﬁrst revisit the 3-step process of computing lexical representations for multilingual NMT, which is illustrated in Figure 1.
Then, we discuss various design choices for each step, as well as the desiderata of an ideal lexical representation for multilingual NMT.
2. L EXICAL U NIT S EGMENTATION
The ﬁrst step to compute the neural representation for a sentence is to segment the sentence into lexical units.
There are three popular options with different granularities:
• Word-based method splits an input sequence into words, often based on white spaces or punctuation.
This is perhaps the natural choice for lexical unit segmentation.
Early work in NMT all employ this method (Sutskever et al., 2014; Bahdanau et al., 2015).
• Character-based method splits an input sequence into characters (Lee et al., 2017).
• Subword-based method splits each word into pieces from a small vocabulary of frequently occurring patterns (Sennrich et al., 2016; Kudo, 2018).
2. E MBEDDING L OOK - UP
After a sentence is segmented into lexical units, NMT models generally look up embeddings from a dictionary to turn each lexical unit into a high-dimensional vector.
In the context of multilingual
NMT, the lexical unit segmentation method affects this dictionary in different ways.
In word-based segmentation, since the number of unique words for each language is unbounded, while computer memory is not, previous work, e.g. Sutskever et al.
(2014), resorts to a ﬁxed-size vocabulary of the most frequent words, while mapping out-of-vocabulary words to an (cid:104) unk (cid:105) token.
For multilingual NMT settings, where multiple languages are processed, the number of words mapped to (cid:104) unk (cid:105) signiﬁcantly increases.
Moreover, different languages, even related languages, have very few words that have exactly the same spelling, which leads to the same concept being represented by multiple and independent parameters.
This disadvantage hurts the translation model’s ability to learn the same concept in multiple languages.
Meanwhile, character-based segmentation can effectively reduce the vocabulary size, while maximizing the potential for parameter sharing between languages with identical or similar character sets.
However, character segmentation is based on the strong assumption that neural networks can infer meaningful semantic boundaries and compose characters into meaningful words.
This puts a large amount of pressure on neural models, requiring larger model sizes and training data.
Additionally, training character-based NMT systems is often slow, due to the longer character sequences (Cherry et al., 2018).
Subword-based segmentation is a middle ground between word and character segmentation.
However, in multilingual translation, the subword segmentation can be sub-optimal, as the subwords from high-resource languages, i.e. languages with more training data, might dominate the subword vocabulary, so that the words in low-resource language can be split into extremely small pieces.
Therefore, existing methods for lexical unit segmentation lead to difﬁculties in building an effective embedding look-up strategy for multilingual NMT.
2. O PTIONAL E NCODING T RANSFORMATIONS
Most commonly, the embedding vectors looked up from the embedding table are used as the ﬁnal lexical representation.
However, it is also possible to have multiple versions of embeddings for a single lexicon and combine them through operations such as attentional weighted sum (Gu et al., 2018).
Without loss of generality, we can assume there is always a transformation applied to the embedding vectors, and models that do not use such a transformation can be treated as using the identity transformation.
2. D ESIDERATA
To efﬁciently utilize parameters for multilingual NMT, the lexical representation should have two properties.
First, for maximal accuracy, the lexical representation should be able to accurately represent words in all of the languages under consideration .
Second, for better cross-lingual learning and generalization, such a representation should maximize the sharing of parameters across languages .
These two conﬂicting objectives are difﬁcult to achieve through existing methods.
The most common method of using lookup embeddings can only share information through lexical units that overlap between the languages.
Subword segmentation strikes a middle ground, but has many potential problems for multilingual NMT, as already discussed in Section 2.2.
Although Gu et al.
(2018)’s method of latent encoding increases lexical level parameter sharing, it still relies on subwords as its fundamental units, and thus inherits the previously stated problems of sub-word segmentation.
We also ﬁnd in experiments in Section 4 that it is actually less robust than simple lookup when large monolingual data to pre-train embeddings is not available, which is the case for many low-resourced languages.
Next, in Section 3, we propose a novel lexical representation strategy that achieves both desiderata.
S OFT D ECOUPLED E NCODING
Given the conﬂict between sharing lexical features and preserving language speciﬁc properties, we propose SDE, a general framework to represent lexical units for multilingual NMT.
Speciﬁcally, following the linguistic concept of the “arbitrariness of the sign” (Chandler, 2007), SDE decomposes the modeling of each word into two stages: (1) modeling the language-speciﬁc spelling of the word, and (2) modeling the language-agnostic semantics of the word.
This decomposition is based on the need for distinguished treatments between a word’s semantics and its spelling.
Semantic representation is language-agnostic.
For example, the English “hello” and the French “bonjour” deliver the same greeting message, which is invariant with respect to the language.
SDE shares such semantic representations among languages by querying a list of shared concepts, which are loosely related to the linguistic concept of “sememes” (Greimas, 1983).
This design is implemented using an attention mechanism, where the query is the lexical unit representation, and the keys and the values come from an embedding matrix shared among all languages.
Meanwhile, the word spellings are more sophisticated.
Here, we identify two important observations about word spellings.
First, words in related languages can have similar spellings, e.g. the English word “color” and the French word “coleur”.
In order to effectively share parameters among languages, a word spelling model should utilize this fact.
Second, and not contradicting the ﬁrst point, related languages can also exhibit consistent spelling shifts.
For instance, “Christopher”, a common name in English, has the spelling “Kry ˇ stof” in Czech.
This necessitates a learnable rule to convert the spelling representations between such pairs of words in related languages.
To account for both points, we use a language-speciﬁc transformation on top of a ﬁrst encoding layer based on character n -grams.
3. E
[Table]
Before we describe our speciﬁc architecture in detail (Section 3.2), given these desiderata discussed above, we summarize the designs of several existing methods for lexical representation and our proposed
SDE framework in Table 1.
Without a preprocessing step of subword segmentation, SDE can capture the lexical similarities of two related languages through the character n -gram embedding while preserving the semantic meaning of lexicons through a shared latent embedding.
3. D ETAILS OF S OFT D ECOUPLED E NCODING
As demonstrated in Figure 2, given a word w in a multilingual corpus from language L i , SDE constructs the embedding of w in three phases.
Lexical Embedding.
We maintain an embedding matrix W c ∈ R C × D , where D is the embedding dimension and C is the number of n -grams in our vocabulary.
Out-of-vocab n -grams are mapped to a designated token (cid:104) unk (cid:105) .
W c is shared among all languages.
Following Wieting et al.
(2016), for each word w , we ﬁrst compute the bag of character n -grams of w , denoted by BoN ( w ) , which is a sparse vector whose coordinates are the appearance counts of each character n -gram in w .
For instance, the characters n -grams with n = 1 , 2 , 3 , 4 of the word “puppy” are shown in Figure 2.
We then look up and add the rows of W c according to their corresponding counts, and apply a tanh activation function on the result
[Equation]
Language-speciﬁc Transformation.
To account for spelling shifts between languages (c.f. Section 3), we apply an language-speciﬁc transformation to normalize away these differences.
We use a simple fully-connected layer for this transformation.
In particular, for language L i , we have
[Equation] where W L i ∈ R D
× D is the transformation matrix speciﬁc to language L i .
Latent Semantic Embedding.
Finally, to model the shared semantics of words among languages, we employ an embedding matrix W s ∈ R S × D , where S is the number of core semantic concepts we assume a language can express.
Similar to the lexical embedding matrix W c , the semantic embedding matrix W s is also shared among all languages.
For each word w , its language-speciﬁc embedding c
i ( w ) is passed as a query for an attention mechanism (Luong et al., 2015) to compute a weighted sum over the latent embeddings
[Equation]
Finally, to ease the optimization of our model, we follow Vaswani et al.
and add the residual connection from c
i ( w ) into e latent ( w ) , forming the Soft Decoupled Encoding embedding of w
[Equation]
E XPERIMENT
We build upon a standard seq2seq NMT model for all experiments.
Except for the experiments in
Section 4.6, we run each experiment with 3 different random seeds, and conduct signiﬁcance tests for the results using the paired bootstrap (Clark et al., 2011).
4. D ATASETS
[Table]
We use the 58-language-to-English TED corpus for experiments.
Following the settings of prior works on multilingual
NMT (Neubig & Hu, 2018; Qi et al., 2018), we use three low-resource language datasets: Azerbaijani (aze), Belarusian (bel), Galician (glg) to English, and a slightly higher-resource dataset, namely Slovak (slk) to English.
Each lowresource language is paired with a related high-resource language: Turkish (tur), Russian (rus), Portuguese (por), and Czech (ces) respectively.
Table 2 shows the statistics of each dataset.
4. B ASELINES
For the baseline, we use the standard lookup embeddings for three granularities of lexical units: (1) word: with a ﬁxed word vocabulary size of 64,000 for the concatenated bilingual data; (2) subjoint: with BPE of 64,000 merge operations on the concatenated bilingual data; and (3) sub-sep: with BPE separately on both languages, each with 32,000 merge operations, effectively creating a vocabulary of size 64,000.
We use all three settings to compare their performances and to build a competitive baselines.
We also implement the latent embedding method of Gu et al.
We use 32,000 character n -gram with n = { 1 , 2 , 3 , 4 , 5 } from each language and a latent embedding size of 10,000.
4. R ESULTS
Table 3 presents the results of SDEand of other baselines.
For the three baselines using lookup, subsep achieves the best performance for three of the four languages.
Sub-joint is worse than sub-sep although it allows complete sharing of lexical units between languages, probably because sub-joint leads to over-segmentation for the low-resource language.
Our reimplementation of universal encoder (Gu et al., 2018) does not perform well either, probably because the monolingual embedding is not trained on enough data, or the hyperparamters for their method are harder to tune.
Meanwhile, SDE outperforms the best baselines for all four languages, without using subword units or extra monolingual data
[Table]
[Table]
scores after removing each component from
SDE-com.
Statistical signiﬁcance is indicated with ∗ ( p < 0 . 0001 ) and † ( p < 0 . 005 ), compared with the full model in the ﬁrst row.
4 Edit Distance 0 .0 .0 .0 . C o un t / T o t a l aze bel glg por 4. A BLATION S TUDIES
We next ablate various features of SDE by removing each of the three key components, and show the results in Table 4.
Removing the latent semantic embedding and lexical embedding consistently harms the performance of the model.
The effect of the language speciﬁc transformation is smaller and is language dependent.
Removing the language speciﬁc transformation does not lead to significant difference for glg.
However, it leads to a 0.8 gain in BLEU for aze, a 0.3 gain for slk, but about a 0.6 decrease for bel.
A further inspection of the four language pairs shows that the language speciﬁc transform is more helpful for training on languages with fewer words of the same spelling.
To quantify this, we extract bilingual dictionaries of the low-resource languages and their paired high-resource languages, and measure the edit distance between the character strings of the aligned words.
We use FastAlign (Dyer et al., 2013) to extract aligned words between the source language and English from the parallel training data, then match the English side of two related languages
Gu et al.
(2018) used extra monolingual data from the Wikipedia dump. We have also tried testing our reimplementation of their method with trained monolingual embedding from the Wikipedia dump, but achieved similar performance. to get their dictionary.
Figure 3 shows the percentage of the word pairs grouped by their edit distance.
Among the four languages, bel has the lowest percentage of words that are exactly the same with their corresponding words in the high-resource language (0 edit distance), indicating that the language speciﬁc transform is most important for divergent languages.
4. E FFECT OF S UBWORDS
Although using subwords in NMT can eliminate unknown words and control the vocabulary size, it might not be optimal for languages with rich morphology (Ataman & Federico, 2018).
We also show in Table 3 that certain choices when using subwords on multilingual data (e.g. whether to train the segmentation on the concatenated multilingual data or separately) have a large effect on the performance of the model.
SDE achieves superior performance without using subwords, so it can avoid the risk of sub-optimal segmentation.
Still, we test SDE with subwords to study its effect on our framework.
The results are shown in Table 5.
We test two methods of using subwords: 1)
[Table]
we use sub-sep as lexical units, and encode its lexical representation using the character n -grams of the subwords; 2) we use words as lexical units, but use its subword pieces instead of the character n -grams to construct its lexical representation.
We use SDE-sub to indicate the second way of using subwords.
When using SDE with sub-sep as lexical units, the performance slightly increases for aze, but decreases for the other three language.
Therefore, it is generally better to directly use words as lexical units for SDE.
When we use words as lexical units but replace character n -gram with subwords, the performance on two of the languages doesn’t change signiﬁcantly, while the performance decreases for bel and increases for glg.
We also examine the performance of SDE and the best baseline sub-sep with different vocabulary size and found that SDE is also competitive with a small character n -gram vocabulary of size
8K.
Details can be found in Appendix A.2.
4. T RAINING ON A LL L ANGUAGES
[Table]
To further compare SDE’s ability to generalize to different languages, we train both SDE and sub-sep on the low-resource languages paired with all four high-resource languages.
The results are listed in Table 6.
For bel, SDE trained on all languages is able to improve over just training with bilingual data by around 0.6 BLEU.
This is the best result on bel, with around 3 BLEU over the best baseline.
The performance of sub-sep, on the other hand, decreases by around 1.5 BLEU when training on all languages for bel.
The performance of both methods decreases for aze when using all languages.
SDE only slightly loses 0.1 BLEU while sub-sep loses over 3 BLEU.
4. W HY DOES SDE WORK BETTER ?
The SDE framework outperforms the strong sub-sep baseline because it avoids sub-optimal segmentation of the multilingual data.
We further inspect the improvements by calculating the word
F-measure of the translated target words based on two properties of their corresponding source
[Figure]
glg [Figure] words: 1) the number of subwords they were split into; 2) the edit distance between their corresponding words in the related high-resource language.
From Figure 4 left , we can see that SDE is better at predicting words that were segmented into a large number of subwords.
Figure 4 right shows that the gain peaks on the second bucket for 2 languages and the ﬁrst bucket for bel and slk, which implies that SDE shows more improvements for words with small but non-zero edit distance from the high-resource language.
This is intuitive: words similar in spelling but with a few different characters can be split into very different subword segments, while SDE can leverage the lexical similarities of word pairs with slightly different spelling.
4. Q UALITATIVE A NALYSIS
Table 7 lists a few translations for both sub-sep and SDE.
We can see that SDE is better at capturing functional words like “if” and “would”.
Moreover, it translates “climatologist” to a related word “weather”, probably from the preﬁx “climat” in glg, while sub-sep gives the totally unrelated translation of “college friend”.
Some examples of bilingual lexicons that can be better captured by
SDE can be found in Appendix A.3.
[Figure]
[Table]
R ELATED W ORKS
In multilingual NMT [Table] several approaches have been proposed to enhance parameter sharing of lexical representations.
Zoph et al.
(2016) randomly assigns embedding of a pretrained NMT model to the vocabulary of the language to adapt, which shows improvements over retraining the new embeddings from scratch.
Nguyen & Chiang (2018) propose to match the embedding of the word piece that overlaps with the vocabulary of the new language.
When training directly on concatenated data, it is also common to have a shared vocabulary of multilingual data (Neubig & Hu, 2018;
Qi et al., 2018).
Gu et al.
(2018) propose to enhance parameter sharing in lexical representation by a latent embedding space shared by all languages.
Several prior works have utilized character level embeddings for machine translation (Cherry et al., 2018; Lee et al., 2017; Ataman & Federico, 2018), language modeling (Kim et al., 2016; J´ozefowicz et al., 2016), and semantic parsing (Yih et al., 2014).
Speciﬁcally for NMT, fully character-level
NMT can effectively reduce the vocabulary size while showing improvements for mulitlingual
NMT (Lee et al., 2017), but it often requires much longer to train (Cherry et al., 2018).
Ataman & Federico (2018) shows that character n -gram encoding of words can improve over BPE for morphologically rich languages.
C ONCLUSION
Existing methods of lexical representation for multilingual NMT hinder parameter sharing between words that share similar surface forms and/or semantic meanings.
We show that SDE can intelligently leverage the word similarities between two related languages by softly decoupling the lexical and semantic representations of the words.
Our method, used without any subword segmentation, shows signiﬁcant improvements over the strong multilingual NMT baseline on all languages tested.
Acknowledgements: The authors thank David Mortensen for helpful comments, and Amazon for providing GPU credits.
This material is based upon work supported in part by the Defense Advanced Research Projects Agency Information Innovation Ofﬁce (I2O) Low Resource Languages for Emergent Incidents (LORELEI) program under Contract No. HR0011-15-C0114.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the U.S. Government.
The U.S.
Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
A A PPENDIX A. T RAINING D ETAILS
• We use a 1-layer long-short-term-memory (LSTM) network with a hidden dimension of 512 for both the encoder and the decoder.
• The word embedding dimension is kept at 128, and all other layer dimensions are set to 512.
• We use a dropout rate of 0.3 for the word embedding and the output vector before the decoder softmax layer.
• The batch size is set to be 1500 words.
We evaluate by development set BLEU score for every 2500 training batches.
• For training, we use the Adam optimizer with a learning rate of 0.001.
We use learning rate decay of 0.8, and stop training if the model performance on development set doesn’t improve for 5 evaluation steps.
A. E FFECT OF V OCABULARY S IZE
We examine the performance of SDE and the best baseline sub-sep, with a character n -gram vocabulary and sub-word vocabulary respectively, of size of 8K, 16K
[Figure]
and 32K. We use character n -gram of n = { 1 , 2 , 3 , 4 } for 8K and 16K vocabularies
[Figure]
and n = { 1 , 2 , 3 , 4 , 5 } for the 32K vocabulary.
Figure 5 shows that for all four languages, SDE outperforms sub-sep with all three vocabulary sizes.
This shows that SDE is also competitive with a relatively small character n -gram vocabulary.
[Figure] [Figure]
. E XAMPLE B ILINGUAL W ORDS
Table 8 lists some words and their subwords from bel and its related language rus.
We can see that subwords fail to capture all the lexical similarities between these words, and sometimes the word pairs are segmented into different number of pieces.
bel rus eng word subword word subword фiнансавыя фiнансавы я финансовых финансовы х ﬁnancial стадыён стады ён стадион стадион stadium розных розны х разных разны х different паказаць паказа ць показать показать show
A. A NALYSIS OF A TTENTION OVER L ATENT E MBEDDING S PACE
In this section we compare the attention distribution over the latent embedding space of related languages, with the intuition that words that mean the same thing should have similar attention distributions.
We calculate the KL divergence of the attention distribution for word pairs in both the LRL and HRL.
Figure 6 shows that the lowest KL divergence is generally on the diagonals representing words with identical meanings, which indicates that similar words from two related languages
[Figure] [Figure]
[Figure]
[Figure]
tend to have similar attention over the latent embedding space.
Note that this is the case even for words with different spellings.
For example, Figure 6 right shows that the KL divergence between the glg word “m ´ usicos” (meaning “musicians”), and the por word of the closet meaning among the words shown here, “jogadores” (meaning “players”), is the smallest, although their spellings are quite different.
A. A NALYSIS OF W ORD V ECTORS FROM SDE
In this section, we qualitatively examine the location of word vectors at different stages of SDE.
We reduce the word vectors to two dimensions using t-SNE (van der Maaten & Hinton, 2008).
In particular, we focus on two groups of words shown in Table 9 from glg-por, where each word from glg is paired with two words from por, one with the same meaning but different spelling, while the other has similar spelling but different meaning.
Figure 7 left shows the embeddings derived from the character n -grams.
At this stage, the word “cando” is closer to “caindo”, which has a different meaning, than “quando”, which has the same meaning but a slightly more different spelling.
The word “etiqueta” lies in the middle of “r´otulo” and “riqueza”.
After the whole encoding process, the location of the words are shown in Figure 7 right .
At the ﬁnal stage, the word “cando” moves closer to “quando”, which has the same meaning, than “caindo”, which is more similar in spelling.
The word “etiqueta” is also much closer to “r´otulo”, the word with similar meaning, and grows further apart from “riqueza”.
[Figure]