T OWARDS N EURAL P HRASE - BASED M ACHINE
T
RANSLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we present Neural Phrase-based Machine Translation (NPMT).
Our method explicitly models the phrase structures in output sequences using SleepWAke Networks (SWAN), a recently proposed segmentation-based sequence modeling method.
To mitigate the monotonic alignment requirement of SWAN, we introduce a new layer to perform (soft) local reordering of input sequences.
Different from existing neural machine translation (NMT) approaches, NPMT does not use attention-based decoding mechanisms.
Instead, it directly outputs phrases in a sequential order.
Our experiments show that NPMT achieves superior performances on IWSLT 2014 German-English/English-German and IWSLT
English-Vietnamese translation tasks compared with strong NMT baselines.
We also observe that our method produces meaningful phrases in output languages.
I NTRODUCTION
A word can be considered as a basic unit in languages. However, in many cases, we often need a phrase to express a concrete meaning.
For example, consider understanding the following sentence, “machine learning is a field of computer science”.
It may become easier to comprehend if we segment it as “[machine learning] [is] [a field of] [computer science]”, where the words in the bracket ‘[]’ are regarded as “phrases”.
These phrases have their own meanings, and can often be reused in other contexts.
The goal of this paper is to explore the use of phrase structures aforementioned for neural networkbased machine translation systems (Sutskever et al., 2014; Bahdanau et al., 2015).
To this end, we develop a neural machine translation method that explicitly models phrases in target language sequences.
Traditional phrase-based statistical machine translation (SMT) approaches have been shown to consistently outperform word-based ones (Koehn et al., 2003; Koehn, 2009; Lopez, 2008).
However, modern neural machine translation (NMT) methods (Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) do not have an explicit treatment on phrases, but they still work surprisingly well and have been deployed to industrial systems (Zhou et al., 2016; Wu et al., 2016).
The proposed Neural Phrase-based Machine Translation (NPMT) method tries to explore the advantages from both kingdoms.
It builds upon Sleep-WAke Networks (SWAN), a segmentation-based sequence modeling technique described in Wang et al.
(2017a), where segments (or phrases) are automatically discovered given the data.
However, SWAN requires monotonic alignments between inputs and outputs. This is often not an appropriate assumption in many language pairs.
To mitigate this issue, we introduce a new layer to perform (soft) local reordering on input sequences.
Experimental results show that NPMT outperforms attention-based NMT baselines in terms of the BLEU score (Papineni et al., 2002) on IWSLT 2014 German-English/English-German and IWSLT
English-Vietnamese translation tasks.
We believe our method is one step towards the full integration of the advantages from neural machine translation and phrase-based SMT.
This paper is organized as follows.
Section 2 presents the neural phrase-based machine translation model.
Section 3 demonstrates the usefulness of our approach on several language pairs.
We conclude our work with some discussions in Section 4.
[Figure]
sie wollen die entscheidung wirklich richtig treffen sie wirklich wollen treffen die entscheidung richtig you really want to make the decision right
[Figure]
N EURAL PHRASE - BASED MACHINE TRANSLATION
We first give an overview of the proposed NMPT architecture and some related work on incorporating phrases into NMT.
We then describe the two key building blocks in NPMT: 1) SWAN, and 2) the soft reordering layer which alleviates the monotonic alignment requirement of SWAN.
In the context of machine translation, we use “segment” and “phrase” interchangeably.
2.1 T HE OVERALL ARCHITECTURE OF NMPT
Figure 1(a) shows the overall architecture of NPMT.
The input sequence is first turned into embedding representations and then they go through a (soft) reordering layer (described below in Section 2.3).
We then pass these “reordered” activations to the bi-directional RNN layers, which are finally fed into the SWAN layer to directly output target language in terms of segments (or phrases).
While it is be possible to replace bi-directional RNN layers with other layers (Gehring et al., 2017), in this paper, we have only explored this particular setting to demonstrate our proposed idea.
We only explicitly consider output segments using SWAN while input segments are implicitly modeled via empty segments in SWAN.
This is conceptually different from the traditional phrase-based
SMT where both inputs and outputs are phrases (or segments).
We leave the option of exploring explicit input segments as future work.
There have been several works that propose different ways to incorporate phrases into attentionbased neural machine translation, such as Tang et al.
; Wang et al.
(2017b)
; Dahlmann et al.
These approaches typically use predefined phrases (obtained by external methods, e.g., phrase-based SMT) to guide or modify the existing attention-based decoder.
The major difference from our approach is that, in NPMT, we do not use attention-based decoding mechanisms, and our phrase structures for the target language are automatically discovered from the training data.
Another line of related work is the segment-to-segment neural transduction model (SSNT)
(Yu et al., 2016), which shows promising results on a Chinese-to-English translation task under a noisy channel framework (Yu et al., 2017).
In SSNT, the segments are implicit, and the monotonic alignments between the inputs and outputs are achieved using latent variables.
The latent variables are marginalized out during training using dynamic programming.
[Figure]
2.2 M ODELING PHRASES WITH SWAN
Here we review the SWAN model proposed in Wang et al.
(2017a).
SWAN defines a probability distribution for the output sequence given an input sequence.
It models all valid output segmentations of the output sequence as well as the monotonic alignments between the output segments and the input sequence.
Empty segments are allowed in the output segmentations.
It does not make any assumption on the lengths of input or output sequence.
Assume input sequence is x 1: T (cid:48) and output sequence is y 1: T .
Let S y denote the set containing all valid segmentations of y 1: T , with the constraint that the number of segments in a segmentation is the same as the input sequence length, T (cid:48) .
Let a t to denote a segment or phrase in the target sequence.
Empty segments are allowed to ensure that we can correctly align segment a t to input element x t .
Otherwise, we might not have a valid alignment for the input and output pair.
See Figure 2 for an example of the emitted segmentation of y 1: T .
The probability of the sequence y 1: T is defined as the sum of the probabilities of all the segmentations in S y (cid:44) { a 1: T (cid:48) :
π ( a 1: T (cid:48) )
=
y 1: T } ,p ( y 1: T | x 1: T (cid:48) )
(cid:44) (cid:88)
a
[Equation]
y
T (cid:48) (cid:89) t = p ( a t | x t ) , (1) where the p ( a t | x t ) is the segment probability given input element x t , which is modeled using a recurrent neural network (RNN) with an additional softmax layer.
π ( · ) is the concatenation operator and the symbol $ , end of a segment, is ignored in the concatenation operator π ( · ) .
(An empty segment, which only contains $ will thus be ignored as well.)
Since | S y | is exponentially large, direct summation quickly becomes infeasible when T or T (cid:48) is not small.
Instead, Wang et al. (2017a) developed an exact dynamic programming algorithm to tackle the computation.
The key idea is that although the number of possible segmentations is exponentially large, the number of possible segments is polynomial— O ( T 2 ) .
In other words, it is possible to first compute all possible segment probabilities, p ( a t | x t ) , ∀ a t , x t , and then use dynamic programming to calculate the output sequence probability p ( y 1: T | x 1: T (cid:48) ) in Eq.
The feasibility of using dynamic programming is due to a property of segmentations—a segmentation of a subsequence is also part of the segmentation of the entire sequence.
In practice, a maximum length for a segment a t is also enforced to reduce the computational complexity, since the length of useful segments is often not very long.
Wang et al.
(2017a) also discussed a way to carry over information across segments using a separate RNN, which we will not elaborate here.
We refer the readers to the original paper for the algorithmic details.
SWAN defines a conditional probability for an output sequence given an input one.
It can be used in many sequence-to-sequence tasks.
In practice, a sequence encoder like a bi-directional RNN can be used to process the raw input sequence (like speech signals or source language) to obtain x 1: T (cid:48) that is to be passed into SWAN for decoding.
For example, Wang et al.
(2017a) demonstrated the usefulness of SWAN in the context of speech recognition.
Greedy decoding for SWAN is straightforward.
We first note that p ( a t | x t ) is modeled as an RNN with an additional softmax layer.
Then for each x t in x 1: T (cid:48) , we can use a left-to-right greedy decoding algorithm (Sutskever et al., 2014) for p ( a t | x t ) to produce an output segment (possibly empty).
We then concatenate these output segments to form the greedy decoding of the entire output sequence.
See Wang et al.
(2017a)
for the algorithmic details of the beam search decoder.
2.3 L OCAL REORDERING OF INPUT SEQUENCES
SWAN assumes a monotonic alignment between the output segments and the input elements.
For speech recognition experiments in Wang et al.
(2017a)
, this is a reasonable assumption.
However, for machine translation, this is usually too restricted.
In neural machine translation literature, attention mechanisms were proposed to address alignment problems (Bahdanau et al., 2015; Luong et al., 2015; Raffel et al., 2017).
But it is not clear how to apply a similar attention mechanism to SWAN due to the use of segmentations for output sequences.
One may note that in NPMT, a bi-directional RNN encoder for the source language can partially mitigate the alignment issue for SWAN, since it can access every source word.
However, from our empirical studies, it is not enough to obtain the best performance.
Here we augment SWAN with a reordering layer that does (soft) local reordering of the input sequence.
This new model leads to promising results on the IWSLT 2014 German-English/English-German, and IWSLT 2015 EnglishVietnamese machine translation tasks.
One additional advantage of using SWAN is that since SWAN does not using attention mechanism, decoding can be much faster, since now we remove the need to query the entire input source for every output word (Raffel et al., 2017).
We now describe the details of the local reordering layer shown in Figure 3(a).
Denote the input to the local reordering layer by e 1: T (cid:48) and the output of this layer by h 1: T (cid:48) .
We compute h t as
[Equation]
where σ ( · ) is the sigmoid function, and 2 τ + 1 is the local reordering window size.
Notation [ e t − τ ; . .
; e t ; . . .
; e t + τ ] is the concatenation of vectors e t − τ , . . .
, e t , . . .
, e t
+ τ .
For i = 0 , . . .
, 2 τ , notation w i is the parameter for sigmoid function at position i of the input window.
It decides the weight of e t − τ
+ i through gate σ (cid:0) w Ti [ e t − τ ; . . .
; e t ; . . .
; e t + τ ] (cid:1) .
The final output h t is a weighted linear combination of the input elements, e t − τ , . . .
, e t , . . .
, e t + τ , in the window followed by a nonlinear transformation by the tanh( · ) function.
Figure 3(b) illustrates how local reordering works.
Here we want to (softly) select an input element from a window given all information available in this window.
Suppose we have two adjacent windows, ( e 1 , e 2 , e 3 ) and ( e 2 , e 3 , e 4 ) .
If we pick e 3 in the first window and e 2 in the second, e 2 and e 3 are effectively reordered.
Our layer is different from the typical attention mechanism (Bahdanau et al., 2015) in three ways. First, we do not have a query to begin with as in standard attention mechanisms.
Second, we do not normalize the weights for the input elements e t − τ , . . .
, e t , . . .
, e t + τ .
This provides the reordering capability and can shuffle everything if needed.
Third, the gate of any position i in the reordering window is determined by all input elements e t − τ , . . .
, e t , . . .
, e t + τ in the window.
We provides a visualizing example of the reordering layer gates in Appendix A.
One related work to our reordering layer is the Gated Linear Units (GLU)
(Dauphin et al., 2017) which can control the information flow of the output of a traditional convolutional layer.
But GLU does not have a mechanism to decide which input element from the convolutional window to choose.
From our experiments, neither GLU nor traditional convolutional layer helped our setup of using SWAN.
Another related work is the self-attention mechanism (Vaswani et al., 2017); this could potentially provide a soft reordering capability and we leave the exploration of this in the future work.
e te te t e t+ e t+ σ t-2 σ tσ t σ t+1 σ t+ h t (a) eeeheh(b)
E XPERIMENTS
In this section, we evaluate our model on the IWSLT 2014 German-English (Cettolo et al., 2014),
IWSLT 2014 English-German, IWSLT 2015 English-Vietnamese machine translation tasks.
We note that, in this paper, we limited the application of our model to relatively small datasets to demonstrate the usefulness of our method.
We plan to explore more large scale experiments in future work.
3.1 IWSLT14 G ERMAN -E NGLISH
We evaluate our model on the German-English machine translation track of the IWSLT 2014 evaluation campaign (Cettolo et al., 2014).
The data comes from translated TED talks, and the dataset contains roughly 153K training sentences, 7K development sentences, and 7K test sentences.
We use the same preprocessing and dataset splits as in Ranzato et al.
; Wiseman & Rush (2016);
Bahdanau et al.
We report our IWSLT 2014 German-English experiments using one reordering layer with window size 7, two-bi-directional GRU encoder (Gated recurrent unit, Chung et al.
with 256 hidden units, and a two-unidirectional GRU decoder with 512 hidden units.
We add dropout with a rate of 0 . 5 in the GRU layer.
We chose GRU since baselines for comparisons were using GRU.
The maximum segment length is set to 6.
Batch size is set as 32 and the Adam algorithm (Kingma &
Ba, 2014) is used for optimization with initial learning rate as 0.001.
For decoding, we use greedy search and beam search with a beam size of 10.
As reported in Maas et al.
; Bahdanau et al.
(2017), we find that penalizing candidate sentences that are too short was required to obtain the best results.
All hyperparameters are chosen based on the development set.
NPMT takes about 2–3 days to run to convergence (40 epochs) on a 4 M40 GPU machine.
The results are summarized in Table 1.
In addition to previous reported baselines in the literature, we also explored the best hyperparameter using the same model architecture (except the reordering layer) using sequence-to-sequence model with attention as reported as LL ∗ of Table 1.
NPMT achieves state-of-the-art results on this dataset as far as we know.
Compared to the supervised sequence-to-sequence model, LL (Bahdanau et al., 2017), NPMT achieves 2.4 BLEU gain in the greedy setting and 2.25 BLEU gain using beam-search.
Our results are also better than those from the actor-critic based methods in Bahdanau et al.
But we note that our proposed method is orthogonal to the actor-critic method.
So it is possible to further improve our results using the actor-critic method.
We also run two following experiments to verify the sources of the gain.
The first is to add a reordering layer to the original sequence-to-sequence model with attention, which gives us BLEU scores of 25.55 (greedy) and 26.91 (beam search).
The second is to remove the reordering layer from NPMT, which gives us BLEU scores of 27.79 (greedy) and 29.28 (beam search).
This shows that the reordering layer and SWAN are
BLEU
Greedy
Beam Search
[Table]
al
[Table] [Table]
In greedy decoding, we can estimate the average segment length 3 for the output.
The average segment length is around 1.4–1.5, indicating phrases with more than one word are being decoded.
Figure 2 shows an example of the input and decoding results with NPMT.
We can observe phraselevel translation being captured by the model (e.g., “danke” → “thank you”).
The model also knows when to sleep before outputting a phrase (e.g., “das” → $).
We use the index of words in the source sentence to indicate where the output phrases are from.
Table 2 shows some sampled examples.
We can observe there are many informative segments in the decoding results, e.g., “tens of thousands of”, “the best thing”, “a little”, etc.
There are also mapping from phrase to phrase, word to phrases, and phrase to word in the examples.
Following the analysis, we show the most frequent phrase mappings in Appendix B.
We also explore an option of adding a language-model score during beam search as the traditional statistical machine translation does.
This option might not make much sense in attention-based approaches, since the decoder itself is usually a neural network language model.
In SWAN, however, there is no language models directly involved in the segmentation modeling,
and we find it useful to have an external language model during beam search.
We use a 4-order language model trained using KenLM implementation (Heafield et al., 2013) for English target training data.
So the final beam search score we use is
[Equation]
where we empirically find that λ 1 = 1 . 2 and λ 2 = 0 .
2 gives good performance, which is tuned on the development set.
If no external language models are used, we set λ 2 = 0 .
This scoring function is similar to the one for speech recognition in Hannun et al.
3.2 IWSLT14 E NGLISH -G ERMAN
We also evaluate our model on the opposition direction, English-German, which translates from a more segmented text to a more inflectional one.
Following the setup in Section 3.1, we use the same dataset with the opposite source and target languages.
We use the same model architecture,
SWAN thank you , but $ the best thing is still coming $ .
danke , aber das beste noch kommt .
Soft reordering + Bi-directional RNN
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
die
[Table]
vondna
[Table]
herstellen
[Table] [Table]
bis
[Table] [Table]
lang
[Table]
aber
[Table]
ein
UNK
[Table]
prozess
[Table] [Table]
also 30 je
[Table]
man
[Table]
ein
[Table]
st¨uck
[Table]
,umso 38 mehr 39 fehler 40 passieren 41 .
[Table]
1 there are • 3 tens of thousands of • 4 machines • 6 around • 8 the world • 9 can make •little • 11 pieces • 12 of • 13 dna • 16 , • 17 30 • 18 to • 19 50 • 20 letters • 21 long • 22 , but • 23 it ’s • 26 a more UNK • 27 process • 28 , • 29 so • 31 the longer • 32 you make •a piece • 36 , • 38 the more • 39 mistakes • 40 happen • 41 .
target ground truth
there are tens of thousands of machines around the world that make small pieces of dna – 30 to 50 letters - in length - and it ’s a UNK process , so the longer you make the piece , the more errors there are .
optimization algorithm and beam search size as the German-English translation task.
NPMT takes about 2–3 days to run to convergence (40 epochs) on a 4 M40 GPU machine.
Given there is no previous sequence-to-sequence attention model baseline for this setup
[Table]
we create a strong one and tune hyperparameters on the development set.
The results are shown in Table 3.
Based on the development set
[Table]
we set λ 1 = 1 and λ 2 = 0 .
15 in Eq.
Our model outperforms sequence-to-sequence model with attention by 2.46 BLEU and 2.49 BLEU in greedy and beam search cases.
We can also use a 4-order language model trained using KenLM implementation for
German target training data, which further improves the performance.
Some sampled examples are shown in Table 4.
Several informative segments/phrases can be found in the decoding results, e
[Table]
g., “some time ago” → “vor enniger zeit”.
[Table]
[Table]
3.3 IWSLT15 E NGLISH -V IETNAMESE
In this section, we evaluate our model on the IWSLT 2015 English to Vietnamese machine translation task.
The data is from translated TED talks, and the dataset contains roughly 133K training sentence pairs provided by the IWSLT 2015 Evaluation Campaign (Cettolo et al., 2015).
Following the same preprocessing steps in Luong & Manning (2015); Raffel et al.
(2017), we use the TED tst2012 (1553 sentences) as a validation set for hyperparameter tuning and TED tst2013 (1268 sentences) as a test set.
We use the one reordering layer with window size 7, two-bi-directional LSTM (Long shortterm memory, Hochreiter & Schmidhuber (1997)) encoder with 512 hidden units, and a threeunidirectional LSTM decoder with 512 hidden units.
We add dropout with a rate of 0 . 4 in the
LSTM layer.
We chose LSTM since baselines for comparisons were using LSTM.
The maximum segment length is set to 7.
Batch size is set as 48 and the Adam algorithm Kingma & Ba (2014) is used for optimization with initial learning rate as 0.001.
For decoding, we use greedy decoding and beam search with a beam size of 10.
The results are shown in Table 5.
Based on the development set, we set λ 1 = 0 .
7 and λ 2 = 0 .
15 in Eq.
NPMT takes about one day to run to convergence
(15 epochs) on a 4 M40 GPU machine.
Our model outperforms sequence-to-sequence model with attention by 1.41 BLEU and 1.59 BLEU in greedy and beam search cases.
We also use a 4-order language model trained using KenLM implementation for Vietnamese target training data, which further improves the BLEU score.
Note that our reordering layer relaxes the monotonic assumption as in Raffel et al.
and is able to decode in linear time.
Empirically we outperform models with monotonic attention.
Table 6 shows some sampled examples
[Table] [Table]
C ONCLUSION
We proposed NPMT, a neural phrase-based machine translation system that models phrase structures in the target language using SWAN.
We also introduced a local reordering layer to mitigate the monotonic alignment requirement in SWAN.
Our experimental results showed promising results on a IWSLT German-English, English-German, English-Vietnamese translation tasks.
The results suggest that NPMT can potentially be extended to explore the structures in other challenging sequence-to-sequence problems.
In future work, we will explore two directions: 1) speed up NPMT and apply it to larger datasets and more language pairs; 2) investigate how to learn input and output phrases simultaneously [Table]
A R EORDERING L AYER A NALYSIS
To further understand the behavior of the reordering layer, we examine the values of the gate σ (cid:0) w Ti [ e t − τ ; . . .
; e t ; . . .
; e t + τ ] (cid:1) in Eq.
We study the NPMT English-German model in Section 3.2.
In Figure 5, we show an example that translates from “can you translate it ?”
to ”k¨onnen man es ¨ubersetzen ?”, where the mapping between words are as follows: “can → k¨onnen”, “you → man”, “translate → ¨ubersetzen”, “it → es” and “?
→ ?”.
Note that the example needs to be reordered from “translate it” to ”es ¨ubersetzen”.
Each row of Figure 5 represents a window of size 7 that is centered at a source sentence word.
The values in the matrix represent the gate values for the corresponding words.
The gate values will later be multiplied with the embedding e t − τ
+ i of
Eq.
(2) and contribute to the hidden vector h t .
The y-axis represents the word/phrases emitted from the corresponding position.
We can observe that the gates mostly focus on the central word since the first part of the sentence only requires monotonic alignment.
Interestingly, the model outputs “$” (empty) when the model has the word “translate” in the center of the window.
Then, the model outputs “es” when the model encounters “it”.
Finally, in the last window (top row), the model not only has a large gate value to the center input “?”, but the model also has a relatively large gate value to the word “translate” in order to output the translation “¨ubersetzen ?”.
This shows an example of the reordering effect achieved by using the gating mechanism of the reordering layer.
[Figure]
B P HRASE MAPPING EXAMPLES
Following the examples of Table
[Figure]
we analyze the decoding results on the test set of the GermanEnglish translation task.
Given we do not have explicit input segments in NPMT
[Figure]
we assume input words that emit “$” symbol are within the same group as the next non-’$’ word.
For example, in
Figure
[Figure]
input words “das beste” are considered as an input segment.
We then can aggregate all the input, output segments (phrases) and sort them based on the frequency.
Tables 7 and 8 show the most-frequent input, output phrase mappings
[Figure]
One → One
One → Many Many → One
Many → Many
Many → Many ∗ , → , es → it ’s , dass → that die UNK → the UNK wissen sie → you know .
→ .
UNK → the UNK in der → in der UNK → the UNK in diesem → in this und → and und → , and
UNK .
→ .
ein UNK
→ a UNK die welt → the world
UNK → UNK das → this is
UNK , → , das UNK → the UNK ist es → it ’s aber → but das, → that ’s , die → that eine UNK → a UNK ” .
→ . ” ”
→ ”
UNK → a UNK ist .
→ .
in UNK → in UNK ein paar → a few ist → is ich → i think in den → in den UNK → the UNK gibt es → there ’s der → of es → it was ist , → , wissen sie → you know der welt →
the world von → of dies → this is sind .
→ .
in diesem → in this die frage → the question mit → with es → there ’s , wenn → if dem UNK → the UNK haben wir → we have
Phrases with
Phrases with 4 words auf der ganzen → all over the auf der ganzen → a little bit of gibt eine menge → a lot of wei ß nicht , was → what ’s going to be dann hat er → he doesn ’t have tun , das wir → we can ’t do , die man → you can do tat , das ich → i didn ’t do das k¨onnen wir → we can do that zu verbessern , die → that can be done