Paper:
A Teacher-Student Framework for Zero-Resource Neural Machine Translation
While end-to-end neural machine translation (NMT) has made remarkable progress recently, it still suffers from the data scarcity problem for low resource language pairs and domains. In this paper, we propose a method for zero-resource NMT by assuming that parallel sentences have close probabilities of generating a sentence in a third language. Based on the assumption, we are able to train a source-to-target NMT model without parallel corpora available (“student”) guided by an existing pivot-to-target NMT model (“teacher”) on a source-pivot parallel corpus. Experimental results show that the proposed method significantly improves over a baseline pivot-based model by +3.0 BLEU points across various language pairs.
1 Introduction
Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2014), which directly models the translation process in an end-to-end way, has attracted intensive attention from the community. Although NMT has achieved state-of-theart translation performance on resource-rich language pairs such as English-French and GermanEnglish (Luong et al., 2015; Jean et al., 2015; Wu et al., 2016), it still suffers from the unavailability of large-scale parallel corpora for translating low resource languages. Due to the large parameter space, neural models usually learn poorly from low-count events, resulting in a poor choice for low resource language pairs. Zoph et al. (2016) indicate that NMT obtains much worse translation quality than a statistical machine translation
(SMT) system on low-resource languages. As a result, a number of authors have endeavored to explore methods for translating language pairs without parallel corpora available. These methods can be roughly divided into two broad categories: multilingual and pivot-based. Firat et al. (2016b) present a multi-way, multilingual model with shared attention to achieve zeroresource translation. They fine-tune the attention part using pseudo bilingual sentences for the zeroresource language pair. Another direction is to develop a universal NMT model in multilingual scenarios (Johnson et al., 2016; Ha et al., 2016). They use parallel corpora of multiple languages to train one single model, which is then able to translate a language pair without parallel corpora available. Although these approaches prove to be effective, the combination of multiple languages in modeling and training leads to significantly increased complexity.
Another direction is to achieve source-to-target NMT without parallel data via a pivot, which is either text (Cheng et al., 2016) or image (Nakayama and Nishida, 2016b). Cheng et al. (2016) propose a pivot-based method to zero-resource NMT: it first translates the source language to a pivot language, which is then translated to the target language. Nakayama and Nishida (2016b) show that using multimedia information as pivot also significantly benefit zero-resource translation. However, pivot-based approaches usually need to divide the decoding process into two steps, which is not only more computationally expensive, but also potentially suffers from the error propagation problem (Zhu et al., 2013).
In this paper, we propose a new method for zero-resource neural machine translation. Our method assumes that parallel sentences should have close probabilities of generating a sentence in a third language. To train a source-to-target NMT
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
(a) (b)
X YZ Z
X
Y
P (z|x;✓x!z) P (y|z;✓z!y)
P (y|z;✓z!y)
P (y|x;✓x!y)
Figure 1: (a) The pivot-based approach and (b) the teacher-student approach to zero-resource neural machine translation. X, Y, and Z denote source, target, and pivot languages, respectively. We use a dashed line to denote that there is a parallel corpus available for the connected language pair. Solid lines with arrows represent translation directions. The pivot-based approach leverages a pivot to achieve indirect source-to-target translation: it first translates x into z, which is then translated into y. Our approach directly trains the intended source-to-target model (“student”) on a source-pivot parallel corpus, with the guidance of an existing pivot-to-target model (“teacher”). Our training algorithm is based on the translation equivalence assumption: if x is a translation of z, then P (y|x;θx→y) should be close to P (y|z;θz→y).
model without parallel corpora (“student”), we leverage an existing pivot-to-target NMT model (“teacher”) to guide the learning process of the student model on a source-pivot parallel corpus. As compared with the pivot-based approaches (Cheng et al., 2016; Nakayama and Nishida, 2016b), our method allows direct modeling of the intended NMT model, without the need to divide training or decoding into two steps. This strategy not only improves efficiency but also avoids error propagation in decoding. Experiments on the Europarl and WMT datasets show that our approach achieves significant improvements in terms of both translation quality and decoding efficiency over a baseline pivot-based approach to zeroresource NMT on Spanish-French and GermanFrench translation tasks.
2 Background
Neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2014) advocates the use of neural networks to model the translation process in an end-to-end manner. As a data-driven approach, NMT treats parallel corpora as the major source for acquiring translation knowledge.
Let x be a source-language sentence and y be a target-language sentence. We use P (y|x;θx→y) to denote a source-to-target neural translation model, where θx→y is a set of model parame-
ters. Given a source-target parallel corpus Dx,y, which is a set of parallel source-target sentences, the model parameters can be learned by maximizing the log-likelihood of the parallel corpus:
θ̂x→y = argmax θx→y { ∑ 〈x,y〉∈Dx,y logP (y|x;θx→y) }
Given learned model parameters θ̂x→y, the decision rule for finding the translation with the highest probability for a source sentence x is given by
ŷ = argmax y
{ P (y|x; θ̂x→y) } (1)
As a data-driven approach, NMT heavily relies on the availability of large-scale parallel corpora to deliver state-of-the-art translation performance (Wu et al., 2016). Zoph et al. (2016) report that NMT obtains much lower BLEU scores than SMT if only small-scale parallel corpora are available. The heavy dependence on the quantity of training data poses a severe challenge for NMT to translation zero-resource language pairs.
Simple and easy-to-implement, pivot-based methods have been widely used in SMT for translating zero-resource language pairs (De Gispert and Marino, 2006; Cohn and Lapata, 2007; Utiyama and Isahara, 2007; Wu and Wang, 2007;
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
Bertoldi et al., 2008; Wu and Wang, 2009; Zahabi et al., 2013; Kholy et al., 2013). As pivotbased methods are agnostic to model structures, they have been adapted to NMT recently (Cheng et al., 2016; Nakayama and Nishida, 2016b).
Figure 1 illustrates the basic idea of pivot-based approaches to zero-resource NMT (Cheng et al., 2016; Nakayama and Nishida, 2016b). Let X, Y, and Z to denote source, target, and pivot languages. We use dashed lines to denote language pairs with parallel corpora available and solid lines with arrows to denote translation directions.
Intuitively, the source-to-target translation can be indirectly modeled by bridging two NMT models via a pivot:
P (y|x;θx→z,θz→y) = ∑ z P (z|x;θx→z)P (y|z;θz→y) (2)
As shown in Figure 1, the pivot-based approaches assume that the source-to-target parallel corpus Dx,z and the pivot-to-target parallel corpus Dz,y are available. As it is impractical to enumerate all possible pivot sentences, the two NMT models are trained separately in practice:
θ̂x→z = argmax θx→z { ∑ 〈x,z〉∈Dx,z logP (z|x;θx→z) }
θ̂z→y = argmax θz→y { ∑ 〈z,y〉∈Dz,y logP (y|z;θz→y) }
Due to the exponential search space of pivot sentences, the decoding process of translating an unseen source sentence x has to be divided into two steps:
ẑ = argmax z
{ P (z|x; θ̂x→z) } (3)
ŷ = argmax y
{ P (y|ẑ; θ̂z→y) } (4)
Despite its simplicity, the above two-step decoding process potentially suffers from the error propagation problem (Zhu et al., 2013): the translation errors made in the first step (i.e., source-topivot translation) will affect the second step (i.e., pivot-to-target translation).
Therefore, it is necessary to explore methods for direct modeling of source-to-target translation without parallel corpora available.
3 Approach
3.1 Assumptions
In this work, we propose to directly model the intended source-to-target neural translation based on a teacher-student framework. The basic idea is to use a pre-trained pivot-to-target model (“teacher”) to guide the learning process of a source-to-target model without training data available (“student”) on a source-pivot parallel corpus.
As shown in Figure 1(b), we still assume that a source-to-pivot parallel corpus Dx,z and a pivot-to-source parallel corpus Dz,y are available. Unlike pivot-based approaches, we first use the pivot-to-target parallel corpus Dz,y to obtain a teacher model P (y|z; θ̂z→y), where θ̂z→y is a set of learned model parameters. Then, the teacher model “teaches” the student model P (y|x;θx→y) based on the following assumption. Assumption 1 If a source sentence x is a translation of a pivot sentence z, then the probability of generating a target sentence y from x should be close to that from its counterpart z.
We can move a further step to introduce a wordlevel assumption:
Assumption 2 If a source word x is a translation of a pivot word z, then the probability of generating a target sentence y from x should be close to that from its counterpart z.
The two assumptions are empirically verified in our experiments (see Table 2). In the following subsections, we will introduce two approaches to zero-resource neural machine translation based on the two assumptions.
3.2 Sentence-Level Teaching
Given a source-pivot parallel corpus Dx,z , our training objective based on Assumption 1 is defined as follows:
JSENT(θx→y) = ∑ 〈x,z〉∈Dx,z KL ( P (y|z; θ̂z→y) ∣∣∣∣∣∣P (y|x;θx→y)) (5) where the KL divergence sums over all possible target sentences:
KL ( P (y|z; θ̂z→y) ∣∣∣∣∣∣P (y|x;θx→y)) =
∑ y∈Y P (y|z; θ̂z→y) log P (y|z; θ̂z→y) P (y|x;θx→y) (6)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
As the teacher model parameters are fixed, the training objective can be equivalently written as
JSENT(θx→y) = − ∑ 〈x,z〉∈Dx,z Ey|z;θ̂z→y [ logP (y|x;θx→y) ] (7)
In training, our goal is to find a set of source-totarget model parameters that maximizes the training objective:
θ̂x→y = argmax θx→y
{ JSENT(θx→y) } (8)
However, a major difficulty is the intractability in calculating the gradients because of the exponential search space of target sentences. To address this problem, it is possible to construct a subspace by either sampling (Shen et al., 2016), generating a k-best list (Yong et al., 2016) or mode approximation (Kim and Rush, 2016). Then, standard stochastic gradient descent algorithms can be used to optimize model parameters.
3.3 Word-Level Teaching
Instead of minimizing the KL divergence between the teacher and student models at the sentence level, we further define a training objective based on Assumption 2 at the word level:
JWORD(θx→y) = ∑ 〈x,z〉∈Dx,z Ey|z;θ̂z→y [ J(x,y, z, θ̂z→y,θx→y) ] (9)
where
J(x,y, z, θ̂z→y,θx→y)
= |y|∑ j=1 KL ( P (y|z,y<j ; θ̂z→y) ∣∣∣∣∣∣ P (y|x,y<j ;θx→y) ) (10)
Equation (9) suggests that the teacher model P (y|z,y<j ; θ̂z→y) “teaches” the student model P (y|x,y<j ;θx→y) in a word-by-word way. Note that the KL-divergence between two models is defined at the word level:
KL ( P (y|z,y<j ; θ̂z→y) ∣∣∣∣∣∣P (y|x,y<j ;θx→y)) = ∑ y∈Vy P (y|z,y<j ; θ̂z→y) log P (y|z,y<j ; θ̂z→y) P (y|x,y<j ;θx→y)
Corpus Direction Train Dev. Test
Europarl Es→ En 850K 2,000 2,000 De→ En 840K 2,000 2,000 En→ Fr 900K 2,000 2,000
WMT Es→ En 6.78M 3,003 3,003 En→ Fr 9.29M 3,003 3,003
Table 1: Data statistics. For the Europarl corpus, we evaluate our approach on Spanish-French (EsFr) and Germany-French (De-Fr) translation tasks. For the WMT corpus, we our approach on the Spanish-French (Es-Fr) translation task. English is used a pivot language in all experiments.
Therefore, our goal is to find a set of source-totarget model parameters that maximizes the training objective:
θ̂x→y = argmax θx→y
{ JWORD(θx→y) } (11)
We use similar approach for approximating the full search space with sentence-level teaching.
4 Experiments
4.1 Setup
We evaluate our approach on the Europarl and WMT corpora. To compare with the pivot-based methods, we use the same dataset as in the paper (Cheng et al., 2016). The evaluation metric is case-insensitive BLEU (Papineni et al., 2002) as calculated by the multi-bleu.perl script. All the experiments treat English as the pivot language and French as the target language.
For the Europarl corpus (Koehn, 2005), we evaluate our proposed model on Spanish-French (Es-Fr) and Germany-French (De-Fr) translation tasks in a zero-resource scenario. To build nonoverlapping source-to-pivot and pivot-to-target datasets, we split pivot sentences shared by the original source-to-pivot and pivot-to-target corpora into two equal parts and merge them separately to the remaining source-to-pivot and pivotto-target corpora. Both the development and test datasets are from shared task 2006. All the sentences are tokenized by the tokenize.perl script. To deal with out-of-vocabulary words, we also adopt byte pair encoding (BPE) (Sennrich et al., 2016) to split words into sub-words. The size of sub-words is set to 30K for each language.
For the WMT corpus, we evaluate our approach on a Spanish-French (Es-Fr) translation task with
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Approx. P (y|x;θx→y) .
random init. trained
JSENT greedy 313.0 53.5 beam 323.5 51.8 JWORD greedy 274.0 37.1 beam 288.7 36.5 sampling 268.6 40.5
Table 2: Approximate KL divergence from source-to-target to pivot-to-target model. We use a random initialized source-to-target model for comparison.
a zero-resource setting. We combine the following corpora to form the Es-En and En-Fr parallel corpora: Common Crawl, News Commentary, Europarl v7 and UN. All the sentences are tokenized by the tokenize.perl script. Newstest2011 serves as the development set and newstest2012 and newstest2013 serve as test sets. We use case-sensitive BLEU to evaluate translation results. BPE is also used to reduce the vocabulary size. The size of sub-words is set to 43K, 33K, 43K for Spanish, English and French, respectively. See Table 1 for detailed statistics for both Europarl and WMT corpora.
We leverage a tutorial NMT open-source code implemented by Theano for all the experiments. 1 and compare our approach with state-of-the-art multilingual method (Firat et al., 2016b) and pivotbased method (Cheng et al., 2016) with the following variations:
1. Sentence-Level Teaching: for simplicity, we use mode as suggested in (Kim and Rush, 2016) to approximate the target sentence space, which is the result of running beam search on the pivot sentences and taking the highest-scoring target sentence with the teacher model. Beam size with K = 1 (greedy decoding) and K = 5 are investigated in our paper, denoted as sent-greedy and sent-beam, respectively.
2. Word-Level Teaching: we use the same mode approximation approach as in sentence-level teaching, denoted as word-greedy (beam search with K=1) and word-beam (beam search with K=5) respectively. Besides, Monte Carlo estimation by sampling from the
1dl4mt-tutorial: https://github.com/nyu-dl
teacher model is also investigated since it introduces more diverse data, denoted as wordsampling.
4.2 Assumptions Verification
To verify the assumptions in Section 3.1, we train a source-to-target translation model P (y|x;θx→y) and a pivot-to-target translation model P (y|z;θz→y) using trilingual corpus. Then we use these two models to calculate JSENT and JWORD, which reveals how close the sentencelevel and word-level distributions are, respectively. Since it is difficult to measure JSENT and JWORD directly , we instead calculate their approximations as illustrated in Section 4.1. We report average KL divergence with different approximations evaluated on the aligned trilingual development set Exzy with 2000 sentences from shared task 2006 as shown in Table 1. We also report KL Divergence from a randomly initialized NMT model Prandom to pivot-to-target P (y|z,θz→y) for comparison.
The results agree with our assumptions since significantly smaller KL divergence is observed for KL ( P (y|z;θz→y)||P (y|x;θx→y) ) . For sentence-level distributions, smaller KL divergence for beam search approximation is observed compared with greedy approximation, indicating that mode produced by the beam search results of P (y|z;θz→y) have higher probabilities in P (y|x;θx→y) compared with the greedy results. Thus, we suspect that the sent-beam method will surpass the sent-greedy method. For word-level results, KL divergence of sampling approximation is the highest, indicating poorer performance than the other two methods. However, sampling methods introduce more data diversity at the target side. Thus it is harder to decide which factor dominates the training process. We leave further discussion to Section 4.3.
4.3 Results on the Europarl Corpus
4.3.1 Comparison with pivot-based methods
Table 3 gives BLEU scores on the Europarl corpus of our best performing sentence-level method (sent-beam) and word-level method (wordsampling) compared with pivot-based methods (Cheng et al., 2016). The same data preprocessing are applied in (Cheng et al., 2016) as in our experiments. We find that our sentence-level method and word-level method outperform all of their zero-resource approaches across language
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Method BLEU
Es→ Fr De→ Fr
Cheng.# pivot 29.79 23.70 hard 29.93 23.88 soft 30.57 23.79 likelihood† 32.59 25.93 Ours sent-beam 31.64 24.39 word-sampling 33.86 27.03
Table 3: Comparison with previous work on Spanish-French and German-French translation tasks from Europarl corpus. English is treated as the pivot language. † denotes using parallel source-to-target corpus. # denotes methods in Cheng et al. (2016).
pairs. Our word-sampling method improves their best zero-resource results on Spanish-French translation by 3.29 BLEU and Germany-French translation by 3.15 BLEU. Besides that, our word-sampling method surprisingly obtains improvement over the likelihood method, which leverages source-to-target parallel corpus and also takes much longer to train. The significant improvements can be explained by the error propagation problem of pivot-based methods that translation error of the source-pivot translation system will be transferred to the pivot-target translation.
4.3.2 Comparison of Sentence-Level and Word-Level Methods
Table 4 shows BLEU scores on the Europarl corpus of our proposed five methods.
For sentence-level approaches, sent-beam outperforms sent-greedy by 0.59 BLEU over es→ fr translation and 2.51 BLEU over de→ fr translation. The results are in line with our observations in Section 4.2 that sentence-level KLD by beam approximation is smaller than that by greedy approximation. However, as the time complexity grows linearly in the number of beams k, the better performance is achieved at the expense of beam search time.
For word-level experiments, we observe that word-sampling performs much better than the other two methods: 1.94 BLEU on es→ fr translation and 2.65 BLEU on de→ fr translation; 1.88 BLEU on es→ fr translation and 2.84 BLEU on de→ fr translation. Althrough word-level KLD calculated by Monte Carlo estimation is larger
Methods #sents
BLEU de-en en-fr de-fr
MLE − − 840K 26.06 - - 1.5M 29.02 Ours 840K 900K - 27.03
Table 5: Comparison on German-French translation tasks from Europarl corpus with stardard NMT.
than that by mode approximation, word-sampling introduces more data diversity for training, which dominates the effect of KLD difference.
4.3.3 BLEU Score over Iterations
Figure 3 shows valid loss 2 and BLEU score over iterations. We observe that word-level models tend to have lower valid loss compared with sentencelevel methods. Generally, models with lower valid loss are inclined to have higher BLEU. Our results indicate that this is not necessarily the case: sentbeam converges to +0.82 the BLEU on validation set with -13 valid loss compared with the wordbeam method. Kim and Rush (2016) claims a similar observation in data distillation for NMT and provides an explanation that student distributions are more peaked for sentence-level models. This is indeed the case in our result: on Germany-French translation task the argmax for the sent-beam student model (on average) accounts approximately for 3.49% of the total probability mass, while the corresponding number is 1.25% for the wordbeam student model and 2.60% for the teacher model. However, argmax for the sentence-level model in (Kim and Rush, 2016) comprises around 15% output probability, while ours only accounts for 3.49%. It possibly reveals that it is more diffi-
2Valid loss: the average NLL of sentence pairs on the validation set.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Method # Training Sents
# Para. BLEU
Es→ En En→ Fr Newstest2012 Newstest2013 Existing zero-resource NMT systems
Cheng et al. (2016)† pivot 6.78M 9.29M - 19.81 - Cheng et al. (2016)† likelihood 6.78M 9.29M 100K 21.48 - Firat et al. (2016b) one-to-one 34.71M 65.77M - 17.59 17.61 Firat et al. (2016b)† many-to-one 34.71M 65.77M - 21.33 21.19
Our zero-resource NMT systems word-sampling 6.78M 9.29M - 25.26 25.06
Table 6: Comparison with previous work on Spanish-French translation in zero resource scenario over WMT corpus. The BLEU scores are case sensitive. †: the pivot path involved in the translation path during testing.
cult to model around the teacher’s mode for crosstask teaching than in data distillation.
4.3.4 Comparison with Standard NMT
Table 5 gives the comparison on German-French zero-resource translation between word-sampling approach and standard NMT model trained by MLE. Our proposed method even slightly outperforms standard NMT system trained with parallel source-to-target corpus when this corpus contains similar number of sentences as source-topivot corpus, demonstrating that the effectiveness of our methods to learn from teacher model. This phenomenon can be explained by that the total size of corpus involved in our method is twice that of standard MLE. The translation knowledge contained in other language pairs benefit source-totarget translation, as observed in multilingual scenario (Johnson et al., 2016; Ha et al., 2016).
4.3.5 BLEU Score with Low-Resource Source-to-Pivot Corpus
Our proposed method can also be applied to zeroresource NMT with low source-to-pivot corpus. Specifically, the size of source-pivot corpus is magnitude smaller than that of pivot-target corpus. This setting makes sense in applications. For example, there are significantly fewer Urdu-English corpus available than English-French corpus.
To fulfill this task, we combine our best performing word-sampling method with the initialization and parameter freezing strategy proposed in (Zoph et al., 2016). We set the size of DeEn corpus to be 100K and use the same teacher model trained with 900K corpus. Table 7 gives the BLEU score of our method on De-Fr translation compared with three other methods. Noting that our task is much harder than (Zoph et al.,
2016) (transfer) since they are given parallel DeFr corpus directly. Surprisingly, our method outperforms all other methods. We significantly improve the baseline pivot method by 5.63 BLEU and the state-of-the-art low resource method transfer by 0.56 BLEU. We leave further evaluation of this method for future work. 3
4.4 Results on the WMT Corpus
Word-sampling method obtains the best performance in our five proposed approaches according to experiments on the Europarl corpus. To further investigate this approach, we conduct experiment on large scale WMT corpus for es→ fr translation. Table 6 shows the results of our word-sampling method in comparison with other state-of-the-art baselines. Cheng et al.(2016) use the same training&develpment sets and the same
3We test initialization trick in our original settings and observe that with initialization, the BLEU starts to increase very fast (21.25 Bleu with 10000 iterations while without initialization, the number is 2.5). However the model converges to similar point as without initialization.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
0 3 6 9 1 2 1 53 0
6 0
9 0
1 2 0
1 5 0
1 8 0
2 1 0
0 3 6 9 1 2 1 50
5
1 0
1 5
2 0
2 5 3 0 V a lid L o ss
I t e r a t i o n s
s e n t  g r e e d y s e n t  b e a m w o r d  g r e e d y w o r d  b e a m w o r d  s a m p l i n g
×1 04 ×1 04
B L
E U
I t e r a t i o n s
s e n t  g r e e d y s e n t  b e a m w o r d  g r e e d y w o r d  b e a m w o r d  s a m p l i n g
Figure 2: Validation loss and BLEU across iterations of our proposed methods.
preprocessing as ours. Firat et al.(2016b) uses a much larger training sets. Our method obtains significant improvement over the pivot baseline by 5.45 BLEU on newstest2012 and over many-toone 3.87 BLEU on newstest2013, not to mention that both method are depending on source-pivottarget decoding path.
5 Related Work
Training NMT system in zero-resource scenario by leveraging some other languages has attracted intensive attention in last year. Firat et al. (2016b) propose an approach which genuinely delivers the multi-way, multilingual NMT model proposed by (Firat et al., 2016a) to zero-resource translation. They use the multi-way NMT model trained by other language pairs to generate pseudo parallel corpus and fine-tune the attention mechanism of the multi-way NMT model to enable zero-resource translation. Several authors propose a universal encoder-decoder network in multilingual scenarios to perform zero-shot learning. This universal model extracts translation knowledge from multiple different languages, making zero-resource translation feasible without directly training (Johnson et al., 2016; Ha et al., 2016).
Besides multilingual NMT, another import line of research is pivot-based methods. Inspired by the widely used pivot idea in SMT (De Gispert and Marino, 2006; Cohn and Lapata, 2007; Utiyama and Isahara, 2007; Wu and Wang, 2007; Bertoldi et al., 2008; Zahabi et al., 2013; Kholy et al., 2013), Cheng et al.(2016) propose pivotbased NMT by simultaneously improving source
to pivot and pivot to target translation quality in order to improve source to target translation quality. Nakayama and Nishida (2016a) achieve zeroresource machine translation by utilizing image as pivot.
In (Kim and Rush, 2016), the author first introduce knowledge distilling in neural machine translation. They suggest to generate pseudo corpus to train the student network. Our work is highly related to theirs. However, we focus on zeroresource learning instead of model compression.
6 Conclusion
In this paper, we propose a novel framework to transfer the knowledge of the teacher model trained with rich-resource language pair into the student model with zero-resource with the help of source-to-pivot corpus. We introduce sentencelevel and word-level teaching to guide the learning process of the student model. Our experiments on Europarl Corpus and WMT corpus across languages show that our proposed word-level sampling method can significantly outperforms the state-of-the-art pivot-based methods and multilingual methods in terms of both translation quality and decoding efficiency.
We also analyze zero-resource translation with low source-to-pivot corpus, and combine our word-level sampling method with initialization and parameter freezing suggested by (Zoph et al., 2016). The experiments on Europarl Corpus show that our approach obtains an significant improvement over pivot method significantly.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Review:
This paper proposes a novel strategy for zero-resource translation where (source, pivot) and (pivot, target) parallel corpora are available. A teacher model for p(target|pivot) is first trained on the (pivot, target) corpus, then a student model for p(target|source) is trained to minimize relative entropy with respect to the teacher on the (source, pivot) corpus. When using word-level relative entropy over samples from the teacher, this approach is shown to outperform previous variants on standard pivoting, as well as other zero-resource strategies.  This is a good contribution: a novel idea, clearly explained, and with convincing empirical support. Unlike some previous work, it makes fairly minimal assumptions about the nature of the NMT systems involved, and hence should be widely applicable.  I have only a few suggestions for further experiments. First, it would be interesting to see how robust this approach is to more dissimilar source and pivot languages, where intuitively the true p(target|source) and p(target|pivot) will be further apart. Second, given the success of introducing word-based diversity, it was surprising not to see a sentence n-best or sentence-sampling experiment. This would be more costly, but not much more so since you re already doing beam search with the teacher. Finally, related to the previous, it might be interesting to explore transition from word-based diversity to sentence-based as the student converges and no longer needs the signal from low-probability words.  Some further comments:  line 241: Despite its simplicity to Due to its simplicity  277: target sentence y to target word y  442: I assume that K=1 and K=5 mean that you compare probabilities of the most probable and 5 most probable words in the current context. If so, how is the current context determined - greedily or with a beam?  Section 4.2. The comparison with an essentially uniform distribution doesn t seem very informative here: it would be extremely surprising if p(y|z) were not significantly closer to p(y|x) than to uniform. It would be more interesting to know to what extent p(y|z) still provides a useful signal as p(y|x) gets better. This would be easy to measure by comparing p(y|z) to models for p(y|x) trained on different amounts of data or for different numbers of iterations. Another useful thing to explore in this section would be the effect of the mode approximation compared to n-best for sentence-level scores.  555: It  odd that word beam does worse than word greedy, since word beam should be closer to word sampling. Do you have an explanation for this?  582: The claimed advantage of sent-beam here looks like it may just be noise, given the high variance of these curves.