switch -GLAT: M ULTILINGUAL P ARALLEL M ACHINE T RANSLATION VIA C ODE -S WITCH D ECODER
A BSTRACT
Multilingual machine translation aims to develop a single model for multiple language directions.
However, existing multilingual models based on Transformer are limited in terms of both translation performance and inference speed.
In this paper, we propose switch -GLAT, a non-autoregressive multilingual machine translation model with a code-switch decoder.
It can generate contextual codeswitched translations for a given source sentence, and perform code-switch backtranslation, greatly boosting multilingual translation performance.
In addition, its inference is highly efﬁcient thanks to its parallel decoder.
Experiments show that our proposed switch -GLAT outperform the multilingual Transformer with as much as 0 .
74 BLEU improvement and 6 .
2 x faster decoding speed in inference.
I NTRODUCTION
Neural machine translation (NMT) is the current state-of-the-art approach (Bahdanau et al., 2014;
Sutskever et al., 2014) for machine translation in both academia (Bojar et al., 2017) and industry (Hassan et al., 2018).
Recent works (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Lin et al., 2020) extend the approach to support multilingual translation, i.e. training a single model that can translate across multiple language directions.
Multilingual models are appealing for several reasons.
First, they can reduce the online translation service number, enabling simpler deployment (Arivazhagan et al., 2019) when plenty of translation directions are required.
Additionally, multilingual training makes it possible to transfer knowledge from high-resource languages to low-resource ones, thus improving the translation quality of low-resource directions (Zoph et al., 2016; Johnson et al., 2017;
Wang & Neubig, 2019).
However, most multilingual NMT systems are built upon the autoregressive architecture, which translates from left to right and thus is not efﬁcient enough in terms of translation speed.
Such efﬁciency problem is more serious in multilingual setting because all translation directions suffer from this slow inference speed.
A straightforward solution to improve the multilingual translation efﬁciency is to develop multilingual non-autoregressive translation (NAT).
NAT generates translation outputs in parallel (Gu et al., 2018), which leads to signiﬁcantly faster translation speed.
Thanks to the recent progress of
NAT (Ghazvininejad et al., 2019; Gu et al., 2019; Deng & Rush, 2020), current state-of-the-art NAT models have achieved comparable BLEU scores (Li et al., 2018; Wei et al., 2019;
Qian et al., 2020) with their auto-regressive counterparts.
Among them, the glancing transformer (GLAT) proposed by Qian et al.
(2020) is a representative work, which even outperforms many strong autoregressive translation systems in BLEU score on German-English translation task of WMT
(Qian et al., 2021).
In this paper, we argue that multilingual NAT models is not only superior in efﬁciency, but also can achieve better multilingual translation accuracy, due to its capability of generating high quality codeswitched translations (Lin et al., 2020;
Yang et al., 2020; Jose et al., 2020).
In particular, we propose switch -GLAT, a carefully designed multilingual version of GLAT, which can outperform multilingual
Transformer in both speed and translation quality.
Generally, the main idea of switch -GLAT is to
H
ParallelDecoder
[Figure]
n t i o n
Decoder
Encoder
G l a n c
i ng
S a m p li ng veryx 4 cutex 5 Thex 1 catx 2 isxh 4 süß y 5 Diey 1 h 2 istyy 4 sehr y 5 y 1 y 2 katze y
Encoder
H
ParallelDecoder
Glancing
ComputeDistance
H'
ParallelDecoder
[Figure]
words
ReplaceInput H
Compute Loss
Glancing
[Figure]
employ a code-switch decoder , which can generate contextual code-switched translations (instead of using dictionary for replacement) for a given source sentence, and then perform code-switch back-translation to boost the multilingual translation performance.
In more details, the glancing sampling module in GLAT enables switch -GLAT to generate partial translation based on others.
Then combined with its multilingual and non-autoregressive characteristics, switch -GLAT could output code-switched translations with the employment of token-level language tags instead of sentencelevel ones in auto-regressive models.
Ultimately, switch -GLAT can generate contextual translated words at arbitrary positions of the target sentence in arbitrary languages.
This greatly improves the multilingual translation performance when we reverse the pairs of source to contextual code-switched target sentences, for training in a back-translation fashion (so called code-switch back-translation).
We conduct extensive experiments on 3 merged translation datasets: WMT with four language pairs (both close languages and distant ones) and WMT with 10 language pairs.
switch -GLAT shows consistent improvements over autoregressive multilingual baselines on all datasets, validating that switch -GLAT can achieve better multilingual translation performance simultaneously with a faster decoding speed.
We further evaluated the cross-lingual representations through word induction and sentence retrieval tasks.
The results demonstrated the proposed code-switch back-translation beneﬁts better-aligned cross-lingual representations.
B ACKGROUND
Multilingual Neural Machine Translation (MNMT)
Given a source sentence X
= { x 1 , x 2 , ..., x M } with length M and its target sentence Y =
{ y 1 , y 2 , ..., y N } with length N , MNMT leverages the standard bilingual neural machine translation models and extends the source and target inputs respectively with a source and target language token src and tgt .
This results in
X (cid:48)
= { src , x 1 , x 2 , ..., x M } and Y (cid:48)
= { tgt , y 1 , y 2 , ..., y N } .
MNMT is generally modeled from X (cid:48) to Y (cid:48) with Transformer (Vaswani et al., 2017).
Transformer consists of stacked encoder and decoder layers, which are jointly trained to maximize the conditional probability of Y (cid:48) given X (cid:48) :
[Equation]
where θ are the trainable model parameters.
Glancing Transformer prposed by Qian et al.
(2020) is a NAT architecture which achieves top results in machine translation with 8 x ∼ 15 x speedup.
It performs two-pass decoding in training but is still fully non-autoregressive in inference.
In the ﬁrst decoding pass, given the encoder F e and decoder F d , H 0 d
= { h 01 , h 02 , ..., h 0 N } is the decoder input either gathered from the encoder output using soft copy (Wei et al., 2019) or full mask (Ghazvininejad et al., 2019), and then Y is predicted as:
[Equation]
where θ are the trainable model parameters.
Then glancing transformer (GLAT) adopts glancing sampling strategy to sample a subset of Y according to its distance with ˆ Y , thus resulting in GS ( Y, ˆ Y ) .
[Figure]
Finally, GLAT predicts the target sequence Y based on this subset and source sentence X in the second decoding pass as follows:
[Equation]
where H
d is updated accordingly and GS ( Y, ˆ Y ) denotes the remaining subset of Y after removing the sampled tokens.
The overall architecture of GLAT is shown in Figure 1.
The twice decoding in training makes GLAT capable of predicting some partial translations based on others.
Then combined with multilingual and non-autoregressive characteristics, GLAT has the potential to generate contextual code-switched outputs.
In the next section, we will show how to modify the GLAT architecture to achieve this.
P ROPOSED M ETHOD : switch -GLAT
This section will detail the proposed switch -GLAT.
The training objective can be factorized into two parts.
One aims to make the model have multilingual translation ability, and the other to achieve better-aligned cross-lingual representations to boost translation performance, respectively denoted as
L multi and L csbt .
The overall objective can be formulated as follows:
[Equation]
where λ plays the role of a “temperature” to schedule the importance of code-switch back-translation.
As the training progresses, λ is gradually increased, which allows for more complex data instances to be involved, i.e. code-switched translations incorporating more languages and higher switching ratios.
This process encourages model to align similar words from different languages into the same vector space, thus boosting machine translation performance.
The overall architecture is shown in Figure 2.
3. C ODE -S WITCH D ECODER
switch -GLAT leverages the general parallel decoder of GLAT and extends it into a code-switch decoder by employing the token-level language tag.
The code-switch decoder ﬁrst gains multilingual translation ability through multilingual training, and then it can generate code-switched translations in arbitrary languages with the help of token-level language tag.
Speciﬁcally, given a multilingual corpora D
= { D l } Ll =1 consisting of L language pairs, the loss
L multi is then deﬁned as:
[Equation]
where D l = { ( X lj , Y lj ) } N l j =1 is a parallel corpus with size N l and L l len ( j ) is the length prediction loss of the j -th pair.
P ( L lj ) is the real length distribution of target sentence, and ˆ P ( L lj ) is the predicted one based on the concatenation of encoder output as well as source and target language embeddings.
θ M are trainable model parameters.
L tag is the accordingly updated GLAT training loss (Equation 3) incorporating the source and target language tags:
[Equation]
To involve the indicative language tag, we add it to the ﬁrst layer input and ﬁnal layer output at each position of both encoder and decoder as follows:
[Equation]
where f
i denotes the ﬁrst encoder layer input at position i and h
j denotes the ﬁrst decoder layer input at position j .
Correspondingly, f Ki and h Kj denote the last layer output.
src and tgt are respectively the source and target individual tags, while E src and E tgt are their corresponding representations.
The overall prediction process of pair ( X lj , Y lj ) is illustrated in the left module of Figure 2.
Through the multilingual training process, switch -GLAT can translate between different languages using the indicative language tags, of which the decoder is called code-switch decoder.
It has the ability to generate contextual translated words in arbitrary languages due to its token-level characteristics.
3. C ODE -S WITCH B ACK -T RANSLATION
Thanks to the code-switch decoder, we can perform code-switch back-translation (CSBT), which is critical because it encourages model to align the produced words and the original ones into the same vector space according to their similar context information.
Better-aligned cross-lingual representations beneﬁt better translation performance.
Speciﬁcally, a subset D S = {
( X i , Y i , l src i , l tgt i ) } Si =1 of size S is ﬁrst sampled, where l src i and l tgt i are respectively source and target languages of i -th pair.
Then, Y i is masked with a given rate P M , leading to ˜ Y i .
Subsequently, the masked positions of ˜ Y i can be decoded into a third randomly sampled language by leveraging the token-level language tag.
Thus, the ﬁnal decoded sequence ˆ Y i consists of contextual tokens from mixed languages, which will in turn be taken as the source side input and the original source sentence as the target side input.
This process results in a code-switch back-translation corpus D C
= { ( ˆ Y i , X i ) }
Si =1 , which is illustrated in the middle module of Figure 2.
The dynamically generated code-switch translations can augment data instance distribution to enhance model training as illustrated in the right module of Figure 2.
Loss L csbt is deﬁned as:
[Equation]
As the training continues, the masked rate P M and number of mixed languages are gradually increased.
Speciﬁcally, the value of P M is iterated from 0 .
1 to 0 .
5 with step size 0 .
1 every 10 epochs.
In the ﬁrst iteration of P M , the number of mixed languages is set to 1 .
Afterwards, it will be increased to one-third of the total.
Through this process, abundant code-switched sentences can be generated, which helps to learn better-aligned cross-lingual representations.
3. S CHEDULED C ODE -S WITCH T RAINING
The inﬂuence of multilingual training and code-switch back-translation is balanced by λ .
As the training progresses, λ is increased and more complicated data instances are involved, which means that code-switch back-translation becomes more important.
We use the step function to evolve the value of λ :
[Equation]
where λ ( t ) is the importance value at step t and E is a pre-deﬁned changing point to incorporate code-switched translations.
λ (0) is the starting value and we set it to 0 .
As the training goes on, the code-switch back-translation and model training are iterated, leading to well-aligned cross-lingual representations and improved machine translation performance.
E XPERIMENTS 4. D ATASETS
To better measure the effects of multilingualism at different levels, we test our proposed switch -GLAT on the following three merged datasets: (1) WMT-EDF: We collect 4 language pairs from WMTEnglish (En) ↔ German (De) and English (En) ↔ French (Fr).
All three languages belong to
Indo-European language family and are relatively close on linguistics.
(2) WMT-EFZ: We also collect 4 language pairs from WMT-14 English (En) ↔ French (Fr) and WMT-17 English (En) ↔ Chinese (Zh), which are distant languages on linguistics and their relationships are more difﬁcult to learn.
WMT-many: We also gather 10 language pairs from WMT-14 English (En) ↔ German (De), English (En) ↔ French (Fr), WMT-16 English
(En) ↔ Russian (Ru), English (En) ↔ Romanian (Ro) and
WMT-17 English (En) ↔ Chinese (Zh) to test switch -GLAT on more diverse language pairs.
4. B ASELINE M ODELS
We compare switch -GLAT against the following representative multilingual baselines : (1) MTransformer : We set a multilingual transformer baseline, which also uses token-level language tags on both source and target sides.
(2) CLSR : Zhang et al.
(2021) present CLSR that learns to route between language-speciﬁc or shared pathways from the data itself to obtain the sharing structure.
(3) Adapter :
Bapna et al.
(2019) propose to inject a lightweight adapter layer for each language in
MNMT to extract some language-speciﬁc features.
We reimplement Adapter since its code is not available currently.
MNAT :
We also set a multilingual NAT baseline (Gu et al., 2018) modeled by adding token-level language tags to the vanilla NAT.
All model settings are the same as switch -GLAT.
Besides, we also report the performance of two typical bilingual models : (1) Transformer : We use the code released by Vaswani et al.
(2017) to implement Transformer.
(2) GLAT :
GLAT is the backbone of our switch -GLAT and it achieves comparable results to Transformer.
To better analyze the inﬂuence of different components in our switch -GLAT, we also conduct ablation tests as follows: (1) switch -GLAT-w/o-glancing : switch -GLAT removes glancing sampling strategy in training.
(2) switch -GLAT-w/o-CSBT : switch -GLAT is trained without code-switch back-translation, which means the model is only trained on the golden parallel corpora.
(3) switchGLAT-w/o-CCS : switch -GLAT constructs the code-switched data by simply replacing aligned words using a bilingual dictionary instead of contextualized code-switching.
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
0 ×
[Table]
[Table] [Table] [Table]
[Table]
We use a new evaluation script. Though the average BLEU scores are 0 . 0 ∼ 0 . 8 points lower than before, the overall conclusions are not changed.
[Table]
[Figure]
s
[Figure]
4. T RAINING AND I NFERENCE D ETAILS
We use 6 layers for encoder and parallel decoder.
The model hidden size d model and feed-forward hidden size d ff are set to 512 and 2048 respectively.
The number of attention head is set to 8 .
All the datasets are tokenized and segmented into subword units using BPE encodings (Sennrich et al., 2016).
The vocabulary size is set to 85 k for WMT-EDF and 95 k for WMT-EFZ/many.
Following the previous works (Gu et al., 2018; Li et al., 2020; Ma et al., 2019), the sequence-level knowledge distillation is used for all datasets, and the Transformer-base architecture (Vaswani et al., 2017) is employed as the teacher model.
The changing point E is set to 300 , 000 steps and sampling number S is set to 300 , 000 for each pair.
The mini-batch size is set to 64 k tokens and the maximum training step is 1 , 200 , 000 .
The model is trained with 8 NVIDIA Tesla V 100 GPU cards.
We follow the default parameters of Adam optimizer (Kingma & Ba, 2014) and learning rate schedule in Vaswani et al.
Dropout annealing strategy (Rennie et al., 2015) is applied to stable training and the initialized dropout rate is set to 0 .
In training, data from different language pairs are sampled according to a multinomial distribution rebalanced by a temperature of 0 .
3 (Conneau et al., 2019).
In inference, following Qian et al.
(2020), switch -GLAT performs one-iteration parallel decoding.
Moreover, we also leverage the common practice of noise parallel decoding (Gu et al., 2018) and the number of length reranking candidate is set to 5 .
4. M AIN R ESULTS 4.4. R ESULTS ON WMT-EDF/EFZ
The left and right halves of Table 1 respectively show the results of WMT-EDF and WMT-EFZ.
switch -GLAT outperforms all multilingual competitors on the average score in both settings.
Speciﬁcally, in WMT-EDF scenario, our model achieves the best average BLEU score of 33 . 34 among all multilingual competitors, especially exceeding multilingual Transformer with a fairly signiﬁcant margin of 0 .
35 point.
Moreover, it performs better than CLSR and Adapter with an improvements of 1 . 33 and 1 . 26 points respectively.
The same phenomena can be observed in WMT-EFZ scenario where the language pairs are more distant on linguistics.
Our interpretation is that switch -GLAT has the ability to produce context-dependent code-switched translations with the code-switch decoder.
The augmented code-switched translations can help model align the generated tokens and the original ones into the same vector space due to their similar context information, thus boosting machine translation performance (Mikolov et al., 2013).
switch -GLAT can largely speed up decoding.
In WMT-EDF scenario, the decoding speed tested on Fr-En pair is reported in Table 1, which is evaluated in the throughput of each model (full speed values are illustrated in Figure 3 (a) and Figure 3 (b)).
Table 1 shows that the decoding speed of switch -GLAT is 6 . 2 times faster than multilingual transformer.
Additionally, Figure 3 (a) and (b) also illustrate that the throughput of our model is signiﬁcantly larger than multilingual transformer on all directions.
These validate that the switch -GLAT can greatly improve decoding efﬁciency thanks to its parallel decoder.
Note that the length predictor is jointly trained incorporating source and target language embeddings, thus leading to different length preference in different models.
switch -GLAT performs better than GLAT on average.
In both scenarios, switch -GLAT achieves higher performance than GLAT on average, especially X-to-En directions.
These indicate that multilingual training makes it possible to transfer knowledge so that performance of X-to-En direction is enhanced due to the augmented data in English on the target side.
4.4. R ESULTS ON WMT- MANY
shows the results on the gathered WMT dataset from 10 language pairs.
switch -GLAT performs better than multilingual baselines in most cases.
It is primarily because switch -GLAT is able to produce diverse contextual code-switched translations, which encourage the model to automatically align the generated word vectors and the original ones due to their similar context information.
Well-aligned representations are beneﬁcial for improving machine translation performance.
Notably, Adapter performs much worse than multilingual Transformer on En-Ro and
En-Ru directions as the original paper interprets that Adapter do not perform well in low-resource settings (the reverse directions are enhanced by the English data from other language pairs).
switch -GLAT performs slightly worse than GLAT.
In this scenario, the failure of switch -GLAT may be owing to the limitation of model capacity.
However, switch -GLAT achieves better or comparable performance with GLAT when the model translates from a foreign language to English.
One interpretation is that all the parallel corpora including English on either source or target side can lead to a better parameter estimation of the English decoder (Firat et al., 2016).
switch -GLAT can enlarge the power of efﬁcient decoding.
The decoding speed in this setting are illustrated in Figure 3 (c), showing that the throughput of our model is much larger than multilingual transformer on 10 directions.
It demonstrates that switch -GLAT can enlarge the power of decoding efﬁciency since all translation directions can be accelerated within a single model.
4. A BLATION STUDY
The last three lines in Table 1 and Table 2 show the results of ablation study.
Glancing sampling strategy plays an important role in switch -GLAT.
After removing the glancing sampling strategy, the performance drops most.
It validates that glancing sampling is critical in training an advanced parallel multilingual translation model, which makes it suitable to generate high-quality code-switched translations.
Code-switch back-translation promotes multilingual translation.
Excluding code-switch backtranslation or employing code-switched data constructed by dictionary substitution both harms translation performance, of which the former hurts more.
It validates that applying code-switched data can signiﬁcantly increase multilingual translation performance, especially those generated by incorporating context information instead of dictionary replacing.
[Figure] [Figure]
[Table]
Transformer
[Table]
Avg means
the average accuracy.
A NALYSIS
In this section, we will make a rigorous analysis to reveal the reasons why our switch -GLAT can achieve better translation performance than multilingual transformer.
All tasks are tested in WMTEDF scenario.
5. V ISUALIZATION
There is much smaller gap in the cross-lingual representations learned by our switch -GLAT.
To intuitively illustrate how well the cross-lingual representations are aligned, we plot a two-dimensional projection of representations learned by four models in Figure 4.
Here multilingual models are trained on the merged WMT-EDF dataset, while bilingual ones on the English → German part.
We use the t-SNE algorithm (van der Maaten Laurens & Hinton, 2008) to perform the projection.
English and German words are gold word pairs from Open Multilingual WordNet (OMW) datasets (Bond &
Foster, 2013).
English words are displayed in blue color and German words in red.
We ﬁnd there is much more overlapping between blue and red areas in Figure 4 (d), validating that switch -GLAT can produce better-aligned cross-lingual representations than other models.
5. C ROSS -L INGUAL W ORD I NDUCTION
The cross-lingual representations are aligned well at word level.
We assess the cross-lingual word induction performance to see how well the similar words from different languages are close to each other in the learned vector space (Gouws et al., 2015).
Speciﬁcally, golden word pairs are extracted from the OMW datasets (Bond & Foster, 2013), resulting in 10 , 585 En-De and 12 , 511 EnFr pairs.
Top- 1 accuracy is shown in the left half of Table 3.
It shows that switch -GLAT signiﬁcantly outperforms all autoregressive multilingual competitors on average, indicating that cross-lingual representations in our model are aligned well at word level.
5. P ARALLEL S ENTENCE R ETRIEVAL
The cross-lingual representations are aligned well at sentence level.
To further assess the quality of the cross-lingual representations at sentence level, we employ parallel sentence retrieval (PSR) task.
PSR aims to extract parallel sentences from a comparable corpus between two languages.
Following the settings of XTREME (Hu et al., 2020), we use the Tatoeba dataset (Artetxe & Schwenk, 2019).
Cosine similarity is leveraged to search the nearest neighbour and the search accuracy of each model is reported in the right half of Table 3.
It can be seen that switch -GLAT achieves the highest accuracy among all competitors, validating that better contextualized embeddings are obtained in switch -GLAT as the sentence embeddings are calculated with regarding all tokens in it.
R ELATED W ORK 6. M ULTILINGUAL N EURAL M ACHINE T RANSLATION
Early works extend the NMT model proposed by Bahdanau et al.
to a multilingual version based either on LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Cho et al., 2014).
Dong et al.
(2015) propose a one-to-many translation model (from English into 4 languages) by adding a dedicated decoder per target language, showing improvements over strong single-pair baseline.
Zoph & Knight (2016) present a multi-source NMT which signiﬁcantly outperforms a single-source model.
Firat et al.
(2016) propose a many-to-many model involving 10 language pairs by leveraging separate encoder and decoder per language as well as a sharing attention mechanism.
Further, Ha et al.
(2016) and
Johnson et al.
(2017) propose to train a multilingual model with a shared encoder-decoder-attention network across all languages to perform many-to-many translation.
Recently, Transformer (Vaswani et al., 2017) is applied as the backbone of MNMT by many works.
The Multi-Distillation (Tan et al., 2019) is trained to simultaneously ﬁt the training data and match the outputs of individual bilingual models through knowledge distillation.
Bapna et al.
(2019) present to inject a language-speciﬁc adaptation layer into a pretrained model to keep the language-speciﬁc features.
Aharoni et al.
(2019) propose a multilingual NMT model involving up to 102 languages which achieves good zero-shot performance.
All the above MNMT systems are based on autoregressive architectures which follow a sequential generation process and are limited to slow decoding speed.
6. N ON - AUTOREGRESSIVE N EURAL M ACHINE T RANSLATION
Recently, NAT has attracted much attention due to its ability of parallel decoding.
A fully nonautoregressive model is ﬁrst proposed by Gu et al.
(2018), based on which latent variables are introduced to deal with dependencies among target words (Ma et al., 2019; Bao et al., 2019;
Ran et al., 2021).
Other branches of fully non-autoregressive systems either try to transfer knowledge from autoregressive models (Li et al., 2018; Wei et al., 2019;
Guo et al., 2020; Sun & Yang, 2020), or apply different training objectives (Libovick ` y & Helcl, 2018;
Shao et al., 2020; Ghazvininejad et al., 2020;
Qian et al., 2020).
Besides, some works build NAT systems with structured prediction to model interdependencies among output words.
Sun et al.
(2019) propose to incorporate a structured inference module into the non-autoregressive models.
Deng & Rush (2020) present a Markov transformer to perform cascaded decoding.
Additionally, serveral semi-autoregressive models are also constructed to reﬁne the outputs with multi-pass iterative decoding (Lee et al., 2018;
Gu et al., 2019; Ghazvininejad et al., 2019; Kasai et al., 2020).
All these works are limited to bilingual translation tasks.
To our best knowledge, switch -GLAT is the ﬁrst one to perform multilingual neural machine translation based on a non-autoregressive architecture.
C ONCLUSION
This paper proposes switch -GLAT, a non-autoregressive multilingual neural machine translation model.
It incorporates a switch-decoder, which can produce contextualized code-switched sentences and perform code-switch back-translation.
Through this process, the multilingual translation performance and cross-lingual representations can both be improved.
Besides, its parallel decoder enables a highly efﬁcient inference.
Experiments on three datasets suggest that switch -GLAT is superior to multilingual transformer in both effectiveness and efﬁciency.
Further analyses also demonstrate the effectiveness of the learned contextual cross-lingual representations.
A A PPENDIX A. C OMPARING WITH VANILLA BACK - TRANSLATION
We conduct ablation study of using vanilla back-translation ( switch -GLAT-with-vanilla BT) in
Table 4.
It shows that though the model using vanilla back-translation outperforms the one not using any augmentation strategy ( switch -GLAT-w/o-CSBT), it can be further improved by leveraging code-switch back-translation, validating that our proposed code-switch back-translation is favorable
[Table]
A. A PPLYING CSBT TO A UTOREGRESSIVE MODEL
We also apply our proposed code-switch back-translation (CSBT) to multilingual transformer, and the results are shown in Table 5.
It shows that CSBT also boosts the performance of multilingual
Transformer.
Though M-Transformer-with-CSBT performs slightly better than our switch-GLAT, our model could signiﬁcantly improve inference efﬁciency
[Table]
[Figure]
[Table]
A. T RANSFER A BILITY
To validate that our model can improve the performance of low-resource languages, we train our switch -GLAT on the subset of TED dataset that is merged from English (En)-Spanish (Es), EnglishFrench (Fr) and English-Portuguese (Pt) pairs.
En-Es and En-Fr pairs have about four times as much data as En-Pt pair.
The results are shown in Table 6.
It can be seen that our switch-GLAT can signiﬁcantly improve the performance of low-resource languages (Pt-En) with as much as 19 . 91 and 0 .
95 BLEU points respectively compared with GLAT and Transformer.
To further validate the zero-shot ability of our model, we evaluated it (trained on WMT-many) on the test set of WMT 2019 German-French.
Table 7 shows that the zero-shot ability of switch-GLAT and multilingual Transformer are both poor, but applying self pivot to these two models can achieve good performance.
[Table] [Table]
A. R ESULTS WITH A B IG M ODEL
As capacity is crucial for multilingual models
[Table]
we conduct experiments to show how switch -GLAT performance compares as model size grows.
We adopt a standard Transformer-large architecture (Vaswani et al., 2017)
[Table]
which employs 6 -layer encoder, 6 -layer decoder and 1024 dimension on 16 heads.
The results are shown in Table 8.
It shows that in a multi-direction scenario
[Table]
GLAT performs better than switch -GLAT when trained on a base model
[Table]
but increasing model size brings opposite results
[Table]
demonstrating that model capacity is quite critical for multilingual models.
A. A PPENDED V ISUALIZATION
We also visualize the representations learned by CLSR since it is also a representative multilingual baseline.
Again, we ﬁnd there is much more overlapping between blue and red areas in Figure(d), validating that switch -GLAT can produce better-aligned cross-lingual representations than other models.
A. H YPER - PARAMETER S ELECTION
We tune the training step of changing lambda (100000 to max steps with step size 100000), sample number S (100000 to max data size with step size 200000) and initialized dropout rate (0.1 to 0.
[Figure]
[Table]
with step size 0.1)with grid search on the validation set of WMT-EDF, and found that the values described in the paper performed best.
[Table]
The longest journey begins with the ﬁrst step. The longest journey begins with the ﬁrst step.
[Table]
The longest voyage begins with the ﬁrst step. The longest journey begins with the ﬁrst étape.
[Table]
Le plus long viaje begins with the ﬁrst step.
The longest journey empezar with the premier pisar.
[Table]
Die längste Reise empieza con the ﬁrst step.
The höchste viajar empezar with das ﬁrst step.
A. C ASE S TUDY
To gain an insight on how well our model can generate code-switched sentences, we provide some examples in Table 9.
It shows that constructing code-switched sentences by dictionary replacement may cause errors.
For example, pisar is a verb while the original word step is a noun, and using a verb here is not appropriate.