IOT: I NSTANCE - WISE L AYER R EORDERING FOR T RANSFORMER S TRUCTURES
A BSTRACT
With sequentially stacked self-attention, (optional) encoder-decoder attention, and feed-forward layers, Transformer achieves big success in natural language processing (NLP), and many variants have been proposed.
Currently, almost all these models assume that the layer order is ﬁxed and kept the same across data samples.
We observe that different data samples actually favor different orders of the layers.
Based on this observation, in this work, we break the assumption of the ﬁxed layer order in Transformer and introduce instance-wise layer reordering into model structure.
Our Instance-wise Ordered Transformer (IOT) can model variant functions by reordered layers, which enables each sample to select the better one to improve the model performance under the constraint of almost same number of parameters.
To achieve this, we introduce a light predictor with negligible parameter and inference cost to decide the most capable and favorable layer order for any input sequence.
Experiments on 3 tasks (neural machine translation, abstractive summarization, and code generation) and 9 datasets demonstrate consistent improvements of our method.
We further show that our method can also be applied to other architectures beyond Transformer.
Our code is released at Github 1 .
I NTRODUCTION
Transformer (Vaswani et al., 2017) has been the dominant architecture in deep learning models (Hassan et al., 2018; Ng et al., 2019; Carion et al., 2020;
Radford et al., 2019; Dai et al., 2019; Lee et al., 2019; Devlin et al., 2018;
Yang et al., 2019; Cai & Lam, 2019).
A Transformer model is stacked by several identical blocks, and each block consists of sequentially ordered layers: the self-attention ( SA ), encoder-decoder attention ( ED ) (decoder only) and feed-forward ( FF ) layer.
Recently, various modiﬁcations have been proposed, where the focus is on replacing or inserting some components (e.g., attention layer/layer norm/position encoding) in standard Transformer (Wu et al., 2019;
Lu et al., 2019; Shaw et al., 2018; So et al., 2019; Ahmed et al., 2017).
[Table]
Despite these Transformer alternatives have achieved improved performances, one critical element is almost neglected in current models, which is how to arrange the components within a Transformer network, i.e., the layer order also matters.
As pointed by He et al.
(2016b), different orders of ReLU, batch normalization and residual connection signiﬁcantly affect the performance of ResNet (He et al., 2016a).
Therefore, we ask: What if we reorder the sequential layers in Transformer (e.g., SA → FF or FF → SA of encoder, SA → FF → ED or FF → ED → SA of decoder)?
What is the best order for these different layers?
/IOT
[Table]
We ﬁrst conduct preliminary experiments.
We vary the three layers in decoder with all six variants (each with a unique order of the three layers) and train these models.
Results on IWSLT
German → English translation are reported in Table 1.
As we can see, their performances are similar and no one is outstanding.
The corpus BLEU variance is only 0 . 0045 , which means that simply reordering the layers and training over the whole corpus impacts little.
Press et al.
(2019) also reported this for machine translation, but they stopped here.
This seems to be a negative answer.
However, we take a further step and ask one more question: Does different data favor different ordered layers?
That is, we investigate whether each speciﬁc data has its own preference for one particular order.
Intuitively, putting various data patterns in one order should not be the best choice.
For example, harder samples may favor a particular order while easier ones favor another one.
Thus, for each order, we count the ratio of samples that achieve the best score with that order.
In Table 1, we ﬁnd they almost lie on a uniform distribution (e.g., 17 .
9 % samples achieve the best BLEU with order SA → ED → FF ).
Besides, we calculate the BLEU variance for each sample, and average all these variances, the result is 114 .
76 , which is much larger than above corpus variance ( 0 . 0045 ).
These both mean the data indeed has its own preference to different orders.
In Table 2, we present translations from all decoders for on example with BLEU and TER score to give an evidence.
Motivated by above observations, in this work, we present Instance-wise Ordered Transformer (IOT), in which the layer order is determined by the speciﬁc data through instance-wise learning.
To achieve this, we utilize a light predictor to predict the conﬁdence for each order, given the corresponding classiﬁcation losses as training signals.
However, directly training the predictor with conventional (i.e., NMT) loss tends to quickly converge to a bad order, and ignore explorations on others.
Thus, we introduce an exploration loss and an exploitation loss to make an effective training while keeping an unambiguous prediction for each data so that the best order can be decided during inference.
We evaluate our approach on 3 sequence generation tasks, including neural machine translation (NMT), abstractive summarization (ABS) and code generation (CG).
For NMT, we work on 8 IWSLT and 2 WMT tasks, both on low-resource and rich-resource scenarios.
Our method can consistently obtain 1 .
BELU score improvements over Transformer.
For ABS, IOT also outperforms Transformer and other baselines on Gigaword dataset.
For CG tasks, the results on 2 large-scale real-world code datasets (Java and Python) collected from Github surpass the state-of-the-art performances.
These all demonstrate the effectiveness of our IOT.
Furthermore, we provide detailed studies to verify that the instance-wise learning and order selection make a reasonable and necessary modeling.
The contributions of this work can be summarized as follows:
• We are the ﬁrst to leverage instance-wise learning for layer order selection in a Transformer model (with shared parameters), and we demonstrate the instance-wise learning is critical.
• We demonstrate our learning approach can be universally applied to other structures beside Transformer (e.g., Dynamic Convolutions), as long as there are multiple different layers.
• Experiments on 3 sequence generation tasks and datasets verify the effectiveness of IOT with consistent performance improvements.
R ELATED W ORK
Architecture Exploration Inventing novel architectures by human designing or automatic searching plays an important role in deep learning.
Speciﬁc to Transformer structures, various modiﬁcations have been proposed.
For example, human knowledge powered designs include DynamicConv (Wu et al., 2019), Macaron Network (Lu et al., 2019), Reformer (Kitaev et al., 2020) and others (Fonollosa et al., 2019;
Ahmed et al., 2017; Shaw et al., 2018).
As for automatic searching, neural architecture
[Figure]
search can discover networks with state-of-the-art performances but always with complicated computation, i.e., Evolved Transformer (So et al., 2019).
The underlying principle is to add or replace some components of Transformer.
For instance, Wu et al.
(2019) replace self-attention with dynamic convolution, So et al.
(2019) add a separate convolution layer in a new branch.
Different from them, we, instead, only focus on the selection of layer orders for each data sample so as to improve the model performance, without a heavy modiﬁcation.
Besides, our approach is structure agnostic, which can be universally applied to other structures, only if multiple different layers exist.
Instance-wise Learning Deep learning models are trained over large-scale datasets, and data samples are often treated equally without modeling the difference between them.
Some works attempt to weight each data with different importance (Ren et al., 2018;
Hu et al., 2019; Chang et al., 2017) or feed data with curriculum learning according to its difﬁculty (Bengio et al., 2009; Fan et al., 2018).
However, they often explicitly manipulate the data during training only, while no distinction exists in inference, and under one ﬁxed model.
Elbayad et al.
(2020) take a step further and propose the depth-adaptive Transformer, which can forecast different depths of the network by predicting the required computation for a particular data.
Similarly, Liu et al.
(2020) propose a sample-wise adaptive mechanism to dynamically calculate the number of required layers.
They both aim at reducing the computation cost and speed up the inference.
Schwartz et al.
, Bapna et al.
and Shazeer et al.
all leverage conditional computation for each sample to control the computation and accuracy tradeoff during inference.
Instead, we pay attention to the variant modeling functions and perform instance-wise order selection in order to boost the Transformer performance.
The most related work is Press et al.
(2019), which manually generates randomly ordered Transformer encoders and ﬁnds the Sandwich Transformer can slightly reduce the perplexity of language modeling.
However, they ﬁnd that Sandwich Transformer pattern has no effect on NMT task.
Besides, it still performs over the whole corpus without considering each speciﬁc data.
We, instead, investigate on various sequence-to-sequence generation tasks and greatly improve the task performances through instance-wise learning, so as to discover the optimal ordered Transformer for each particular data.
I NSTANCE - WISE O RDERED T RANSFORMER
The overall framework of IOT is presented in Figure 1.
In comparison with the standard Transformer,
IOT only incorporates light-weighted predictors and reorders the encoder/decoder with weight tying, under the constraint of almost same number of parameters and exempt from heavy modiﬁcations.
In this section, we introduce the details of IOT, including training, inference and discussions.
Notations Sequence-to-sequence learning aims to map one sequence x
= [ x 1 , x 2 , ..., x T x ] into another sequence y =
[ y 1 , y 2 , ..., y T y ] , where x i , y j denotes the i -th and j -th token of x and y , T x and
T y are the corresponding lengths.
Given one sentence pair ( x, y ) and a learning model M , we can deﬁne the training objective as minimizing the cross-entropy loss L M = − (cid:80) T y j =1 log P ( y j | y <
j , x ) .
Besides, D KL ( P (cid:107) Q ) denotes the Kullback-Leibler (KL) divergence between distributions P and Q .
3. I NSTANCE - WISE E NCODER /D ECODER
IOT intends to break the ﬁxed order of layers in Transformer.
As shown in introduction, simply reordering the layers w.r.t the whole corpus impacts little, while each data has its own preference to orders.
Therefore, IOT incorporates instance-wise learning to adjust the favorable order for each data.
As shown in Figure 1, both encoder and decoder in IOT consist of several blocks of SA, ED, FF layer with dynamic order, and we assume there are M (e.g., M = 2 ) ordered encoders and N (e.g.,
N = 6 ) ordered decoders (with shared weights).
Inspired by the fact that lower training loss implies the higher proﬁciency conﬁdence for candidate orders, we utilize the cross-entropy loss as signals to learn the conﬁdence.
That is, we calculate conﬁdence γ m and λ n for each encoder enc m , decoder dec n (resulted model M m,n ), and use them to weight the training loss L M m,n .
To calculate the conﬁdence, we add a simple and light predictor to help distinguish the orders.
Training Given one source sequence x = [ x 1 , x 2 , ..., x T x ] , we ﬁrst map each token into word embedding e =
[ e 1 , e 2 , ..., e T x ] , where e i ∈ R d , and then apply one light encoder predictor π enc to predict the conﬁdence of encoder orders using sentence embedding s e = 1 T x (cid:80) T x i
=1 e i .
Concretely, π enc takes s e as input and predicts γ m for enc m by Gumbel-softmax (Jang et al., 2016):
[Equation]
where g m is sampled from Gumbel distribution: g m = − log( − log U m ) , U m ∼ Uniform(0, 1),
W e ∈ R
d
× M is the weight matrix, τ e is a constant temperature to control the distribution to be identical approximation with categorical distribution.
Simultaneously, the token embeddings e will feed to the encoders to get hidden states h
= [ h 1 , h 2 , ..., h T x ] , then we can calculate decoder order conﬁdence λ n by one predictor π dec in the same way as π enc :
[Equation]
where s d = 1 T x (cid:80) T x i =1 h i and W d is the weight matrix.
For each ordered path through enc m and dec n , we can obtain the training loss L M m,n , and the ﬁnal cross-entropy loss is weighted by conﬁdence γ m and λ n with L M m,n , formulately as:
[Equation]
Inference During inference, we directly replace the Gumbel-softmax used in training with argmax , in order to choose the most capable encoder and decoder for each sequence x :
[Equation]
Discussion The decoding process is almost the same as standard Transformer, with only little overhead for order predictions.
One may concern the training cost is increased through our training.
As we present in Section 5.1, the cost is actually affordable with a fast convergence.
Currently, we reorder the layers of the encoder/decoder block and stack the same ordered block L times (see Figure 1).
A complex extension is to reorder all L blocks of encoder/decoder
and we take it as future work.
3. A UXILIARY L OSSES
As we can see, the predictors are trained in an unsupervised way, and we observe they lean to be lazy so that all samples quickly converge to one same order during training, without a senseful learning.
Thus, to make an effective training and inference, we introduce exploration and exploitation losses.
(1) Exploration : ﬁrst, we explore the diverse capability of all orders with help of a loss L D to encourage all orders to participate in training.
The spirit is the same as to encourage exploration in reinforcement learning.
The expected softmax probability E x [ π x ] (encoder/decoder) from the predictor should approximate the uniform distribution Q
=
[ 1 N , 1 N , . .
, 1 N ] (e.g., decoder orders), and we achieve this by minimizing KL-divergence between the statistical average E x [ π x ] and Q :
[Equation]
where ( π x ) n is the probability of n -th decoder order for data x .
For encoder order, it is ( π x ) m processed in a same way as decoder.
(2) Exploitation: different from L D to keep all orders effectively trained, during inference, the output distribution π x for each data should be able to make an unambiguous argmax selection.
We then introduce another loss L S to constrain each π x to be far away from the uniform distribution Q .
Concretely, we maximize the KL-divergence between each probability π x and Q :
[Equation]
Note that we clamp the value of probability π x since the KL value is theoretically unbounded.
With above auxiliary losses, the ﬁnal training objective is to minimize:
[Equation]
where c 1 and c 2 are coefﬁcients to make a trade-off between L D and L S .
In this way, we can achieve effective training, while keeping the ability to distinguish the favorable order for each data.
Discussion L D and L S aim to keep effective training and unambiguous inference.
There are several alternatives.
The ﬁrst is to simply decay the temperature τ in Equation (1) and (2), and remove the auxiliary losses.
However, we do not notice obvious gain.
Second is to linearly decay c 1 only and remove L S , which is able to fully train all orders at the beginning and loose this constraint gradually.
We ﬁnd this is also beneﬁcial, but our two losses method performs better.
E XPERIMENTS
We conduct experiments on 3 sequence generation tasks: neural machine translation (both lowresource and rich-resource), code generation and abstractive summarization.
The main settings of each experiment are introduced here, and more details can be found in Appendix A.
4. D ATASET
Neural Machine Translation For the low-resource scenario, we conduct experiments on IWSLT
English ↔ German (En ↔ De), English ↔ Spanish (En ↔ Es), IWSLT17 English ↔ French (En ↔ Fr),
English ↔ Chinese (En ↔ Zh) translations.
The training data includes 160 k , 183 k , 236 k , 235 k sentence pairs for each language pair respectively.
For the rich-resource scenario, we work on WMT
En → De and WMT16 Romanian → English (Ro → En) translations.
For WMT14 En → De, we ﬁlter out 4 .
5 M sentence pairs for training and concatenate newstest2012 and newstest2013 as dev set, newstest2014 as test set.
For WMT16 Ro → En, we concatenate the 0 .
6 M bilingual pairs and 2 .
M back translated data 2 for training, newsdev2016/newstest2016 serve as dev/test set.
Code Generation Code generation aims to map natural language sentences to programming language code.
We work on
one Java
Hu et al., 2018) and one Python dataset (Wan et al., 2018), following
Wei et al.
(2019) to process the two datasets.
The Java dataset is collected from Java projects on
Github, and the Python dataset is collected by Barone & Sennrich (2017).
We split each dataset with ratio 0 . 8 : 0 . 1 : 0 . 1 as training, dev and test set.
Abstractive Summarization Abstractive summarization is to summarize one long sentence into a short one.
The dataset we utilized is a widely acknowledged one: Gigaword summarization, which is constructed from a subset of Gigaword corpus (Graff et al., 2003) and ﬁrst used by Rush et al.
The training data consists of 3 . 8 M article-headline pairs, while the dev and test set consist of 190 k and 2 k pairs respectively.
4. M ODEL AND O PTIMIZATION
For IWSLT translation tasks, we use transformer iwslt de en setting as model conﬁguration.
The number of block, embedding size and feed-forward network (FFN) size are 6 , 512 , 1024 .
WMT tasks use transformer vaswani wmt en de big conﬁguration, with 6 blocks, embedding size 1024 and FFN size 4096 .
Optimization and learning scheduler are the default settings in
[Table] [Table]
respectively.
Others are the same as NMT.
For summarization, we take transformer wmt en de , with 6 blocks, embedding size 512 and FFN size 2048 .
Dropout (Srivastava et al., 2014) is set to be 0 . 3 .
Other settings are also the same as NMT task.
Implementation is developed on Fairseq (Ott et al., 2019).
We ﬁrst grid search c 1 , c 2 on IWSLT
De → En dev set, and then apply them on other tasks.
The best setting is c 1 = 0 .
1 , c 2 = 0 .
01 , and the importance study of c 1 , c 2 is shown in Appendix B.1.
4. E VALUATION
We use multi-bleu.perl to evaluate IWSLT14 En ↔ De and all WMT tasks for a fair comparison with previous works.
For other NMT tasks, we use sacre-bleu for evaluation.
During inference, we follow Vaswani et al.
(2017) to use beam size 4 and length penalty 0 .
6 for WMT
En → De, beam size 5 and penalty 1 . 0 for other tasks.
For code generation, the evaluation is based on two metrics, the sentence BLEU computes the n-gram precision of a candidate sequence to the reference, and the percentage of valid code (PoV) that can be parsed into an abstract syntax tree (AST).
As for summarization, the generated summarization is evaluated by ROUGE-1/2/L F1 score (Lin, 2004).
4. M AIN R ESULTS
Encoder/Decoder Orders Encoder block only contains SA and FF layers, the resulted max number of encoder layer orders M is 2 , while for decoder, the max order variants N is 6 .
Therefore, we ﬁrst evaluate the utilization of encoder orders, decoders orders, and both orders on IWSLT
De → En translation, in order to see the impacts of different number of order candidates and their combinations.
In Table 3 (a), we can see that 2 ordered encoders improve the result, and 6 ordered decoders achieve more gain.
This meets our expectation, since the search space is limited when there are only 2 ordered encoders.
However, if we train both encoder and decoder orders (e.g., M = 2 , N = 6 ), the results (e.g., 35 . 30 ) can not surpass the 6 decoders only ( 35 . 60 ).
We suspect the search space is too large so that training becomes hard, and decoder orders play a more important role than encoder orders for sequence generation.
Therefore, we turn to investigate different decoder order candidates (refer to Appendix A.3 for detailed combinations) in Table 3 (b).
Results show that N = 4 , 5 , 6 achieve similar strong performances (results on other tasks/datasets are in Appendix A.4).
Thus, considering the efﬁciency and improvements, we utilize N = 4 ordered decoders (order 1 , 2 , 4 , 6 in Table 1) to reduce training cost in later experiments.
NMT
Results BLEU scores on 8 IWSLT low-resource tasks are shown in Table 4.
As we can see,
IOT achieves more than 1 .
BLEU points improvement on all tasks (e.g., 1 . 7 on Fr → En).
The consistent gains on various language pairs well demonstrate the generalization and effectiveness of our method.
We then present comparison with other works on IWSLT
De → En task in Table(a), and IOT is also better than several human designed networks.
The results of WMT14 En → De and WMT16 Ro → En are reported in Table 6.
We also compare with existing works, such as the unsupervised Ro → En based on pre-trained cross-lingual language model (Lample & Conneau, 2019).
[Table] [Table]
[Table]
Code Generation Results The results are shown in Table 5(b)
[Table]
We can observe that Transformer obtains better result than the
LSTM-based work (Wei et al., 2019).
Compared with Transformer,
IOT can further improve the quality of generated code.
Speciﬁcally, IOT boosts Transformer with 0 .
93 BLEU/ 2 .
86% PoV gain on Java generation and 0 .
75 BLEU/ 1 .
25% PoV gain on Python respectively.
Again, these results well demonstrate the effectiveness of our method.
Abstractive Summarization Results The IOT performances on summarization task are shown in Table 7.
From the results, we can see IOT achieves 0 . 8 , 0 .
7 and 1 .
scores gain of ROUGE-1,
ROUGE-2 and ROUGE-L metrics over standard Transformer on
Gigaword summarization.
IOT also surpasses other works such as reinforcement learning based method (Wang et al., 2018), which again veriﬁes our approach is simple yet effective.
[Table]
S TUDY AND A NALYSIS 5. I NFERENCE /T RAINING C OST
As discussed before, our approach only increases negligible parameters and inference time cost.
Here we compare the detailed inference time and model size of our framework to the standard Transformer.
The detailed parameter numbers and inference time on IWSLT
En ↔ De test set are shown in
Table 8.
Since we only add one linear layer and softmax layer as the predictor, the number of extra parameters is M × hidden size (encoder predictor) or N × hidden size (decoder predictor), which is negligible compared to other model parameters.
Therefore, IOT introduces more model diversity and improves the performance, but under the constraint of almost same number of parameters.
As for the inference time, the only difference is from the one-pass order prediction and the cost is extremely low compared with heavy autoregressive generation process, which can be seen from Table 8.
Apart from the inference cost, one may concern about the training cost since IOT trains multiple orders in one model.
To see the inﬂuence, we provide several statistics here.
Speciﬁcally, on the four
IWSLT
En → X translation tasks, we analyze the cost by counting the training time for each epoch, the epoch number when model convergences, and the corresponding total training time.
The numbers
[Table] [Table]
are presented in Table 9, and we can have several observations.
Take IWSLT
En → De translation as an example, (1) jointly optimizing different orders indeed introduces more training cost for each epoch.
Transformer baseline costs 277 .
1 s per epoch training, while our IOT costs 475 .
6 s and 685 .
4 s with N = 2 and N = 3 orders respectively, the increased cost ratio is about 1 .
72 × and 2 .
47 × (but less than 2 . 0 and 3 . 0 ).
However, we ﬁnd that with the shared parameters between these orders, the model convergence also becomes faster.
Transformer needs 67 epochs when converge, while our IOT only needs 42(0 .
63 × ) and 39(0 . 58 × ) epochs for N = 2 and N = 3 orders, much fewer than Transformer.
The total training cost actually is not increased much.
IOT ( N = 2 ) and IOT ( N = 3 ) are about 1 .
08 × and 1 .
44 × training time compared with Transformer baseline (the ratio for IOT ( N = 3 ) is only 1 . 10 on IWSLT
En → Es).
From these observations, we can see that the increased training cost is affordable due to the fast convergence.
5. C ASE V ERIFICATION
We perform a study with N = 3 to verify that IOT has made a necessary instance-wise order selection.
We ﬁrst split IWSLT
En ↔ De dev set into 3 subsets according to the prediction of π dec , and then we decode each subset use all 3 ordered decoders, and report the BLEU results.
As shown in Figure 2, each subset indeed achieves the best score on the corresponding predicted order (outperforms other orders by 0 .
4 BLEU).
We also do the same study on the test set, and the predicted order outperforms others by 0 .
8 BLEU.
These well prove that IOT makes a reasonable prediction.
Besides, we ﬁnd that the predicted orders correlate to different sentence difﬁculties.
In our case, the set 1 sentences belong to decoder 1 achieve highest BLEU than other sets, which means set 1 is relatively simple to translate, and vice versa for samples in set 2 .
These imply that different difﬁculty sentences have different structure preferences.
We provide statistics and examples in Appendix B.2.
5. A PPLY ON A NOTHER S TRUCTURE (D YNAMIC C ONV )
As we discussed, our instance-wise layer reordering is structure agnostic.
In this subsection, we evaluate this by applying our approach on DynamicConv network (Wu et al., 2019) beyond standard
[Figure]
Transformer, which replaces the self-attention with dynamic convolution.
We train layer ordered
DynamicConv on N = 2 , 3 , 4 decoders and test the performances.
The BLEU score of standard
DynamicConv is 35 . 20 , and with our instance-wise order learning, we achieve 35 .
82 , 35 .for
N = 2 , 3 , 4 ordered decoders respectively (near 0 .
7 point gain).
Therefore, this study veriﬁes our claim that our approach can be applied to other structures, as long as multiple different layers exist.
5. D ISCUSSIONS
[Table]
Ensemble Since our framework involves multiple orders (with shared parameters), which is also done in ensemble framework, we make a comparison with ensemble.
The ensemble method trains multiple models with different parameters separately in an independent way.
While our work trains orders in a joint way with an intention to make them more diverse.
More importantly, from the view of time and memory cost, the ensemble framework increases N times which is totally different from ours.
In this sense, our method can be combined with ensemble to further boost performance.
The competitive results on IWSTL
En ↔ De test set are shown in Table 10.
We can clearly conclude that IOT and ensemble are complementary to each other.
Regularization IOT consists of different ordered blocks in a weight tying method, which may looks like a parameter regularization to some extent.
However, we show that IOT is more than regularization and can be complementary with other regularization methods.
Setting (1): We ﬁrst train a Transformer model on IWSLT
De → En task with all shared decoder orders, but without instance-wise learning, and test the performance with each order.
We ﬁnd the BLEU scores on test set are near 34 .
80 for each order, much worse than IOT, which means that simply regularizing the shared parameters for different orders is not the main contribution to performance improvement, and our instance-wise learning is critical.
Setting (2): Another experiment is that we train Transformer with LayerDrop (Fan et al., 2019), a dropout technique to regularize the layer parameters.
The test
BLEU is 35 .
40 , which achieves about 0 .
8 score improvement over Transformer.
After applying
IOT with LayerDrop, we obtain further gains than IOT only ( 35 . 62 ) to reach a BLEU score 36 . 13 .
Therefore, this demonstrates IOT is not only regularization and can be smoothly integrated with other regularization methods.
More details and experiments on other tasks are shown in Appendix B.3.
C ONCLUSION
In this work, we propose Instance-wise Ordered Transformer, which leverages instance-wise learning to reorder the layers in Transformer for each data.
Compared with standard Transformer, IOT only introduces slightly increased time cost.
Experiments on 3 sequence generation tasks and 9 datasets demonstrate the effectiveness of IOT.
We also verify that our approach can be universally applied to other structures, such as DynamicConv.
In future, we plan to work on more complicated reordering in each block, as well as other tasks such as multi-lingual translation and text classiﬁcation.
A E XPERIMENTAL S ETTINGS AND M ORE R ESULTS A. D ETAILED D ATA S ETTINGS
Neural Machine Translation Following the common practice (Ott et al., 2019), we lowercase all words for IWSLT
En ↔ De.
For IWSLT
En ↔ De, En ↔ Es, IWSLT
En ↔ Fr, we use a joint source and target vocabulary with 10 k byte-pair-encoding (BPE) (Sennrich et al., 2015) operations, and for IWSLT
En ↔ Zh, we use a seperate source and target vocabulary.
For all WMT tasks, sentences are encoded by a joint source and target vocabulary of 32 k tokens.
Code Generation
In the Java dataset, the numbers of training, validation and test sequences are 69 , 708 , 8 , 714 and 8 , 714 respectively, and the corresponding numbers for Python are 55 , 538 , 18 , 505 and 18 , 502 .
All samples are tokenized.
We use the downloaded Java dataset without further processing, and use Python standard AST module to further process the python code.
The source and target vocabulary sizes in natural language to Java code generation are 27 k and 50 k , and those for natural language to Python code generation are 18k and 50 k .
In this case, following Wei et al.
(2019), we do not apply subword tokenization like BPE to the sequences.
Abstractive Summarization
The Gigaword corpus represents for a headline generation task, each source article contains about 31 .
4 tokens on average, while the target headline contains near 8 .tokens per sentence.
The training data consists of 3 . 8 M article-headline pairs, while the validation and test set consist of 190 k and 2 k pairs respectively.
We preprocess the dataset in a same way as
NMT task.
The words in the source article and target headline are concatenated to make a joint BPE vocabulary.
After preprocessing, there are 29 k subword tokens in the vocabulary.
A. D ETAILED M ODEL /T RAINING C ONFIGURATIONS
Model Conﬁguration
The detailed model conﬁgurations are as follows:
• transformer iwslt de en setting: 6 blocks in encoder and decoder, embedding size 512 , feed-forward size 1024 , attention heads 4 , dropout value 0 .
3 , weight decay 0 . 0001 .
• transformer vaswani wmt en de big setting: 6 blocks in encoder and decoder, embedding size 1024 , feed-forward size 4096 , attention heads 16 , dropout value 0 .
3 , attention dropout 0 .
1 , relu dropout 0 .
• transformer wmt en de big setting: 6 blocks in encoder and decoder, embedding size 0124 , feed-forward size 4096 , attention heads 16 , dropout value 0 .
Optimization We adopt the default optimization setting in Vaswani et al.
Adam (Kingma & Ba, 2014) optimizer with β 1 = 0 .
9 , β 2 = 0 .
98 and (cid:15) = 10 − 9 .
The learning rate scheduler is inverse sqrt with warmup steps 4 , 000 , default learning rate is 0 . 0005 .
Label smoothing (Szegedy et al., 2016) is used with value 0 .
As introduced, to learn the predictors, we clamp the softmax output with value 0 .
A. R ESULTS OF O RDER C OMBINATIONS
We show in the paper that different number of orders (e.g., N = 4 or N = 5 ) have varied performances.
Therefore, one necessary point is about the different combinations of these N decoders.
Here, we work on N = 5 IOT model to show the results of different order candidates.
We ﬁrst present each ordered decoder in Table 11 again (same as in Table 1).
For the N = 5 ordered decoders with IOT model, we show the performances with 5 combined orders selected from all six variants on dev set of IWSLT
De → En and En → De translations.
The results are reported in Table 12.
We can see the different combinations achieve similar strong performances, which shows that our
[Table] [Table]
approach is robust towards different order combinations.
This also demonstrate that the importance of IOT is the diversity among order candidates that can help each data distinguish them.
For other N ordered decoders, the patterns are similar.
Therefore, here we only report the N combinations used for IOT experiments in the paper as follows: IOT ( N = 2 ) is combined by order 4 , 6 ( ED → SA → FF and FF → ED → SA ), and IOT ( N = 3 ) is order 1 , 4 , 6 , IOT ( N = 4 ) is order 1 , 2 , 4 , 6 , and IOT ( N = 5 ) is order 1 , 2 , 4 , 5 , 6 .
A. R ESULTS OF D IFFERENT N UMBER OF D ECODERS
The results of N = 4 ordered decoders (order 1 , 2 , 4 , 6 ) are mainly reported in the paper.
Here, we also show results of other N decoders for all tasks, along with the Transformer baseline.
The results of different N decoders for WMT14 En → De and WMT16 Ro → En translations, code generation task, and Gigaword summarization are reported in Table 14, Table 15 and Tablerespectively.
As we can see, more ordered decoders can bring better performance, which supports the effectiveness of our framework and demonstrates the data has its own favor towards different orders.
Considering the efﬁciency, we do not perform experiments with more than 4 decoders for these tasks.
[Table] [Table]
[Table] [Table]
[Table]
B M ORE S TUDIES B. I MPACT OF W EIGHTED A UXILIARY L OSSES
We conduct another study on IWSLT
De
→ En dev set to investigate the impact of our proposed auxiliary losses controlled by weight c 1 and c 2 .
The values of c 1 and c 2 are varied between [0 . 0 , 0 . 05 , 0 . 1 , 0 . 5] and [0 . 0 , 0 . 005 , 0 . 01 , 0 . 05] respectively, and the results are presented in Table 17.
It can be seen that the best conﬁguration is c 1 = 0 .
1 and c 2 = 0 .
Therefore, we report the leading results in the paper with c 1 = 0 .
1 , c 2 = 0 .
The results also clearly demonstrate that the two additional losses are necessary to make our framework effective.
B. D ATA E XAMPLES V ERIFICATION
As discussed in Section 5.2, the data split by the corresponding predicted order is in different pattern.
For example, the difﬁculty of each set is different.
We therefore analyze the split data and calculate some statistics among these subsets.
Speciﬁcally, we ﬁrst count the sentence number S , the tokens
T , and the distinct vocabulary D
i in each subset.
We show these numbers in Table 18, along with corresponding averaged BLEU score (see Figure 2).
We can see that the vocabulary size of set 1 is the smallest, and set 2 is the largest, which means there are more distinct words in set 2 .
This leads the generation of set 2 to be harder than set 1 , which maps the BLEU score ranking among these sets.
Besides, we also calculate the token frequency f ij for token j in each own subset i , and sum the frequency of top 20 tokens in each subset, F i = (cid:80) 20 j =1 f ij , to give another evidence.
The results also show that F 1 is the highest, which means the tokens in set 1 contains most frequent words to make an easy learning, while set 2 is harder since F 2 is small.
[Table]
[Table]
We further take a look at the data and ﬁnd the sentences in set 1 are mostly “simple sentences”, and set 2 contains many “emphatic sentences”, while set 3 is somehow mixed.
In Table 19, we provide some sentence examples belong to each subset to give more clariﬁcations.
B. R EGULARIZATION
In Section 5.4, we have provided an example of regularization experiments on IWLST
De → En translation, which demonstrates that our IOT is not only regularization and can be smoothly integrated with other regularization methods.
To give more evidences and more details, we extend the regularization experiments on all IWSLT translation tasks (IWSLT14 En ↔ De, IWSLT
En ↔ Es,
IWSLT17 En ↔ Zh, IWSLT17 En ↔ Fr), and WMT16 Ro → En translation.
The two speciﬁc settings of the experiments are as follows.
Setting (1): The ﬁrst experiment is “ordered Transformer” without instance awareness.
That is, all the reordered architectures are trained on the whole same corpus with equal weights, and the parameters for these reordered architectures are shared.
More specifically, the decoder block has different ways to order SA , ED , and FF layers (e.g., FF → SA → ED ,
SA → ED → FF , etc), but the parameters for the reordered blocks are shared.
Mathematically, the loss function is: L C =
(cid:80) Nn =1 ( λ n · L M n ) , where L M n is the model loss function for n -th ordered decoder.
Compared with Eqn (3), the weight λ n is ﬁxed to be 1 here.
At inference, we ﬁrst ﬁnd out the best order according to the dev performance and apply it on the test set.
We cannot use instance-wise reordered model in this setting, while our proposed IOT can.
The experiments are conducted with transformer iwslt de en conﬁguration for IWSLT translations, and transformer vaswani wmt en de big conﬁguration for WMT16 Ro → En translation.
Setting (2): We integrate another regularization technique ‘LayerDrop’ (Fan et al., 2019) into both the Transformer baseline and our IOT ( N = 4 ) method, while other settings remain unchanged.
The study results of these two settings are represented in Table 20.
From the results, we have same conclusions as discussed in Section 5.4.
Simply sharing the parameters of different decoders as a regularization cannot boost the model performance (“Transformer + (1)” in
Table 20), while our IOT can further improve the performance with other regularization methods.
B. R OBUSTNESS
An impact of IOT training besides performance gain is that the model can be more robust compared to one order only.
In Table 21, we provide one example to prove the robustness.
We train one
Transformer model by decoder order 1 , and to decode the sentences with all orders in inference.
Obviously, only decoding with order 1 leads to good performance, while other orders can not achieve reasonable scores since the layer order is changed and the feature exaction becomes incorrect.
As for
IOT, the generated sequences remain stable and high results for each order.
B. V ISUALIZATION
To better understand the difference between IOT and standard Transformer, we investigate on the training process and provide visualization results about model optimization and performance improvements.
Speciﬁcally, we plot the curve of training loss, validation loss, as well as the validation
BLEU score and test BLEU score along the training epochs on IWSLT
De → En translation dataset.
The loss curves are visualized in Figure 3, and the BLEU curves are presented in Figure 4.
From the validation loss curves of Figure 3(b) and 3(c), we can ﬁrst see that our IOT ( N = 3 ) training converges faster than Transformer baseline and shows the advantage of IOT, which is consistent to
[Table]
[Table]
is the order trained by Transformer.
our analysis in Section 5.1.
The converged (smallest) validation loss value seems to be similar to
Transformer baseline, but please note that the loss computation of IOT is different from Transformer baseline.
As shown in Eqn (3), the loss function of IOT is a weighted sum of loss values for each order, while for Transformer, it is only one order loss.
Therefore, when we turn to the comparison of validation BLEU score, the superiority of our IOT can be clearly veriﬁed.
From the BLEU score curves in Figure
[Figure]
it is obvious that IOT achieves better BLEU score than standard Transformer along the whole training epochs, on both validation and test sets.
These visualized results well demonstrate the effectiveness of our IOT approach.
[Figure] [Figure]