Original: H INT - BASED T RAINING FOR N ON -A UTOREGRESSIVE T RANSLATION
Revised: H INT - BASED T RAINING FOR N ON -A UTOREGRESSIVE T RANSLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine translation is an important real-world application, and neural networkbased AutoRegressive Translation (ART) models have achieved very promising accuracy.
Revised: Machine translation is an important real-world application, and neural networkbased AutoRegressive Translation (ART) models have achieved very promising accuracy.

Original: Due to the unparallelizable nature of the autoregressive factorization,
Revised: Due to the unparallelizable nature of the autoregressive factorization,

Original: ART models have to generate tokens one by one during decoding and thus suffer from high inference latency.
Revised: ART models have to generate tokens one by one during decoding and thus suffer from high inference latency.

Original: Recently, Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time.
Revised: Recently, Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time.

Original: However, they could only achieve inferior accuracy compared with ART models.
Revised: However, they could only achieve inferior accuracy compared with ART models.

Original: To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a welltrained ART model to train the NART model.
Revised: To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a welltrained ART model to train the NART model.

Original: We deﬁne two hints for the machine translation task: hints from hidden states and hints from word alignments, and use such hints to regularize the optimization of NART models.
Revised: We deﬁne two hints for the machine translation task: hints from hidden states and hints from word alignments, and use such hints to regularize the optimization of NART models.

Original: Experimental results show that the NART model trained with hints could achieve signiﬁcantly better translation performance than previous NART models on several tasks.
Revised: Experimental results show that the NART model trained with hints could achieve signiﬁcantly better translation performance than previous NART models on several tasks.

Original: In particular, for the WMT14 En-De and De-En task, we obtain BLEU scores of 25. and 29.52 respectively, which largely outperforms the previous non-autoregressive baselines.
Revised: In particular, for the WMT14 En-De and De-En task, we obtain BLEU scores of 25. and 29.52 respectively, which largely outperforms the previous non-autoregressive baselines.

Original: It is even comparable to a strong LSTM-based ART model (24.60 on
Revised: It is even comparable to a strong LSTM-based ART model (24.60 on

Original: WMT14 En-De), but one order of magnitude faster in inference.
Revised: WMT14 En-De), but one order of magnitude faster in inference.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural machine translation has attracted much attention from the research community (Bahdanau et al., 2014; 2016; Kalchbrenner et al., 2016; Gehring et al., 2016) and has been gradually adopted by industry in the past several years (Wu et al., 2016).
Revised: Neural machine translation has attracted much attention from the research community (Bahdanau et al., 2014; 2016; Kalchbrenner et al., 2016; Gehring et al., 2016) and has been gradually adopted by industry in the past several years (Wu et al., 2016).

Original: Despite the huge variety of model architectures (Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017), given a source sentence x
Revised: Despite the huge variety of model architectures (Bahdanau et al., 2014; Gehring et al., 2017; Vaswani et al., 2017), given a source sentence x

Original: = ( x 1 , ..., x T x ) and a target sentence y =
Revised: = ( x 1 , ..., x T x ) and a target sentence y =

Original: ( y 1 , ..., y T y ) , most neural machine translation models decompose and estimate the conditional probability P ( y | x ) in an universal autoregressive manner:
Revised: ( y 1 , ..., y T y ) , most neural machine translation models decompose and estimate the conditional probability P ( y | x ) in an universal autoregressive manner:

Original: [Equation]
Revised: [Equation]

Original: where y <t represents the ﬁrst t − 1 words of y .
Revised: where y <t represents the ﬁrst t − 1 words of y .

Original: During inference, given an input sentence, those models generate the translation results sequentially, token by token from left to right.
Revised: During inference, given an input sentence, those models generate the translation results sequentially, token by token from left to right.

Original: We call all such models AutoRegressive neural machine Translation (ART) models.
Revised: We call all such models AutoRegressive neural machine Translation (ART) models.

Original: A state-of-the-art ART model, Transformer (Vaswani et al., 2017), is shown in the left part of Figure 1.
Revised: A state-of-the-art ART model, Transformer (Vaswani et al., 2017), is shown in the left part of Figure 1.

Original: A well-known limitation of the ART models is that the inference process can hardly be parallelized, and the inference time is linear with respect to the length of the target sequence.
Revised: A well-known limitation of the ART models is that the inference process can hardly be parallelized, and the inference time is linear with respect to the length of the target sequence.

Original: As a result, the
Revised: As a result, the

Original: ART models suffer from long inference time (Wu et al., 2016), which is sometimes unaffordable for industrial applications.
Revised: ART models suffer from long inference time (Wu et al., 2016), which is sometimes unaffordable for industrial applications.

Original: Consequently, people start to develop Non-AutoRegressive neural machine
Revised: Consequently, people start to develop Non-AutoRegressive neural machine

Original: Translation (NART) models to speed up the inference process (Gu et al., 2017; Kaiser et al., 2018;
Revised: Translation (NART) models to speed up the inference process (Gu et al., 2017; Kaiser et al., 2018;

Original: Lee et al., 2018).
Revised: Lee et al., 2018).

Original: These models use the general encoder-decoder framework: the encoder takes a source sentence x as input and generates a set of contextual embeddings and predicted length T y ; conditioned on the contextual embeddings, the decoder takes a transformed copy of x as input and predicts the target tokens at all the positions independently in parallel according to the following decomposition:
Revised: These models use the general encoder-decoder framework: the encoder takes a source sentence x as input and generates a set of contextual embeddings and predicted length T y ; conditioned on the contextual embeddings, the decoder takes a transformed copy of x as input and predicts the target tokens at all the positions independently in parallel according to the following decomposition:

Original: [Equation]
Revised: [Equation]

Original: While the NART models achieve signiﬁcant speedup during inference (Gu et al., 2017), their accuracy is considerably lower than their ART counterpart.
Revised: While the NART models achieve signiﬁcant speedup during inference (Gu et al., 2017), their accuracy is considerably lower than their ART counterpart.

Original: Most of the previous works attribute the poor performance to this unavoidable conditional independence assumption of the NART model.
Revised: Most of the previous works attribute the poor performance to this unavoidable conditional independence assumption of the NART model.

Original: To tackle this issue, they try to improve the expressiveness and accuracy of the decoder input in different ways: Gu et al.
Revised: To tackle this issue, they try to improve the expressiveness and accuracy of the decoder input in different ways: Gu et al.

Original: (2017) introduce fertilities from statistical machine translation models into the
Revised: (2017) introduce fertilities from statistical machine translation models into the

Original: NART models, Lee et al.
Revised: NART models, Lee et al.

Original: (2018) base the decoding process of their proposed model on an iterative reﬁnement process, and Kaiser et al.
Revised: (2018) base the decoding process of their proposed model on an iterative reﬁnement process, and Kaiser et al.

Original: (2018) take a step further to embed an autoregressive submodule that consists of discrete latent variables into their model.
Revised: (2018) take a step further to embed an autoregressive submodule that consists of discrete latent variables into their model.

Original: Although such methods provide better expressiveness of decoder inputs and improve the ﬁnal translation accuracy, the inference speed of these models will be hurt due to the overhead of the introduced modules, which contradicts with the original purpose of introducing the NART models, i.e., to parallelize and speed up neural machine translation models.
Revised: Although such methods provide better expressiveness of decoder inputs and improve the ﬁnal translation accuracy, the inference speed of these models will be hurt due to the overhead of the introduced modules, which contradicts with the original purpose of introducing the NART models, i.e., to parallelize and speed up neural machine translation models.

Original: Different from previous works that develop new submodules for decoder input, we improve the translation model from another perspective.
Revised: Different from previous works that develop new submodules for decoder input, we improve the translation model from another perspective.

Original: We aim to provide more guided signals during optimization.
Revised: We aim to provide more guided signals during optimization.

Original: That is, we do not introduce any new prediction submodule but introduce better regularization.
Revised: That is, we do not introduce any new prediction submodule but introduce better regularization.

Original: The reason we tackle the problem from this perspective lies in two points: First, the encoder input (source words) contains all semantic information for translation, and the decoder input in the
Revised: The reason we tackle the problem from this perspective lies in two points: First, the encoder input (source words) contains all semantic information for translation, and the decoder input in the

Original: NART model can be considered as a middle layer between input and output.
Revised: NART model can be considered as a middle layer between input and output.

Original: It is not clear how much gain can be achieved by developing a sophisticated submodule for a middle layer in a deep neural network.
Revised: It is not clear how much gain can be achieved by developing a sophisticated submodule for a middle layer in a deep neural network.

Original: Second, the encoder-decoder-based NART model is already over-parameterized.
Revised: Second, the encoder-decoder-based NART model is already over-parameterized.

Original: We believe that such neural network still has great ability and space to be better optimized if we can provide it with stronger and richer signals, for example, from a much better ART model: Once we have a well-trained ART model, we actually know rich information about the contexts to make the prediction at each time step and the natural word alignments between bilingual sentences.
Revised: We believe that such neural network still has great ability and space to be better optimized if we can provide it with stronger and richer signals, for example, from a much better ART model: Once we have a well-trained ART model, we actually know rich information about the contexts to make the prediction at each time step and the natural word alignments between bilingual sentences.

Original: All the information could be invaluable towards the improved training of a NART model.
Revised: All the information could be invaluable towards the improved training of a NART model.

Original: To well leverage an ART model, we use the hint-based training framework (Romero et al., 2014;
Revised: To well leverage an ART model, we use the hint-based training framework (Romero et al., 2014;

Original: Chen et al., 2017), in which the information from hidden layers of teacher model (referred as hints ) are used to guide the training process of a student model.
Revised: Chen et al., 2017), in which the information from hidden layers of teacher model (referred as hints ) are used to guide the training process of a student model.

Original: However, hint-based training was developed for image classiﬁcation models and it is challenging to deﬁne and use hints for translation.
Revised: However, hint-based training was developed for image classiﬁcation models and it is challenging to deﬁne and use hints for translation.

Original: First, the translation model is composed of stacked encoder layers, attention layers, and stacked decoder layers.
Revised: First, the translation model is composed of stacked encoder layers, attention layers, and stacked decoder layers.

Original: It is not clear how to deﬁne hints in such an encoder-decoder framework.
Revised: It is not clear how to deﬁne hints in such an encoder-decoder framework.

Original: Second, the
Revised: Second, the

Original: NART and ART models are of different architectures on the decoding stage.
Revised: NART and ART models are of different architectures on the decoding stage.

Original: It is not obvious how to leverage hints from the teacher to the training of student with a different architecture.
Revised: It is not obvious how to leverage hints from the teacher to the training of student with a different architecture.

Original: We ﬁnd that directly applying hints used in the classiﬁcation tasks fails.
Revised: We ﬁnd that directly applying hints used in the classiﬁcation tasks fails.

Original: In this paper, we ﬁrst investigate the causes of the bad performance of the NART model, and then deﬁne hints targeting to solve the problems.
Revised: In this paper, we ﬁrst investigate the causes of the bad performance of the NART model, and then deﬁne hints targeting to solve the problems.

Original: According to our empirical study, we ﬁnd that the hidden states of the NART model differ from the ART model: the positions where the NART model outputs incoherent tokens will have very high hidden states similarity.
Revised: According to our empirical study, we ﬁnd that the hidden states of the NART model differ from the ART model: the positions where the NART model outputs incoherent tokens will have very high hidden states similarity.

Original: Also, the attention distributions of the NART model are more ambiguous than those of ART model.
Revised: Also, the attention distributions of the NART model are more ambiguous than those of ART model.

Original: Based on these observations, we design two kinds of hints from the hidden states and attention distributions of the ART model, to help the training of the NART model.
Revised: Based on these observations, we design two kinds of hints from the hidden states and attention distributions of the ART model, to help the training of the NART model.

Original: We have conducted experiments on the widely used WMT14 English-to-German/German-toEnglish (En-De/De-En) task and IWSLT14 German-to-English task.
Revised: We have conducted experiments on the widely used WMT14 English-to-German/German-toEnglish (En-De/De-En) task and IWSLT14 German-to-English task.

Original: For WMT14 En-De task, our proposed method achieves a BLEU score of 25.20 which signiﬁcantly outperforms the nonautoregressive baseline models and is even comparable to a strong ART baseline, Google’s LSTMbased translation model (24.60 Wu et al.
Revised: For WMT14 En-De task, our proposed method achieves a BLEU score of 25.20 which signiﬁcantly outperforms the nonautoregressive baseline models and is even comparable to a strong ART baseline, Google’s LSTMbased translation model (24.60 Wu et al.

Original: For WMT14 De-En task, we also achieve signiﬁcant performance gains, reaching 29.52 in terms of BLEU.
Revised: For WMT14 De-En task, we also achieve signiﬁcant performance gains, reaching 29.52 in terms of BLEU.

Original:  R ELATED W ORKS 2. A UTO R EGRESSIVE T RANSLATION
Revised:  R ELATED W ORKS 2. A UTO R EGRESSIVE T RANSLATION

Original: Given a sentence x = ( x 1 , . . .
Revised: Given a sentence x = ( x 1 , . . .

Original: , x T x ) from the source language, the straight-forward way for translation is to generate the words in the target language y
Revised: , x T x ) from the source language, the straight-forward way for translation is to generate the words in the target language y

Original: =
Revised: =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y T y ) one by one from left to right.
Revised: , y T y ) one by one from left to right.

Original: This is also known as the autoregressive factorization in which the joint probability is decomposed into a chain of conditional probabilities, as in the Eqn.
Revised: This is also known as the autoregressive factorization in which the joint probability is decomposed into a chain of conditional probabilities, as in the Eqn.

Original: Deep neural networks are widely used to model such conditional probabilities based on the encoder-decoder framework.
Revised: Deep neural networks are widely used to model such conditional probabilities based on the encoder-decoder framework.

Original: The encoder takes the source tokens ( x 1 , . . .
Revised: The encoder takes the source tokens ( x 1 , . . .

Original: , x T x ) as input and encodes x into a set of context states c = ( c 1 , . . .
Revised: , x T x ) as input and encodes x into a set of context states c = ( c 1 , . . .

Original: c T x ) .
Revised: c T x ) .

Original: The decoder takes c and subsequence y <t as input and estimates P ( y t | y <t , c ) according to some parametric function.
Revised: The decoder takes c and subsequence y <t as input and estimates P ( y t | y <t , c ) according to some parametric function.

Original: [Figure]
Revised: [Figure]

Original: FFN
Revised: FFN

Original: [Figure]
Revised: [Figure]

Original: SoftMax SoftMax
Revised: SoftMax SoftMax

Original: SoftMax SoftMax
Revised: SoftMax SoftMax

Original: Emb Emb
Revised: Emb Emb

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: sos
Revised: sos

Original: [Figure]
Revised: [Figure]

Original: yy
Revised: yy

Original: [Figure]
Revised: [Figure]

Original: NART
Revised: NART

Original: [Figure]
Revised: [Figure]

Original: Emb
Revised: Emb

Original: [Figure]
Revised: [Figure]

Original: There are many design choices in the encoder-decoder framework based on different types of layers, e.g., recurrent neural network(RNN)-based (Bahdanau et al., 2014), convolution neural network(CNN)-based (Gehring et al., 2017) and recent self-attention based (Vaswani et al., 2017) approaches.
Revised: There are many design choices in the encoder-decoder framework based on different types of layers, e.g., recurrent neural network(RNN)-based (Bahdanau et al., 2014), convolution neural network(CNN)-based (Gehring et al., 2017) and recent self-attention based (Vaswani et al., 2017) approaches.

Original: We show a self-attention based network (Transformer) in the left part of Figure 1.
Revised: We show a self-attention based network (Transformer) in the left part of Figure 1.

Original: While the ART models have achieved great success in terms of translation quality, the time consumption during inference is still far away from satisfactory.
Revised: While the ART models have achieved great success in terms of translation quality, the time consumption during inference is still far away from satisfactory.

Original: During training, the ground truth pair ( x, y ) is exposed to the model, and thus the prediction at different positions can be estimated in parallel based on CNN or self-attention networks.
Revised: During training, the ground truth pair ( x, y ) is exposed to the model, and thus the prediction at different positions can be estimated in parallel based on CNN or self-attention networks.

Original: However, during inference, given a source sentence x , the decoder has to generate tokens sequentially, as the decoder inputs y <t must be inferred on the ﬂy.
Revised: However, during inference, given a source sentence x , the decoder has to generate tokens sequentially, as the decoder inputs y <t must be inferred on the ﬂy.

Original: Such autoregressive behavior becomes the bottleneck of the computational time (Wu et al., 2016).
Revised: Such autoregressive behavior becomes the bottleneck of the computational time (Wu et al., 2016).

Original: 2. N ON -A UTO R EGRESSIVE T RANSLATION
Revised: 2. N ON -A UTO R EGRESSIVE T RANSLATION

Original: In order to speed upto the inference process, a line of works begin to develop non-autoregressive translation models.
Revised: In order to speed up the inference process, a line of works begin to develop non-autoregressive translation models.

Original: These models follow the encoder-decoder framework and inherit the encoder structure from the autoregressive models.
Revised: These models follow the encoder-decoder framework and inherit the encoder structure from the autoregressive models.

Original: After generating the context states c by the encoder, a separate module will be used to predict the target sentence length T y and decoder inputs z
Revised: After generating the context states c by the encoder, a separate module will be used to predict the target sentence length T y and decoder inputs z

Original: =
Revised: =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z T y ) by a parametric function: ( T y , z ) ∼ f z ( x, c ; θ ) , which is either deterministic or stochastic.
Revised: , z T y ) by a parametric function: ( T y , z ) ∼ f z ( x, c ; θ ) , which is either deterministic or stochastic.

Original: The decoder will then predict y based on following probabilistic decomposition
Revised: The decoder will then predict y based on following probabilistic decomposition

Original: [Equation]
Revised: [Equation]

Original: Different conﬁgurations of T y and z enable the decoder to produce different target sentence y given the same input sentence x , which increases the output diversity of the translation models.
Revised: Different conﬁgurations of T y and z enable the decoder to produce different target sentence y given the same input sentence x , which increases the output diversity of the translation models.

Original: Previous works mainly pay attention to different design choices of f z .
Revised: Previous works mainly pay attention to different design choices of f z .

Original: Gu et al.
Revised: Gu et al.

Original: (2017) introduce fertilities , corresponding to the number of target tokens occupied by each of the source tokens, and use a non-uniform copy of encoder inputs as z according to the fertility of each input token.
Revised: (2017) introduce fertilities , corresponding to the number of target tokens occupied by each of the source tokens, and use a non-uniform copy of encoder inputs as z according to the fertility of each input token.

Original: The prediction of fertilities is done by a separated neural network-based module.
Revised: The prediction of fertilities is done by a separated neural network-based module.

Original: Lee et al.
Revised: Lee et al.

Original: (2018) deﬁne z by a sequence of generated target sentences y (0) , . .
Revised: (2018) deﬁne z by a sequence of generated target sentences y (0) , . .

Original: , y ( L ) , where each y ( i ) is a reﬁnement of y ( i − 1) .
Revised: , y ( L ) , where each y ( i ) is a reﬁnement of y ( i − 1) .

Original: Kaiser et al.
Revised: Kaiser et al.

Original: (2018) use a sequence of autoregressively generated discrete latent variables as inputs of the decoder.
Revised: (2018) use a sequence of autoregressively generated discrete latent variables as inputs of the decoder.

Original: While the expressiveness of z improved by different kinds of design choices, the computational overhead of z will hurt the inference speed of the NART models.
Revised: While the expressiveness of z improved by different kinds of design choices, the computational overhead of z will hurt the inference speed of the NART models.

Original: Comparing to the more than 15 × speed up in Gu et al.
Revised: Comparing to the more than 15 × speed up in Gu et al.

Original: (2017), which uses a relatively simpler design choice of z , the speedup of
Revised: (2017), which uses a relatively simpler design choice of z , the speedup of

Original: Kaiser et al.
Revised: Kaiser et al.

Original: (2018) is reduced to about 5 × , and the speedup of Lee et al.
Revised: (2018) is reduced to about 5 × , and the speedup of Lee et al.

Original: (2018) is reduced to about
Revised: (2018) is reduced to about

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: ) NART with Hints i_ know_that_we_can_ do_it_and_as_as_ &_ apos_;_s_m_ concerned_ , _ that_ &_ apos_ ;_s_ something_that_the_ world_needs_now_ .
Revised: ) NART with Hints i_ know_that_we_can_ do_it_and_as_as_ &_ apos_;_s_m_ concerned_ , _ that_ &_ apos_ ;_s_ something_that_the_ world_needs_now_ .

Original: _ <EOS> 1.
Revised: _ <EOS> 1.

Original: 2 × .
Revised: 2 × .

Original: This contradicts with the design goal of the NART models: to parallelize and speed up neural machine translation models.
Revised: This contradicts with the design goal of the NART models: to parallelize and speed up neural machine translation models.

Original:  H INT - BASED T RAINING FOR N ON -A UTOREGRESSIVE T RANSLATION
Revised:  H INT - BASED T RAINING FOR N ON -A UTOREGRESSIVE T RANSLATION

Original: In this section, we introduce the proposed hint-based training algorithm that leverages a well-trained
Revised: In this section, we introduce the proposed hint-based training algorithm that leverages a well-trained

Original: ART model to train the NART model.
Revised: ART model to train the NART model.

Original: Our model mostly follows Transformer (Vaswani et al., 2017), with an additional positional attention layer proposed by Gu et al.
Revised: Our model mostly follows Transformer (Vaswani et al., 2017), with an additional positional attention layer proposed by Gu et al.

Original: (2017), as shown in the right part of Figure 1.
Revised: (2017), as shown in the right part of Figure 1.

Original: To avoid overhead, we use simple linear combinations of source token embeddings as z , which has no learnable parameters.
Revised: To avoid overhead, we use simple linear combinations of source token embeddings as z , which has no learnable parameters.

Original: Details about the model can be found in the appendix.
Revised: Details about the model can be found in the appendix.

Original: We ﬁrst describe the observations we ﬁnd about the ART and NART models, and then discuss what kinds of information can be used as hints and how to use them to help the training of the NART model.
Revised: We ﬁrst describe the observations we ﬁnd about the ART and NART models, and then discuss what kinds of information can be used as hints and how to use them to help the training of the NART model.

Original: 3. O BSERVATION : I LLED S TATES AND A TTENTIONS
Revised: 3. O BSERVATION : I LLED S TATES AND A TTENTIONS

Original: According to the case study in Gu et al.
Revised: According to the case study in Gu et al.

Original: (2017) and the observations based on our trained model, the translations of the NART models contain incoherent phrases and miss meaningful tokens on the source side.
Revised: (2017) and the observations based on our trained model, the translations of the NART models contain incoherent phrases and miss meaningful tokens on the source side.

Original: As shown in Table 3, these patterns are not commonly appeared in ART models.
Revised: As shown in Table 3, these patterns do not commonly appear in ART models.

Original: We aim to answer why the NART model tends to produce incoherent phrases (e.g. repetitive words) and miss relevant translations.
Revised: We aim to answer why the NART model tends to produce incoherent phrases (e.g. repetitive words) and miss relevant translations.

Original: To study the ﬁrst problem, we visualize the cosine similarities between decoder hidden states of a certain layer in both ART and NART models for sampled cases.
Revised: To study the ﬁrst problem, we visualize the cosine similarities between decoder hidden states of a certain layer in both ART and NART models for sampled cases.

Original: Mathematically, for a set of hidden states r 1 , . .
Revised: Mathematically, for a set of hidden states r 1 , . .

Original: , r T , where T is the number of positions, the pairwise cosine similarity can be derived by cos ij = (cid:104)
Revised: , r T , where T is the number of positions, the pairwise cosine similarity can be derived by cos ij = (cid:104)

Original: r i , r j (cid:105) /
Revised: r i , r j (cid:105) /

Original: ( (cid:107) r i (cid:107) · (cid:107) r j (cid:107) ) .
Revised: ( (cid:107) r i (cid:107) · (cid:107) r j (cid:107) ) .

Original: We then plot the heatmap of the resulting matrix cos , and a typical example is shown in Figure 2.
Revised: We then plot the heatmap of the resulting matrix cos , and a typical example is shown in Figure 2.

Original: From the ﬁgure, we can see that the cosine similarities between the hidden states at different positions in the NART model are larger than those of the ART model, which indicates that the hidden states across positions in the NART model are “similar”.
Revised: From the ﬁgure, we can see that the cosine similarities between the hidden states at different positions in the NART model are larger than those of the ART model, which indicates that the hidden states across positions in the NART model are “similar”.

Original: Positions with highly-correlated hidden states are more likely to generate the same word and make the NART model output repetitive tokens, e.g., the yellow area on the top-left of Figure 2(b).
Revised: Positions with highly-correlated hidden states are more likely to generate the same word and make the NART model output repetitive tokens, e.g., the yellow area on the top-left of Figure 2(b).

Original: However, this problem does not happen in the teacher model.
Revised: However, this problem does not happen in the teacher model.

Original: According to our statistics, 70% of the cosine similarities between hidden states in the teacher model are less than 0.25, and 95% are less than 0.5.
Revised: According to our statistics, 70% of the cosine similarities between hidden states in the teacher model are less than 0.25, and 95% are less than 0.5.

Original: To study the second problem, we visualize the encoder-decoder attentions for sampled cases.
Revised: To study the second problem, we visualize the encoder-decoder attentions for sampled cases.

Original: Good attentions between the source and target sentences are usually considered to lead to accurate translation while poor ones may cause bad translation with wrong tokens.
Revised: Good attentions between the source and target sentences are usually considered to lead to accurate translation while poor ones may cause bad translation with wrong tokens.

Original: As shown in Figure 3, the attentions of the ART model almost covers all source tokens, while the attentions of the NART model do not cover “farm” but with two “morning”.
Revised: As shown in Figure 3, the attentions of the ART model almost covers all source tokens, while the attentions of the NART model do not cover “farm” but with two “morning”.

Original: This directly makes the translation result worse j e d e n _ m o r g e n _ f li e g e n _
Revised: This directly makes the translation result worse j e d e n _ m o r g e n _ f li e g e n _

Original: s i e _ 240 _ k il o m e t e r _ z u r _ f a r m _ .
Revised: s i e _ 240 _ k il o m e t e r _ z u r _ f a r m _ .

Original: _ < E O S > (a) ART Teacher every_ morning_ , _ they_ fly_240_ miles_to_ the_farm_ .
Revised: _ < E O S > (a) ART Teacher every_ morning_ , _ they_ fly_240_ miles_to_ the_farm_ .

Original: _ <EOS> j e d e n _ m o r g
Revised: _ <EOS> j e d e n _ m o r g

Original: e
Revised: e

Original: n _ f li e g e n _ s i e _ 240 _ k il o m e t e r _ z u r _ f a r m _ .
Revised: n _ f li e g e n _ s i e _ 240 _ k il o m e t e r _ z u r _ f a r m _ .

Original: _ < E O S > (
Revised: _ < E O S > (

Original: [Figure]
Revised: [Figure]

Original: every
Revised: every

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: fly
Revised: fly

Original: [Figure]
Revised: [Figure]

Original: miles
Revised: miles

Original: [Figure]
Revised: [Figure]

Original: j e d e n _ m o r g e n _ f li e g e n _
Revised: j e d e n _ m o r g e n _ f li e g e n _

Original: s i e _ 240 _ k il o m e t e r _ z u r _ f a r m _ .
Revised: s i e _ 240 _ k il o m e t e r _ z u r _ f a r m _ .

Original: _ < E O S > (c) NART
Revised: _ < E O S > (c) NART

Original: [Figure]
Revised: [Figure]

Original: Hints every_ morning_ , _ they_ fly_
Revised: Hints every_ morning_ , _ they_ fly_

Original: [Figure]
Revised: [Figure]

Original: miles_to
Revised: miles_to

Original: [Figure]
Revised: [Figure]

Original: the_farm_ .
Revised: the_farm_ .

Original: _ <EOS> 0.
Revised: _ <EOS> 0.

Original: in the NART model.
Revised: in the NART model.

Original: These phenomena inspire us to use the intermediate hidden information in the
Revised: These phenomena inspire us to use the intermediate hidden information in the

Original: ART model to guide the learning process of the NART model
Revised: ART model to guide the learning process of the NART model

Original: . 3. H INT -B ASED T RAINING WITH A UTOREGRESSIVE T EACHER
Revised: . 3. H INT -B ASED T RAINING WITH A UTOREGRESSIVE T EACHER

Original: The empirical study in the previous section motivates us to leverage intermediate hidden information from a teacher translation model to help the training of a student model, which is usually referred to as hint-based training .
Revised: The empirical study in the previous section motivates us to leverage intermediate hidden information from a teacher translation model to help the training of a student model, which is usually referred to as hint-based training .

Original: Hint-based training (Romero et al., 2014; Wang et al., 2016; Chen et al., 2017) is popularly used to transfer complicated nonlinear mappings from one convolutional neural network to another.
Revised: Hint-based training (Romero et al., 2014; Wang et al., 2016; Chen et al., 2017) is popularly used to transfer complicated nonlinear mappings from one convolutional neural network to another.

Original: In our scenario, we focus on how to deﬁne hints from a well-trained ART teacher model and use it to guide the training process of a NART student model.
Revised: In our scenario, we focus on how to deﬁne hints from a well-trained ART teacher model and use it to guide the training process of a NART student model.

Original: We study layer-tolayer hints and assume both the teacher model and the student model have an M -layer encoder and an N -layer decoder, despite that the stacked components are quite different.
Revised: We study layer-tolayer hints and assume both the teacher model and the student model have an M -layer encoder and an N -layer decoder, despite that the stacked components are quite different.

Original: Without loss of generality, we discuss our proposed method on a given paired sentence ( x, y ) .
Revised: Without loss of generality, we discuss our proposed method on a given paired sentence ( x, y ) .

Original: In real experiments, losses are averaged over all training data.
Revised: In real experiments, losses are averaged over all training data.

Original: For the teacher model, we use a tr t,l,h as the encoder-to-decoder attention distribution of h -th head in the l -th decoder layer at position t , and use r tr t,l as the output of the l -th decoder layer after feed forward network at position t .
Revised: For the teacher model, we use a tr t,l,h as the encoder-to-decoder attention distribution of h -th head in the l -th decoder layer at position t , and use r tr t,l as the output of the l -th decoder layer after feed forward network at position t .

Original: Correspondingly, a st t,l,h and r st t,l are used for the student model.
Revised: Correspondingly, a st t,l,h and r st t,l are used for the student model.

Original: We propose a hint-based training framework that contains two kinds of hints: hints from hidden states and hints from word alignments.
Revised: We propose a hint-based training framework that contains two kinds of hints: hints from hidden states and hints from word alignments.

Original: Hints from hidden states The discrepancy of hidden states between ART and NART models motivates us to use hidden states of ART model as a hint for the learning process of the NART model.
Revised: Hints from hidden states The discrepancy of hidden states between ART and NART models motivates us to use hidden states of ART model as a hint for the learning process of the NART model.

Original: One of the straight-forward methods is to regularize the L 1 or L 2 distance between each pair of hidden states in ART and NART models.
Revised: One of the straight-forward methods is to regularize the L 1 or L 2 distance between each pair of hidden states in ART and NART models.

Original: However, since the decoder input and network components are completely different in ART and NART models, we ﬁnd using straight-forward regression method on hidden states hurts the learning of the translation model and fails.
Revised: However, since the decoder input and network components are completely different in ART and NART models, we ﬁnd using straight-forward regression method on hidden states hurts the learning of the translation model and fails.

Original: Therefore, we design a more implicit loss to help the student refrain from the incoherent translation results by acting towards the teacher in the hidden-state level.
Revised: Therefore, we design a more implicit loss to help the student refrain from the incoherent translation results by acting towards the teacher in the hidden-state level.

Original: Speciﬁcally, we have
Revised: Speciﬁcally, we have

Original: [Equation]
Revised: [Equation]

Original: where φ is a penalty function.
Revised: where φ is a penalty function.

Original: In particular, we let
Revised: In particular, we let

Original: [Equation]
Revised: [Equation]

Original: where − 1 ≤ γ st , γ tr ≤ 1 are two thresholds controlling whether to penalize or not.
Revised: where − 1 ≤ γ st , γ tr ≤ 1 are two thresholds controlling whether to penalize or not.

Original: We design this loss since we only want to penalize hidden states that are highly similar in the NART model, but not similar in the ART model.
Revised: We design this loss since we only want to penalize hidden states that are highly similar in the NART model, but not similar in the ART model.

Original: We have tested several alternative choices of − log(1 − d st ) , e.g., exp( d st ) , from which we ﬁnd similar experimental results.
Revised: We have tested several alternative choices of − log(1 − d st ) , e.g., exp( d st ) , from which we ﬁnd similar experimental results.

Original: Hints from word alignments
Revised: Hints from word alignments

Original: Attention mechanism greatly boosts the performance of the ART models (Bahdanau et al., 2014) and becomes a crucial building block.
Revised: Attention mechanism greatly boosts the performance of the ART models (Bahdanau et al., 2014) and becomes a crucial building block.

Original: Many papers discover that the attentions provide reasonable word/phrase alignments between the source and target lead to better performance when predicting target tokens.
Revised: Many papers discover that the attentions provide reasonable word/phrase alignments between the source and target lead to better performance when predicting target tokens.

Original: As we observe that meaningful words in the source sentence are sometimes untranslated by the NART model, and the corresponding positions often suffer from ambiguous attention distributions as shown in Figure 3, we use the word alignment information from the ART model to help the training of the NART model.
Revised: As we observe that meaningful words in the source sentence are sometimes untranslated by the NART model, and the corresponding positions often suffer from ambiguous attention distributions as shown in Figure 3, we use the word alignment information from the ART model to help the training of the NART model.

Original: In particular, we minimize KL-divergence between the per-head encoder-to-decoder attention distributions of the teacher and the student to encourage the student to have similar word alignments to the teacher model, i.e.
Revised: In particular, we minimize KL-divergence between the per-head encoder-to-decoder attention distributions of the teacher and the student to encourage the student to have similar word alignments to the teacher model, i.e.

Original: [Equation]
Revised: [Equation]

Original: Our ﬁnal training loss L is a weighted sum of two parts stated above and the negative log-likelihood loss L nll deﬁned on bilingual sentence pair ( x, y ) , i.
Revised: Our ﬁnal training loss L is a weighted sum of two parts stated above and the negative log-likelihood loss L nll deﬁned on bilingual sentence pair ( x, y ) , i.

Original: [Equation]
Revised: [Equation]

Original: where λ and µ are hyperparameters controlling the weight of different loss terms.
Revised: where λ and µ are hyperparameters controlling the weight of different loss terms.

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETTINGS
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETTINGS

Original: We evaluate our methods on two widely used public machine translation datasets: IWSLT
Revised: We evaluate our methods on two widely used public machine translation datasets: IWSLT

Original: German-to-English (De-En) (Huang et al., 2017; Bahdanau et al., 2016) and WMT14 English-toGerman (En-De) dataset
Revised: German-to-English (De-En) (Huang et al., 2017; Bahdanau et al., 2016) and WMT14 English-toGerman (En-De) dataset

Original: (Wu et al., 2016; Gehring et al., 2017).
Revised: (Wu et al., 2016; Gehring et al., 2017).

Original: IWSLT

Revised: IWSLT

Original: De-En is a relatively smaller dataset comparing to WMT14 En-De.
Revised: De-En is a relatively smaller dataset comparing to WMT14 En-De.

Original: To compare with previous works, we also reverse
Revised: To compare with previous works, we also reverse

Original: WMT14 English-to-German dataset and obtain WMT14 German-to-English dataset.
Revised: WMT14 English-to-German dataset and obtain WMT14 German-to-English dataset.

Original: We pretrain Transformer (Vaswani et al., 2017) as the autoregressive teacher model on each dataset.
Revised: We pretrain Transformer (Vaswani et al., 2017) as the autoregressive teacher model on each dataset.

Original: The teacher models achieve 33.26/27.30/31.29 in terms of BLEU in IWSLT

Revised: The teacher models achieve 33.26/27.30/31.29 in terms of BLEU in IWSLT

Original: De-En, WMT
Revised: De-En, WMT

Original: En-De, De-En test set, respectively.
Revised: En-De, De-En test set, respectively.

Original: The student model shares the same number of layers in encoder/decoder, size of hidden states/embeddings and number of heads as the teacher models in each task.
Revised: The student model shares the same number of layers in encoder/decoder, size of hidden states/embeddings and number of heads as the teacher models in each task.

Original: Following Gu et al.
Revised: Following Gu et al.

Original: (2017), we replace the target sentences in all datasets by the decoded output of the teacher models.
Revised: Kim & Rush (2016), we replace the target sentences in all datasets by the decoded output of the teacher models.

Original: Hyperparameters for hints based training ( γ st , γ tr , λ, µ ) are determined to make the scales of three loss components similar after initialization.
Revised: Hyperparameters for hints based training ( γ st , γ tr , λ, µ ) are determined to make the scales of three loss components similar after initialization.

Original: We also employ label smoothing of value (cid:15) ls = 0 .(Szegedy
Revised: We also employ label smoothing of value (cid:15) ls = 0 .(Szegedy

Original: et al., 2016) in all experiments.
Revised: et al., 2016) in all experiments.

Original: We use Adam optimizer and follow the optimizer setting and learning rate schedule in Vaswani et al.
Revised: We use Adam optimizer and follow the optimizer setting and learning rate schedule in Vaswani et al.

Original: Models for WMT14/IWSLT14 tasks are trained on 8/1 NVIDIA M40 GPUs respectively.
Revised: Models for WMT14/IWSLT14 tasks are trained on 8/1 NVIDIA M40 GPUs respectively.

Original: We implement our model based on the open-sourced tensor2tensor (Vaswani et al., 2018) and plan to release it in the near future.
Revised: We implement our model based on the open-sourced tensor2tensor (Vaswani et al., 2018) and plan to release it in the near future.

Original: More experimental settings can be found in the appendix.
Revised: More experimental settings can be found in the appendix.

Original: 4. I NFERENCE
Revised: 4. I NFERENCE

Original: During training, T y does not need to be predicted as the target sentence is given.
Revised: During training, T y does not need to be predicted as the target sentence is given.

Original: During testing, we have to predict the length of the target sentence for each source sentence.
Revised: During testing, we have to predict the length of the target sentence for each source sentence.

Original: In many languages, the length of the target sentence can be roughly estimated from the length of the source sentence.
Revised: In many languages, the length of the target sentence can be roughly estimated from the length of the source sentence.

Original: For example, if the source sentence is very long, its translation is also a long sentence.
Revised: For example, if the source sentence is very long, its translation is also a long sentence.

Original: We provide a simple method to avoid the computational overhead, which uses input length to determine target sentence length:
Revised: We provide a simple method to avoid the computational overhead, which uses input length to determine target sentence length:

Original: T y =
Revised: T y =

Original: T x + C , where C is a constant bias determined by the average length differences between the source and target sentences in the training data.
Revised: T x + C , where C is a constant bias determined by the average length differences between the source and target sentences in the training data.

Original: We can also predict the target length ranging from [( T x + C ) − B, ( T x + C ) + B ] , where B is the halfwidth.
Revised: We can also predict the target length ranging from [( T x + C ) − B, ( T x + C ) + B ] , where B is the halfwidth.

Original: By doing this, we can obtain multiple translation results with different lengths.
Revised: By doing this, we can obtain multiple translation results with different lengths.

Original: [Table]
Revised: [Table]

Original: Once we have multiple translation results, we additionally use our ART teacher model to evaluate each result and select the one that achieves the highest probability.
Revised: Once we have multiple translation results, we additionally use our ART teacher model to evaluate each result and select the one that achieves the highest probability.

Original: As the evaluation is fully parallelizable (since it is identical to the parallel training of the ART model), this rescoring operation will not hurt the non-autoregressive property of the NART model.
Revised: As the evaluation is fully parallelizable (since it is identical to the parallel training of the ART model), this rescoring operation will not hurt the non-autoregressive property of the NART model.

Original: We use BLEU score (Papineni et al., 2002) as our evaluation measure.
Revised: We use BLEU score (Papineni et al., 2002) as our evaluation measure.

Original: During inference, we set C to 2 , − 2 , 2 for WMT14 En-De, De-En and IWSLT14 De-En datasets respectively, according to the average lengths of different languages in the training sets.
Revised: During inference, we set C to 2 , − 2 , 2 for WMT14 En-De, De-En and IWSLT14 De-En datasets respectively, according to the average lengths of different languages in the training sets.

Original: When using the teacher to rescore, we set B = 4 and thus have 9 candidates in total.
Revised: When using the teacher to rescore, we set B = 4 and thus have 9 candidates in total.

Original: We also evaluate the average per-sentence decoding latencies on one NVIDIA TITAN
Revised: We also evaluate the average per-sentence decoding latencies on one NVIDIA TITAN

Original: Xp GPU card by decoding on WMT14 En-De test sets with batch size 1 for our ART teacher model and NART models, and calculate the speedup based on them.
Revised: Xp GPU card by decoding on WMT14 En-De test sets with batch size 1 for our ART teacher model and NART models, and calculate the speedup based on them.

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: We compare our model with several baselines: LSTM-based, convolution-based, self attentionbased ART models, the fertility based (FT) NART model, the deterministic iterative reﬁnement based (IR) NART model, and the Latent Transformer (LT) which is not fully non-autoregressive by incorporating an autoregressive sub-module in the NART model architecture.
Revised: We compare our model with several baselines: LSTM-based, convolution-based, self attentionbased ART models, the fertility based (FT) NART model, the deterministic iterative reﬁnement based (IR) NART model, and the Latent Transformer (LT) which is not fully non-autoregressive by incorporating an autoregressive sub-module in the NART model architecture.

Original: The experimental results are shown in the Table 1.
Revised: The experimental results are shown in the Table 1.

Original: Across different datasets, our method achieves state-of-the-art performances with signiﬁcant improvements over previous proposed non-autoregressive models.
Revised: Across different datasets, our method achieves state-of-the-art performances with signiﬁcant improvements over previous proposed non-autoregressive models.

Original: Speciﬁcally, our method outperforms fertility based NART model with 6.54/7.11 BLEU score improvements on WMT En-De and
Revised: Speciﬁcally, our method outperforms fertility based NART model with 6.54/7.11 BLEU score improvements on WMT En-De and

Original: De-En tasks in similar settings.
Revised: De-En tasks in similar settings.

Original: Comparing to the ART models, our method achieves comparable results with state-of-the-art LSTM-based sequence-to-sequence model on WMT En-De task.
Revised: Comparing to the ART models, our method achieves comparable results with state-of-the-art LSTM-based sequence-to-sequence model on WMT En-De task.

Original: Apart from the translation accuracy, our model achieves a speedup of 30.2 (output a single sentence) or 17.8 (teacher rescoring) times over the ART counterparts.
Revised: Apart from the translation accuracy, our model achieves a speedup of 30.2 (output a single sentence) or 17.8 (teacher rescoring) times over the ART counterparts.

Original: Note that our speedups signiﬁcantly
Revised: Note that our speedups signiﬁcantly

Original: [Table]
Revised: [Table]

Original: outperform all previous works, because of our lighter design of the NART model: without any computationally expensive module trying to improve the expressiveness
Revised: outperform all previous works, because of our lighter design of the NART model: without any computationally expensive module trying to improve the expressiveness

Original: [Table]
Revised: [Table]

Original: We provide some case studies for the NART models with and without hints in Table 3.
Revised: We provide some case studies for the NART models with and without hints in Table 3.

Original: More cases can be found in the appendix.
Revised: More cases can be found in the appendix.

Original: From the ﬁrst case, we can see that the model without hints translates the meaning of “as far as I’m concerned” to a set of meaningless tokens.
Revised: From the ﬁrst case, we can see that the model without hints translates the meaning of “as far as I’m concerned” to a set of meaningless tokens.

Original: In the second case, the model without hints omits the phrase “the farm” and replaces it with a repetitive phrase “every morning” .
Revised: In the second case, the model without hints omits the phrase “the farm” and replaces it with a repetitive phrase “every morning” .

Original: In the third case, the model without hints mistakenly puts the word “uploaded” to the beginning of the sentence, whereas our model correctly translates the source sentence.
Revised: In the third case, the model without hints mistakenly puts the word “uploaded” to the beginning of the sentence, whereas our model correctly translates the source sentence.

Original: In all cases, hint-based training helps the NART model to generate better target sentences.
Revised: In all cases, hint-based training helps the NART model to generate better target sentences.

Original: [Table]
Revised: [Table]

Original: We also visualize the hidden state cosine similarities and attention distributions for the NART model with hint-based training, as shown in Figure 2(c) and 3(c).
Revised: We also visualize the hidden state cosine similarities and attention distributions for the NART model with hint-based training, as shown in Figure 2(c) and 3(c).

Original: With hints from hidden states, the hidden states similarities of the NART model decrease in general, and especially for the positions where the original NART model outputs incoherent phrases.
Revised: With hints from hidden states, the hidden states similarities of the NART model decrease in general, and especially for the positions where the original NART model outputs incoherent phrases.

Original: The attention distribution of the NART model after hint-based training is more similar to the ART teacher model and less ambiguous comparing with the NART model without hints.
Revised: The attention distribution of the NART model after hint-based training is more similar to the ART teacher model and less ambiguous comparing with the NART model without hints.

Original: Finally, we study the effectiveness of different parts and compare it with a NART model without hints.
Revised: Finally, we study the effectiveness of different parts and compare it with a NART model without hints.

Original: We conduct an ablation study on IWSLT14 De-En task and the results are shown in Table 2.
Revised: We conduct an ablation study on IWSLT14 De-En task and the results are shown in Table 2.

Original: The hints from word alignments provide an improvement of about 1.6 BLEU points, and the hints from hidden states improve the results by about 0.8 points in terms of BLEU.
Revised: The hints from word alignments provide an improvement of about 1.6 BLEU points, and the hints from hidden states improve the results by about 0.8 points in terms of BLEU.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Non-autoregressive translation (NART) models have suffered from low-quality translation results.
Revised: Non-autoregressive translation (NART) models have suffered from low-quality translation results.

Original: In this paper, we proposed to use hints from well-trained autoregressive translation (ART) models to enhance the training of NART models.
Revised: In this paper, we proposed to use hints from well-trained autoregressive translation (ART) models to enhance the training of NART models.

Original: Our results on WMT14 En-De and De-En signiﬁcantly outperform previous NART baselines, and achieve comparable accuracy to an LSTM-based ART model, with one order of magnitude faster in inference.
Revised: Our results on WMT14 En-De and De-En signiﬁcantly outperform previous NART baselines, and achieve comparable accuracy to an LSTM-based ART model, with one order of magnitude faster in inference.

Original: In the future, we will focus on designing new architectures and new training methods for NART models to achieve comparable accuracy as the state-of-the-art ART models such as Transformer.
Revised: In the future, we will focus on designing new architectures and new training methods for NART models to achieve comparable accuracy as the state-of-the-art ART models such as Transformer.

Original: A N ETWORK A RCHITECTURE
Revised: A N ETWORK A RCHITECTURE

Original: The NART model is developed using the general encoder-decoder framework which is the same as the ART models.
Revised: The NART model is developed using the general encoder-decoder framework which is the same as the ART models.

Original: Figure 1 shows the architecture of an ART and a NART model based on
Revised: Figure 1 shows the architecture of an ART and a NART model based on

Original: Transformer.
Revised: Transformer.

Original: We introduce several components that could help in understanding our proposed hintbased training algorithm.
Revised: We introduce several components that could help in understanding our proposed hintbased training algorithm.

Original: More details about general architectures can be found in Vaswani et al.
Revised: More details about general architectures can be found in Vaswani et al.

Original: (2017); Gu et al.
Revised: (2017); Gu et al.

Original: Encoder and decoder Same as the ART model, the encoder of the NART model takes the embeddings of source tokens as inputs 2 and generates a set of context vectors.
Revised: Encoder and decoder Same as the ART model, the encoder of the NART model takes the embeddings of source tokens as inputs 2 and generates a set of context vectors.

Original: As discussed in Section 2.2, the NART model needs to predict z given length T y and source sentence x .
Revised: As discussed in Section 2.2, the NART model needs to predict z given length T y and source sentence x .

Original: We use a simple and efﬁcient method to predict z = ( z 1 , . .
Revised: We use a simple and efﬁcient method to predict z = ( z 1 , . .

Original: , z T y ) .
Revised: , z T y ) .

Original: Given source sentence x = ( x 1 , . . .
Revised: Given source sentence x = ( x 1 , . . .

Original: , x T x ) and target length T y , we denote e ( x i ) as the embedding of x i .
Revised: , x T x ) and target length T y , we denote e ( x i ) as the embedding of x i .

Original: We linearly combine the embeddings of all the source tokens to generate z as follows:
Revised: We linearly combine the embeddings of all the source tokens to generate z as follows:

Original: [Equation]
Revised: [Equation]

Original: where w ij is the normalized weight that controls the contribution of e ( x i ) to z j according to
Revised: where w ij is the normalized weight that controls the contribution of e ( x i ) to z j according to

Original: [Equation]
Revised: [Equation]

Original: where j (cid:48)
Revised: where j (cid:48)

Original: ( i ) =
Revised: ( i ) =

Original: ( T y /T x ) · i .
Revised: ( T y /T x ) · i .

Original: τ is a hyperparameter to control the “sharpness” of the weight distribution.
Revised: τ is a hyperparameter to control the “sharpness” of the weight distribution.

Original: We use f z ( x, T y , τ ) for this weighted average function to be consistent as in the Eqn.
Revised: We use f z ( x, T y , τ ) for this weighted average function to be consistent as in the Eqn.

Original: Three types of multi-head attention The ART and NART models share two types of multi-head attentions: multi-head self attention and multi-head encoder-to-decoder attention.
Revised: Three types of multi-head attention The ART and NART models share two types of multi-head attentions: multi-head self attention and multi-head encoder-to-decoder attention.

Original: The NART model speciﬁcally uses multi-head positional attention to model local word orders within the sentence (Vaswani et al., 2017; Gu et al., 2017).
Revised: The NART model speciﬁcally uses multi-head positional attention to model local word orders within the sentence (Vaswani et al., 2017; Gu et al., 2017).

Original: A general attention mechanism can be formulated as querying a dictionary with key-value pairs (Vaswani et al., 2017), e.g.,
Revised: A general attention mechanism can be formulated as querying a dictionary with key-value pairs (Vaswani et al., 2017), e.g.,

Original: [Equation]
Revised: [Equation]

Original: where d model is the dimension of hidden representations and Q (Query), K (Key), V (Value) differ among three types of attentions.
Revised: where d model is the dimension of hidden representations and Q (Query), K (Key), V (Value) differ among three types of attentions.

Original: For self attention, Q , K and V are hidden representations of the previous layer.
Revised: For self attention, Q , K and V are hidden representations of the previous layer.

Original: For encoder-to-decoder attention, Q is hidden representations of the previous layer, whereas K and V are context vectors from the encoder.
Revised: For encoder-to-decoder attention, Q is hidden representations of the previous layer, whereas K and V are context vectors from the encoder.

Original: For positional attention, positional embeddings are used as Q and K , and hidden representations of the previous layer are used as V .
Revised: For positional attention, positional embeddings are used as Q and K , and hidden representations of the previous layer are used as V .

Original: The multi-head variant of attention allows the model to jointly attend to information from different representation subspaces, and is deﬁned as
Revised: The multi-head variant of attention allows the model to jointly attend to information from different representation subspaces, and is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where W Qh ∈ R d model × d k , W Kh ∈ R d model × d k , W Vh ∈ R d model × d k , and W Oh ∈ R d model ×
Revised: where W Qh ∈ R d model × d k , W Kh ∈ R d model × d k , W Vh ∈ R d model × d k , and W Oh ∈ R d model ×

Original: Hd v are project parameter matrices, H is the number of heads, and d k and d v are the numbers of dimensions.
Revised: Hd v are project parameter matrices, H is the number of heads, and d k and d v are the numbers of dimensions.

Original: In addition to multi-head attentions, the encoder and decoder also contain fully connected feedforward network (FFN) layers with ReLU activations, which are applied to each position separately and identically.
Revised: In addition to multi-head attentions, the encoder and decoder also contain fully connected feedforward network (FFN) layers with ReLU activations, which are applied to each position separately and identically.

Original: Compositions of self attention, encoder-to-decoder attention, positional attention, and position-wise feed-forward network are stacked to form the encoder and decoder of the ART model and the NART model, with residual connections (He et al., 2016) and layer normalization (Ba et al., 2016).
Revised: Compositions of self attention, encoder-to-decoder attention, positional attention, and position-wise feed-forward network are stacked to form the encoder and decoder of the ART model and the NART model, with residual connections (He et al., 2016) and layer normalization (Ba et al., 2016).

Original: B E XTRA E XPERIMENTAL S ETTINGS
Revised: B E XTRA R ELATED W ORKS ON K NOWLEDGE D ISTILLATION AND H INT -B ASED T RAINING

Revised: Knowledge Distillation (KD) was ﬁrst proposed by Hinton et al. (2015), which trains a small student network from a large (possibly ensemble) teacher network . The training objective of the student network contains two parts.

Revised: The ﬁrst part is the standard classiﬁcation loss, e.g, the cross entropy loss deﬁned on the student network and the training data.

Revised: The second part is deﬁned between the output distributions of the student network and the teacher network, e.g, using KL-divergence .

Revised: Kim & Rush (2016) introduces the KD framework to neural machine translation models.

Revised: They replace the ground truth target sentence by the generated sentence from a well-trained teacher model.

Revised: Sentencelevel KD is also proved helpful for non-autoregressive translation in multiple previous works (Gu et al., 2017; Lee et al., 2018).

Revised: However, knowledge distillation only uses the outputs of the teacher model, but ignores the rich hidden information inside a teacher model.

Revised: Romero et al.

Revised: (2014) introduced hint-based training to leverage the intermediate representations learned by the teacher model as hints to improve the training process and ﬁnal performance of the student model.

Revised: Hu et al.

Revised: (2018) used the attention weights as hints to train a small student network for reading comprehension.

Revised: C E XTRA E XPERIMENTAL S ETTINGS

Original: Dataset speciﬁcations The training/validation/test sets of the IWSLT14 dataset 3 contain about 153K/7K/7K sentence pairs, respectively.
Revised: Dataset speciﬁcations The training/validation/test sets of the IWSLT14 dataset 3 contain about 153K/7K/7K sentence pairs, respectively.

Original: The training set of the WMT14 dataset 4 contains 4.5M parallel sentence pairs.
Revised: The training set of the WMT14 dataset 4 contains 4.5M parallel sentence pairs.

Original: Newstest2014 is used as the test set, and Newstest2013 is used as the validation set.
Revised: Newstest2014 is used as the test set, and Newstest2013 is used as the validation set.

Original: In both datasets, tokens are split into a 32000 word-piece dictionary (Wu et al., 2016) which is shared in source and target languages.
Revised: In both datasets, tokens are split into a 32000 word-piece dictionary (Wu et al., 2016) which is shared in source and target languages.

Original: Model speciﬁcations For the WMT14 dataset, we use the default network architecture of the base Transformer model in Vaswani et al.
Revised: Model speciﬁcations For the WMT14 dataset, we use the default network architecture of the base Transformer model in Vaswani et al.

Original: (2017), which consists of a 6-layer encoder and 6-layer decoder.
Revised: (2017), which consists of a 6-layer encoder and 6-layer decoder.

Original: The size of hidden nodes and embeddings are set to 512.
Revised: The size of hidden nodes and embeddings are set to 512.

Original: For the IWSLT14 dataset, we use a smaller architecture, which consists of a 5-layer encoder, and a 5-layer decoder.
Revised: For the IWSLT14 dataset, we use a smaller architecture, which consists of a 5-layer encoder, and a 5-layer decoder.

Original: The size of hidden states and embeddings are set to 256 and the number of heads is set to 4.
Revised: The size of hidden states and embeddings are set to 256 and the number of heads is set to 4.

Original: Hyperparameter speciﬁcations Hyperparameters ( τ, γ st , γ tr , λ, µ ) are determined to make the scales of three loss components  after initialization.
Revised: Hyperparameter speciﬁcations Hyperparameters ( τ, γ st , γ tr , λ, µ ) are determined to make the scales of three loss components similar after initialization.

Original: Speciﬁcally, we use τ = 0 .
Revised: Speciﬁcally, we use τ = 0 .

Original: 3 , γ st = 0 .
Revised: 3 , γ st = 0 .

Original: 1 , γ tr = 0 .
Revised: 1 , γ tr = 0 .

Original: 9 , λ = 5 . 0 , µ = 1 . 0 for IWSLT

Revised: 9 , λ = 5 . 0 , µ = 1 . 0 for IWSLT

Original: De-En, τ = 0 .
Revised: De-En, τ = 0 .

Original: 3 , γ st = 0 .
Revised: 3 , γ st = 0 .

Original: 5 , γ tr = 0 .
Revised: 5 , γ tr = 0 .

Original: 9 , λ = 5 . 0 , µ = 1 . 0 for WMT14 De-En and WMT14 En-De.
Revised: 9 , λ = 5 . 0 , µ = 1 . 0 for WMT14 De-En and WMT14 En-De.

Original: BLEU scores We use tokenized case-sensitive BLEU (Papineni et al., 2002) 5 for WMT14 En-De and De-En datasets, and use tokenized case-insensitive BLEU for
Revised: BLEU scores We use tokenized case-sensitive BLEU (Papineni et al., 2002) 5 for WMT14 En-De and De-En datasets, and use tokenized case-insensitive BLEU for

Original: IWSLT

Revised: IWSLT

Original: De-En dataset, which is a common practice in literature.
Revised: De-En dataset, which is a common practice in literature.

Original: C E XTRA C ASES
Revised: D E XTRA E XPERIMENTS

Original: [Table] -En. [Table] [Table]
Revised: Repetitive words According to the empirical analysis, the percentage of repetitive words drops from 8.3% to 6.5% by our proposed hint-based training algorithm on the test set of IWSLT14 De-En, which is a more than 20% reduction.

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: This shows that our proposed method effectively improve the quality of the translation outputs.

Original: one

Original: [Table] ago [Table] or so
Revised: Performance on long sentences We select the source sentences whose lengths are at least 40 in the test set of IWSLT14 De-En dataset, and test the models trained with different kinds of hints on this subsampled set.

Original: [Table] up properly . [Table] [Table]
Revised: The results are shown in Table 4.

Original: a year ago or something

Original: [Table]

Original: a study

Original: [Table]

Original: me

Original: [Table]

Original: right .

Original: [Table] [Table] [Table] now , when i
Revised: As can be seen from the table, our model

Original: [Table]

Original: here , i see the athletes that

Original: [Table]

Original: in this

Original: [Table]

Original: and the sports species .

Original: [Table]

Original: here , [Table] athletes that [Table]
Revised: BLEU scores are calculated by the scripts at https: outperforms the baseline model by more than 3 points in term of BLEU (20.63 v.s. 17.48).

Original: in this this ogand

Original: [Table]

Original: kinds of sports . [Table] now if i [Table]
Revised: Note that the incoherent patterns like repetitive words are a common phenomenon among sentences of all lengths, rather than a special problem for long sentences.

Original: at here , i see the athlettes that

Original: [Table]
Revised: [Table]

Original: out in this output and the species of sports .

Original: [Table] [Table] sometimes those ideas get ahead of our thinking in ways that are important . ART: sometimes these ideas forecast our thinking in a very important way .
Revised: E E XTRA C ASES

Original: [Table]

Original: : sometimes these ideas make of our thinking , in a very important way .

Original: [Table]
Revised: [Table]