PNAT: N ON - AUTOREGRESSIVE T RANSFORMER BY
P OSITION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Non-autoregressive models are promising on various text generation tasks.
Previous work hardly considers to explicitly model the positions of generated words.
However, the position modeling is an essential problem in non-autoregressive text generation.
In this study, we propose PNAT, which incorporates positions as a latent variable into the text generative process.
Experimental results show that
PNAT achieves top results on machine translation and paraphrase generation tasks, outperforming several strong baselines.
I NTRODUCTION
Transformer (Vaswani et al., 2017) has been widely used in many text generation tasks, which is ﬁrst proposed in neural machine translation, achieving great success for its promising performance.
Nevertheless, the auto-regressive property of Transformer has been a bottleneck.
Speciﬁcally, the decoder of Transformer generates words sequentially, and the latter words are conditioned on previous ones in a sentence.
Such bottleneck prevents the decoder from higher efﬁciency in parallel computation, and imposes strong constrains in text generation, with which the generation order has to be left to right (or right to left) (Shaw et al., 2018; Vaswani et al., 2017).
Recently, many researches (Gu et al., 2018;
Lee et al., 2018; Wang et al., 2019; Wei et al., 2019) are devoted to break the auto-regressive bottleneck by introducing non-autoregressive Transformer (NAT) for neural machine translation, where the decoder generates all words simultaneously instead of sequentially.
Intuitively, NAT abandons feeding previous predicted words into decoder state at the next time step, but directly copy encoded representation at source side to the decoder inputs (Gu et al., 2018).
However, without the auto-regressive constrain, the search space of the output sentence becomes larger, which brings the performance gap between NAT and auto-regressive Transformer (AT).
Related works propose to include some inductive priors or learning techniques to boost the performance of NAT.
But most of previous work hardly consider explicitly modeling the position of output words during text generation.
We argue that position prediction is an essential problem of NAT.
Current NAT approaches do not explicitly model the position of output words, and may ignore the reordering issue in generating output sentences.
Compared to machine translation, the reorder problem is much more severe in tasks such as table-to-text and dialog generations.
Additionally, it is straightforward to explicitly model word positions in output sentences, as position embeddings are used in Transformer, which is natively non-autoregressive, to include the order information.
Intuitively, if output positions are explicitly modeled, the predicted position combined with Transformer to realize non-autoregressive generation would become more natural.
In this paper, we propose non-autoregressive transformer by position learning (PNAT).
PNAT is simple yet effective, which explicitly models positions of output words as latent variables in the text generation.
Speciﬁcally, we introduce a heuristic search process to guide the position learning, and max sampling is adopted to inference the latent model.
The proposed PNAT is motivated by learning syntax position (also called syntax distance)
(Shen et al., 2018).
Shen et al.
(2018) show that syntax position of words in a sentence could be predicted by neural networks in a non-autoregressive fashion, which even obtains top parsing accuracy among strong parser baselines.
Given the observations above, we try to directly predict the positions of output words to build a NAT model for text generation.
Our proposed PNAT takes following advantages:
• We propose PNAT, which ﬁrst includes positions of output words as latent variables for text generation.
Experiments show that PNAT achieves very top results in non-autoregressive NMT, outperforming many strong baselines.
PNAT also obtains better results than AT in paraphrase generation task.
• Further analysis shows that PNAT has great potentials.
With the increase of position prediction accuracy, performances of PNAT could increase signiﬁcantly.
The observations may shed light on the future direction of NAT.
• Thanks to the explicitly modeling of position, we could control the generation by facilitating the position latent variable, which may enable interesting applications such as controlling one special word left to another one.
We leave this as future work.
B ACKGROUND 2. A UTOREGRESSIVE D ECODING
A target sequence Y =
y 1: M is decomposed into a series of conditional probabilities autoregressively, each of which is parameterized using neural networks.
This approach has become a de facto standard in language modeling(Sundermeyer et al., 2012), and has been also applied to conditional sequence modeling p ( Y | X ) by introducing an additional conditional variable X = x 1: N : [Equation]
[Equation]
With different choices of neural network architectures such as recurrent neural networks (RNNs) (Bahdanau et al., 2014; Cho et al., 2014), convolutional neural networks (CNNs) (Krizhevsky et al., 2012;
Gehring et al., 2017), as well as self-attention based transformer (Vaswani et al., 2017), the autoregressive decoding has achieved great success in tasks such as machine translation (Bahdanau et al., 2014), paraphrase generation (Gupta et al., 2018), speech recognition (Graves et al., 2013), etc.
2. N ON - AUTOREGRESSIVE D ECODING
Autoregressive model suffers from the issue of slow decoding in inference, because tokens are generated sequentially and each of them depends on previous ones.
As a solution to this issue, Gu et al.
(2018) proposed Non-Autoregressive Transformer (denoted as NAT) for machine translation, breaking the dependency among the target tokens through time by decoding all the tokens simultaneously.
Put simply, NAT (Gu et al., 2018) factorizes the conditional distribution over a target sequence into a series of conditionally independent distributions with respect to time:
[Equation]
which allows trivially ﬁnding the most likely target sequence by arg max Y p
( Y | X ) for each timestep t , effectively bypassing computational overhead and sub-optimality in decoding from an autoregressive model.
Although non-autoregressive models achieves 15 × speedup in machine translation compared with autoregressive models, it comes at the expense of potential performance degradation (Gu et al., 2018).
The degradation results from the removal of conditional dependencies within the decoding sentence( y t depend on y <t ).
Without such dependencies, the decoder is hard to leverage the inherent sentence structure in prediction.
2. L ATENT V ARIABLES FOR N ON -A UTOREGRESSIVE D ECODING
A non-autoregressive model could be incorporated with conditional dependency as latent variable to alleviate the degradation resulted from the absence of dependency:
[Equation]
[Figure] [Figure] querywiththecorrespondingkey.
We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queriesandkeysofdimension d k ,andvaluesofdimension d v .
Wecomputethedotproductsofthe querywithallkeys,divideeachby p d k ,andapplyasoftmaxfunctiontoobtaintheweightsonthe values. [Figure]
[Figure]
,thedecoderinsertsathirdsub-layer,whichperformsmulti-headattentionovertheoutputoftheencoderstack. Similartotheencoder,weemployresidualconnections aroundeachofthesub-layers,followedbylayernormalization. Wealsomodifytheself-attentionsub-layerinthedecoderstacktopreventpositionsfromattendingtosubsequentpositions. Thismasking,combinedwithfactthattheoutputembeddingsareoffsetbyoneposition,ensuresthatthe predictionsforposition i candependonlyontheknownoutputsatpositionslessthan i .
3.2 Attention
Anattentionfunctioncanbedescribedasmappingaqueryandasetofkey-valuepairstoanoutput,wherethequery,keys,values,andoutputareallvectors.
Theoutputiscomputedasaweightedsumofthevalues,wheretheweightassignedtoeachvalueiscomputedbyacompatibilityfunctionofthe querywiththecorrespondingkey.
3.2.1 ScaledDot-ProductAttention
[-Postion Predictor  y 
-]We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queriesandkeysofdimension d k ,andvaluesofdimension d v . Wecomputethedotproductsofthe querywithallkeys,divideeachby p d k ,andapplyasoftmaxfunctiontoobtaintheweightsonthe values.
[Figure]
For example, NAT-FT (Gu et al., 2018) models the inherent sentence structure with a latent fertility variable, which represents how many target tokens that a source token would translate to.
Lee et al.
(2018) introduces L intermediate predictions Y 1: L as random variables , and to reﬁne the predictions from Y 1 to Y L in a iterative manner.
PNAT: P OSITION -B ASED N ON -A UTOREGRESSIVE T RANSFORMER
We propose position-based non-autoregressive transformer ( PNAT ), an extension to transformer incorporated with non auto-regressive decoding and position learning.
3. M ODELING P OSITION WITH L ATENT V ARIABLES
Languages are usually inconsistent with each other in word order.
Thus reordering is usually required when translating a sentence from a language to another.
In NAT family, words representations or encoder states at source side are copied to the target side to feed into decoder as its input.
Previously,
Gu et al.
(2018) utilizes positional attention which incorporates positional encoding into decoder attention to perform local reordering.
But such implicitly reordering mechanism by position attention may cause a repeated generation problem, because position learning module is not optimized directly, and is likely to be misguided by target supervision.
To tackle with this problem, we propose to explicitly model the position as a latent variable.
We rewrite the target sequence Y with its corresponding position latent variable z
=
z 1: M as a set Y z =
y z 1 : z M .
The conditional probability P ( Y | X ) is factorized with respect to the position latent variable:
[Equation]
where π ( M ) is a set consisting of permutations with M elements.
At decoding time, the factorization allows us to decode sentences in parallel by pre-predicting the corresponding position variables z .
3. M ODEL A RCHITECTURE
As shown in Figure 1, PNAT is composed of four modules: an encoder stack, a bridge block, a position predictor as well as a decoder stack.
Before detailing each component of PNAT model, we overview the architecture for a brief understanding.
Like most sequence-to-sequence models, PNAT ﬁrst encodes a source sequence X = x 1: N into its contextual word representations E =
e 1: N with the encoder stack.
With generated contextual word representation E at source side, the bridge block is leveraged to computed the target length M as well as the corresponding features D = d 1: M , which is fed into the decoder as its input.
It is worth noting that the decoder inputs D is computed without reordering.
Thus the position predictor is introduced to deal with this issue by predicting a permutation z
= z 1: M over D .
Finally, PNAT generates the target sequence from the decoder input D and its permutation z .
Encoder and Decoder Given a source sentence X with length N , PNAT encoder produces its contextual word representations E .
The contextual word representations E are further used in computing target length M and decoder initial states D , and are also used as memory of attention at decoder side.
Generally, PNAT decoder can be considered as a transformer with a broader vision, because it leverages future word information that is blind to the autoregressive transformer.
Intuitively, we use relative position encoding in self-attention(Shaw et al., 2018), rather than absolute one that is more likely to cause position errors.
Following Shaw et al.
(2018) with a clipping distance d (usually d ≥ 2 ) set for relative positions, we preserve d = 4 relations.
Bridge The bridge module predicts the target length M , and initializes the decoder inputs D from the source representations E .
The target length M could be estimated from the source encoder representation:
[Equation]
5) where φ ( · ) produces a categorical distribution ranged in [ − B, B ] .
Then, we adopt the method proposed by Li et al.
(2019) to compute D .
Given the source representation E and the estimated target length M , we linearly combine the embeddings of the neighboring source tokens to generate
D as follows:
[Equation]
where w ji is a normalized weight that reﬂects the contribution of e i to d j , and τ is a hyperparameter indicating the sharpness of the weight distribution.
Position Predictor For the proposed PNAT, we model position permutations with a position predictor.
As shown in Figure 1, the position predictor takes the decoder inputs D and the source representation E to predict a permutation z .
The position predictor has a sub-encoder which stacks multiple layers of encoder units to predict its predicted input R = r 1: M .
Depending on whether the decision-making process is autoregressive or non-autoregressive, there are two types of position predictor: autoregressive position predictor and non-autoregressive position predictor.
With the predicted inputs R , we conduct an autoregressive position predictor, denoted as AR-Predictor .
The AR-Predictor searches a permutation z with:
[Equation]
(8) where θ is the parameter of AR-Predictor, which includes a RNN-based model incorporated with a pointer network (Vinyals et al., 2015).
We also build a non-autoregressive version for the position predictor, denoted as NAR-Predictor, to model the position permutation probabilities with: [Equation]
[Equation]
To obtain the permutation z , AR-Predictor performs greedy search whereas NAR-Predictor performs direct arg max .
3. T RAINING
Training requires maximizing the marginalized likelihood in Eqn. 4.
However, this is intractable since we need to enumerate all the M !
permutations of tokens.
We therefore optimize this objective by Monte Carlo sampling method with a heuristic search algorithm.
Heuristic Search for Positions Intuitively, each target token should have a corresponding decoder input, and meanwhile each decoder input should be assigned to a target token.
Based on this idea, we design a heuristic search algorithm to allocate positions.
Given the decoder inputs and its target tokens, we ﬁrst estimate the similarity between each pair of the decoder input d i and the target token embedding y j , which is also the weights of the target word classiﬁer:
[Equation]
Based on the cosine similarity matrix, HSP is designed to ﬁnd a perfect matching between decoder inputs and target tokens:
[Equation]
Here we apply a greedy algorithm to select the pair with the highest similarity score iteratively until a permutation is generated.
The intuition behind is that, if the decoder input d i is already the most similar one to a target word, it would be easier to keep and even reinforce this association in learning the model.
Objective Function With the heuristically discovered positions as reference positions z ref , the position predictor could be trained with a position loss:
[Equation]
Besides, grounding on the referenced positions, the generative process of target sequences is optimized by:
[Equation]
Finally, combining two loss functions mentioned above, a full-ﬂedged loss is derived as
[Equation]
[Equation]
E XPERIMENTS
We test PNAT on several benchmark sequence generation tasks.
We ﬁrst describe the experimental design and training details, and then present the main results, followed by some deep studies.
4. E XPERIMENTAL D ESIGN
To show the generation ability of PNAT, we conduct experiments on the popular machine translation and paraphrase generation tasks.
These sequence generation task evaluation models from different perspectives.
Translation tasks test the ability of semantic transforming across bilingual corpus.
While paraphrase task focuses on substitution between the same languages while keeping the semantics.
Machine Translation We valid the effectiveness of PNAT on the most widely used benchmarks for machine translation — IWSLT
English-German(196k pairs) and WMT14 English-German(4.5M pairs).
The dataset is processed by Lee et al.
(2018) with Moses script (Koehn et al., 2007), and the words are segmented into subword units using byte-pair encoding (BPE) (Sennrich et al., 2016).
Paraphrase Generation We conduct experiments following previous work (Miao et al., 2019) for paraphrase generation.
We make use of the established Quora dataset 1 to evaluate on the paraphrase generation task.
We consider the supervised paraphrase generation and split the Quora dataset in the standard setting.
We sample 100k pairs sentence as training data, and holds out 3k, 30k for validation and testing, respectively.
4. T RAINING D ETAILS
Model Setting For machine translation, we follow the settings from Gu et al.
In the case of
IWSLT
English-German, we use a small model ( d model = 278, d hidden = 507 , p dropout = 0.1, n layer =and n head = 2) for Transformer(Vaswani et al., 2017) and NAT models.
For WMT14 English-German, we use transformer-base by Vaswani et al.
(2017) ( d model = 512, d hidden = 512 , p dropout = 0.1, n layer = 6).
We also use inverse square root learning rate scheduling(Vaswani et al., 2017) for the WMT14, and using linear annealing(from 3 e − 4 to 1 e − 5 ) suggested by Lee et al.
(2018) for the IWSLT task.
For paraphrase generation, we follow the settings from Miao et al.
(2019), and set the 300-dimensional
GRU with 2 layer for Seq-to-Seq (GRU).
We empirically select a Transformer and NAT models with hyperparameters ( d model = 400, d hidden = 800 , p dropout = 0.1, n layer = 3 and n head = 4).
Each mini-batch consists of approximately 2k tokens for IWSLT16 and paraphrase, 8k tokens for WMT14.
All of our experiments are based on the open-source implementation of dl4mt-nonauto 2 .
The hyperparameter α used in Eqn.
14 was be set to 1.0 for WMT14, 0.3 for IWSLT16 and Quora.
Knowledge Distillation Sequence-level knowledge distillation is applied to alleviate multimodality in the training dataset, using Transformer as a teacher (Hinton et al., 2015).
Previous studies on non-autoregressive generation (Gu et al., 2018; Lee et al., 2018; Wei et al., 2019) have used translations produced by a pre-trained Transformer model as the training data, which signiﬁcantly improves the performance.
We follow this setting in translation tasks and use the released distillation data from https://github.com/nyu-dl/dl4mt-nonauto .
Length Parallel Decoding At the inference stage, we follow the common practice of noisy parallel decoding (Gu et al., 2018), which generates a number of decoding candidates in parallel and selects the best via re-scoring using a pre-trained autoregressive model.
For PNAT, we ﬁrst use the bridge to predict the target length as ˆ M , then generate multiple generation candidates by predicting different target length M ∈ [ ˆ M − ∆ M, ˆ M + ∆ M ] , which was called LPD (length parallel decoding).
The model generates several outputs in parallel, then we use the pre-trained autoregressive model to identify the best overall output.
4. M AIN R ESULTS
Machine Translation We include 5 NAT efforts as our competitors, the NAT with fertility (NATFT), the NAT with iterative reﬁnement (IR-NAT), the NAT with regularization (NAT-REG), the
NAT with enhanced decoder input (ENAT), and the NAT with learning from auto-regressive model (imitate-NAT)
(Gu et al., 2018;
Lee et al., 2018; Wang et al., 2019; Guo et al., 2019; Wei et al., 2019).
For all our tasks, we obtain the performance of competitors by either directly using the performance ﬁgures reported in the previous works if they are available or producing them by using the open source implementation of baseline algorithms on our datasets.
Clearly, PNAT signiﬁcantly outperforms all the competitors with a big margin.
The results are shown in the Table 1.
We basically compare the proposed PNAT against the autoregressive counterpart both in terms of generation quality, which is measured with BLEU (Papineni et al., 2002) and inference speed.
Our best results are obtained with length parallel decoding which employ autoregressive model to rerank the multiple generation candidates of different target length.
On the large scale WMT14 DE-EN task, PNAT (+LPD) surpass the imitate-NMT by 0.6 BLEU score.
Without reranking, the gap has increased to 1 BLEU score (26.65 v.s. 25.67).
The experiments shows the power of explicitly position modeling which reduces the gap between non-autoregressive and the autoregressive models.
[Table]
Paraphrase Generation Given a sentence, paraphrase generation aims to synthesize another sentence that is different from the given one, but conveys the same meaning.
Comparing with translation task, paraphrase generation prefers a more similar order between source and target sentence, which possibly learn a trivial position model.
PNAT can potentially yield better results with the position model to infer the relatively ordered alignment relationship.
The results of paraphrase
[Table]
generation are shown in Table 2.
In consist with our intuition, PNAT achieves the best result on this task and even surpass transformer around 3.5 BLEU.
The comparison between NAT-base and PNAT shows that explicit position modeling in PNAT plays a crucial role in generating sentences.
The NAT model is not power enough to capture the latent position relationship.
It is also worth mentioning that, PNAT with searched-position achieves surprisingly good performance on this task to show the potential of PNAT.
As suggest by the result, a stronger position model may lead to substantial improvements [Table] [Table]
4. A NALYSIS Effectiveness of Position Modeling
We analysis the accuracy of our position modeling and its inﬂuence for the quality of generation on the WMT14 DE-EN task.
For evaluating the position accuracy, we adopt the heuristic searched position as the searched position reference which is the training target of the position predictor.
PNAT requires the position information of two places.
The ﬁrst is the mutual relative relationship between the states that will be used during decoding.
And the second is to reorder the decoded output after decoding.
We then propose the corresponding metrics for evaluation, which is the relative position accuracy (with relation threshold r = 4 ) and the permutation accuracy.
As shown in Table 3, better position accuracy always yields better generation performance.
The position model is crucial to the success of PNAT.
The non-autoregressive position model is less effective than the current autoregressive position model, both in the accuracy of the permutation and the relative position.
Notably PNAT is promising, as PNAT achieves a large improvement with the searched position.
Even though the current PNAT with a simple AR-Predictor have surpassed the previous NAT model, the position accuracy is still less desirable (say, less than 30%) and has a great exploration space.
Impact of Knowledge Distillation Previous NAT model relies heavily on Knowledge Distillation to reduce the difﬁculty of learning from complex patterns.
Generally, NAT can not get satisfactory results without Knowledge Distillation.
One important reason is that without explicit sequential information the model capability of NAT is not powerful enough to capture the complex patterns. [Table]
As showed in Table 4, without knowledge distillation, NAT achieves only 11.02 BLEU score on
WMT14.
Limited by the model capability, NAT has to introduce several tricks to obtain better results.
While for PNAT, we achieve 24.04 BLEU score even without knowledge distillation.
We believe that position learning greatly contributes to improve the model capability of NAT model.
Repeated Generation Analysis Previous NAT often suffers from the repeated generation problem due to the lack of sequential position information.
NAT is less effective to distinguish adjacent decoder hidden states, which is copied from the adjacent source representation.
To further study this problem, we proposed to evaluate the gains of simply remove the repeated tokens. As shown in Table 5, we perform the repeated generation analysis [-in IWSLT
-]on the paraphrase generation tasks. Removing repeated tokens has little impact for PNAT model, with only 0.05 BLEU differences.
However for the NAT-base model, the gap comes with almost 1 BLEU (0.89).
The results clearly demonstrate that the explicitly position model essentially learns the sequential information for sequence generation.
{+Convergence Efﬁciency We also perform the training efﬁciency analysis in IWSLT
+}
DE-EN
Translation task.
The learning curves are shown in 2.
The curve of the PNAT is on the top-left corner.
Remarkably, PNAT has the best convergence speed compared with the NAT competitors and even a strong autoregressive model.
The results are in line with our intuition, that the position
[Table]
learning brings meaningful information of position relationship and beneﬁts the generation of the target sentence.
[Figure]
R ELATED W ORK
Gu et al.
ﬁrst develop a non-autoregressive Transformer for neural machine translation(NMT) tasks, which produces the outputs in parallel and the inference speed is thus signiﬁcantly boosted.
Due to the removal of the dependencies between the target outputs, it comes at the cost that the translation quality is largely sacriﬁces.
A line of work has been proposed to mitigate such performance degradation.
Some previous work is focused on enhancing the decoder inputs by replacing the target words as inputs, such as Guo et al.
and Lee et al.
Lee et al.
(2018) proposed a method of iterative reﬁnement based on latent variable model and denoising autoencoder.
Guo et al.
(2019) enhance decoder input by introducing phrase table in statistical machine translation and embedding transformation.
Another part of previous work focus on improving the supervision of NAT’s decoder states, including imitation learning from autoregressive models (Wei et al., 2019) or regularizing the decoder state with backward reconstruction error (Wang et al., 2019).
Unlike previous work, we explicitly model the position, which has shown its importance to the autoregressive model and can well model the dependence between states.
To the best of our knowledge, PNAT is the ﬁrst work to explicitly model position information for non-autoregressive text generation.
C ONCLUSION
We proposed PNAT, a non-autoregressive transformer by explicitly modeled positions, which bridge the performance gap between the non-autoregressive decoding and autoregressive decoding.
Speciﬁcally, we model the position as latent variables, and training with heuristic searched positions with MC algorithms.
As a result, PNAT leads to signiﬁcant improvement and move more close to the performance gap between the NAT and AT on machine translation tasks.
Besides, the experimental results of the paraphrase generation task show that the performance of the PNAT can exceed that of the autoregressive model, and at the same time, it also has a large improvement space.
According to our further analysis on effectiveness of position modeling, in future work, we can still enhance the performance of the NAT model by strengthening position learning