D ISCOVERING N ON - MONOTONIC A UTOREGRESSIVE O RDERINGS WITH V ARIATIONAL I NFERENCE
A BSTRACT
The predominant approach for language modeling is to encode a sequence of tokens from left to right , but this eliminates a source of information: the order by which the sequence was naturally generated.
One strategy to recover this information is to decode both the content and ordering of tokens.
Some prior work supervises content and ordering with hand-designed loss functions to encourage speciﬁc orders or bootstraps from a predeﬁned ordering.
These approaches require domain-speciﬁc insight.
Other prior work searches over valid insertion operations that lead to ground truth sequences during training, which has high time complexity and cannot be efﬁciently parallelized.
We address these limitations with an unsupervised learner that can be trained in a fully-parallelizable manner to discover high-quality autoregressive orders in a data driven way without a domain-speciﬁc prior.
The learner is a neural network that performs variational inference with the autoregressive ordering as a latent variable.
Since the corresponding variational lower bound is not differentiable, we develop a practical algorithm for end-to-end optimization using policy gradients.
Strong empirical results with our solution on sequence modeling tasks suggest that our algorithm is capable of discovering various autoregressive orders for different sequences that are competitive with or even better than ﬁxed orders.
I NTRODUCTION
Autoregressive models have a rich history.
Early papers that studied autoregressive models, such as (Uria et al., 2016) and (Germain et al., 2015), showed an interest in designing algorithms that did not require a gold-standard autoregressive order to be known upfront by researchers.
However, these papers were overshadowed by developments in natural language processing that demonstrated the power of the left-to-right autoregressive order (Cho et al., 2014; Sutskever et al., 2014a).
Since then, the left-to-right autoregressive order has been essential for application domains such as image captioning (Vinyals et al., 2015b; Xu et al., 2015), machine translation (Luong et al., 2015; Bahdanau et al., 2015) and distant ﬁelds like image synthesis (van den Oord et al., 2016).
However, interest in non left-to-right autoregressive orders is resurfacing (Welleck et al., 2019b; Stern et al., 2019), and evidence (Vinyals et al., 2016; G¯u et al., 2018; Alvarez-Melis & Jaakkola, 2017) suggests adaptive orders may produce more accurate autoregressive models.
These positive results make designing algorithms that can leverage adaptive orders an important research domain.
Inferring autoregressive orderings in a data-driven manner is challenging.
Modern benchmarks for machine translation (Stahlberg, 2019) and other tasks (Oda et al., 2015) are not labelled with gold-standard orders, and left-to-right seems to be the default.
This could be explained if domainindependent methodology for identifying high-quality orders is an open question.
Certain approaches (Stern et al., 2019; Welleck et al., 2019b; Ruis et al., 2020) use hand-designed loss functions to promote a genre of orders—such as balanced binary trees.
These loss functions incorporate certain domain-assumptions: for example, they assume the balanced binary tree order will not disrupt learning.
Learning disruption is an important consideration, because prior work shows that poor orders may prohibitively slow learning (Chen et al., 2018).
Future approaches to inferring autoregressive orders should withhold domain knowledge, to promote their generalization.
To our best knowledge, we propose the ﬁrst domain-independent unsupervised learner that discovers high-quality autoregressive orders through fully-parallelizable end-to-end training without domainspeciﬁc tuning.
We provide three main contributions that stabilize this learner.
First, we propose an encoder architecture that conditions on training examples to output autoregressive orders using techniques in combinatorical optimization.
Second, we propose Variational Order Inference that learns an approximate posterior over autoregressive orders.
Finally, we develop a practical algorithm for solving the resulting non-differentiable ELBO end-to-end with policy gradients.
Empirical results with our solution on image captioning, code generation, text summarization, and machine translation tasks suggest that with similar hyperparameters, our algorithm is capable of recovering autoregressive orders that are even better than ﬁxed orders.
Case studies suggest that our learned orders depend adaptively on content, and resemble a type of best-ﬁrst generation order, which ﬁrst decodes focal objects and names.
Our experimental framework is available at this link.
R ELATED W ORKS
Autoregressive Models Autoregressive models decompose the generation of a high dimensional probability distribution by generating one dimension at a time, with a predeﬁned order.
Combined with high capacity neural networks, this approach to modeling complex distributions has been very successful (Sutskever et al., 2011; Mikolov et al., 2012).
Recent works have achieved great improvements with autoregressive models in many applications, including language modeling (Radford et al., 2018; 2019; Brown et al., 2020), machine translation (Sutskever et al., 2014b) and image captioning (Karpathy & Fei-Fei, 2015).
Most previous works on autoregressive models use a ﬁxed ordering pre-deﬁned by the designer with left-to-right emerging as the primary choice.
In contrast, our method is capable of learning arbitrary orderings conditioned on data and is more ﬂexible.
Non-Monotonic Autoregressive Orderings Ford et al.
(2018b) shows that a sub-optimal ordering can severely limit the viability of a language model and propose to ﬁrst generate a partially ﬁlled sentence template and then ﬁll in missing tokens.
Previous works have also studied bidirectional decoding (Sun et al., 2017;
Zhou et al., 2019; Mehri & Sigal, 2018) and syntax trees based decoding (Yamada & Knight, 2001; Charniak et al., 2003; Dyer et al., 2016; Aharoni & Goldberg, 2017; Wang et al., 2018) in the natural language setting.
However, all of the works mentioned above do not learn the orderings and instead opt to use heuristics to deﬁne them.
Chan et al.
(2019) performs language modeling according to a known prior, such as balanced binary tree, and does not allow arbitrary sequence generation orders.
Welleck et al.
(2019a) proposes to use a tree-based recursive generation method to learn arbitrary generation orders.
However, their performance lags behind that of left-toright.
Gu et al.
(2019a) proposes Transformer-InDIGO to allow non-monotonic sequence generation by ﬁrst pretraining with pre-deﬁned orderings, such as left-to-right, then ﬁne-tuning use Searched Adaptive Order (SAO) to ﬁnd alternative orderings.
They report that without pretraining, the learned orders degenerate.
In addition, they perform beam search when decoding each token during training, which cannot be efﬁciently parallelized at the sequence length dimension.
Emelianenko et al.
(2019) proposes an alternative to SAO, but suffers from similar poor time complexity.
In contrast, our method learns high-quality autoregressive orderings directly from data under fully-parallelizable end-to-end training.
Variational Methods Our method optimizes the evidence lower bound, or ELBO in short.
ELBO is a quantity that is widely used as an optimization proxy in the machine learning literature, where the exact quantity is hard to compute or optimize.
Variational methods have achieved great success in machine learning, such as VAE (Kingma & Welling, 2013) and β -VAE (Higgins et al., 2017).
Combinatorial Optimization Recent works have studied gradient-based optimization in the combinatorial space of permutations (Mena et al., 2018; Grover et al., 2019; Linderman et al., 2018).
These works have been applied in tasks such as number sorting, jigsaw puzzle solving, and neural signal identiﬁcation in worms.
To our best knowledge, we are the ﬁrst to build on these techniques to automatically discover autoregressive orderings in vision and language datasets.
P RELIMINARIES
The goal of autoregressive sequence modelling is to model an ordered sequence of target values y
= ( y 1 , y 2 . . .
, y n ) : y i ∈ R , possibly conditioned on an ordered sequence of source values x = ( x 1 , x 2 . . .
, x m ) : x i ∈ R , where ( x , y ) is sampled from the dataset D .
Inspired by Vinyals et al.
(2015a)
and Gu et al.
(2019a)
, we formulate the generation process of y as a 2 n step process, where at time step 2 t − 1 we generate a value, and at timestep 2 t we select a not-yet-chosen position in { 1 , 2 , · · · , n } to insert the value.
Thus, we introduce the latent sequence variable z
= ( z 1 , z 2 . . .
, z n ) :
z ∈ S n , where S n is the set of one-dimensional permutations of { 1 , 2 , · · · , n } , and z t is deﬁned as the absolute position of the value generated at time step 2 t − 1 in the naturally ordered y .
Then p ( y , z | x ) denotes the probability of generating y in the ordering of z given the source sequence x .
We can thus factorize p ( y , z | x ) using the chain rule:
[Equation]
For example, p ( y 1 , y 2 , z 1 = 2 , z 2 = 1 | x ) =
p ( y 2 | x ) p ( z 1 | y 2 , x ) p ( y 1 | z 1 , y 2 , x ) p ( z 2 | y 1 , z 1 , y 2 , x ) is deﬁned as the probability of generating y 2 in the ﬁrst step, then inserting y 2 into absolute position 2 , then generating y 1 , and ﬁnally inserting y 1 into absolute position 1 .
Note that in practice, the length of y is usually varied.
Therefore, we do not ﬁrst create a ﬁxed-length sequence of blanks and then replace the blanks with actual values.
Instead, we dynamically insert a new value at a position relative to the previous values.
One common approach to predict such relative position is Pointer Network (Vinyals et al., 2015a).
In other words, at timestep t , we insert the value at position r t relative to the previous generated values.
Here, for any z ∈ S n , r = ( r 1 , r 2 , . . .
, r n ) is constructed such that there is a bijection between S n and the set of all constructed r .
Due to such bijection, we can use z and r interchangeably.
We will use z throughout the paper.
V ARIATIONAL O RDER I NFERENCE (VOI)
[Figure]
Starting from just the original data y in natural order, we can use variational inference to create an objective (2) that allows us to recover latent order z , parametrized by two neural networks θ and φ .
The encoder network φ samples autoregressive orders given the ground truth data, which the decoder network θ uses to recover y .
More speciﬁcally, φ is a non-autoregressive network (permutation generator in Fig. 1) that takes in the source sequence x and the entire ground truth target sequence y and outputs latent order
z in a single forward pass.
θ is an autoregressive network (autoregressive decoder in Fig. 1) that takes in x and predicts both the target sequence y and the ordering z through the factorization in Equation (1).
We name this process Variational Order Inference (VOI).
[Equation]
Here, H q φ is the entropy term.
During training, we train φ and θ jointly to maximize the ELBO in (2).
During testing, we only keep the decoder θ .
To optimize the decoder network θ in (2), for each y , we ﬁrst sample K latents z 1 , z 2 , . .
, z K from q φ ( ·| y , x ) .
We then update θ using the Monte-Carlo gradient estimate
E y [Equation] [Equation]
Algorithm
1: Given: encoder network φ with learning rate α φ , decoder network θ with learning rate α θ , entropy coefﬁcient β , batch of training data ( X , Y ) =
{ ( x b , y b ) } Nb =1 sampled from dataset D 2: Set gradient accumulators g φ = 0 , g θ =3: for ( x , y ) ∈
( X , Y ) do (cid:46)
In practice, this is done through parallel tensor operations 4:
X =
φ ( y , x ) 5:
Sample K doubly stochastic matrices B 1 , B 2 , . .
, B K ∈ B n × n from G .
S .
( X, τ ) 6:
Obtain P 1 , P 2 , . .
, P K ∈ P n × n from B 1 , B 2 , . . .
, B K using Hungarian Algorithm 7:
Obtain latents z 1 , z 2 , . .
, z K = f − 1 len ( y ) ( P 1 ) , f − 1 len ( y ) ( P 2 ) , . . .
, f − 1 len ( y ) ( P K ) 8: g θ = g θ
+ 1 N · K (cid:80)
Ki =1 ∇ θ log p θ ( y , z
i | x ) 9:
Calculate log q φ ( z i | y , x ) = (cid:104) X, P i (cid:105) F − log( perm (exp ( X )))
≈ (cid:104) X, P i (cid:105) F − log( perm B (exp( X )))
Calculate b ( y , x ) = 1 K (cid:80) Ki =
[Equation]
[Equation] 12: end [Equation]
[Equation]
[Equation]
Optimizing the encoder network φ is tricky.
Since z is a discrete latent variable, the gradient from log p θ ( y , z ) does not ﬂow through z .
Thus, we formulate (2) in a reinforcement learning setting with a one-step Markov Decision Process ( S , A , R ) .
Under our setting, the state space S =
D ; for each state ( x , y ) ∈ D , the action space
A ( x , y )
=
S length ( y ) with entropy term H q φ ( ·| y , x ) ; the reward function R (( x , y ) , z ∈ S length ( y ) )
= log p θ ( y , z | x ) .
We can then set the optimization objective
L ( φ ) to be (2).
In practice, we ﬁnd that adding an entropy coefﬁcient β and gradually annealing it can speed up the convergence of decoder while still obtaining good autoregressive orders.
To compute ∇ φ L ( φ ) , we derive the policy gradient with baseline formulation (Sutton et al., 2000): [Equation]
[Equation]
where b ( y , x ) is the baseline function independent of action z .
The reason we use a state-dependent baseline b ( y , x ) instead of a global baseline b is that the the length of y can have a wide range, causing signiﬁcant reward scale difference.
In particular, we set b ( y , x ) =
E z ∼ q φ [log p θ ( y , z i | x )] .
If we sample K ≥ 2 latents for each y , then we can use its Monte-Carlo estimate 1 K (cid:80)
Ki =1 log p θ ( y , z i | x ) .
Since we use policy gradient to optimize φ , we still need a closed form for the distribution q φ ( z | y , x ) .
Before we proceed, we deﬁne P n × n as the set of n × n permutation matrices, where exactly one entry in each row and column is 1 and all other entries are 0 ; B n × n as the set of n × n doubly stochastic matrices, i.e. non-negative matrices whose sum of entries in each row and in each column equals 1 ; R + n × n as the set of non-negative n × n matrices.
Note that we have the relationship
P n × n ⊂ B n × n ⊂ R + n × n .
To obtain q φ ( z | y , x ) , we ﬁrst write z in two-dimensional form.
For each z ∈ S n , let f n ( z ) ∈ P n × n be constructed such that f n ( z ) i
= one hot( z i ) , where f n ( z ) i is the i -th row of f n ( z ) .
Thus f n is a natural bijection from S n to P n × n , and we can rewrite q φ as a distribution over P n × n such that q φ
[Equation]
f n ( z ) | y , x ) = q φ ( z | y , x ) . Next, we need to model the distribution of q φ ( ·| y , x ) .
Inspired by (Mena et al., 2018), we model q φ ( ·| y , x ) as a Gumbel-Matching distribution G .
M .
( X ) over P n × n , where X = φ
( y , x ) ∈ R n × n is the output of φ .
Then for P ∈ P n × n ,
[Equation]
where (cid:104) X, P (cid:105) F = trace
( X T P ) is the Frobenius inner product of X and P .
To obtain samples in P n × n from the Gumbel-Matching distribution, (Mena et al., 2018) relaxes P n × n to B n × n by deﬁning the Gumbel-Sinkhorn distribution G .
S .
( X, τ ) : τ > 0 over B n × n , and proves that
G .
S .
( X, τ ) converges almost surely to G .
M .
( X ) as τ → 0 + .
Therefore, to approximately sample from G .
M .
( X ) , we ﬁrst sample from G .
S .
( X, τ ) , then apply Hungarian algorithm (Munkres, 1957) to obtain P ∈ G .
M .
( X ) .
Further details are presented in Appendix A.
The Gumbel-Matching distribution allows us to obtain the numerator for the closed form of q φ ( z | y , x ) =
q φ ( f − 1 n ( P ) | y , x ) , which equals exp (cid:104) X, P (cid:105) F .
However, the denominator is intractable to compute and equals (cid:80) P ∈P n × n exp (cid:104) X, P (cid:105) F .
Upon further examination, we can express it as perm (exp( X )) , the matrix permanent of exp( X ) , and approximate it using perm B (exp( X )) , its Bethe permanent.
We present details about matrix permanent and Bethe permanent along with the proof that the denominator of q φ ( ·| y , x ) equals perm (exp( X )) in Appendix B.
After we approximate q φ , we can now optimize φ using the policy gradient in (3).
We present a diagram of our architecture in Figure 1, and a pseudocode of our algorithm in Algorithm 1.
Note that even though latent space S n is very large and contains n !
permutations, in practice, if p θ ( y , z ∗ | x ) ≥ p θ ( y , z | x ) ∀ z ∈ S n , then p θ ( y , z | x ) tends to increase as the edit distance between z and z ∗ decreases.
Therefore, φ does not need to search over the entire latent to obtain good permutations, making variational inference over S n feasible.
E XPERIMENTS
Encoder and Decoder Architecture.
We present encoder and decoder architectures for Variational
Order Inference on conditional sequence generation tasks, which we focus on in this work.
Note that Algorithm 1 is also applicable to unconditional sequence generation domains, such as image generation, through different encoder and decoder architectures.
We leave this for future work.
For decoder θ , we use the Transformer-InDIGO
(Gu et al., 2019a) architecture, which builds on
Transformer with relative position representations (Shaw et al., 2018) to allow sequence generation through insertion operations.
Note that “encoder” and “decoder” in this section refer to the two networks φ and θ in Algorithm 1, respectively, instead of Transformer’s encoder and decoder.
Also, we obtain orderings through the output of encoder instead of through Searched Adaptive Order (SAO).
To our best effort, we were unable to obtain the ofﬁcial implementation of TransformerInDIGO, so we reimplemented the algorithm based on the paper’s descriptions.
As a side note, rather than generating y autoregressively in 2 n steps as done in Transformer-InDIGO, it is possible to use a non-autoregressive decoder instead and improve the decoding speed.
There have recently been many works on non-autoregressive conditional sequence generation (Gu et al., 2018; 2019b; Ma et al., 2019; Bao et al., 2019).
To train a non-autoregressive decoder Transformer, we can incorporate the ordering information generated by our encoder network into the decoder
Transformer’s encoder latent output.
We leave this for future work.
For encoder φ , we adopt the Transformer (Vaswani et al., 2017) architecture.
Note that our encoder generates latents based on the entire ground truth target sequence y .
Therefore, it does not need to mask out subsequent positions during attention.
We also experiment with different position embedding schemes (see Section 7) and ﬁnd that Transformer-XL’s
(Dai et al., 2019) relative positional encoding performs the best, so we replace the sinusoid encoding in the original Transformer.
Tasks.
We evaluate our approach on challenging sequence generation tasks: natural language to code generation (NL2Code) (Ling et al., 2016), image captioning, text summarization, and machine translation.
For NL2Code, we[-use Django (Oda et al., 2015). For image captioning, we use COCO 
 (Lin et al., 2015).-]
Baselines.
We compare our approach with several
Preprocessing.
For Django, we adopt the same preprocessing steps as described in (Gu et al., 2019a), and we use all unique words as the vocabulary.
For MS-COCO, we ﬁnd that the baseline in
Gu et al.
(2019a) is much lower than commonly used in the vision and language community.
There- fore, instead of using Resnet-18, we use the pretrained Faster-RCNN checkpoint using a ResNetFPN backbone provided by TorchVision to extract 512-dimensional feature vectors for each object detection.
To make our model spatially-aware, we also concatenate the bounding box coordinates for every detection before feeding into our Transformers’ encoder.
For Gigaword and WMT, we learn 32k byte-pair encoding (BPE, Sennrich et al.
on tokenized data.
[Figure]
[Figure]
InDIGO
[Figure]
Ours -
[Table]
Training.
For our decoder, we set d model = 512 , d hidden = 2048 , 6 layers for both Transformer’s encoder and decoder [Table] and 8 attention heads.
This is the same model conﬁguration as TransformerBase (Vaswani et al., 2017) and as described in Gu et al. (2019a).
Our encoder also uses the same conﬁguration.
[-For all experiments, we apply dropout = 0.
-]
For our model trained with Variational Order Inference , we sample K = 4 latents for each training sample.
An ablation on the choices of K is presented in Section 7.
For WMT, many previous works on nonsequential orderings (Stern et al., 2019) and nonautoregressive sequence generation (Gu et al., 2019b) have found sequence-level knowledge distillation (Kim & Rush, 2016) helpful.
Therefore, we ﬁrst train the L2R model on the original WMT corpus, then create a new training corpus using beam search.
We ﬁnd that this improves the BLEU of VOI model by about 2.
Even though the training set changed, the orderings learned by VOI are very similar to the ones trained on the original corpus.
More detailed training processes are described in Appendix C.
During training, our encoder outputs the latent ordering through one single forward pass, and our decoder can predict all tokens with their positions given by the latent ordering in one single forward pass.
If we let N denote the batch size, l denote the length of each target sequence, and d denote the size of hidden vector, then one single forward pass of our model has computation complexity
O ( NKdl 2 ) , while Transformer-InDIGO trained with SAO has
complexity O
( Ndl 3 ) .
Since K (cid:28) l in general, our algorithm has better theoretical computational complexity during training.
During evaluation, we only keep the decoder to iteratively generate the next position and token, which is as efﬁcient as any standard ﬁxed-order autoregressive models.
We also empirically compare VOI’s runtime with that of SAO and ﬁxed-order baselines (e.g. L2R).
We implement SAO as described in Gu et al. (2019a).
We test the runtime on a single GPU in order to accurately measure the number of ops required.
For training speed per iteration, we use a batch size of 8.
For ordering search time, we use a batch size of 1 to avoid padding tokens in the input for accurate measure.
We observe that VOI is signiﬁcantly faster than SAO, which searches orderings sequentially.
In practice, as we distribute VOI across more GPUs, the K factor in the runtime is effectively divided by the number of GPUs used (if we ignore the parallelization overhead), so we can achieve further speedups.
Results.
We compare VOI against predeﬁned orderings along with Transformer-InDIGO trained with SAO in Table 1.
The metrics we used include BLEU-4 (Papineni et al., 2002), Meteor (Denkowski & Lavie, 2014), Rouge (Lin, 2004), CIDEr (Vedantam et al., 2015), and TER (Snover et al., 2006).
The ”accuracy” reported for Django is deﬁned as the percentage of perfect matches in code generation.
Our results illustrate consistently better performance across ﬁxed orderings.
Most notably, CIDEr for MS-COCO, BLEU for Django, and Rouge-1 for Gigaword reveal the largest improvements in performance.
O RDER A NALYSIS
In this section, we analyze the generation orders learned by Variational Order Inference on a macro level by comparing the similarity of our learned orders with predeﬁned orders deﬁned in Section 5, and on a micro level, by inspecting when the model generates certain types of tokens. the same set of predeﬁned orders (dashed lines). In the right plot, with identical setup, we measure Normalized
Levenshtein Distance .
We observe that Variational Order Inference favors left-to-right decoding above the other predeﬁned orders—this corresponds to the blue lines.
However, with a max Order
Rank Correlation of 0.6, it appears left-to-right is not a perfect explanation. The comparably high Order Rank Correlation of 0.3 with rare-tokens-ﬁrst order suggests a complex strategy. 6. U NDERSTANDING T HE M ODEL G LOBALLY
We ﬁnd that prior work (Gu et al., 2019a; Welleck et al., 2019a; Gu et al., 2018) tends to study autoregressive orders by evaluating performance on validation sets, and by visualizing the model’s generation steps.
We provide similar visualizations in Appendix F.3.
However, this does not merit a quantitative understanding of the strategy that was learned.
We address this limitation by introducing methodology to quantitatively study decoding strategies learned by non-monotonic autoregressive models.
We introduce Normalized Levenshtein Distance and Order Rank Correlation , to measure similarity between decoding strategies.
Given two generation orders w , z ∈ S n of the same sequence y , where n is the length of y , we deﬁne the Normalized Levenshtein Distance .
[Equation]
The function lev ( w , z ) is the Levenshtein distance, and z 1: removes the ﬁrst element of z .
This metric has the property that a distance of 0 implies that two orders w and z are the same, while a distance of 1 implies that the same tokens appear in distant locations in w and z .
Our second metric
Order Rank Correlation , is the Spearman’s rank correlation coefﬁcient between w and z .
[Equation]
A correlation of 1 implies that w and z are the same; a correlation of − 1 implies that w and z are reversed; and a correlation of 0 implies that w and z are not correlated.
In Figure 3, we apply these metrics to analyze our models learnt through Variational Order Inference .
particular nouns verbs
Discussion.
The experiment in Figure 3 conﬁrms our model’s behavior is not well explained by predeﬁned orders.
Interestingly, as the generated sequences increase in length, the Normalized Levenshtein Distance decreases, reaching a ﬁnal value of 0.57, indicating that approximately half of the tokens are already arranged according to a left-to-right generation order.
However, the Order
Rank Correlation barely increases, so we can infer that while individual tokens are close to their left-to-right generation index, their relative ordering is not preserved.
Our hypothesis is that certain phrases are generated from left-to-right, but their arrangement follows a best-ﬁrst strategy.
6. U NDERSTANDING T HE M ODEL L OCALLY
To complement the study of our model at a global level, we perform a similar study on the micro token level.
Our hope is that a per-token metric can help us understand if and when our Variational
Order Inference is adaptively choosing between left-to-right and rare-ﬁrst order.
We also hope to evaluate our hypothesis that Variational Order Inference is following a best-ﬁrst strategy.
Discussion.
The experiment in Figure 4 demonstrates that Variational Order Inference prefers decoding descriptive tokens ﬁrst—such as nouns, numerals, adverbs, verbs, and adjectives.
In addition, the unknown part of speech is typically decoded ﬁrst, and we ﬁnd this typically corresponds to special tokens such as proper names.
Our model appears to capture the salient content ﬁrst, which is illustrated by nouns ranking second in the generation order statistics.
For image captioning, nouns typically correspond to focal objects, which suggests our model has an object-detection phase.
Evidence of this phase supports our previous hypothesis that a best-ﬁrst strategy is learned.
6. U NDERSTANDING T HE M ODEL V IA P ERTURBATIONS
[Table]
In this section, we study the question: to what extent is the generation order learned by Variational Order Inference dependent on the content of the conditioning variable x ?
This question is important because simply knowing that our model has learned a best-ﬁrst does not illuminate whether that strategy depends only on the target tokens y being generated, or if it also depends on the content of x .
An adaptive generation order should depend on both.
Discussion.
In this experiment, we ﬁrst obtain a sequence y generated by our VOI given the source image x .
We then freeze y , which allows the model to infer a new generation order for y when different features of x are removed.
Figure 6.3 shows that for a particular case, removing a single region-feature (feature number 0, which corresponds to the bus) from x changes the model-predicted generation order by as much as 0.7 Normalized Levenshtein Distance .
These results conﬁrm that our model appears to learn an adaptive strategy, which depends on both the tokens y being generated and the content of the conditioning variable x , which is an image in this experiment.
A BLATION S TUDIES
[Table]
[Table]
In Section 5, we introduced the speciﬁc encoder and decoder architectures we use for conditional sequence generation tasks.
In this section, we present ablation studies to support the architecture design of our encoder and modeling q φ with GumbelMatching distribution.
We consider 4 different positional encoding schemes for the encoder Transformer φ : the sinusoid encoding in the original Transformer (Vaswani et al., 2017), the sinusoid encoding with positional attention module (Gu et al., 2018), the relative positional encoding in Shaw et al.
(2018), and the relative positional encoding proposed in Transformer-XL (Dai et al., 2019).
Besides modeling q φ ( ·| x , y ) as Gumbel-Matching distribution and using Bethe permanent to approximate its denominator, we also consider modeling using Plackett-Luce distribution (Plackett, 1975; Luce, 1959) and sample using techniques recently proposed in Grover et al.
Plackett-Luce distribution has tractable density, so we can compute the exact q φ efﬁciently without using approximation techniques.
To analyze the encoder’s ability to learn autoregressive orderings, we ﬁrst train a decoder with
Common-First order on one batch of MS-COCO until it perfectly generates each sentence.
We then ﬁx the decoder and initialize an encoder.
We train the encoder for 15k gradient steps using the procedure in Algorithm 1 to recover the ground truth Common-First order, and we report the ﬁnal
Normalized Levenshtein Distance against the ground truth in Table 2.
We observe that modeling q φ with Gumbel-Matching distribution signiﬁcantly outperforms modeling with Plackett-Luce, despite the former requiring denominator approximation.
We also observe that under Gumbel-Matching modeling distribution, the relative position encoding in Transformer-XL signiﬁcantly outperforms other encoding schemes.
Thus we combine these two techniques in our architecture design.
[Table]
In addition, we analyze how the choice of K , the number of latents per training sample, affects model performance.
We use the same setting as above and apply Transformer-XL relative position encoding, and we report the results in Table 3.
We observe that the encoder more accurately ﬁts to the ground truth order as K increases, until a value of around 10.
Since a very large K can slow the model down while only bringing marginal improvement, a choice of K from 4 to 10 is sufﬁcient.
C ONCLUSION
We propose, to our best knowledge, the ﬁrst unsupervised learner that learns high-quality autoregressive orders through fully-parallelizable end-to-end training without domain-speciﬁc tuning.
We propose a procedure named Variational Order Inference that uses the Variational Lower Bound with the space of autoregressive orderings as latent.
Building on techniques in combinatorical optimization, we develop a practical policy gradient algorithm to optimize the encoder of the variational objective, and we propose an encoder architecture that conditions on training examples to output autoregressive orders.
Empirical results demonstrate that our model is capable of discovering autoregressive orders that are competitive with or even better than ﬁxed and predeﬁned orders.
In addition, the global and local analysis of the orderings learned through Variational Order Inference suggest that they resemble a type of best-ﬁrst generation order, characterized by prioritizing the generation of descriptive tokens and deprioritizing the generation of modiﬁer tokens.
A PPENDIX A G UMBEL -M ATCHING D ISTRIBUTION AND ITS S AMPLING
In Section 4, We model the distribution of q φ ( ·| y , x ) as a Gumbel-Matching distribution G .
M .
( X ) over P n × n , where X = φ
( y , x ) ∈ R n × n is the latent output.
To obtain samples in P n × n from the Gumbel-Matching distribution, Mena et al.
(2018) relaxes
P n × n to B n × n by deﬁning the Gumbel-Sinkhorn distribution G . S .
( X, τ ) : τ > 0 over B n × n .
Here we reproduce the following deﬁnitions and theorems with similar notations from Sinkhorn (1964) and Mena et al.
Deﬁnition A.1.
Let X ∈ R n × n and A ∈ R + n × n .
The Sinkhorn Operator S is deﬁned as
[Equation]
Here, (cid:11) is the element-wise division between two matrices, and T r and T c are row and column normalizations of a non-negative matrix, respectively.
Therefore, iteratively applying T is equivalent to iteratively normalizing a non-negative matrix by column and row.
Theorem A.2.
(Sinkhorn, 1964)
The range of S is B n × n .
Theorem A.3.
(Mena et al., 2018) Let X ∈ R n × n , τ > 0 .
The Gumbel-Sinkhorn distribution
G .
S .
( X, τ ) is deﬁned as follows:
[Equation]
where (cid:15) is a matrix of i.i.d.
standard Gumbel noise.
Moreover, G . S .
( X, τ ) converges almost surely to G .
M .
( X ) as τ → 0 + .
To approximately sample from G .
M .
( X ) , we ﬁrst sample from G .
S .
( X, τ ) .
Even though theoretically, T needs to be applied inﬁnite number of times to obtain a matrix in B n × n , Mena et al.
(2018) reports that 20 iterations of T are enough in practice.
We ﬁnd that in our experiments, 20 iterations are not enough to obtain a matrix in B n × n , but 100 − 200 iterations are enough.
After we obtain the matrix in B n × n , we apply Hungarian algorithm (Munkres, 1957) to obtain P ∈ G .
M .
( X ) .
Finally, we need to calculate the entropy term H q φ in L φ in Equation 3.
This can be approximated using the technique in Appendix B.3 of Mena et al.
B M ATRIX P ERMANENT AND ITS A PPROXIMATION WITH B ETHE P ERMANENT
In this section, we present details about matrix permanent and bethe permanent, which we use as an approximation to the denominator of q φ ( ·| y , x ) .
Deﬁnition B.1.
Let A ∈ R n × n .
The permanent of A is deﬁned as follows:
[Equation]
Theorem B.2.
The denominator of q φ ( ·| y , x ) equals perm (exp( X )) .
Proof.
(cid:88)
[Equation]
n
[Equation]
(cid:88) σ ∈ S n exp( n (cid:88) i =
X i,σ ( i ) )
= (cid:88) σ ∈ S n n (cid:89)
i =
(exp( X ))
i,σ ( i ) = perm (exp( X ))
Deﬁnition B.3.
(Vontobel, 2010; Anari & Rezaei, 2019) Let A ∈ R + n × n .
The bethe permanent of
A is deﬁned as follows:
[Equation]
Theorem B.4.
(Anari & Rezaei, 2019) Let A ∈ R + n × n .
Then, √ 2 − n perm ( A ) ≤ perm B ( A ) ≤ perm ( A ) .
The γ in Deﬁnition B.3 can be calculated using the message passing algorithm in Lemma 29 of
Vontobel (2010).
An efﬁcient implementation has recently been introduced in Appendix C of Mena et al.
Therefore, we can use perm B (exp ( X )) to approximate the denominator of q φ ( ·| y , x ) , and we can then use policy gradient to compute ∇ φ L ( φ ) in Equation (3).
C D ETAILED T RAINING P ROCESS AND H YPERPARAMETER S ETTINGS
{+For all experiments, we apply dropout = 0.
 (Srivastava et al., 2014)+} and label smoothing = 0.1.
We apply Adam Optimizer (Kingma & Ba, 2015) with β 1 = 0 .
99 , β 2 = 0 .
999 for MS-COCO, and β 1 = 0 .
99 , β 2 = 0 .
98 for all other tasks.
For baseline experiments, we use a batch size of 64 for
Django and MS-COCO, and 128 for Gigaword and WMT. We decrease the learning rate linearly from 1e-4 to zero. We train the baseline until the performance plateaus. For our VOI model, we train on Django for a total of 350 epochs (120k gradient steps), MS-COCO for 20 epochs (350k gradient steps), Gigaword for 16 epochs (1M gradient steps), and WMT16 RoEn for 120 epochs (1.3M gradient steps).
We use a batch size of 36 for MS-COCO and Django,for Gigaword, and 54 for WMT.
We sample K = 4 latents per training sample for the ﬁrst three datasets, and K = 3 for WMT. Due to constraints in computational resource, we were unable to scale WMT to larger batch size and larger K . We also did not experiment with larger batch size for COCO, Django, and Gigaword.
We leave the investigations of larger batch sizes and larger K for future work.
We set the initial decoder learning rate to be 5e-5 and the encoder learning rate to be 5e-6.
We train the VOI encoder and decoder with shared embedding for about the ﬁrst 15-20% of steps (i.e.
4 epochs for COCO, 50 epochs for Django, 3 epochs for Gigaword, and 20 epochs for WMT).
We then separate the embeddings for the rest of the training steps.
When the embedding is shared, we set the entropy coefﬁcient β = 0 .
3 for all tasks.
After we separate the embeddings, for MS-COCO, we anneal β with a log-linear schedule from 0. to 0.03.
We decrease the learning rate to (3e-5, 3e-6) for the decoder and the encoder respectively after epoch 13, when the encoder starts to sample very similar permutations for a single training data.
We observe that training either VOI or the ﬁxed ordering models for too long leads to overﬁtting.
Finetuning VOI with the encoder ﬁxed does not help and causes the performance to slightly drop.
For Django, we set the learning rates to be (3e-5, 3e-6). We log-anneal β to 0.03 for the ﬁrst 90% of steps and then anneal β to 0.003 for the rest of the steps.
We ﬁnd that the latter allows the encoder to commit to a single ordering on sequences of longer length and slightly improves performance.
We ﬁnally ﬁx the encoder and ﬁnetune the decoder for 50 epochs with a larger batch size of 64 and learning rate linearly annealing to zero.
This ﬁnetuning step improves the BLEU score by about 0.6.
For Gigaword and WMT, we add a cosine alignment loss between the decoder and the encoder’s embedding matrices to the loss of the encoder. We set the cosine alignment loss coefﬁcient to be
100.0 for Gigaword and 10.0 for WMT. Intuitively, since the Gigaword and WMT vocabularies are much larger than those of MS-COCO and Django, and they contain many rare words, this loss allows the encoder to leverage the semantic information recently-learnt from the decoder to better discover autoregressive orderings.
For Gigaword, we anneal β log-linearly from 0.3 to 0.03 in 8 epochs (500k gradient steps).
We then ﬁx the encoder and ﬁne-tune the decoder with a batch size of 128 for 5 epochs with learning rate linearly decreasing from 7e-5 to 0.
We observe that, compared to COCO and Django, this ﬁnetuning step signiﬁcantly improves VOI’s performance and raises the ROUGE score by around 1.5 to 2.0. For WMT, we anneal β log-linearly from 0.3 to 7e-4 in 80 epochs (900k gradient steps).
We decrease the learning rates from (5e-5, 5e-6) to (3e-5, 3e-6) at epoch 40 when the encoder starts sampling very similar permutations. We then ﬁx the encoder and ﬁnetune the decoder with a batch size of 128 for 20 epochs with learning rate linearly decreasing from 3e-5 to 0.
We observe that this ﬁnetuning step also signiﬁcantly beneﬁts VOI’s performance and improves the BLEU score by around 1.5 points. Due to resource constraints, we did not tune our hyperparameters and training schedules very carefully, and we leave the discovery of better training schemes for future work.
D E XAMPLE A RCHITECTURE FOR C ONDITIONAL S EQUENCE G ENERATION
In Section 5, we introduced the speciﬁc encoder and decoder architectures used for the conditional sequence generation tasks in our paper.
To further illustrate the architecture of Variational Order
Inference , we present a diagram of the architecture instantiated for COCO 2017.
E P ARTS O F S PEECH M APPINGS
The parts of speech used in our Order Analysis section correspond to the NLTK Universal Tagset.
In the below table, we provide mappings for the tag identiﬁers used in our main paper.
More information about the speciﬁc NLTK tags can be found at the following url: http://www.nltk.
org/book/ch05.html .
F V ISUALIZATIONS OF S EQUENCE G ENERATION F. COCO
We visualize the generation order inferred by Variational Order Inference for COCO.
Sequences are generated using beam search over both tokens and their insertion positions, using a beam size of 3.
Bounding boxes that correspond to region-features calculated using bottom-up attention are superimposed on the image, with an opacity value proportional to the magnitude of their softmax attention value in the ﬁnal cross-attention layer in the language model.
Tag
Meaning
English Examples
ADJ adjective new, good, high, special, big, local
ADP adposition on, of, at, with, by, into, under
ADV adverb really, already, still, early, now
CONJ conjunction and, or, but, if, while, although
DET determiner, article the, a, some, most, every, no, which
NOUN noun year, home, costs, time, Africa
NUM numeral twenty-four, fourth, 1991, 14:
PRT particle at, on, out, over per, that, up, with
PRON pronoun he, their, her, its, my, I, us
VERB verb is, say, told, given, playing, would .
punctuation marks .
X other ersatz, esprit, dunno, gr8, univeristy
Image ID:two people standing in the snow on snowboards .
two people standing in the snow on snowboards . two people standing in the snow on snowboards .
two people standing in the snow on snowboards . two people standing in the snow on snowboards .
two people standing in the snow on snowboards . two people standing in the snow on snowboards .
two people standing in the snow on snowboards . two people standing in the snow on snowboards .
Decoded Text
Image ID: [Figure] [Figure] slope .
[Figure] riding [Figure] slope .
[Figure]
riding skis down a snow covered slope . a woman riding skis down a snow covered slope .
a woman riding skis down a snow covered slope . a woman riding skis down a snow covered slope .
a woman riding skis down a snow covered slope . a woman riding skis down a snow covered slope .
a woman riding skis down a snow covered slope .
Decoded Text
Image ID:a kitchen with white cabinets and white appliances .
a kitchen with white cabinets and white appliances . a kitchen with white cabinets and white appliances .
a kitchen with white cabinets and white appliances . a kitchen with white cabinets and white appliances .
a kitchen with white cabinets and white appliances . a kitchen with white cabinets and white appliances .
a kitchen with white cabinets and white appliances . a kitchen with white cabinets and white appliances .
Decoded Text
Image ID:a person standing on a ledge near the water .
a person standing on a ledge near the water . a person standing on a ledge near the water .
a person standing on a ledge near the water . a person standing on a ledge near the water .
a person standing on a ledge near the water . a person standing on a ledge near the water .
a person standing on a ledge near the water . a person standing on a ledge near the water .
a person standing on a ledge near the water .
Decoded Text
Image ID:a woman holding a cell phone in her hand .
a woman holding a cell phone in her hand . a woman holding a cell phone in her hand .
a woman holding a cell phone in her hand . a woman holding a cell phone in her hand .
a woman holding a cell phone in her hand . a woman holding a cell phone in her hand .
a woman holding a cell phone in her hand . a woman holding a cell phone in her hand .
a woman holding a cell phone in her hand .
Decoded Text
Image ID:a desktop computer with a laptop on a desk .
a desktop computer with a laptop on a desk . a desktop computer with a laptop on a desk .
a desktop computer with a laptop on a desk . a desktop computer with a laptop on a desk .
a desktop computer with a laptop on a desk . a desktop computer with a laptop on a desk .
a desktop computer with a laptop on a desk . a desktop computer with a laptop on a desk .
a desktop computer with a laptop on a desk .
Decoded Text
Image ID:a bedroom with a bed and a table .
a bedroom with a bed and a table . a bedroom with a bed and a table . a bedroom with a bed and a table . a bedroom with a bed and a table .
a bedroom with a bed and a table .
a bedroom with a bed and a table . a bedroom with a bed and a table . a bedroom with a bed and a table .
Decoded Text
[Figure] young [Figure] slope a young
[Figure]
riding skis [Figure] slope a [Figure]
slope a young [Figure] riding skis [Figure]
slope a young [Figure] riding skis [Figure]
slope
[Figure]
young person [Figure] slope [Figure]
slope Decoded
[Figure]
Image ID:a kitchen with white appliances and wood cabinets .
a kitchen with white appliances and wood cabinets . a kitchen with white appliances and wood cabinets .
a kitchen with white appliances and wood cabinets . a kitchen with white appliances and wood cabinets .
a kitchen with white appliances and wood cabinets . a kitchen with white appliances and wood cabinets .
a kitchen with white appliances and wood cabinets . a kitchen with white appliances and wood cabinets .
Decoded Text
Image ID:a couple of people that are sitting on a bench a couple of people that are sitting on a bench a couple of people that are sitting on a bench a couple of people that are sitting on a bench a couple of people that are sitting on a bench a couple of people that are sitting on a bench a couple of people that are sitting on a bench a couple of people that are sitting on a bench a couple of people that are sitting on a bench a couple of people that are sitting on a bench
Decoded Text
Image ID:a woman in a crowd is using her cell phone .
a woman in a crowd is using her cell phone . a woman in a crowd is using her cell phone . a woman in a crowd is using her cell phone .
a woman in a crowd is using her cell phone . a woman in a crowd is using her cell phone .
a woman in a crowd is using her cell phone . a woman in a crowd is using her cell phone .
a woman in a crowd is using her cell phone . a woman in a crowd is using her cell phone .
a woman in a crowd is using her cell phone .
Decoded Text
Image ID:a desktop computer sitting on top of a desk .
a desktop computer sitting on top of a desk . a desktop computer sitting on top of a desk .
a desktop computer sitting on top of a desk . a desktop computer sitting on top of a desk .
a desktop computer sitting on top of a desk . a desktop computer sitting on top of a desk .
a desktop computer sitting on top of a desk . a desktop computer sitting on top of a desk .
a desktop computer sitting on top of a desk .
Decoded Text
Image ID:a bed and a table in a room .
a bed and a table in a room . a bed and a table in a room .
a bed and a table in a room . a bed and a table in a room .
a bed and a table in a room . a bed and a table in a room .
a bed and a table in a room . a bed and a table in a room .
Decoded Text
[Figure]
Decoded [Figure] Image ID:a kitchen area with a stove , refrigerator and dishwasher .
a kitchen area with a stove , refrigerator and dishwasher . a kitchen area with a stove , refrigerator and dishwasher .
a kitchen area with a stove , refrigerator and dishwasher . a kitchen area with a stove , refrigerator and dishwasher .
a kitchen area with a stove , refrigerator and dishwasher . a kitchen area with a stove , refrigerator and dishwasher .
a kitchen area with a stove , refrigerator and dishwasher . a kitchen area with a stove , refrigerator and dishwasher .
a kitchen area with a stove , refrigerator and dishwasher . a kitchen area with a stove , refrigerator and dishwasher .
Decoded Text
Image ID:a group of people sitting on top of a pier .
a group of people sitting on top of a pier .
a group of people sitting on top of a pier . a group of people sitting on top of a pier . a group of people sitting on top of a pier .
a group of people sitting on top of a pier . a group of people sitting on top of a pier .
a group of people sitting on top of a pier . a group of people sitting on top of a pier .
a group of people sitting on top of a pier . a group of people sitting on top of a pier .
Decoded Text
Image ID:a woman holding a cell phone while talking on a crowd .
a woman holding a cell phone while talking on a crowd . a woman holding a cell phone while talking on a crowd .
a woman holding a cell phone while talking on a crowd . a woman holding a cell phone while talking on a crowd .
a woman holding a cell phone while talking on a crowd . a woman holding a cell phone while talking on a crowd .
a woman holding a cell phone while talking on a crowd . a woman holding a cell phone while talking on a crowd .
a woman holding a cell phone while talking on a crowd . a woman holding a cell phone while talking on a crowd .
a woman holding a cell phone while talking on a crowd .
Decoded Text
Image ID:a computer monitor sitting on top of a wooden desk .
a computer monitor sitting on top of a wooden desk .
a computer monitor sitting on top of a wooden desk . a computer monitor sitting on top of a wooden desk . a computer monitor sitting on top of a wooden desk .
a computer monitor sitting on top of a wooden desk . a computer monitor sitting on top of a wooden desk . a computer monitor sitting on top of a wooden desk . a computer monitor sitting on top of a wooden desk .
a computer monitor sitting on top of a wooden desk . a computer monitor sitting on top of a wooden desk .
Decoded Text
Image ID:a bed sitting in a room next to a window . a bed sitting in a room next to a window . a bed sitting in a room next to a window . a bed sitting in a room next to a window .
a bed sitting in a room next to a window . a bed sitting in a room next to a window . a bed sitting in a room next to a window . a bed sitting in a room next to a window .
a bed sitting in a room next to a window . a bed sitting in a room next to a window . a bed sitting in a room next to a window .
Decoded Text
[Figure]
ID:
[Figure]
riding skis
[Figure] a [Figure] covered slope .
a man riding skis down a [Figure] covered slope .
[Figure] skis [Figure] a
[Figure] slope . [Figure] skis
[Figure] a [Figure] slope
[Figure] slope . [Figure] slope .
[Figure] slope . [Figure] slope .
[Figure] slope . [Figure] slope .
Decoded Text
Image ID:a kitchen with a stove and a refrigerator .
a kitchen with a stove and a refrigerator . a kitchen with a stove and a refrigerator .
a kitchen with a stove and a refrigerator . a kitchen with a stove and a refrigerator .
a kitchen with a stove and a refrigerator . a kitchen with a stove and a refrigerator .
a kitchen with a stove and a refrigerator . a kitchen with a stove and a refrigerator .
Decoded Text
Image ID:a woman sitting on a dock looking at the water .
a woman sitting on a dock looking at the water . a woman sitting on a dock looking at the water .
a woman sitting on a dock looking at the water . a woman sitting on a dock looking at the water .
a woman sitting on a dock looking at the water . a woman sitting on a dock looking at the water .
a woman sitting on a dock looking at the water . a woman sitting on a dock looking at the water .
a woman sitting on a dock looking at the water . a woman sitting on a dock looking at the water .
Decoded Text
Image ID:the woman is talking on her cell phone .
the woman is talking on her cell phone . the woman is talking on her cell phone .
the woman is talking on her cell phone . the woman is talking on her cell phone .
the woman is talking on her cell phone . the woman is talking on her cell phone .
the woman is talking on her cell phone . the woman is talking on her cell phone .
Decoded Text
Image ID:a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse a laptop on a desk with a keyboard and a mouse
Decoded Text
Image ID:a room with a bed , chairs , and tables .
a room with a bed , chairs , and tables . a room with a bed , chairs , and tables .
a room with a bed , chairs , and tables . a room with a bed , chairs , and tables . a room with a bed , chairs , and tables .
a room with a bed , chairs , and tables . a room with a bed , chairs , and tables .
a room with a bed , chairs , and tables . a room with a bed , chairs , and tables .
a room with a bed , chairs , and tables .
Decoded Text
F. D JANGO
We visualize the latent generation order inferred by Variational Order Inference for Django.
Sequences are generated using a beam search over both the tokens and their insertion positions, using a beam size of 3.
Text on which the model is conditioned is provided on the left for each example.
[Figure]
raise AttributeError ( '_STR:0_' % self .
name )
[Figure]
AttributeError ( '_STR:0_' % self .
name ) raise AttributeError ( '_STR:0_' % self .
name ) raise AttributeError ( '_STR:0_' % self .
name ) raise AttributeError ( '_STR:0_' % self .
name )
[Figure]
AttributeError ( '_STR:0_' % self .
name )
[Figure]
AttributeError ( '_STR:0_' % self .
name )
[Figure]
AttributeError ( '_STR:0_' % self .
name )
[Figure]
( '_STR:0_' % self .
name )
[Figure]
Generation order inferred by Ours-VOI for a pseudocode sample from the Django natural language to code test set with the sample id 154 .
F. G IGAWORD
We visualize the latent generation order inferred by Variational Order Inference for Gigaword.
Sequences are generated using a beam search over both the tokens and their insertion positions, using a beam size of 3.
Text on which the model is conditioned is provided on the left for each example.
for every i and arg in enumerated iterable args ,
Conditioned Text for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass
Decoded Text
[Figure]
% self [Figure] [Figure] [Figure]
[Figure] [Figure]
( '_STR
[Figure]
_' % self .
[Figure]
( '_STR:0_' % self .
[Figure]
( '_STR:0_'
[Figure]
( '_STR:0_
[Figure]
( '_STR:0_' % self .
[Figure]
[Figure]
for every i and arg in enumerated iterable args , Conditioned Text for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass for i , arg in enumerate ( args ) :pass Decoded Text raise
[Figure]
argument
[Figure]
_STR:0_ , formated with self. [Figure] [ self . [Figure]
[Figure] raise AttributeError ( '_STR:0_' % name self .
__class__ .
__name__ ) raise AttributeError ( '_STR:0_' % name self .
__class__ .
__name__ ) raise AttributeError ( '_STR:0_' % name self .
__class__ .
__name__ ) raise AttributeError ( '_STR:0_' % name self .
__class__ .
__name__ ) raise AttributeError ( '_STR:0_' % name self .
__class__ .
__name__ ) raise AttributeError ( '_STR:0_' % name self .
__class__ .
__name__ ) raise AttributeError ( '_STR:0_' % name self .
__class__ .
[Figure]
for every i and arg in enumerated iterable
[Figure]
[Figure]
for , arg i in args enumerate ( ) :pass for , arg i in args enumerate ( ) :pass for , arg i in args enumerate ( ) :pass for , arg i in args enumerate ( ) :pass for ,
arg i in args enumerate ( ) :pass for , arg i in args enumerate ( ) :pass for ,
arg i in args enumerate ( ) :pass for , arg i in args enumerate ( ) :pass for ,
arg i in args enumerate ( ) :pass for , arg i in args enumerate ( ) :pass
Decoded Text
[Figure] [Figure]
for every i and arg in enumerated iterable args ,
Conditioned Text for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass for i , in arg enumerate ( args ) :pass
Decoded Text prime minister of antigua and bar@@ bud@@ a baldwin spencer left here monday for hong kong , winding up his four @-@ day visit to shanghai .
Conditioned Text antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong Decoded
[Figure] the sri lankan navy has taken into custody two indian fishing trawlers which were poaching sri lanka &apos; s northwestern coast and arrested # # indian fishermen on board , a local newspaper reported on wednesday .
Conditioned Text sri lankan navy arrests two indian trawlers sri lankan navy arrests two indian trawlers sri lankan navy arrests two indian trawlers sri lankan navy arrests two indian trawlers sri lankan navy arrests two indian trawlers sri lankan navy arrests two indian trawlers sri lankan navy arrests two indian trawlers
Decoded Text prime minister of antigua and bar@@ bud@@ a baldwin spencer left here monday for hong kong , winding up his four @-@ day visit to shanghai .
Conditioned Text antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong antigua pm leaves shanghai for hong kong
Decoded Text the sri lankan navy has taken into custody two indian fishing trawlers which were poaching sri lanka &apos; s northwestern coast and arrested # # indian fishermen on board , a local newspaper reported on wednesday .
Conditioned Text sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen
Decoded Text prime minister of antigua and bar@@ bud@@ a baldwin spencer left here monday for hong kong , winding up his four @-@ day visit to shanghai .
Conditioned Text antigua pm ends visit to shanghai antigua pm ends visit to shanghai antigua pm ends visit to shanghai antigua pm ends visit to shanghai antigua pm ends visit to shanghai antigua pm ends visit to shanghai
Decoded Text the sri lankan navy has taken into custody two indian fishing trawlers which were poaching sri lanka &apos; s northwestern coast and arrested # # indian fishermen on board , a local newspaper reported on wednesday .
Conditioned Text sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen
Decoded Text prime minister of antigua and bar@@ bud@@ a baldwin spencer left here monday for hong kong , winding up his four @-@ day visit to shanghai .
Conditioned Text antigua ends prime minister visit to china antigua ends prime minister visit to china antigua ends prime minister visit to china antigua ends prime minister visit to china antigua ends prime minister visit to china antigua ends prime minister visit to china antigua ends prime minister visit to china
Decoded Text the sri lankan navy has taken into custody two indian fishing trawlers which were poaching sri lanka &apos; s northwestern coast and arrested # # indian fishermen on board , a local newspaper reported on wednesday .
Conditioned Text sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen sri lankan navy arrests # # indian fishermen
Decoded Text