Original: P OSTERIOR A TTENTION M ODELS FOR S EQUENCE TO S EQUENCE L EARNING
Revised: P OSTERIOR A TTENTION M ODELS FOR S EQUENCE TO S EQUENCE L EARNING

Original: Anonymous authors
Revised: sunita@iitb.ac.in

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modern neural architectures critically rely on attention for mapping structured inputs to sequences.
Revised: Modern neural architectures critically rely on attention for mapping structured inputs to sequences

Original: In this paper we show that prevalent attention architectures do not adequately model the dependence among the attention and output variables along the length of a predicted sequence.
Revised: In this paper we show that prevalent attention architectures do not adequately model the dependence among the attention and output tokens across a predicted sequence.

Original: We present an alternative architecture called Posterior Attention Models that after a principled factorization of the full joint distribution of the attention and output variables propose two major changes.
Revised: We present an alternative architecture called Posterior Attention Models that after a principled factorization of the full joint distribution of the attention and output variables, proposes two major changes.

Original: First, the position where attention is marginalized is changed from the input to the output.
Revised: First, the position where attention is marginalized is changed from the input to the output.

Original: Second, the attention propagated to the next decoding stage is a posterior attention distribution conditioned on the output.
Revised: Second, the attention propagated to the next decoding stage is a posterior attention distribution conditioned on the output.

Original: Empirically on ﬁve translation and two morphological inﬂection tasks the proposed posterior attention models yield better predictions and alignment accuracy than existing attention models.
Revised: Empirically on ﬁve translation and two morphological inﬂection tasks the proposed posterior attention models yield better BLEU score and alignment accuracy than existing attention models

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Attention is a critical module of modern neural models for sequence to sequence learning as applied to tasks like translation, grammar error correction, morphological inﬂection, and speech to text conversion.
Revised: Attention is a critical module of modern neural models for sequence to sequence learning as applied to tasks like translation, grammar error correction, morphological inﬂection, and speech to text conversion.

Original: Attention speciﬁes what part of the input is relevant for each output.
Revised: Attention speciﬁes what part of the input is relevant for each output.

Original: Many variants of attention have been proposed including soft (Bahdanau et al., 2014; Luong et al., 2015), sparse (Martins & Astudillo, 2016), local (Luong et al., 2015), hard (Xu et al., 2015; Zaremba & Sutskever, 2015), monotonic hard (Yu et al., 2016; Aharoni & Goldberg, 2017), hard non-monotonic Wu et al. ; Shankar et al. (2018), variational Deng et al. (2018), The most prevalent of these is soft attention that computes attention for each output as a multinomial distribution over the input states.
Revised: Many variants of attention have been proposed including soft (Bahdanau et al., 2014; Luong et al., 2015), sparse (Martins & Astudillo, 2016), local (Luong et al., 2015), hard (Xu et al., 2015; Zaremba & Sutskever, 2015), monotonic hard (Yu et al., 2016; Aharoni & Goldberg, 2017), hard non-monotonic (Wu et al., 2018; Shankar et al., 2018), and variational (Deng et al., 2018), The most prevalent of these is soft attention that computes attention for each output as a multinomial distribution over the input states.

Original: The multinomial probabilities serve as weights, and an attention weighted sum of input states serves as relevant context for the output and subsequent attention.
Revised: The multinomial probabilities serve as weights, and an attention weighted sum of input states serves as relevant context for the output and subsequent attention.

Original: Soft attention is end to end differentiable, easy to implement, and hence widely popular.
Revised: Soft attention is end to end differentiable, easy to implement, and hence widely popular.

Original: Hard attention and sparse attentions are difﬁcult to implement and not popularly used.
Revised: Hard attention and sparse attentions are difﬁcult to implement and not popularly used.

Original: In this paper we revisit the statistical soundness of the way soft attention and other variants capture the dependence between attention and output variables, and among multiple attention variables along the length of the sequence.
Revised: In this paper we revisit the statistical soundness of the way soft attention and other variants capture the dependence between attention and output variables, and among multiple attention variables along the length of the sequence.

Original: Our investigation leads to a more principled model that we call the Posterior Attention Model (PAM).
Revised: Our investigation leads to a more principled model that we call the Posterior Attention Model (PAM).

Original: We start with an explicit joint distribution of all output and attention variables in a predicted sequence.
Revised: We start with an explicit joint distribution of all output and attention variables in a predicted sequence.

Original: We then propose a tractable approximation that retains the advantages of forward dependence and token-level decomposition and thus leads to efﬁcient training and inference.
Revised: We then propose a tractable approximation that retains the advantages of forward dependence and token-level decomposition and thus leads to efﬁcient training and inference.

Original: The computations performed at each decode step has two important differences with existing models.
Revised: The computations performed at each decode step has two important differences with existing models.

Original: First, at each decoding step the probability of the output token is a mixture of output probability for each attention.
Revised: First, at each decoding step the probability of the output token is a mixture of output probability for each attention.

Original: In contrast, existing models take a mixture of the input, and compute a single output distribution from this diffused mixed input.
Revised: In contrast, existing models take a mixture of the input, and compute a single output distribution from this diffused mixed input.

Original: We show that our direct coupling of output and attention gives the beneﬁt of hard attention without its computational challenges.
Revised: We show that our direct coupling of output and attention gives the beneﬁt of hard attention without its computational challenges.

Original: Second, we introduce the notion of a posterior attention distribution, that is, the attention distribution conditioned on the current output.
Revised: Second, we introduce the notion of a posterior attention distribution, that is, the attention distribution conditioned on the current output.

Original: We show that it is both statistically sounder and more accurate to condition subsequent attention on the output corrected posterior attention, rather than the output independent prior attention as in existing models.
Revised: We show that it is both statistically sounder and more accurate to condition subsequent attention on the output corrected posterior attention, rather than the output independent prior attention as in existing models.

Original: We evaluate the posterior attention model on ﬁve translation tasks and two morphological inﬂection tasks.
Revised: We evaluate the posterior attention model on ﬁve translation tasks and two morphological inﬂection tasks.

Original: We show that posterior attention provides improved BLEU score, higher alignment accuracy, and better input coverage.
Revised: We show that posterior attention provides improved BLEU score, higher alignment accuracy, and better input coverage.

Original: We also empirically analyze the reasons behind the improved performance of the posterior attention model.
Revised: We also empirically analyze the reasons behind the improved performance of the posterior attention model.

Original: We discover that the entropy of posterior attention is much lower than entropy of soft attention.
Revised: We discover that the entropy of posterior attention is much lower than entropy of soft attention.

Original: This is a signiﬁcant ﬁnding that challenges the current practice of computing attention distribution without considering the output token.
Revised: This is a signiﬁcant ﬁnding that challenges the current practice of computing attention distribution without considering the output token.

Original: The running time overhead of posterior attention is only 40% over existing soft-attention.
Revised: The running time overhead of posterior attention is only 40% over existing soft-attention.

Original:  J OINT D ISTRIBUTION FOR A TTENTION AND O UTPUT V ARIABLES
Revised:  J OINT D ISTRIBUTION FOR A TTENTION AND O UTPUT V ARIABLES

Original: Our goal is to model the conditional distribution
Revised: Our goal is to model the conditional distribution

Original: Pr( y | x ) of an output sequence y =
Revised: Pr( y | x ) of an output sequence y =

Original: y 1 , . .
Revised: y 1 , . .

Original: , y n given an input sequence x = x 1 , . . .
Revised: , y n given an input sequence x = x 1 , . . .

Original: , x m .
Revised: , x m .

Original: Each output y t is a discrete token from a typically large vocabulary V .
Revised: Each output y t is a discrete token from a typically large vocabulary V .

Original: Each x j can be any abstract input.
Revised: Each x j can be any abstract input.

Original: Typically a RNN encodes the input sequence into a sequence of state vectors x 1 , . . .
Revised: Typically a RNN encodes the input sequence into a sequence of state vectors x 1 , . . .

Original: , x m , which we jointly denote as x 1: m .
Revised: , x m , which we jointly denote as x 1: m .

Original: Each y t depends not only on other tokens in the sequence, but on some speciﬁc focused part of the input sequence.
Revised: Each y t depends not only on other tokens in the sequence, but on some speciﬁc focused part of the input sequence.

Original: A hidden variable a t , called the attention variable, denotes which part of x 1: m the output y t depends on.
Revised: A hidden variable a t , called the attention variable, denotes which part of x 1: m the output y t depends on.

Original: We denote the set of all attention as a = a 1 , . . .
Revised: We denote the set of all attention as a = a 1 , . . .

Original: , a n .
Revised: , a n .

Original: During training the input x and output y are observed but the attention a is hidden.
Revised: During training the input x and output y are observed but the attention a is hidden.

Original: Hence, we write Pr( y | x ) as
Revised: Hence, we write Pr( y | x ) as

Original: [Equation]
Revised: [Equation]

Original: The number of variables involved in this summation is daunting, and we need to approximate.
Revised: The number of variables involved in this summation is daunting, and we need to approximate.

Original: We ﬁrst review how existing soft attention-based encoder decoder models handle this challenge.
Revised: We ﬁrst review how existing soft attention-based encoder decoder models handle this challenge.

Original: 2. E XISTING A TTENTION - BASED E NCODER D ECODER M ODEL
Revised: 2. E XISTING A TTENTION - BASED E NCODER D ECODER M ODEL

Original: Existing Encoder-Decoder (ED) networks factorize Pr( y | x 1: m ) by applying chain rule on y variables as (cid:81) nt =1 Pr( y t | x 1: m , y 1 , . . .
Revised: Existing Encoder-Decoder (ED) networks factorize Pr( y | x 1: m ) by applying chain rule on y variables as (cid:81) nt =1 Pr( y t | x 1: m , y 1 , . . .

Original: , y t − 1 ) .
Revised: , y t − 1 ) .

Original: A decoder RNN summarizes the variable length history y 1 , . . .
Revised: A decoder RNN summarizes the variable length history y 1 , . . .

Original: , y t − 1 as a decoder state s t , so that Pr( y | x 1: m ) =
Revised: , y t − 1 as a decoder state s t , so that Pr( y | x 1: m ) =

Original: (cid:81) nt =1 Pr( y t | x 1: m , s t ) .
Revised: (cid:81) nt =1 Pr( y t | x 1: m , s t ) .

Original: The distribution of each attention variable a t is computed as a function of the decoder state and encoder state as:
Revised: The distribution of each attention variable a t is computed as a function of the decoder state and encoder state as:

Original: Pr( a | x 1: m , s t ) ∝ e A θ ( x a , s t ) .
Revised: Pr( a | x 1: m , s t ) ∝ e A θ ( x a , s t ) .

Original: Here A θ ( ., . ) is an end-to-end trained function of input state x a and decoder state s t .
Revised: Here A θ ( ., . ) is an end-to-end trained function of input state x a and decoder state s t .

Original: We will use the short form P t ( a ) for Pr( a t | x 1: m , s t ) .
Revised: We will use the short form P t ( a ) for Pr( a t | x 1: m , s t ) .

Original: Thereafter, an attention weighted sum of the input states (cid:80) a P t ( a ) x a called input context c t is computed.
Revised: Thereafter, an attention weighted sum of the input states (cid:80) a P t ( a ) x a called input context c t is computed.

Original: The distribution of y t is computed from c t (capturing attention) and s t capturing previous y as:
Revised: The distribution of y t is computed from c t (capturing attention) and s t capturing previous y as:

Original: [Equation]
Revised: [Equation]

Original: Next, c t is fed to the decoder RNN along with y t for computing the next state: s t +1 =
Revised: Next, c t is fed to the decoder RNN along with y t for computing the next state: s t +1 =

Original: RNN ( s t , c t , y t ) .
Revised: RNN ( s t , c t , y t ) .

Original: Figure 1[left] summarizes the compute equations of the encoder-decoder model.
Revised: Figure 1[left] summarizes the compute equations of the encoder-decoder model.

Original: If we view Equationas an approximation of the full joint distribution in Equation 1, we ﬁnd that the treatment of the attention variables has been rather ad hoc.
Revised: If we view Equationas an approximation of the full joint distribution in Equation 1, we ﬁnd that the treatment of the attention variables has been rather ad hoc.

Original: Attention was introduced as an after-thought of factorizing on the y t variables, the interaction among various a t s is not expressed, and the inﬂuence of a t on y t by diffusing the inputs is unprincipled. 
Revised: Attention was introduced as an after-thought of factorizing on the y t variables, the interaction among various a t s is not expressed, and the inﬂuence of a t on y t by diffusing the inputs is unprincipled. 2. L ATENT A TTENTION M ODELS

Revised: There have been a number of recent works (Wu et al., 2018;

Revised: Deng et al., 2018; Shankar et al., 2018) which model attention as latent-alignment variable in the joint distribution Equation 1.

Original: We next present a sounder model of the interaction of the various attention and output variables, while being more efﬁcient than the full joint method of
Revised: The model becomes more tractable by assuming that the output y t at each step is dependent only on a t and previous outputs y <t i.e. P ( y t | y < t , a <t , a t ) = P ( y t | y <t , a t ) . Both Shankar et al.

Revised: ; Wu et al.

Revised: (2018) further assume that attention at each time step is independent of attention at other timesteps, and marginalize over all attentions at each time-step as in Equation 3.

Revised: Deng et al.

Revised: (2018) also rely on the same assumption but instead of direct marginalization use variational methods.

Original: Equation 1. 
Revised: All these models can be considered as a neuralization of IBM Model 1. [Equation] Such mean-ﬁeld assumption while making the model signiﬁcantly simpler ignore relationships between attentions which is undesirable. Moreover as we will see in the experiments, they also ignore consistency between attention and output variables.

Revised: We next present a principled model of the interaction of the various attention and output variables, which is as efﬁcient as the mean ﬁeld factorization approach while allowing more realistic latent behavior.

Original: We call our proposed approach: Posterior Attention Models or PAM.
Revised: We call our proposed approach: Posterior Attention Models or PAM.

Original: 2. P OSTERIOR A TTENTION M ODELS
Revised: 2. P OSTERIOR A TTENTION M ODELS

Original: Our goal is to express the joint distribution as a product of tractable terms computed at each time step much like in existing ED model, but via a less ad hoc treatment of the attention variables a 1 , . .
Revised: Our goal is to express the joint distribution as a product of tractable terms computed at each time step much like in existing ED model, but via a less ad hoc treatment of the attention variables a 1 , . .

Original: . , a n .
Revised: . , a n .

Original: We use y <t , a <t to denote all output and attention variables before t that is, y 1 , , . . .
Revised: We use y <t , a <t to denote all output and attention variables before t that is, y 1 , , . . .

Original: y t − 1 , a 1 , . . .
Revised: y t − 1 , a 1 , . . .

Original: a t − 1 .
Revised: a t − 1 .

Original: Here and in the rest of the paper we will drop x 1: m to use the shorter form P ( y ) for Pr( y | x 1: m ) .
Revised: Here and in the rest of the paper we will drop x 1: m to use the shorter form P ( y ) for Pr( y | x 1: m ) .

Original: We ﬁrst factorize Eq 1 via chain rule, like in ED but jointly on both a and y .
Revised: We ﬁrst factorize Eq 1 via chain rule, like in ED but jointly on both a and y .

Original: [Equation]
Revised: [Equation]

Original: We then make two mild assumptions: First,that the output y tat each step is dependent only on a t and previous outputs y <t and is independent of all other attention variables, i.e., P ( y t | y <t , a <t , a t ) = P ( y t | y <t , a t ) .
Revised: We then make two mild assumptions: First, the same local attention assumption that y t is dependent only on a t and previous outputs y <t as detailed earlier Wang et al.

Original: Second, a Markovian assumption on the attention variables i.e., P ( a n | y <n , a <n ) =
Revised: Second, a Markovian assumption on the attention variables i.e., P ( a n | y <n , a <n ) =

Original: P ( a n | y <n , a n − 1 ) .
Revised: P ( a n | y <n , a n − 1 ) .

Original: These together allows us to simplify the above joint as:
Revised: These together allows us to simplify the above joint as:

Original: [Equation]
Revised: [Equation]

Original: The last equality is after applying the same rewrite recursively on P ( y <n ) .
Revised: The last equality is after applying the same rewrite recursively on P ( y <n ) .

Original: Thus, we have expressed the joint distribution as a product of factors that apply at each decoding step t while conditioning only on previous outputs and attention.
Revised: Thus, we have expressed the joint distribution as a product of factors that apply at each decoding step t while conditioning only on previous outputs and attention.

Original: The term (cid:80) a t − 1 P ( a t | a t − 1 , y <t ) P ( a t − 1 | y <t ) =
Revised: The term (cid:80) a t − 1 P ( a t | a t − 1 , y <t ) P ( a t − 1 | y <t ) =

Original: P ( a t | y <t ) is the attention at step ’ t ’ conditioned on all previous outputs.
Revised: P ( a t | y <t ) is the attention at step ’ t ’ conditioned on all previous outputs.

Original: For reasons that will soon become clear we call this the prior attention at t and denote as Prior t ( a ) .
Revised: For reasons that will soon become clear we call this the prior attention at t and denote as Prior t ( a ) .

Original: We call P ( a t − 1 | y <t ) =
Revised: We call P ( a t − 1 | y <t ) =

Original: P ( a t − 1 | y <
Revised: P ( a t − 1 | y <

Original: ( t − 1) , y t − 1 ) as the posterior attention Postr ( a t − 1 ) since this is the attention distribution after observing the output label at the corresponding step, unlike in prior attention.
Revised: ( t − 1) , y t − 1 ) as the posterior attention Postr ( a t − 1 ) since this is the attention distribution after observing the output label at the corresponding step, unlike in prior attention.

Original: We expect this attention to be more accurate than the prior that is computed without knowledge of the output token at that step.
Revised: We expect this attention to be more accurate than the prior that is computed without knowledge of the output token at that step.

Original: We compute posterior attention at any t using prior attention at t − 1 by applying Bayes rule as follows:[Equation]
Revised: We compute posterior attention at any t using prior attention at t − 1 by applying Bayes rule as follows:

Original: [Equation]
Revised: [Equation]

Original: The above equations give us the important result that the attention at step t should be computed from the posterior attention of the previous step.
Revised: The above equations give us the important result that the attention at step t should be computed from the posterior attention of the previous step.

Original: Intuitively, also it makes sense because attention reﬂects an alignment of the input and output, and its distribution will improve if the output is known.
Revised: Intuitively, also it makes sense because attention reﬂects an alignment of the input and output, and its distribution will improve if the output is known.

Original: We get into details of computing such coupled attention in Section 2.2.1.
Revised: We get into details of computing such coupled attention in Section 2.3.1.

Original: 2.2. C OMPUTATION OF P RIOR A TTENTION D ISTRIBUTION
Revised: 2.3. C OMPUTATION OF P RIOR A TTENTION D ISTRIBUTION

Original: We use the RNN to summarize y <t as a ﬁxed length vector s t as in current ED models.
Revised: We use the RNN to summarize y <t as a ﬁxed length vector s t as in current ED models.

Original: We then discuss three different methods we explored for computing (cid:80) a t − 1 P ( a t | s t , a t − 1 , y t 
Revised: We then discuss three different methods we explored for computing (cid:80) a t − 1 P ( a t | s t , a t − 1 , y t − 1 )

Original: [Equation]

Original: Postr t − 1 ( a t − 1 ) .
Revised: Postr t − 1 ( a t − 1 ) .

Original: Our methods are designed to be light-weight in terms of the number of extra parameters they consume beyond the default soft-attention methods to have the fairest comparison.
Revised: Our methods are designed to be light-weight in terms of the number of extra parameters they consume beyond the default soft-attention methods to have the fairest comparison.

Original: Postr-Joint The simplest of these uses the same decoder RNN to absorb the posterior attention of the previous step.
Revised: Postr-Joint The simplest of these uses the same decoder RNN to absorb the posterior attention of the previous step.

Original: We linearize the function using the ﬁrst order Taylor expansion to efﬁciently approximate computation of Prior t ( a ) similar to the deterministic technique of Xu et al.
Revised: We linearize the function using the ﬁrst order Taylor expansion to efﬁciently approximate computation of Prior t ( a ) similar to the deterministic technique of Xu et al.

Original: Prior
Revised: Prior

Original: [Equation]
Revised: [Equation]

Original: The above equation suggests that the decoder RNN state should be updated as s t =
Revised: The above equation suggests that the decoder RNN state should be updated as s t =

Original: RNN
Revised: RNN

Original: [Equation]
Revised: [Equation]

Revised: s

Original:  The computation here is thus similar to existing ED model’s but the crucial difference is that the context used to update the RNN is computed from posterior attention, and not the prior attention.
Revised: [Equation] y t [Equation] The computation here is thus similar to existing ED model’s but the crucial difference is that the context used to update the RNN is computed from posterior attention, and not the prior attention.

Original: We will see that this leads to large improvement in accuracy.
Revised: We will see that this leads to large improvement in accuracy.

Original: Proximity biased Next we experiment with models that explicitly couple adjacent attention.
Revised: Proximity biased Next we experiment with models that explicitly couple adjacent attention.

Original: These models utilize an index based coupling between attention positions of the form
Revised: These models utilize an index based coupling between attention positions of the form

Original: [Equation]
Revised: [Table]

Original: where A θ ( x a t , s t ) is the attention logit computed from the previous RNN step, k ( a t , a t − 1 ) is the attention coupling energy and Z is the normalization constant.
Revised: where A θ ( x a t , s t ) is the attention logit computed from the previous RNN step, k ( a t , a t − 1 ) is the attention coupling energy and Z is the normalization constant.

Original: In the proximity based attention the coupling energy k ( a t , a t − 1 ) is given by I ( | a t − a t − 1 | < 3) δ a t − a t − 1 .
Revised: In the proximity based attention the coupling energy k ( a t , a t − 1 ) is given by I ( | a t − a t − 1 | < 3) δ a t − a t − 1 .

Original: This model provides a greater focus on attending states within a window of size ﬁve centered around the recently attended input state.
Revised: This model provides a greater focus on attending states within a window of size ﬁve centered around the recently attended input state.

Original: We label this model as Prox-Postr-Joint in our experiments.
Revised: We label this model as Prox-Postr-Joint in our experiments.

Original: Monotonicity biased
Revised: Monotonicity biased

Original: This method differs from the above proximity-based attention only in how it deﬁnes the coupling energy k ( a t , a t − 1 ) .
Revised: This method differs from the above proximity-based attention only in how it deﬁnes the coupling energy k ( a t , a t − 1 ) .

Original: As the name implies, in this method k ( a t , a t − 1 ) is a monotonic energy given by I ( a t > a t − 1 ) δ a t − a t − 1 − 1 .
Revised: As the name implies, in this method k ( a t , a t − 1 ) is a monotonic energy given by I ( a t > a t − 1 ) δ a t − a t − 1 − 1 .

Original: This model provides a positive exponentially [Equation] [Equation] [Equation]
Revised: This model provides a positive exponentially decaying bias towards encoder states which are to the right of the current attended state, thus inﬂuecning attention to be more monotonic.

Original: (cid:80)

Original: [Equation] Pr( y | x 1: m ) = n (cid:89) t = m (cid:88) a = P ( y t | s t , x a ) Prior t ( a ) (10) s t +1 =
Revised: As we shall see tasks with natural monotonic attention beneﬁt from this form of bias.

Original: RNN ( s t , y t , (cid:88)

Original: a

Original: Postr t ( a ) x a ) (11)

Original: Postr t ( a ) =

Original: P ( y t | s t , x a ) Prior t ( a ) (cid:80) a (cid:48) P ( y t | s t , x a (cid:48) )

Original: Prior t ( a (cid:48) ) (12)

Original: Prior t ( a t ) =

Original: (cid:88) a (cid:48) P ( a t | s t − 1 , a (cid:48) ) Postr t − 1 ( a (cid:48) ) (13) P ( a t | s t − 1 , a (cid:48) )
Revised: This model is denoted as Mono-Postr-Joint in our experiments.

Original: = See Section 2.2.

Original: 2. P UTTING IT ALL TOGETHER
Revised: 2. P UTTING IT ALL TOGETHER

Original: In Figure 1 we put together the ﬁnal set of equations that are used to compute the output distribution and contrast with existing attention model.
Revised: In Figure 1 we put together the ﬁnal set of equations that are used to compute the output distribution and contrast with existing attention model.

Original: We call this overall architecture as Posterior Attention
Revised: We call this overall architecture as Posterior Attention

Original: Model (PAM).
Revised: Model (PAM).

Original: First note that in PAM, we explicitly compute a joint distribution of output and attention at each step and marginalize out the attention.
Revised: First note that in PAM, we explicitly compute a joint distribution of output and attention at each step and marginalize out the attention.

Original: Thus, the output is a mixture of multiple output distributions each of which is a function of one focused input (like in hard attention), and not a diffused sum of the input (like in soft attention).
Revised: Thus, the output is a mixture of multiple output distributions each of which is a function of one focused input (like in hard attention), and not a diffused sum of the input (like in soft attention).

Original: This difference in the way attention is marginalized is not only statistically sound, but also leads to higher accuracy.
Revised: This difference in the way attention is marginalized is not only statistically sound, but also leads to higher accuracy.

Original: The only downside of the joint model is that we need to compute m softmaxes for each output y t , and this may be impractical when the vocabulary size is large.
Revised: The only downside of the joint model is that we need to compute m softmaxes for each output y t , and this may be impractical when the vocabulary size is large.

Original: A simple and effective ﬁx to this is to select the top-K attentions based on
Revised: A simple and effective ﬁx to this is to select the top-K attentions based on

Original: Prior t and compute the ﬁnal output distribution as.
Revised: Prior t and compute the ﬁnal output distribution as.

Original: [Equation]
Revised: [Equation]

Original: Small values of K (order 6), sufﬁce to provide good performance
Revised: Small values of K (order 6), sufﬁce to provide good performance

Original: The second difference is that the attention distribution that is propagated to the next step is posterior to observing the current output.
Revised: The second difference is that the attention distribution that is propagated to the next step is posterior to observing the current output.

Original: We derived this from a principled rewrite of the joint distribution, and were pleasantly surprised to see signiﬁcant accuracy gains by this subtle difference in the way the decoder state is updated.
Revised: We derived this from a principled rewrite of the joint distribution, and were pleasantly surprised to see signiﬁcant accuracy gains by this subtle difference in the way the decoder state is updated.

Original: Computing the posterior attention does not incur any additional overheads because the joint attention-output distribution was already materialized in the ﬁrst equation.
Revised: Computing the posterior attention does not incur any additional overheads because the joint attention-output distribution was already materialized in the ﬁrst equation.

Original: However, due to the sparsity induced by the top-K operation on attention probabilities, the posterior probabilities are unrealistically sparse.
Revised: However, due to the sparsity induced by the top-K operation on attention probabilities, the posterior probabilities are unrealistically sparse.

Original: As such we augment the posterior attention using input from standard attention, by using an equally weighted combination of the two distributions.
Revised: As such we augment the posterior attention using input from standard attention, by using an equally weighted combination of the two distributions.

Original: Third, the prior attention distribution is explicitly conditioned on the previous attention.
Revised: Third, the prior attention distribution is explicitly conditioned on the previous attention.

Original: This allowed us to incorporate various application-speciﬁc natural biases like proximity and monotonicity of adjacent attentions.
Revised: This allowed us to incorporate various application-speciﬁc natural biases like proximity and monotonicity of adjacent attentions.

Original: 2. A LTERNATIVE R EWRITES
Revised: 2. A LTERNATIVE R EWRITES

Original: Our rewrite although somewhat unconventional was derived to satisfy two important goals: First, explain the need to propagate posterior attention to subsequent steps.
Revised: Our rewrite although somewhat unconventional was derived to satisfy two important goals: First, explain the need to propagate posterior attention to subsequent steps.

Original: Second, to factorize the joint as the product of the local distribution at each time t which allows efﬁcient gradient updates and minimal changes to existing beam-search inference.
Revised: Second, to factorize the joint as the product of the local distribution at each time t which allows efﬁcient gradient updates and minimal changes to existing beam-search inference.

Original: A more conventional rewrite for handling Markovian dependencies is the standard forward algorithm which works as follows.
Revised: A more conventional rewrite for handling Markovian dependencies is the standard forward algorithm which works as follows.

Original: First we write:
Revised: First we write:

Revised: [Equation]

Revised: [Equation]

Revised: Pr( y | x 1:

Original: [Equation]
Revised: [Equation]

Revised: ) = n (cid:89) t = m (cid:88) a =

Revised: P ( y t | s t , x a ) Prior t ( a ) (11) s t +1 =

Revised: RNN ( s t , y t , (cid:88)

Revised: a

Revised: Postr t ( a ) x a ) (12)

Revised: Postr t ( a ) =

Revised: P ( y t | s t , x a ) Prior t ( a ) (cid:80) a (cid:48) P ( y t | s t , x a (cid:48) )

Revised: Prior t ( a (cid:48) ) (13)

Revised: Prior t ( a t ) =

Revised: (cid:88) a (cid:48)

Revised: P ( a t | s t − 1 , a (cid:48) )

Revised: Postr t − 1 ( a (cid:48) ) (14)

Revised: P ( a t | s t − 1 , a (cid:48) )

Revised: = See Section 2.3.

Original: Then use the forward algorithm to compute:
Revised: Then use the forward algorithm to compute:

Original: [Equation]
Revised: [Equation]

Original: which then gives the joint distribution as P ( y )
Revised: which then gives the joint distribution as P ( y )

Original: = (cid:80) a α n ( a ) .
Revised: = (cid:80) a α n ( a ) .

Original: This expression is neither in the outer product form, nor does it motivate the need for posterior attention.
Revised: This expression is neither in the outer product form, nor does it motivate the need for posterior attention.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The de facto standard for sequence to sequence learning via neural networks is the encoder decoder model.
Revised: The de facto standard for sequence to sequence learning via neural networks is the encoder decoder model.

Original: Ever since their ﬁrst introduction in Bahdanau et al.
Revised: Ever since their ﬁrst introduction in Bahdanau et al.

Original: (2014), many different attention models have been proposed.
Revised: (2014), many different attention models have been proposed.

Original: We discuss them here.
Revised: We discuss them here.

Original: Soft Attention is the de-facto mechanism for seq2seq learning et al (2018).
Revised: Soft Attention is the de-facto mechanism for seq2seq learning et al (2018).

Original: It was proposed for translation in Bahdanau et al.
Revised: It was proposed for translation in Bahdanau et al.

Original: and reﬁned further in Luong et al. (2015).
Revised: and reﬁned further in Luong et al. (2015).

Original: The output derives from an attention averaged context.
Revised: The output derives from an attention averaged context.

Original: The advantage is end to end differentiability.
Revised: The advantage is end to end differentiability.

Original: Hard Attention was proposed in Xu et al.
Revised: Hard Attention was proposed in Xu et al.

Original: and attends to exactly one input state for an output.
Revised: and attends to exactly one input state for an output.

Original: The merit of hard attention is that the output is determined from a single input rather than an average of all inputs.
Revised: The merit of hard attention is that the output is determined from a single input rather than an average of all inputs.

Revised: Accordingly it has proven useful in when explicit focus is beneﬁcial such as model adaptation Shankar & Sarawagi (2018) and catastrophic forgetting Serrà et al.

Original: However due to non-differentiability, training Hard-Attention requires the REINFORCE Williams (1992) algorithm and is subject to high variance, requiring careful tricks to train reliably.
Revised: However due to non-differentiability, training Hard-Attention requires the REINFORCE Williams (1992) algorithm and is subject to high variance, requiring careful tricks to train reliably.

Original: Yu et al.
Revised: Yu et al.

Original: (2016) keep the encoder and decoder independent to allow for easier marginalization.
Revised: (2016) keep the encoder and decoder independent to allow for easier marginalization.

Original: Aharoni & Goldberg (2017) use a monotonic hard attention and avoid the problem, by supervising hard attention with external alignment information.
Revised: Aharoni & Goldberg (2017) use a monotonic hard attention and avoid the problem, by supervising hard attention with external alignment information.

Original: Our model in equation 10 uses hard attention on the encoder states.
Revised: Our model in equation 11 uses hard attention on the encoder states.

Original: However unlike standard hard attention we do not use a one-hot attention and instead are computing the exact marginalization.
Revised: However unlike standard hard attention we do not use a one-hot attention and instead are computing the exact marginalization.

Original: Sparse/Local Attention
Revised: Sparse/Local Attention

Original: Many attempts have been made to bridge the gap between soft and hard attention.
Revised: Many attempts have been made to bridge the gap between soft and hard attention.

Original: Luong et al.
Revised: Luong et al.

Original: (2015) proposes local attention that averages a window of input.
Revised: (2015) proposes local attention that averages a window of input.

Original: This has been reﬁned later to include syntax (Chen et al., 2017; Sennrich & Haddow, 2016; Chen et al., 2018) and has been explored for image captioning in Gregor et al.
Revised: This has been reﬁned later to include syntax (Chen et al., 2017; Sennrich & Haddow, 2016; Chen et al., 2018) and has been explored for image captioning in Gregor et al.

Original: A related idea to hard attention is to make it sparse using sparsity inducing operators (Martins & Astudillo, 2016; Niculae & Blondel, 2017).
Revised: A related idea to hard attention is to make it sparse using sparsity inducing operators (Martins & Astudillo, 2016; Niculae & Blondel, 2017).

Original: However, all sparse/local attention methods continue to compute P ( y ) from an attention weighted sum of inputs like in soft attention.
Revised: However, all sparse/local attention methods continue to compute P ( y ) from an attention weighted sum of inputs like in soft attention.

Original: Recurrent Attention Yang et al.
Revised: Recurrent Attention Yang et al.

Original: (2016) have previously modeled relationship between the attentions at different time steps by using a recurrent history mechanism.
Revised: (2016) have previously modeled relationship between the attentions at different time steps by using a recurrent history mechanism.

Original: The attention history of an input word and its surrounding words are captured in a summary vector by an RNN, which is provided as further input to the attention mechanism for incorporating dependence on history.
Revised: The attention history of an input word and its surrounding words are captured in a summary vector by an RNN, which is provided as further input to the attention mechanism for incorporating dependence on history.

Original: While both works model dependence between attention at different steps, our principled rewrite of the joint distribution shows that posterior attention should be the link to the next attention.
Revised: While both works model dependence between attention at different steps, our principled rewrite of the joint distribution shows that posterior attention should be the link to the next attention.

Original: Structured Attention Networks Similar to this work, Kim et al.

Original: (2017) interpret attention as latent structural variable.

Original: The authors then take advantage of easy inference in certain graphical models to implement forms of segmental and syntactic attention.

Original: These works only focus on attention at each step independently whereas our focus is modeling the dependency among adjacent attention.

Original: Moreover our posterior attention framework is independent of how the prior attention at each position is modeled.

Original: In this paper we assumed a multinomial distribution but the structured distribution of

Original: Kim et al.

Original: (2017) can also beneﬁt from our posterior coupling.

Original: Latent Attention Models There has been a number of recent works which model attention as latentalignments in (Wu et al., 2018;

Original: Deng et al., 2018; Shankar et al., 2018).

Original:  Our model can be considered as a generalization of the work of Wang et al.
Revised: Latent Attention Models Our model can be considered as a generalization of the work of Wang et al.

Original: (2018) to the case where attention is also provided to the RNN.
Revised: (2018) to the case where attention is also provided to the RNN.

Original: The model in Shankar et al.
Revised: The model in Shankar et al.

Original: and related input-feeding model of Wu et al.

Original: (2018) also factorize the joint distribution and are identical to our Prior-Joint model.
Revised: (2018) also factorizes the joint distribution and are identical to our Prior-Joint model.

Original: However unlike these models we explicitly model the posterior attention distribution and attention coupling.
Revised: However unlike these models we explicitly model the posterior attention distribution and attention coupling.

Original: Deng et al.
Revised: Deng et al.

Original: (2018) proposes to learn the posterior attention via variational methods.
Revised: (2018) proposes to learn the posterior attention via variational methods.

Original: A key difference is while their model tries to supervise attention using a posterior inference network via a KL term, we directly use the actual posterior for computing attention in the subsequent steps.
Revised: A key difference is while their model tries to supervise attention using a posterior inference network via a KL term, we directly use the actual posterior for computing attention in the subsequent steps.

Original: Their method requires training a separate inference network to model the posterior whereas we require no additional network parameters.

Original: In our method, posterior attention is used in identical roles across training and inference, unlike in Deng et al.
Revised: In our method, posterior attention is used in identical roles across training and inference, unlike in Deng et al.

Original: (2018)’s that rely on variational training.
Revised: (2018)’s that rely on variational training.

Original: E XPERIMENTS
Revised: Structured Attention Networks Similar to this work, Kim et al. (2017) interpret attention as latent structural variable. The authors then take advantage of easy inference in certain graphical models to implement forms of segmental and syntactic attention.

Original: [Table] [Table] [Table] [Table]
Revised: These works only focus on attention at each step independently whereas our focus is modeling the dependency among adjacent attention.

Original: [Table] [Table]
Revised: Moreover our posterior attention framework is independent of how the prior attention at each position is modeled.

Original: [Table] [Table] [Table] [Table]
Revised: In this paper we assumed a multinomial distribution but the structured distribution of

Original: [Table] [Table] [Table] [Table]
Revised: Kim et al.

Original: [Table] [Table] [Table] [Table]
Revised: (2017) can also beneﬁt from our posterior coupling.

Original:  [Table] [Table] [Table] [Table]
Revised:  E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: We compare our posterior attention model on two sequence to sequence learning tasks: machine translation and morphological inﬂection.
Revised: We compare our posterior attention model on two sequence to sequence learning tasks: machine translation and morphological inﬂection.

Original: We compare on the following methods:
Revised: We compare on the following methods:

Original: [Table] : attention Luong et al. (2015).
Revised: Soft: attention Luong et al. (2015).

Original: Sparse: attention Niculae & Blondel (2017).[Table] : Our [Table]
Revised: Sparse: attention Niculae & Blondel (2017).

Original: [Table]

Original: [Table] -Joint: monotonic PAM (§ [Table]
Revised: Postr-Joint: Our default PAM ( 2.3.1).

Original: [Table] [Table] biased PAM(§ 2.2.1)
Revised: Mono-Postr-Joint: monotonic PAM (§ 2.3.1) Prox-Postr-Joint: Proximity biased PAM(§ 2.3.1) Prior-Joint: suppresses

Original: Prior-Joint: supresses the dependence on previous attention analogous to current ED models. This reduces the model to that proposed in Shankar et al.
Revised: the dependence on previous attention like in current ED models, thus reducing to methods proposed in Shankar et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: Variational: method of Deng et al.
Revised: Variational: method of Deng et al.

Revised: (2018) using author’s code with our tokenization and architecture.

Original: 4. M ACHINE TRANSLATION
Revised: 4. M ACHINE TRANSLATION

Original: We experiment on ﬁve language pairs from three datasets:
Revised: We experiment on ﬁve language pairs from three datasets:

Original: IWSLT
Revised: IWSLT

Original: English ↔ Vietnamese ,
Revised: English ↔ Vietnamese ,

Original: IWSLT
Revised: IWSLT

Original: German ↔ English
Revised: German ↔ English

Original: Cettolo et al.
Revised: Cettolo et al.

Original: (2015); and WAT17 Japanese → English
Revised: (2015); and WAT17 Japanese → English

Original: Nakazawa et al.
Revised: Nakazawa et al.

Original: We use a 2 layer bi-directional encoder and 2 layer decoder with 512 LSTM units and 0.2 dropout with vanilla SGD optimizer.
Revised: We use a 2 layer bi-directional encoder and 2 layer decoder with 512 LSTM units and 0.2 dropout with vanilla SGD optimizer.

Original: We use word level encoding for all translation tasks.
Revised: We use word level encoding for all translation tasks.

Original: Instead of tuning hyper-parameters for our method we used Soft-Attention tuned parameters.
Revised: Instead of tuning hyper-parameters for our method we used Soft-Attention tuned parameters.

Original: Our results are in Table 1 where we show perplexity (PPL) and BLEU with beam size 4 and 10.
Revised: Our results are in Table 1 where we show perplexity (PPL) and BLEU with beam size 4 and 10.

Original: All
Revised: All

Original: Postr-Joint variants and Prior-Joint outperform soft attention and sparse-attention by large margins.
Revised: Postr-Joint variants and Prior-Joint outperform soft attention and sparse-attention by large margins.

Original: Moreover models with posterior attention show improvement over those which use prior attention.
Revised: Moreover models with posterior attention show improvement over those which use prior attention.

Original: We also observe small improvements over the more sophisticated  variational attention model likely due to the use of the exact posterior during inference instead.
Revised: We also observe small improvements over the more sophisticated and compute-intensive variational attention model likely due to the use of the exact posterior during inference instead.

Original: 1 This clearly shows the performance advantage of joint modeling and posterior attention.
Revised: 1 This clearly shows the performance advantage of joint modeling and posterior attention.

Original: We shall analyze the reasons for these improvements later.
Revised: We shall analyze the reasons for these improvements later.

Original: Next we explore the impact of different coupling models discussed in 2.2.1.
Revised: Next we explore the impact of different coupling models discussed in 2.3.1.

Original: For that focus on methods Postr-Joint, Prox-Postr-Joint, and Mono-Postr-Joint in Table 1.
Revised: For that focus on methods Postr-Joint, Prox-Postr-Joint, and Mono-Postr-Joint in Table 1.

Original: We obtain some gains over
Revised: We obtain some gains over

Original: Postr-Joint by explicitly modeling attention coupling.
Revised: Postr-Joint by explicitly modeling attention coupling.

Original: For language-pairs with a natural monotonic alignment like German-English, Mono-Postr-Joint slightly outperforms other models by (0.1-0.
Revised: For language-pairs with a natural monotonic alignment like German-English, Mono-Postr-Joint slightly outperforms other models by (0.1-0.

Original: BLEU points).
Revised: BLEU points).

Original: English-Vietnamese is a more non-monotonic pair and as expected we do not ﬁnd gains by incorporating a monotonic bias.
Revised: English-Vietnamese is a more non-monotonic pair and as expected we do not ﬁnd gains by incorporating a monotonic bias.

Original: 4. M ORPHOLOGICAL I NFLECTION
Revised: 4. M ORPHOLOGICAL I NFLECTION

Original: To demonstrate the use of our model beyond translation, we next consider the task of generating morphological inﬂections.
Revised: To demonstrate the use of our model beyond translation, we next consider the task of generating morphological inﬂections.

Original: We use inﬂection forms for German Nouns (de-N) and German Verbs (de-V) from Durrett & DeNero (2013)’s.
Revised: We use inﬂection forms for German Nouns (de-N) and German Verbs (de-V) from Durrett & DeNero (2013)’s.

Original: A model is trained separately for each type of inﬂection to predict the inﬂected character sequence.
Revised: A model is trained separately for each type of inﬂection to predict the inﬂected character sequence.

Original: We train a one layer encoder and decoder with 128 hidden
Revised: We train a one layer encoder and decoder with 128 hidden

Original: LSTM units each with a dropout rate of 0.2 using Adam and measure 0/1 accuracy.
Revised: LSTM units each with a dropout rate of 0.2 using Adam and measure 0/1 accuracy.

Original: We also ran the 100 units wide two layer LSTM with hard-monotonic attention model Aharoni & Goldberg (2017) labeled Hard-Mono 2 .
Revised: We also ran the 100 units wide two layer LSTM with hard-monotonic attention model Aharoni & Goldberg (2017) labeled Hard-Mono 2 .

Original: [Table]
Revised: [Table]

Original: Using joint modeling we get signiﬁcant gains (0.3 points) even against task-speciﬁc hard-monotonic attention, showing that our approach is more general than translation.
Revised: Using joint modeling we get signiﬁcant gains (0.3 points) even against task-speciﬁc hard-monotonic attention, showing that our approach is more general than translation.

Original: Moreover when we use
Revised: Moreover when we use

Original: Mono-Postr-Joint which has a structural bias towards task-speciﬁc monotonic attention, we obtain immense improvements (upto 1 accuracy point) over joint models.
Revised: Mono-Postr-Joint which has a structural bias towards task-speciﬁc monotonic attention, we obtain immense improvements (upto 1 accuracy point) over joint models.

Original: 4. E XPLAINING WHY WE SCORE ABOVE S OFT A TTENTION
Revised: 4. E XPLAINING WHY WE SCORE ABOVE S OFT A TTENTION

Original: We attempt to get more insights on why posterior attention models score over soft attention in end to end accuracy.
Revised: We attempt to get more insights on why posterior attention models score over soft attention in end to end accuracy.

Original: We show that the main reason is better alignment of input and output because of a more precise attention model.
Revised: We show that the main reason is better alignment of input and output because of a more precise attention model.

Original: We demonstrate that by ﬁrst showing some anecdotes of better alignment, then showing that posterior attention is more focused (has lower entropy), provides better alignment accuracy, and better input coverage.
Revised: We demonstrate that by ﬁrst showing some anecdotes of better alignment, then showing that posterior attention is more focused (has lower entropy), provides better alignment accuracy, and better input coverage.

Original: For these runs we perform experiments in the teacher forcing setup so as to compare two models’ distributions under identical inputs.
Revised: For these runs we perform experiments in the teacher forcing setup so as to compare two models’ distributions under identical inputs.

Original: Anecdotal Examples Fig2 presents the heatmap of difference between Postr-Joint and SoftAttention on two different sentences.
Revised: Anecdotal Examples Fig2 presents the heatmap of difference between Postr-Joint and SoftAttention on two different sentences.

Original: In each ﬁgure the red regions represents where Postr-Joint has greater attention and blue where soft-attention has greater focus.
Revised: In each ﬁgure the red regions represents where Postr-Joint has greater attention and blue where soft-attention has greater focus.

Original: One can observe that Soft-Attention is far more diffused.
Revised: One can observe that Soft-Attention is far more diffused.

Original: More importantly, we can see that Postr-Joint is able to correct mistakes and provides the appropriate context for the next step.
Revised: More importantly, we can see that Postr-Joint is able to correct mistakes and provides the appropriate context for the next step.

Original: For example in Fig2a Soft-Attention (blue) has maximum focus on the source word ’generationen’ when the target word is innovation which corresponds to ’innovationen’; on the other hand Postr-Joint is able to correct this.
Revised: For example in Fig2a Soft-Attention (blue) has maximum focus on the source word ’generationen’ when the target word is innovation which corresponds to ’innovationen’; on the other hand Postr-Joint is able to correct this.

Original: Similarly while producing the phrase ’but the same’ Postr-Joint focuses the attention on the source word ’dasselbe’
Revised: Similarly while producing the phrase ’but the same’ Postr-Joint focuses the attention on the source word ’dasselbe’

Original: Fig2b.
Revised: Fig2b.

Original: This provides insight into as to how by providing better contexts via incorporating the target, posterior attention can outperform prior attention.
Revised: This provides insight into as to how by providing better contexts via incorporating the target, posterior attention can outperform prior attention.

Original: Attention Entropy Vs Accuracy We expect Soft-Attn to be worse hit by high attention uncertainty than other models.
Revised: Attention Entropy Vs Accuracy We expect Soft-Attn to be worse hit by high attention uncertainty than other models.

Original: This, if true, could illustrate that P ( y t | x t ) distribution can be learned more easily
Revised: This, if true, could illustrate that P ( y t | x t ) distribution can be learned more easily

Original: [Figure]
Revised: [Figure]

Original:  Alignment accuracy Failure of attention to produce latent structures which correspond to linguistic structures has been noted by Koehn & Knowles (2017); Ghader & Monz (2017).Based on few examples, we hypothesize that Posterior Attention should be able to produce better alignments.
Revised: relatively stabler Alignment accuracy Failure of attention to produce latent structures which correspond to linguistic structures has been noted by Koehn & Knowles (2017); Ghader & Monz (2017).Based on few examples, we hypothesize that Posterior Attention should be able to produce better alignments.

Original: To test this we used the RWTH German-English dataset which provides alignment information manually tagged by experts, and compare the alignment accuracy for Soft, Prior-Joint and Postr-Joint attentions.
Revised: To test this we used the RWTH German-English dataset which provides alignment information manually tagged by experts, and compare the alignment accuracy for Soft, Prior-Joint and Postr-Joint attentions.

Original: Following the procedure in Ghader & Monz (2017) the most attended source word for each target word is taken as the aligned word.
Revised: Following the procedure in Ghader & Monz (2017) the most attended source word for each target word is taken as the aligned word.

Original: We used the AER metric Koehn (2010) to compare these against the expert alignments.
Revised: We used the AER metric Koehn (2010) to compare these against the expert alignments.

Original: Table 3 presents the AER accuracy for different models.
Revised: Table 3 presents the AER accuracy for different models.

Original: One can read off that Postr-Joint model beats the second best model ( Prior-Joint ) by more than 10%, and dwarfs soft-attention by a huge margin, proving that posterior alignments are signiﬁcantly more compatible with true alignments. 
Revised: One can read off that Postr-Joint model beats the second best model ( Prior-Joint ) by more than 10%, and dwarfs soft-attention by a huge margin, proving that posterior alignments are signiﬁcantly more compatible with true alignments. Attention AER

Original: [Table]
Revised: [Table]

Original:  Fraction of covered tokens A natural expectation for translation is that by the time the entire output sentence has been produced, attention would have covered the entire input sequence.
Revised: [Figure] Fraction of covered tokens A natural expectation for translation is that by the time the entire output sentence has been produced, attention would have covered the entire input sequence.

Original: A loss based on this precise heuristic was used in Chorowski & Jaitly (2016) to improve the performance of a attention based seq2seq model for speech transcription.
Revised: A loss based on this precise heuristic was used in Chorowski & Jaitly (2016) to improve the performance of a attention based seq2seq model for speech transcription.

Original: In this experiment we try to indirectly assess reliability of different attention models via measuring whether cumulatively attention has focused on the entire input sequence.
Revised: In this experiment we try to indirectly assess reliability of different attention models via measuring whether cumulatively attention has focused on the entire input sequence.

Original: We plot the frequency distribution of the coverage in Fig4.
Revised: We plot the frequency distribution of the coverage in Fig4.

Original: Note that in soft attention model, there are many sentences which do not receive enough attention during the entire decoding process.
Revised: Note that in soft attention model, there are many sentences which do not receive enough attention during the entire decoding process.

Original: Prior-Joint and Postr-Joint have similar behavior with few instances of one outperforming the other, however both outperform soft attention by huge margins.
Revised: Prior-Joint and Postr-Joint have similar behavior with few instances of one outperforming the other, however both outperform soft attention by huge margins.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We show in this paper that none of the existing attention models adequately model the dependence of the output and attention along the length of the output for general sequence prediction tasks.
Revised: We show in this paper that none of the existing attention models adequately model the dependence of the output and attention along the length of the output for general sequence prediction tasks.

Original: We propose a factorization of the joint distribution, and develop practical approximations that allows[Figure] the joint distribution to decompose over output tokens, much like in existing attention.
Revised: We propose a factorization of the joint distribution, and develop practical approximations that allows the joint distribution to decompose over output tokens, much like in existing attention.

Original: Our more principled probabilistic joint modeling of the dependency structure leads to three important differences.
Revised: Our more principled probabilistic joint modeling of the dependency structure leads to three important differences.

Original: First, the output token distribution is obtained by aggregating predictions across all attention.
Revised: First, the output token distribution is obtained by aggregating predictions across all attention.

Original: Second, the concept of conditioning attention on the current output i.e. a posterior attention for inferring the next output becomes important.
Revised: Second, the concept of conditioning attention on the current output i.e. a posterior attention for inferring the next output becomes important.

Original: Our experiments show that it is sounder, more meaningful and more accurate to condition subsequent attention distribution on the posterior attention.
Revised: Our experiments show that it is sounder, more meaningful and more accurate to condition subsequent attention distribution on the posterior attention.

Original: Thirdly, via directly exposing attention coupling, we have a principled way to directly incorporate task-speciﬁc structural biases and prior knowledge into attention.
Revised: Thirdly, via directly exposing attention coupling, we have a principled way to directly incorporate task-speciﬁc structural biases and prior knowledge into attention.

Original: We experimented with some simple biases and found boosts in related tasks.
Revised: We experimented with some simple biases and found boosts in related tasks.

Original: Our work opens avenues for future work in scaling these techniques to large-scale models and multi-headed attention.
Revised: Our work opens avenues for future work in scaling these techniques to large-scale models and multi-headed attention.

Original: Another promising line is to incorporate more complex biases like phrasal structure or image segments into joint attention models.
Revised: Another promising line is to incorporate more complex biases like phrasal structure or image segments into joint attention models.

Revised: Acknowledgements We thank NVIDIA Corporation and Flipkart for supporting this research

Original: A PPENDIX
Revised: A PPENDIX

Original: d a ss e l b e g il t a ll e r d i n g s a u c h f ü
Revised: d a ss e l b e g il t a ll e r d i n g s a u c h f ü

Original: r e r n e u e r b a r e e n e r
Revised: r e r n e u e r b a r e e n e r

Original: g i e n .
Revised: g i e n .

Original: butthe same goes for renewables as well . (a) d a ss e l b e g il t a ll e r d i n g s a u c h f ü r e r n e u e r b a r e e n e r g i e n .
Revised: but

Original: [Figure]
Revised: [Figure]

Original:  as well .
Revised: goes for renewables as well .

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] i
Revised: (a) d a ss e l b e g il t a ll e r d

Original: [Figure] i nn o [Figure] a t i 
Revised: i n g s a u c h f ü r e r n e u e r b a r e e n e r g i e n .

Original: [Figure]
Revised: [Figure]

Original:  n e n [Figure] o [Figure]
Revised: w i r b a u e n a u f i nn o v a t i o n e n v o n g e n e r a t

Original: g
Revised: i

Original: [Figure]

Original: n
Revised: o

Original: [Figure]

Original:  r  a t i [Figure] i nn
Revised: n e n v o r un s a u f .

Original: [Figure]

Original: i
Revised: and

Original: [Figure]
Revised: [Figure]

Original: n

Original: [Figure]
Revised: [Figure]

Original: d)