Original: P OSTERIOR A TTENTION M ODELS FOR S EQUENCE TO S EQUENCE L EARNING
Revised: P OSTERIOR A TTENTION M ODELS FOR S EQUENCE TO S EQUENCE L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modern neural architectures critically rely on attention for mapping structured inputs to sequences.
Revised: Modern neural architectures critically rely on attention for mapping structured inputs to sequences.

Original: In this paper we show that prevalent attention architectures do not adequately model the dependence among the attention and output variables along the length of a predicted sequence.
Revised: In this paper we show that prevalent attention architectures do not adequately model the dependence among the attention and output variables along the length of a predicted sequence.

Original: We present an alternative architecture called Posterior Attention Models that relying on a principled factorization of the full joint distribution of the attention and output variables propose two major changes.
Revised: We present an alternative architecture called Posterior Attention Models that after a principled factorization of the full joint distribution of the attention and output variables propose two major changes.

Original: First, the position where attention is marginalized is changed from the input to the output.
Revised: First, the position where attention is marginalized is changed from the input to the output.

Original: Second, the attention propagated to the next decoding stage is a posterior attention distribution conditioned on the output.
Revised: Second, the attention propagated to the next decoding stage is a posterior attention distribution conditioned on the output.

Original: Empirically on ﬁve translation and two morphological inﬂection tasks the proposed posterior attention models yield better predictions and alignment accuracy than existing attention models.
Revised: Empirically on ﬁve translation and two morphological inﬂection tasks the proposed posterior attention models yield better predictions and alignment accuracy than existing attention models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Attention is a critical module of modern neural models for sequence to sequence learning as applied to tasks like translation, grammar error correction, morphological inﬂection, and speech to text conversion.
Revised: Attention is a critical module of modern neural models for sequence to sequence learning as applied to tasks like translation, grammar error correction, morphological inﬂection, and speech to text conversion.

Original: Attention speciﬁes what part of the input is relevant for each output.
Revised: Attention speciﬁes what part of the input is relevant for each output.

Original: Many variants of attention have been proposed including soft (Bahdanau et al., 2014; Luong et al., 2015), sparse (Martins & Astudillo, 2016), local (Luong et al., 2015), hard (Xu et al., 2015; Zaremba & Sutskever, 2015),and monotonic hardattention (Yu et al., 2016; Aharoni & Goldberg, 2017).
Revised: Many variants of attention have been proposed including soft (Bahdanau et al., 2014; Luong et al., 2015), sparse (Martins & Astudillo, 2016), local (Luong et al., 2015), hard (Xu et al., 2015; Zaremba & Sutskever, 2015), monotonic hard (Yu et al., 2016; Aharoni & Goldberg, 2017), hard non-monotonic Wu et al. ; Shankar et al.

Revised: (2018), variational Deng et al.

Original:  The most prevalent of these is soft attention that computes attention for each output as a multinomial distribution over the input states.
Revised: (2018), The most prevalent of these is soft attention that computes attention for each output as a multinomial distribution over the input states.

Original: The multinomial probabilities serve as weights, and an attention weighted sum of input states serves as relevant context for the output and subsequent attention.
Revised: The multinomial probabilities serve as weights, and an attention weighted sum of input states serves as relevant context for the output and subsequent attention.

Original: Soft attention is end to end differentiable, easy to implement, and hence widely popular.
Revised: Soft attention is end to end differentiable, easy to implement, and hence widely popular.

Original: Hard attention and sparse attentions are difﬁcult to implement and not popularly used.
Revised: Hard attention and sparse attentions are difﬁcult to implement and not popularly used.

Original: In this paper we revisit the statistical soundness of the way soft attention and other variants capture the dependence between attention and output variables, and among multiple attention variables along the length of the sequence.
Revised: In this paper we revisit the statistical soundness of the way soft attention and other variants capture the dependence between attention and output variables, and among multiple attention variables along the length of the sequence.

Original: Our investigation leads to a more principled model that we call the Posterior Attention Model (PAM).
Revised: Our investigation leads to a more principled model that we call the Posterior Attention Model (PAM).

Original: We start with an explicit joint distribution of all output and attention variables in a predicted sequence.
Revised: We start with an explicit joint distribution of all output and attention variables in a predicted sequence.

Original: We then propose a tractable approximation that retains the advantages of forward dependence and token-level decomposition that leads to efﬁcient training and inference.
Revised: We then propose a tractable approximation that retains the advantages of forward dependence and token-level decomposition and thus leads to efﬁcient training and inference.

Original: However, the computations performed at each decode step has two important differences with existing models.
Revised: The computations performed at each decode step has two important differences with existing models.

Original: First, at each decoding step the probability of the output token is a mixture of output probability for each attention.
Revised: First, at each decoding step the probability of the output token is a mixture of output probability for each attention.

Original: In contrast, existing models take a mixture of the input, and compute a single output distribution from this diffused mixed input.
Revised: In contrast, existing models take a mixture of the input, and compute a single output distribution from this diffused mixed input.

Original: We show that our direct coupling of output and attention gives the beneﬁt of hard attention without its computational challenges.
Revised: We show that our direct coupling of output and attention gives the beneﬁt of hard attention without its computational challenges.

Original: Second, we introduce the notion of a posterior attention distribution, that is, the attention distribution conditioned on the current output.
Revised: Second, we introduce the notion of a posterior attention distribution, that is, the attention distribution conditioned on the current output.

Original: We show that it is both statistically sounder and more accurate to condition subsequent attention on the output corrected posterior attention, rather than the output independent prior attention as in existing models.
Revised: We show that it is both statistically sounder and more accurate to condition subsequent attention on the output corrected posterior attention, rather than the output independent prior attention as in existing models.

Original: We evaluate the posterior attention model on ﬁve translation tasks and two morphological inﬂection tasks.
Revised: We evaluate the posterior attention model on ﬁve translation tasks and two morphological inﬂection tasks.

Original: We show that posterior attention provides improved BLEU score, higher alignment accuracy, and better input coverage.
Revised: We show that posterior attention provides improved BLEU score, higher alignment accuracy, and better input coverage.

Original: We also empirically analyze the reasons behind the improved performance of the posterior attention model.
Revised: We also empirically analyze the reasons behind the improved performance of the posterior attention model.

Original: We discover that the entropy of posterior attention is much lower than entropy of soft attention.
Revised: We discover that the entropy of posterior attention is much lower than entropy of soft attention.

Original: This is a signiﬁcant ﬁnd that challenges the current practice of computing attention distribution without considering the output token.
Revised: This is a signiﬁcant ﬁnding that challenges the current practice of computing attention distribution without considering the output token.

Original: The running time overhead of posterior attention is only 40% over existing soft-attention.
Revised: The running time overhead of posterior attention is only 40% over existing soft-attention.

Original:  J OINT D ISTRIBUTION FOR A TTENTION AND O UTPUT V ARIABLES
Revised:  J OINT D ISTRIBUTION FOR A TTENTION AND O UTPUT V ARIABLES

Original: Our goal is to model the conditional distribution
Revised: Our goal is to model the conditional distribution

Original: Pr( y | x ) of an output sequence y =
Revised: Pr( y | x ) of an output sequence y =

Original: y 1 , . .
Revised: y 1 , . .

Original: , y n given an input sequence x = x 1 , . . .
Revised: , y n given an input sequence x = x 1 , . . .

Original: , x m .
Revised: , x m .

Original: Each output y t is a discrete token from a typically large vocabulary V .
Revised: Each output y t is a discrete token from a typically large vocabulary V .

Original: Each x j can be any abstract input.
Revised: Each x j can be any abstract input.

Original: Typically a RNN encodes the input sequence into a sequence of state vectors x 1 , . . .
Revised: Typically a RNN encodes the input sequence into a sequence of state vectors x 1 , . . .

Original: , x m , which we jointly denote as x 1: m .
Revised: , x m , which we jointly denote as x 1: m .

Original: Each y t depends not only on other tokens in the sequence, but on some speciﬁc focused part of the input sequence.
Revised: Each y t depends not only on other tokens in the sequence, but on some speciﬁc focused part of the input sequence.

Original: A hidden variable a t , called the attention variable, denotes which part of x 1: m the output y t depends on.
Revised: A hidden variable a t , called the attention variable, denotes which part of x 1: m the output y t depends on.

Original: We denote the set of all attention as a = a 1 , . . .
Revised: We denote the set of all attention as a = a 1 , . . .

Original: , a n .
Revised: , a n .

Original: During training the input x and output y are observed but the attention a is hidden.
Revised: During training the input x and output y are observed but the attention a is hidden.

Original: Hence, we write Pr( y | x ) as
Revised: Hence, we write Pr( y | x ) as

Original: [Equation]
Revised: [Equation]

Original: The number of variables involved in this summation is daunting, and we need to approximate.
Revised: The number of variables involved in this summation is daunting, and we need to approximate.

Original: We ﬁrst review how existing soft attention-based encoder decoder models handle this challenge.
Revised: We ﬁrst review how existing soft attention-based encoder decoder models handle this challenge.

Original: 2. E XISTING A TTENTION - BASED E NCODER D ECODER M ODEL
Revised: 2. E XISTING A TTENTION - BASED E NCODER D ECODER M ODEL

Original: Existing Encoder-Decoder (ED) networks factorize Pr( y | x 1: m ) by applying chain rule on y variables as (cid:81) nt =1 Pr( y t | x 1: m , y 1 , . . .
Revised: Existing Encoder-Decoder (ED) networks factorize Pr( y | x 1: m ) by applying chain rule on y variables as (cid:81) nt =1 Pr( y t | x 1: m , y 1 , . . .

Original: , y t − 1 ) .
Revised: , y t − 1 ) .

Original: A decoder RNN summarizes the variable length history y 1 , . . .
Revised: A decoder RNN summarizes the variable length history y 1 , . . .

Original: , y t − 1 as a decoder state s t , so that Pr( y | x 1: m ) =
Revised: , y t − 1 as a decoder state s t , so that Pr( y | x 1: m ) =

Original: (cid:81) nt =1 Pr( y t | x 1: m , s t ) .
Revised: (cid:81) nt =1 Pr( y t | x 1: m , s t ) .

Original: The distribution of each attention variable a t is computed as a function of the decoder state and encoder state as:
Revised: The distribution of each attention variable a t is computed as a function of the decoder state and encoder state as:

Original: Pr( a | x 1: m , s t ) ∝ e A θ ( x a , s t ) .
Revised: Pr( a | x 1: m , s t ) ∝ e A θ ( x a , s t ) .

Original: Here A θ ( ., . ) is an end-to-end trained function of input state x a and decoder state s t .
Revised: Here A θ ( ., . ) is an end-to-end trained function of input state x a and decoder state s t .

Original: We will use the short form P t ( a ) for Pr( a t | x 1: m , s t ) .
Revised: We will use the short form P t ( a ) for Pr( a t | x 1: m , s t ) .

Original: Thereafter, an attention weighted sum of the input states (cid:80) a P t ( a ) x a called input context c t is computed.
Revised: Thereafter, an attention weighted sum of the input states (cid:80) a P t ( a ) x a called input context c t is computed.

Original: The distribution of y t is computed from c t (capturing attention) and s t capturing previous y as:
Revised: The distribution of y t is computed from c t (capturing attention) and s t capturing previous y as:

Original: [Equation]
Revised: [Equation]

Original: Next, c t is fed to the decoder RNN along with y t for computing the next state: s t +1 =
Revised: Next, c t is fed to the decoder RNN along with y t for computing the next state: s t +1 =

Original: RNN ( s t , c t , y t ) .
Revised: RNN ( s t , c t , y t ) .

Original: Figure 1[left] summarizes the compute equations of the encoder-decoder model.
Revised: Figure 1[left] summarizes the compute equations of the encoder-decoder model.

Original: If we view Equationas an approximation of the full joint distribution in Equation 1, we ﬁnd that the treatment of the attention variables has been rather ad hoc.
Revised: If we view Equationas an approximation of the full joint distribution in Equation 1, we ﬁnd that the treatment of the attention variables has been rather ad hoc.

Original: Attention was introduced as an after-thought ofﬁrst factorizing on the y t variables, the interaction among multiple a t s is not expressed statistically, and the inﬂuence of a t on y t by diffusing the inputs is less than satisfactory.
Revised: Attention was introduced as an after-thought of factorizing on the y t variables, the interaction among various a t s is not expressed, and the inﬂuence of a t on y t by diffusing the inputs is unprincipled.

Original: We next present astatistically sounder model of the interaction of the various attention and output variables, while being more efﬁcientto compute than the full joint method of Equation 1.
Revised: We next present a sounder model of the interaction of the various attention and output variables, while being more efﬁcient than the full joint method of Equation 1.

Original: We call our proposed approach: Posterior Attention Models or PAM.
Revised: We call our proposed approach: Posterior Attention Models or PAM.

Original: 2. P OSTERIOR A TTENTION M ODELS
Revised: 2. P OSTERIOR A TTENTION M ODELS

Original: Our goal is to express the joint distribution as a product of tractable terms computed at each time step much like in existing ED model, but via a less ad hoc treatment of the attention variables a 1 , . .
Revised: Our goal is to express the joint distribution as a product of tractable terms computed at each time step much like in existing ED model, but via a less ad hoc treatment of the attention variables a 1 , . .

Original: . , a n .
Revised: . , a n .

Original: We use y <t , a <t to denote all output and attention variables before t that is, y 1 , , . . .
Revised: We use y <t , a <t to denote all output and attention variables before t that is, y 1 , , . . .

Original: y t − 1 , a 1 , . . .
Revised: y t − 1 , a 1 , . . .

Original: a t − 1 .
Revised: a t − 1 .

Original: Here and in the rest of the paper we will drop x 1: m to use the shorter form P ( y ) for Pr( y | x 1: m ) .
Revised: Here and in the rest of the paper we will drop x 1: m to use the shorter form P ( y ) for Pr( y | x 1: m ) .

Original: We ﬁrst factorize Eq 1 via chain rule, like in ED but jointly on both a and y .
Revised: We ﬁrst factorize Eq 1 via chain rule, like in ED but jointly on both a and y .

Original: [Equation]
Revised: [Equation]

Original: We then make the mild assumption that the output y t at each step is dependent only on a t and previous outputs y <t and is independent of all other attention variables.
Revised: We then make two mild assumptions: First, that the output y t at each step is dependent only on a t and previous outputs y <t and is independent of all other attention variables, i.e., P ( y t | y <t , a <t , a t ) =

Original: That is, P ( y t | y <t , a  <n , a n ) =
Revised: P ( y t | y <t , a t ) . Second, a Markovian assumption on the attention variables i.e., P ( a n | y <n , a <n ) =

Original: P ( y t | y <t , a t ) .
Revised: P ( a n | y <n , a n − 1 ) .

Original: This allows us to factorize the above joint as:
Revised: These together allows us to simplify the above joint as:

Original: [Equation]
Revised: [Equation]

Original: The last equality is after applying the same rewrite recursively on P ( y <n ) .
Revised: The last equality is after applying the same rewrite recursively on P ( y <n ) .

Original: Thus, we have expressed the joint distribution in terms of factors that apply at each decoding step t while conditioned only on previous outputs and attention.
Revised: Thus, we have expressed the joint distribution as a product of factors that apply at each decoding step t while conditioning only on previous outputs and attention.

Original: We can use the same RNN trick to summarize y <t as a ﬁxed length vector.

Original: Themain intractable termin the above is (cid:80) a <t P ( a t | a <t , y <t ) P ( a <t | y <t ) =
Revised: The term (cid:80) a t − 1 P ( a t | a t − 1 , y <t ) P ( a t − 1 | y <t ) =

Original: P ( a t | y <t ). This is  attention at step ’ t ’ conditioned on all previous outputs.
Revised: P ( a t | y <t ) is the attention at step ’ t ’ conditioned on all previous outputs.

Original: For reasons that will soon become clear we call this the prior attention at t and denote as Prior t ( a ) .
Revised: For reasons that will soon become clear we call this the prior attention at t and denote as Prior t ( a ) .

Original: In existing models, this attention is computed independently at each step using the RNN state (Eq 10) whereas we propose to expand out the expressions and more carefully capture the dependencies among attention variables. 2.2. C OMPUTATION OF A TTENTION D ISTRIBUTION We take a step beyond existing RNN-based dependence transfer and explicitly model the dependency among adjacent attention. We fall back on the decoder RNN to approximate the dependence on the attention and outputs before the immediate past.
Revised: We call P ( a t − 1 | y <t ) =

Original: This gives us: [Equation] The RNN state is an approximate summary and its update will be discussed shortly. The Prior ( a t ) becomes:
Revised: P ( a t − 1 | y <

Original: [Equation] We call P ( a t − 1 | y <t ) = P ( a t − 1 | y < ( t − 1) , y t − 1 ) as the posterior attention Postr ( a t − 1 ) since this is the attention distribution after observing the output label atthat step, and not just the previous steps as in prior attention.
Revised: ( t − 1) , y t − 1 ) as the posterior attention Postr ( a t − 1 ) since this is the attention distribution after observing the output label at the corresponding step, unlike in prior attention.

Original: We expect this attention to be more accurate than the prior that is computed without knowledge of the output token at that step.
Revised: We expect this attention to be more accurate than the prior that is computed without knowledge of the output token at that step.

Original: We compute posterior attention at any t using prior attention at t − 1 by applying Bayes rule as follows:
Revised: We compute posterior attention at any t using prior attention at t − 1 by applying Bayes rule as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The above equation gives us the important insight that the attention at step t should be computed from the posterior attention of the previous step.
Revised: The above equations give us the important result that the attention at step t should be computed from the posterior attention of the previous step.

Original: Intuitively, also it makes sense because attention reﬂects an alignment of the input and output, and its distribution will improve if the output is known.
Revised: Intuitively, also it makes sense because attention reﬂects an alignment of the input and output, and its distribution will improve if the output is known.

Original: For an exact method to compute the P ( a  t | a t − 1 , y t − 1 , s t− 1 ) term in the equation of prior attention, we would need to design an attention logic with four arguments: x a t ,x a t − 1 , y t− 1 , and s n − 1 .
Revised: We get into details of computing such coupled attention in Section 2.2.1. 2.2. C OMPUTATION OF P RIOR A TTENTION D ISTRIBUTION We use the RNN to summarize y <t as a ﬁxed length vector s t as in current ED models. We then discuss three different methods we explored for computing (cid:80) a t − 1 P ( a t | s t , a t − 1 , y t

Original: In contrast, existing models compute attention logits
Revised: [Equation]

Original: A θ (x a, s t  ) with only two arguments.
Revised: Postr t − 1 ( a t − 1 ) .

Original: To avoid the extra attention parameters, we designed three light-weight methods of capturing these dependencies. In all these variants we introduce only a handful of extra parameters which can be learned end-to-end.
Revised: Our methods are designed to be light-weight in terms of the number of extra parameters they consume beyond the default soft-attention methods to have the fairest comparison.

Original: Postr-Joint The simplest of these uses the same decoder RNN to absorb the posterior attention of the previous step.
Revised: Postr-Joint The simplest of these uses the same decoder RNN to absorb the posterior attention of the previous step.

Original: Essentially, we used deterministic attention technique used by Xu et al. (2015) to efﬁciently approximate computation of Prior t ( a ) using ﬁrst order Taylor expansion.
Revised: We linearize the function using the ﬁrst order Taylor expansion to efﬁciently approximate computation of Prior t ( a ) similar to the deterministic technique of Xu et al.

Original: Prior
Revised: Prior

Original: [Equation]
Revised: [Equation]

Original: The above equation suggests that the decoder RNN state should be updated as s t =
Revised: The above equation suggests that the decoder RNN state should be updated as s t =

Original: RNN( s t − 1 , [Equation] y t
Revised: RNN

Original: [Equation]
Revised: [Equation]

Original: The computation here is thus similar to existing ED model’s but the crucial difference is that the context used to update the RNN is computed from posterior attention, and not the prior attention.
Revised: The computation here is thus similar to existing ED model’s but the crucial difference is that the context used to update the RNN is computed from posterior attention, and not the prior attention.

Original: We will see that this gives rise to signiﬁcant improvement in accuracy.
Revised: We will see that this leads to large improvement in accuracy.

Original:  Next we experiment with models that explicitly couple adjacent attention.
Revised: Proximity biased Next we experiment with models that explicitly couple adjacent attention.

Original: These models utilize a separate index based coupling between attention positionsand are of the form
Revised: These models utilize an index based coupling between attention positions of the form

Original: [Equation]
Revised: [Equation]

Original:  A θ ( x a t , s t− 1 ) is the attention logit computed from the previous RNN step and k ( a t , a t − 1 ) is the attention coupling energy. We experiment with two types of coupling energies:
Revised: where A θ ( x a t , s t ) is the attention logit computed from the previous RNN step, k ( a t , a t − 1 ) is the attention coupling energy and Z is the normalization constant.

Original: Proximity biased coupling  k ( a t , a t − 1 ) is given by I ( | a t − a t − 1 | < 3) δ a t − a t − 1 .
Revised: In the proximity based attention the coupling energy k ( a t , a t − 1 ) is given by I ( | a t − a t − 1 | < 3) δ a t − a t − 1 .

Original: This model has a natural bias towards attending on inputs where attention has focused recently.
Revised: This model provides a greater focus on attending states within a window of size ﬁve centered around the recently attended input state.

Original: We label this model as Prox-Postr-Joint in our results below
Revised: We label this model as Prox-Postr-Joint in our experiments.

Original: Monotonicity biased coupling  k ( a t , a t − 1 ) is a monotonic energy given by I ( a t > a t − 1 ) δ a t − a t − 1 − 1 .
Revised: This method differs from the above proximity-based attention only in how it deﬁnes the coupling energy k ( a t , a t − 1 ) . As the name implies, in this method k ( a t , a t − 1 ) is a monotonic energy given by I ( a t > a t − 1 ) δ a t − a t − 1 − 1 .

Original: This model biases attention towards a monotonic attention which keeps moving ahead.
Revised: This model provides a positive exponentially [Equation] [Equation] [Equation]

Revised: (cid:80)

Original: As we shall see tasks with natural monotonic attention beneﬁt from this form of bias.
Revised: [Equation] Pr( y | x 1: m ) = n (cid:89) t = m (cid:88) a = P ( y t | s t , x a ) Prior t ( a ) (10) s t +1 =

Revised: RNN ( s t , y t , (cid:88)

Revised: a

Revised: Postr t ( a ) x a ) (11)

Revised: Postr t ( a ) =

Revised: P ( y t | s t , x a ) Prior t ( a ) (cid:80) a (cid:48) P ( y t | s t , x a (cid:48) )

Revised: Prior t ( a (cid:48) ) (12)

Revised: Prior t ( a t ) =

Original: This model is denoted as Mono-Postr-Joint in our experiments.
Revised: (cid:88) a (cid:48) P ( a t | s t − 1 , a (cid:48) ) Postr t − 1 ( a (cid:48) ) (13) P ( a t | s t − 1 , a (cid:48) )

Revised: = See Section 2.2.

Original: 2. P UTTING IT ALL TOGETHER
Revised: 2. P UTTING IT ALL TOGETHER

Original: In Figure 1 we put together the ﬁnal set of equations that are used to compute the output distribution and contrast with existing attention model.
Revised: In Figure 1 we put together the ﬁnal set of equations that are used to compute the output distribution and contrast with existing attention model.

Original: We call this overall architecture as Posterior Attention
Revised: We call this overall architecture as Posterior Attention

Original: Model (PAM).
Revised: Model (PAM).

Original: First note that in PAM, we explicitly compute a joint distribution of output and attention at each step and marginalize out the attention.
Revised: First note that in PAM, we explicitly compute a joint distribution of output and attention at each step and marginalize out the attention.

Original: Thus, the output is a mixture of multiple output distributions each of which is a function of one focused input (like in hard attention), and not a diffused sum of the input (like in soft attention).
Revised: Thus, the output is a mixture of multiple output distributions each of which is a function of one focused input (like in hard attention), and not a diffused sum of the input (like in soft attention).

Original: This difference in the way attention is marginalized is not only statistically sound, but also leads to higher accuracy.
Revised: This difference in the way attention is marginalized is not only statistically sound, but also leads to higher accuracy.

Original: The only downside of the joint model is that we need to compute m softmaxes for each output y t , and this may be impractical when the vocabulary size is large.
Revised: The only downside of the joint model is that we need to compute m softmaxes for each output y t , and this may be impractical when the vocabulary size is large.

Original: A simple and effective ﬁx to this is to select the Top-K attentions based on
Revised: A simple and effective ﬁx to this is to select the top-K attentions based on

Original: Prior t and compute the ﬁnal output distribution as.
Revised: Prior t and compute the ﬁnal output distribution as.

Original: [Equation]
Revised: [Equation]

Original: Small values of K (order 6), sufﬁce to provide good performance1 .
Revised: Small values of K (order 6), sufﬁce to provide good performance

Original: The second difference is that the attention distribution that is propagated to the next step is posterior to observing the current output.
Revised: The second difference is that the attention distribution that is propagated to the next step is posterior to observing the current output.

Original: We derived this from a principled rewrite of the joint distribution, and were pleasantly surprised to see signiﬁcant accuracy gains by this subtle difference in the way the decoder state is updated.
Revised: We derived this from a principled rewrite of the joint distribution, and were pleasantly surprised to see signiﬁcant accuracy gains by this subtle difference in the way the decoder state is updated.

Original: Computing the posterior attention does not incur any additional overheads because the joint attention-output distribution was already materialized in the ﬁrst equation.
Revised: Computing the posterior attention does not incur any additional overheads because the joint attention-output distribution was already materialized in the ﬁrst equation.

Original: However, due to the sparsity induced by the top-k operation on attention probabilities, the posterior probabilities are unrealistically sparse.
Revised: However, due to the sparsity induced by the top-K operation on attention probabilities, the posterior probabilities are unrealistically sparse.

Original: As such we augment the posterior attention using input from standard attention, by using a equally weighted combination of the two distributions.
Revised: As such we augment the posterior attention using input from standard attention, by using an equally weighted combination of the two distributions.

Original: Third, the prior attention distribution is explicitly conditioned on the previous attention.
Revised: Third, the prior attention distribution is explicitly conditioned on the previous attention.

Original: This allowed us to incorporate various application-speciﬁc natural biases like proximity and monotonicity of adjacent attentions.
Revised: This allowed us to incorporate various application-speciﬁc natural biases like proximity and monotonicity of adjacent attentions.

Original:  R ELATED W ORK
Revised: 2. A LTERNATIVE R EWRITES

Revised: Our rewrite although somewhat unconventional was derived to satisfy two important goals: First, explain the need to propagate posterior attention to subsequent steps. Second, to factorize the joint as the product of the local distribution at each time t which allows efﬁcient gradient updates and minimal changes to existing beam-search inference. A more conventional rewrite for handling Markovian dependencies is the standard forward algorithm which works as follows.

Revised: First we write:

Revised: [Equation]

Revised: Then use the forward algorithm to compute:

Revised: [Equation]

Revised: which then gives the joint distribution as P ( y )

Revised: = (cid:80) a α n ( a ) .

Revised: This expression is neither in the outer product form, nor does it motivate the need for posterior attention.

Revised:  R ELATED W ORK

Original: The de facto standard for sequence to sequence learning via neural networks is the encoder decoder model.
Revised: The de facto standard for sequence to sequence learning via neural networks is the encoder decoder model.

Original: Ever since their ﬁrst introduction in Bahdanau et al.
Revised: Ever since their ﬁrst introduction in Bahdanau et al.

Original: (2014), many different attention models have been proposed.
Revised: (2014), many different attention models have been proposed.

Original: We discuss them here.
Revised: We discuss them here.

Original: Soft Attention is the de-facto mechanism for seq2seq learning et al (2018).
Revised: Soft Attention is the de-facto mechanism for seq2seq learning et al (2018).

Original: It was proposed for translation in Bahdanau et al.
Revised: It was proposed for translation in Bahdanau et al.

Original: and reﬁned further in Luong et al. (2015).
Revised: and reﬁned further in Luong et al. (2015).

Original: The output derives from an attention averaged context.
Revised: The output derives from an attention averaged context.

Original: The advantage is end to end differentiability.
Revised: The advantage is end to end differentiability.

Original: Pr

Original: [Equation]

Original: [Equation]

Original: = n

Original: [Equation]

Original: t =

Original: Pr(

Original: [Equation]

Original: t |

Original: [Equation]

Original: t , m (cid:88)

Original: a

Original: [Equation]

Original: t ( a [Equation] 8) s t [Equation]
Revised: Hard Attention was proposed in Xu et al.

Original: = RNN

Original: [Equation]

Original: t

Original: [Equation]

Original: t

Original: [Equation]

Original: a

Original: P t ( a

Original: [Equation]

Original: P t [Equation] = e A θ ( x a [Equation]
Revised: and attends to exactly one input state for an output.

Original: t ) (cid:80) mr =

Original: [Equation]

Original: e

Original: [Equation]

Original: θ ( x r

Original: [Equation]

Original: t ) (10)

Original: Pr( y | x 1: m ) = n (cid:89) t

Original: = m (cid:88) a =

Original: P ( y t | s t , x j ) Prior t ( a ) (11) s t +1 =

Original: RNN ( s t , y t , (cid:88)

Original: a

Original: Postr t ( a ) x a ) (12)

Original: Postr t ( a ) =

Original: P ( y t | s t , x a ) Prior t ( a ) (cid:80) a (cid:48) P ( y t | s t , x a (cid:48) )

Original: Prior t ( a (cid:48) ) (13)

Original: Prior t ( a t ) =

Original: (cid:88) a (cid:48)

Original: P ( a t | s t − 1 , a (cid:48) )

Original: Postr t − 1 ( a (cid:48) ) (14)

Original: P ( a t | s t − 1 , a (cid:48) )

Original: = See Section 2.2.

Original: Hard Attention was proposed

Original: The merit of hard attention is that the output is determined from a single input rather than an average of all inputs.
Revised: The merit of hard attention is that the output is determined from a single input rather than an average of all inputs.

Original: However due to non-differentiability , training Hard-Attention requires the REINFORCE
Revised: However due to non-differentiability, training Hard-Attention requires the REINFORCE

Original: Williams (1992) algorithm and is subject to high variance, requiring careful tricks to train reliably.
Revised: Williams (1992) algorithm and is subject to high variance, requiring careful tricks to train reliably.

Original: Yu et al.
Revised: Yu et al.

Original: (2016) keep the encoder and decoder independent to allow for easier marginalization.
Revised: (2016) keep the encoder and decoder independent to allow for easier marginalization.

Original: Aharoni &
Revised: Aharoni &

Original: Goldberg (2017) use a monotonic hard attention and avoid the problem, by supervising hard attention with external alignment information.
Revised: Goldberg (2017) use a monotonic hard attention and avoid the problem, by supervising hard attention with external alignment information.

Revised: Our model in equation 10 uses hard attention on the encoder states.

Revised: However unlike standard hard attention we do not use a one-hot attention and instead are computing the exact marginalization.

Original: Sparse/Local Attention
Revised: Sparse/Local Attention

Original: Many attempts have been made to bridge the gap between soft and hard attention.
Revised: Many attempts have been made to bridge the gap between soft and hard attention.

Original: Luong et al.
Revised: Luong et al.

Original: (2015) proposes local attention that averages a window of input.
Revised: (2015) proposes local attention that averages a window of input.

Original: This has been reﬁned later to include syntax (Chen et al., 2017; Sennrich & Haddow, 2016; Chen et al., 2018) and has been explored for image captioning in Gregor et al.
Revised: This has been reﬁned later to include syntax (Chen et al., 2017; Sennrich & Haddow, 2016; Chen et al., 2018) and has been explored for image captioning in Gregor et al.

Original: A related idea to harden attention is to make it sparse using sparsity inducing operators (Martins & Astudillo, 2016; Niculae & Blondel, 2017).
Revised: A related idea to hard attention is to make it sparse using sparsity inducing operators (Martins & Astudillo, 2016; Niculae & Blondel, 2017).

Original: However, all sparse/local attention methods continue to compute P ( y ) from an attention weighted sum of inputs like in soft attention.
Revised: However, all sparse/local attention methods continue to compute P ( y ) from an attention weighted sum of inputs like in soft attention.

Original: Recurrent Attention Yang et al.
Revised: Recurrent Attention Yang et al.

Original: (2016) have previously modeled relationship between the attentions at different time steps by using a recurrent history mechanism.
Revised: (2016) have previously modeled relationship between the attentions at different time steps by using a recurrent history mechanism.

Original: The attention history of an input word and its surrounding words are captured in a summary vector by an RNN, which is provided as further input to the attention mechanism for incorporating dependence on history.
Revised: The attention history of an input word and its surrounding words are captured in a summary vector by an RNN, which is provided as further input to the attention mechanism for incorporating dependence on history.

Original: While both works model dependence between attention at different steps, our principled rewrite of the joint distribution shows that posterior attention should be the link to the next attention.
Revised: While both works model dependence between attention at different steps, our principled rewrite of the joint distribution shows that posterior attention should be the link to the next attention.

Original: Structured Attention Networks Similar to this work, Kim et al.
Revised: Structured Attention Networks Similar to this work, Kim et al.

Original: (2017) interpret attention as latent structural variable.
Revised: (2017) interpret attention as latent structural variable.

Original: The authors then take advantage of easy inference in certain graphical models to implement forms of segmental and syntactic attention.
Revised: The authors then take advantage of easy inference in certain graphical models to implement forms of segmental and syntactic attention.

Original: Liu & Lapata (2018) extend the same technique to non-projective dependencies in document modeling.

Original: However these work only focus on attention at each step independently whereas our focus is modeling the dependency among adjacent attention.
Revised: These works only focus on attention at each step independently whereas our focus is modeling the dependency among adjacent attention.

Original: Moreover our posterior attention framework is independent of how the prior attention at each position is modeled.
Revised: Moreover our posterior attention framework is independent of how the prior attention at each position is modeled.

Original: In this paper we assumed a multinomial distribution but the structured distribution of Kim et al.
Revised: In this paper we assumed a multinomial distribution but the structured distribution of Kim et al.

Original: (2017) can also beneﬁt from our posterior coupling.
Revised: (2017) can also beneﬁt from our posterior coupling.

Original: Variational Posterior Attention Schulz et al.
Revised: Latent Attention Models There has been a number of recent works which model attention as latentalignments in (Wu et al., 2018; Deng et al., 2018; Shankar et al., 2018). Our model can be considered as a generalization of the work of Wang et al.

Original: (2018)deploy a stochastic decoder based on chaining multiple latent variables, and use a variational approach totrain the model.
Revised: (2018) to the case where attention is also provided to the RNN. The model in Shankar et al.

Original: The motivation behind the  modelis the existence of several ﬂuent translations for a sentence which can differ in their syntactic forms.
Revised: and related input-feeding model of Wu et al. (2018) also factorize the joint distribution and are identical to our Prior-Joint model. However unlike these models we explicitly model the posterior attention distribution and attention coupling. Deng et al.

Original: Bahuleyan et al.
Revised: (2018) proposes to learn the posterior attention via variational methods. A key difference is while their model tries to supervise attention using a posterior inference network via a KL term, we directly use the actual posterior for computing attention in the subsequent steps. Their method requires training a separate inference network to model the posterior whereas we require no additional network parameters.

Original: and Zhou & Neubig (2017) proposes variational attention but attention is still a function of x. Their goal is also to increase diversity. Our goal is to improve performance by relying on the posterior.
Revised: In our method, posterior attention is used in identical roles across training and inference, unlike in Deng et al. (2018)’s that rely on variational training.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We compare our posterior attention model on two sequence to sequence learning tasks: machine translation and morphological inﬂection.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: Methods compared
Revised: [Table] [Table] [Table] [Table]

Original: Soft: This is the standard soft attention mechanism with Luong attention.
Revised: [Table]

Original: Sparse: This is the sparse-attention model presented in Niculae & Blondel (2017).
Revised: [Table] [Table] [Table]

Original: Postr-Joint: This is ourdefault posterior attention network as described in 2.2.1 . We try two more variants of PAMbased on explicit coupling called Mono-Postr-Joint: which refers to  the monotonic biased model in 2.2.
Revised: We compare our posterior attention model on two sequence to sequence learning tasks: machine translation and morphological inﬂection. We compare on the following methods:

Original: Prox-Postr-Joint: which refers to the explicitly coupled proximity biased model described in 2.2. Prior-Joint: This is our model minus the posterior attention.
Revised: [Table]

Original: That is, we use joint attention output distribution as in Eq 11 but prior and RNN updates are as in soft attention. 4. M ACHINE TRANSLATION
Revised: : attention Luong et al. (2015). Sparse: attention Niculae & Blondel (2017).

Original: [Table]
Revised: [Table]

Original: We experiment on ﬁve language pairs from three datasets:
Revised: : Our [Table] [Table]

Original: [Table]
Revised: [Table]

Original: and
Revised: -Joint: monotonic PAM (§ [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: biased

Revised: PAM(§ 2.2.1)

Revised: Prior-Joint: supresses the dependence on previous attention analogous to current ED models.

Revised: This reduces the model to that proposed in

Original: Nakazawa et al. 
Revised: Shankar et al. ; Wu et al. Variational: method of Deng et al. 4. M ACHINE TRANSLATION

Revised: We experiment on ﬁve language pairs from three datasets:

Revised: IWSLT

Revised: English ↔ Vietnamese ,

Revised: IWSLT

Revised: German ↔ English

Revised: Cettolo et al.

Revised: (2015); and WAT17 Japanese → English

Revised: Nakazawa et al.

Original: We use a 2 layer bi-directional encoder and 2 layer decoder with 512 LSTM units and 0.2 dropout with vanilla SGD optimizer.
Revised: We use a 2 layer bi-directional encoder and 2 layer decoder with 512 LSTM units and 0.2 dropout with vanilla SGD optimizer.

Revised: We use word level encoding for all translation tasks.

Original: Instead of tuning hyper-parameters for our method we used Soft-Attention tuned parameters.
Revised: Instead of tuning hyper-parameters for our method we used Soft-Attention tuned parameters.

Original: Overall Comparison Our results are in Table 1 where we show perplexity (PPL) and BLEU with beam size 4 and 10.
Revised: Our results are in Table 1 where we show perplexity (PPL) and BLEU with beam size 4 and 10.

Original: All Postr-Joint variants and Prior-Joint outperform soft attention and sparse-attention by large margins.
Revised: All Postr-Joint variants and Prior-Joint outperform soft attention and sparse-attention by large margins.

Original: Moreover models with posterior attention show improvement over those which use prior attention.
Revised: Moreover models with posterior attention show improvement over those which use prior attention.

Revised: We also observe small improvements over the more sophisticated variational attention model likely due to the use of the exact posterior during inference instead.

Original:  This clearly shows the performance advantage of joint modeling and posterior attention.
Revised: 1 This clearly shows the performance advantage of joint modeling and posterior attention.

Original: We shall analyze the reasons for these improvements later.
Revised: We shall analyze the reasons for these improvements later.

Original: Comparing Attention Coupling Next we explore the impact of different coupling models discussed in 2.2.1.
Revised: Next we explore the impact of different coupling models discussed in 2.2.1.

Original: For that focus on methods Postr-Joint, Prox-Postr-Joint, and Mono-Postr-Joint in Table 1.
Revised: For that focus on methods Postr-Joint, Prox-Postr-Joint, and Mono-Postr-Joint in Table 1.

Original: We obtain some gains over Postr-Joint by explicitly modeling attention coupling.
Revised: We obtain some gains over Postr-Joint by explicitly modeling attention coupling.

Original: For language-pairs with a natural monotonic alignment like GermanEnglish, Mono-Postr-Joint slightly outperforms other models by (0.1-0.2 BLEU points).
Revised: For language-pairs with a natural monotonic alignment like German-English, Mono-Postr-Joint slightly outperforms other models by (0.1-0. BLEU points).

Original: EnglishVietnamese is a more non-monotonic pair and as expected we do not ﬁnd gains by incorporating a monotonic bias.
Revised: English-Vietnamese is a more non-monotonic pair and as expected we do not ﬁnd gains by incorporating a monotonic bias.

Original: 4. M ORPHOLOGICAL I NFLECTION
Revised: 4. M ORPHOLOGICAL I NFLECTION

Original: To demonstrate the use of our model beyond translation, we next consider the task of generating morphological inﬂections.
Revised: To demonstrate the use of our model beyond translation, we next consider the task of generating morphological inﬂections.

Original: We useDurrett & DeNero (2013)’s dataset containing inﬂection forms for German Nouns (de-N) and German Verbs (de-V).
Revised: We use inﬂection forms for German Nouns (de-N) and German Verbs (de-V) from Durrett & DeNero (2013)’s.

Original: The models are trained separately for each type of inﬂectionfor each dataset to predict the inﬂected character sequence.
Revised: A model is trained separately for each type of inﬂection to predict the inﬂected character sequence.

Original: We train a one layer encoder and decoder with 128 hidden LSTM units each with a dropout rate of 0.2 using Adam and measure 0/1 accuracy.
Revised: We train a one layer encoder and decoder with 128 hidden LSTM units each with a dropout rate of 0.2 using Adam and measure 0/1 accuracy.

Original: We also ran the 100 units wide two layer LSTM with hard-monotonic attention model Aharoni & Goldberg (2017) labeled Hard-Mono 2 .
Revised: We also ran the 100 units wide two layer LSTM with hard-monotonic attention model Aharoni & Goldberg (2017) labeled Hard-Mono 2 .

Original: [Table]
Revised: [Table]

Original: Using joint modeling we get signiﬁcant gains (0.3 points) even against task-speciﬁc hard-monotonic attention, showing that our approach is more general than translation.
Revised: Using joint modeling we get signiﬁcant gains (0.3 points) even against task-speciﬁc hard-monotonic attention, showing that our approach is more general than translation.

Original: Moreover when we use
Revised: Moreover when we use

Original: Mono-Postr-Joint which has a structural bias towards task-speciﬁc monotonic attention, we obtain immense improvements (upto 1 accuracy point) over joint models.
Revised: Mono-Postr-Joint which has a structural bias towards task-speciﬁc monotonic attention, we obtain immense improvements (upto 1 accuracy point) over joint models.

Original: 4. E XPLAINING WHY WE SCORE ABOVE S OFT A TTENTION
Revised: 4. E XPLAINING WHY WE SCORE ABOVE S OFT A TTENTION

Original: We attempt to get more insights on why posterior attention models score over soft attention in end to end accuracy.
Revised: We attempt to get more insights on why posterior attention models score over soft attention in end to end accuracy.

Original: We show that the main reason is better alignment of input and output because of a more precise attention model.
Revised: We show that the main reason is better alignment of input and output because of a more precise attention model.

Original: We demonstrate that by ﬁrst showing some anecdotes of better alignment, then showing that posterior attention is more focused (has lower entropy), provides better alignment accuracy, and better input coverage.
Revised: We demonstrate that by ﬁrst showing some anecdotes of better alignment, then showing that posterior attention is more focused (has lower entropy), provides better alignment accuracy, and better input coverage.

Original: For these runs we perform experiments in the teacher forcing setup so as to compare two models’ distributions under identical inputs.
Revised: For these runs we perform experiments in the teacher forcing setup so as to compare two models’ distributions under identical inputs.

Original: Anecdotal Examples Fig2 presents the heatmap of difference between Postr-Joint and SoftAttention on some representative sentences.
Revised: Anecdotal Examples Fig2 presents the heatmap of difference between Postr-Joint and SoftAttention on two different sentences.

Original: Thus the red regions represent where Postr-Joint has greater attention and blue where soft-attention has greater focus.
Revised: In each ﬁgure the red regions represents where Postr-Joint has greater attention and blue where soft-attention has greater focus.

Original: One can observegenerally that Soft-Attention is far more diffused.
Revised: One can observe that Soft-Attention is far more diffused.

Original: More importantly, we can see that Postr-Joint is able to correct mistakes and provides the appropriate context for the next step.
Revised: More importantly, we can see that Postr-Joint is able to correct mistakes and provides the appropriate context for the next step.

Original: For example in Fig2a Soft-Attention (blue) has maximum focus on the source word ’generationen’ when the target word is innovation which corresponds to ’innovationen’; on the other hand Postr-Joint is able to correct this.
Revised: For example in Fig2a Soft-Attention (blue) has maximum focus on the source word ’generationen’ when the target word is innovation which corresponds to ’innovationen’; on the other hand Postr-Joint is able to correct this.

Original: Similarly while producing the phrase ’but the same’ Postr-Joint focuses the attention on the source word ’dasselbe’ Fig2b.
Revised: Similarly while producing the phrase ’but the same’ Postr-Joint focuses the attention on the source word ’dasselbe’ Fig2b.

Original: This provides insight into as to how by providing better contexts via incorporating the target, posterior attention can outperform prior attention.
Revised: This provides insight into as to how by providing better contexts via incorporating the target, posterior attention can outperform prior attention.

Original: Attention Entropy Vs Accuracy We expect Soft-Attn to be worse hit by high attention uncertainty than other models.
Revised: Attention Entropy Vs Accuracy We expect Soft-Attn to be worse hit by high attention uncertainty than other models.

Original: This, if true, could illustrate that P ( y t | x t ) distribution can be learned more easilyif the input is ’pure’, rather than diffused via pre-aggregation. To this end we plot the accuracy of
Revised: This, if true, could illustrate that P ( y t | x t ) distribution can be learned more easily

Original: Postr-Joint, Prior-Joint and Soft-Attn under increasing attention entropy in Figure 3 on the EnglishGerman pair.

Original: As one can expect the accuracy drops off quickly as attention uncertainty rises.

Original: The plot also presents the histogram of the fraction of cases with different attention uncertainties.

Original: Soft attention models (blue) have signiﬁcantly higher number of cases of high attention uncertainty, leading to low performance.

Original: One of the primary means by which joint models outperformed soft-attention is by

Original: [Figure]
Revised: [Figure]

Original: Alignment accuracy Failure of attention to produce latent structures which correspond to linguistic structures has been noted by Koehn & Knowles (2017); Ghader & Monz (2017).Based on few examples, we hypothesize that Posterior Attention should be able to produce better alignments.
Revised: Alignment accuracy Failure of attention to produce latent structures which correspond to linguistic structures has been noted by Koehn & Knowles (2017); Ghader & Monz (2017).Based on few examples, we hypothesize that Posterior Attention should be able to produce better alignments.

Original: To test this we used the RWTH German-English dataset which provides alignment information manually tagged by experts, and compare the alignment accuracy for Soft, Prior-Joint and Postr-Joint attentions.
Revised: To test this we used the RWTH German-English dataset which provides alignment information manually tagged by experts, and compare the alignment accuracy for Soft, Prior-Joint and Postr-Joint attentions.

Original: Following the procedure in Ghader & Monz (2017) the most attended source word for each target word is taken as the aligned word.
Revised: Following the procedure in Ghader & Monz (2017) the most attended source word for each target word is taken as the aligned word.

Original: We used the AER metric Koehn (2010) to compare these against the expert alignments.
Revised: We used the AER metric Koehn (2010) to compare these against the expert alignments.

Original: Table 3 presents the AER accuracy for different models.
Revised: Table 3 presents the AER accuracy for different models.

Original: One can read off that Postr-Joint model beats the second best model ( Prior-Joint ) by more than 10%, and dwarfs soft-attention by a huge margin, proving that posterior alignments are signiﬁcantly more compatible with true alignments.
Revised: One can read off that Postr-Joint model beats the second best model ( Prior-Joint ) by more than 10%, and dwarfs soft-attention by a huge margin, proving that posterior alignments are signiﬁcantly more compatible with true alignments.

Original: [Table]
Revised: [Table]

Original: Fraction of covered tokens A natural expectation for translation is that by the time the entire output sentence has been produced, attention would have covered the entire input sequence.
Revised: Fraction of covered tokens A natural expectation for translation is that by the time the entire output sentence has been produced, attention would have covered the entire input sequence.

Original: A loss based on this precise heuristic was used in Chorowski & Jaitly (2016) to improve the performance of a attention based seq2seq model for speech transcription.
Revised: A loss based on this precise heuristic was used in Chorowski & Jaitly (2016) to improve the performance of a attention based seq2seq model for speech transcription.

Original: In this experiment we try to indirectly assess reliability of different attention models via measuring whether cumulatively attention has focused on the entire input sequence.
Revised: In this experiment we try to indirectly assess reliability of different attention models via measuring whether cumulatively attention has focused on the entire input sequence.

Original: We plot the frequency distribution of the coverage in Fig4.
Revised: We plot the frequency distribution of the coverage in Fig4.

Original: Note that in soft attention model, there are many sentences which do not receive enough attention during the entire decoding process.
Revised: Note that in soft attention model, there are many sentences which do not receive enough attention during the entire decoding process.

Original: Prior-Joint and Postr-Joint have similar behavior with few instances of one outperforming the other, however both outperform soft attention by huge margins.
Revised: Prior-Joint and Postr-Joint have similar behavior with few instances of one outperforming the other, however both outperform soft attention by huge margins.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We show in this paper that none of the existing attention models adequately model the dependence of the output and attention along the length of the output for general sequence prediction tasks.
Revised: We show in this paper that none of the existing attention models adequately model the dependence of the output and attention along the length of the output for general sequence prediction tasks.

Original: We propose a factorization of the joint distribution, and develop practical approximations that allows  the joint distribution to decompose over output tokens, much like in existing attention.
Revised: We propose a factorization of the joint distribution, and develop practical approximations that allows [Figure] the joint distribution to decompose over output tokens, much like in existing attention.

Original: Our more principled probabilistic joint modeling of the dependency structure leads to three important differences.
Revised: Our more principled probabilistic joint modeling of the dependency structure leads to three important differences.

Original: First, the output token distribution is obtained by aggregating predictions across all attention.
Revised: First, the output token distribution is obtained by aggregating predictions across all attention.

Original: Second, the concept of conditioning attention on the current output i.e. a posterior attention for inferring the next output becomes important.
Revised: Second, the concept of conditioning attention on the current output i.e. a posterior attention for inferring the next output becomes important.

Original: Our experiments show that it is sounder, more meaningful and more accurate to condition subsequent attention distribution on the posterior attention.
Revised: Our experiments show that it is sounder, more meaningful and more accurate to condition subsequent attention distribution on the posterior attention.

Original: Thirdly, via[Figure] directly exposing attention coupling, we have a principled way to directly incorporate task-speciﬁc structural biases and prior knowledge into attention.
Revised: Thirdly, via directly exposing attention coupling, we have a principled way to directly incorporate task-speciﬁc structural biases and prior knowledge into attention.

Original: We experimented with some simple biases and found boosts in related tasks.
Revised: We experimented with some simple biases and found boosts in related tasks.

Original: Our work opens avenues for future work in scaling these techniques to large-scale models and multi-headed attention.
Revised: Our work opens avenues for future work in scaling these techniques to large-scale models and multi-headed attention.

Original: Another promising line is to incorporate more complex biases like phrasal structure or image segments into joint attention models.
Revised: Another promising line is to incorporate more complex biases like phrasal structure or image segments into joint attention models.

Original: A PPENDIX
Revised: A PPENDIX

Original: d a ss e l b e g il t a ll e r d i n g s a u c h f ü
Revised: d a ss e l b e g il t a ll e r d i n g s a u c h f ü

Original: r e r n e u e r b a r e e n e r
Revised: r e r n e u e r b a r e e n e r

Original: g i e n .
Revised: g i e n .

Original: but the same goes for renewables as well .
Revised: but the same goes for renewables as well .

Original: (a) d a ss e l b e g il t a ll e r d
Revised: (a) d a ss e l b e g il t a ll e r d

Original: i n g s a u c h f ü r e r n e u e r b a r e e n e r g i e n .
Revised: i n g s a u c h f ü r e r n e u e r b a r e e n e r g i e n .

Original: [Figure]
Revised: [Figure]

Original: as well .
Revised: as well .

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: i nn o
Revised: i nn o

Original: [Figure]
Revised: [Figure]

Original: a t i
Revised: a t i

Original: [Figure]
Revised: [Figure]

Original: n e n
Revised: n e n

Original: [Figure]
Revised: [Figure]

Original: o
Revised: o

Original: [Figure]
Revised: [Figure]

Original: g
Revised: g

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: r a t i
Revised: r a t i

Original: [Figure]
Revised: [Figure]

Original: i nn
Revised: i nn

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: d)
Revised: d)