Original: M ONOTONIC M ULTIHEAD A TTENTION
Revised: M ONOTONIC M ULTIHEAD A TTENTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence.
Revised: Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence.

Original: Recent approaches for this task either apply a ﬁxed policy on a state-of-the art Transformer model, or a learnable monotonic attention on a weaker recurrent neural network-based structure.
Revised: Recent approaches for this task either apply a ﬁxed policy on a state-of-the art Transformer model, or a learnable monotonic attention on a weaker recurrent neural network-based structure.

Original: In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead attention.
Revised: In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead attention.

Original: We also introduce two novel and interpretable approaches for latency control that are speciﬁcally designed for multiple attention heads.
Revised: We also introduce two novel and interpretable approaches for latency control that are speciﬁcally designed for multiple attention heads.

Original: We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach.
Revised: We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach.

Original: We analyze how the latency controls affect the attention span and we study the relationship between the speed of a head and the layer it belongs to.
Revised: We analyze how the latency controls affect the attention span and we study the relationship between the speed of a head and the layer it belongs to.

Original: Finally, we motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency.
Revised: Finally, we motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Simultaneous machine translation adds the capability of a live interpreter to machine translation: a simultaneous machine translation model starts generating a translation before it has ﬁnished reading the entire source sentence.
Revised: Simultaneous machine translation adds the capability of a live interpreter to machine translation: a simultaneous machine translation model starts generating a translation before it has ﬁnished reading the entire source sentence.

Original: Such models are useful in any situation where translation needs to be done in real time.
Revised: Such models are useful in any situation where translation needs to be done in real time.

Original: For example, simultaneous models can translate live video captions or facilitate conversations between people speaking different languages.
Revised: For example, simultaneous models can translate live video captions or facilitate conversations between people speaking different languages.

Original: In a usual neural machine translation model, the encoder ﬁrst reads the entire sentence, and then the decoder writes the target sentence.
Revised: In a usual neural machine translation model, the encoder ﬁrst reads the entire sentence, and then the decoder writes the target sentence.

Original: On the other hand, a simultaneous neural machine translation model alternates between reading the input and writing the output using either a ﬁxed or learned policy.
Revised: On the other hand, a simultaneous neural machine translation model alternates between reading the input and writing the output using either a ﬁxed or learned policy.

Original: Monotonic attention mechanisms fall into the learned policy category.
Revised: Monotonic attention mechanisms fall into the learned policy category.

Original: Recent work exploring monotonic attention variants for simultaneous translation include: hard monotonic attention (Raffel et al., 2017), monotonic chunkwise attention (MoChA) (Chiu & Raffel, 2018) and monotonic inﬁnite lookback attention (MILk) (Arivazhagan et al., 2019).
Revised: Recent work exploring monotonic attention variants for simultaneous translation include: hard monotonic attention (Raffel et al., 2017), monotonic chunkwise attention (MoChA) (Chiu & Raffel, 2018) and monotonic inﬁnite lookback attention (MILk) (Arivazhagan et al., 2019).

Original: MILk in particular has shown better quality / latency trade-offs than ﬁxed policy approaches, such as wait- k (Ma et al., 2019) or wait-if-* (Cho &
Revised: MILk in particular has shown better quality / latency trade-offs than ﬁxed policy approaches, such as wait- k (Ma et al., 2019) or wait-if-* (Cho &

Original: Esipova, 2016) policies.
Revised: Esipova, 2016) policies.

Original: MILk also outperforms hard monotonic attention and MoChA; while the other two monotonic attention mechanisms only consider a ﬁxed reading window, MILk computes a softmax attention over all previous encoder states, which may be the key to its improved latencyquality tradeoffs.
Revised: MILk also outperforms hard monotonic attention and MoChA; while the other two monotonic attention mechanisms only consider a ﬁxed reading window, MILk computes a softmax attention over all previous encoder states, which may be the key to its improved latencyquality tradeoffs.

Original: These monotonic attention approaches also provide a closed form expression for the expected alignment between source and target tokens.
Revised: These monotonic attention approaches also provide a closed form expression for the expected alignment between source and target tokens.

Original: However, monotonic attention-based models, including the state-of-the-art MILk, were built on top of RNN-based models.
Revised: However, monotonic attention-based models, including the state-of-the-art MILk, were built on top of RNN-based models.

Original: RNN-based models have been outperformed by the recent state-of-the-art
Revised: RNN-based models have been outperformed by the recent state-of-the-art

Original: Transformer model (Vaswani et al., 2017), which features multiple encoder-decoder attention layers and multihead attention at each layer.
Revised: Transformer model (Vaswani et al., 2017), which features multiple encoder-decoder attention layers and multihead attention at each layer.

Original: We thus propose monotonic multihead attention (MMA), which combines the strengths of multilayer multihead attention and monotonic attention.
Revised: We thus propose monotonic multihead attention (MMA), which combines the strengths of multilayer multihead attention and monotonic attention.

Original: We propose two variants, Hard MMA (MMA-H) and
Revised: We propose two variants, Hard MMA (MMA-H) and

Original: Inﬁnite Lookback MMA (MMA-IL).
Revised: Inﬁnite Lookback MMA (MMA-IL).

Original: MMA-H is designed with streaming systems in mind where the attention span must be limited.
Revised: MMA-H is designed with streaming systems in mind where the attention span must be limited.

Original: MMA-IL emphasizes the quality of the translation system.
Revised: MMA-IL emphasizes the quality of the translation system.

Original: We also propose two novel latency regularization methods.
Revised: We also propose two novel latency regularization methods.

Original: The ﬁrst encourages the model to be faster by directly minimizing the average latency.
Revised: The ﬁrst encourages the model to be faster by directly minimizing the average latency.

Original: The second encourages the attention heads to maintain similar positions, preventing the latency from being dominated by a single or a few heads.
Revised: The second encourages the attention heads to maintain similar positions, preventing the latency from being dominated by a single or a few heads.

Original: The main contributions of this paper are: (1) A novel monotonic attention mechanism, monotonic multihead attention, which enables the Transformer model to perform online decoding.
Revised: The main contributions of this paper are: (1) A novel monotonic attention mechanism, monotonic multihead attention, which enables the Transformer model to perform online decoding.

Original: This model leverages the power of the Transformer and the efﬁciency of monotonic attention.
Revised: This model leverages the power of the Transformer and the efﬁciency of monotonic attention.

Original: (2) Better latencyquality tradeoffs compared to the MILk model, the previous state-of-the-art, on two standard translation benchmarks, IWSLT15 English-Vietnamese (En-Vi) and WMT15 German-English (De-En).
Revised: (2) Better latencyquality tradeoffs compared to the MILk model, the previous state-of-the-art, on two standard translation benchmarks, IWSLT15 English-Vietnamese (En-Vi) and WMT15 German-English (De-En).

Original: (3) Analyses on how our model is able to control the attention span and on the relationship between the speed of a head and the layer it belongs to.
Revised: (3) Analyses on how our model is able to control the attention span and on the relationship between the speed of a head and the layer it belongs to.

Original: We motivate the design of our model with an ablation study on the number of decoder layers and the number of decoder heads.
Revised: We motivate the design of our model with an ablation study on the number of decoder layers and the number of decoder heads.

Original:  M ONOTONIC M ULTIHEAD A TTENTION M ODEL
Revised:  M ONOTONIC M ULTIHEAD A TTENTION M ODEL

Original: In this section, we review the monotonic attention-based approaches in RNN-based encoder-decoder models.
Revised: In this section, we review the monotonic attention-based approaches in RNN-based encoder-decoder models.

Original: We then introduce the two types of Monotonic Multihead Attention (MMA) for Transformer models: MMA-H and MMA-IL.
Revised: We then introduce the two types of Monotonic Multihead Attention (MMA) for Transformer models: MMA-H and MMA-IL.

Original: Finally, we introduce strategies to control latency and coverage.
Revised: Finally, we introduce strategies to control latency and coverage.

Original: 2. M ONOTONIC A TTENTION
Revised: 2. M ONOTONIC A TTENTION

Original: The hard monotonic attention mechanism (Raffel et al., 2017) was ﬁrst introduced in order to achieve online linear time decoding for RNN-based encoder-decoder models.
Revised: The hard monotonic attention mechanism (Raffel et al., 2017) was ﬁrst introduced in order to achieve online linear time decoding for RNN-based encoder-decoder models.

Original: We denote the input sequence as x = { x 1 , ..., x T } , and the corresponding encoder states as m = { m 1 , ..., m T } , with T being the length of the source sequence.
Revised: We denote the input sequence as x = { x 1 , ..., x T } , and the corresponding encoder states as m = { m 1 , ..., m T } , with T being the length of the source sequence.

Original: The model generates a target sequence y =
Revised: The model generates a target sequence y =

Original: { y 1 , ..., y U } with
Revised: { y 1 , ..., y U } with

Original: U being the length of the target sequence.
Revised: U being the length of the target sequence.

Original: At the i -th decoding step, the decoder only attends to one encoder state m t
Revised: At the i -th decoding step, the decoder only attends to one encoder state m t

Original: i with t i = j .
Revised: i with t i = j .

Original: When generating a new target token y i , the decoder chooses whether to move one step forward or to stay at the current position based on a Bernoulli selection probability p i,j ,
Revised: When generating a new target token y i , the decoder chooses whether to move one step forward or to stay at the current position based on a Bernoulli selection probability p i,j ,

Original: so that t i ≥ t i − 1 .
Revised: so that t i ≥ t i − 1 .

Original: Denoting the decoder state at the i -th position, starting from j = t i − 1 , t i − 1 + 1 , t i − 1 + 2 , ... , this process can be calculated as follows:
Revised: Denoting the decoder state at the i -th position, starting from j = t i − 1 , t i − 1 + 1 , t i − 1 + 2 , ... , this process can be calculated as follows:

Original: [Equation]
Revised: [Equation]

Original: When z i,j = 1 , we set t i =
Revised: When z i,j = 1 , we set t i =

Original: j and start generating a target token y i ; otherwise, we set t i =
Revised: j and start generating a target token y i ; otherwise, we set t i =

Original: j + 1 and repeat the process.
Revised: j + 1 and repeat the process.

Original: During training, an expected alignment α is introduced to replace the softmax attention.
Revised: During training, an expected alignment α is introduced to replace the softmax attention.

Original: It can be calculated in a recurrent manner, shown in Equation 4:
Revised: It can be calculated in a recurrent manner, shown in Equation 4:

Original: [Equation]
Revised: [Equation]

Original: Raffel et al.
Revised: Raffel et al.

Original: (2017) also introduce a closed-form parallel solution for the recurrence relation in
Revised: (2017) also introduce a closed-form parallel solution for the recurrence relation in

Original: Equation 5:
Revised: Equation 5:

Original: [Equation]
Revised: [Equation]

Original: where cumprod ( x ) =
Revised: where cumprod ( x ) =

Original: [1 , x 1 , x 1 x 2 , ..., (cid:81)
Revised: [1 , x 1 , x 1 x 2 , ..., (cid:81)

Original: | x |− 1 i =1 x i ] and cumsum ( x ) =
Revised: | x |− 1 i =1 x i ] and cumsum ( x ) =

Original: [ x 1 , x 1 + x 2 , ..., (cid:80) | x | i =
Revised: [ x 1 , x 1 + x 2 , ..., (cid:80) | x | i =

Original: 1 x i ] .
Revised: 1 x i ] .

Original: In practice, the denominator in Equation 5 is clamped into a range of [ (cid:15), 1] to avoid numerical instabilities introduced by cumprod .
Revised: In practice, the denominator in Equation 5 is clamped into a range of [ (cid:15), 1] to avoid numerical instabilities introduced by cumprod .

Original: Although this monotonic attention mechanism achieves online linear time decoding, the decoder can only attend to one encoder state.
Revised: Although this monotonic attention mechanism achieves online linear time decoding, the decoder can only attend to one encoder state.

Original: This limitation can diminish translation quality as there may be insufﬁcient information for reordering.
Revised: This limitation can diminish translation quality as there may be insufﬁcient information for reordering.

Original: Moreover, the model lacks a mechanism to adjust latency based on different requirements at decoding time.
Revised: Moreover, the model lacks a mechanism to adjust latency based on different requirements at decoding time.

Original: To address these issues, Chiu & Raffel (2018) introduce Monotonic Chunkwise Attention (MoChA), which allows the decoder to apply softmax attention to a ﬁxed-length subsequence of encoder states.
Revised: To address these issues, Chiu & Raffel (2018) introduce Monotonic Chunkwise Attention (MoChA), which allows the decoder to apply softmax attention to a ﬁxed-length subsequence of encoder states.

Original: Alternatively, Arivazhagan et al.
Revised: Alternatively, Arivazhagan et al.

Original: (2019) introduce Monotonic Inﬁnite Lookback Attention (MILk) which allows the decoder to access encoder states from the beginning of the source sequence.
Revised: (2019) introduce Monotonic Inﬁnite Lookback Attention (MILk) which allows the decoder to access encoder states from the beginning of the source sequence.

Original: The expected attention for the MILk model is deﬁned in Equation 6.
Revised: The expected attention for the MILk model is deﬁned in Equation 6.

Original: [Equation]
Revised: [Equation]

Original: 2. M ONOTONIC M ULTIHEAD A TTENTION
Revised: 2. M ONOTONIC M ULTIHEAD A TTENTION

Original: Previous monotonic attention approaches are based on RNN encoder-decoder models with a single attention and haven’t explored the power of the Transformer model.
Revised: Previous monotonic attention approaches are based on RNN encoder-decoder models with a single attention and haven’t explored the power of the Transformer model.

Original: The Transformer architecture (Vaswani et al., 2017) has recently become the state-of-the-art for machine translation (Barrault et al., 2019).
Revised: The Transformer architecture (Vaswani et al., 2017) has recently become the state-of-the-art for machine translation (Barrault et al., 2019).

Original: An important feature of the Transformer is the use of a separate multihead attention module at each layer.
Revised: An important feature of the Transformer is the use of a separate multihead attention module at each layer.

Original: Thus, we propose a new approach, Monotonic Multihead Attention (MMA), which combines the expressive power of multihead attention and the low latency of monotonic attention.
Revised: Thus, we propose a new approach, Monotonic Multihead Attention (MMA), which combines the expressive power of multihead attention and the low latency of monotonic attention.

Original: Multihead attention allows each decoder layer to have multiple heads, where each head can compute a different attention distribution.
Revised: Multihead attention allows each decoder layer to have multiple heads, where each head can compute a different attention distribution.

Original: Given queries Q , keys K and values V , multihead attention
Revised: Given queries Q , keys K and values V , multihead attention

Original: MultiHead ( Q, K, V ) is deﬁned in Equation 7.
Revised: MultiHead ( Q, K, V ) is deﬁned in Equation 7.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: The attention function is the scaled dot-product attention, deﬁned in Equation 8:
Revised: The attention function is the scaled dot-product attention, deﬁned in Equation 8:

Original: [Equation]
Revised: [Equation]

Original: There are three applications of multihead attention in the Transformer model:
Revised: There are three applications of multihead attention in the Transformer model:

Original: The Encoder contains self-attention layers where all of the queries, keys and values come from previous layers.
Revised: The Encoder contains self-attention layers where all of the queries, keys and values come from previous layers.

Original: The Decoder contains self-attention layers that allow each position in the decoder to attend to all positions in the decoder up to and including that position.
Revised: The Decoder contains self-attention layers that allow each position in the decoder to attend to all positions in the decoder up to and including that position.

Original: The Encoder-Decoder attention contains multihead attention layers where queries come from the previous decoder layer and the keys and values come from the output of the encoder.
Revised: The Encoder-Decoder attention contains multihead attention layers where queries come from the previous decoder layer and the keys and values come from the output of the encoder.

Original: Every decoder layer has a separate encoder-decoder attention.
Revised: Every decoder layer has a separate encoder-decoder attention.

Original: For MMA, we assign each head to operate as a separate monotonic attention in encoder-decoder attention.
Revised: For MMA, we assign each head to operate as a separate monotonic attention in encoder-decoder attention.

Original: For a transformer with L decoder layers and H attention heads per layer, we deﬁne the selection process of the h -th head encoder-decoder attention in the l -th decoder layer as
Revised: For a transformer with L decoder layers and H attention heads per layer, we deﬁne the selection process of the h -th head encoder-decoder attention in the l -th decoder layer as

Original: [Equation]
Revised: [Equation]

Original: where W l,h is the input projection matrix, d k is the dimension of the attention head.
Revised: where W l,h is the input projection matrix, d k is the dimension of the attention head.

Original: We make the selection process independent for each head in each layer.
Revised: We make the selection process independent for each head in each layer.

Original: We then investigate two types of
Revised: We then investigate two types of

Original: MMA, MMA-H(ard) and MMA-IL(inﬁnite lookback).
Revised: MMA, MMA-H(ard) and MMA-IL(inﬁnite lookback).

Original: For MMA-H, we use Equation 4 in order to calculate the expected alignment for each layer each head, given p l,hi,j .
Revised: For MMA-H, we use Equation 4 in order to calculate the expected alignment for each layer each head, given p l,hi,j .

Original: For MMA-IL, we calculate the softmax energy for each head as follows:
Revised: For MMA-IL, we calculate the softmax energy for each head as follows:

Original: [Equation]
Revised: [Equation]

Original: and then use Equation 6 to calculate the expected attention.
Revised: and then use Equation 6 to calculate the expected attention.

Original: Each attention head in MMA-H hardattends to one encoder state.
Revised: Each attention head in MMA-H hardattends to one encoder state.

Original: On the other hand, each attention head in MMA-IL can attend to all previous encoder states.
Revised: On the other hand, each attention head in MMA-IL can attend to all previous encoder states.

Original: Thus, MMA-IL allows the model to leverage more information for translation, but MMA-H may be better suited for streaming systems with stricter efﬁciency requirements.
Revised: Thus, MMA-IL allows the model to leverage more information for translation, but MMA-H may be better suited for streaming systems with stricter efﬁciency requirements.

Original: Finally, our models use unidirectional encoders: the encoder self-attention can only attend to previous states, which is also required for simultaneous translation.
Revised: Finally, our models use unidirectional encoders: the encoder self-attention can only attend to previous states, which is also required for simultaneous translation.

Original: At inference time, our decoding strategy is shown in Algorithm 1.
Revised: At inference time, our decoding strategy is shown in Algorithm 1.

Original: For each l, h , at decoding step i , we apply the sampling processes discussed in subsection 2.1 individually and set the encoder step at t l,hi .
Revised: For each l, h , at decoding step i , we apply the sampling processes discussed in subsection 2.1 individually and set the encoder step at t l,hi .

Original: Then a hard alignment or partial softmax attention from encoder states, shown in Equation 13, will be retrieved to feed into the decoder to generate the i -th token.
Revised: Then a hard alignment or partial softmax attention from encoder states, shown in Equation 13, will be retrieved to feed into the decoder to generate the i -th token.

Original: The model will write a new target token only after all the attentions have decided to write.
Revised: The model will write a new target token only after all the attentions have decided to write.

Original: In other words, the heads that have decided to write must wait until the others have ﬁnished reading.
Revised: In other words, the heads that have decided to write must wait until the others have ﬁnished reading.

Original: [Equation]
Revised: [Equation]

Original: where c l,hi
Revised: where c l,hi

Original: [Equation]
Revised: [Equation]

Original: Figure 1 illustrates a comparison between our model and the monotonic model with one attention head.
Revised: Figure 1 illustrates a comparison between our model and the monotonic model with one attention head.

Original: Compared with the monotonic model, the MMA model is able to set attention to different positions so that it can still attend to previous states while reading each new token.
Revised: Compared with the monotonic model, the MMA model is able to set attention to different positions so that it can still attend to previous states while reading each new token.

Original: Each head can adjust its speed on-the-ﬂy.
Revised: Each head can adjust its speed on-the-ﬂy.

Original: Some heads read new inputs, while the others can stay in the past to retain the source history information.
Revised: Some heads read new inputs, while the others can stay in the past to retain the source history information.

Original: Even with the hard alignment variant (MMA-H), the model is still able to preserve the history information by setting heads to past states.
Revised: Even with the hard alignment variant (MMA-H), the model is still able to preserve the history information by setting heads to past states.

Original: In contrast, the hard monotonic model, which only has one head, loses the previous information at the attention layer.
Revised: In contrast, the hard monotonic model, which only has one head, loses the previous information at the attention layer.

Original: 2. L ATENCY C ONTROL
Revised: 2. L ATENCY C ONTROL

Original: Effective simultaneous machine translation must balance quality and latency.
Revised: Effective simultaneous machine translation must balance quality and latency.

Original: At a high level, latency measures how many source tokens the model has to read until a translation is generated.
Revised: At a high level, latency measures how many source tokens the model has to read until a translation is generated.

Original: The model we have introduced in subsection 2.2 is not able to control latency on its own.
Revised: The model we have introduced in subsection 2.2 is not able to control latency on its own.

Original: While MMA allows simultaneous translation by having a read or write schedule for each head, the overall latency is determined by the fastest head, i.e. the head that reads the most.
Revised: While MMA allows simultaneous translation by having a read or write schedule for each head, the overall latency is determined by the fastest head, i.e. the head that reads the most.

Original: It is possible that a head always reads new input without producing output, which would result in the maximum possible latency.
Revised: It is possible that a head always reads new input without producing output, which would result in the maximum possible latency.

Original: Note that the attention behaviors in MMA-H and MMA-IL can be different.
Revised: Note that the attention behaviors in MMA-H and MMA-IL can be different.

Original: In MMA-IL, a head reaching the end of the sentence will provide the model with maximum information about the source sentence.
Revised: In MMA-IL, a head reaching the end of the sentence will provide the model with maximum information about the source sentence.

Original: On the other hand, in the case of MMA-H, reaching the end of sentence for a head only gives a hard alignment to the end-of-sentence token, which provides very little information to the decoder.
Revised: On the other hand, in the case of MMA-H, reaching the end of sentence for a head only gives a hard alignment to the end-of-sentence token, which provides very little information to the decoder.

Original: Furthermore, it is possible that an MMA-H attention head stays at the beginning of sentence
Revised: Furthermore, it is possible that an MMA-H attention head stays at the beginning of sentence

Original: Input
Revised: Input

Original: 1: while do 2:
Revised: 1: while do 2:

Original: 3: h sequence 4: for l ← 1 to L do 5: for h ← 1 to H do 6: for j ← t l,hi − 1 to | x | do 7: p l,hi,j = Sigmoid (cid:0) MonotonicEnergy
Revised: 3: h sequence 4: for l ← 1 to L do 5: for h ← 1 to H do 6: for j ← t l,hi − 1 to | x | do 7: p l,hi,j = Sigmoid (cid:0) MonotonicEnergy

Original: −
Revised: −

Original: ,m j ) (cid:1) 8: if p l,hi,j
Revised: ,m j ) (cid:1) 8: if p l,hi,j

Original: 5 then 9: t i =
Revised: 5 then 9: t i =

Original: j 10: c l hi =
Revised: j 10: c l hi =

Original: f context ( h t i ) 11:
Revised: f context ( h t i ) 11:

Original: Break 12: else 13: if j > t max then 14:
Revised: Break 12: else 13: if j > t max then 14:

Original: Read token x j 15:
Revised: Read token x j 15:

Original: Calculate state h j and append to h 16: t max =
Revised: Calculate state h j and append to h 16: t max =

Original: j 17: c li =
Revised: j 17: c li =

Original: Concat ( c l, 1 i , c l, 2 i , ..., c l,Hi ) 18: s li =
Revised: Concat ( c l, 1 i , c l, 2 i , ..., c l,Hi ) 18: s li =

Original: DecoderLayer l ( s l 1: i − 1 , s l − 1 1: i − 1 , c li ) 19: y i
Revised: DecoderLayer l ( s l 1: i − 1 , s l − 1 1: i − 1 , c li ) 19: y i

Original: =
Revised: =

Original: Output ( s Li ) 20: i
Revised: Output ( s Li ) 20: i

Original: = i +without moving forward.
Revised: = i +without moving forward.

Original: Such a head would not cause latency issues but would degrade the model quality since the decoder would not have any information about the input.
Revised: Such a head would not cause latency issues but would degrade the model quality since the decoder would not have any information about the input.

Original: In addition, this behavior is not suited for streaming systems.
Revised: In addition, this behavior is not suited for streaming systems.

Original: To address these issues, we introduce two latency control methods.
Revised: To address these issues, we introduce two latency control methods.

Original: The ﬁrst one is weighted average latency, shown in Equation 14:
Revised: The ﬁrst one is weighted average latency, shown in Equation 14:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where g l,hi = (cid:80) | x | j =1 jα i,j .
Revised: where g l,hi = (cid:80) | x | j =1 jα i,j .

Original: Then we calculate the latency loss with a differentiable latency metric C .
Revised: Then we calculate the latency loss with a differentiable latency metric C .

Original: [Equation]
Revised: [Equation]

Original: Like Arivazhagan et al.
Revised: Like Arivazhagan et al.

Original: (2019), we use the Differentiable Average Lagging.
Revised: (2019), we use the Differentiable Average Lagging.

Original: It is important to notice that, unlike the original latency augmented training in Arivazhagan et al.
Revised: It is important to notice that, unlike the original latency augmented training in Arivazhagan et al.

Original: (2019), Equation 15 is not the expected latency metric given C , but weighted average C on all the attentions.
Revised: (2019), Equation 15 is not the expected latency metric given C , but weighted average C on all the attentions.

Original: The real expected latency is ˆ g = max l,h (cid:0) g l,h (cid:1) instead of ¯ g , but using this directly would only affect the speed of the fastest head.
Revised: The real expected latency is ˆ g = max l,h (cid:0) g l,h (cid:1) instead of ¯ g , but using this directly would only affect the speed of the fastest head.

Original: Equation 15, however, can control every head in a way that the faster heads will be automatically assigned to larger weights and slower heads will also be moderately regularized.
Revised: Equation 15, however, can control every head in a way that the faster heads will be automatically assigned to larger weights and slower heads will also be moderately regularized.

Original: heads from getting faster.
Revised: heads from getting faster.

Original: However, for MMA-H models, we found that the latency of are mainly due to outliers that skip almost every token.
Revised: However, for MMA-H models, we found that the latency of are mainly due to outliers that skip almost every token.

Original: The weighted average latency loss is not sufﬁcient to control the outliers.
Revised: The weighted average latency loss is not sufﬁcient to control the outliers.

Original: We therefore introduce the head divergence loss, the average variance of expected delays at each step, deﬁned in Equation 16:
Revised: We therefore introduce the head divergence loss, the average variance of expected delays at each step, deﬁned in Equation 16:

Original: [Equation]
Revised: [Equation]

Original: (16) where ¯ g i = 1 LH (cid:80) g i
Revised: (16) where ¯ g i = 1 LH (cid:80) g i

Original: The ﬁnal objective function is presented in Equation 17:
Revised: The ﬁnal objective function is presented in Equation 17:

Original: [Equation]
Revised: [Equation]

Original: (17) where λ avg , λ var are hyperparameters that control both losses.
Revised: (17) where λ avg , λ var are hyperparameters that control both losses.

Original: Intuitively, while λ avg controls the overall speed, λ var controls the divergence of the heads.
Revised: Intuitively, while λ avg controls the overall speed, λ var controls the divergence of the heads.

Original: Combining these two losses, we are able to dynamically control the range of attention heads so that we can control the latency and the reading buffer.
Revised: Combining these two losses, we are able to dynamically control the range of attention heads so that we can control the latency and the reading buffer.

Original: For MMA-IL model, we only use L avg ; for MMA-H we only use L var .
Revised: For MMA-IL model, we only use L avg ; for MMA-H we only use L var .

Original:  E XPERIMENTAL S ETUP 3. E VALUATION M ETRICS
Revised:  E XPERIMENTAL S ETUP 3. E VALUATION M ETRICS

Original: We evaluate our model using quality and latency.
Revised: We evaluate our model using quality and latency.

Original: For translation quality, we use tokenized BLEUfor IWSLT

Revised: For translation quality, we use tokenized BLEUfor IWSLT

Original: En-Vi and detokenized BLEU with SacreBLEU (Post, 2018) for WMT15 De-En.
Revised: En-Vi and detokenized BLEU with SacreBLEU (Post, 2018) for WMT15 De-En.

Original: For latency, we use three different recent metrics, Average Proportion (AP) (Cho & Esipova, 2016),
Revised: For latency, we use three different recent metrics, Average Proportion (AP) (Cho & Esipova, 2016),

Original: Average Lagging (AL)
Revised: Average Lagging (AL)

Original: (Ma et al., 2019) and Differentiable Average Lagging (DAL) (Arivazhagan et al., 2019) 4 .
Revised: (Ma et al., 2019) and Differentiable Average Lagging (DAL) (Arivazhagan et al., 2019) 4 .

Original: We remind the reader of the metric deﬁnitions in Appendix A.2.
Revised: We remind the reader of the metric deﬁnitions in Appendix A.2.

Original: 3. D ATASETS
Revised: 3. D ATASETS

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: We evaluate our method on two standard machine translation datasets, IWSLT14 En-Vi and WMT
Revised: We evaluate our method on two standard machine translation datasets, IWSLT14 En-Vi and WMT

Original: De-En.
Revised: De-En.

Original: Statistics of the datasets can be found in Table 1.
Revised: Statistics of the datasets can be found in Table 1.

Original: For each dataset, we apply tokenization with the Moses (Koehn et al., 2007) tokenizer and preserve casing.
Revised: For each dataset, we apply tokenization with the Moses (Koehn et al., 2007) tokenizer and preserve casing.

Original: IWSLT15 English-Vietnamese TED talks from IWSLT 2015 Evaluation Campaign (Cettolo et al., 2016).
Revised: IWSLT15 English-Vietnamese TED talks from IWSLT 2015 Evaluation Campaign (Cettolo et al., 2016).

Original: We follow the same settings from Luong & Manning (2015) and Raffel et al.
Revised: We follow the same settings from Luong & Manning (2015) and Raffel et al.

Original: We replace words with frequency less than 5 by < unk > .
Revised: We replace words with frequency less than 5 by < unk > .

Original: We use tst2012 as a validation set tst2013 as a test set.
Revised: We use tst2012 as a validation set tst2013 as a test set.

Original: WMT15 German-English
Revised: WMT15 German-English

Original: We follow the setting from Arivazhagan et al.
Revised: We follow the setting from Arivazhagan et al.

Original: We apply byte pair encoding (BPE) (Sennrich et al., 2016) jointly on the source and target to construct a shared vocabulary with 32K symbols.
Revised: We apply byte pair encoding (BPE) (Sennrich et al., 2016) jointly on the source and target to construct a shared vocabulary with 32K symbols.

Original: We use newstest2013 as validation set and newstest2015 as test set.
Revised: We use newstest2013 as validation set and newstest2015 as test set.

Original: 3. M ODELS
Revised: 3. M ODELS

Original: We evaluate MMA-H and MMA-IL models on both datasets.
Revised: We evaluate MMA-H and MMA-IL models on both datasets.

Original: The MILK model we evaluate on
Revised: The MILK model we evaluate on

Original: IWSLT

Revised: IWSLT

Original: En-Vi is based on Luong et al.
Revised: En-Vi is based on Luong et al.

Original: (2015) rather than RNMT+ (Chen et al., 2018).
Revised: (2015) rather than RNMT+ (Chen et al., 2018).

Original: In general, our ofﬂine models use unidirectional encoders, i.e. the encoder self-attention can only attend to previous states, and greedy decoding.
Revised: In general, our ofﬂine models use unidirectional encoders, i.e. the encoder self-attention can only attend to previous states, and greedy decoding.

Original: We report ofﬂine model performance in Table 2 and the effect of using unidirectional encoders and greedy decoding in Table 3.
Revised: We report ofﬂine model performance in Table 2 and the effect of using unidirectional encoders and greedy decoding in Table 3.

Original: For MMA models, we replace the encoder-decoder layers with MMA and keep other hyperparameter settings the same as the ofﬂine model.
Revised: For MMA models, we replace the encoder-decoder layers with MMA and keep other hyperparameter settings the same as the ofﬂine model.

Original: Detailed hyperparameter settings can be found in subsection A.1.
Revised: Detailed hyperparameter settings can be found in subsection A.1.

Original: We use the Fairseq library (Ott et al., 2019) 6 for our implementation.
Revised: We use the Fairseq library (Ott et al., 2019) 6 for our implementation.

Original: Code will be released upon publication.
Revised: Code will be released upon publication.

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section, we present the main results of our model in terms of latency-quality tradeoffs, ablation studies and analyses.
Revised: In this section, we present the main results of our model in terms of latency-quality tradeoffs, ablation studies and analyses.

Original: In the ﬁrst study, we analyze the effect of the variance loss on the attention span.
Revised: In the ﬁrst study, we analyze the effect of the variance loss on the attention span.

Original: Then, we study the effect of the number of decoder layers and decoder heads on quality and latency.
Revised: Then, we study the effect of the number of decoder layers and decoder heads on quality and latency.

Original: We also provide a case study for the behavior of attention heads in an example.
Revised: We also provide a case study for the behavior of attention heads in an example.

Original: Finally, we study the relationship between the rank of an attention head and the layer it belongs to.
Revised: Finally, we study the relationship between the rank of an attention head and the layer it belongs to.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: IWSLT

Original: [Figure]
Revised: [Figure]

Original: U
Revised: [Figure]

Original: IWSLT15 English-Vietnamese

Original: [Figure]
Revised: [Figure]

Original: Average Proportion

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We plot the quality-latency curves for MMA-H and MMA-IL in Figure 2.
Revised: We plot the quality-latency curves for MMA-H and MMA-IL in Figure 2.

Original: The BLEU and latency scores on the test sets are generated by setting a latency range and selecting the checkpoint with best BLEU score on the validation set.
Revised: The BLEU and latency scores on the test sets are generated by setting a latency range and selecting the checkpoint with best BLEU score on the validation set.

Original: We use differentiable average lagging (Arivazhagan et al., 2019) when setting the latency range.
Revised: We use differentiable average lagging (Arivazhagan et al., 2019) when setting the latency range.

Original: We ﬁnd that for a given latency, our models obtain a better translation quality.
Revised: We ﬁnd that for a given latency, our models obtain a better translation quality.

Original: While MMA-IL tends to have a decrease in quality as the latency decreases,
Revised: While MMA-IL tends to have a decrease in quality as the latency decreases,

Original: MMA-H has a small gain in quality as latency decreases.
Revised: MMA-H has a small gain in quality as latency decreases.

Original: The reason is that a larger latency does not necessarily mean an increase in source information available to the model.
Revised: The reason is that a larger latency does not necessarily mean an increase in source information available to the model.

Original: In fact, the large latency is introduced by outlier attention heads, which read the entire source sentence and point to the end of the source sentence.
Revised: In fact, the large latency is from the outlier attention heads, which skip the entire source sentence and point to the end of the source sentence.

Original: These outlier heads not only increase the latency but they also do not provide useful information.
Revised: The outliers not only increase the latency but they also do not provide useful information.

Original: We introduce the attention variance loss to eliminate the outliers, as such a loss makes the attention heads focus on the current context for translating the new target token.
Revised: We introduce the attention variance loss to eliminate the outliers, as such a loss makes the attention heads focus on the current context for translating the new target token.

Original: This is why when the latency decreases (fewer outliers), the MMA-H model has an increase on translation quality.

Original: It is interesting to observe that even MMA-H has a better latency-quality tradeoff than MILkeven though each head only attends to only one state.
Revised: It is interesting to observe that even MMA-H has a better latency-quality tradeoff than MILkeven though each head only attends to only one state.

Original: Although MMA-H is not yet able to handle an arbitrarily long input (without resorting to segmenting the input), since both the encoder and decoder self-attention have an inﬁnite lookback, that model represents a good step in that direction.[Figure] [Figure]
Revised: Although MMA-H is not yet able to handle an arbitrarily long input (without resorting to segmenting the input), since both the encoder and decoder self-attention have an inﬁnite lookback, that model represents a good step in that direction.

Original: 4. A TTENTION S PAN
Revised: 4. A TTENTION S PAN

Original: In subsection 2.3, we introduced the attention variance loss to MMA-H in order to prevent outlier attention heads from increasing the latency or increasing the attention span.
Revised: In subsection 2.3, we introduced the attention variance loss to MMA-H in order to prevent outlier attention heads from increasing the latency or increasing the attention span.

Original: We have already  evaluated the effectiveness of this method on latency in subsection 4.1.
Revised: We have already [Figure] [Figure] evaluated the effectiveness of this method on latency in subsection 4.1.

Original: We also want to measure the difference between the fastest and slowest heads at each decoding step.
Revised: We also want to measure the difference between the fastest and slowest heads at each decoding step.

Original: We deﬁne the average attention span in Equation 18:
Revised: We deﬁne the average attention span in Equation 18:

Original: [Equation]
Revised: [Equation]

Original: It estimates the reading buffer we need for streaming translation.
Revised: It estimates the reading buffer we need for streaming translation.

Original: We show the relation between the average attention span (averaged over the IWSLT and WMT test sets) versus L var in Figure 3.
Revised: We show the relation between the average attention span (averaged over the IWSLT and WMT test sets) versus L var in Figure 3.

Original: As expected, the average attention span is reduced as we increase L var .
Revised: As expected, the average attention span is reduced as we increase L var .

Original: 4. E FFECT ON NUMBER OF LAYERS AND NUMBER OF HEADS
Revised: 4. E FFECT ON NUMBER OF LAYERS AND NUMBER OF HEADS

Original: One motivation to introduce MMA is to adapt the Transformer, which is the current state-of-the-art model for machine translation, to online decoding.
Revised: One motivation to introduce MMA is to adapt the Transformer, which is the current state-of-the-art model for machine translation, to online decoding.

Original: Important features of the Transformer architecture include having a separate attention layer for each decoder layer block and multihead attention.
Revised: Important features of the Transformer architecture include having a separate attention layer for each decoder layer block and multihead attention.

Original: In this section, we test the effect of these two components on the ofﬂine, MMA-H, and MMA-IL models from a quality and latency perspective.
Revised: In this section, we test the effect of these two components on the ofﬂine, MMA-H, and MMA-IL models from a quality and latency perspective.

Original: We report quality as measured by detokenized BLEU and latency as measured by DAL on the WMT13 validation set in Figure 4.
Revised: We report quality as measured by detokenized BLEU and latency as measured by DAL on the WMT13 validation set in Figure 4.

Original: We set λ avg = 0 .
Revised: We set λ avg = 0 .

Original: 2 for
Revised: 2 for

Original: MMA-IL and λ var = 0 .
Revised: MMA-IL and λ var = 0 .

Original: 2 for MMA-H.
Revised: 2 for MMA-H.

Original: The ofﬂine model beneﬁts from having more than one decoder layer.
Revised: The ofﬂine model beneﬁts from having more than one decoder layer.

Original: In the case of 1 decoder layer, increasing the number of attention heads is beneﬁcial but in the case of 3 and 6 decoder layers, we do not see much beneﬁt from using more than 2 heads.
Revised: In the case of 1 decoder layer, increasing the number of attention heads is beneﬁcial but in the case of 3 and 6 decoder layers, we do not see much beneﬁt from using more than 2 heads.

Original: The best performance is obtained for 3 layers and 2 heads, which effectively means 6 attention heads.
Revised: The best performance is obtained for 3 layers and 2 heads (6 effective heads).

Original: The MMA-IL model behaves similarly to the ofﬂine model, and the best performance is observed with 6 layers and 4 heads (24 effective heads).
Revised: The MMA-IL model behaves similarly to the ofﬂine model, and the best performance is observed with 6 layers and 4 heads (24 effective heads).

Original: For MMA-H, with 1 layer, performance improves with more heads.
Revised: For MMA-H, with 1 layer, performance improves with more heads.

Original: With 3 layers, the single-head setting is the most effective (which means 3 effective heads).
Revised: With 3 layers, the single-head setting is the most effective (3 effective heads).

Original: Finally, with 6 layers, the best performance is reached with 16 heads (96 effective heads).
Revised: Finally, with 6 layers, the best performance is reached with 16 heads (96 effective heads).

Original: The general trend we observe is that performance improves as we increase the number of effective heads, either from multiple layers or multihead attention up to a certain point, then either plateaus or degrades.
Revised: The general trend we observe is that performance improves as we increase the number of effective heads, either from multiple layers or multihead attention, up to a certain point, then either plateaus or degrades.

Original: This motivates the introduction of the MMA model.
Revised: This motivates the introduction of the MMA model.

Original: We also note that latency increases with the number of effective attention heads.
Revised: We also note that latency increases with the number of effective attention heads.

Original: This is due to having ﬁxed loss weights: when more heads are involved, we should increase λ var or λ avg to better control latency.
Revised: This is due to having ﬁxed loss weights: when more heads are involved, we should increase λ var or λ avg to better control latency.

Original: 4. A TTENTION BEHAVIORS
Revised: 4. A TTENTION BEHAVIORS

Original: We characterize attention behaviors by providing a running example of MMA-H and MMA-IL, shown in Figure 5.
Revised: We characterize attention behaviors by providing a running example of MMA-H and MMA-IL, shown in Figure 5.

Original: Each curve represents the path that an attention head goes through at inference time.
Revised: Each curve represents the path that an attention head goes through at inference time.

Original: For MMA-H, shown in Figure 5a, we found that when the source and target tokens have the 
Revised: For MMA-H, shown in Figure 5a, we found that when the source and target tokens have the same order, the attention heads behave linearly and the distance between fastest head and slowest head is small. For example, this can be observed from partial sentence pair “I also didn ’t know that” and target tokens “ Tôi cũng không biết rằng ”, which have the same order. However, when the source tokens and target tokens have different orders, such as “the second step” and “ bước ( step ) thứ hai

Original: B LE
Revised: B LE

Original: U
Revised: U

Original: Ofﬂine Model 1decoderlayer3decoderlayers6decoderlayers 12     # heads
Revised: Ofﬂine Model 1decoderlayer3decoderlayers6decoderlayers 12     # heads

Original: MMA-H 1decoderlayer3decoderlayers6decoderlayers 12 

Revised: MMA-H 1decoderlayer3decoderlayers6decoderlayers 12

Original: DA
Revised: DA

Original: L
Revised: L

Original: MMA-H (latency) 1decoderlayer3decoderlayers6decoderlayers
Revised: MMA-H (latency) 1decoderlayer3decoderlayers6decoderlayers

Original: B LE
Revised: B LE

Original: U
Revised: U

Original: MMA-IL 1decoderlayer3decoderlayers
Revised: MMA-IL 1decoderlayer3decoderlayers

Original: 6decoderlayers 12 

Revised: 6decoderlayers 12

Original: DA
Revised: DA

Original: L
Revised: L

Original: [Figure] -IL (
Revised: MMA-IL (

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: same order , the attention heads behave linearly and the distance between fastest head and slowest head is small. For example, this can be observed from partial sentence pair “I also didn ’t know that” and target tokens “ Tôi cũng không biết rằng ”, which have the same order. However, when the source tokens and target tokens have different orders, such as “the second step” and “ bước ( step ) thứ hai ( second ) ”, the model will generate “ bước ( step ) ”
Revised: ( second ) ”, the model will generate “ bước ( step ) ”

Original: ﬁrst and some heads will stay in the past to retain the information for later reordered translation “ thứ hai ( second ) ”.
Revised: ﬁrst and some heads will stay in the past to retain the information for later reordered translation “ thứ hai ( second ) ”.

Original: We can also see that the attention heads move near diagonally, which potentially can be applied for streaming inputs.
Revised: We can also see that the attention heads have a near-diagonal trajectory, which is appropriate for streaming inputs.

Original: The behaviors of the heads in MMA-IL models are shown as Figure 5b.
Revised: The behaviors of the heads in MMA-IL models are shown in Figure 5b.

Original: Notice that we remove the partial softmax alignment in this ﬁgure.
Revised: Notice that we remove the partial softmax alignment in this ﬁgure.

Original: Because we don’t expect streaming capability for MMA-IL, some heads stop at early position of the source sentence to retain the history information.
Revised: Because we don’t expect streaming capability for MMA-IL, some heads stop at early position of the source sentence to retain the history information.

Original: Moreover, because MMA-IL has more information when generating a new target token, it tends to produce better translation quality.
Revised: Moreover, because MMA-IL has more information when generating a new target token, it tends to produce better translation quality.

Original: In this example, the MMA-IL model has better translation on “isolate the victim” than MMA-H (“ là cô lập nạn nhân ” vs “ là tách biệt nạn nhân ”)[Figure] [Figure]
Revised: In this example, the MMA-IL model has better translation on “isolate the victim” than MMA-H (“ là cô lập nạn nhân ” vs “ là tách biệt nạn nhân ”)

Original: [Figure]
Revised: [Figure]

Original:  R ANK OF THE HEADS
Revised: 4. R ANK OF THE HEADS

Original: In Figure 6, we calculate the average and standard deviation of rank of each head when generating every target token.
Revised: In Figure 6, we calculate the average and standard deviation of rank of each head when generating every target token.

Original: For MMA-IL, we ﬁnd thatsome heads  are generally faster than others, this can be also observed from the case example from Figure 5b.
Revised: For MMA-IL, we ﬁnd that heads in lower layers tend to have higher rank and are thus slower.

Original: However, in MMA-H, the difference of the average rank are smaller.
Revised: However, in MMA-H, the difference of the average rank are smaller.

Original: Furthermore,  the standard deviation is very large which means that the order of the heads in MMA-H changes frequently over the inference process.
Revised: Furthermore, [Figure] the standard deviation is very large which means that the order of the heads in MMA-H changes frequently over the inference process.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recent work on simultaneous machine translation falls into three categories.
Revised: Recent work on simultaneous machine translation falls into three categories.

Original: In the ﬁrst one, models use a rule-based policy for reading input and writing output.
Revised: In the ﬁrst one, models use a rule-based policy for reading input and writing output.

Original: Cho & Esipova (2016) propose a Wait-If-* policy to enable an ofﬂine model to decode simultaneously.
Revised: Cho & Esipova (2016) propose a WaitIf-* policy to enable an ofﬂine model to decode simultaneously.

Original: Ma et al.
Revised: Ma et al.

Original: (2019) propose a wait- k policy where the model ﬁrst reads k tokens, then alternates between read and write actions.
Revised: (2019) propose a wait- k policy where the model ﬁrst reads k tokens, then alternates between read and write actions.

Original: Dalvi et al.
Revised: Dalvi et al.

Original: (2018) propose an incremental decoding method, also based on a rule-based schedule.
Revised: (2018) propose an incremental decoding method, also based on a rule-based schedule.

Original: In the second category, models learn the policy with reinforcement learning.
Revised: In the second category, a ﬂexible policy is learnt from data.

Original: Grissom II et al.
Revised: Grissom II et al.

Original: (2014) introduce a Markov chain to phrase-based machine translation models for simultaneous machine translation, in which they apply reinforcement learning to learn the read-write policy based on states.
Revised: (2014) introduce a Markov chain to phrase-based machine translation models for simultaneous machine translation, in which they apply reinforcement learning to learn the read-write policy based on states.

Original: Gu et al.
Revised: Gu et al.

Original: (2017) introduce an agent which learns to make decisions on when to translate from the interaction with a pre-trained ofﬂine neural machine translation model.
Revised: (2017) introduce an agent which learns to make decisions on when to translate from the interaction with a pre-trained ofﬂine neural machine translation model.

Original: Luo et al.
Revised: Luo et al.

Original: (2017) used continuous rewards policy gradient for online alignments for speech recognition.
Revised: (2017) used continuous rewards policy gradient for online alignments for speech recognition.

Original: Lawson et al.
Revised: Lawson et al.

Original: (2018) proposed a hard alignment with variational inference for online decoding.
Revised: (2018) proposed a hard alignment with variational inference for online decoding.

Original: Alinejad et al.
Revised: Alinejad et al.

Original: (2018) propose a new operation predict which predicts future source tokens.
Revised: (2018) propose a new operation predict which predicts future source tokens.

Revised: Zheng et al.

Revised: (2019b) introduce restricted dynamic oracle and restricted imitation learning for simultaneous translation.

Revised: Zheng et al.

Revised: (2019a) train the agent with action sequence from labels that generated based on the rank of the gold target word given partial input.

Original: Models from the last category leverage monotonic attention and replace the softmax attention with an expected attention calculated from a stepwise Bernoulli selection probability.
Revised: Models from the last category leverage monotonic attention and replace the softmax attention with an expected attention calculated from a stepwise Bernoulli selection probability.

Original: Raffel et al.
Revised: Raffel et al.

Original: ﬁrst introduce the concept of monotonic attention for online linear time decoding, where the attention only attends to one encoder state at a time.
Revised: ﬁrst introduce the concept of monotonic attention for online linear time decoding, where the attention only attends to one encoder state at a time.

Original: Chiu & Raffel (2018) extended that work to let the model attend to a chunk of encoder state.
Revised: Chiu & Raffel (2018) extended that work to let the model attend to a chunk of encoder state.

Original: Arivazhagan et al.
Revised: Arivazhagan et al.

Original: (2019) also make use of the monotonic attention but introduce an inﬁnite lookback to improve the translation quality.
Revised: (2019) also make use of the monotonic attention but introduce an inﬁnite lookback to improve the translation quality.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose two variants of the monotonic multihead attention model for simultaneous machine translation.
Revised: In this paper, we propose two variants of the monotonic multihead attention model for simultaneous machine translation.

Original: By introducing two new targeted loss terms which allow us to control both latency and attention span, we are able to leverage the power of the Transformer architecture to achieve better quality-latency trade-offs than the previous state-of-the-art model.
Revised: By introducing two new targeted loss terms which allow us to control both latency and attention span, we are able to leverage the power of the Transformer architecture to achieve better quality-latency trade-offs than the previous state-of-the-art model.

Original: We also present detailed ablation studies demonstrating the efﬁcacy and rationale of our approach.
Revised: We also present detailed ablation studies demonstrating the efﬁcacy and rationale of our approach.

Original: By introducing these stronger simultaneous sequence-to-sequence models, we hope to facilitate important applications, such as high-quality real-time interpretation between human speakers.
Revised: By introducing these stronger simultaneous sequence-to-sequence models, we hope to facilitate important applications, such as high-quality real-time interpretation between human speakers.

Original: A A PPENDIX A. H YPERPARAMETERS
Revised: A A PPENDIX A. H YPERPARAMETERS

Original: The hyperparameters we used for ofﬂine and monotonic transformer models are deﬁned in Table 4.
Revised: The hyperparameters we used for ofﬂine and monotonic transformer models are deﬁned in Table 4.

Original: A. L ATENCY M ETRICS D EFINITIONS
Revised: A. L ATENCY M ETRICS D EFINITIONS

Original: Given the delays g
Revised: Given the delays g

Original: = { g 1 , g 2 , ..., g | y | } of generating each target token, AP, AL and DAL are deﬁned in Table5.
Revised: = { g 1 , g 2 , ..., g | y | } of generating each target token, AP, AL and DAL are deﬁned in Table

Original: A. D ETAILED RESULTS

Original: We provide the detailed results in Figure 2 as Table 6 and Table 7.

Original: A. T HRESHOLD OF READING ACTION

Original: We explore a simple method that can adjust system’s latency at inference time without training new models.

Original: In Algorithm 1 line 8, 0.5 was used as an threshold.

Original: One can set different threshold p during the inference time to control the latency.

Original: We run the pilot experiments on IWSLT15 En-Vi dataset and the results are shown as Table 8.

Original: Although this method doesn’t require training new model, it dramatically hurts the translation quality.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: | x || y | | y | (cid:88)
Revised: | x || y | | y | (cid:88)

Original: i = g i
Revised: i = g i

Original: [Table]
Revised: [Table]

Original: τ τ (cid:88) i =
Revised: τ τ (cid:88) i =

Original: g i − i − 1 | y | / | x | where τ = arg max i ( g i =
Revised: g i − i − 1 | y | / | x | where τ = arg max i ( g i =

Original: | x | )
Revised: | x | )

Original: Differentiable [Table] Lagging | y | | y | (cid:88)
Revised: Differentiable Average Lagging | y | | y | (cid:88)

Original: i = g (cid:48)
Revised: i = g (cid:48)

Original: i − i − 1 | y | / | x | where g (cid:48) i =
Revised: i − i − 1 | y | / | x | where g (cid:48) i =

Original:  
Revised:  

Original: g i i =max( g i , g (cid:48)
Revised: g i i =max( g i , g (cid:48)

Original: i − 1 + | y | | x | ) i <
Revised: i − 1 + | y | | x | ) i <

Revised: A. D ETAILED RESULTS

Original: [Table]
Revised: We provide the detailed results in Figure 2 as Table 6 and Table 7. A. T HRESHOLD OF READING ACTION We explore a simple method that can adjust system’s latency at inference time without training new models. In Algorithm 1 line 8, 0.5 was used as an threshold.

Revised: One can set different threshold p during the inference time to control the latency.

Revised: We run the pilot experiments on IWSLT15 En-Vi dataset and the results are shown as Table 8.

Revised: Although this method doesn’t require training new model, it dramatically hurts the translation quality.

Revised: [Table]

Revised: [Table]

Revised: A. A VERAGE LOSS FOR MMA-H

Original: [Table]
Revised: We explore applying a simple average instead of a weighted average loss to MMA-H. The results are shown in Figure 7 and Table 9. We ﬁnd that even with very large weights, we are unable to reduce the overall latency. In addition, we ﬁnd that the weighted average loss severely affects the translation quality negatively.

Revised: On the other hand, the divergence loss we propose in Equation 16 can efﬁciently reduce the latency while retaining relatively good translation quality for MMA-H models

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: latency loss.
Revised: [Table] [Figure] [Figure] [Table]