Original: L EARNING M ONOTONIC A LIGNMENTS WITH S OURCE -A WARE GMM A TTENTION
Revised: L EARNING M ONOTONIC A LIGNMENTS WITH S OURCE -A WARE GMM A TTENTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers with soft attention have been widely adopted to various sequence-tosequence  tasks.
Revised: Transformers with soft attention have been widely adopted in various sequence-tosequence (Seq2Seq) tasks.

Original: While the soft attention is effective for learning semantic similarities between queries and keys based on their contents, it does not explicitly model the order of elements in sequences which is crucial for monotonic sequence-to-sequence tasks.
Revised: Whereas soft attention is effective for learning semantic similarities between queries and keys based on their contents, it does not explicitly model the order of elements in sequences which is crucial for monotonic (Seq2Seq) tasks.

Original: Learning monotonic alignments between input and output sequences could be beneﬁcial for long-form and online inference applications that are still challengingproblem for the conventional soft attention algorithm.
Revised: Learning monotonic alignments between input and output sequences may be beneﬁcial for longform and online inference applications that are still challenging for the conventional soft attention algorithm.

Original: In this paper, we focus on monotonic sequence-tosequence tasks and propose Source-Aware Gaussian Mixture Model (SAGMM) attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence.
Revised: Herein, we focus on monotonic Seq2Seq tasks and propose a sourceaware Gaussian mixture model attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence.

Original: We show that the proposed attention mechanism improves the performance on the online and long-form speech recognition problems without performance degradation in ofﬂine in-distribution speech recognition through experiments.
Revised: We experimentally demonstrate that the proposed attention mechanism improved the performance on the online and long-form speech recognition problems without performance degradation in ofﬂine in-distribution speech recognition.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For a recentfew years,the transformer models with soft attention have been widely adopted to various sequence generation tasks (Raffel et al., 2019; Vaswani et al., 2017; Parmar et al., 2018; Karita et al., 2019).
Revised: In recent years, transformer models with soft attention have been widely adopted in various sequence generation tasks (Raffel et al., 2019; Vaswani et al., 2017; Parmar et al., 2018; Karita et al., 2019).

Original: The soft attention does not explicitly model the order of elements in the sequence and attends all encoder outputs for each decoder step.
Revised: Soft attention does not explicitly model the order of elements in a sequence and attends all encoder outputs for each decoder step.

Original: However, the order of elements is crucial to understand monotonic sequenceto-sequence (Seq2Seq) tasks, e.g. automatic speech recognition (ASR), video analysis, and lip reading.
Revised: However, the order of elements is crucial for understanding monotonic sequence-to-sequence (Seq2Seq) tasks, such as automatic speech recognition (ASR), video analysis, and lip reading.

Original: Learning monotonic alignments helps model to attend to a subset of the encoder output without performance degradation in these tasks.
Revised: Learning monotonic alignments enables the model to attend to a subset of the encoder output without performance degradation in these tasks.

Original: In comparison,the soft attention is not suitable for streaming inference applications since the softmax operation needs to wait until all encoder outputs are produced.
Revised: In comparison, soft attention is not suitable for streaming inference applications because the softmax operation needs to wait until all encoder outputs are produced.

Original: Figure 1 (b) shows the attention plot from soft-attention.
Revised: Figure 1 (b) shows the attention plot for soft-attention.

Original: The soft attention learns  alignments between queries and keys based on their similarities while it requires all encoder tokens before attention score calculation.
Revised: Soft attention learns the alignments between queries and keys based on their similarities; it requires all encoder tokens prior to the attention score calculation.

Original: Furthermore,the soft attention has difﬁculty to decode long-form sequences which are not considered in  training corpus.
Revised: Furthermore, soft attention cannot easily decode long-form sequences that are not considered in the training corpus.

Original: The Gaussian Mixture Model (GMM) attention Graves (2013); Battenberg et al.
Revised: The Gaussian Mixture Model (GMM) attention Graves (2013); Battenberg et al.

Original: Chiu et al.
Revised: Chiu et al.

Original: (2019) has been proposed to learn the monotonic mapping between encoder and decoder states for long-form sequence generation.
Revised: (2019) have been proposed for learning the monotonic mapping between encoder and decoder states for long-form sequence generation.

Original: The GMM attention is a pure location-aware algorithm in whichthe encoder contents are not considered during attention score calculation.
Revised: The GMM attention is a pure location-aware algorithm in which encoder contents are not considered during attention score calculation.

Original: However, each element in  encoder output sequences has different amount of information and should be attended considering their contents.
Revised: However, each element in the encoder output sequence contains different amounts of information and should be attended considering their contents.

Original: As shown in ﬁgure(c), the GMM attention fails to learn  detailed alignments and attends to many tokens simultaneously.
Revised: In ﬁgure 1 (c), the GMM attention fails to learn the detailed alignments and attends to many tokens simultaneously.

Original: In this paper, we adopt the GMM attention mechanism to  modern transformer structure and propose SourceAware Gaussian Mixture Model (SAGMM) attention which considers both contents and orders of source sequences.
Revised: In this study, we adopted the GMM attention mechanism to the modern transformer structure and proposed the Source-Aware Gaussian Mixture Model (SAGMM) attention which considers both contents and orders of source sequences.

Original: Each component in  SAGMM is multi-modal and discard non-informative tokens in  attention window.
Revised: Each component in the SAGMM is multi-modal and discards non-informative tokens in the attention window.

Original: For online inference, we propose  truncated SAGMM (SAGMM-tr) by discarding long-tail of the attention score in  SAGMM.
Revised: For online inference, we propose a truncated SAGMM (SAGMM-tr) that discards the long-tail of the attention score in the SAGMM.

Original: To the best of our knowledge, this is the ﬁrst attempt to adopt the GMM-based attention to online sequence generation tasks.
Revised: To the best of our knowledge, this is the ﬁrst attempt to adopt a GMM-based attention to online sequence generation tasks.

Original: Learning accurate monotonic alignments helps SAGMM-tr to attend to a relevant subset of sequences for each decoder step and improves the performance of  model in  streaming and long-form sequence generation tasks.
Revised: Learning accurate monotonic alignments enables the SAGMM-tr to attend to a relevant subset of sequences for each decoder step and improves the performance of the model in terms of streaming and long-form sequence generation tasks.

Original: Figure 1 (d) shows the monotonic alignments learned by SAGMM-tr which enables online inference.
Revised: Figure 1 (d) shows the monotonic alignments learned by the SAGMM-tr, enabling online inference.

Original: The experiments on streaming and long-form ASR showed substantial performance improvements compared to conventional algorithms without performance degradation on ofﬂine in-distribution ASR.
Revised: Experiments involving streaming and long-form ASR showed substantial performance improvements compared with conventional algorithms without performance degradation in ofﬂine in-distribution ASR.

Original: We also tested  SAGMM-tr in  machine translation task and showed the performance of  proposed algorithm in non-monotonic tasks.
Revised: Furthermore, we tested the SAGMM-tr in a machine translation task and demonstrated the performance of the proposed algorithm in non-monotonic tasks.

Original:  S OURCE - AWARE GMM ATTENTION 2. S OFT ATTENTION
Revised:  S OURCE - AWARE GMM ATTENTION 2. S OFT ATTENTION

Original: In this paper, we abbreviate the head index h during attention score calculation for simplicity.
Revised: Herein, we abbreviate the head index h during attention score calculation for simplicity.

Original: In the dotproduct multi-head soft attention Vaswani et al.
Revised: In dot-product multi-head soft attention Vaswani et al.

Original: (2017) without relative positional encoding , the attention score from soft attention α Soft is derived from the query matrix Q ∈ R
Revised: (2017) without relative positional encoding , the attention score from soft attention α Soft is derived from the query matrix Q ∈ R

Original: I × d and key matrix K ∈ R J × d as follows:
Revised: I × d and key matrix K ∈ R J × d as follows:

Original: [Equation]
Revised: [Equation]

Original:  where d, I , and J are  feature dimension, decoder and encoder sequence length, respectively.
Revised: (1) where d, I , and J are the feature dimension, decoder and encoder sequence length, respectively.

Original: The attention context matrix form h -th head H h and the multi-head output M is given by
Revised: The attention context matrix from the h -th head H h and the multi-head output M are expressed as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where α hSoft denotes  α Soft for  h -th head, V h ∈ R J × d is value matrix for  h -th head, and n hdenotes the number of heads.
Revised: ) where α hSoft denotes the α Soft for the h -th head, V h ∈ R J × d the value matrix for the h -th head, and n h the number of heads.

Original: In this paper, we adopt the relative positional encoding Shaw et al.
Revised: In this study, we adopted relative positional encoding Shaw et al.

Original: Dai et al.
Revised: Dai et al.

Original: (2019) for self-attention layers. The relative positional encoding gives a more strong baseline for the longform sequence generation.
Revised: (2019) which provides a stronger baseline for long-form sequence generation for self-attention layers.

Original: 2. GMM ATTENTION
Revised: 2. GMM ATTENTION

Original: The previous studies on GMM attention Battenberg et al.
Revised: The previous studies regarding GMM attention Battenberg et al.

Original: Chiu et al.
Revised: Chiu et al.

Original: (2019) are based on early content-based attention (Cho et al., 2014).
Revised: (2019) were based on early content-based attention (Cho et al., 2014).

Original: Li et al.
Revised: Li et al.

Original: (2020) adopt the GMM attention to the transformer framework, but they do not provide detailed descriptions.
Revised: (2020) adopted the GMM attention to the transformer framework, but did not provide detailed descriptions.

Original: In this paper, we adopt v2 model in Battenberg et al.
Revised: Herein, we adopt v2 model in Battenberg et al.

Original: (2020) which improves the performance of  attention mechanismproposed in Graves (2013).
Revised: (2020) which improved the performance of the original GMM attention mechanism Graves (2013).

Original: We deﬁne the GMM attention as a variant ofthe multi-head attention by considering a Gaussian distribution component as an attention score of single-head in  multi-head mechanism.Our GMM model is similar to
Revised: We deﬁne the GMM attention as a variant of multi-head attention by considering a Gaussian distribution component as an attention score of single-head in a multi-head mechanism.

Original:  Battenberg et al.
Revised: In the study by Battenberg et al.

Original: (2020) when the value matrix is shared for all Gaussian components. The multi-head value matrices helps to attend to information from different representation subspaces (Vaswani et al., 2017).
Revised: (2020), the value matrix was shared for all Gaussian components, whereas multi-head value matrices were multiplied with the probability from corresponding components in this study to attend to information from different representation subspaces (Vaswani et al., 2017).

Revised: Hence, the multi-head GMM attention introduced here is a more generalized algorithm compared with the early GMM attention.

Original: Let us denote the i -th row of Q as Q i ∈ R 1 × d .
Revised: Let us denote the i -th row of Q as Q i ∈ R 1 × d .

Original: The normal distribution parameters for  i -th step are given by
Revised: The normal distribution parameters for the i -th step are expressed

Original: [Equation]
Revised: [Equation]

Original: where ζ ( x ) is a softplus function of x , and W ∆ ∈ R d × 1 , W σ ∈ R d × 1 ,  W φ ∈ R d × 1 .
Revised: where ζ ( x ) is a softplus function of x ; W ∆ ∈ R d × 1 , W σ ∈ R d × 1 , and W φ ∈ R d × 1 .

Original: The softplus function is adopted similar to  Battenberg et al.
Revised: The softplus function was adopted, similar to the study of Battenberg et al.

Original: (2020) in which softplus activation showed better performance than  exponential operation.
Revised: (2020) in which softplus activation demonstrated better performances than the exponential operation.

Original: A mixture component of the GMM attention, from  i -th decoder step to  j -th encoder token, is derived from the α GMM (cid:107) i,j is deﬁned as follows:
Revised: A mixture component of the GMM attention, from the i -th decoder step to the j -th encoder token, is derived from the α GMM (cid:107) i,j is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where softmax h denotes  softmax function over heads.
Revised: where softmax h denotes the softmax function over heads.

Original: The conventional GMM attention mechanism is analogous to integral of source sequences with an uniform axis spacing as shown in Figure 2 (a).
Revised: The conventional GMM attention mechanism is analogous to integral of source sequences with a uniform axis spacing as shown in Figure 2 (a).

Original: In this ﬁgure, each rectangle denotes the attention score with given
Revised: In this ﬁgure, each rectangle denotes the attention score with speciﬁed

Original: Gaussian component parameters.
Revised: Gaussian component parameters.

Original: The GMM attention assumes that each encoder output is equally important.
Revised: The GMM attention assumes that each encoder output is equally important.

Original: However, this assumption is not satisﬁed for many input modalities, e.g. speech and videos from real environments.
Revised: However, this assumption is not satisﬁed for many input modalities, e.g. speech and videos from real environments.

Original: Moreover, the number of modes in  GMM attention is limited by the number of mixture components.
Revised: Moreover, the number of modes in the GMM attention is limited by the number of mixture components.

Original: To learn robust alignments for monotonic Seq2Seq tasks, we propose  SAGMM which considers both contents and locations during attention mechanism.
Revised: To learn robust alignments for monotonic Seq2Seq tasks, we propose the SAGMM which considers both contents and locations for the attention mechanism.

Original: 2. S OURCE - AWARE GMM ATTENTION
Revised: 2. S OURCE - AWARE GMM ATTENTION

Original: Figure 2 (b) shows the scheme of  proposed SAGMM attention.
Revised: Figure 2 (b) shows the scheme of the proposed SAGMM attention.

Original: Compared to the GMM attention, the SAGMM is analogous to  integral of normal distribution with non-uniform spacing based on encoder contents.
Revised: Compared with the GMM attention, the SAGMM is analogous to an integral of normal distribution with non-uniform spacing based on encoder contents.

Original: In the ﬁgure 2 (b), the width of rectangle δ j controls the model to selectively attend to the informative tokens during  attention score calculation.
Revised: In the ﬁgure 2 (b), the width of rectangle δ j controls the model to selectively attend to the informative tokens during the attention score calculation.

Original: This content-aware property of  SAGMM helps the model learning stable monotonic alignments from the training corpus.
Revised: This content-aware property of the SAGMM enables the model to learn stable monotonic alignments from the training corpus.

Original: Furthermore, SAGMM can easily discard non-informative tokens and aggregate information distributed over several remote tokens.
Revised: Furthermore, SAGMM can easily discard non-informative tokens and aggregate information distributed over several remote tokens.

Original: InSAGMM, the  normal distribution parameter ∆ i , σ i , φ i , and µ i are derived fromthe same equation with equation 4 and equation 5.
Revised: In the SAGMM, normal distribution parameter ∆ i , σ i , φ i , and µ i are derived from equation 4 and equation 5.

Original: To encode the contents of  source sequences, the weight for each encoder output δ j is given from the j -th row of the key matrix K as follows:
Revised: To encode the contents of the source sequences, the weight for each encoder output δ j is provided from the j -th row of the key matrix K as follows:

Original: [Equation]
Revised: [Equation]

Original: where W δ ∈ R
Revised: where W δ ∈ R

Original: d
Revised: d

Original: × 1 and the sigmoid function is introduced to smoothly bound the maximum weight δ j similar to Dong & Xu (2020).
Revised: × 1 and the sigmoid function are introduced to smoothly bound the maximum weight δ j similar to Dong & Xu (2020).

Original: Then, the probability of normal distribution N ( ν j ; µ i , σ i ) is calculated from its cumulative sum ν j given by
Revised: Subsequently, the probability of normal distribution N ( ν j ; µ i , σ i ) is calculated from its cumulative sum ν j expressed as

Original: [Equation]
Revised: [Equation]

Original: Finally, the SAGMM attention score α SAGMM is deﬁned as follows:
Revised: Finally, the SAGMM attention score α SAGMM is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where δ j where δ j is multiplied to describe uneven step size in summation.
Revised: where δ j where δ j is multiplied to describe uneven step sizes in summation.

Original: In the early stage of training, we introduce length penalty loss to help learningalignments between µ and ν deﬁned as
Revised: In the early stage of training, we introduced a length penalty loss to facilitate alignment learning between µ and ν ; it is expressed as

Original: [Equation]
Revised: [Equation]

Original: with λ length = 0 .
Revised: with λ length = 0 .

Original: 0005 where I and J denote the length of  decoder and encoder sequences, respectively.
Revised: 0005 where I and J denote the length of the decoder and encoder sequences, respectively.

Original: We turned off  L length after 200K  training steps in  experiments.
Revised: We turned off the L length after 200K of training steps in the experiments.

Original: Finally, we modify 5 as µ i = µ i − 1 + min ( max (∆ i , 0) , 3) to scalethe µ i and ν j similar to  token indices.
Revised: Finally, we modiﬁed 5 as µ i = µ i − 1 + min ( max (∆ i , 0) , 3) to scale µ i and ν j similar to the token indices.

Original: This modiﬁcation helps the interpretability of µ and ν by scaling them.
Revised: This modiﬁcation facilitates the interpretability of µ and ν .

Original: The attention score for each encoder token in  SAGMM is determined independently since they do not rely on  softmax operation over  encoder outputs.
Revised: The attention score for each encoder token in the SAGMM was determined independently because they do not rely on the softmax operation over the encoder outputs.

Original: Note that (cid:80) j δ j N ( ν j ; µ i , σ i ) approximates the integral of  Gaussian distribution.
Revised: It is noteworthy that (cid:80) j δ j N ( ν j ; µ i , σ i ) approximates the integral of the Gaussian distribution.

Original: Thus the sum of  attention weights is close to 1, which helps numerical stability and learning without using softmax.
Revised: Hence, the sum of the attention weights is approximately 1, thereby facilitating numerical stability and learning without using softmax.

Original: 2. SAGMM- TR FOR ONLINE INFERENCE
Revised: 2. SAGMM- TR FOR ONLINE INFERENCE

Original: Since the attention score in SAGMM is generated without softmax normalization over all encoder tokens, we can simply build the attention for streaming inference by cropping the long-tail of the Gaussian distribution.
Revised: Since the attention score in SAGMM is generated without softmax normalization over all encoder tokens, we can simply build the attention for streaming inference by cropping the long-tail of the Gaussian distribution.

Original: In SAGMM-tr, the normal distribution is truncated to limit  past and future context as follows:
Revised: In SAGMM-tr, the normal distribution is truncated to limit the past and future contexts as follows:

Original: [Equation]
Revised: [Equation]

Original: Discarding the tokens with threshold  2 √ σ i removes about 5%percent of  attention score.
Revised: Discarding the tokens with threshold of 2 √ σ i removes approximately 5% of the attention score.

Original: In online inference with  SAGMM-tr, H hi can be calculated after ν j becomes larger than µ i + 2 √ σ i .
Revised: In online inference with the SAGMM-tr, H hi can be calculated after ν j exceeds µ i + 2 √ σ i .

Original: Note that when ν j ≥ µ i + 2 √ σ i satisﬁes in for a current token j = β i , then the equationalso satisﬁes for all j > β i andwe can calculate H hi  without  waiting future contexts.
Revised: It is noteworthy that when ν j ≥ µ i + 2 √ σ i satisﬁes in for a current token j = β i , then the equation satisﬁes for all j > β i and H hi can be emitted without for waiting future contexts.

Original: We initialize the parameters of SAGMM-tr attention with those from  SAGMM model and ﬁne-tuned them until the performance converges.
Revised: We started from the SAGMM model and ﬁne-tuned SAGMM-tr until the performance converged.

Original: We found that the models with  SAGMM-tr showed slightly better performance than SAGMM and mainly reported the results with SAGMM-tr model.
Revised: We discovered that the models with the SAGMM-tr demonstrated slightly better performances than the SAGMM; hence, we mainly report the results involving the SAGMM-tr model herein.

Original: For online speech recognition with SAGMM-tr, we randomly concatenate the 1 -vector after the end of  source sequence with a probability p eos in the training stage and trained to emit the end-of-sequence token EOS  only with the utterances containing  1 -vector.
Revised: For online speech recognition using the SAGMM-tr, we randomly concatenated the 1 -vector after the end of the source sequence with a probability p eos in the training stage and performed training to emit the endof-sequence ( EOS ) token only with the utterances containing the 1 -vector.

Original: The 1 -vector concatenation helps to suppress the EOS token in a long silence part.
Revised: The 1 -vector concatenation suppresses the EOS token in the long silence part.

Original: Finally, we build the uni-directional encoder whose maximum latency is similar to  Zhang et al.
Revised: Finally, we built a unidirectional encoder whose maximum latency was similar to those of Zhang et al.

Original: Dong et al.
Revised: Dong et al.

Original: by adopting the block-wise mask on self-attention layers.
Revised: by adopting a block-wise mask on self-attention layers.

Original: The detailed explanation for the block-wise masking is given in  Appendix.
Revised: A detailed explanation on the block-wise masking is provided in the Appendix.

Original: Algorithm 1 in appendix shows a pseudo code for the SAGMM-tr attention in inference stage.
Revised: Algorithm 1 in the Appendix shows a pseudo code for the SAGMM-tr attention in inference stage.

Original: The number of tokens we need towaits in equation 14 - 16 is given by model parameters.
Revised: The number of tokens required to 16 was determined by model parameters.

Original: To strictly limit the maximum latency of model, we also trained and tested  SAGMM-tr with  ﬁxed attention window width c  in experiments.
Revised: We trained and tested the SAGMM-tr with a ﬁxed attention window width c , to demonstrate the performance in environments with maximum latency constraint.

Original: In this version, equation 14 - 16 are modiﬁed as follows:
Revised: In this version, equation 14 - 16 were modiﬁed as follows:

Original: [Equation]
Revised: [Equation]

Original: In equation 20, we wait c
Revised: In equation 20, we wait c

Original: 2 additional tokens before producing  output.
Revised: 2 additional tokens before producing the output.

Original: We compared the performance of  SAGMM-tr with adaptive and ﬁxed window length in experiments.
Revised: We compared the performance of the SAGMM-tr with adaptive and ﬁxed window widths through experiments.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Chiu & Raffel (2018) propose monotonic chunkwise attention (MoChA) based on hard monotonic sampling over attention window (Raffel et al., 2017).
Revised: Chiu & Raffel (2018) propose monotonic chunkwise attention (MoChA) based on hard monotonic sampling over attention window (Raffel et al., 2017).

Original: The monotonic multi-head attention Ma et al.
Revised: The monotonic multi-head attention Ma et al.

Original: ; Inaguma et al.
Revised: ; Inaguma et al.

Original: (2020) and monotonic inﬁnite lookback attention Arivazhagan et al.
Revised: (2020) and monotonic inﬁnite lookback attention Arivazhagan et al.

Original: (2019) are based on the concept similar to  (Chiu & Raffel, 2018).
Revised: (2019) are based on the concept similar to that reported by (Chiu & Raffel, 2018).

Original: MoChA replaces hard sampling operation to probability distribution over memory in the training stage.
Revised: MoChA replaces hard sampling operation to probability distribution over memory in the training stage.

Original: In contrast,  SAGMM uses the same operations for training and inference.
Revised: By contrast, the SAGMM uses the same operations for training and inference.

Original: Also, MoChA relies on  numerically unstable cumulative product which requires additional efforts for training (Miao et al., 2019; Raffel et al., 2017).
Revised: Furthermore, MoChA relies on a numerically unstable cumulative product that requires additional effort for training (Miao et al., 2019; Raffel et al., 2017).

Original: In contrast,  SAGMM adopts the stable cumulative sum to describe monotonic alignments.
Revised: By contrast, the SAGMM adopts a stable cumulative sum to describe monotonic alignments.

Original: Finally, MoCha is not effective for long-form speech recognition compared to GMM attention Chiu et al.
Revised: Finally, MoCha is not effective for long-form speech recognition compared with GMM attention Chiu et al.

Original: (2019) while our model outperformed  GMM in  “test-long” set in experiments.
Revised: Meanwhile, our model outperformed the GMM in the “test-long” set in experiments.

Original: Dong & Xu (2020) introduce the encoder output weight for ASR task.
Revised: Dong & Xu (2020) introduce the encoder output weight for ASR task.

Original: Compared to their work, our attention mechanism considers the decoder content during attention score calculation.
Revised: By contrast, our attention mechanism considers the decoder content during attention score calculation.

Original:  SAGMMalso allows the overlap between successive attention windows which is helpful to generate semantics that span several decoder tokens.
Revised: Furthermore, the SAGMM allows the overlap between successive attention windows, facilitating the semantics that span several decoder tokens.

Original: Lee et al.
Revised: Lee et al.

Original: (2020) propose the online  attention mechanismwithout softmax operation by aggregating the information of the encoded sequence with update gate.
Revised: (2020) proposed an online softmax-free attention mechanism by aggregating the information of the encoded sequence using an update gate.

Original: Similar to soft attention, their model does not learn monotonic alignment.
Revised: Similar to soft attention, their approach does not explicitly model monotonic alignments.

Original: Also, the update gate for online inference in Lee et al.
Revised: Furthermore, the update gate for online inference in Lee et al.

Original: (2020) is uni-modal.
Revised: (2020) is uni-modal.

Original: The location-aware attention Chorowski et al.
Revised: The location-aware attention Chorowski et al.

Original: ; Watanabe et al.
Revised: ; Watanabe et al.

Original: (2017); Moritz et al.
Revised: (2017); Moritz et al.

Original: (2019) is proposed to employ positional information to the content-based attention mechanism. The location-aware attention introduces  attention scorein the previous step as additional argument for the current step.
Revised: (2019) which introduces previous attention score as additional argument to the current step was proposed to employ positional information in a content-based attention mechanism.

Original: However, it it not streamable attention.
Revised: The location-aware attention improved the performance of ofﬂine inference, whereas we introduced monotonicity in our study for online inference.

Original: The non-attentive neural network-based approaches which do not rely on encoder-decoder attention have been studied by Connectionist Temporal Classiﬁcation (CTC) Graves et al.
Revised: Non-attentive neural network-based approaches that do not rely on encoder-decoder attention have been investigated based on Connectionist Temporal Classiﬁcation (CTC) Graves et al.

Original: ; Liptchinsky et al.
Revised: ; Liptchinsky et al.

Original:  Li et al.
Revised: (2017); Li et al.

Original: (2019),and RNN-transducer (RNN-T)
Revised: (2019), RNN-transducer (RNN-T)

Original: Graves (2012), Transformer-transducer (Transformer-T)
Revised: Graves (2012), Transformer-transducer (Transformer-T)

Original: Zhang et al.
Revised: Zhang et al.

Original: ; Yeh et al.
Revised: ; Yeh et al.

Original: (2019), and Imputer Chan et al.
Revised: (2019), and Imputer Chan et al.

Original: We focus on the Seq2Seq approach which do not require an assumption on sequence lengths and uses the simple beam search algorithm for inference.
Revised: Herein, we focus on the Seq2Seq approach which does not require an assumption regarding sequence lengths and uses the simple beam search algorithm for inference.

Original: Finally, the block-wise inference has been widely studied with manually deﬁned decoding blocks 
Revised: Block-wise inference has been widely investigated using manually deﬁned decoding blocks Jaitly et al.

Original: Jaitly et al. (2015); Tsunoo et al.
Revised: ; Tsunoo et al.

Original: (2020) or joint decoding with CTC model Moritz et al.
Revised: (2020) or joint decoding using a CTC model Moritz et al.

Original: We do not usethe joint CTC decoding Kim et al.
Revised: We doid not use joint CTC decoding Kim et al.

Original: (2016) nor human supervision for the blocked inference.
Revised: (2016) nor human supervision for the blocked inference.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We conducted experiments inthe speech command Warden (2018) and LibriSpeech Panayotov et al.
Revised: We conducted experiments in speech command Warden (2018) and LibriSpeech Panayotov et al.

Original: (2015) datasets to compare the performance of our model in standard, online and long-form ASR tasks.
Revised: (2015) datasets to compare the performances of our model in standard, online and long-form ASR tasks.

Original: We only compare the end-to-end speech recognition model withoutthe external language model (LM) rescoring.
Revised: We only compared the end-to-end speech recognition model without external language model (LM) rescoring.

Original: We also performed preliminary experiments on the translation task to show the performance of the SAGMM-tr on non-monotonic tasks.
Revised: Furthermore, we performed preliminary experiments on the translation task to demonstrate the performance of the SAGMM-tr on non-monotonic tasks.

Original: 4. E XPERIMENTS ON CONCATENATED S PEECH C OMMAND DATASET
Revised: 4. E XPERIMENTS ON CONCATENATED S PEECH C OMMAND DATASET

Original: First, we performed a speech recognition experiment with limited vocabulary to show that the common soft attention algorithm has difﬁculty to decode utterances with unseen sequence lengths.
Revised: First, we performed a speech recognition experiment with limited vocabulary to demonstrate the difﬁculty of using the typical soft attention algorithm in decoding utterances with unseen sequence lengths.

Original: The speech command dataset consists of 1 second long speeches from various speakers saying single words from a vocabulary of 30 words.
Revised: The speech command dataset consists of 1 second long speeches from various speakers uttering single words from a vocabulary of 30 words.

Original: We built 100K utterances of training corpusWarden (2018) by concatenating randomly chosen 5-9 utterances in the training data. 500 test utteranceswere also built by concatenating { 3 , 7 , 10 , 15 , 20 } randomly chosen words in the test set.
Revised: We built 100K utterances in the training corpusWarden (2018) by concatenating randomly selected 5-9 utterances in the training data and 500 test utterances by concatenating { 3 , 7 , 10 , 15 , 20 } randomly selected words in the test set.

Original: We cropped both the start and end of chosen utterances randomly from 0.05s to 0.15s before concatenating them to prevent the SAGMM from ﬁnding a trivial solution.
Revised: We cropped both the start and end of selected utterances randomly from 0.05s to 0.15s before concatenation to prevent the SAGMM from yielding a trivial solution.

Original: We compared the word error rate (WER) of transformers with soft and SAGMM attentions on test sets.
Revised: We compared the word error rate (WER) of transformers with soft and SAGMM attentions on test sets.

Original: Detailed hyperparameters for the experiment are described in Appendix.

Original: In table 1, the soft attention mechanism tends to memorize the sequence length distribution from the training corpus and fails to decode correctly in the unseen sequence length.
Revised: As shown in Table 1, the soft attention mechanism tends to memorize the sequence length distribution from the training corpus and fails to decode correctly in the unseen sequence length.

Original: On the other hand, the proposed SAGMM algorithm was robust to sequence length mismatches.
Revised: Meanwhile, the proposed SAGMM algorithm was robust to sequence length mismatches.

Original: The decoded examples for soft attention and  SAGMM are shown in  Appendix. 
Revised: Detailed hyperparameters for the experiment, decoded examples for soft attention and the SAGMM are shown in the Appendix. number of concatenated words.

Original: [Table]
Revised: [Table]

Original: 4. E XPERIMENTS ON L IBRI S PEECH DATASET
Revised: 4. E XPERIMENTS ON L IBRI S PEECH DATASET

Original: We conducted experiments onthe LibriSpeech whose training dataset consists of 960 hours of read audio books (Panayotov et al., 2015).
Revised: We conducted experiments on LibriSpeech whose training dataset comprised 960h of read audio books (Panayotov et al., 2015).

Original: We validated the performance of  models on dev sets and reported  WERs in  tests sets with the ﬁxed beam width  4 .
Revised: We validated the performance of the models on development sets and reported the WERs in the tests sets with a ﬁxed beam width of 4 .

Original: First, we compared the performance of  SAGMM-tr with conventional algorithms inthe ofﬂine speech recognition.
Revised: First, we compared the performance of the SAGMM-tr with conventional algorithms in ofﬂine speech recognition.

Original: We trained the multi-head transformers similar tothe previous experiments.
Revised: We trained the multi-head transformers similarly to previous experiments.

Original: The encoder consisted of 10 layers of self-attention block with 768 hidden nodes and 3,072 ﬁlters.
Revised: The encoder comprised 10 layers of self-attention blocks, with 768 hidden nodes and 3,072 ﬁlters.

Original: The decoder was constructed by stacking 4 self-attention blocks with the same node and ﬁlter sizes.
Revised: The decoder was constructed by stacking 4 self-attention blocks with the same node and ﬁlter sizes.

Original: The encoder and decoder self-attention employed soft-attention with relative positional encoding (Dai et al., 2019).
Revised: The encoder and decoder self-attention employed soft-attention with relative positional encoding (Dai et al., 2019).

Original: We trained and tested models which employedthe soft attention, GMM attention, and SAGMM-tr attention mechanisms as encoderdecoder attention.
Revised: We trained and tested models that employed soft attention, GMM attention, and SAGMM-tr attention mechanisms as the encoder-decoder attention.

Original: We adopt the auxiliary CTC loss on the encoder output with λ ctc = 0 .
Revised: We adopted the auxiliary CTC loss on the encoder output with λ ctc = 0 .

Original: 1 for  fair comparison with the previous attention-based models Lee et al.
Revised: 1 for a fair comparison with the previous attention-based models Lee et al.

Original: ; Dong & Xu (2020);
Revised: ; Dong & Xu (2020);

Original: Kim et al.
Revised: Kim et al.

Original: The  SAGMM-tr model without CTC loss was 3.84% on test-clean dataset.
Revised: The WER of the SAGMM-tr model without CTC loss was 3.84% on test-clean dataset.

Original: The detail hyperparameters for the experiment and  performance of SAGMM-tr without CTC loss are shown in  Appendix.
Revised: Detailed hyperparameters used in the experiment and the performance of SAGMM-tr without CTC loss are shown in the Appendix.

Original: Table 2 shows the performance of various non-streamable models.
Revised: Table 2 shows the performance of various non-streamable models.

Original: In  non-streamable models, our baseline transformer showed similar performance with previous studies considering the number of parameters.
Revised: In the non-streamable models, our baseline transformer demonstrated performances similar to those of previous studies considering the number of parameters.

Original: We also compared the performance of GMM and SAGMM-tr models with the bi-directional encoder to  other streamable encoder-decoder attention algorithms.
Revised: Furthermore, we compared the performances of GMM and SAGMM-tr models with a bidirectional encoder to those of other streamable decoder algorithms.

Original: The performance ofSAGMM-tr attention matched our transformer model and showed better performance than other streamable Seq2Seq models. From the results,
Revised: The performance of the SAGMM-tr [Table] Streamable decoders

Original: [Table]
Revised: [Table]

Revised: attention was consistent with that of our transformer model, whereas it was better compared with those of other streamable Seq2Seq models.

Original:  SAGMM-tr successfully described the monotonic alignments between speech and transcription without attending  all encoder outputs.
Revised: The results showed that the SAGMM-tr successfully described the monotonic alignments between speech and transcription without attending to all encoder outputs.

Original: Next, we tested the performance of  SAGMM-tr in  online ASR task.
Revised: Next, we investigated the performance of the SAGMM-tr in an online ASR task.

Original: We trainedthe single-head and multihead SAGMM-tr models with the block-wise masked encoder.
Revised: We trained single-head and multi-head SAGMM-tr models using the block-wise masked encoder.

Original: The truncated GMM model was also tested. However, the model attended to the future context as shown in A.4 and failed to learn the correct alignments.
Revised: Additionally, the truncated GMM model was tested, but discarded in experiments because the model attended to the future context as shown in A.4 and failed to learn the correct alignments.

Original: In the multi-head SAGMM-tr, several heads in the ﬁrst decoder layer were not trained and softmax ( φ hi ) went to zero.
Revised: In the multi-head SAGMM-tr, several heads in the ﬁrst decoder layer were not trained and softmax ( φ hi ) approached zero.

Original: We simply pruned these heads after the performance on dev set  converged and ﬁne-tuned foradditional 30K  steps.
Revised: Therefore, we pruned these heads after the performance on the development set had converged and ﬁne-tuned for 30K additional steps.

Original: We believe  trainingwith φ hi and head pruning helps to prevent over-ﬁtting when the desired monotonic alignments do not require many heads especially in the ﬁrst decoder layer.
Revised: We believe that training and head pruning with φ hi can prevent over-ﬁtting when the desired monotonic alignments do not require many heads, particularly in the ﬁrst decoder layer.

Original: Table 3 shows the performance of models with various online inference approaches.
Revised: Table 3 shows the performances of models based on various online inference approaches.

Original: From the table, the single-head SAGMM-tr outperformed other CTC and attention-based algorithms.
Revised: As shown, the single-head SAGMM-tr outperformed other CTC and attention-based algorithms.

Original: The multi-head mechanism further improved the performance of SAGMM-tr.
Revised: The multi-head mechanism further improved the performance of SAGMM-tr.

Original: Compared to the similar model withthe bidirectional encoder, the block-wise maskon encoder self-attention layers increased the WER on testclean set by 0 .
Revised: Adopting unidireciotnal encoder with block-wise mask increased the WER on the test-clean set by 0 .

Original: Our model showed slightly worse performance than state-of-the-art algorithm with transformer-T (Zhang et al., 2020).
Revised: Our model demonstrated slightly worse performance than state-of-the-art algorithms with transformer-T (Zhang et al., 2020).

Original: Subword regularization Kudo (2018); Hannun et al.
Revised: Subword regularization Kudo (2018); Hannun et al.

Original: and sequence-level loss Sabour et al.
Revised: and sequence-level loss Sabour et al.

Original: ; Prabhavalkar et al.
Revised: ; Prabhavalkar et al.

Original: (2018) can help to match theperformance of SAGMM-tr with the best transducer models.
Revised: (2018) may enable the SAGMM-tr to achieve performances similar to the best transducer models.

Revised: Furthermore, the

Original: The SAGMM-tr canalso be adopted to various monotonic Seq2Seq tasks easily since it can be inferred via  simple beam search without probability marginalization in transducers and does not require  assumption that the target shouldnot be longer than the source.
Revised: SAGMM-tr can be adopted in various monotonic Seq2Seq tasks easily; this is because it can be inferred via a simple beam search without probability marginalization in transducers and does not require the assumption that the target should be equal or shorter than the source.

Original: Figure 3 shows the attention plots of the single-head SAGMM-tr which successfully captures alignments between speech and transcription for an utterance with a long silence.
Revised: Figure 3 shows the attention plots of the single-head SAGMM-tr which successfully captures alignments between speech and transcription for an utterance with a long silence.

Original:  Figure 1also shows the attention alignments from multi-head soft attention, truncated GMM attention and SAGMM-tr attentions. Detailed
Revised: Furthermore, Figure 1 shows an Figure speech Each subﬁgure shows attention plot from each encoder-decoder attention layer in decoder with four blocks.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  plots are shown in Appendix A.4.
Revised: attention alignments from multi-head soft attention, truncated GMM attention and SAGMM-tr attentions(the full plots are shown in Appendix A.4).

Original: In soft attention, several heads attends to the speech absence frames constantly for all decoder steps.
Revised: In soft attention, several heads attend to the speech absence frames constantly for every decoder step.

Original: These heads might help utterance-level normalization on  attention context vectors.
Revised: These heads might facilitate in the utterance-level normalization on the attention context vectors.

Revised: In future studied, the role of silence-attending heads in soft attention should be investigated and introduced to the SAGMM-tr attention.

Original: The truncated GMM model failed to learn the alignment and attended to an wide range of encoder tokens.
Revised: The truncated GMM model failed to learn the alignment and attended to a wide range of encoder tokens.

Original: Finally,  SAGMM-tr attention learned  monotonic alignments successfully.
Revised: Finally, the SAGMM-tr attention learned the monotonic alignments successfully.

Original: Due to the monotonic constraint, all heads in SAGMM-tr did not attend silence.

Original: The future work should explore the role of heads attending silence in soft attention and introduce them to the SAGMM-tr attention.

Original: The SAGMM-tr attention plots in  lower layers are more blurred.
Revised: The SAGMM-tr attention plots in the lower layers were more blurred compared with the higher layers.

Original: We suppose that the lower layers ﬁnd the relevant subsets in the sequence and the higher layers use the context vector fromlower layers to select the most relevant tokens in the subset.
Revised: We suppose that the lower layers found the relevant subsets in the sequence and the higher layers utilized the context vectors from the lower encoder-decoder attentions and relatively more focused, partly resembling the behavior of soft attention.

Original: The multi-head SAGMM-tr model showed more blurred alignments compared to the single-head model.
Revised: The multi-head SAGMM-tr model showed more blurred alignments compared with the single-head model.

Original: Our hypothesis for this phenomenon is that several heads attended larger window to provide context information.
Revised: Our hypothesis for this phenomenon is that several heads attended larger windows to provide context information.

Original: The ﬁxed  window length scheme described in equation 17 - 20 or regularization on σ i would be also helpful to minimize the  latency of online inference.
Revised: The ﬁxed attention window width in equation 17 - 20 or regularization on σ i can minimize the maximum latency of online inference.

Original: In this paper, we tested the ﬁxed  window algorithm.
Revised: In this study, we tested the ﬁxed attention window width.

Original: We started with the SAGMM-tr models trained for the online ASR experiment and measured the WERs before and after ﬁne-tuning with the constant window width ( c = 15 frames ) .
Revised: We began with the SAGMM-tr models trained for the online ASR experiment and measured the WERs before and after ﬁne-tuning with a constant window width ( c = 15 frames ) .

Original: From the results in table 4 (a),  SAGMM-tr attention mechanism can bealso adopted to the models with  ﬁxed attention window width.
Revised: From the results in Table 4 (a), the SAGMM-tr attention mechanism can be adopted to the models with a ﬁxed attention window width.

Original: Finally, we concatenated utterances from the same speaker in  “test-clean” set to build  “test-long” set for the long-form speech recognition.
Revised: Finally, we concatenated utterances from the same speaker in the “test-clean” set to build a “test-long” set for the long-form speech recognition.

Original: The average length of utterances in the test-long set was 54 seconds.
Revised: The average length of utterances in the test-long set was 54s.

Original: In the SAGMM-tr model, the beam search algorithm was slightly modiﬁed from Algorithm 1 to suppress the EOS token until the attention window covers ν h,J .
Revised: In the SAGMM-tr model, the beam search algorithm was slightly modiﬁed from Algorithm 1 to suppress the EOS token until the attention window encompassed ν h,J .

Original: The performance of  GMM attention did not improved by the same modiﬁcation since GMM attention attends to ν h,J before the end of  sentence.
Revised: The performance of the GMM attention did not improved by the same modiﬁcation since GMM attention attends to ν h,J prior to the end of a sentence.

Original: From the table 4 (b),  SAGMM-tr outperformed the conventional soft and GMM attention models in longform speech recognition.
Revised: As shown in Table 4 (b), the SAGMM-tr outperformed the conventional soft and GMM attention models in long-form speech recognition.

Original: Comparing the SAGMM-tr models, the multi-head SAGMM-tr showed worse performance than the single-head model.
Revised: Comparing the SAGMM-tr models, the multi-head SAGMM-tr showed worse performance than the single-head model.

Original: The performance degradation of  multi-head mechanism on “testlong” set might come from accumulated attention windows range mismatches.
Revised: The performance degradation of the multi-head mechanism on the “test-long” set might arise from accumulated attention windows range mismatches.

Original: Future works should explore techniques to regularize the attention window ranges in multi-headed models.
Revised: Attention window ranges in multi-headed models should be investigated in future studies.

Original: 4. E XPERIMENTS ONA NON - MONOTONIC TASK
Revised: 4. E XPERIMENTS ON NON - MONOTONIC TASK

Original: The proposed algorithm focuses on strict monotonic tasks which do not allow reordering.
Revised: The proposed algorithm focuses on strict monotonic tasks that do not allow reordering.

Original: However, the selfattention layers in the encoder could reorder the semantics in input sequences before monotonic alignments from the SAGMM-tr.
Revised: However, the selfattention layers in the encoder can reorder the semantics in input sequences before monotonic alignments from the SAGMM-tr.

Original: We conducted machine translation experiments on the WMT EN-DE environment to show the performance of  SAGMM-tr attention in non-monotonic tasks.
Revised: We conducted machine translation experiments on the WMT EN-DE environment to demonstrate the performance of the SAGMM-tr attention in non-monotonic tasks.

Original: While the machine translation is a non-monotonic task, various approaches have been proposed to improve  performance of online inference and simultaneous translation (Arivazhagan et al., 2019; Elbayad et al., 2020).
Revised: Whereas the machine translation is a non-monotonic task, various approaches have been proposed to improve the performance of online inference and simultaneous translation (Arivazhagan et al., 2019; Elbayad et al., 2020).

Original: In this paper, we conduct simple experiments to ﬁgure out if the GMM attention can deals with non-monotonic tasks.
Revised: In this study, we conduct simple experiments to ﬁgure out if the GMM attention can manage non-monotonic tasks.

Original: We trained the encoder-decoder model similar to the transformer-base in Vaswani et al.
Revised: We trained the encoder-decoder model similar to the transformer-base in Vaswani et al.

Original: (2017) except  the head size  4 of the encoder-decoder and decoder self-attention.
Revised: (2017) except that the head size was 4 in the encoder-decoder and decoder self-attention.

Original: We trained  model with 4.5M pairs of sentences and measured the BLEU score on the newstest2013 set after  100K  training steps.
Revised: We trained a model with 4.5M pairs of sentences and measured the BLEU score on the newstest2013 set after performing 100K of training steps.

Original: For  uni-directional encoder, the block-wise mask with M = 5 is applied to  SAGMM-tr model with  bi-directional encoder and  ﬁne-tuned foradditional 40K  steps.
Revised: For the uni-directional encoder, a block-wise mask with M = 5 was applied to the SAGMM-tr model with a bi-directional encoder and then ﬁne-tuned for 40K additional steps.

Original: [Table]
Revised: [Table]

Original: [Table] -enc) 23.
Revised: -tr (uni-enc) 23.

Original: Table 5 shows the performance of soft and SAGMM-tr attention models on the newstest2013 set.
Revised: Table 5 shows the performances of soft and SAGMM-tr attention models on the newstest2013 set.

Original: As expected, the performance of  SAGMM-tr did not match that ofthe conventional transformer models with soft attention.
Revised: As expected, the performance of the SAGMM-tr was not consistent with that of conventional transformer models with soft attention.

Original: The uni-directional encoder also deteriorates the performance of SAGMM-tr similar to  speech recognition experiments.
Revised: Furthermore, the uni-directional encoder deteriorated the performance of the SAGMM-tr, similar to the speech recognition experiments.

Original: Since SAGMM-tr does not considerthe local reordering,  the proposed algorithm has difﬁculty in learning locally non-monotonic functions.
Revised: Because the SAGMM-tr does not consider local reordering, it is difﬁcult for the proposed algorithm to learn locally non-monotonic functions.

Original: Our future works will improve the performance of SAGMM-tr on local reordering tasks by positive constraint relaxation on ∆ i .
Revised: In our future studies, we will improve the performance of SAGMM-tr for local reordering tasks by positive constraint relaxation on ∆ i .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Wehave proposed the SAGMM which attends the subset of source sequences according to the normal distribution on the weighted encoder output axis, considering both contents and order of elements in sequences.
Revised: We proposed the SAGMM, which attends the subset of source sequences according to the normal distribution on a weighted encoder output axis, considering both contents and order of elements in sequences.

Original: We have also proposed  SAGMM-tr for online/real-time inference applications.
Revised: Furthermore, we proposed the SAGMM-tr for online/real-time inference applications.

Original: From the results on various speech recognition tasks,  the proposed attention mechanism is shown to be able to learnthe monotonic alignments between source and target sequences withouta human supervision.
Revised: Based on results on various speech recognition tasks, it was discovered that the proposed attention mechanism can learn monotonic alignments between source and target sequences without human supervision.

Original: The performance of  transformer with  SAGMM-tr improved for online and long-form speech recognition without performance degradation in the standard ofﬂine ASR task.
Revised: The performance of a transformer with the SAGMM-tr improved for online and long-form speech recognition without performance degradation in the standard ofﬂine ASR task.

Original: In future works, we would like to adoptthe SAGMM-based attention mechanisms for natural language processing tasks which allow local reordering duringthe sequence generation.
Revised: In future studies, we plan to adopt SAGMM-based attention mechanisms for natural language processing tasks that allow local reordering during sequence generation.

Original: We are also interested in latency minimization for online inference by controlling µ and σ in both the training and inference stages.
Revised: We are also interested in latency minimization for online inference by controlling µ and σ in both the training and inference stages.

Original: A. P SEUDO CODE FOR ONLINE INFERENCE WITH SAGMM- TR ATTENTION
Revised: A. P SEUDO CODE FOR ONLINE INFERENCE WITH SAGMM- TR ATTENTION

Original: Algorithm h - head.
Revised: Algorithm h - head.

Original: Data: encoder key and value vectors K hj , V hj for j ∈ { 1 , 2 , ...
Revised: Data: encoder key and value vectors K hj , V hj for j ∈ { 1 , 2 , ...

Original: J } , i = 1 , µ h 

Revised: J } , i = 1 , µ h

Original: = ν h 

Revised: = ν h

Original: = 0 , y 

Revised: = 0 , y

Original: = SOS while y i − 1 (cid:54) =
Revised: = SOS while y i − 1 (cid:54) =

Original: EOS do
Revised: EOS do

Original: Q hi = feedforward ( y i − 1 ) /
Revised: Q hi = feedforward ( y i − 1 ) /

Original: * Lower decoder layers */ ∆ hi , σ hi , φ hi =
Revised: * Lower decoder layers */ ∆ hi , σ hi , φ hi =

Original: GMMparam (cid:0) Q hi (cid:1) , µ hi = µ hi − 1 + min (cid:0) max (cid:0) ∆ hi , 0 (cid:1) , 3 (cid:1) /
Revised: GMMparam (cid:0) Q hi (cid:1) , µ hi = µ hi − 1 + min (cid:0) max (cid:0) ∆ hi , 0 (cid:1) , 3 (cid:1) /

Original: * equation 4 */ pos hs , = µ hi − 2 (cid:112) σ hi , pos he = µ hi + 2 (cid:112) σ hi , Attend h =for j = 1 to J do δ hj
Revised: * equation 4 */ pos hs , = µ hi − 2 (cid:112) σ hi , pos he = µ hi + 2 (cid:112) σ hi , Attend h =for j = 1 to J do δ hj

Original: = sigmoid (cid:0) K hj W hδ (cid:1) , ν hj =
Revised: = sigmoid (cid:0) K hj W hδ (cid:1) , ν hj =

Original: ν hj − 1 + δ hj if ν hj > pos he then
Revised: ν hj − 1 + δ hj if ν hj > pos he then

Original: Break else if ν hj > pos hs then α
Revised: Break else if ν hj > pos hs then α

Original: [Equation]
Revised: [Equation]

Original: /* equation 14, 15 */
Revised: /* equation 14, 15 */

Original: Attend h =
Revised: Attend h =

Original: Attend h
Revised: Attend h

Original: + α hSAGMM - tr (cid:107)
Revised: + α hSAGMM - tr (cid:107)

Original: [Equation]
Revised: [Equation]

Original: hj y i
Revised: hj y i

Original: =
Revised: =

Original: Output (cid:0) Decoder (cid:0) softmax h (cid:0) φ hi (cid:1) Attend h (cid:1)(cid:1) , i
Revised: Output (cid:0) Decoder (cid:0) softmax h (cid:0) φ hi (cid:1) Attend h (cid:1)(cid:1) , i

Original: = i
Revised: = i

Original: +
Revised: +

Original: A. E XPERIMENT CONFIGURATIONS
Revised: A. E XPERIMENT CONFIGURATIONS

Original: For all speech recognition experiments, the 80-dimensional log-Mel ﬁlterbank features were extracted with a 25ms window and shifted every 10ms.
Revised: For all speech recognition experiments, the 80-dimensional log-Mel ﬁlterbank features were extracted with a 25ms window and shifted every 10ms.

Original: Each input vector stacked 3 log-Mel feature vectors, downsampled to 30ms frame rate.
Revised: Each input vector stacked 3 log-Mel feature vectors, downsampled to 30ms frame rate.

Original: In the speech command dataset experiment, We trained the multi-headed transformer Vaswani et al.
Revised: In the speech command dataset experiment, We trained the multi-headed transformer Vaswani et al.

Original: Dong et al.
Revised: Dong et al.

Original: (2018) with soft attention and SAGMM attention for encoder-decoder alignments.
Revised: (2018) with soft attention and SAGMM attention for encoder-decoder alignments.

Original: We adopt the relative positional encoding for both the encoder and decoder self-attention (Dai et al., 2019).
Revised: We adopt the relative positional encoding for both the encoder and decoder self-attention (Dai et al., 2019).

Original: The encoder consisted of 6 layers of self-attention block with 512 nodes and 2,048 ﬁlter size, respectively.
Revised: The encoder consisted of 6 layers of self-attention block with 512 nodes and 2,048 ﬁlter size, respectively.

Original: The decoder was constructed by stacking 3 self-attention block with the same node and ﬁlter sizes.
Revised: The decoder was constructed by stacking 3 self-attention block with the same node and ﬁlter sizes.

Original: The number of heads for encoder self attention, encoder-decoder attention, and decoder self attention were { 8 , 4 , 4 } , respectively.
Revised: The number of heads for encoder self attention, encoder-decoder attention, and decoder self attention were { 8 , 4 , 4 } , respectively.

Original: We employed label smoothing of value (cid:15) ls = 0 . 15 , and applied parallel scheduled sampling with probability 0.2 after 100k (Duckworth et al., 2019).
Revised: We employed label smoothing of value (cid:15) ls = 0 . 15 , and applied parallel scheduled sampling with probability 0.2 after 100k (Duckworth et al., 2019).

Original: We used the Adam optimizer with β 1 = 0 .
Revised: We used the Adam optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 and (cid:15) = 10 − 9 ) .
Revised: 98 and (cid:15) = 10 − 9 ) .

Original: The learning rate was linearly increased to 0.1 until 16k steps and set constant in following training steps.
Revised: The learning rate was linearly increased to 0.1 until 16k steps and set constant in following training steps.

Original: We used the tensorﬂow framework and trained the model until 500K step with 4 m40 GPUs.
Revised: We used the tensorﬂow framework and trained the model until 500K step with 4 m40 GPUs.

Original: We averaged 5 last checkpoints before the inference.
Revised: We averaged 5 last checkpoints before the inference.

Original: The dropout ratio for attention weights, rectiﬁed linear units, output of sub-layers, and neural network input after positional encoding were 0.2, 0.1, 0.2, and 0.2, respectively.
Revised: The dropout ratio for attention weights, rectiﬁed linear units, output of sub-layers, and neural network input after positional encoding were 0.2, 0.1, 0.2, and 0.2, respectively.

Original: The Specaug algorithm Park et al.
Revised: The Specaug algorithm Park et al.

Original: (2019) was also applied during training.
Revised: (2019) was also applied during training.

Original: The width of beam was set to 4 in all experiments.
Revised: The width of beam was set to 4 in all experiments.

Original: The output token segments are grepheme of the reference.
Revised: The output token segments are grepheme of the reference.

Original: In the LibriSpeech experiment, The learning rate was linearly increased to 0.1 until 16k steps and set constant until the model converges on dev-clean dataset.
Revised: In the LibriSpeech experiment, The learning rate was linearly increased to 0.1 until 16k steps and set constant until the model converges on dev-clean dataset.

Original: Then, we lowered learning rate to 0.02 and ﬁne-tuned until the model converges.
Revised: Then, we lowered learning rate to 0.02 and ﬁne-tuned until the model converges.

Original: We used the tensorﬂow framework and the model is trained from 7 to 10 days onp40 GPUs.
Revised: We used the tensorﬂow framework and the model is trained from 7 to 10 days onp40 GPUs.

Original: We averaged { 5 , 10 , 15 } last checkpoints before the performance evaluation.
Revised: We averaged { 5 , 10 , 15 } last checkpoints before the performance evaluation.

Original: The 1K wordpiece vocabularies extracted from training dataset is used for output token segmentation.
Revised: The 1K wordpiece vocabularies extracted from training dataset is used for output token segmentation.

Original: Other parameters are the same with those in the speech command dataset experiment.
Revised: Other parameters are the same with those in the speech command dataset experiment.

Original: A. A TTENTION PLOT FOR VARIOUS ENCODER - DECODER MODELS
Revised: A. A TTENTION PLOT FOR VARIOUS ENCODER - DECODER MODELS

Original: To show the SAGMM-tr is able to learn alignments between encoder and decoders, we plot the attention score for single and multi-head models.
Revised: To show the SAGMM-tr is able to learn alignments between encoder and decoders, we plot the attention score for single and multi-head models.

Original: We used the utterance “Chapter eleven ¡long pause¿ the morrow brought a very sober looking morning the sun making only a few efforts to appear and Catherine augured from it everything most favourable to her wishes” in LibriSpeech.
Revised: We used the utterance “Chapter eleven ¡long pause¿ the morrow brought a very sober looking morning the sun making only a few efforts to appear and Catherine augured from it everything most favourable to her wishes” in LibriSpeech.

Original: The truncated GMM model fails to learn the true alignment attended several distinct windows in 5.
Revised: The truncated GMM model fails to learn the true alignment attended several distinct windows in 5.

Original: Surprisingly, the performance of truncated GMM was 4.03% and 10.16% for test-clean and test-other, respectively.
Revised: Surprisingly, the performance of truncated GMM was 4.03% and 10.16% for test-clean and test-other, respectively.

Original: The decent performance of truncated GMM implies that the GMM learns to attend as many frames as possible instead of learning alignments between sequences.
Revised: The decent performance of truncated GMM implies that the GMM learns to attend as many frames as possible instead of learning alignments between sequences.

Original: Therefore, the truncated GMM attention is not suitable for streaming inference.
Revised: Therefore, the truncated GMM attention is not suitable for streaming inference.

Original: the morrow brought a very sober looking morning the sun making only a few efforts to appear and
Revised: the morrow brought a very sober looking morning the sun making only a few efforts to appear and

Original: Catherine augured from it everything most favourable to her wishes” in LibriSpeech
Revised: Catherine augured from it everything most favourable to her wishes” in LibriSpeech.

Original: The attention  in time domain longer than the original wave due to 1 -vector concatenation
Revised: The attention plot in time domain longer than the original wave due to 1 -vector concatenation

Original: In the soft attention model, the several heads learns monotonic alignments between inputs and outputs as shown in Figure 6.
Revised: In the soft attention model, the several heads learns monotonic alignments between inputs and outputs as shown in Figure 6.

Original: However, other heads attend to the whole speech presence or absence intervals.
Revised: However, other heads attend to the whole speech presence or absence intervals.

Original: We assume these heads help utterance-level normalization.
Revised: We assume these heads help utterance-level normalization.

Original: While a few heads show monotonicity, the soft attention is not suitable for online inference due to softmax operation over all frames.
Revised: While a few heads show monotonicity, the soft attention is not suitable for online inference due to softmax operation over all frames.

Original: The attention alignment ﬁgure for single and multi-headed SAGMM-tr model is shown in Figures 7, 8 and 9.
Revised: The attention alignment ﬁgure for single and multi-headed SAGMM-tr model is shown in Figures 7, 8 and 9.

Original: From the ﬁgures, both models learn the optimal alignments between input wave and transcription and distinguish the word boundaries without relying on human knowledge.
Revised: From the ﬁgures, both models learn the optimal alignments between input wave and transcription and distinguish the word boundaries without relying on human knowledge.

Original: The width of attention windows in multi-head SAGMM-tr with adaptive window width are longer than those in the single-head model.
Revised: The width of attention windows in multi-head SAGMM-tr with adaptive window width are longer than those in the single-head model.

Original: The additional objective function for regularizing σ i would be helpful to reduce the window length in ﬁgure 8.
Revised: The additional objective function for regularizing σ i would be helpful to reduce the window width in ﬁgure 8.

Original: In the multi-head model with ﬁxed window length, the model learns the multi-head monotonic alignment without increasing the window width.
Revised: In the multi-head model with ﬁxed window width, the model learns the multi-head monotonic alignment without increasing the window width.

Original: Considering the attention window width as a hyperparameter from external knowledge, it is shown that additional information could improve the model latency in SAGMM-tr while the model can also be trained without human knowledge.
Revised: Considering the attention window width as a hyperparameter from external knowledge, it is shown that additional information could improve the model latency in SAGMM-tr while the model can also be trained without human knowledge.

Original: A. A BLATION STUDIES
Revised: A. A BLATION STUDIES

Original: We tested the three ablation experiments on SAGMM-tr attention in Figure 6.
Revised: We tested the three ablation experiments on SAGMM-tr attention in Figure 6.

Original: First, we tested the model without CTC loss on encoder output.
Revised: First, we tested the model without CTC loss on encoder output.

Original: The CTC loss improves the word error rate performances of model similar to previous studies, but the model showed decent performance without the CTC loss.
Revised: The CTC loss improves the word error rate performances of model similar to previous studies, but the model showed decent performance without the CTC loss.

Original: Second, we tested the SAGMM-tr model with uni-directional encoder without initialization from the bi-directional encoder.
Revised: Second, we tested the SAGMM-tr model with uni-directional encoder without initialization from the bi-directional encoder.

Original: The result shows slightly worse performance compared to the original SAGMM-tr.
Revised: The result shows slightly worse performance compared to the original SAGMM-tr.

Original: Finally, we train the
Revised: Finally, we train the

Original: SAGMM model in the training and decoded with the SAGMM-tr decoding scheme without ﬁne-tuning.
Revised: SAGMM model in the training and decoded with the SAGMM-tr decoding scheme without ﬁne-tuning.

Original: The performance degradation in 6 shows that the mismatch between training and test stages should be removed by the ﬁne-tuning with truncated normal distribution.
Revised: The performance degradation in 6 shows that the mismatch between training and test stages should be removed by the ﬁne-tuning with truncated normal distribution.

Original: [Table]
Revised: [Table]

Original: A TTENTION PLOTS FOR SAGMM- TR IN MACHINE TRANSLATION EXPERIMENTS
Revised: A TTENTION PLOTS FOR SAGMM- TR IN MACHINE TRANSLATION EXPERIMENTS

Original: We plot the ﬁrst three decoder layers of the SAGMM-tr model with bi-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”
Revised: We plot the ﬁrst three decoder layers of the SAGMM-tr model with bi-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”

Original: and model output “Der Patient muss wirklich verstanden werden, um den Grad des Risikos seines Krebses zu verstehen, indem er ihm die M ¨
Revised: and model output “Der Patient muss wirklich verstanden werden, um den Grad des Risikos seines Krebses zu verstehen, indem er ihm die M ¨

Original: oglichkeiten zur Verf ¨ ugung stellt, nicht notwendigerweise mit Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen und sich stattdessen in solchen F ¨ allen f ¨ ur eine aktive ¨Uberwachung der Krankheit entscheiden.”.
Revised: oglichkeiten zur Verf ¨ ugung stellt, nicht notwendigerweise mit Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen und sich stattdessen in solchen F ¨ allen f ¨ ur eine aktive ¨Uberwachung der Krankheit entscheiden.”.

Original: We also plot the ﬁrst three encoder-decoder attention and encoder-self attention layers of the SAGMM-tr model with uni-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”
Revised: We also plot the ﬁrst three encoder-decoder attention and encoder-self attention layers of the SAGMM-tr model with uni-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”

Original: and model output “Der Patient muss wirklich verstanden werden, um das Risiko seines
Revised: and model output “Der Patient muss wirklich verstanden werden, um das Risiko seines

Original: Krebses zu verstehen, indem er ihm die M¨oglichkeiten zur Verf¨ugung stellt, nicht notwendigerweise mit
Revised: Krebses zu verstehen, indem er ihm die M¨oglichkeiten zur Verf¨ugung stellt, nicht notwendigerweise mit

Original: Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen, und stattdessen in solchen F¨allen
Revised: Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen, und stattdessen in solchen F¨allen

Original: f¨ur eine aktive ¨Uberwachung der Krankheit zu entscheiden.”.
Revised: f¨ur eine aktive ¨Uberwachung der Krankheit zu entscheiden.”.

Original: From the ﬁgures 10, 11 and 12, the transformers with SAGMM-tr approximates the machine translation task as a strict monotonic task and ﬁnds optimal alignments under the assumption.
Revised: From the ﬁgures 10, 11 and 12, the transformers with SAGMM-tr approximates the machine translation task as a strict monotonic task and ﬁnds optimal alignments under the assumption.

Original: Since the machine translation is not strictly monotonic task, the performance of the SAGMM-tr attention deteriorates compared to the soft attention.
Revised: Since the machine translation is not strictly monotonic task, the performance of the SAGMM-tr attention deteriorates compared to the soft attention.

Original: However, the transformer with SAGMM-tr attention and uni-directional encoder enables online inference for simultaneous translation.
Revised: However, the transformer with SAGMM-tr attention and uni-directional encoder enables online inference for simultaneous translation.

Original: A. D ECODED EXAMPLES OF SOFT ATTENTION AND SAGMM MODELS ON SPEECH COMMAND AND L IBRI S PEECH DATASETS
Revised: A. D ECODED EXAMPLES OF SOFT ATTENTION AND SAGMM MODELS ON SPEECH COMMAND AND L IBRI S PEECH DATASETS

Original: Table 7 and 8 show the decoded examples from the models with unseen sequence lengths.
Revised: Table 7 and 8 show the decoded examples from the models with unseen sequence lengths.

Original: From the tables, soft attention has difﬁculty to generate the sequences that are signiﬁcantly shorter or longer than the training corpus distribution.
Revised: From the tables, soft attention has difﬁculty to generate the sequences that are signiﬁcantly shorter or longer than the training corpus distribution.

Original: In contrast, the SAGMM-tr is robust to the sequence length mismatch and generates the transcription to arbitrary length.
Revised: In contrast, the SAGMM-tr is robust to the sequence length mismatch and generates the transcription to arbitrary length.

Original: [Table]
Revised: [Table]

Original: Transcription Reference
Revised: Transcription Reference

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in a chord words was it their colours they were voyaging across the deserts of the sky a host of nomads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and woodbegirt and citadelled and of entrenched and marshalled races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in a chord words was it their colours they were voyaging across the deserts of the sky a host of nomads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and woodbegirt and citadelled and of entrenched and marshalled races

Original: Soft attention
Revised: Soft attention

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves to love him as ourselves with the same amount and intensity of love the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and martialed races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves to love him as ourselves with the same amount and intensity of love the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and martialed races

Original: [Table]
Revised: [Table]

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleys and wood be girt and set and of intrenched and marshalled races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleys and wood be girt and set and of intrenched and marshalled races

Original: SAGMM-tr(mh)
Revised: SAGMM-tr(mh)

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the command of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay there beyond the irish sea europe of strange tongues and valleyed and wood girt and citadel and of intrenched and marched and races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the command of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay there beyond the irish sea europe of strange tongues and valleyed and wood girt and citadel and of intrenched and marched and races