Original: L EARNING M ONOTONIC A LIGNMENTS WITH S OURCE -A WARE GMM A TTENTION
Revised: L EARNING M ONOTONIC A LIGNMENTS WITH S OURCE -A WARE GMM A TTENTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers with soft attention have been widely adopted in various sequence-tosequence (Seq2Seq) tasks.
Revised: Transformers with soft attention have been widely adopted in various sequence-tosequence (Seq2Seq) tasks.

Original: Whereas soft attention is effective for learning semantic similarities between queries and keys based on their contents, it does not explicitly model the order of elements in sequences which is crucial for monotonic (Seq2Seq) tasks.
Revised: Whereas soft attention is effective for learning semantic similarities between queries and keys based on their contents, it does not explicitly model the order of elements in sequences which is crucial for monotonic Seq2Seq tasks.

Original: Learning monotonic alignments between input and output sequences may be beneﬁcial for longform and online inference applications that are still challenging for the conventional soft attention algorithm.
Revised: Learning monotonic alignments between input and output sequences may be beneﬁcial for longform and online inference applications that are still challenging for the conventional soft attention algorithm.

Original: Herein, we focus on monotonic Seq2Seq tasks and propose a sourceaware Gaussian mixture model attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence.
Revised: Herein, we focus on monotonic Seq2Seq tasks and propose a sourceaware Gaussian mixture model attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence.

Original: We experimentally demonstrate that the proposed attention mechanism improved the performance on the online and long-form speech recognition problems without performance degradation in ofﬂine in-distribution speech recognition.
Revised: We experimentally demonstrate that the proposed attention mechanism improved the performance on the online and long-form speech recognition problems without performance degradation in ofﬂine in-distribution speech recognition.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, transformer models with soft attention have been widely adopted in various sequence generation tasks (Raffel et al., 2019; Vaswani et al., 2017; Parmar et al., 2018; Karita et al., 2019).
Revised: In recent years, transformer models with soft attention have been widely adopted in various sequence generation tasks (Raffel et al., 2019; Vaswani et al., 2017; Parmar et al., 2018; Karita et al., 2019).

Original: Soft attention does not explicitly model the order of elements in a sequence and attends all encoder outputs for each decoder step.
Revised: Soft attention does not explicitly model the order of elements in a sequence and attends all encoder outputs for each decoder step.

Original: However, the order of elements is crucial for understanding monotonic sequence-to-sequence (Seq2Seq) tasks, such as automatic speech recognition (ASR), video analysis, and lip reading.
Revised: However, the order of elements is crucial for understanding monotonic sequence-to-sequence (Seq2Seq) tasks, such as automatic speech recognition (ASR), video analysis, and lip reading.

Original: Learning monotonic alignments enables the model to attend to a subset of the encoder output without performance degradation in these tasks.
Revised: Learning monotonic alignments enables the model to attend to a subset of the encoder output without performance degradation in these tasks.

Original: In comparison, soft attention is not suitable for streaming inference applications because the softmax operation needs to wait until all encoder outputs are produced.
Revised: In comparison, soft attention is not suitable for streaming inference applications because the softmax operation needs to wait until all encoder outputs are produced.

Original: Figure 1 (b) shows the attention plot for soft-attention.
Revised: Figure 1 (b) shows the attention plot for soft-attention.

Original: Soft attention learns the alignments between queries and keys based on their similarities; it requires all encoder tokens prior to the attention score calculation.
Revised: Soft attention learns the alignments between queries and keys based on their similarities; it requires all encoder tokens prior to the attention score calculation.

Original: Furthermore, soft attention cannot easily decode long-form sequences that are not considered in the training corpus.
Revised: Furthermore, soft attention cannot easily decode long-form sequences that are not considered in the training corpus.

Original: The Gaussian Mixture Model (GMM) attention Graves (2013); Battenberg et al.
Revised: The Gaussian Mixture Model (GMM) attention Graves (2013); Battenberg et al.

Original: Chiu et al.
Revised: Chiu et al.

Original: (2019) have been proposed for learning the monotonic mapping between encoder and decoder states for long-form sequence generation.
Revised: (2019) have been proposed for learning the monotonic mapping between encoder and decoder states for long-form sequence generation.

Original: The GMM attention is a pure location-aware algorithm in which encoder contents are not considered during attention score calculation.
Revised: The GMM attention is a pure location-aware algorithm in which encoder contents are not considered during attention score calculation.

Original: However, each element in the encoder output sequence contains different amounts of information and should be attended considering their contents.
Revised: However, each element in the encoder output sequence contains different amounts of information and should be attended considering their contents.

Original: In ﬁgure 1 (c), the GMM attention fails to learn the detailed alignments and attends to many tokens simultaneously.
Revised: In ﬁgure 1 (c), the GMM attention fails to learn the detailed alignments and attends to many tokens simultaneously.

Original: In this study, we adopted the GMM attention mechanism to the modern transformer structure and proposed the Source-Aware Gaussian Mixture Model (SAGMM) attention which considers both contents and orders of source sequences.
Revised: In this study, we adopted the GMM attention mechanism to the modern transformer structure and proposed the Source-Aware Gaussian Mixture Model (SAGMM) attention which considers both contents and orders of source sequences.

Original: Each component in the SAGMM is multi-modal and discards non-informative tokens in the attention window.
Revised: Each component in the SAGMM is multi-modal and discards non-informative tokens in the attention window.

Original: For online inference, we propose a truncated SAGMM (SAGMM-tr) that discards the long-tail of the attention score in the SAGMM.
Revised: For online inference, we propose a truncated SAGMM (SAGMM-tr) that discards the long-tail of the attention score in the SAGMM.

Original: To the best of our knowledge, this is the ﬁrst attempt to adopt a GMM-based attention to online sequence generation tasks.
Revised: To the best of our knowledge, this is the ﬁrst attempt to adopt a GMM-based attention to online sequence generation tasks.

Original: Learning accurate monotonic alignments enables the SAGMM-tr to attend to a relevant subset of sequences for each decoder step and improves the performance of the model in terms of streaming and long-form sequence generation tasks.
Revised: Learning accurate monotonic alignments enables the SAGMM-tr to attend to a relevant subset of sequences for each decoder step and improves the performance of the model in terms of streaming and long-form sequence generation tasks.

Original: Figure 1 (d) shows the monotonic alignments learned by the SAGMM-tr, enabling online inference.
Revised: Figure 1 (d) shows the monotonic alignments learned by the SAGMM-tr, enabling online inference.

Original: Experiments involving streaming and long-form ASR showed substantial performance improvements compared with conventional algorithms without performance degradation in ofﬂine in-distribution ASR.
Revised: Experiments involving streaming and long-form ASR showed substantial performance improvements compared with conventional algorithms without performance degradation in ofﬂine in-distribution ASR.

Original: Furthermore, we tested the SAGMM-tr in a machine translation task and demonstrated the performance of the proposed algorithm in non-monotonic tasks.
Revised: Furthermore, we tested the SAGMM-tr in a machine translation task and demonstrated the performance of the proposed algorithm in non-monotonic tasks.

Original:  S OURCE - AWARE GMM ATTENTION 2. S OFT ATTENTION
Revised:  S OURCE - AWARE GMM ATTENTION 2. S OFT ATTENTION

Original: Herein, we abbreviate the head index h during attention score calculation for simplicity.
Revised: Herein, we abbreviate the head index h during attention score calculation for simplicity.

Original: In dot-product multi-head soft attention Vaswani et al.
Revised: In dot-product multi-head soft attention Vaswani et al.

Original: (2017) without relative positional encoding , the attention score from soft attention α Soft is derived from the query matrix Q ∈ R
Revised: (2017) without relative positional encoding , the attention score from soft attention α Soft is derived from the query matrix Q ∈ R

Original: I × d and key matrix K ∈ R J × d as follows:
Revised: I × d and key matrix K ∈ R J × d as follows:

Original: [Equation]
Revised: [Equation]

Original: (1) where d, I , and J are the feature dimension, decoder and encoder sequence length, respectively.
Revised: (1) where d, I , and J are the feature dimension, decoder and encoder sequence length, respectively.

Original: The attention context matrix from the h -th head H h and the multi-head output M are expressed as
Revised: The attention context matrix from the h -th head H h and the multi-head output M are expressed as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where α hSoft denotes the α Soft for the h -th head, V h ∈ R J ×
Revised: ) where α hSoft denotes the α Soft for the h -th head, V h ∈ R J ×

Original: d the value matrix for the h -th head, and n h the number of heads.
Revised: d the value matrix for the h -th head, and n h the number of heads.

Original: In this study, we adopted relative positional encoding Shaw et al.
Revised: In this study, we adopted relative positional encoding Shaw et al.

Original: Dai et al.
Revised: Dai et al.

Original: (2019) which provides a stronger baseline for long-form sequence generation for self-attention layers.
Revised: (2019) which provides a stronger baseline for long-form sequence generation for self-attention layers.

Original: 2. GMM ATTENTION
Revised: 2. GMM ATTENTION

Original: The previous studies regarding GMM attention Battenberg et al.
Revised: The previous studies regarding GMM attention Battenberg et al.

Original: Chiu et al.
Revised: Chiu et al.

Original: (2019) were based on early content-based attention (Cho et al., 2014).
Revised: (2019) were based on early content-based attention (Cho et al., 2014).

Original: Li et al.
Revised: Li et al.

Original: (2020) adopted the GMM attention to the transformer framework, but did not provide detailed descriptions.
Revised: (2020) adopted the GMM attention to the transformer framework, but did not provide detailed descriptions.

Original: Herein, we adopt v2 model in Battenberg et al.
Revised: Herein, we adopt v2 model in Battenberg et al.

Original: (2020) which improved the performance of the original GMM attention mechanism Graves (2013).
Revised: (2020) which improved the performance of the original GMM attention mechanism Graves (2013).

Original: We deﬁne the GMM attention as a variant of multi-head attention by considering a Gaussian distribution component as an attention score of single-head in a multi-head mechanism.
Revised: We deﬁne the GMM attention as a variant of multi-head attention by considering a Gaussian distribution component as an attention score of single-head in a multi-head mechanism.

Original: In the study by Battenberg et al.
Revised: In the study by Battenberg et al.

Original: (2020), the value matrix was shared for all Gaussian components, whereas multi-head value matrices were multiplied with the probability from corresponding components in this study to attend to information from different representation subspaces (Vaswani et al., 2017).
Revised: (2020), the value matrix was shared for all Gaussian components, whereas multi-head value matrices were multiplied with the probability from corresponding components in this study to attend to information from different representation subspaces (Vaswani et al., 2017).

Original: Hence, the multi-head GMM attention introduced here is a more generalized algorithm compared with the early GMM attention.
Revised: Hence, the multi-head GMM attention introduced here is a more generalized algorithm compared with the early GMM attention.

Original: Let us denote the i -th row of Q as Q i ∈ R 1 × d .
Revised: Let us denote the i -th row of Q as Q i ∈ R 1 × d .

Original: The normal distribution parameters for the i -th step are expressed
Revised: The normal distribution parameters for the i -th step are expressed

Original: [Equation]
Revised: [Equation]

Original: where ζ ( x ) is a softplus function of x ; W ∆ ∈ R d × 1 , W σ ∈ R d × 1 , and W φ ∈ R d × 1 .
Revised: where ζ ( x ) is a softplus function of x ; W ∆ ∈ R d × 1 , W σ ∈ R d × 1 , and W φ ∈ R d × 1 .

Original: The softplus function was adopted, similar to the study of Battenberg et al.
Revised: The softplus function was adopted, similar to the study of Battenberg et al.

Original: (2020) in which softplus activation demonstrated better performances than the exponential operation.
Revised: (2020) in which softplus activation demonstrated better performances than the exponential operation.

Original: A mixture component of the GMM attention, from the i -th decoder step to the j -th encoder token, is derived from the α GMM (cid:107) i,j is deﬁned as follows:
Revised: A mixture component of the GMM attention, from the i -th decoder step to the j -th encoder token, is derived from the α GMM (cid:107) i,j is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where softmax h denotes the softmax function over heads.
Revised: where softmax h denotes the softmax function over heads.

Original: The conventional GMM attention mechanism is analogous to integral of source sequences with a uniform axis spacing as shown in Figure 2 (a).
Revised: The conventional GMM attention mechanism is analogous to integral of source sequences with a uniform axis spacing as shown in Figure 2 (a).

Original: In this ﬁgure, each rectangle denotes the attention score with speciﬁed
Revised: In this ﬁgure, each rectangle denotes the attention score with speciﬁed

Original: Gaussian component parameters.
Revised: Gaussian component parameters.

Original: The GMM attention assumes that each encoder output is equally important.
Revised: The GMM attention assumes that each encoder output is equally important.

Original: However, this assumption is not satisﬁed for many input modalities, e.g. speech and videos from real environments.
Revised: However, this assumption is not satisﬁed for many input modalities, e.g. speech and videos from real environments.

Original: Moreover, the number of modes in the GMM attention is limited by the number of mixture components.
Revised: Moreover, the number of modes in the GMM attention is limited by the number of mixture components.

Original: To learn robust alignments for monotonic Seq2Seq tasks, we propose the SAGMM which considers both contents and locations for the attention mechanism.
Revised: To learn robust alignments for monotonic Seq2Seq tasks, we propose the SAGMM which considers both contents and locations for the attention mechanism.

Original: 2. S OURCE - AWARE GMM ATTENTION
Revised: 2. S OURCE - AWARE GMM ATTENTION

Original: Figure 2 (b) shows the scheme of the proposed SAGMM attention.
Revised: Figure 2 (b) shows the scheme of the proposed SAGMM attention.

Original: Compared with the GMM attention, the SAGMM is analogous to an integral of normal distribution with non-uniform spacing based on encoder contents.
Revised: Compared with the GMM attention, the SAGMM is analogous to an integral of normal distribution with non-uniform spacing based on encoder contents.

Original: In the ﬁgure 2 (b), the width of rectangle δ j controls the model to selectively attend to the informative tokens during the attention score calculation.
Revised: In the ﬁgure 2 (b), the width of rectangle δ j controls the model to selectively attend to the informative tokens during the attention score calculation.

Original: This content-aware property of the SAGMM enables the model to learn stable monotonic alignments from the training corpus.
Revised: This content-aware property of the SAGMM enables the model to learn stable monotonic alignments from the training corpus.

Original: Furthermore, SAGMM can easily discard non-informative tokens and aggregate information distributed over several remote tokens.
Revised: Furthermore, SAGMM can easily discard non-informative tokens and aggregate information distributed over several remote tokens.

Original: In the SAGMM, normal distribution parameter ∆ i , σ i , φ i , and µ i are derived from equation 4 and equation 5.
Revised: In the SAGMM, normal distribution parameter ∆ i , σ i , φ i , and µ i are derived from equation 4 and equation 5.

Original: To encode the contents of the source sequences, the weight for each encoder output δ j is provided from the j -th row of the key matrix K as follows:
Revised: To encode the contents of the source sequences, the weight for each encoder output δ j is provided from the j -th row of the key matrix K as follows:

Original: [Equation]
Revised: [Equation]

Original: where W δ ∈ R
Revised: where W δ ∈ R

Original: d
Revised: d

Original: × 1 and the sigmoid function are introduced to smoothly bound the maximum weight δ j similar to Dong & Xu (2020).
Revised: × 1 and the sigmoid function are introduced to smoothly bound the maximum weight δ j similar to Dong & Xu (2020).

Original: Subsequently, the probability of normal distribution N ( ν j ; µ i , σ i ) is calculated from its cumulative sum ν
Revised: Subsequently, the probability of normal distribution N ( ν j ; µ i , σ i ) is calculated from its cumulative sum ν

Original: j expressed as
Revised: j expressed as

Original: [Equation]
Revised: [Equation]

Original: Finally, the SAGMM attention score α SAGMM is deﬁned as follows:
Revised: Finally, the SAGMM attention score α SAGMM is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where δ j where δ j is multiplied to describe uneven step sizes in summation.
Revised: where δ j where δ j is multiplied to describe uneven step sizes in summation.

Original: In the early stage of training, we introduced a length penalty loss to facilitate alignment learning between µ and ν ; it is expressed as
Revised: In the early stage of training, we introduced a length penalty loss to facilitate alignment learning between µ and ν ; it is expressed as

Original: [Equation]
Revised: [Equation]

Original: with λ length = 0 .
Revised: with λ length = 0 .

Original: 0005 where I and J denote the length of the decoder and encoder sequences, respectively.
Revised: 0005 where I and J denote the length of the decoder and encoder sequences, respectively.

Original: We turned off the L length after 200K of training steps in the experiments.
Revised: We turned off the L length after 200K of training steps in the experiments.

Original: Finally, we modiﬁed 5 as µ i = µ i − 1 + min
Revised: Finally, we modiﬁed 5 as µ i = µ i − 1 + min

Original: ( max (∆ i , 0) , 3) to scale µ i and ν j similar to the token indices.
Revised: ( max (∆ i , 0) , 3) to scale µ i and ν j similar to the token indices.

Original: This modiﬁcation facilitates the interpretability of µ and ν .
Revised: This modiﬁcation facilitates the interpretability of µ and ν .

Original: The attention score for each encoder token in the SAGMM was determined independently because they do not rely on the softmax operation over the encoder outputs.
Revised: The attention score for each encoder token in the SAGMM was determined independently because they do not rely on the softmax operation over the encoder outputs.

Original: It is noteworthy that (cid:80) j δ j N ( ν j ; µ i , σ i ) approximates the integral of the Gaussian distribution.
Revised: It is noteworthy that (cid:80) j δ j N ( ν j ; µ i , σ i ) approximates the integral of the Gaussian distribution.

Original: Hence, the sum of the attention weights is approximately 1, thereby facilitating numerical stability and learning without using softmax.
Revised: Hence, the sum of the attention weights is approximately 1, thereby facilitating numerical stability and learning without using softmax.

Original: 2. SAGMM- TR FOR ONLINE INFERENCE
Revised: 2. SAGMM- TR FOR ONLINE INFERENCE

Original: Since the attention score in SAGMM is generated without softmax normalization over all encoder tokens, we can simply build the attention for streaming inference by cropping the long-tail of the Gaussian distribution.
Revised: Since the attention score in SAGMM is generated without softmax normalization over all encoder tokens, we can simply build the attention for streaming inference by cropping the long-tail of the Gaussian distribution.

Original: In SAGMM-tr, the normal distribution is truncated to limit the past and future contexts as follows:
Revised: In SAGMM-tr, the normal distribution is truncated to limit the past and future contexts as follows:

Original: [Equation]
Revised: [Equation]

Original: Discarding the tokens with threshold of 2 √ σ i removes approximately 5% of the attention score.
Revised: Discarding the tokens with threshold of 2 √ σ i removes approximately 5% of the attention score.

Original: In online inference with the SAGMM-tr, H hi can be calculated after ν j exceeds µ i + 2 √ σ i .
Revised: In online inference with the SAGMM-tr, H hi can be calculated after ν j exceeds µ i + 2 √ σ i .

Original: It is noteworthy that when ν j ≥ µ i + 2 √ σ i satisﬁes in for a current token j = β i , then the equation satisﬁes for all j > β i and H hi can be emitted without for waiting future contexts.
Revised: It is noteworthy that when ν j ≥ µ i + 2 √ σ i satisﬁes in for a current token j = β i , then the equation satisﬁes for all j > β i and H hi can be emitted without for waiting future contexts.

Original: We started from the SAGMM model and ﬁne-tuned SAGMM-tr until the performance converged.
Revised: We started from the SAGMM model and ﬁne-tuned SAGMM-tr until the performance converged.

Original: We discovered that the models with the SAGMM-tr demonstrated slightly better performances than the SAGMM; hence, we mainly report the results involving the SAGMM-tr model herein.
Revised: We discovered that the models with the SAGMM-tr demonstrated slightly better performances than the SAGMM; hence, we mainly report the results involving the SAGMM-tr model herein.

Original: For online speech recognition using the SAGMM-tr, we randomly concatenated the 1 -vector after the end of the source sequence with a probability p eos in the training stage and performed training to emit the endof-sequence ( EOS ) token only with the utterances containing the 1 -vector.
Revised: For online speech recognition using the SAGMM-tr, we randomly concatenated the 1 -vector after the end of the source sequence with a probability p eos in the training stage and performed training to emit the endof-sequence ( EOS ) token only with the utterances containing the 1 -vector.

Original: The 1 -vector concatenation suppresses the EOS token in the long silence part.
Revised: The 1 -vector concatenation suppresses the EOS token in the long silence part.

Original: Finally, we built a unidirectional encoder whose maximum latency was similar to those of Zhang et al.
Revised: Finally, we built a unidirectional encoder whose maximum latency was similar to those of Zhang et al.

Original: Dong et al.
Revised: Dong et al.

Original: by adopting a block-wise mask on self-attention layers.
Revised: by adopting a block-wise mask on self-attention layers.

Original: A detailed explanation on the block-wise masking is provided in the Appendix.
Revised: A detailed explanation on the block-wise masking is provided in the Appendix.

Original: Algorithm 1 in the Appendix shows a pseudo code for the SAGMM-tr attention in inference stage.
Revised: Algorithm 1 in the Appendix shows a pseudo code for the SAGMM-tr attention in inference stage.

Original: The number of tokens required to 16 was determined by model parameters.
Revised: The number of tokens required to 16 was determined by model parameters.

Original: We trained and tested the
Revised: We trained and tested the

Original: SAGMM-tr with a ﬁxed attention window width c , to demonstrate the performance in environments with maximum latency constraint.
Revised: SAGMM-tr with a ﬁxed attention window width c , to demonstrate the performance in environments with maximum latency constraint.

Original: In this version, equation 14 - 16 were modiﬁed as follows:
Revised: In this version, equation 14 - 16 were modiﬁed as follows:

Original: [Equation]
Revised: [Equation]

Original: In equation 20, we wait c
Revised: In equation 20, we wait c

Original: 2 additional tokens before producing the output.
Revised: 2 additional tokens before producing the output.

Original: We compared the performance of the SAGMM-tr with adaptive and ﬁxed window widths through experiments.
Revised: We compared the performance of the SAGMM-tr with adaptive and ﬁxed window widths through experiments.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Chiu & Raffel (2018) propose monotonic chunkwise attention (MoChA) based on hard monotonic sampling over attention window (Raffel et al., 2017).
Revised: Chiu & Raffel (2018) propose monotonic chunkwise attention (MoChA) based on hard monotonic sampling over attention window (Raffel et al., 2017).

Original: The monotonic multi-head attention Ma et al.
Revised: The monotonic multi-head attention Ma et al.

Original: ; Inaguma et al.
Revised: ; Inaguma et al.

Original: (2020) and monotonic inﬁnite lookback attention Arivazhagan et al.
Revised: (2020) and monotonic inﬁnite lookback attention Arivazhagan et al.

Original: (2019) are based on the concept similar to that reported by (Chiu & Raffel, 2018).
Revised: (2019) are based on the concept similar to that reported by (Chiu & Raffel, 2018).

Original: MoChA replaces hard sampling operation to probability distribution over memory in the training stage.
Revised: MoChA replaces hard sampling operation to probability distribution over memory in the training stage.

Original: By contrast, the SAGMM uses the same operations for training and inference.
Revised: By contrast, the SAGMM uses the same operations for training and inference.

Original: Furthermore, MoChA relies on a numerically unstable cumulative product that requires additional effort for training (Miao et al., 2019; Raffel et al., 2017).
Revised: Furthermore, MoChA relies on a numerically unstable cumulative product that requires additional effort for training (Miao et al., 2019; Raffel et al., 2017).

Original: By contrast, the SAGMM adopts a stable cumulative sum to describe monotonic alignments.
Revised: By contrast, the SAGMM adopts a stable cumulative sum to describe monotonic alignments.

Original: Finally, MoCha is not effective for long-form speech recognition compared with GMM attention Chiu et al.
Revised: Finally, MoCha is not effective for long-form speech recognition compared with GMM attention Chiu et al.

Original: Meanwhile, our model outperformed the GMM in the “test-long” set in experiments.
Revised: Meanwhile, our model outperformed the GMM in the “test-long” set in experiments.

Original: Dong & Xu (2020) introduce the encoder output weight for ASR task.
Revised: Dong & Xu (2020) introduce the encoder output weight for ASR task.

Original: By contrast, our attention mechanism considers the decoder content during attention score calculation.
Revised: By contrast, our attention mechanism considers the decoder content during attention score calculation.

Original: Furthermore, the SAGMM allows the overlap between successive attention windows, facilitating the semantics that span several decoder tokens.
Revised: Furthermore, the SAGMM allows the overlap between successive attention windows, facilitating the semantics that span several decoder tokens.

Original: Lee et al.
Revised: Lee et al.

Original: (2020) proposed an online softmax-free attention mechanism by aggregating the information of the encoded sequence using an update gate.
Revised: (2020) proposed an online softmax-free attention mechanism by aggregating the information of the encoded sequence using an update gate.

Original: Similar to soft attention, their approach does not explicitly model monotonic alignments.
Revised: Similar to soft attention, their approach does not explicitly model monotonic alignments.

Original: Furthermore, the update gate for online inference in Lee et al.
Revised: Furthermore, the update gate for online inference in Lee et al.

Original: (2020) is uni-modal.
Revised: (2020) is uni-modal.

Original: The location-aware attention Chorowski et al.
Revised: The location-aware attention Chorowski et al.

Original: ; Watanabe et al.
Revised: ; Watanabe et al.

Original: (2017); Moritz et al.
Revised: (2017); Moritz et al.

Original: (2019) which introduces previous attention score as additional argument to the current step was proposed to employ positional information in a content-based attention mechanism.
Revised: (2019) which introduces previous attention score as additional argument to the current step was proposed to employ positional information in a content-based attention mechanism.

Original: The location-aware attention improved the performance of ofﬂine inference, whereas we introduced monotonicity in our study for online inference.
Revised: The location-aware attention improved the performance of ofﬂine inference, whereas we introduced monotonicity in our study for online inference.

Original: Non-attentive neural network-based approaches that do not rely on encoder-decoder attention have been investigated based on Connectionist Temporal Classiﬁcation (CTC) Graves et al.
Revised: Non-attentive neural network-based approaches that do not rely on encoder-decoder attention have been investigated based on Connectionist Temporal Classiﬁcation (CTC) Graves et al.

Original: ; Liptchinsky et al.
Revised: ; Liptchinsky et al.

Original: (2017); Li et al.
Revised: (2017); Li et al.

Original: (2019), RNN-transducer (RNN-T)
Revised: (2019), RNN-transducer (RNN-T)

Original: Graves (2012), Transformer-transducer (Transformer-T)
Revised: Graves (2012), Transformer-transducer (Transformer-T)

Original: Zhang et al.
Revised: Zhang et al.

Original: ; Yeh et al.
Revised: ; Yeh et al.

Original: (2019), and Imputer Chan et al.
Revised: (2019), and Imputer Chan et al.

Original: Herein, we focus on the Seq2Seq approach which does not require an assumption regarding sequence lengths and uses the simple beam search algorithm for inference.
Revised: Herein, we focus on the Seq2Seq approach which does not require an assumption regarding sequence lengths and uses the simple beam search algorithm for inference.

Original: Block-wise inference has been widely investigated using manually deﬁned decoding blocks Jaitly et al.
Revised: Block-wise inference has been widely investigated using manually deﬁned decoding blocks Jaitly et al.

Original: ; Tsunoo et al.
Revised: ; Tsunoo et al.

Original: (2020) or joint decoding using a CTC model Moritz et al.
Revised: (2020) or joint decoding using a CTC model Moritz et al.

Original: We doid not use joint CTC decoding Kim et al.
Revised: We doid not use joint CTC decoding Kim et al.

Original: (2016) nor human supervision for the blocked inference.
Revised: (2016) nor human supervision for the blocked inference.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We conducted experiments in speech command Warden (2018) and LibriSpeech Panayotov et al.
Revised: We conducted experiments in speech command Warden (2018) and LibriSpeech Panayotov et al.

Original: (2015) datasets to compare the performances of our model in standard, online and long-form ASR tasks.
Revised: (2015) datasets to compare the performances of our model in standard, online and long-form ASR tasks.

Original: We only compared the end-to-end speech recognition model without external language model (LM) rescoring.
Revised: We only compared the end-to-end speech recognition model without external language model (LM) rescoring.

Original: Furthermore, we performed preliminary experiments on the translation task to demonstrate the performance of the SAGMM-tr on non-monotonic tasks.
Revised: Furthermore, we performed preliminary experiments on the translation task to demonstrate the performance of the SAGMM-tr on non-monotonic tasks.

Original: 4. E XPERIMENTS ON CONCATENATED S PEECH C OMMAND DATASET
Revised: 4. E XPERIMENTS ON CONCATENATED S PEECH C OMMAND DATASET

Original: First, we performed a speech recognition experiment with limited vocabulary to demonstrate the difﬁculty of using the typical soft attention algorithm in decoding utterances with unseen sequence lengths.
Revised: First, we performed a speech recognition experiment with limited vocabulary to demonstrate the difﬁculty of using the typical soft attention algorithm in decoding utterances with unseen sequence lengths.

Original: The speech command dataset consists of 1 second long speeches from various speakers uttering single words from a vocabulary of 30 words.
Revised: The speech command dataset consists of 1 second long speeches from various speakers uttering single words from a vocabulary of 30 words.

Original: We built 100K utterances in the training corpusWarden (2018) by concatenating randomly selected 5-9 utterances in the training data and 500 test utterances by concatenating { 3 , 7 , 10 , 15 , 20 } randomly selected words in the test set.
Revised: We built 100K utterances in the training corpusWarden (2018) by concatenating randomly selected 5-9 utterances in the training data and 500 test utterances by concatenating { 3 , 7 , 10 , 15 , 20 } randomly selected words in the test set.

Original: We cropped both the start and end of selected utterances randomly from 0.05s to 0.15s before concatenation to prevent the SAGMM from yielding a trivial solution.
Revised: We cropped both the start and end of selected utterances randomly from 0.05s to 0.15s before concatenation to prevent the SAGMM from yielding a trivial solution.

Original: We compared the word error rate (WER) of transformers with soft and SAGMM attentions on test sets.
Revised: We compared the word error rate (WER) of transformers with soft and SAGMM attentions on test sets.

Original: As shown in Table 1, the soft attention mechanism tends to memorize the sequence length distribution from the training corpus and fails to decode correctly in the unseen sequence length.
Revised: As shown in Table 1, the soft attention mechanism tends to memorize the sequence length distribution from the training corpus and fails to decode correctly in the unseen sequence length.

Original: Meanwhile, the proposed
Revised: Meanwhile, the proposed

Original: SAGMM algorithm was robust to sequence length mismatches.
Revised: SAGMM algorithm was robust to sequence length mismatches.

Original: Detailed hyperparameters for the experiment, decoded examples for soft attention and the SAGMM are shown in the Appendix.
Revised: Detailed hyperparameters for the experiment, decoded examples for soft attention and the SAGMM are shown in the Appendix.

Original: number of concatenated words.
Revised: number of concatenated words.

Original: [Table]
Revised: [Table]

Original: 4. E XPERIMENTS ON L IBRI S PEECH DATASET
Revised: 4. E XPERIMENTS ON L IBRI S PEECH DATASET

Original: We conducted experiments on LibriSpeech whose training dataset comprised 960h of read audio books (Panayotov et al., 2015).
Revised: We conducted experiments on LibriSpeech whose training dataset comprised 960h of read audio books (Panayotov et al., 2015).

Original: We validated the performance of the models on development sets and reported the
Revised: We validated the performance of the models on development sets and reported the

Original: WERs in the tests sets with a ﬁxed beam width of 4 .
Revised: WERs in the tests sets with a ﬁxed beam width of 4 .

Original: First, we compared the performance of the SAGMM-tr with conventional algorithms in ofﬂine speech recognition.
Revised: First, we compared the performance of the SAGMM-tr with conventional algorithms in ofﬂine speech recognition.

Original: We trained the multi-head transformers similarly to previous experiments.
Revised: We trained the multi-head transformers similarly to previous experiments.

Original: The encoder comprised 10 layers of self-attention blocks, with 768 hidden nodes and 3,072 ﬁlters.
Revised: The encoder comprised 10 layers of self-attention blocks, with 768 hidden nodes and 3,072 ﬁlters.

Original: The decoder was constructed by stacking 4 self-attention blocks with the same node and ﬁlter sizes.
Revised: The decoder was constructed by stacking 4 self-attention blocks with the same node and ﬁlter sizes.

Original: The encoder and decoder self-attention employed soft-attention with relative positional encoding (Dai et al., 2019).
Revised: The encoder and decoder self-attention employed soft-attention with relative positional encoding (Dai et al., 2019).

Original: We trained and tested models that employed soft attention, GMM attention, and SAGMM-tr attention mechanisms as the encoder-decoder attention.
Revised: We trained and tested models that employed soft attention, GMM attention, and SAGMM-tr attention mechanisms as the encoder-decoder attention.

Original: We adopted the auxiliary CTC loss on the encoder output with λ ctc = 0 .
Revised: We adopted the auxiliary CTC loss on the encoder output with λ ctc = 0 .

Original: 1 for a fair comparison with the previous attention-based models Lee et al.
Revised: 1 for a fair comparison with the previous attention-based models Lee et al.

Original: ; Dong & Xu (2020);
Revised: ; Dong & Xu (2020);

Original: Kim et al.
Revised: Kim et al.

Original: The WER of the SAGMM-tr model without CTC loss was 3.84% on test-clean dataset.
Revised: The WER of the SAGMM-tr model without CTC loss was 3.84% on test-clean dataset.

Original: Detailed hyperparameters used in the experiment and the performance of SAGMM-tr without CTC loss are shown in the Appendix.
Revised: Detailed hyperparameters used in the experiment and the performance of SAGMM-tr without CTC loss are shown in the Appendix.

Original: Table 2 shows the performance of various non-streamable models.
Revised: Table 2 shows the performance of various non-streamable models.

Original: In the non-streamable models, our baseline transformer demonstrated performances similar to those of previous studies considering the number of parameters.
Revised: In the non-streamable models, our baseline transformer demonstrated performances similar to those of previous studies considering the number of parameters.

Original: Furthermore, we compared the performances of GMM and SAGMM-tr models with a bidirectional encoder to those of other streamable decoder algorithms.
Revised: Furthermore, we compared the performances of GMM and SAGMM-tr models with a bidirectional encoder to those of other streamable decoder algorithms.

Original: The performance of the SAGMM-tr
Revised: The performance of the SAGMM-tr

Original: [Table]
Revised: [Table]

Original: Streamable decoders
Revised: Streamable decoders

Original: [Table]
Revised: [Table]

Original: attention was consistent with that of our transformer model, whereas it was better compared with those of other streamable Seq2Seq models.
Revised: attention was consistent with that of our transformer model, whereas it was better compared with those of other streamable Seq2Seq models.

Original: The results showed that the SAGMM-tr successfully described the monotonic alignments between speech and transcription without attending to all encoder outputs.
Revised: The results showed that the SAGMM-tr successfully described the monotonic alignments between speech and transcription without attending to all encoder outputs.

Original: Next, we investigated the performance of the SAGMM-tr in an online ASR task.
Revised: Next, we investigated the performance of the SAGMM-tr in an online ASR task.

Original: We trained single-head and multi-head SAGMM-tr models using the block-wise masked encoder.
Revised: We trained single-head and multi-head SAGMM-tr models using the block-wise masked encoder.

Original: Additionally, the truncated GMM model was tested, but discarded in experiments because the model attended to the future context as shown in A.4 and failed to learn the correct alignments.
Revised: Additionally, the truncated GMM model was tested, but discarded in experiments because the model attended to the future context as shown in A.4 and failed to learn the correct alignments.

Original: In the multi-head SAGMM-tr, several heads in the ﬁrst decoder layer were not trained and softmax ( φ hi ) approached zero.
Revised: In the multi-head SAGMM-tr, several heads in the ﬁrst decoder layer were not trained and softmax ( φ hi ) approached zero.

Original: Therefore, we pruned these heads after the performance on the development set had converged and ﬁne-tuned for 30K additional steps.
Revised: Therefore, we pruned these heads after the performance on the development set had converged and ﬁne-tuned for 30K additional steps.

Original: We believe that training and head pruning with φ hi can prevent over-ﬁtting when the desired monotonic alignments do not require many heads, particularly in the ﬁrst decoder layer.
Revised: We believe that training and head pruning with φ hi can prevent over-ﬁtting when the desired monotonic alignments do not require many heads, particularly in the ﬁrst decoder layer.

Original: Table 3 shows the performances of models based on various online inference approaches.
Revised: Table 3 shows the performances of models based on various online inference approaches.

Original: As shown, the single-head SAGMM-tr outperformed other CTC and attention-based algorithms.
Revised: As shown, the single-head SAGMM-tr outperformed other CTC and attention-based algorithms.

Original: The multi-head mechanism further improved the performance of SAGMM-tr.
Revised: The multi-head mechanism further improved the performance of SAGMM-tr.

Original: Adopting unidireciotnal encoder with block-wise mask increased the WER on the test-clean set by 0 .
Revised: Adopting unidireciotnal encoder with block-wise mask increased the WER on the test-clean set by 0 .

Original: Our model demonstrated slightly worse performance than state-of-the-art algorithms with transformer-T (Zhang et al., 2020).
Revised: Our model demonstrated slightly worse performance than state-of-the-art algorithms with transformer-T (Zhang et al., 2020).

Original: Subword regularization
Revised: Subword regularization

Original: Kudo (2018); Hannun et al.
Revised: Kudo (2018); Hannun et al.

Original: and sequence-level loss Sabour et al.
Revised: and sequence-level loss Sabour et al.

Original: ; Prabhavalkar et al.
Revised: ; Prabhavalkar et al.

Original: (2018) may enable the SAGMM-tr to achieve performances similar to the best transducer models.
Revised: (2018) may enable the SAGMM-tr to achieve performances similar to the best transducer models.

Original: Furthermore, the
Revised: Furthermore, the

Original: SAGMM-tr can be adopted in various monotonic Seq2Seq tasks easily; this is because it can be inferred via a simple beam search without probability marginalization in transducers and does not require the assumption that the target should be equal or shorter than the source.
Revised: SAGMM-tr can be adopted in various monotonic Seq2Seq tasks easily; this is because it can be inferred via a simple beam search without probability marginalization in transducers and does not require the assumption that the target should be equal or shorter than the source.

Original: Figure 3 shows the attention plots of the single-head SAGMM-tr which successfully captures alignments between speech and transcription for an utterance with a long silence.
Revised: Figure 3 shows the attention plots of the single-head SAGMM-tr which successfully captures alignments between speech and transcription for an utterance with a long silence.

Original: Furthermore, Figure 1 shows an
Revised: Furthermore, Figure 1 shows an

Original: Figure speech
Revised: Figure speech

Original: Each subﬁgure shows attention plot from each encoder-decoder attention layer in decoder with four blocks.
Revised: Each subﬁgure shows attention plot from each encoder-decoder attention layer in decoder with four blocks.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: attention alignments from multi-head soft attention, truncated GMM attention and SAGMM-tr attentions(the full plots are shown in Appendix A.4).
Revised: attention alignments from multi-head soft attention, truncated GMM attention and SAGMM-tr attentions(the full plots are shown in Appendix A.4).

Original: In soft attention, several heads attend to the speech absence frames constantly for every decoder step.
Revised: In soft attention, several heads attend to the speech absence frames constantly for every decoder step.

Original: These heads might facilitate in the utterance-level normalization on the attention context vectors.
Revised: These heads might facilitate in the utterance-level normalization on the attention context vectors.

Original: In future studied, the role of silence-attending heads in soft attention should be investigated and introduced to the SAGMM-tr attention.
Revised: In future studied, the role of silence-attending heads in soft attention should be investigated and introduced to the SAGMM-tr attention.

Original: The truncated GMM model failed to learn the alignment and attended to a wide range of encoder tokens.
Revised: The truncated GMM model failed to learn the alignment and attended to a wide range of encoder tokens.

Original: Finally, the SAGMM-tr attention learned the monotonic alignments successfully.
Revised: Finally, the SAGMM-tr attention learned the monotonic alignments successfully.

Original: The SAGMM-tr attention plots in the lower layers were more blurred compared with the higher layers.
Revised: The SAGMM-tr attention plots in the lower layers were more blurred compared with the higher layers.

Original: We suppose that the lower layers found the relevant subsets in the sequence and the higher layers utilized the context vectors from the lower encoder-decoder attentions and relatively more focused, partly resembling the behavior of soft attention.
Revised: We suppose that the lower layers found the relevant subsets in the sequence and the higher layers utilized the context vectors from the lower encoder-decoder attentions and relatively more focused, partly resembling the behavior of soft attention.

Original: The multi-head SAGMM-tr model showed more blurred alignments compared with the single-head model.
Revised: The multi-head SAGMM-tr model showed more blurred alignments compared with the single-head model.

Original: Our hypothesis for this phenomenon is that several heads attended larger windows to provide context information.
Revised: Our hypothesis for this phenomenon is that several heads attended larger windows to provide context information.

Original: The ﬁxed attention window width in equation 17 - 20 or regularization on σ i can minimize the maximum latency of online inference.
Revised: The ﬁxed attention window width in equation 17 - 20 or regularization on σ i can minimize the maximum latency of online inference.

Original: In this study, we tested the ﬁxed attention window width.
Revised: In this study, we tested the ﬁxed attention window width.

Original: We began with the SAGMM-tr models trained for the online ASR experiment and measured the WERs before and after ﬁne-tuning with a constant window width ( c = 15 frames ) .
Revised: We began with the SAGMM-tr models trained for the online ASR experiment and measured the WERs before and after ﬁne-tuning with a constant window width ( c = 15 frames ) .

Original: From the results in Table 4 (a), the SAGMM-tr attention mechanism can be adopted to the models with a ﬁxed attention window width.
Revised: From the results in Table 4 (a), the SAGMM-tr attention mechanism can be adopted to the models with a ﬁxed attention window width.

Original: Finally, we concatenated utterances from the same speaker in the “test-clean” set to build a “test-long” set for the long-form speech recognition.
Revised: Finally, we concatenated utterances from the same speaker in the “test-clean” set to build a “test-long” set for the long-form speech recognition.

Original: The average length of utterances in the test-long set was 54s.
Revised: The average length of utterances in the test-long set was 54s.

Original: In the SAGMM-tr model, the beam search algorithm was slightly modiﬁed from Algorithm 1 to suppress the
Revised: In the SAGMM-tr model, the beam search algorithm was slightly modiﬁed from Algorithm 1 to suppress the

Original: EOS token until the attention window encompassed ν h,J .
Revised: EOS token until the attention window encompassed ν h,J .

Original: The performance of the GMM attention did not improved by the same modiﬁcation since GMM attention attends to ν h,J prior to the end of a sentence.
Revised: The performance of the GMM attention did not improved by the same modiﬁcation since GMM attention attends to ν h,J prior to the end of a sentence.

Original: As shown in Table 4 (b), the SAGMM-tr outperformed the conventional soft and GMM attention models in long-form speech recognition.
Revised: As shown in Table 4 (b), the SAGMM-tr outperformed the conventional soft and GMM attention models in long-form speech recognition.

Original: Comparing the SAGMM-tr models, the multi-head SAGMM-tr showed worse performance than the single-head model.
Revised: Comparing the SAGMM-tr models, the multi-head SAGMM-tr showed worse performance than the single-head model.

Original: The performance degradation of the multi-head mechanism on the “test-long” set might arise from accumulated attention windows range mismatches.
Revised: The performance degradation of the multi-head mechanism on the “test-long” set might arise from accumulated attention windows range mismatches.

Original: Attention window ranges in multi-headed models should be investigated in future studies.
Revised: Attention window ranges in multi-headed models should be investigated in future studies.

Original: 4. E XPERIMENTS ON NON - MONOTONIC TASK
Revised: 4. E XPERIMENTS ON NON - MONOTONIC TASK

Original: The proposed algorithm focuses on strict monotonic tasks that do not allow reordering.
Revised: The proposed algorithm focuses on strict monotonic tasks that do not allow reordering.

Original: However, the selfattention layers in the encoder can reorder the semantics in input sequences before monotonic alignments from the SAGMM-tr.
Revised: However, the selfattention layers in the encoder can reorder the semantics in input sequences before monotonic alignments from the SAGMM-tr.

Original: We conducted machine translation experiments on the WMT EN-DE environment to demonstrate the performance of the SAGMM-tr attention in non-monotonic tasks.
Revised: We conducted machine translation experiments on the WMT EN-DE environment to demonstrate the performance of the SAGMM-tr attention in non-monotonic tasks.

Original: Whereas the machine translation is a non-monotonic task, various approaches have been proposed to improve the performance of online inference and simultaneous translation (Arivazhagan et al., 2019; Elbayad et al., 2020).
Revised: Whereas the machine translation is a non-monotonic task, various approaches have been proposed to improve the performance of online inference and simultaneous translation (Arivazhagan et al., 2019; Elbayad et al., 2020).

Original: In this study, we conduct simple experiments to ﬁgure out if the GMM attention can manage non-monotonic tasks.
Revised: In this study, we conduct simple experiments to ﬁgure out if the GMM attention can manage non-monotonic tasks.

Original: We trained the encoder-decoder model similar to the transformer-base in Vaswani et al.
Revised: We trained the encoder-decoder model similar to the transformer-base in Vaswani et al.

Original: (2017) except that the head size was 4 in the encoder-decoder and decoder self-attention.
Revised: (2017) except that the head size was 4 in the encoder-decoder and decoder self-attention.

Original: We trained a model with 4.5M pairs of sentences and measured the BLEU score on the newstest2013 set after performing 100K of training steps.
Revised: We trained a model with 4.5M pairs of sentences and measured the BLEU score on the newstest2013 set after performing 100K of training steps.

Original: For the uni-directional encoder, a block-wise mask with M = 5 was applied to the SAGMM-tr model with a bi-directional encoder and then ﬁne-tuned for 40K additional steps.
Revised: For the uni-directional encoder, a block-wise mask with M = 5 was applied to the SAGMM-tr model with a bi-directional encoder and then ﬁne-tuned for 40K additional steps.

Original: [Table]
Revised: [Table]

Original: -tr (uni-enc) 23.
Revised: -tr (uni-enc) 23.

Original: Table 5 shows the performances of soft and SAGMM-tr attention models on the newstest2013 set.
Revised: Table 5 shows the performances of soft and SAGMM-tr attention models on the newstest2013 set.

Original: As expected, the performance of the SAGMM-tr was not consistent with that of conventional transformer models with soft attention.
Revised: As expected, the performance of the SAGMM-tr was not consistent with that of conventional transformer models with soft attention.

Original: Furthermore, the uni-directional encoder deteriorated the performance of the SAGMM-tr, similar to the speech recognition experiments.
Revised: Furthermore, the uni-directional encoder deteriorated the performance of the SAGMM-tr, similar to the speech recognition experiments.

Original: Because the
Revised: Because the

Original: SAGMM-tr does not consider local reordering, it is difﬁcult for the proposed algorithm to learn locally non-monotonic functions.
Revised: SAGMM-tr does not consider local reordering, it is difﬁcult for the proposed algorithm to learn locally non-monotonic functions.

Original: In our future studies, we will improve the performance of SAGMM-tr for local reordering tasks by positive constraint relaxation on ∆ i .
Revised: In our future studies, we will improve the performance of SAGMM-tr for local reordering tasks by positive constraint relaxation on ∆ i .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed the SAGMM, which attends the subset of source sequences according to the normal distribution on a weighted encoder output axis, considering both contents and order of elements in sequences.
Revised: We proposed the SAGMM, which attends the subset of source sequences according to the normal distribution on a weighted encoder output axis, considering both contents and order of elements in sequences.

Original: Furthermore, we proposed the SAGMM-tr for online/real-time inference applications.
Revised: Furthermore, we proposed the SAGMM-tr for online/real-time inference applications.

Original: Based on results on various speech recognition tasks, it was discovered that the proposed attention mechanism can learn monotonic alignments between source and target sequences without human supervision.
Revised: Based on results on various speech recognition tasks, it was discovered that the proposed attention mechanism can learn monotonic alignments between source and target sequences without human supervision.

Original: The performance of a transformer with the SAGMM-tr improved for online and long-form speech recognition without performance degradation in the standard ofﬂine ASR task.
Revised: The performance of a transformer with the SAGMM-tr improved for online and long-form speech recognition without performance degradation in the standard ofﬂine ASR task.

Original: In future studies, we plan to adopt SAGMM-based attention mechanisms for natural language processing tasks that allow local reordering during sequence generation.
Revised: In future studies, we plan to adopt SAGMM-based attention mechanisms for natural language processing tasks that allow local reordering during sequence generation.

Original: We are also interested in latency minimization for online inference by controlling µ and σ in both the training and inference stages.
Revised: We are also interested in latency minimization for online inference by controlling µ and σ in both the training and inference stages.

Original: A. P SEUDO CODE FOR ONLINE INFERENCE WITH SAGMM- TR ATTENTION
Revised: A. P SEUDO CODE FOR ONLINE INFERENCE WITH SAGMM- TR ATTENTION

Original: Algorithm h - head.
Revised: Algorithm h - head.

Original: Data: encoder key and value vectors K hj , V hj for j ∈ { 1 , 2 , ...
Revised: Data: encoder key and value vectors K hj , V hj for j ∈ { 1 , 2 , ...

Original: J } , i = 1 , µ h 

Revised: J } , i = 1 , µ h

Original: = ν h 

Revised: = ν h

Original: = 0 , y 

Revised: = 0 , y

Original: = SOS while y i − 1 (cid:54) =
Revised: = SOS while y i − 1 (cid:54) =

Original: EOS do
Revised: EOS do

Original: Q hi = feedforward ( y i − 1 ) /
Revised: Q hi = feedforward ( y i − 1 ) /

Original: * Lower decoder layers */ ∆ hi , σ hi , φ hi =
Revised: * Lower decoder layers */ ∆ hi , σ hi , φ hi =

Original: GMMparam (cid:0) Q hi (cid:1) , µ hi = µ hi − 1 + min (cid:0) max (cid:0) ∆ hi , 0 (cid:1) , 3 (cid:1) /
Revised: GMMparam (cid:0) Q hi (cid:1) , µ hi = µ hi − 1 + min (cid:0) max (cid:0) ∆ hi , 0 (cid:1) , 3 (cid:1) /

Original: * equation 4 */ pos hs , = µ hi − 2 (cid:112) σ hi , pos he = µ hi + 2 (cid:112) σ hi , Attend h =for j = 1 to J do δ hj
Revised: * equation 4 */ pos hs , = µ hi − 2 (cid:112) σ hi , pos he = µ hi + 2 (cid:112) σ hi , Attend h =for j = 1 to J do δ hj

Original: = sigmoid (cid:0) K hj W hδ (cid:1) , ν hj =
Revised: = sigmoid (cid:0) K hj W hδ (cid:1) , ν hj =

Original: ν hj − 1 + δ hj if ν hj > pos he then
Revised: ν hj − 1 + δ hj if ν hj > pos he then

Original: Break else if ν hj > pos hs then α
Revised: Break else if ν hj > pos hs then α

Original: [Equation]
Revised: [Equation]

Original: /* equation 14, 15 */
Revised: /* equation 14, 15 */

Original: Attend h =
Revised: Attend h =

Original: Attend h
Revised: Attend h

Original: + α hSAGMM - tr (cid:107)
Revised: + α hSAGMM - tr (cid:107)

Original: [Equation]
Revised: [Equation]

Original: hj y i
Revised: hj y i

Original: =
Revised: =

Original: Output (cid:0) Decoder (cid:0) softmax h (cid:0) φ hi (cid:1) Attend h (cid:1)(cid:1) , i
Revised: Output (cid:0) Decoder (cid:0) softmax h (cid:0) φ hi (cid:1) Attend h (cid:1)(cid:1) , i

Original: = i
Revised: = i

Original: +
Revised: +

Original: A. E XPERIMENT CONFIGURATIONS
Revised: A. E XPERIMENT CONFIGURATIONS

Original: For all speech recognition experiments, the 80-dimensional log-Mel ﬁlterbank features were extracted with a 25ms window and shifted every 10ms.
Revised: For all speech recognition experiments, the 80-dimensional log-Mel ﬁlterbank features were extracted with a 25ms window and shifted every 10ms.

Original: Each input vector stacked 3 log-Mel feature vectors, downsampled to 30ms frame rate.
Revised: Each input vector stacked 3 log-Mel feature vectors, downsampled to 30ms frame rate.

Original: In the speech command dataset experiment, We trained the multi-headed transformer Vaswani et al.
Revised: In the speech command dataset experiment, We trained the multi-headed transformer Vaswani et al.

Original: Dong et al.
Revised: Dong et al.

Original: (2018) with soft attention and SAGMM attention for encoder-decoder alignments.
Revised: (2018) with soft attention and SAGMM attention for encoder-decoder alignments.

Original: We adopt the relative positional encoding for both the encoder and decoder self-attention (Dai et al., 2019).
Revised: We adopt the relative positional encoding for both the encoder and decoder self-attention (Dai et al., 2019).

Original: The encoder consisted of 6 layers of self-attention block with 512 nodes and 2,048 ﬁlter size, respectively.
Revised: The encoder consisted of 6 layers of self-attention block with 512 nodes and 2,048 ﬁlter size, respectively.

Original: The decoder was constructed by stacking 3 self-attention block with the same node and ﬁlter sizes.
Revised: The decoder was constructed by stacking 3 self-attention block with the same node and ﬁlter sizes.

Original: The number of heads for encoder self attention, encoder-decoder attention, and decoder self attention were { 8 , 4 , 4 } , respectively.
Revised: The number of heads for encoder self attention, encoder-decoder attention, and decoder self attention were { 8 , 4 , 4 } , respectively.

Original: We employed label smoothing of value (cid:15) ls = 0 . 15 , and applied parallel scheduled sampling with probability 0.2 after 100k (Duckworth et al., 2019).
Revised: We employed label smoothing of value (cid:15) ls = 0 . 15 , and applied parallel scheduled sampling with probability 0.2 after 100k (Duckworth et al., 2019).

Original: We used the Adam optimizer with β 1 = 0 .
Revised: We used the Adam optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 and (cid:15) = 10 − 9 ) .
Revised: 98 and (cid:15) = 10 − 9 ) .

Original: The learning rate was linearly increased to 0.1 until 16k steps and set constant in following training steps.
Revised: The learning rate was linearly increased to 0.1 until 16k steps and set constant in following training steps.

Original: We used the tensorﬂow framework and trained the model until 500K step with 4 m40 GPUs.
Revised: We used the tensorﬂow framework and trained the model until 500K step with 4 m40 GPUs.

Original: We averaged 5 last checkpoints before the inference.
Revised: We averaged 5 last checkpoints before the inference.

Original: The dropout ratio for attention weights, rectiﬁed linear units, output of sub-layers, and neural network input after positional encoding were 0.2, 0.1, 0.2, and 0.2, respectively.
Revised: The dropout ratio for attention weights, rectiﬁed linear units, output of sub-layers, and neural network input after positional encoding were 0.2, 0.1, 0.2, and 0.2, respectively.

Original: The Specaug algorithm Park et al.
Revised: The Specaug algorithm Park et al.

Original: (2019) was also applied during training.
Revised: (2019) was also applied during training.

Original: The width of beam was set to 4 in all experiments.
Revised: The width of beam was set to 4 in all experiments.

Original: The output token segments are grepheme of the reference.
Revised: The output token segments are grepheme of the reference.

Original: In the LibriSpeech experiment, The learning rate was linearly increased to 0.1 until 16k steps and set constant until the model converges on dev-clean dataset.
Revised: In the LibriSpeech experiment, The learning rate was linearly increased to 0.1 until 16k steps and set constant until the model converges on dev-clean dataset.

Original: Then, we lowered learning rate to 0.02 and ﬁne-tuned until the model converges.
Revised: Then, we lowered learning rate to 0.02 and ﬁne-tuned until the model converges.

Original: We used the tensorﬂow framework and the model is trained from 7 to 10 days onp40 GPUs.
Revised: We used the tensorﬂow framework and the model is trained from 7 to 10 days onp40 GPUs.

Original: We averaged { 5 , 10 , 15 } last checkpoints before the performance evaluation.
Revised: We averaged { 5 , 10 , 15 } last checkpoints before the performance evaluation.

Original: The 1K wordpiece vocabularies extracted from training dataset is used for output token segmentation.
Revised: The 1K wordpiece vocabularies extracted from training dataset is used for output token segmentation.

Original: Other parameters are the same with those in the speech command dataset experiment.
Revised: Other parameters are the same with those in the speech command dataset experiment.

Original: A. A TTENTION PLOT FOR VARIOUS ENCODER - DECODER MODELS
Revised: A. A TTENTION PLOT FOR VARIOUS ENCODER - DECODER MODELS

Original: To show the SAGMM-tr is able to learn alignments between encoder and decoders, we plot the attention score for single and multi-head models.
Revised: To show the SAGMM-tr is able to learn alignments between encoder and decoders, we plot the attention score for single and multi-head models.

Original: We used the utterance “Chapter eleven ¡long pause¿ the morrow brought a very sober looking morning the sun making only a few efforts to appear and Catherine augured from it everything most favourable to her wishes” in LibriSpeech.
Revised: We used the utterance “Chapter eleven ¡long pause¿ the morrow brought a very sober looking morning the sun making only a few efforts to appear and Catherine augured from it everything most favourable to her wishes” in LibriSpeech.

Original: The truncated GMM model fails to learn the true alignment attended several distinct windows in 5.
Revised: The truncated GMM model fails to learn the true alignment attended several distinct windows in 5.

Original: Surprisingly, the performance of truncated GMM was 4.03% and 10.16% for test-clean and test-other, respectively.
Revised: Surprisingly, the performance of truncated GMM was 4.03% and 10.16% for test-clean and test-other, respectively.

Original: The decent performance of truncated GMM implies that the GMM learns to attend as many frames as possible instead of learning alignments between sequences.
Revised: The decent performance of truncated GMM implies that the GMM learns to attend as many frames as possible instead of learning alignments between sequences.

Original: Therefore, the truncated GMM attention is not suitable for streaming inference.
Revised: Therefore, the truncated GMM attention is not suitable for streaming inference.

Original: the morrow brought a very sober looking morning the sun making only a few efforts to appear and
Revised: the morrow brought a very sober looking morning the sun making only a few efforts to appear and

Original: Catherine augured from it everything most favourable to her wishes” in LibriSpeech.
Revised: Catherine augured from it everything most favourable to her wishes” in LibriSpeech.

Original: The attention plot in time domain longer than the original wave due to 1 -vector concatenation
Revised: The attention plot in time domain longer than the original wave due to 1 -vector concatenation

Original: In the soft attention model, the several heads learns monotonic alignments between inputs and outputs as shown in Figure 6.
Revised: In the soft attention model, the several heads learns monotonic alignments between inputs and outputs as shown in Figure 6.

Original: However, other heads attend to the whole speech presence or absence intervals.
Revised: However, other heads attend to the whole speech presence or absence intervals.

Original: We assume these heads help utterance-level normalization.
Revised: We assume these heads help utterance-level normalization.

Original: While a few heads show monotonicity, the soft attention is not suitable for online inference due to softmax operation over all frames.
Revised: While a few heads show monotonicity, the soft attention is not suitable for online inference due to softmax operation over all frames.

Original: The attention alignment ﬁgure for single and multi-headed SAGMM-tr model is shown in Figures 7, 8 and 9.
Revised: The attention alignment ﬁgure for single and multi-headed SAGMM-tr model is shown in Figures 7, 8 and 9.

Original: From the ﬁgures, both models learn the optimal alignments between input wave and transcription and distinguish the word boundaries without relying on human knowledge.
Revised: From the ﬁgures, both models learn the optimal alignments between input wave and transcription and distinguish the word boundaries without relying on human knowledge.

Original: The width of attention windows in multi-head SAGMM-tr with adaptive window width are longer than those in the single-head model.
Revised: The width of attention windows in multi-head SAGMM-tr with adaptive window width are longer than those in the single-head model.

Original: The additional objective function for regularizing σ i would be helpful to reduce the window width in ﬁgure 8.
Revised: The additional objective function for regularizing σ i would be helpful to reduce the window width in ﬁgure 8.

Original: In the multi-head model with ﬁxed window width, the model learns the multi-head monotonic alignment without increasing the window width.
Revised: In the multi-head model with ﬁxed window width, the model learns the multi-head monotonic alignment without increasing the window width.

Original: Considering the attention window width as a hyperparameter from external knowledge, it is shown that additional information could improve the model latency in SAGMM-tr while the model can also be trained without human knowledge.
Revised: Considering the attention window width as a hyperparameter from external knowledge, it is shown that additional information could improve the model latency in SAGMM-tr while the model can also be trained without human knowledge.

Original: A. A BLATION STUDIES
Revised: A. A BLATION STUDIES

Original: We tested the three ablation experiments on SAGMM-tr attention in Figure 6.
Revised: We tested the three ablation experiments on SAGMM-tr attention in Figure 6.

Original: First, we tested the model without CTC loss on encoder output.
Revised: First, we tested the model without CTC loss on encoder output.

Original: The CTC loss improves the word error rate performances of model similar to previous studies, but the model showed decent performance without the CTC loss.
Revised: The CTC loss improves the word error rate performances of model similar to previous studies, but the model showed decent performance without the CTC loss.

Original: Second, we tested the SAGMM-tr model with uni-directional encoder without initialization from the bi-directional encoder.
Revised: Second, we tested the SAGMM-tr model with uni-directional encoder without initialization from the bi-directional encoder.

Original: The result shows slightly worse performance compared to the original SAGMM-tr.
Revised: The result shows slightly worse performance compared to the original SAGMM-tr.

Original: Finally, we train the
Revised: Finally, we train the

Original: SAGMM model in the training and decoded with the SAGMM-tr decoding scheme without ﬁne-tuning.
Revised: SAGMM model in the training and decoded with the SAGMM-tr decoding scheme without ﬁne-tuning.

Original: The performance degradation in 6 shows that the mismatch between training and test stages should be removed by the ﬁne-tuning with truncated normal distribution.
Revised: The performance degradation in 6 shows that the mismatch between training and test stages should be removed by the ﬁne-tuning with truncated normal distribution.

Original: [Table]
Revised: [Table]

Original: A TTENTION PLOTS FOR SAGMM- TR IN MACHINE TRANSLATION EXPERIMENTS
Revised: A TTENTION PLOTS FOR SAGMM- TR IN MACHINE TRANSLATION EXPERIMENTS

Original: We plot the ﬁrst three decoder layers of the SAGMM-tr model with bi-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”
Revised: We plot the ﬁrst three decoder layers of the SAGMM-tr model with bi-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”

Original: and model output “Der Patient muss wirklich verstanden werden, um den Grad des Risikos seines Krebses zu verstehen, indem er ihm die M ¨
Revised: and model output “Der Patient muss wirklich verstanden werden, um den Grad des Risikos seines Krebses zu verstehen, indem er ihm die M ¨

Original: oglichkeiten zur Verf ¨ ugung stellt, nicht notwendigerweise mit Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen und sich stattdessen in solchen F ¨ allen f ¨ ur eine aktive ¨Uberwachung der Krankheit entscheiden.”.
Revised: oglichkeiten zur Verf ¨ ugung stellt, nicht notwendigerweise mit Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen und sich stattdessen in solchen F ¨ allen f ¨ ur eine aktive ¨Uberwachung der Krankheit entscheiden.”.

Original: We also plot the ﬁrst three encoder-decoder attention and encoder-self attention layers of the SAGMM-tr model with uni-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”
Revised: We also plot the ﬁrst three encoder-decoder attention and encoder-self attention layers of the SAGMM-tr model with uni-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”

Original: and model output “Der Patient muss wirklich verstanden werden, um das Risiko seines
Revised: and model output “Der Patient muss wirklich verstanden werden, um das Risiko seines

Original: Krebses zu verstehen, indem er ihm die M¨oglichkeiten zur Verf¨ugung stellt, nicht notwendigerweise mit
Revised: Krebses zu verstehen, indem er ihm die M¨oglichkeiten zur Verf¨ugung stellt, nicht notwendigerweise mit

Original: Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen, und stattdessen in solchen F¨allen
Revised: Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen, und stattdessen in solchen F¨allen

Original: f¨ur eine aktive ¨Uberwachung der Krankheit zu entscheiden.”.
Revised: f¨ur eine aktive ¨Uberwachung der Krankheit zu entscheiden.”.

Original: From the ﬁgures 10, 11 and 12, the transformers with SAGMM-tr approximates the machine translation task as a strict monotonic task and ﬁnds optimal alignments under the assumption.
Revised: From the ﬁgures 10, 11 and 12, the transformers with SAGMM-tr approximates the machine translation task as a strict monotonic task and ﬁnds optimal alignments under the assumption.

Original: Since the machine translation is not strictly monotonic task, the performance of the SAGMM-tr attention deteriorates compared to the soft attention.
Revised: Since the machine translation is not strictly monotonic task, the performance of the SAGMM-tr attention deteriorates compared to the soft attention.

Original: However, the transformer with SAGMM-tr attention and uni-directional encoder enables online inference for simultaneous translation.
Revised: However, the transformer with SAGMM-tr attention and uni-directional encoder enables online inference for simultaneous translation.

Original: A. D ECODED EXAMPLES OF SOFT ATTENTION AND SAGMM MODELS ON SPEECH COMMAND AND L IBRI S PEECH DATASETS
Revised: A. D ECODED EXAMPLES OF SOFT ATTENTION AND SAGMM MODELS ON SPEECH COMMAND AND L IBRI S PEECH DATASETS

Original: Table 7 and 8 show the decoded examples from the models with unseen sequence lengths.
Revised: Table 7 and 8 show the decoded examples from the models with unseen sequence lengths.

Original: From the tables, soft attention has difﬁculty to generate the sequences that are signiﬁcantly shorter or longer than the training corpus distribution.
Revised: From the tables, soft attention has difﬁculty to generate the sequences that are signiﬁcantly shorter or longer than the training corpus distribution.

Original: In contrast, the SAGMM-tr is robust to the sequence length mismatch and generates the transcription to arbitrary length.
Revised: In contrast, the SAGMM-tr is robust to the sequence length mismatch and generates the transcription to arbitrary length.

Original: [Table]
Revised: [Table]

Original: Transcription Reference
Revised: Transcription Reference

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in a chord words was it their colours they were voyaging across the deserts of the sky a host of nomads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and woodbegirt and citadelled and of entrenched and marshalled races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in a chord words was it their colours they were voyaging across the deserts of the sky a host of nomads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and woodbegirt and citadelled and of entrenched and marshalled races

Original: Soft attention
Revised: Soft attention

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves to love him as ourselves with the same amount and intensity of love the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and martialed races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves to love him as ourselves with the same amount and intensity of love the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and martialed races

Original: [Table]
Revised: [Table]

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleys and wood be girt and set and of intrenched and marshalled races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleys and wood be girt and set and of intrenched and marshalled races

Original: SAGMM-tr(mh)
Revised: SAGMM-tr(mh)

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the command of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay there beyond the irish sea europe of strange tongues and valleyed and wood girt and citadel and of intrenched and marched and races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the command of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay there beyond the irish sea europe of strange tongues and valleyed and wood girt and citadel and of intrenched and marched and races