Original: L EARNING M ONOTONIC A LIGNMENTS WITH S OURCE -A WARE GMM A TTENTION
Revised: L EARNING M ONOTONIC A LIGNMENTS WITH S OURCE -A WARE GMM A TTENTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers with soft attention have been widely adopted to various sequence-tosequence tasks.
Revised: Transformers with soft attention have been widely adopted to various sequence-tosequence tasks.

Original: While the soft attention is effective for learning semantic similarities between queries and keys based on their contents, it does not explicitly model the order of elements in sequences which are crucial for monotonic sequence-to-sequence tasks.
Revised: While the soft attention is effective for learning semantic similarities between queries and keys based on their contents, it does not explicitly model the order of elements in sequences which is crucial for monotonic sequence-to-sequence tasks.

Original: Learning monotonic alignments between input and output sequences could be beneﬁcial for long-form and online inference applications that are still challenging problem for the conventional soft attention algorithm.
Revised: Learning monotonic alignments between input and output sequences could be beneﬁcial for long-form and online inference applications that are still challenging problem for the conventional soft attention algorithm.

Original: In this paper, we focus on monotonic sequence-tosequence task and propose Source-Aware Gaussian Mixture Model (SAGMM) attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence.
Revised: In this paper, we focus on monotonic sequence-tosequence tasks and propose Source-Aware Gaussian Mixture Model (SAGMM) attention in which the attention scores are monotonically calculated considering both the content and order of the source sequence.

Original: We show that the proposed attention mechanism solves the online and long-form speech recognition problems without performance degradation in ofﬂine in-distribution speech recognition through experiments.
Revised: We show that the proposed attention mechanism improves the performance on the online and long-form speech recognition problems without performance degradation in ofﬂine in-distribution speech recognition through experiments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For a recent few years, the transformer models with soft attention have been widely adopted to various sequence generation tasks (Raffel et al., 2019; Vaswani et al., 2017; Parmar et al., 2018; Karita et al., 2019).
Revised: For a recent few years, the transformer models with soft attention have been widely adopted to various sequence generation tasks (Raffel et al., 2019; Vaswani et al., 2017; Parmar et al., 2018; Karita et al., 2019).

Original: The soft attention does not explicitly model the order of elements in the sequence and attends all encoder outputs for each decoder step.
Revised: The soft attention does not explicitly model the order of elements in the sequence and attends all encoder outputs for each decoder step.

Original: However, the order of elements is crucial to understand monotonic sequenceto-sequence (Seq2Seq) tasks, e.g. automatic speech recognition (ASR), video analysis, and lip reading.
Revised: However, the order of elements is crucial to understand monotonic sequenceto-sequence (Seq2Seq) tasks, e.g. automatic speech recognition (ASR), video analysis, and lip reading.

Original: Learning monotonic alignments helps model to attend  subset of the encoder output without performance degradation in these tasks.
Revised: Learning monotonic alignments helps model to attend to a subset of the encoder output without performance degradation in these tasks.

Original: In comparison, the soft attention is not suitable for streaming inference applications since the softmax operation needs to wait until all encoder outputs are produced.
Revised: In comparison, the soft attention is not suitable for streaming inference applications since the softmax operation needs to wait until all encoder outputs are produced.

Revised: Figure 1 (b) shows the attention plot from soft-attention.

Revised: The soft attention learns alignments between queries and keys based on their similarities while it requires all encoder tokens before attention score calculation.

Original: Furthermore, the soft attention has difﬁculty to generate long-form sequences which are not considered in training corpus.
Revised: Furthermore, the soft attention has difﬁculty to decode long-form sequences which are not considered in training corpus.

Original: The Gaussian Mixture Model (GMM) attention Graves (2013); Battenberg et al.
Revised: The Gaussian Mixture Model (GMM) attention Graves (2013); Battenberg et al.

Original: Chiu et al.
Revised: Chiu et al.

Original: (2019) has been proposed to learn the monotonic mapping between encoder and decoder states for long-form sequence generation.
Revised: (2019) has been proposed to learn the monotonic mapping between encoder and decoder states for long-form sequence generation.

Original: The GMM attention is a pure location-aware algorithm in which the model selects attention windows cumulatively without considering source contents.
Revised: The GMM attention is a pure location-aware algorithm in which the encoder contents are not considered during attention score calculation.

Original: This pure location-based property leads to mismatch between attention parameters from decoders and information distribution in encoder outputs.
Revised: However, each element encoder output sequences has different amount of information and should be attended considering their contents.

Revised: As shown in ﬁgure 1 (c), the GMM attention fails to learn detailed alignments and attends to many tokens simultaneously.

Original: In this paper, we adopt the GMM attention mechanism to modern transformer structure and propose SourceAware Gaussian Mixture Model (SAGMM) attention which considers both contents and orders of source sequences.
Revised: In this paper, we adopt the GMM attention mechanism to modern transformer structure and propose SourceAware Gaussian Mixture Model (SAGMM) attention which considers both contents and orders of source sequences.

Original:  SAGMM is multi-modal attention that can also discard non-informative tokens in attention window.
Revised: Each component in SAGMM is multi-modal and discard non-informative tokens in attention window.

Original: For online inference, we propose truncated SAGMM (SAGMM-tr) by discarding long-tail of the attention score in SAGMM.
Revised: For online inference, we propose truncated SAGMM (SAGMM-tr) by discarding long-tail of the attention score in SAGMM.

Original: To the best of our knowledge, this is the ﬁrst attempt to adopt the GMMbased attention to online sequence generation tasks.
Revised: To the best of our knowledge, this is the ﬁrst attempt to adopt the GMM-based attention to online sequence generation tasks.

Original: Learning accurate monotonic alignmentsalso helps SAGMM-tr to attend  subset oflong sequences for each decoder step and improves the performance of model in  long-form sequence generation tasks.
Revised: Learning accurate monotonic alignments helps SAGMM-tr to attend to a relevant subset of sequences for each decoder step and improves the performance of model in streaming and long-form sequence generation tasks.

Revised: Figure 1 (d) shows the monotonic alignments learned by SAGMM-tr which enables online inference.

Original: The experiments on streaming and long-form ASR showed substantial performance improvements compared to conventional algorithms without performance degradation on ofﬂine in-distribution ASR.
Revised: The experiments on streaming and long-form ASR showed substantial performance improvements compared to conventional algorithms without performance degradation on ofﬂine in-distribution ASR.

Original: We also tested SAGMM-tr in machine translation task and showed the performance of proposed algorithm in non-monotonic tasks.
Revised: We also tested SAGMM-tr in machine translation task and showed the performance of proposed algorithm in non-monotonic tasks.

Original:  S OURCE - AWARE GMM ATTENTION 2. S OFT ATTENTION
Revised:  S OURCE - AWARE GMM ATTENTION 2. S OFT ATTENTION

Original: In this paper, we abbreviate the head index h during attention score calculation for simplicity.
Revised: In this paper, we abbreviate the head index h during attention score calculation for simplicity.

Original: In the dotproduct multi-head soft attention Vaswani et al.
Revised: In the dotproduct multi-head soft attention Vaswani et al.

Original: (2017) without relative positional encoding , the attention score from soft attentionSoft α  is derived from the query matrix Q ∈ R I × d and key matrix K ∈ R J × d as follows:
Revised: (2017) without relative positional encoding , the attention score from soft attention α Soft is derived from the query matrix Q ∈ R I × d and key matrix K ∈ R J × d as follows:

Original: [Equation]
Revised: [Equation]

Original: where d, I , and J area feature dimension, decoder and encoder sequence length, respectively.
Revised: where d, I , and J are feature dimension, decoder and encoder sequence length, respectively.

Original: Themultihead attention concatenates output vector from each head  and apply a linear transform before a residual connection as
Revised: The attention context matrix form h -th head H h and the multi-head output M is given by

Original: [Equation]
Revised: [Equation]

Original: whereSoft α h denotesSoft α from h -th head and V h ∈ R J × d is value matrix for h -th head.
Revised: where α hSoft denotes α Soft for h -th head, V h ∈ R J × d is value matrix for h -th head, and n h denotes the number of heads.

Original: In this paper, we adopt the relative positional encoding Shaw et al.
Revised: In this paper, we adopt the relative positional encoding Shaw et al.

Original: Dai et al.
Revised: Dai et al.

Original: (2019) for self-attention layers.
Revised: (2019) for self-attention layers.

Original: The relative positional encoding gives a more strong baseline for the long-form sequence generation.
Revised: The relative positional encoding gives a more strong baseline for the longform sequence generation.

Original: 2. GMM ATTENTION
Revised: 2. GMM ATTENTION

Original: The previous studies on GMM attention Battenberg et al.
Revised: The previous studies on GMM attention Battenberg et al.

Original: Chiu et al.
Revised: Chiu et al.

Original: (2019) are based on early content-based attention (Cho et al.,2014; Sutskever et al., 2014).
Revised: (2019) are based on early content-based attention (Cho et al., 2014).

Original: Li et al.
Revised: Li et al.

Original: (2020) adopt the GMM attention to the transformer framework, but they do not provide detailed descriptions.
Revised: (2020) adopt the GMM attention to the transformer framework, but they do not provide detailed descriptions.

Original: In this paper, we deﬁne the
Revised: In this paper, we adopt v2 model in Battenberg et al.

Revised: (2020) which improves the performance of attention mechanism proposed in Graves (2013).

Original:  GMM attention as a variant of the multi-head attention in which each normal distribution corresponds to attention score of a single head.
Revised: We deﬁne the GMM attention as a variant of the multi-head attention by considering a Gaussian distribution component as an attention score of single-head in multi-head mechanism. Our GMM model is similar to Battenberg et al.

Revised: (2020) when the value matrix is shared for all Gaussian components.

Revised: The multi-head value matrices helps to attend to information from different representation subspaces (Vaswani et al., 2017).

Original: Let us denote the i -th row of Q as Q i ∈ R 1 × d .
Revised: Let us denote the i -th row of Q as Q i ∈ R 1 × d .

Original: The normal distribution parameters for i -th step are given by
Revised: The normal distribution parameters for i -th step are given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where ζ ( x ) is a softplus function of x , and
Revised: where ζ ( x ) is a softplus function of x , and

Original: W ∆ ∈ R d × 1 , W Σ ∈ R d × 1 , W φ ∈ R d × 1 .
Revised: W ∆ ∈ R d × 1 , W σ ∈ R d × 1 , W φ ∈ R d × 1 .

Revised: The softplus function is adopted similar to Battenberg et al.

Revised: (2020) in which softplus activation showed better performance than exponential operation.

Original: The GMM attention score for j -th encoder output in i -th decoder step GMMα i,j is deﬁned as follows:
Revised: The GMM attention score for j -th encoder output in i -th decoder step α GMM (cid:107) i,j is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: In contrast to the early GMM Battenberg et al. (2020), the distinct value matrix V h for each head in equation 8 helps jointly attending to information from different representation subspaces (Vaswani et al., 2017). The multi-head output Multihead is calculated from equation 3 .
Revised: where softmax h denotes softmax function over heads.

Original: The conventional GMM attention mechanism is identical to integral over source sequences with an uniform key width as shown in Figure 1 (a).
Revised: The conventional GMM attention mechanism is analogous to integral of source sequences with an uniform axis spacing as shown in Figure 2 (a).

Original: In this ﬁgure, each rectangle denotes the attention scoreof GMM attention with given Gaussian  parameters.
Revised: In this ﬁgure, each rectangle denotes the attention score with given Gaussian component parameters.

Original: The GMM attention assumes that each encoder output is equally important.
Revised: The GMM attention assumes that each encoder output is equally important.

Original: However, this assumption is not satisﬁed for many input modalities, e.g. speech and videos from real environments.
Revised: However, this assumption is not satisﬁed for many input modalities, e.g. speech and videos from real environments.

Original: Moreover, the  GMM attentionscore is uni-modal and cannot discard the non-informative tokens in attention window.
Revised: Moreover, the number of modes in GMM attention is limited by the number mixture components.

Original: To learn robust alignments for monotonic Seq2Seq tasks, we propose SAGMM which considers both contents and locations during attention mechanism.
Revised: To learn robust alignments for monotonic Seq2Seq tasks, we propose SAGMM which considers both contents and locations during attention mechanism.

Original: 2. S OURCE - AWARE GMM ATTENTION
Revised: 2. S OURCE - AWARE GMM ATTENTION

Original: Similar to the GMM mechanism, the attention score of  SAGMM is derived from theGaussian probability distribution.
Revised: Figure 2 (b) shows the scheme of proposed SAGMM attention. Compared to the GMM attention, the

Original: However, we introduce non-uniform weights for encoder outputs to consider irregular information distribution over source sequences. The SAGMM can skip or focus on  the sub-sequences according totheir relative importance via the non-uniform key axis.
Revised: SAGMM is analogous to integral of normal distribution with non-uniform spacing based on encoder contents. In the ﬁgure 2 (b), the width of rectangle δ j controls the model to selectively attend to the informative tokens during attention score calculation.

Original: This content-aware property of SAGMM helps the model learning stable monotonic alignments from the training corpus.
Revised: This content-aware property of SAGMM helps the model learning stable monotonic alignments from the training corpus.

Original: Furthermore,the attention score of SAGMMis multi-modal that the model can  discard non-informative tokens to aggregate information distributed over several remote frames.
Revised: Furthermore, SAGMM can easily discard non-informative tokens and aggregate information distributed over several remote tokens.

Original: In SAGMM, the normal distribution paramter ∆ i , Σ i , φ i , and µ i are derived from the same equation with equation 4 and equation 5.
Revised: In SAGMM, the normal distribution parameter ∆ i , σ i , φ i , and µ i are derived from the same equation with equation 4 and equation 5.

Original: To encode the contents of source sequences, the weight for each encoder output δ j is given from the j -th row of the key matrix K as follows:
Revised: To encode the contents of source sequences, the weight for each encoder output δ j is given from the j -th row of the key matrix K as follows:

Original: [Equation]
Revised: [Equation]

Original: ) where W δ ∈ R d × 1 and σ ( x ) denotes a sigmoid function of x .
Revised: (9) where W δ ∈ R d × 1 and the sigmoid function is introduced to smoothly bound the maximum weight δ j similar to Dong & Xu (2020).

Original: Then, the probability of normal distribution N ( ν j ; µ i , Σ i ) is calculated from its cumulative sum ν j given by
Revised: Then, the probability of normal distribution N ( ν j ; µ i , σ i ) is calculated from its cumulative sum ν j given by

Original: [Equation]
Revised: [Equation]

Original: Finally, the SAGMM attention score SAGMMα i,j is deﬁned as follows:
Revised: Finally, the SAGMM attention score α SAGMM is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: Figure 1 (b) shows the scheme of proposed SAGMM attention. In this ﬁgure, the area of rectangles are derived from both the normal distribution parameters and weights from the encoder outputs. The widths of rectangles δ j and their cumulative sum ν jare derived from the encoder outputs considering the contents and order of source sequences. In this view, the proposed algorithm is source-aware attention in contrast to pure location-aware GMM attention that adopts source token index j .
Revised: where δ j where δ j is multiplied to describe uneven step size in summation.

Original: In the early stage of training, we introduce length penalty loss to help learning alignments between µ and ν deﬁned as
Revised: In the early stage of training, we introduce length penalty loss to help learning alignments between µ and ν deﬁned as

Original: [Equation]
Revised: [Equation]

Original: with λ length = 0 .0005 .
Revised: with λ length = 0 .

Revised: 0005 where I and J denote the length of decoder and encoder sequences, respectively.

Original: We turned off L length after 200K training steps in experiments.
Revised: We turned off L length after 200K training steps in experiments.

Original: Finally, we modify 5 as µ i = µ i − 1 + relu3 (∆ i ) to scale the µ i and ν j similar to token indices.
Revised: Finally, we modify 5 as µ i = µ i − 1 + min ( max (∆ i , 0) , 3) to scale the µ i and ν j similar to token indices.

Original: This modiﬁcation helps the interpretability of µ and ν by scaling them.
Revised: This modiﬁcation helps the interpretability of µ and ν by scaling them.

Original: While SAGMM does not include the softmax operation over encoder outputs, following equation is satisﬁed for all i :
Revised: The attention score for each encoder token in SAGMM is determined independently since they do not rely on softmax operation over encoder outputs.

Original: [Equation]
Revised: Note that (cid:80) j δ j N ( ν j ; µ i , σ i ) approximates the integral of

Original: In experiments, the model with SAGMM attention easily converged without tuning on the sum of attention scores. Since SAGMM does not rely on the softmax operation, we propose the SAGMM-tr for online inference applications by cropping the long-tail part of normal distribution.
Revised: Gaussian distribution. Thus the sum of attention weights is close to 1, which helps numerical stability and learning without using softmax.

Original: 2. SAGMM- TR FOR ONLINE INFERENCE
Revised: 2. SAGMM- TR FOR ONLINE INFERENCE

Revised: Since the attention score in SAGMM is generated without softmax normalization over all encoder tokens, we can simply build the attention for streaming inference by cropping the long-tail of the Gaussian distribution.

Original: In SAGMM-tr, the normal distribution is truncated to limit past and future context as follows:
Revised: In SAGMM-tr, the normal distribution is truncated to limit past and future context as follows:

Original: [Equation]
Revised: [Equation]

Original: Discarding the tokens with threshold 2 √ Σ i removes about 5% percent of attention score.
Revised: Discarding the tokens with threshold 2 √ σ i removes about 5% percent of attention score.

Original: We initialize the parameters of SAGMM-tr attention with those from SAGMM model and ﬁne-tuned them until the performance converges.

Original: We found that the models with SAGMM-tr showed slightly better performance than SAGMM and mainly reported the results with SAGMM-tr model. Algorithm 1: Decoding process of SAGMM-tr attention mechanism for a single head. The multi-headed layer can be calculated by running single-head algorithm in parallel.
Revised: In online inference with SAGMM-tr, H hi can be calculated after ν j becomes larger than µ i + 2 √ σ i .

Original: Data: encoder key and value vectors K j , V j  for  j∈ { 1 , , ...J } , i = 1 , µ 0 = ν 0 = 0 , [-y 
 = SOS while y-] i− 1 (cid:54) =
Revised: Note that when ν j ≥ µ i + 2 √ σ i satisﬁes in for a current token j = β i , [-y 
 = SOS while y-]then the equation also satisﬁes for all j > β i

Original: EOS do

Original: Q i = feedforward ( y i − 1 ) / * Lower decoder layers */ ∆ i , Σ i , φ i = GMMparam ( Q i ) , µ i = µ i − 1 + relu3 (∆ i ) / * equation 4 */ pos s , = µ i − 2 √ Σ i , pos e = µ i + 2 √ Σ i , Attend =for j = 1 to J do δ j = σ ( K j W δ ) , ν j = ν
Revised: and we can calculate H hi without waiting future contexts.

Original: j − 1 + δ j if ν j > pos e then Break else if ν j > pos s then [Equation] /* equation 15, 16 */
Revised: We initialize the parameters of SAGMM-tr attention with those from SAGMM model and ﬁne-tuned them until the performance converges.

Original: [Equation] y i = Output ( Decoder ( Attend )) , i = i +
Revised: We found that the models with SAGMM-tr showed slightly better performance than SAGMM and mainly reported the results with SAGMM-tr model.

Original: The ﬁxed length window with hyperparameters can be also adopted to truncate a Gaussian distribution.

Original: We tested the window width c = 15 centered by µ i and compared the recognition performance of the models to adaptive window width in experiments.

Original: Forthe online speech recognition with SAGMM-tr, we randomly concatenate the 1 -vector after the end of source sequence with a probability p eos in the training stage and trained to emit the end-of-sequence token EOS only with the utterances containing 1 -vector.
Revised: For online speech recognition with SAGMM-tr, we randomly concatenate the 1 -vector after the end of source sequence with a probability p eos in the training stage and trained to emit the end-of-sequence token EOS only with the utterances containing 1 -vector.

Original: The 1 -vector concatenation helps to suppress the EOS token in a long silence part.
Revised: The 1 -vector concatenation helps to suppress the EOS token in a long silence part.

Original: Finally, we build the uni-directional encoder whose maximum latency is similar to Zhang et al.
Revised: Finally, we build the uni-directional encoder whose maximum latency is similar to Zhang et al.

Original: Dong et al.
Revised: Dong et al.

Original: by adopting the block-wise mask on self-attention layers.
Revised: by adopting the block-wise mask on self-attention layers.

Original: The detailed explanation for the block-wise masking is given in Appendix.
Revised: The detailed explanation for the block-wise masking is given in Appendix.

Original: Algorithm 1  shows a psuedo code for the SAGMM-tr attention inthe inference stage.
Revised: Algorithm 1 in appendix shows a pseudo code for the SAGMM-tr attention in inference stage.

Revised: The number of tokens we need to waits in equation 14 - 16 is given by model parameters.

Revised: To strictly limit the maximum latency of model, we also trained and tested SAGMM-tr with ﬁxed attention window width c in experiments.

Revised: In this version, equation 14 - 16 are modiﬁed as follows:

Original: R ELATED WORKS
Revised: [Equation] In equation 20, we wait c 2 additional tokens before producing output. We compared the performance of

Revised: SAGMM-tr with adaptive and ﬁxed window length in experiments.

Original: Chiu & Raffel (2018) propose monotonic chunkwise attention (MoChA) based on hard monotonic sampling over attention window (Raffel et al., 2017b).
Revised: Chiu & Raffel (2018) propose monotonic chunkwise attention (MoChA) based on hard monotonic sampling over attention window (Raffel et al., 2017).

Original: The monotonic multi-head attention Ma et al.
Revised: The monotonic multi-head attention Ma et al.

Original: ; Inaguma et al.
Revised: ; Inaguma et al.

Original: (2020) and monotonic inﬁnite lookback attention Arivazhagan et al.
Revised: (2020) and monotonic inﬁnite lookback attention Arivazhagan et al.

Original: (2019) are based on the concept similar to (Chiu & Raffel, 2018).
Revised: (2019) are based on the concept similar to (Chiu & Raffel, 2018).

Original: MoChA replaces hard sampling operation to probability distribution over memory in the training stage.
Revised: MoChA replaces hard sampling operation to probability distribution over memory in the training stage.

Original: In contrast, SAGMM uses the same operations for training and inference.
Revised: In contrast, SAGMM uses the same operations for training and inference.

Original: Also,
Revised: Also,

Original: MoChA relies on numerically unstable cumulative product which requires additional efforts for training (Miao et al., 2019; Raffel et al., 2017a).
Revised: MoChA relies on numerically unstable cumulative product which requires additional efforts for training (Miao et al., 2019; Raffel et al., 2017).

Original: In contrast, SAGMM adopts the stable cumulative sum to describe monotonic alignments.
Revised: In contrast, SAGMM adopts the stable cumulative sum to describe monotonic alignments.

Original: Finally, MoCha is not effective for long-form speech recognition compared to GMM attention Chiu et al.
Revised: Finally, MoCha is not effective for long-form speech recognition compared to GMM attention Chiu et al.

Original: (2019) while our model outperformed GMM in “test-long” set in experiments.
Revised: (2019) while our model outperformed GMM in “test-long” set in experiments.

Original: Dong & Xu (2020) introduce the encoder output weight for ASR task.
Revised: Dong & Xu (2020) introduce the encoder output weight for ASR task.

Original: Compared to their work, our attention mechanism considers the decoder content during attention score calculation.
Revised: Compared to their work, our attention mechanism considers the decoder content during attention score calculation.

Original: SAGMM also allows the overlap between successive attention windows which is helpful to generate semantics that span several decoder tokens.
Revised: SAGMM also allows the overlap between successive attention windows which is helpful to generate semantics that span several decoder tokens.

Original: Lee et al.
Revised: Lee et al.

Original: (2020) propose the  attention mechanismwhich enables early inference without softmax operation.
Revised: (2020) propose the online attention mechanism without softmax operation by aggregating the information of the encoded sequence with update gate.

Original: Their approach does not explicitly model monotonic alignment and the update gate for online inference in Lee et al.
Revised: Similar to soft attention, their model does not learn monotonic alignment. Also, the update gate for online inference in Lee et al.

Original: (2020) is uni-modal similar to conventional GMM attention.
Revised: (2020) is uni-modal. The location-aware attention Chorowski et al.

Revised: ; Watanabe et al.

Revised: (2017); Moritz et al.

Revised: (2019) is proposed to employ positional information to the content-based attention mechanism.

Revised: The location-aware attention introduces attention score in the previous step as additional argument for the current step.

Original:  The non-attentive neural network-based approaches which do not rely on encoder-decoder attention have been studied by Connectionist Temporal Classiﬁcation (CTC) Graves et al.
Revised: However, it it not streamable attention. The non-attentive neural network-based approaches which do not rely on encoder-decoder attention have been studied by Connectionist Temporal Classiﬁcation (CTC) Graves et al.

Original: ; Liptchinsky et al.
Revised: ; Liptchinsky et al.

Original: Li et al.
Revised: Li et al.

Original: (2019), and RNN-transducer (RNN-T)
Revised: (2019), and RNN-transducer (RNN-T)

Original: Graves (2012), Transformer-transducer (Transformer-T)
Revised: Graves (2012), Transformer-transducer (Transformer-T)

Original: Zhang et al.
Revised: Zhang et al.

Original: ; Yeh et al.
Revised: ; Yeh et al.

Original: (2019), and Imputer Chan et al.
Revised: (2019), and Imputer Chan et al.

Original: We focus on the Seq2Seq approach which do not require an assumption on sequence lengths and uses the simple beam search algorithm forthe inference.
Revised: We focus on the Seq2Seq approach which do not require an assumption on sequence lengths and uses the simple beam search algorithm for inference.

Original: Finally, the block-wise inference has been widely studied with manually deﬁned decoding blocks
Revised: Finally, the block-wise inference has been widely studied with manually deﬁned decoding blocks

Original: Jaitly et al. (2015); Tsunoo et al.
Revised: Jaitly et al. (2015); Tsunoo et al.

Original: (2020) or joint decoding with CTC model Moritz et al.
Revised: (2020) or joint decoding with CTC model Moritz et al.

Original: We do not use the joint CTC decoding Kim et al.
Revised: We do not use the joint CTC decoding Kim et al.

Original: (2016) nor human supervision for the blocked inference.
Revised: (2016) nor human supervision for the blocked inference.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We conducted experiments in the speech command Warden (2018) and LibriSpeech Panayotov et al.
Revised: We conducted experiments in the speech command Warden (2018) and LibriSpeech Panayotov et al.

Original: (2015) datasets to compare the performance of our model in standard, online and long-form ASR tasks.
Revised: (2015) datasets to compare the performance of our model in standard, online and long-form ASR tasks.

Original: We only compare the end-to-end speech recognition model without the external language model  rescoring.
Revised: We only compare the end-to-end speech recognition model without the external language model (LM) rescoring.

Original: We also performed preliminary experiments on the translation task to show the performance of the SAGMM-tr on non-monotonic tasks.
Revised: We also performed preliminary experiments on the translation task to show the performance of the SAGMM-tr on non-monotonic tasks.

Original: 4. E XPERIMENTS ON CONCATENATED S PEECH C OMMAND DATASET
Revised: 4. E XPERIMENTS ON CONCATENATED S PEECH C OMMAND DATASET

Original: First, we performed a speech recognition experiment with limited vocabulary to show that the common soft attention algorithm has difﬁculty to decode utterances with unseen sequence lengths.
Revised: First, we performed a speech recognition experiment with limited vocabulary to show that the common soft attention algorithm has difﬁculty to decode utterances with unseen sequence lengths.

Original: The speech command dataset consists of 1 seconds speeches from various speakers saying single words from  30 vocabulary.
Revised: The speech command dataset consists of 1 second long speeches from various speakers saying single words from a vocabulary of 30 words.

Original: We built 100K utterances of training corpusWarden (2018) by concatenating randomly chosen 5-9 utterances in the training data.
Revised: We built 100K utterances of training corpusWarden (2018) by concatenating randomly chosen 5-9 utterances in the training data.

Original: 500 test utterances were also built by concatenating { 3 , 7 , 10 , 15 , 20 } randomly chosen words in the test set.
Revised: 500 test utterances were also built by concatenating { 3 , 7 , 10 , 15 , 20 } randomly chosen words in the test set.

Original: We cropped both the start and end of chosen utterances randomly from 0.05s to 0.15s before concatenating them to prevent the SAGMM from ﬁnding a trivial solution.
Revised: We cropped both the start and end of chosen utterances randomly from 0.05s to 0.15s before concatenating them to prevent the SAGMM from ﬁnding a trivial solution.

Original: We compared the word error rate (WER) of transformers with soft and SAGMM attentions on test sets.
Revised: We compared the word error rate (WER) of transformers with soft and SAGMM attentions on test sets.

Original: Detailed hyperparameters for the experiment are described in Appendix.
Revised: Detailed hyperparameters for the experiment are described in Appendix.

Original: In table 1, the soft attention mechanism tends to memorize the sequence length distribution from the training corpus and fails to decode cor- rectly in the unseen sequence length.
Revised: In table 1, the soft attention mechanism tends to memorize the sequence length distribution from the training corpus and fails to decode correctly in the unseen sequence length.

Original: On the other hand, the proposed SAGMM algorithm was robust to sequence length mismatches.
Revised: On the other hand, the proposed SAGMM algorithm was robust to sequence length mismatches.

Original: The decoded examples for soft attention and SAGMM are shown in Appendix.
Revised: The decoded examples for soft attention and SAGMM are shown in Appendix.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS ON L IBRI S PEECH DATASET
Revised: 4. E XPERIMENTS ON L IBRI S PEECH DATASET

Original: We conducted experiments on the LibriSpeech whose training dataset consists of 960 hours of read audio books (Panayotov et al., 2015).
Revised: We conducted experiments on the LibriSpeech whose training dataset consists of 960 hours of read audio books (Panayotov et al., 2015).

Original: We validated the performance of models on dev sets and reported WERs in tests sets with the ﬁxed beam width 4 .
Revised: We validated the performance of models on dev sets and reported WERs in tests sets with the ﬁxed beam width 4 .

Original: First, we compared the performance of SAGMM-tr with conventional algorithms in the ofﬂine speech recognition.
Revised: First, we compared the performance of SAGMM-tr with conventional algorithms in the ofﬂine speech recognition.

Original: We trained the multi-head transformers similar to the previous experiments.
Revised: We trained the multi-head transformers similar to the previous experiments.

Original: The encoder consisted of 10 layers of self-attention block with 768 hidden nodes and 3,072 ﬁlters.
Revised: The encoder consisted of 10 layers of self-attention block with 768 hidden nodes and 3,072 ﬁlters.

Original: The decoder was constructed by stacking 4 self-attention blocks with the same node and ﬁlter sizes.
Revised: The decoder was constructed by stacking 4 self-attention blocks with the same node and ﬁlter sizes.

Revised: The encoder and decoder self-attention employed soft-attention with relative positional encoding (Dai et al., 2019).We trained and tested models which employed the soft attention, GMM attention, and SAGMM-tr attention mechanisms as encoderdecoder attention.

Original: We adopt the  CTC loss on the encoder output with λ ctc = 0 .
Revised: We adopt the auxiliary CTC loss on the encoder output with λ ctc = 0 .

Original: 1 for fair comparison with the previous attention-based models Lee et al.
Revised: 1 for fair comparison with the previous attention-based models Lee et al.

Original:  Dong & Xu (2020); Kim et al.
Revised: ; Dong & Xu (2020); Kim et al.

Revised: The SAGMM-tr model without CTC loss was 3.84% on test-clean dataset.

Original: The detail hyperparameters for the experiment and performance of SAGMM-tr without CTC loss are shown in Appendix.
Revised: The detail hyperparameters for the experiment and performance of SAGMM-tr without CTC loss are shown in Appendix.

Original: Table 2 shows the performance of various non-streamable models.
Revised: Table 2 shows the performance of various non-streamable models.

Original: In non-streamable models, our baseline transformer showed similar performance with previous studies considering the number of parameters.
Revised: In non-streamable models, our baseline transformer showed similar performance with previous studies considering the number of parameters.

Original: We also compared the performance of GMM and SAGMM-tr models with the bi-directional encoder to other streamable models.
Revised: We also compared the performance of GMM and SAGMM-tr models with the bi-directional encoder to other streamable encoder-decoder attention algorithms.

Original: The performance of SAGMM-tr attention matched our transformer model and showed better performance than other streamable Seq2Seq models.
Revised: The performance of SAGMM-tr attention matched our transformer model and showed better performance than other streamable Seq2Seq models.

Original: From the results,  SAGMM-tr successfully described the monotonic alignments between speech and transcription without attending all encoder outputs.
Revised: From the results, [Table] SAGMM-tr successfully described the monotonic alignments between speech and transcription without attending all encoder outputs.

Original: Next, we tested the performance of SAGMM-tr in online ASR task.
Revised: Next, we tested the performance of SAGMM-tr in online ASR task.

Original: We trained the single-head and multihead SAGMM-tr models with the block-wise masked encoder.
Revised: We trained the single-head and multihead SAGMM-tr models with the block-wise masked encoder.

Original: The truncated GMM model was also tested.
Revised: The truncated GMM model was also tested.

Original: However, the model attended to the future context as shown in A.3 and failed to learn the correct alignments.
Revised: However, the model attended to the future context as shown in A.4 and failed to learn the correct alignments.

Original: In the multi-head SAGMM-tr, several heads in the ﬁrst decoder layer were not trained and softmax ( φ hi ) went to zero.
Revised: In the multi-head SAGMM-tr, several heads in the ﬁrst decoder layer were not trained and softmax ( φ hi ) went to zero.

Original: We simply pruned these heads after the performance on dev set converged and ﬁne-tuned for additional 30K steps.
Revised: We simply pruned these heads after the performance on dev set converged and ﬁne-tuned for additional 30K steps.

Original: Figure 2 shows the attention plots of the single-head SAGMM-tr which successfully captures alignments between speech and transcription for an utterance with a long silence. Attention plots for soft attention, GMM, and multi-head SAGMM-tr models are in Appendix A.3.
Revised: We believe training with φ hi and head pruning helps to prevent over-ﬁtting when the desired monotonic alignments do not require many heads especially in the ﬁrst decoder layer.

Original: Table 3 shows the performance of models with various online inference approaches.
Revised: Table 3 shows the performance of models with various online inference approaches.

Original: From the table, the single-head SAGMM-tr outperformed other CTC and attention-based algorithms.
Revised: From the table, the single-head SAGMM-tr outperformed other CTC and attention-based algorithms.

Original: The multi-head mechanism further improved the performance of SAGMM-tr.
Revised: The multi-head mechanism further improved the performance of SAGMM-tr.

Original: Compared to the similar model with the bidirectional encoder, the block-wise mask on encoder self-attention layers increased the WER on testclean set by 0 .
Revised: Compared to the similar model with the bidirectional encoder, the block-wise mask on encoder self-attention layers increased the WER on testclean set by 0 .

Original: Our model showed slightly worse performance than state-of-the-art algorithm with transformer-T (Zhang et al., 2020).
Revised: Our model showed slightly worse performance than state-of-the-art algorithm with transformer-T (Zhang et al., 2020).

Original: Subword regularization Kudo (2018); Hannun et al.and [Table] [Table]
Revised: Subword regularization Kudo (2018); Hannun et al.

Original: [Table]

Original: SAGMM-

Original: [Table]

Original: 121M

Original: [Table]

Original:  sequence-level loss Sabour et al.
Revised: and sequence-level loss Sabour et al.

Original: ; Prabhavalkar et al.
Revised: ; Prabhavalkar et al.

Original: (2018) can help to match the performance of SAGMM-tr with the best transducer models.
Revised: (2018) can help to match the performance of SAGMM-tr with the best transducer models.

Original: The SAGMM-tr can also be adopted to various monotonic
Revised: The SAGMM-tr can also be adopted to various monotonic

Original: Seq2Seq tasks easily since it can be inferred via simple beam search without probability marginalization in transducers and does not require assumption that the target should not be longer than the source.
Revised: Seq2Seq tasks easily since it can be inferred via simple beam search without probability marginalization in transducers and does not require assumption that the target should not be longer than the source.

Revised: Figure 3 shows the attention plots of the single-head SAGMM-tr which successfully captures alignments between speech and transcription for an utterance with a long silence.

Original: We also tested SAGMM-tr models with ﬁxed attention window width.
Revised: Figure 1 also shows the attention alignments from multi-head soft attention, truncated GMM attention and SAGMM-tr attentions. Detailed [Table] [Table]

Revised: plots are shown in Appendix A.4.

Revised: In soft attention, several heads attends to the speech absence frames constantly for all decoder steps.

Original: We started with the SAGMM-tr models trained for the online ASR experiment and measured the WERs before and after ﬁne-tuning with the constant window width ( c = 15 frames ) .
Revised: These heads might help utterance-level normalization on attention context vectors. The truncated GMM model failed to learn the alignment and attended to an wide range of encoder tokens. Finally, SAGMM-tr attention learned monotonic alignments successfully. Due to the monotonic constraint, all heads in SAGMM-tr did not attend silence.

Original: From the results in table 4 (a), SAGMM-tr attention mechanism can be also adopted to the models with ﬁxed attention window width.
Revised: The future work should explore the role of heads attending silence in soft attention and introduce them to the SAGMM-tr attention. The SAGMM-tr attention plots in lower layers are more blurred. We suppose that the lower layers ﬁnd the relevant subsets in the sequence and the higher layers use the context vector from lower layers to select the most relevant tokens in the subset. The multi-head SAGMM-tr model showed more blurred alignments compared to the single-head model.

Revised: Our hypothesis for this phenomenon is that several heads attended larger window to provide context information.

Original: The attention plots for SAGMM-tr  with the ﬁxed window width are shown in Appendix.
Revised: The ﬁxed window length scheme described in equation 17 - 20 or regularization on σ i would be also helpful to minimize the latency of online inference. In this paper, we tested the ﬁxed window algorithm. We started with the SAGMM-tr models trained for the online ASR experiment and measured the WERs before and after ﬁne-tuning with the constant window width ( c = 15 frames ) . From the results in table 4 (a), SAGMM-tr attention mechanism can be also adopted to the models with ﬁxed attention window width.

Original: Finally, we concatenated utterances from the same speaker in “test-clean” set to build “test-long” set for the long-form speech recognition.
Revised: Finally, we concatenated utterances from the same speaker in “test-clean” set to build “test-long” set for the long-form speech recognition.

Original: The average length of utterances in the test-long set was 54 seconds.
Revised: The average length of utterances in the test-long set was 54 seconds.

Original: In the
Revised: In the

Original: SAGMM-tr model, the beam search algorithm was slightly modiﬁed from Algorithm 1 to suppress the EOS token until the attention window covers ν h,J .
Revised: SAGMM-tr model, the beam search algorithm was slightly modiﬁed from Algorithm 1 to suppress the EOS token until the attention window covers ν h,J .

Original: The performance of GMM attention did not improved by the same modiﬁcation since GMM attention attends to ν h,J before the end of sentence.[Table]
Revised: The performance of GMM attention did not improved by the same modiﬁcation since GMM attention attends to ν h,J before the end of sentence.

Original: From the table 4 (b) [Table] SAGMM-tr outperformed the conventional soft and GMM attention models in longform speech recognition.
Revised: From the table 4 (b), SAGMM-tr outperformed the conventional soft and GMM attention models in longform speech recognition.

Original: Comparing the SAGMM-tr models, the multi-head SAGMM-tr showed worse performance than the single-head model.
Revised: Comparing the SAGMM-tr models, the multi-head SAGMM-tr showed worse performance than the single-head model.

Original: The performance degradation of multi-head mechanism on “testlong” set might come from accumulated attention windows range mismatches.
Revised: The performance degradation of multi-head mechanism on “testlong” set might come from accumulated attention windows range mismatches.

Original: Future works should explore techniques to regularize the attention window ranges in multi-headed models
Revised: Future works should explore techniques to regularize the attention window ranges in multi-headed models.

Original: [Table]

Original: 4. E XPERIMENTS ON A NON - MONOTONIC TASK
Revised: 4. E XPERIMENTS ON A NON - MONOTONIC TASK

Original: The proposed algorithm focuses on strict monotonic tasks which do not allow reordering.
Revised: The proposed algorithm focuses on strict monotonic tasks which do not allow reordering.

Original: However, the selfattention layers in the encoder could reorder the semantics in input sequences before monotonic alignments from the SAGMM-tr.
Revised: However, the selfattention layers in the encoder could reorder the semantics in input sequences before monotonic alignments from the SAGMM-tr.

Original: We conducted machine translation experiments on the WMT EN-DE environment to show the performance of SAGMM-tr attention in non-monotonic tasks.
Revised: We conducted machine translation experiments on the WMT EN-DE environment to show the performance of SAGMM-tr attention in non-monotonic tasks.

Revised: While the machine translation is a non-monotonic task, various approaches have been proposed to improve performance of online inference and simultaneous translation (Arivazhagan et al., 2019; Elbayad et al., 2020).

Revised: In this paper, we conduct simple experiments to ﬁgure out if the GMM attention can deals with non-monotonic tasks.

Original: We trained the encoder-decoder model similar to the transformer-base in Vaswani et al.
Revised: We trained the encoder-decoder model similar to the transformer-base in Vaswani et al.

Original: (2017) except the head size 4 of the encoder-decoder and decoder self-attention.
Revised: (2017) except the head size 4 of the encoder-decoder and decoder self-attention.

Original: We trained model with 4.5M pairs of sentences and measured the BLEU score on the newstest2013 set after 100K training steps.
Revised: We trained model with 4.5M pairs of sentences and measured the BLEU score on the newstest2013 set after 100K training steps.

Original: For uni-directional encoder, the block-wise mask with M = 5 is applied to SAGMM-tr model with bi-directional encoder and ﬁne-tuned for additional 40K steps.
Revised: For uni-directional encoder, the block-wise mask with M = 5 is applied to SAGMM-tr model with bi-directional encoder and ﬁne-tuned for additional 40K steps.

Original: [Table]
Revised: [Table]

Original:  Table 5 shows the performance of soft and SAGMM-tr attention models on the newstest2013 set.
Revised: [Table] -enc) 23. Table 5 shows the performance of soft and SAGMM-tr attention models on the newstest2013 set.

Original: As expected, the performance of SAGMM-tr did not match that of the conventional transformer models with soft attention.
Revised: As expected, the performance of SAGMM-tr did not match that of the conventional transformer models with soft attention.

Original: The uni-directional encoder also deteriorates the performance of SAGMM-tr similar to speech recognition experiments.
Revised: The uni-directional encoder also deteriorates the performance of SAGMM-tr similar to speech recognition experiments.

Original: Since SAGMM-tr does not consider the local reordering, the proposed algorithm has difﬁculty in learning locally non-monotonic functions.
Revised: Since SAGMM-tr does not consider the local reordering, the proposed algorithm has difﬁculty in learning locally non-monotonic functions.

Original: Our future works will improve the performance of SAGMM-tr on local reordering tasks by positive constraint relaxation on ∆ i .
Revised: Our future works will improve the performance of SAGMM-tr on local reordering tasks by positive constraint relaxation on ∆ i .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed the SAGMM which attends the subset of source sequences according to the normal distribution on the weighted encoder output axis, considering both contents and order of elements in sequences.
Revised: We have proposed the SAGMM which attends the subset of source sequences according to the normal distribution on the weighted encoder output axis, considering both contents and order of elements in sequences.

Original: We have also proposed SAGMM-tr for online/real-time inference applications.
Revised: We have also proposed SAGMM-tr for online/real-time inference applications.

Original: From the results on various speech recognition tasks, the proposed attention mechanism is shown to be able to learn the monotonic alignments between source and target sequences without a human supervision.
Revised: From the results on various speech recognition tasks, the proposed attention mechanism is shown to be able to learn the monotonic alignments between source and target sequences without a human supervision.

Original: The performance of transformer with SAGMM-tr improved for online and long-form speech recognition without performance degradation in the standard ofﬂine ASR task.
Revised: The performance of transformer with SAGMM-tr improved for online and long-form speech recognition without performance degradation in the standard ofﬂine ASR task.

Original: In future works, we would like to adopt the SAGMM-based attention mechanisms for natural language processing tasks which allow local reordering during the sequence generation.
Revised: In future works, we would like to adopt the SAGMM-based attention mechanisms for natural language processing tasks which allow local reordering during the sequence generation.

Original: We are also interested in latency minimization for online inference by controlling µ and Σ in both the training and inference stages.
Revised: We are also interested in latency minimization for online inference by controlling µ and σ in both the training and inference stages.

Original: A. E XPERIMENT CONFIGURATIONS
Revised: A. P SEUDO CODE FOR ONLINE INFERENCE WITH SAGMM- TR ATTENTION

Original: {+Algorithm h - head. Data: encoder key and value vectors K hj , V hj for j ∈ { 1 , 2 , ... J } , i = 1 , µ h 
+}
Revised: {+Algorithm h - head. Data: encoder key and value vectors K hj , V hj for j ∈ { 1 , 2 , ... J } , i = 1 , µ h 
+}

Original: {+= ν h 
+}
Revised: {+= ν h 
+}

Original: {+= 0 , y 
+}
Revised: {+= 0 , y 
+}

Revised: = SOS while y i − 1 (cid:54) =

Revised: EOS do

Revised: Q hi = feedforward ( y i − 1 ) /

Revised: * Lower decoder layers */ ∆ hi , σ hi , φ hi =

Revised: GMMparam (cid:0) Q hi (cid:1) , µ hi = µ hi − 1 + min (cid:0) max (cid:0) ∆ hi , 0 (cid:1) , 3 (cid:1) /

Revised: * equation 4 */ pos hs , = µ hi − 2 (cid:112) σ hi , pos he = µ hi + 2 (cid:112) σ hi , Attend h =for j = 1 to J do δ hj

Revised: = sigmoid (cid:0) K hj W hδ (cid:1) , ν hj =

Revised: ν hj − 1 + δ hj if ν hj > pos he then

Revised: Break else if ν hj > pos hs then α

Revised: [Equation]

Revised: /* equation 14, 15 */

Revised: Attend h =

Revised: Attend h

Revised: + α hSAGMM - tr (cid:107)

Revised: [Equation]

Revised: hj y i

Revised: =

Revised: Output (cid:0) Decoder (cid:0) softmax h (cid:0) φ hi (cid:1) Attend h (cid:1)(cid:1) , i

Original:  For all speech recognition experiments, the 80-dimensional log-Mel ﬁlterbank features were extracted with a 25ms window and shifted every 10ms.
Revised: = i + A. E XPERIMENT CONFIGURATIONS For all speech recognition experiments, the 80-dimensional log-Mel ﬁlterbank features were extracted with a 25ms window and shifted every 10ms.

Original: Each input vector stacked 3 log-Mel feature vectors, downsampled to 30ms frame rate.
Revised: Each input vector stacked 3 log-Mel feature vectors, downsampled to 30ms frame rate.

Original: In the speech command dataset experiment, We trained the multi-headed transformer Vaswani et al.
Revised: In the speech command dataset experiment, We trained the multi-headed transformer Vaswani et al.

Original: Dong et al.
Revised: Dong et al.

Original: (2018) with soft attention and SAGMM attention for encoder-decoder alignments.
Revised: (2018) with soft attention and SAGMM attention for encoder-decoder alignments.

Original: We adopt the relative positional encoding for both the encoder and decoder self-attention (Dai et al., 2019).
Revised: We adopt the relative positional encoding for both the encoder and decoder self-attention (Dai et al., 2019).

Original: The encoder consisted of 6 layers of self-attention block with 512 nodes and 2,048 ﬁlter size, respectively.
Revised: The encoder consisted of 6 layers of self-attention block with 512 nodes and 2,048 ﬁlter size, respectively.

Original: The decoder was constructed by stacking 3 self-attention block with the same node and ﬁlter sizes.
Revised: The decoder was constructed by stacking 3 self-attention block with the same node and ﬁlter sizes.

Original: The number of heads for encoder self attention, encoder-decoder attention, and decoder self attention were { 8 , 4 , 4 } , respectively.
Revised: The number of heads for encoder self attention, encoder-decoder attention, and decoder self attention were { 8 , 4 , 4 } , respectively.

Original: We employed label smoothing of value (cid:15) ls = 0 . 15 , and applied parallel scheduled sampling with probability 0.2 after 100k (Duckworth et al., 2019).
Revised: We employed label smoothing of value (cid:15) ls = 0 . 15 , and applied parallel scheduled sampling with probability 0.2 after 100k (Duckworth et al., 2019).

Original: We used the Adam optimizer with β 1 = 0 .
Revised: We used the Adam optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 98 and (cid:15) = 10 − 9 ) .
Revised: 98 and (cid:15) = 10 − 9 ) .

Original: The learning rate was linearly increased to 0.1 until 16k steps and set constant in following training steps.
Revised: The learning rate was linearly increased to 0.1 until 16k steps and set constant in following training steps.

Original: We used the tensorﬂow framework and trained the model until 500K step with 4 m40 GPUs.
Revised: We used the tensorﬂow framework and trained the model until 500K step with 4 m40 GPUs.

Original: We averaged 5 last checkpoints before the inference.
Revised: We averaged 5 last checkpoints before the inference.

Original: The dropout ratio for attention weights, rectiﬁed linear units, output of sub-layers, and neural network input after positional encoding were 0.2, 0.1, 0.2, and 0.2, respectively.
Revised: The dropout ratio for attention weights, rectiﬁed linear units, output of sub-layers, and neural network input after positional encoding were 0.2, 0.1, 0.2, and 0.2, respectively.

Original: The Specaug algorithm Park et al.
Revised: The Specaug algorithm Park et al.

Original: (2019) was also applied during training.
Revised: (2019) was also applied during training.

Original: The width of beam was set to 4 in all experiments.
Revised: The width of beam was set to 4 in all experiments.

Original: The output token segments are grepheme of the reference.
Revised: The output token segments are grepheme of the reference.

Original: In the LibriSpeech experiment, The learning rate was linearly increased to 0.1 until 16k steps and set constant until the model converges on dev-clean dataset.
Revised: In the LibriSpeech experiment, The learning rate was linearly increased to 0.1 until 16k steps and set constant until the model converges on dev-clean dataset.

Original: Then, we lowered learning rate to 0.02 and ﬁne-tuned until the model converges.
Revised: Then, we lowered learning rate to 0.02 and ﬁne-tuned until the model converges.

Original: We used the tensorﬂow framework and the model is trained from 7 to 10 days onp40 GPUs.
Revised: We used the tensorﬂow framework and the model is trained from 7 to 10 days onp40 GPUs.

Original: We averaged { 5 , 10 , 15 } last checkpoints before the performance evaluation.
Revised: We averaged { 5 , 10 , 15 } last checkpoints before the performance evaluation.

Original: The 1K wordpiece vocabularies extracted from training dataset is used for output token segmentation.
Revised: The 1K wordpiece vocabularies extracted from training dataset is used for output token segmentation.

Original: Other parameters are the same with those in the speech command dataset experiment.
Revised: Other parameters are the same with those in the speech command dataset experiment.

Original: A. A TTENTION PLOT FOR VARIOUS ENCODER - DECODER MODELS
Revised: A. A TTENTION PLOT FOR VARIOUS ENCODER - DECODER MODELS

Original: To show the SAGMM-tr is able to learn alignments between encoder and decoders we plot the attention score for single and multi-head models.
Revised: To show the SAGMM-tr is able to learn alignments between encoder and decoders, we plot the attention score for single and multi-head models.

Original: We used the utterance “Chapter eleven ¡long pause¿ the morrow brought a very sober looking morning the sun making only a few efforts to appear and Catherine augured from it everything most favourable to her wishes” in LibriSpeech.
Revised: We used the utterance “Chapter eleven ¡long pause¿ the morrow brought a very sober looking morning the sun making only a few efforts to appear and Catherine augured from it everything most favourable to her wishes” in LibriSpeech.

Original: The truncated GMM model fails to learn the true alignment attended several distinct windows in 4.
Revised: The truncated GMM model fails to learn the true alignment attended several distinct windows in 5.

Original: Surprisingly, the performance of truncated GMM was 4.03% and 10.16% for test-clean and test-other, respectively.
Revised: Surprisingly, the performance of truncated GMM was 4.03% and 10.16% for test-clean and test-other, respectively.

Original: The decent performance of truncated GMM implies that the GMM learns to attend as many frames as possible instead of learning alignments between sequences.
Revised: The decent performance of truncated GMM implies that the GMM learns to attend as many frames as possible instead of learning alignments between sequences.

Original: Therefore, the truncated GMM attention is not suitable for streaming inference.
Revised: Therefore, the truncated GMM attention is not suitable for streaming inference.

Revised: the morrow brought a very sober looking morning the sun making only a few efforts to appear and

Revised: Catherine augured from it everything most favourable to her wishes” in LibriSpeech

Revised: The attention in time domain longer than the original wave due to 1 -vector concatenation

Original: In the soft attention model, the several heads learns monotonic alignments between inputs and outputs as shown in Figure 5.
Revised: In the soft attention model, the several heads learns monotonic alignments between inputs and outputs as shown in Figure 6.

Original: However, other heads attend to the whole speech presence or absence intervals.
Revised: However, other heads attend to the whole speech presence or absence intervals.

Original: We assume these heads help utterance-level normalization.
Revised: We assume these heads help utterance-level normalization.

Original: While a few heads show monotonicity, the soft attention is not suitable for online inference due to softmax operation over all frames.
Revised: While a few heads show monotonicity, the soft attention is not suitable for online inference due to softmax operation over all frames.

Original: The attention alignment ﬁgure for single and multi-headed SAGMM-tr model is shown in Figures 6, 7 and 8.
Revised: The attention alignment ﬁgure for single and multi-headed SAGMM-tr model is shown in Figures 7, 8 and 9.

Original: From the ﬁgures, both models learn the optimal alignments between input wave and transcription and distinguish the word boundaries without relying on human knowledge.
Revised: From the ﬁgures, both models learn the optimal alignments between input wave and transcription and distinguish the word boundaries without relying on human knowledge.

Original: The width of attention windows in multi-head SAGMM-tr with adaptive window width are longer than those in the single-head model.
Revised: The width of attention windows in multi-head SAGMM-tr with adaptive window width are longer than those in the single-head model.

Original: The additional objective function for regularizing Σ i would be helpful to reduce the window length in ﬁgure 7.
Revised: The additional objective function for regularizing σ i would be helpful to reduce the window length in ﬁgure 8.

Original: In the multi-head model with ﬁxed window length, the model learns the multi-head monotonic alignment without increasing the window width.
Revised: In the multi-head model with ﬁxed window length, the model learns the multi-head monotonic alignment without increasing the window width.

Original: Considering the attention window width as a hyperparameter from external knowledge, it is shown that additional information could improve the model latency in SAGMM-tr while the model can also be trained without human knowledge.
Revised: Considering the attention window width as a hyperparameter from external knowledge, it is shown that additional information could improve the model latency in SAGMM-tr while the model can also be trained without human knowledge.

Original: A. A BLATION STUDIES
Revised: A. A BLATION STUDIES

Original: We tested the three ablation experiments on SAGMM-tr attention in Figure 6.
Revised: We tested the three ablation experiments on SAGMM-tr attention in Figure 6.

Original: First, we tested the model without CTC loss on encoder output.
Revised: First, we tested the model without CTC loss on encoder output.

Original: The CTC loss improves the word error rate performances of model similar to previous studies, but the model showed decent performance without the CTC loss.
Revised: The CTC loss improves the word error rate performances of model similar to previous studies, but the model showed decent performance without the CTC loss.

Original: Second, we tested the SAGMM-tr model with uni-directional encoder without initialization from the bi-directional encoder.
Revised: Second, we tested the SAGMM-tr model with uni-directional encoder without initialization from the bi-directional encoder.

Original: The result shows slightly worse performance compared to the original SAGMM-tr.
Revised: The result shows slightly worse performance compared to the original SAGMM-tr.

Original: Finally, we train the
Revised: Finally, we train the

Original: SAGMM model in the training and decoded with the SAGMM-tr decoding scheme without ﬁne-tuning.
Revised: SAGMM model in the training and decoded with the SAGMM-tr decoding scheme without ﬁne-tuning.

Original: The performance degradation in 6 shows that the mismatch between training and test stages should be removed by the ﬁne-tuning with truncated normal distribution.
Revised: The performance degradation in 6 shows that the mismatch between training and test stages should be removed by the ﬁne-tuning with truncated normal distribution.

Original: [Table]
Revised: [Table]

Original: A TTENTION PLOTS FOR SAGMM- TR IN MACHINE TRANSLATION EXPERIMENTS
Revised: A TTENTION PLOTS FOR SAGMM- TR IN MACHINE TRANSLATION EXPERIMENTS

Original: We plot the ﬁrst three decoder layers of the SAGMM-tr model with bi-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”
Revised: We plot the ﬁrst three decoder layers of the SAGMM-tr model with bi-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”

Original: and model output “Der Patient muss wirklich verstanden werden, um den Grad des Risikos seines Krebses zu verstehen, indem er ihm die M ¨ oglichkeiten zur Verf¨ugung stellt, nicht notwendigerweise mit Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen und sich stattdessen in solchen F¨allen f¨ur eine aktive ¨Uberwachung der Krankheit entscheiden.”.
Revised: and model output “Der Patient muss wirklich verstanden werden, um den Grad des Risikos seines Krebses zu verstehen, indem er ihm die M ¨ oglichkeiten zur Verf ¨ ugung stellt, nicht notwendigerweise mit Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen und sich stattdessen in solchen F ¨ allen f ¨ ur eine aktive ¨Uberwachung der Krankheit entscheiden.”.

Original: We also plot the ﬁrst three encoder-decoder attention and encoder-self attention layers of the SAGMM-tr model with uni-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”
Revised: We also plot the ﬁrst three encoder-decoder attention and encoder-self attention layers of the SAGMM-tr model with uni-directional encoder for an example utterance “The patient really needs to be made to understand the degree of risk of his cancer, by offering him the options available, not necessarily treating prostate cancers that are not long-term life threatening, and opting instead, in such cases, for active monitoring of the disease.”

Original: and model output “Der Patient muss wirklich verstanden werden, um das Risiko seines
Revised: and model output “Der Patient muss wirklich verstanden werden, um das Risiko seines

Original: Krebses zu verstehen, indem er ihm die M¨oglichkeiten zur Verf¨ugung stellt, nicht notwendigerweise mit
Revised: Krebses zu verstehen, indem er ihm die M¨oglichkeiten zur Verf¨ugung stellt, nicht notwendigerweise mit

Original: Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen, und stattdessen in solchen F¨allen
Revised: Prostatakrebs umzugehen, die nicht langfristiges Leben bedrohen, und stattdessen in solchen F¨allen

Original: f¨ur eine aktive ¨Uberwachung der Krankheit zu entscheiden.”.
Revised: f¨ur eine aktive ¨Uberwachung der Krankheit zu entscheiden.”.

Original: From the ﬁgures 9, 10 and 11, the transformers with SAGMM-tr approximates the machine translation task as a strict monotonic task and ﬁnds optimal alignments under the assumption.
Revised: From the ﬁgures 10, 11 and 12, the transformers with SAGMM-tr approximates the machine translation task as a strict monotonic task and ﬁnds optimal alignments under the assumption.

Original: Since the machine translation is not strictly monotonic task, the performance of the SAGMM-tr attention deteriorates compared to the soft attention.
Revised: Since the machine translation is not strictly monotonic task, the performance of the SAGMM-tr attention deteriorates compared to the soft attention.

Original: However, the transformer with SAGMM-tr attention and uni-directional encoder enables early inference for simultaneous translation.
Revised: However, the transformer with SAGMM-tr attention and uni-directional encoder enables online inference for simultaneous translation.

Original: A. D ECODED EXAMPLES OF SOFT ATTENTION AND SAGMM MODELS ON SPEECH COMMAND AND L IBRI S PEECH DATASETS
Revised: A. D ECODED EXAMPLES OF SOFT ATTENTION AND SAGMM MODELS ON SPEECH COMMAND AND L IBRI S PEECH DATASETS

Original: Table 7 and 8 show the decoded examples from the models with unseen sequence lengths.
Revised: Table 7 and 8 show the decoded examples from the models with unseen sequence lengths.

Original: From the tables, soft attention has difﬁculty to generate the sequences that are signiﬁcantly shorter or longer than the training corpus distribution.
Revised: From the tables, soft attention has difﬁculty to generate the sequences that are signiﬁcantly shorter or longer than the training corpus distribution.

Original: In contrast, the SAGMM-tr is robust to the sequence length mismatch and generates the transcription to arbitrary length.
Revised: In contrast, the SAGMM-tr is robust to the sequence length mismatch and generates the transcription to arbitrary length.

Original: [Table]
Revised: [Table]

Original: Transcription Reference
Revised: Transcription Reference

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in a chord words was it their colours they were voyaging across the deserts of the sky a host of nomads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and woodbegirt and citadelled and of entrenched and marshalled races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in a chord words was it their colours they were voyaging across the deserts of the sky a host of nomads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and woodbegirt and citadelled and of entrenched and marshalled races

Original: Soft attention
Revised: Soft attention

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves to love him as ourselves with the same amount and intensity of love the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and martialed races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves to love him as ourselves with the same amount and intensity of love the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleyed and martialed races

Original: [Table]
Revised: [Table]

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleys and wood be girt and set and of intrenched and marshalled races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggars weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the commandment of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to love him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay out there beyond the irish sea europe of strange tongues and valleys and wood be girt and set and of intrenched and marshalled races

Original: SAGMM-tr(mh)
Revised: SAGMM-tr(mh)

Original: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the command of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay there beyond the irish sea europe of strange tongues and valleyed and wood girt and citadel and of intrenched and marched and races
Revised: it was idle for him to move himself to be generous towards them to tell himself that if he ever came to their gates stripped of his pride beaten and in beggar’s weeds that they would be generous towards him loving him as themselves idle and embittering ﬁnally to argue against his own dispassionate certitude that the command of love bade us not to love our neighbour as ourselves with the same amount and intensity of love but to him as ourselves with the same kind of love the phrase and the day and the scene harmonized in accord words was it their colors they were voyaging across the deserts of the sky a host of no mads on the march voyaging high over ireland westward bound the europe they had come from lay there beyond the irish sea europe of strange tongues and valleyed and wood girt and citadel and of intrenched and marched and races