U NDERSTANDING THE R OLE OF S ELF A TTENTION FOR E FFICIENT S PEECH R ECOGNITION
A BSTRACT
Self-attention (SA) is a critical component of Transformer neural networks that have succeeded in automatic speech recognition (ASR).
In this paper, we analyze the role of SA in Transformer-based ASR models for not only understanding the mechanism of improved recognition accuracy but also lowering the computational complexity.
We reveal that SA performs two distinct roles: phonetic and linguistic localization.
Especially, we show by experiments that phonetic localization in the lower layers extracts phonologically meaningful features from speech and reduces the phonetic variance in the utterance for proper linguistic localization in the upper layers.
From this understanding, we discover that attention maps can be reused as long as their localization capability is preserved.
To evaluate this idea, we implement the layer-wise attention map reuse on real GPU platforms and achieve up to 1.96 times speedup in inference and 33% savings in training time with noticeably improved ASR performance for the challenging benchmark on LibriSpeech dev/test-other dataset.
I NTRODUCTION
Recent advances in end-to-end automatic speech recognition (ASR) have been driven by Transformer models (Vaswani et al., 2017).
Transformer was ﬁrst introduced for natural language processing (NLP) tasks such as neural machine translation (Vaswani et al., 2017; Ott et al., 2018), language modeling (Dai et al., 2019; Rae et al., 2019), and text generation (Raffel et al., 2020).
Thanks to its superior performance in processing sequence input, Transformer has been widely adopted in various state-of-art ASR models (Zhang et al., 2020b; Ng et al., 2021; Guo et al., 2021).
Self-attention (SA) is a core component of Transformer-based ASR, which dynamically collects information from multiple frames of an audio sequence.
However, the computation and memory costs of SA increase quadratically with the length of a sequence, which is particularly problematic for ASR.
For example, just a 30-second utterance corresponds to about 750 frames with a widely used window stride of 40ms.
Understanding the role of SA may provide essential insights for the efﬁcient design of Transformerbased ASR models.
Extensive studies have examined the behavior of SA in the ﬁeld of NLP (Kovaleva et al., 2019; Park et al., 2019;
Gong et al., 2019; Rogers et al., 2020).
Recently, several studies further attempted to discover the characteristics of SA in the speech domain.
Yang et al.
(2020) revealed that self-attention features in the self-supervised audio Transformer are categorized into global, vertical, and diagonal patterns.
Zhang et al.
(2021b) focused on the diagonality of upper
SA layers in ASR models for improving efﬁciency.
However, these prior works revealed limited insights on the patterns discovered in SA, constraining its use for improving model efﬁciency.
Thus, providing a holistic view of the role of SA is desirable for efﬁcient ASR model design.
In this work, we reveal that SA plays two distinct roles in the success of Transformer-based ASR models: phonetic and linguistic localization, as illustrated in Figure 1.
First, phonetic localization of lower SA layers attends to the phonologically meaningful global context.
Second, linguistic localization of upper SA layers mainly attends to the local context of a near-diagonal attention map.
We hypothesize that the phonetic variance in utterances such as variations in pronunciation is standardized in the lower SA layers so that the upper SA layers can identify local linguistic features for accurate transcription.
To investigate the behavior of SA layers, we propose phoneme attention relationship (PAR) to explain how phoneme localization works quantitatively.
Interestingly, we discover that the phonetic localization represents the traditional knowledge on phonetics; for example, labial, velar, or nasal phonemes tend to attend to each other.
Based on this understanding, we propose a practical method for efﬁcient ASR model design.
We reuse the attention maps of SA layers while preserving the phonetic localization capability of the lower SA layers, resulting in up to 1.96x speedup in inference and 33% savings of training time with considerably improved performance in challenging ASR tasks (LibriSpeech dev/test-other dataset).
Our contributions can be summarized as follows:
• We reveal that SA layers contribute to ASR with two distinct roles: phonetic localization in the lower layers and linguistic localization in the upper layers.
This unique distinction leads to an in-depth analysis of phonetic SA for the ﬁrst time.
We further propose phoneme attention relationship (PAR) to quantitatively identify the role of phonetic localization.
• We propose layer-wise attention map reuse for efﬁcient Transformer-based ASR models.
In particular, we discover that attention map reuse is possible in lower SA layers as long as the phonetic localization property quantiﬁed by PAR is preserved.
We demonstrate with the popular ASR model and dataset that ASR performance can be maintained or slightly improved even if the attention map is reused.
• We implement the attention map reuse on real GPU platforms and achieve up to 1.96x inference speedup and 33% savings in training time, demonstrating that the proposed method is practical.
B ACKGROUND 2. ASR E NCODER AND S ELF -A TTENTION
SA is usually utilized as a module inside the ASR model consisting of stacked Transformer encoder layers.
ASR encoder takes a sequence of short-time Fourier-transformed (STFT) audio features, known as a ‘frame’, as input and extracts a high-level feature of each frame through multiple layers.
As illustrated in Figure 1, the extracted high-level feature changes for each layer; stacked SA layers ﬁrst extract phonetic features from audio features and utilize these features to build linguistic features for the output transcription.
We brieﬂy review the SA 1 computation procedure.
Consider a sequence of d -dimensional column vectors X = { x 1 , x 2 , ...x T } as input.
Each vector corresponds to each frame of speech where the total number of T frames are included.
The input feature vector X is projected to query ( Q ), key ( K ), and value ( V ) of h -th attention head as follows:
[Equation]
W Q,K,V indicates projection matrices for Q , K , V , respectively.
d h =
d/H is the dimension of each attention head where H is the number of attention heads.
The attention map ( A h ), which represents how much frames attend to each other, is computed by scaled dot-product operation followed by softmax.
The resulting attention map takes a form of a 2D matrix where each row is a probability vector.
A single element of the attention map ( A h [ i, j ] ) represents how much i -th frame attends to j -th frame.
The attention head ( d h ) is a weighted sum of V using the attention map as weight.
Note that each attention head corresponds to a different attention map A h ; this multi-head design enables focusing on various perspectives within a single SA layer.
The output O = { o 1 , o 2 , ...
o T } is computed by the projection ( W O ∈ R d × d ) on the concatenated attention heads 2 .
[Equation]
SA layer contains O ( d [Equation] ) parameters.
As shown in equations, SA requires quadratic computation and memory complexity
O ( T 2 ) , in exchange for the ability to access any location in the sequence.
When N layers of SA are stacked, the burden proportionally increases.
2. P REVIOUS W ORK ON S ELF -A TTENTION A NALYSIS
ASR considers both phonetic and linguistic aspects to transform audio input to text output.
However, the studies on NLP mostly analyze the linguistic characteristics of SA, and the studies on selfsupervised audio representation learning (SSAL) mainly focus on the phonetic behaviors of SA.
The valuable ﬁndings from both domains cannot be directly applied to ASR.
NLP The behavior of SA has been widely studied in the NLP domain (Rogers et al., 2020), mostly focused on BERT (Devlin et al., 2019), a self-supervised language representation learning model.
Kovaleva et al.
and Guan et al.
(2020) suggested that attention patterns can be clustered into several groups and the pattern may change depending on the ﬁne-tuning task.
Clark et al.
and
Tenney et al.
(2019) observed attention maps that correspond to linguistic concepts of the language.
Voita et al.
(2019) also characterized linguistic attention heads and connected the knowledge to efﬁcient model structure.
However, studies on NLP only provide analysis on linguistic attention.
SSAL Recently, several studies have been introduced to understand how the audio information is encoded in SSAL models, such as CPC (Oord et al., 2018), Wav2Vec 2.0 (Baevski et al., 2020), Mockingjay (Liu et al., 2020), HuBERT (Hsu et al., 2021), and Audio ALBERT (Chi et al., 2021).
Ma et al.
(2021) and Shah et al.
(2021) demonstrated that a wide spectrum of phonetic information is included in these models.
Especially, Yang et al.
(2020) categorized attention maps into three categories: global, vertical, and diagonal, where diagonal heads attend to local frames and vertical heads either focus or neglect speciﬁc phonemes.
However, Yang et al.
(2020) only discovered attention patterns without an explanation on how phonetic feature extraction is achieved with these patterns.
ASR Previous works have investigated the redundancy of attention maps mainly based on diagonality.
From the observation that attention maps in upper layers show highly diagonal patterns,
Zhang et al.
(2021b) proposed replacing upper SA layers to feed-forward layers without performance loss.
Zhang et al.
(2021a) removed SA heads of high diagonality during training as a regularization but keep every head for the test time.
Similarly, Chang et al.
(2020) introduced adaptive attention span where each attention head equips a different attention span width to reduce the sequence length for the computation, starting from the intuition that some heads only attend to neighboring frames.
These approaches mainly focus on reducing the burden of diagonal and concentrated attention, however, diagonality-based analysis has limitations in optimizing phonetic attention.
We distinguish SA into two groups and provide proper analysis for each.
U NDERSTANDING THE R OLE OF S ELF -A TTENTION IN ASR 3. A NALYSIS S ETUP
We train and evaluate the model on the LibriSpeech-960 (Panayotov et al., 2015) dataset.
The dataset include two types of data, clean and other , where other contains more challenging utterances.
We extract the 80-dimensional log-Mel ﬁlterbank feature from a 25ms window with a stride of 10ms.
We use 128 sub-word tokens as vocabulary, built on SentencePiece (Kudo & Richardson, 2018) library using the byte-pair encoding (Sennrich et al., 2016).
The analyses are performed on LibriSpeech test-clean dataset unless speciﬁed.
We use Conformer-M(medium)
(Gulati et al., 2020) as the baseline ASR encoder, trained with CTC (Graves et al., 2006) loss.
Conformer is a variant of Transformer, which combines an additional convolution module to enhance the ability to collect local neighboring features.
We follow recently introduced state-of-the-art ASR studies (Zhang et al., 2020b; Ng et al., 2021; Guo et al., 2021) that have adopted Conformer as their ASR encoder.
We train the model with AdamW (Loshchilov & Hutter, 2018) optimizer for 200K iterations.
Please see Appendix A.1 and A.2 for the model conﬁguration and training details.
3. D ISTINGUISHING BETWEEN P HONETIC AND L INGUISTIC S ELF A TTENTIONS
To understand the role of SA, we start by examining the attention map A h , which characterizes the functionality of SA.
The attention map indicates how a frame attends the other frames in terms of probabilistic distribution for each attention head.
Thus, analysis on attention maps across the heads of a SA layer would discover important characteristics of SA.
To measure the diagonality of the attention map, we introduce cumulative attention diagonality (CAD) deﬁned as the integral of the sum of attention probabilities constrained by distance as below:
[Equation]
where T is the number of frames in a sequence, r determines the range of distribution in the attention map under test, and h is an index of the attention head.
Appendix B.1.1 provides detailed explanations of above equation and visualizes some CAD examples.
Figure 2 shows the attention diagonality analysis for the SA layers.
There is a clear transition of
CAD from lower layers (layer 1-8) to upper layers (layer 9-16).
We emphasize this is a unique trend observed in ASR models compared to other domains: Kovaleva et al.
classiﬁed attention maps in BERT but did not report the grouping of the same types.
Yang et al.
(2020) also categorized attention maps in SSAL but those categories broadly appear through layers.
In ASR, Zhang et al.
(2021b) observed a similar diagonality pattern for SA layers, but they considered it as an increase of diagonality over layers and did not separate the distinct patterns.
From the diagonality analysis, we categorize the role of SA layers into two parts: phonetic and linguistic localization.
In ASR, linguistic localization refers to the behavior of the attention map that focuses on local (near in distance) frames and aggregates the information for text transcription.
Since these local frames are particularly important in audio to text transcription in ASR, the attention map is characterized with a diagonal pattern.
As shown in Figure 2, the upper layers tend to exhibit diagonally dominant patterns, implying their role as a linguistic localizer.
We visualize two cases of diagonal attention patterns in Figure 1, extracted from layers 12 and 15 of the baseline model.
Our observation is consistent with the prior work such as Zhang et al.
(2021b).
On the other hand, there has been little discussion about the role of lower layers in ASR in the context of phoneme localization.
We discuss this in detail in the next section.
3. C HARACTERISTIC OF P HONETIC L OCALIZATION
Phonetic localization denotes the role of the attention map that focuses on similar (near in content) frames and extracts the phonologically meaningful features.
We observe two characteristics of phonetic localization.
First, phonetic localization is realized as attention to similar phonemes across the sequence.
Second, the localization transforms each corresponding frame more likely to others.
To analyze phonetic attention, we exploit the phoneme information to ﬁnd out the relationship between attention and phonemes.
For the LibriSpeech dataset, we use the frame-level phoneme alignments obtained from Montreal Forced Aligner
(McAuliffe et al., 2017).
Table 1 lists all phoneme classes.
Please visit Appendix A.3 for details on phoneme pre-processing.
Idx.
[Table]
EH ER
[Table]
IY
IH
O
UH UW
L
R
M
N
NG
Idx.
3335 Phn.
B
D
[Table]
V
[Table]
Figure 3 demonstrates the ﬁrst characteristic of phoneme localization.
The attention map presents that the same or similar phonemes tend to assign high attention weight to each other, for example, (‘S’ to ‘Z’), (‘ER’ to ‘R’), and (‘IY’ to ‘IH’).
We evaluate our second statement by phoneme classiﬁcation on hidden layer representations (output of layers, also known as a hidden activation or hidden vector), similar to previous approaches (Baevski et al., 2021; Liu et al., 2021).
We extract hidden layer representations and train a softmax classiﬁer for each layer.
Input is 256-dimensional vector and output contains total 37 output classes (36 phonemes + “silence”).
See Appendix A.5 for the details on training the classiﬁer and visualization of the confusion matrix.
Figure 4 shows the phoneme classiﬁcation accuracy for different layers.
The accuracy increases for the lower layers (layer 1-8) where phonetic localization dominates.
Speciﬁcally, the accuracy of layer 0 (before SA) is only 53.8%, but it consistently increases to 81.7% at layer 8.
The phoneme classiﬁcation accuracy indicates how well hidden vectors can be distinguished according to their phoneme classes.
Therefore, the accuracy increase on lower layers implies that hidden layer representations are more standardized by phonetic localization.
We assume that the accuracy decreases for the upper layers because they perform the linguistic localization and convert phoneme-level information to output text.
3. L AYER - WISE A NALYSIS OF P HONETIC A TTENTIONS
In phonetic attention, we observed that similar phonemes attend to each other and the phonetic features are clustered through layers.
For the next step, we investigate the layer-wise behavior of phonetic attention to understand the contribution of each phonetic attention layer.
We introduce phoneme attention relationship (PAR) to understand how SA processes phonological information by exposing how much each phoneme class attends to the other phonemes on average.
Speciﬁcally, we directly map an attention probability
A h
[ i, j ] to P h
[ p, q ] where i -th frame and j -th frame correspond to phoneme p and q , respectively.
If two phoneme classes p and q are different, we simply transport the probability from A h
[ i, j ] to P h
[ p, q ] .
On the other hand, if two frames are within the same phoneme class p , we exclude consecutive frames of the same class to avoid unnecessarily emphasize the effect of diagonal attention maps.
PAR is computed as below 3 :
[Equation]
C p , C q indicates the set of frame indices that correspond to phoneme class p and q .
E p ( i ) indicates the number of frames that satisﬁes two conditions: belong to the same class p as i -th frame and all frames between itself and i -th frame also belong to the same class.
In other words, C p − E p ( i ) indicates the subset of C p that are not connected to i -th frame by consecutive class p frames.
For the P h [ p, p ] calculation, we do not include consecutive frames of the same class.
We discuss the purpose of this exclusion in Appendix B.2.1.
We visualize the average PAR of the lower half and upper half of layers in Figure 5.
A prominent diagonal component appears, representing that phonemes put high attention to themselves.
Interestingly, we also discover well-known phonological characteristics in the lower half of layers.
For example, labial (B, P), velar (G, K), and alveolar (S, Z) consonants highly attend to each other.
Nasal phonemes (M, N, NG) also show a high correlation.
From the empirical observations, we denote that phonetic attention map creates heterogeneous patterns.
Thanks to the multi-head structure, a single SA layer can capture multiple phonetic relationships.
Please see Figure 12 in Appendix B.2.
for various PAR examples for each head.
If each phonetic attention map corresponds to different relationships, can we reuse phonetic attention maps across multiple layers?
We answer this question by introducing the PAR coverage, which
[Figure]
The right table summarizes the coverage of different reuse conﬁgurations.
A higher average perlayer coverage ratio implies that each layer performs more similarly to the indicates how much each layer covers the phonetic relationship represented in the averaged PAR from the baseline (Figure 5(a)).
The averaged PAR of the baseline is considered to represent all the essential phoneme relationships.
Please refer to Appendix A.4 for details.
To investigate the effect of reuse, we plot the accumulated PAR coverage for different reuse conﬁgurations in Figure 6.
In calculating the accumulated coverage ratio, we take the average on every PAR under a certain layer and compare it with the reference PAR.
The accumulated coverage consistently increases to 1, which means that the missing relationships are fulﬁlled through layers.
We test four conﬁgurations (will be introduced in detail in the next Section)
X × Y , where X layers share the same attention map.
As the number of reuse increases ( 2 × 8 → 4 × 4 → 8 × 2 ), the average per-layer coverage also grows, which implies that each SA layer tries to capture more phonetic relationships to recover the performance.
However, for 8 × 2 , the model fails to fully cover the reference, represented as a low accumulated coverage ratio of 0 . 974 .
In Appendix B.3, we visualize the effect of the reuse on PAR coverage where phonetic features captured in 2 × 8 and 4 × 4 are [Table] 2 8( H4 )
[Table]
4( H4 ) + 4( H4 )
[Table]
[Table]
H4 )
[Table]
8( H4 )
[Table]
[Table]
[Table]
[Table]
L AYER - WISE A TTENTION M AP R EUSE
We propose layer-wise attention map reuse , a method to design an efﬁcient Transformer-based ASR encoder by reducing the heavy SA computation.
The core idea is to reuse the computed attention map from the previous layer.
More speciﬁcally, we reuse attention map of l -th SA layer to ( l +1) , ( l + 2) , ...
( l + M − 1) -th consecutive SA layers.
If a single attention map is shared through M layers, the computation burden of SA can be reduced by M times.
During training, the reused attention map receives gradients from M layers.
This layer-wise reuse is easy to implement and fully supported by modern accelerator hardware.
The idea of reuse attention map through layers have been proposed for NLP (Xiao et al., 2019;
Ying et al., 2021) but not tested for ASR.
We discuss the difference in
Section 5.
For SA layers that receive the pre-computed attention map, query and key are not used and can be removed.
To compensate the parameter size for those layers, we simply double the output dimension of V ( W Vh ∈ R 2 d h × d , W O ∈ R d × 2 d ) .
Table 2 shows the word error rate (WER) on LibriSpeech dev and test dataset.
Conﬁguration “ X × Y ” indicates that X successive layers are grouped to share the same attention map and total Y groups are built.
Therefore, there exist HXY unique attention heads for each model.
We also use the notation ‘ + ’, for example, 4 × 4 is identical to 4 + 4 + 4 + 4 .
For each conﬁguration, we train the model from scratch with the same training setup as the baseline.
Note that the increased number of heads comes with the decreased per-head dimension to keep the parameter size comparable.
Best and Worst We ﬁrst compare the best ( 4 ( H8 ) × 4 ) and the worst conﬁguration ( 8 ( H4 ) × 2 ).
The worst is the most naive setting that just applies very aggressive attention map reuse.
Although the speed is about the same, performance can be improved by increasing the number of heads ( 8 ( H8 ) × 2 ).
In contrast, the best working setting is carefully designed to maximize performance.
For example, equipping the same number of heads ( 2 ( H4 ) × 8 ) does not show similar performance compared to the best conﬁguration.
Sensitivity to Reuse We examine which of the phonetic or linguistic localization is more sensitive to attention map reuse.
Comparing two conﬁgurations with the identical number of heads and head dimensions ( 4+4+8 vs. 8+4+4 ) in the 5th block of Table 2, we conclude the phonetic localization suffers more from increasing the reuse of layers.
In other words, the linguistic localization seems to be more robust to the reuse.
We conjecture that too few phonetic localization heads fail to capture every essential relationship.
Number of Heads in Phonetic Localization To better understand the trade-off between the number of heads and head dimension, we conduct three experiments that only differ on the number of heads in lower layers.
As shown in the 3rd block of Table 2, among the three conﬁgurations,
[Table]
( 4 ( H4 ) × 4 )
surpasses the other two in 3 over 4 benchmarks.
We expect a trade-off between the number of phonetic localization heads and per-head dimension; the former enables more various aspects to be covered while the latter helps richer representation for each head.
Number of Heads in Linguistic Localization
We perform the same experiment as above for the upper layers, shown in the 4th block of Table 2.
Interestingly, we found that increasing the number of heads for linguistic localization tends to improve the overall performance.
In addition, a considerable performance loss is detected when the number of heads is decreased to 2( H2 ).
This observation implies that the previous studies that only reduce linguistic attention may face limitations when the remaining linguistic heads are too few.
Inference and Training Speed Table 3 compares different reuse conﬁgurations.
Both training and inference speed can be greatly improved by reducing the number of attention computation.
The impact becomes more signiﬁcant for longer sequences as T increases.
Our best conﬁguration ( 4 ( H8 ) × 4 ) accelerates the inference by 1.96x times (for 30-second utterance) and reduces training cost by 33%.
Note that the number of parameters is almost equivalent for all conﬁgurations because of the expansion of V dimension.
Inference speed is evaluated on a single RTX-Titan(24GB) GPU and training cost is measured in GPU-hours on A100(40GB) GPU.
R ELATED W ORK
Attention Map Reuse in NLP Xiao et al.
(2019) proposed sharing of attention map through consecutive layers for neural machine translation.
They determine the reuse policy by Jensen-Shannon divergence (JSD) values between two attention maps.
Ying et al.
(2021) propose a similar approach for BERT but with manual reuse conﬁgurations.
In addition to the critical difference in the domain (NLP vs. ASR), both works depend on the similarities of the attention map, however, do not investigate why the similarity is developed.
Efﬁcient Attention Map Computation for ASR Several studies have been proposed to reduce the cost of attention computation for ASR.
Wang et al.
(2021) proposed a prob-sparse SA that only computes the top-k queries that are less uniform.
For the streaming purpose, block processing of input sequence has been widely adopted (Yeh et al., 2021; Shi et al., 2021).
Masked attention, which restricts the attention range to local neighbors, have also been used (Zhang et al., 2020a;
Tripathi et al., 2020; Audhkhasi et al., 2021).
While these approaches focus on reducing the effective sequence length, our method directly reduces SA computation.
C ONCLUSION
In this paper, we analyze the role of self-attention in Transformer-based ASR and show that the role can be distinguished into two types: phonetic and linguistic localization.
Especially, we showed that the phonetic localization captures various phonetic relationships and contributes to the performance by standardizing the features over similar phonemes, veriﬁed by the increasing phoneme classiﬁcation accuracy over lower layers.
The distinguished roles of SA in lower and upper layers also lead to an efﬁcient ASR model that reuses the attention map for multiple SA layers.
The proposed method has achieved a signiﬁcant 1.96 times of speedup in inference and 33% reduced training time, with the reduction of word error rate from 8.40% to 8.05% on the LibriSpeech test-other dataset.
We explain extra details for the model architecture, training procedure, pre-processing steps and experiments for analyses in Appendix A. We also provide the source code for the experiments in supplemental materials. A CKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant funded by
Korea government (MSIT) (No. 2021R1A2C1013513).
This work was also partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by MSIT (No. 2020-0-01373, No. 2021-0-00020-001). This work was also supported in part by
Samsung Advanced Institute of Technology, Samsung Electronics Co.,
Ltd.
This work was also partly supported by the Google AI Focused Research Awards Program awarded to Wonyong Sung.
We gratefully acknowledge the GCP credit support from Google AI and the GPU server support from the Artiﬁcial Intelligence Cluster Agency (AICA, Korea)
A D ETAILS A. M ODEL
Our baseline model is Conformer-M(medium)
(Gulati et al., 2020) trained with CTC (Graves et al., 2006) loss.
Table 4 shows the conﬁguration of the model.
Unspeciﬁed details follow the original
Conformer paper.
We observed that SyncBN (Peng et al., 2018) is critical for the overall performance.
We employ weak attention suppression (WAS) (Shi et al., 2020) of γ = 0 .
5 for faster convergence and improved performance.
[Table]
We decide to use Conformer as our baseline, following recent state-of-the-art ASR models, because these models can beneﬁt the most from the proposed efﬁcient model design.
However, there may be several concerns on the clarity of our analysis on SA because the convolution module is jointly used with SA inside Conformer.
Because the ability to gather information from the entire sequence is only equipped in SA, the analysis results on the role of phonetic heads could not be presented without SA.
The convolution kernel size of 31, which covers about 1.2 seconds, is too short to gather long-range information.
We believe that the convolution module may guide the model to focus on local information ﬁrst at the early stage of the training, however, the role of SA is not much affected by the difference between Conformer and Transformer.
A. T RAINING Optimizer & Scheduler
[Table]
We use AdamW (Loshchilov & Hutter, 2018) optimizer with the inverse square-root learning rate schedule (Vaswani et al., 2017).
Table 4 shows the training conﬁguration.
We linearly increase the learning rate (LR) to the maximum value for 5K iterations and keep LR at maximum for 95K iterations, followed by 100K iterations of LR decrease.
We use 4x A100(40GB) GPUs for the experiments.
To ﬁt the batch size of 480 in these GPUs, we assign 40 samples per GPU and accumulate the gradient of 3 batches.
We don’t use bucketing for generating the mini-batch during training.
We also employ adaptive SpecAugment (Park et al., 2020), stochastic weight averaging (SWA) (Izmailov et al., 2018), and variational noise.
A. P HONEME P RE -P ROCESSING
For phonetic analyses, We employ the collapsed list of phonemes that are included in the LibriSpeech lexicon.
We collapse (‘AA, AO’ to ‘AA’), (‘OW, OY’ to ‘O’), and (‘SH, ZH’ to ‘SH’), which leads to the phoneme classes in Table 1, for better understanding the characteristics.
Because the LibriSpeech dataset does not provide frame-wise phoneme alignments, we extract the phoneme alignment from MFA (McAuliffe et al., 2017) and map these alignments to each frame.
Especially, we exploit the fact that each frame corresponds to a 40ms interval after passing through two convolutional layers (convolutional sub-sampling) of stride 2 in front of the model.
We assign the phoneme class to each frame if the center of the frame is within the phoneme duration, including the ‘silence’ phoneme.
Except for the phoneme classiﬁcation, our analysis excludes ‘silence’ frames by exploiting framelevel phoneme alignments.
Then, we re-normalize the remaining attention probability to preserve the probability sum to 1.
We observe that these silence frames sometimes consume too much probability mass for both linguistic and phonetic heads, which makes our analysis difﬁcult.
Note that this phenomenon of assigning strong attention to ambiguous tokens, such as [CLS] or [SEP], has been also reported in NLP (Kobayashi et al., 2020; Sun & Marasovi´c, 2021).
Recently, Kobayashi et al.
(2020) introduced the concept of norm-based analysis and reported that those [CLS] and [SEP] does not contribute much to the output even though their attention weight is large.
We leave analysis using effective attention as a future work.
[Figure]
in the reference phoneme attention relationship of Figure 5(a).
The coverage ratio R of the target PAR compared to the reference PAR is calculated as below:
[Equation]
P indicates every phoneme class and Q p indicates top-10 phoneme classes in the order of the largest PAR elements for the phoneme p ( R h [ p, :] ).
Top-10 classes are visualized in Figure 7.
We exploit topmost phoneme classes because our interest is at the important relationships.
A. P HONEME C LASSIFICATION
For the phoneme classiﬁcation, we train a single fully-connected layer as a classiﬁer.
We choose the simplest architecture as a classiﬁer to more directly correlate the phoneme accuracy and hidden layer representations.
These representations are extracted from dev-clean and dev-other dataset and evaluated on test-clean and test-other dataset.
We use SGD with a learning rate of 0.1, momentum of 0.9, and weight decay of 1e-3.
The training takes 15 epochs, where the learning rate is multiplied by 0.1 for every 3 epochs.
We visualize confusion matrices of the phoneme classiﬁcation in Figure 8.
As layer proceeds, wrongly classiﬁed phonemes (non-diagonal) disappear and leave a clear diagonal line on the confusion matrix.
B A DDITIONAL R ESULTS AND D ISCUSSIONS B. D IAGONALITY B.1. CAD RESULTS
The CAD is a good indicator of how fast the accumulated attention increases over the distance, directly represents the diagonality of the attention weight.
The CAD is interpreted as the area under the function D ( r ) , where D ( r ) calculates the amount of total attention weight within the restricted range j ∈
[ i − r
( T − 1) , i + r
( T − 1)] .
( T − 1) is the maximum possible distance between two frames where T is the number of frames. [Equation]
[Equation]
If the r is same, a larger D ( r ) means that the attention is more concentrated near the diagonal.
Please note that D ( r ) is a monotonically increasing function whose output is always in the range [0 , 1] .
To help understand the concept of CAD, we visualize two typical examples of the cumulative attention diagonality in Figure 9.
For layer 2, where attention heads perform the phonetic localization,
CAD values are low.
In contrast, for layer 14 that concentrates on linguistic localization, CAD values are much higher.
Figure 10 plots the sorted CAD values.
We determined the threshold (0.75) where the curve of sorted CAD values changes from convex to concave.
The higher CAD value represents that more probability mass is concentrated near the diagonal in the attention map, while the lower CAD value implies that the distribution is more uniformly distributed.
We observe that diagonally concentrated heads take a large portion in upper layers.
B.1. P REVIOUS D IAGONALITY A NALYSIS
Previous studies utilized a different metric to calculate the diagonality of an attention map (Zhang et al., 2021b;
Yang et al., 2020).
In this version of diagonality, the metric is interpreted as the negative normalized average attention distance (span-length).
In Yang et al.
(2020), the diagonality is calculated as below:
[Equation]
We introduced the cumulative attention diagonality (CAD) because the previous diagonality metric lacks information about how attention is distributed by distance.
In other words, CAD is more comprehensive because it provides the overall diagonality value as well as the tendency of the attention according to the distance.
Our CAD metric dearly captures the ﬂat region in lower layers (Figure 2).
In contrast, in Zhang et al.
(2021b), the diagonality increases from the lower layers to the upper layers, which may not be sufﬁcient observation in ASR.
B. P HONEME A TTENTION R ELATIONSHIP B.2. E XCLUDING C ONSECUTIVE F RAMES
We treat P h
[ p, q ]( p (cid:54) =
q ) and P h
[ p, p ] differently because we want to separate the effect of the diagonal (position-based) attention map.
An attention map that highly focuses on the surroundings (diagonal-like) can unintentionally disturb the purpose of PAR in measuring P h [ p, p ] , because it will give a high value to p - p relationship not because the contents are similar, but because the location is close.
Our purpose on PAR is to examine the phonetic (content-based) behavior, so excluding consecutive frames of the same phoneme class better represents SA in lower layers that correspond to the phonetic localization.
B.2. PAR R ESULTS
Figure 11 visualizes how averaged PAR changes through layers.
Lower layers (ﬁrst row in Figure) show noticeable regions that represent the important phoneme relationships, including the diagonal.
Note that the diagonal stands out even though we excluded the consecutive same phoneme classes.
Each lower layer focuses on different aspects, supporting our analysis on the PAR coverage (Figure 6) that the accumulated coverage ratio continuously increases through layers.
In other words, each layer covers a certain part of the reference PAR with less overlap between layers.
On the other hand, the upper layers (second row in Figure) do not show the emphasized pattern.
This is expected for linguistic localization heads that generate highly diagonal attention maps because their attention is mainly assigned to near frames regardless of their phoneme classes.
For the ﬁner understanding, we also visualize how attention heads compose the averaged PAR.
Figure 12 shows that each head corresponds to different phonological properties but their averaged interests mimic the reference PAR.
This emphasizes the importance of multiple attention heads in phonetic localization; each head specializes in capturing the speciﬁc phoneme relationship and contributes differently.
B.2. P REVIOUS P HONEME R ELATIONSHIP A NALYSIS
The idea of phoneme relationship analysis is ﬁrst introduced in Yang et al.
as phoneme relation map (PRM), but our PAR is different from PRM in two ways.
First and the most difference is that PRM do not distinguish consecutive frames and discontinuous frames that corresponds to the same phoneme class.
Therefore, attention heads that only focus on neighbors would also present heavy self-to-self phoneme relationship in PRM, which hinders the clarity of the analysis.
Second,
PRM do not apply correction according to the sequence length.
We multiply T to reduce the effect of sequence length, inspired by the fact that the expectation of (averaged) probability is potentially smaller for longer sequences.
B. A TTENTION M AP R EUSE AND PAR
For the understanding of the performance loss for extreme reuse cases, we visualize the PAR of different conﬁgurations in Figure 13.
We observe that most of the patterns resemble the baseline model, which implies that heads learn similar roles during training.
However, for 8 × 2 , several information is lost; diagonal became unclear and highlighted correlations disappeared.
B. C OMPARISON TO M ASKED A TTENTION
We compare the masked attention (Zhang et al., 2020a; Tripathi et al., 2020; Audhkhasi et al., 2021;
Huang et al., 2020) with the proposed attention map reuse.
Masked attention, similar to block-based attention (Shen et al., 2018; Qiu et al., 2020), is a method to reduce the computational burden of self-attention by restricting the length of the accessible context.
If masked attention is adopted for ASR, each frame only attends to local neighbors; L frames to the left and R frames to the right, denoted as [ − L, R ] .
[Table]
We train the models with the same setting as the Conformer-M baseline (as in Table 2) but with a limited attention range for either lower or upper layers.
Among the 16 layers in the baseline, lower and upper layers consist of 8 layers each and correspond to phonetic localization and linguistic localization, respectively.
For selected layers, we restrict each frame to only attend to near neighbors within the distance of 64 frames.
Table 6 shows the performance on the LibriSpeech dataset.
For lower layers, attention range restriction causes a clear degradation of the recognition accuracy.
We consider that phonetic localization in lower layers demands a wide range of attention.
On the other hand, for upper layers, attention range restriction shows almost comparable performance to the baseline, and even better in some subsets.
We hypothesize that the upper layers for CTC-based ASR may not require a very wide context, because their attention pattern is highly diagonal.
In addition, the restriction-based computational savings seems to be no larger than the proposed attention reuse.
A context range of 128 (64+64) frames corresponds to about 5.1 seconds, where the average utterance length of the corpora is about 7.4 seconds.
Therefore, we expect approximately 30% reduction in attention calculation when masked attention is applied to every layer.
Our attention reuse ( 4 × 4 ) reduces about 75% of the attention computation without any degradation in the performance.