Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Abstract
The recently proposed Conformer model has become the de facto backbone modelfor various downstream speech tasks based on its hybrid attention-convolutionarchitecture that captures both local and global features.
However, through a seriesof systematic studies, we find that the Conformer architecture’s design choicesare not optimal.
After reexamining the design choices for both the macro andmicro-architecture of Conformer, we propose Squeezeformer which consistentlyoutperforms the state-of-the-art ASR models under the same training schemes.
Inparticular, for the macro-architecture, Squeezeformer incorporates (i) the Temporal
U-Net structure which reduces the cost of the multi-head attention modules on longsequences, and (ii) a simpler block structure of feed-forward module followed upby multi-head attention or convolution modules instead of the Macaron structureproposed in Conformer.
Furthermore, for the micro-architecture, Squeezeformer(i) simplifies the activations in the convolutional block, (ii) removes redundant
Layer Normalization operations, and (iii) incorporates an efficient depth-wisedownsampling layer to efficiently sub-sample the input signal.
Squeezeformerachieves state-of-the-art results of 7.5%, 6.5%, and 6.0% word-error-rate (WER)on Librispeech test-other without external language models, which are 3.1%, 1.4%,and 0.6% better than Conformer-CTC with the same number of FLOPs.
Our codeis open sourced and available online [1].
Introduction
The increasing success of end-to-end neural network models has been a huge driving force for thedrastic advancements in various automatic speech recognition (ASR) tasks.
While both convolutionalneural networks (CNN) [17, 24, 26, 33] and Transformers [22, 28, 29, 53, 54] have drawn attention aspopular backbone architectures for ASR models, each of them has several limitations.
Generally, CNNmodels lack the ability to capture global contexts and Transformers involve prohibitive computing andmemory overhead.
To overcome these shortcomings, Conformer [14] has recently proposed a novelconvolution-augmented Transformer architecture.
Due to its ability to synchronously capture globaland local features from audio signals, Conformer has become the de facto model not only for ASRtasks, but also for various end-to-end speech processing tasks [15].
Furthermore, it has also achievedthe state-of-the-art performance in combination with recent developments in semi-supervised learningmethodologies as well [34, 57].
Despite being a key architecture in speech processing tasks, the Conformer architecture has somelimitations that can be improved upon.
First, Conformer still suffers from the quadratic complexityof the attention mechanism limiting it’s efficiency on long sequence lengths This problem is further
[Figure]
M 71.7 57.0 57.0 58.4 58.4 42.
B a s e li n e + T e m p o r a l U - N e t + T X - S t
y l e B l o c k
+ U n
i f i e d A c t i v a t i o n s + S
i m p
li f i e d L a y e r N o r m + D W
S e p .
S u b s a m p li n g + M o d e l
S c a l e - u p 7.
GFLOPs
Squeezeformer-M highlighted by the long sequence lengths of typical audio inputs as alow pointed out in [43].
Furthermore, the Conformer architecture is relatively more complicated than Transformer architectures usedin other domains such as in natural language processing [8, 41, 48] or computer vision [10, 11, 46].
For instance, the Conformer architecture incorporates multiple different normalization schemes andactivation functions, the Macaron structure [31], as well as back-to-back multi-head attention (MHA)and convolution modules.
This level of complexity makes it difficult to efficiently deploy the modelon dedicated hardware platforms for inference [23, 36, 52].
More importantly, this raises the questionof whether such design choices are necessary and optimal for achieving good performance in ASRtasks.
In this paper, we perform a careful and systematic analysis of each of the design choices with thegoal of achieving lower word-error-rate (WER) for a given computational budget.
We developeda much simpler and more efficient hybrid attention-convolution architecture in both its macro andmicro-design that consistently outperforms the state-of-the-art ASR models.
In particular, we makethe following contributions in our proposed Squeezeformer model:
• We find a high temporal redundancy in the learned feature representations of neighboring speechframes especially deeper in the network which results in unnecessary computational overhead.
Toaddress this, we incorporate the temporal U-Net structure in which a downsampling layer doublesthe sampling rate at the middle of the network, and a light upsampling layer recovers the temporalresolution at the end for training stability (§ 3.1.1).• We redesign the hybrid attention-convolution architecture based on our observation that the backto-back MHA and convolution modules with the Macaron structure is suboptimal.
In particular,we propose a simpler block structure similar to the standard Transformer block [8, 48], wherethe MHA and convolution modules are each directly followed by a single feed forward module(§
We finely examined the micro-architecture of the network and found several modifications thatsimplify the model overall and greatly improve the accuracy and efficiency.
This includes (i)activation unification that replaces GLU activations with Swish (§ 3.2.1), (ii) Layer Normalizationsimplification by replacing redundant pre-Layer Normalization layers with a Scaled Post-LN whichincorporates a learnable scaling for the residual path which can be merged with other layers to be
zero-cost during inference (§ 3.2.2), and (iii) incorporation of a depth-wise separable convolutionfor the first sub-sampling layer that results in a significant FLOPs reduction (§ 3.2.3).
ConvolutionSubsampling
ConformerBlock 2N ×
Decoder
Conformer
SqueezeformerBlock
DW Sep. [Figure] Subsampling
[Figure]
N [Figure]
Decoder +
Squeezeformer
Feed Forward Module
PreLN +
MHA Module
PreLN +
Convolution Module
PreLN +
Feed Forward Module
PreLN +
PostLN +
PostLN
MHA Module +
PostLN
Feed Forward Module +
PostLN
Convolution Module +
Feed Forward Module
PostLN
SqueezeformerBlock
ConformerBlock ConformerBlock
[Figure]
• We show that the Squeezeformer architecture scales well with both smaller and larger models andconsistently outperforms other state-of-the-art ASR models when trained under the same settings (Tab. 4.2, § 4.2).
Furthermore, we justify the final model architecture of Squeezeformer with a
reverse ablation study for the design choices (Tab. A.1, § 4.3).
Related Work
The recent advancements in end-to-end ASR can be broadly categorized into (1) model architectureand (2) training methods.
Model Architecture for End-to-end ASR
The recent end-to-end ASR models are typicallycomposed of an encoder , which takes as input a speech signal (i.e., sequence of speech frames) andextracts high-level acoustic features, and a decoder , which converts the extracted features from theencoder into a sequence of text.
The model architecture of the encoder determines the representationalpower of an ASR model and its ability to extract acoustic features from input signals.
A strongarchitecture is critical for overall performance.
One of the popular choices for a backbone model architecture is CNN.
End-to-end deep CNNmodels have been first explored in Jasper [26], and further improved by introducing depthwiseseparable convolution [19, 44, 47] in QuartzNet [24] and the Squeeze-and-Excitation module [21]in CitriNet [33] and ContextNet [17].
However, since CNN often fails to capture global contexts, Transformer [48] models have also been widely adopted in backbone architectures due to their abilityto capture long-range dependencies between speech frames [22, 28, 29, 53, 54].
Recently, [14]has proposed a novel model architecture named Conformer, which augments Transformers withconvolutions to model both global and local dependencies efficiently.
In this work, with the Conformerarchitecture as our starting point, we focus on designing a next-generation model architecture for ASR that is simpler, more accurate, and more efficient.
The hybrid attention-convolution architecture of Conformer has enabled the state-of-the-art results inmany speech tasks.
However, the quadratic complexity of the attention layer still proves to be costprohibitive at larger sequence lengths.
While different approaches have been proposed to reduce thecost of MHA in ASR [6, 43, 55, 56], their main focus is not changing the overall architecture designand their optimizations can also be applied orthogonal to Squeezeformer.
Efficient-Conformer [5]introduces the progressive downsampling scheme and grouped attention to reduce the training andinference costs of Conformer.
Our work incorporates a similar progressive downsampling, but alsointroduces an up-sampling mechanism with skip connections from the earlier layers inspired by the U-Net [42] architecture in computer vision and U-Time [40] architecture for sleep signal analysis. We
[Table]
find this to be critical for training stability and overall performance.
addition experiments, we refactor the Conformer block by both the macro
[Table]
[Table]
Training Methodology for End-to-end ASR.
In the past few years, various self-supervisedlearning methodologies based on contrastive learning [3, 49, 57] or masked prediction [2, 7, 20]have been proposed to push forward the ASR performance.
While a model pre-trained with a selfsupervised task generally outperforms when finetuned on a target ASR task, training strategies arenot the main focus in this work as they can be applied independently to the underlying architecture.
Architecture Design
The Conformer architecture has been widely adopted by the speech community and is used asa backbone for different speech tasks.
At a macro-level, Conformer incorporates the Macaronstructure [31] comprised of four modules per block, as shown in Fig.
2 (Left).
These blocksare stacked multiple times to construct the Conformer architecture.
In this work, we carefullyreexamine the design choices in Conformer, starting first with its macro-architecture, and thenits micro-architecture design.
We choose Conformer-CTC-M as the baseline model for the casestudy, and we compare word-error-rate (WER) on Librispeech test-other and FLOPs on a 30s audioinput as performance metrics for each architecture.
Furthermore, we focus on FLOPs as a proxyfor model efficiency.
While we acknowledge that FLOPs may not always be a linear indicator ofhardware efficiency/runtime, we choose FLOPs as it is hardware agnostic and is statically computable.
However, we do measure the final latency of our changes, ensuring up to 30% consistent improvementin runtime for different versions of Squeezeformer (Tab. 4.2)
3.1 Macro-Architecture Design
We first focus on designing the macro structure of Squeezeformer, i.e., how the blocks and modulesare organized in a global scale.
3.1.1 Temporal U-Net Architecture
The hybrid attention-convolution structure enables Conformer to capture both global and localinteractions.
However, the attention operation has a quadratic FLOP complexity with respect to theinput sequence length.
We propose to lighten this extra overhead by computing attention over areduced sequence length.
In the Conformer model itself, the input sampling rate is reduced from10ms to 40ms with a convolutional subsampling block at the base of the network.
However, this rateis kept constant throughout the network, with all the attention and convolution operations operatingat a constant temporal scale.
To this end, we begin by studying the temporal redundancy in the learned feature representations.
Inparticular, we analyze how the learned feature embeddings per speech frame are differentiated throughthe Conformer model depth.
We randomly sample 100 audio signals from LibriSpeech’s dev-other dataset, and process them through the Conformer blocks, recording their per-block activations.
Wethen measure the average cosine similarity between two neighboring embedding vectors.
The resultsare plotted as the solid lines in Fig.
We observe that the embeddings for the speech frames directlynext to each other have an average similarity of 95% at the top layer, and even those 4 speech framesaway from each other have a similarity of more than 80%.
This reveals that there is an increasingtemporal redundancy as inputs are processed through the Conformer blocks deeper in the network.
We hypothesize that this redundancy in feature embedding vectors causes unnecessary computationaloverhead and the sequence length can be reduced deeper in the network without loss in accuracy.
[Figure] [Figure]
The preLN can be replaced with the adaptive scaling that readjusts the magnitude of the activation that goes into the subsequent module
As our first macro-architecture improvement step, we change the Conformer model to incorporatesubsampling of the embedding vectors after it has been processed by the early blocks of the model.
In particular, we keep the sample rate to be 40ms up to the 7th block, and afterwards we subsampleto a rate of 80ms per input sequence by using a pooling layer.
For the pooling layer we use adepthwise separable convolution with stride 2 and kernel size 3 to merge the redundancies acrossneighboring embeddings.
This decreases the attention complexity by 4 × and also reduces theredundancies of the features.
This temporal downsampling shares similarities with computer visionmodels which often downsample the input image spatially to save compute and develop hierarchicallevel features [11, 18, 27, 45].
However, the temporal downsampling alone leads to divergence and an unstable training behaviour(§ 4.3).
One possible reason for this is the lack of enough resolution for the decoder after subsamplingthe rate to 80ms.
The decoder maps an embedding for each speech frame into a single label, e.g.,character, and therefore requires sufficient resolution for successful decoding of the full sequence.
Inspired from successful architectures for dense prediction in computer vision such as U-Net [42],we incorporate the Temporal U-Net structure to recover the resolution at the end of the networkthrough an upsampling layer as shown in Fig.
This upsampling block takes the embedding vectorsprocessed by the 40ms and 80ms sampling rates, and produces an embedding with a rate of 40ms byadding them together via a skip connection.
To the best of our knowledge, the closest work to our
Temporal U-Net is the approach proposed in [40], in which the U-Net structure is incorporated to afully-convolutional model to downsample sleep signals.
This change not only reduces the total FLOPs by 20% compared to Conformer 1 , but also improvesthe test-other WER by 0.62% from 7.90% to 7.28%
(Tab. 1, 2nd row).
Furthermore, analyzing thecosine similarity shows that the Temporal U-Net architecture prevents the neighboring embeddings
[Table]
from becoming too similar to each others at the later blocks, in particular at the final block directlyconnected to the decoder, as shown in Fig. 3 as the dashed lines.
3.1.2 Transformer-Style Block
The Conformer block consists of a sequence of feed-forward (‘F’), multi-head attention (MHA, ‘M’),convolution (‘C’), and another feed-forward module (‘F’).
We denote this as the FMCF structure.
Note that the convolutional kernel sizes in ASR models are rather large, e.g., 31 in Conformer,which makes its behaviour similar to attention in mixing global information.
This is stark contrastto convolutional kernels in computer vision, that often have small 3 × 3 kernels and hence benefitgreatly from attention’s global processing.
As such, placing the convolution and MHA module witha similar functionality back-to-back (i.e., the MC substructure) does not seem prudent.
Hence, weconsider an MF/CF structure, which is motivated by considering the convolution module as a local
MHA module.
Furthermore, we drop the Macaron structure [31], as MHA modules followed byfeed-forward modules have been more widely adopted in the literature [8, 10, 41, 48].
In a nutshell,we simplify the architecture to be similar to the standard Transformer network and denote the blocks
MF and CF substructures, as shown in Fig.
This modification further improves the test-other WERimprovement by 0.16% from 7.28% to 7.12% and marginally improves the test-clean WER withoutaffecting the FLOPs (Tab. 1, 3rd row).
3.2 Micro-Architecture Design
So far we have designed the macro structure of Squeezeformer by incorporating seminal architectureprinciples from computer vision and natural language processing into Conformer.
In this subsection,we now focus on optimizing the micro structure of the individual modules.
We show that we canfurther simplify the module architectures while improving both efficiency and performance. [Equation]
1 Unified Activations
Conformer model uses Swish activation for most of the blocks.
However, it switches to a Gated
Linear Unit (GLU) for its convolution module.
Such a heterogeneous design seems over-complicatedand unnecessary.
From a practical standpoint, multiple activations complicates hardware deployment,as an efficient implementation requires dedicated logic design, look up tables, or custom approximations [23, 36, 52].
To address this, we propose to replace the GLU activation with Swish, unifyingthe choice of activation function throughout the entire model.
We keep the expansion rate for theconvolution modules.
As shown in the 4th row of Tab. 1, this change does not entail noticeablechanges in WER and FLOPs but only simplifies the architecture.
3.2.2 Simplified Layer Normalizations
Continuing our micro-improvements, we note that the Conformer model incorporates redundant
Layer Normalizations (LayerNorm), as shown in Fig. 4 (left).
This is because the Conformer modelcontains both a Post-LayerNorm (PostLN) that applies LayerNorm in between the residual blocks, aswell as Pre-LayerNorm (PreLN) which applies LayerNorm inside the residual connection.
While it ishypothesized that preLN stabilizes training and postLN benefits performance [50], these two modulesused together lead to redundant back-to-back operations.
Aside from the architectural redundancy,
LayerNorm can be computationally expensive [23, 52] due to its global reduction operations.
However, we found that straightforwardly removing the preLN or postLN leads to training instabilityand convergence failure (§ 4.3).
Investigating the cause of failure, we observe that a typical trained
Conformer model has orders of magnitude difference in the norm of the learnable scale variables ofthe back-to-back preLN and postLN.
In particular, we found that the preLN would scale down theinput signal by a large value, giving more weight to the skip connection.
Therefore, it is importantto use a scale layer when replacing the PreLN component to allow the network to control thisweight.
This idea is also on par with several training stabilization strategies in other domains.
Forinstance, NF-Net [4] proposed adaptive (i.e., learnable) scaling before and after the residual blocks tostabilize training without normalization.
Furthermore, DeepNet [50] also recently proposed to addnon-trainable rule-based scaling to the skip connections, instead of the residual blocks, to stabilize PreLN in Transformers.
[Table]
Inspired by these computer vision advancements, we propose to replace preLN with learnable scalinglayer that scales and shifts the activations, Scaling( x ) =
αx + β , with learnable scale and biasvectors α and β of the feature dimension.
For homogeneity of architectural design, we then replacethe preLN throughout all the modules with the postLN-then-scaling as illustrated in Fig.
2 (Right) andmake the entire model preLN-only.
Note that the learned scaling p can be merged into the weightsof the subsequent linear layer as the architecture illustrated in Fig.
2 (Right) and hence have zeroinference cost.
With the adaptive scaling our model further improves the test-other WER by 0.20%from 7.09% to 6.89% (Tab. 1, 5th row).
3.2.3 Depthwise Separable Subsampling
We now shift our focus from the Conformer blocks to the subsampling block.
While it is easyto overlook this single module at the beginning of the architecture, we note that it accounts for asignificant portion of the overall FLOPs count, up to 28% for Conformer-CTC-M with a 30-secondinput.
This is because the subsampling layer uses two vanilla convolution operations each of whichhas a stride 2.
To reduce the overhead of this layer, we replace the second convolution operation witha depthwise separable convolution while keeping the kernel size and stride the same.
We leave the firstconvolution operation as is since it is equivalent to a depthwise convolution with the input dimension1.
This saves an additional 22% of the baseline FLOPs without a test-other WER drop and even a 0.06improvement in test-clean WER (Tab. 1, 6th row).
An important point to note here is that generallydepthWise separable convolutions are hard to efficiently map to hardware accelerators, in part dueits low arithmetic intensity.
However, given the large FLOPs reduction, we consistently observe anoverall reduction in the total model latency of about 30% as reported in Tab. 1, as compared to thebaseline Conformer models.
We name our final model with all these improvements as Squeezeformer-SM.
Compared to ConformerCTC-M, our initial baseline, Squeezeformer-SM improves WER by 1.01% from 7.90% to 6.89%with 40% less FLOPs.
Given the smaller FLOPs of Squeezeformer-SM, we also scale up the model toa similar FLOP cost as Conformer-CTC-M.
In particular, we scale both depth and width of the modeltogether following the practice in [9].
Scaling up the model achieves additional test-other WER gainof 0.39% from 6.89% to 6.50% (Tab. 1, 7th row), and we name this architecture Squeezeformer-M.
Results
4.1 Experiment Setup
Models.
Following the procedure described in § 3, we construct Squeezeformer variants with differentsize and FLOPs: we apply the macro and micro-architecture changes in § 3.1 and § 3.2, respectively,to construct Squeezeformer-XS, SM, and ML from Conformer-S, M, and L retaining the model size.
Afterwards, we construct Squeezeformer-S, M, and L by scaling up each model to match the FLOPsof the corresponding Conformer.
The detailed architecture configurations are described in Tab. 2.
[Table]
While there are multiple options available for the decoder such as RNN-Transducer (RNN-T) [12] and
Connectionist Temporal Classification (CTC) [13] [Table] we use a CTC decoder whose non-autoregressivedecoding method benefits training and inference latency [33]. However
[Table] the main focus of this workis the model architecture design of the encoder [Table] which is completely orthogonal to the decoder type.
Another subtlety when evaluating models is the use of external language models (LM).
In many priorwork [16, 22, 32, 37, 51, 53], decoders are often augmented with external LMs such as pre-trained4-gram or Transformer, which boosts the final WER by re-scoring the outputs in a more lexicallyaccurate manner.
However, we compare the results without external LMs to fairly compare the truerepresentation power of the model architectures alone - external LMs can be incorporated as anorthogonal optimization afterwards.
Training Details.
Because the training recipes and codes for Conformer have not been open-sourced,we train it to reproduce the best performance numbers as possible.
We train both Conformer-CTC and
Squeezeformer on the Librispeech-960hr [38] for 500 epochs on Google’s cloud TPUs v3 with batchsize 1024 for the small and medium variants and 2048 for the large variants.
We use AdamW [30]optimizer with weight decay 5e-4 for all models.
More details for the training and evaluation setupare given in § A.2 and § A.3.
4.2 Main Results
In Tab.
3 we compare the WER of Squeezeformer with Conformer-CTC and other state-of-the-art
CTC-based ASR models including QuartzNet [24], CitriNet [33], Transformer [28], and Efficient
Conformer [5] on the clean and other datasets.
For simplicity, we denote WER as test-clean/test-otherwithout % throughout the section.
Squeezeformer vs. Conformer.
Our smallest model Squeezeformer-XS outperforms ConformerCTC-S by 0.32/1.49 (3.74/9.09 vs.
with 1.66 × FLOPs reduction.
Comparedwith Conformer-CTC-M, Squeezeformer-S achieves 0.12/0.43 WER improvement (3.08/7.47 vs.3.20/7.90) with 1.47 × smaller size and 2.73 × less FLOPs, and Squeezeformer-SM further improves
WER by 0.41/1.01 (2.79/6.89 vs. 3.20/7.90) with a comparable size and 1.70 × less FLOPs.
Compared with Conformer-CTC-L, Squeezeformer-M shows 0.24/0.05 WER improvement (2.56/6.50vs. 2.80/6.55) with significant size and FLOPs reductions of 2.18 × and 3.90 × , respectively, and
Squeezeformer-ML shows 0.19/0.50 WER improvement (2.61/6.05 vs. 2.80/6.55) with a similar sizeand 1.66 × less FLOPs.
Finally, our largest model Squeezeformer-L improves WER by 0.33/0.58 upon
Conformer-CTC-L with the same FLOPs count, achieving the state-of-the-art result of 2.47/5.97.
Squeezeformer vs. Other ASR Models.
As can be seen in Tab. 3, our model consistently outperforms
QuartzNet, CitriNet, and Transformer with comparable or smaller model sizes and FLOPs counts.
Anotable result is a comparison against Efficient-Conformer: our model outperforms the efficientlydesigned Efficient Conformer by a large margin of 0.49/1.52 (2.79/3.08 vs. 3.57/8.99) with the same
FLOPs count.
The overall results are summarized as a plot in Fig.
1 (Right) where Squeezeformerconsistently outperforms other models across all FLOPs regimes.
4.3 Ablation Studies
In this section, we provide additional ablation studies for the design choices made for individualarchitecture components using Squeezeformer-M as the base model.
Unless specified, we use thesame hyperparameter settings with the main experiment.
Temporal U-Net.
In the 2nd row of Tab. A.1, the model clearly under performs by 0.35/0.87 withoutthe skip connection from the downsampling layer to the upsampling layer.
This shows that thehigh-resolution information collected in the early layers is critical for successful decoding.
The 3rdrow in Tab.
A.1 shows that our model completely fails to converge without the upsampling layer dueto training stability, even with several different peak learning rates of {0.5, 1.0, 1.5}e-3.
LayerNorm.
In the 4th line of Tab. A.1, we show that WER drops significantly by 3.17/7.49 when weapply the PostLN-only scheme without the learned scaling layer.
Another alternative design choiceis to apply the PreLN-only scheme without the learned scaling, which also results in a noticeable
WER degradation of 0.59/1.76 as shown in the 5th line of Tab. A.1.
In both cases, the model fails toconverge, so we report the best WER before divergence.
The results suggest that the learned scalinglayer plays a key role for training stabilization and better WER.
Convolution Module.
When ablating the GLU activation in the convolution modules, anotherpossible design choice is to drop it without replacing with the Swish activation.
This, however, resultsin 0.10/0.22 worse WER as shown in the last line of Tab. A.1.
Conclusions
In this work, we performed a series of systematic ablation studies on the macro and micro architectureof the Conformer architecture and proposed a novel hybrid attention-convolution architecture thatis simpler, and consistently achieves better performance than other models for a wide range ofcomputational budgets.
The key novel components of Squeezeformer’s macro-architecture is theincorporation of the Temporal U-Net structure which downsamples audio signals in the second halfof the network to reduce the temporal redundancy between adjacent features and save compute, aswell reorder the block order to MFCF be more similar to the standard Transformer-style MF/CFblock structure which simplifies the architecture and improves performance.
Furthermore, the microarchitecture of Squeezeformer simplifies the activations throughout the model and removes redundant
LayerNorms with the Scaled Post-LN, which is more efficient and leads to better accuracy.
We alsodrastically reduce the subsampling cost at the beginning of the model by incorporating a depth-wiseseparable convolution.
We perform extensive testing of the proposed architecture and find that
Squeezeformer scales very well across different model sizes and FLOP regimes, surpassing priormodel architectures when trained under the same settings.
Our code along with the checkpoints forall of the trained models is open sourced and available online [1].