Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Abstract
The recently proposed Conformer model has become the de facto backbone modelfor various downstream speech tasks based on its hybrid attention-convolutionarchitecture that captures both local and global features.
However, through a seriesof systematic studies, we find that the Conformer architecture’s design choicesare not optimal.
After reexamining the design choices for both the macro andmicro-architecture of Conformer, we propose Squeezeformer which consistentlyoutperforms the state-of-the-art ASR models under the same training schemes.
Inparticular, for the macro-architecture, Squeezeformer incorporates (i) the Temporal
U-Net structure which reduces the cost of the multi-head attention modules on longsequences, and (ii) a simpler block structure of feed-forward module followed upby multi-head attention or convolution modules instead of the Macaron structureproposed in Conformer.
Furthermore, for the micro-architecture, Squeezeformer(i) simplifies the activations in the convolutional block, (ii) removes redundant
Layer Normalization operations, and (iii) incorporates an efficient depthwise downsampling layer to efficiently sub-sample the input signal.
Squeezeformer achievesstate-of-the-art results of 7.5%, 6.5%, and 6.0% word-error-rate (WER) on LibriSpeech test-other without external language models, which are 3.1%, 1.4%, and0.6% better than Conformer-CTC with the same number of FLOPs.
Our code isopen sourced and available online [1].
Introduction
The increasing success of end-to-end neural network models has been a huge driving force for thedrastic advancements in various automatic speech recognition (ASR) tasks.
While both convolutionalneural networks (CNN) [20, 27, 29, 36, 61] and Transformers [25, 31, 32, 56, 57] have drawn attentionas popular backbone architectures for ASR models, each of them has several limitations.
Generally, CNN models lack the ability to capture global contexts and Transformers involve prohibitive computing and memory overhead.
To overcome these shortcomings, Conformer [17] has recently proposeda novel convolution-augmented Transformer architecture.
Due to its ability to synchronously captureglobal and local features from audio signals, Conformer has become the de facto model not only for ASR tasks, but also for various end-to-end speech processing tasks [18].
Furthermore, it has alsoachieved the state-of-the-art performance in combination with recent developments in self-supervisedlearning methodologies as well [37, 62].
While the Conformer architecture was introduced as a
RNN-Transducer (RNN-T) [15] model in its original setting, it has been adopted with less critique toencoder-only schemes such as Connectionist Temporal Classification (CTC) [16] as well [39].
Despite being a key architecture in speech processing tasks, the Conformer architecture has somelimitations that can be improved upon.
First, Conformer still suffers from the quadratic complexity of
[Figure]
CTC
[Figure]
M 71.7 57.0 57.0 58.4 58.4 42.
B a s e li n e + T e m p o r a l U - N e t + T X - S t
y l e B l o c k
+ U n
i f i e d A c t i v a t i o n s + S
i m p
li f i e d L a y e r N o r m + D W
S e p .
S u b s a m p li n g + M o d e l
S c a l e - u p 7.
GFLOPs
Squeezeformer-M the attention mechanism limiting its efficiency on long sequence lengths. This problem is furtherhighlighted by the long sequence lengths of typical audio inputs as also pointed out in [46].
Furthermore, the Conformer architecture is relatively more complicated than Transformer architectures usedin other domains such as in natural language processing [8, 44, 51] or computer vision [10, 11, 49].
For instance, the Conformer architecture incorporates multiple different normalization schemes andactivation functions, the Macaron structure [34], as well as back-to-back multi-head attention (MHA)and convolution modules.
This level of complexity makes it difficult to efficiently deploy the modelon dedicated hardware platforms for inference [26, 38, 55].
More importantly, this raises the questionof whether such design choices are necessary and optimal for achieving good performance in ASRtasks.
In this paper, we perform a careful and systematic analysis of each of the design choices with thegoal of achieving lower word-error-rate (WER) for a given computational budget.
We developeda much simpler and more efficient hybrid attention-convolution architecture in both its macro andmicro-design that consistently outperforms the state-of-the-art ASR models.
In particular, we makethe following contributions in our proposed Squeezeformer model:
• We find a high temporal redundancy in the learned feature representations of neighboring speechframes especially deeper in the network which results in unnecessary computational overhead.
Toaddress this, we incorporate the temporal U-Net structure in which a downsampling layer halvesthe sampling rate at the middle of the network, and a light upsampling layer recovers the temporalresolution at the end for training stability (§ 3.1.1).• We redesign the hybrid attention-convolution architecture based on our observation that the backto-back MHA and convolution modules with the Macaron structure is suboptimal.
In particular,we propose a simpler block structure similar to the standard Transformer block [8, 51], wherethe MHA and convolution modules are each directly followed by a single feed forward module(§ 3.1.2).•
We finely examined the micro-architecture of the network and found several modifications thatsimplify the model overall and greatly improve the accuracy and efficiency.
This includes (i)activation unification that replaces GLU activations with Swish (§ 3.2.1), (ii) Layer Normalizationsimplification by replacing redundant pre-Layer Normalization layers with a Scaled post-LayerNormalization which incorporates
ConvolutionSubsampling
ConformerBlock 2N ×
Decoder
Conformer
SqueezeformerBlock
DW Sep. Conv Subsampling N -
[Figure]
N ×
Decoder +
Squeezeformer
Feed Forward Module
PreLN +
MHA Module
PreLN +
Convolution Module
PreLN +
Feed Forward Module
PreLN +
PostLN +
PostLN
MHA Module +
PostLN
Feed Forward Module +
PostLN
Convolution Module +
Feed Forward Module
PostLN
SqueezeformerBlock
ConformerBlock ConformerBlock
SqueezeformerBlock
[Figure]
separable convolution for the first sub-sampling layer that results in a significant floating point
operations (FLOPs) reduction (§ 3.2.3). • We show that the Squeezeformer architecture scales well with both smaller and larger models and consistently outperforms other state-of-the-art ASR models when trained under the same settings(Tab.
Furthermore, we justify the final model architecture of Squeezeformer with a
reverse ablation study for the design choices (Tab. A.1, § 4.3).
Related Work
The recent advancements in end-to-end ASR can be broadly categorized into (1) model architectureand (2) training methods.
Model Architecture for End-to-end ASR
The recent end-to-end ASR models are typicallycomposed of an encoder , which takes as input a speech signal (i.e., sequence of speech frames) andextracts high-level acoustic features, and a decoder , which converts the extracted features from theencoder into a sequence of text.
The model architecture of the encoder determines the representationalpower of an ASR model and its ability to extract acoustic features from input signals.
A strongarchitecture is critical for overall performance.
One of the popular choices for a backbone model architecture is CNN.
End-to-end deep CNN modelshave been first explored in [29, 61], and further improved by introducing depth-wise separable convolution [22, 47, 50] in QuartzNet [27] and the Squeeze-and-Excitation module [24] in CitriNet [36]and ContextNet [20].
However, since CNN often fails to capture global contexts, Transformer [51]models have also been widely adopted in backbone architectures due to their ability to capturelong-range dependencies between speech frames [25, 31, 32, 56, 57].
Recently, [17] has proposeda novel model architecture named Conformer, which augments Transformers with convolutions tomodel both global and local dependencies efficiently.
In this work, with the Conformer architectureas our starting point, we focus on designing a next-generation model architecture for ASR that issimpler, more accurate, and more efficient.
The hybrid attention-convolution architecture of Conformer has enabled the state-of-the-art results inmany speech tasks.
However, the quadratic complexity of the attention layer still proves to be costprohibitive at larger sequence lengths.
While different approaches have been proposed to reduce thecost of MHA in ASR [6, 46, 58, 59], their main focus is not changing the overall architecture designand their optimizations can also be applied orthogonal to Squeezeformer.
Efficient-Conformer [5]introduces the progressive downsampling scheme and grouped attention to reduce the training andinference costs of Conformer.
Our work incorporates a similar progressive downsampling, but also
[Table]
Training Methodology for End-to-end ASR.
In the past few years, various self-supervisedlearning methodologies based on contrastive learning [3, 52, 62] or masked prediction [2, 7, 23]have been proposed to push forward the ASR performance.
While a model pre-trained with a selfsupervised task generally outperforms when finetuned on a target ASR task, training strategies arenot the main focus in this work as they can be applied independently to the underlying architecture.
Architecture Design
The Conformer architecture has been widely adopted by the speech community and is used asa backbone for different speech tasks.
At a macro-level, Conformer incorporates the Macaronstructure [34] comprised of four modules per block, as shown in Fig.
2 (Left).
These blocksare stacked multiple times to construct the Conformer architecture.
In this work, we carefullyreexamine the design choices in Conformer, starting first with its macro-architecture, and thenits micro-architecture design.
We choose Conformer-CTC-M as the baseline model for the casestudy, and we compare word-error-rate (WER) on LibriSpeech test-other and FLOPs on a 30s audioinput as performance metrics for each architecture.
Furthermore, we focus on FLOPs as a proxyfor model efficiency.
While we acknowledge that FLOPs may not always be a linear indicator ofhardware efficiency/runtime, we choose FLOPs as it is hardware agnostic and is statically computable.
However, we do measure the final latency of our changes, ensuring up to 30% consistent improvementin runtime for different versions of Squeezeformer (Tab. 4.2)
3.1 Macro-Architecture Design
We first focus on designing the macro structure of Squeezeformer, i.e., how the blocks and modulesare organized in a global scale.
3.1.1 Temporal U-Net Architecture
The hybrid attention-convolution structure enables Conformer to capture both global and localinteractions.
However, the attention operation has a quadratic FLOP complexity with respect to theinput sequence length.
We propose to lighten this extra overhead by computing attention over areduced sequence length.
In the Conformer model itself, the input sampling rate is reduced from10ms to 40ms with a convolutional subsampling block at the base of the network.
However, this rateis kept constant throughout the network, with all the attention and convolution operations operatingat a constant temporal scale.
To this end, we begin by studying the temporal redundancy in the learned feature representations.
Inparticular, we analyze how the learned feature embeddings per speech frame are differentiated through the Conformer model depth.
We randomly sample 100 audio signals from LibriSpeech’s dev-otherdataset, and process them through the Conformer blocks, recording their per-block activations.
Wethen measure the average cosine similarity between two neighboring embedding vectors.
The resultsare plotted as the solid lines in Fig.
We observe that the embeddings for the speech frames directlynext to each other have an average similarity of 95% at the top layer, and even those 4 speech framesaway from each other have a similarity of more than 80%.
This reveals that there is an increasingtemporal redundancy as inputs are processed through the Conformer blocks deeper in the network.
We hypothesize that this redundancy in feature embedding vectors causes unnecessary computationaloverhead and the sequence length can be reduced deeper in the network without loss in accuracy.
[Figure] [Figure]
The preLN can be replaced with the learned scaling that readjusts the magnitude of the activation that goes into the subsequent module.
As our first macro-architecture improvement step, we change the Conformer model to incorporatesubsampling of the embedding vectors after it has been processed by the early blocks of the model.
In particular, we keep the sample rate to be 40ms up to the 7th block, and afterwards we subsampleto a rate of 80ms per input sequence by using a pooling layer.
For the pooling layer we use adepthwise separable convolution with stride 2 and kernel size 3 to merge the redundancies acrossneighboring embeddings.
This decreases the attention complexity by 4 × and also reduces theredundancies of the features.
This temporal downsampling shares similarities with computer visionmodels which often downsample the input image spatially to save compute and develop hierarchicallevel features [11, 21, 30, 48] and with the approach of Efficient Conformer [5].
However, the temporal downsampling alone leads to divergence and an unstable training behaviour(§ 4.3).
One possible reason for this is the lack of enough resolution for the decoder after subsamplingthe rate to 80ms.
The decoder maps an embedding for each speech frame into a single label, e.g.,character, and therefore requires sufficient resolution for successful decoding of the full sequence.
Inspired from successful architectures for dense prediction in computer vision such as U-Net [45],we incorporate the Temporal U-Net structure to recover the resolution at the end of the networkthrough an upsampling layer as shown in Fig.
This upsampling block takes the embedding vectorsprocessed by the 40ms and 80ms sampling rates, and produces an embedding with a rate of 40ms byadding them together via a skip connection.
To the best of our knowledge, the closest work to our
Temporal U-Net is the approach proposed in [43], in which the U-Net structure is incorporated to afully-convolutional model to downsample sleep signals.
This change not only reduces the total FLOPs by 20% compared to Conformer 1 , but also improvesthe test-other WER by 0.62% from 7.90% to 7.28%
(Tab. 1, 2nd row).
Furthermore, analyzing thecosine similarity shows that the Temporal U-Net architecture prevents the neighboring embeddings
The total FLOPs is for the entire model. If we just study the attention block, the Temporal U-Net structure reduces the FLOPs by 2.31 × and 2.53 × FLOPs reduction for processing 30s and 60s audio signals as compared to Conformer-CTC-M baseline, respectively.
from becoming too similar to each others at the later blocks, in particular at the final block directlyconnected to the decoder, as shown in Fig. 3 as the dashed lines.
3.1.2 Transformer-Style Block
The Conformer block consists of a sequence of feed-forward (‘F’), multi-head attention (MHA, ‘M’),convolution (‘C’), and another feed-forward module (‘F’).
We denote this as the FMCF structure.
Note that the convolutional kernel sizes in ASR models are rather large, e.g., 31 in Conformer,which makes its behaviour similar to attention in mixing global information.
This is stark contrastto convolutional kernels in computer vision, that often have small 3 × 3 kernels and hence benefitgreatly from attention’s global processing.
As such, placing the convolution and MHA module witha similar functionality back-to-back (i.e., the MC substructure) does not seem prudent.
Hence, weconsider an MF/CF structure, which is motivated by considering the convolution module as a local
MHA module.
Furthermore, we drop the Macaron structure [34], as MHA modules followed byfeed-forward modules have been more widely adopted in the literature [8, 10, 44, 51].
In a nutshell,we simplify the architecture to be similar to the standard Transformer network and denote the blocks
MF and CF substructures, as shown in Fig.
This modification further improves the test-other WERimprovement by 0.16% from 7.28% to 7.12% and marginally improves the test-clean WER withoutaffecting the FLOPs (Tab. 1, 3rd row).
3.2 Micro-Architecture Design
So far we have designed the macro structure of Squeezeformer by incorporating seminal architectureprinciples from computer vision and natural language processing into Conformer.
In this subsection,we now focus on optimizing the micro structure of the individual modules.
We show that we canfurther simplify the module architectures while improving both efficiency and performance.3.
1 Unified Activations
Conformer uses Swish activation for most of the blocks.
However, it switches to a Gated Linear
Unit (GLU) for its convolution module.
Such a heterogeneous design seems over-complicated andunnecessary.
From a practical standpoint, multiple activations complicates hardware deployment,as an efficient implementation requires dedicated logic design, look up tables, or custom approximations [26, 38, 55].
For instance, on low-end edge devices with no dedicated vector processingunits, supporting additional non-linear operations would require additional look up tables or advancedalgorithms [13, 14].
To address this, we propose to replace the GLU activation with Swish, unifyingthe choice of activation function throughout the entire model.
We keep the expansion rate for theconvolution modules.
As shown in the 4th row of Tab. 1, this change does not entail noticeablechanges in WER and FLOPs but only simplifies the architecture.
3.2.2 Simplified Layer Normalizations
Continuing our micro-improvements, we note that the Conformer model incorporates redundant
Layer Normalizations (LayerNorm), as shown in Fig. 4 (left).
This is because the Conformer modelcontains both a post-LayerNorm (postLN) that applies LayerNorm in between the residual blocks, aswell as pre-LayerNorm (preLN) which applies LayerNorm inside the residual connection.
While it ishypothesized that preLN stabilizes training and postLN benefits performance [53], these two modulesused together lead to redundant back-to-back operations.
Aside from the architectural redundancy,
LayerNorm can be computationally expensive [26, 55] due to its global reduction operations.
However, we found that straightforwardly removing the preLN or postLN leads to training instabilityand convergence failure (§ 4.3).
Investigating the cause of failure, we observe that a typical trained
Conformer model has orders of magnitude difference in the norm of the learnable scale variables ofthe back-to-back preLN and postLN.
In particular, we found that the preLN would scale down theinput signal by a large value, giving more weight to the skip connection.
Therefore, it is importantto use a scale layer when replacing the preLN component to allow the network to control thisweight.
This idea is also on par with several training stabilization strategies in other domains.
Forinstance, NF-Net [4] proposed adaptive (i.e., learnable) scaling before and after the residual blocks tostabilize training without normalization.
Furthermore, DeepNet [53] also recently proposed to add
[Table]
non-trainable rule-based scaling to the skip connections, instead of the residual blocks, to stabilizepreLN in Transformers.
Inspired by these computer vision advancements, we propose to replace preLN with learnable scalinglayer that scales and shifts the activations, Scaling( x ) =
αx + β , with learnable scale and biasvectors α and β of the feature dimension.
For homogeneity of architectural design, we then replacethe preLN throughout all the modules with the postLN-then-scaling as illustrated in Fig.
2 (Right)and make the entire model postLN-only.
Note that the learned scaling parameters can be merged intothe weights of the subsequent linear layer as the architecture illustrated in Fig.
2 (Right) and hencehave zero inference cost.
With the learned scaling our model further improves the test-other WER by0.20% from 7.09% to 6.89% (Tab. 1, 5th row).
3.2.3 Depthwise Separable Subsampling
We now shift our focus from the Conformer blocks to the subsampling block.
While it is easyto overlook this single module at the beginning of the architecture, we note that it accounts for asignificant portion of the overall FLOPs count, up to 28% for Conformer-CTC-M with a 30-secondinput.
This is because the subsampling layer uses two vanilla convolution operations each of whichhas a stride 2.
To reduce the overhead of this layer, we replace the second convolution operationwith a depthwise separable convolution while keeping the kernel size and stride the same.
We leavethe first convolution operation as is since it is equivalent to a depthwise convolution with the inputdimension 1.
This saves an additional 22% of the baseline FLOPs without a test-other WER dropand even a 0.06% improvement in test-clean WER (Tab. 1, 6th row).
An important point to notehere is that generally depthwise separable convolutions are hard to efficiently map to hardwareaccelerators, in part due its low arithmetic intensity.
However, given the large FLOPs reduction, weconsistently observe an overall reduction in the total model latency of about 30% as reported in Tab.
1,as compared to the baseline Conformer models.
We name our final model with all these improvements as Squeezeformer-SM.
Compared to ConformerCTC-M, our initial baseline, Squeezeformer-SM improves WER by 1.01% from 7.90% to 6.89%with 40% less FLOPs.
Given the smaller FLOPs of Squeezeformer-SM, we also scale up the model toa similar FLOP cost as Conformer-CTC-M.
In particular, we scale both depth and width of the modeltogether following the practice in [9].
Scaling up the model achieves additional test-other WER gainof 0.39% from 6.89% to 6.50% (Tab. 1, 7th row), and we name this architecture Squeezeformer-M.
Results
4.1 Experiment Setup
Models.
Following the procedure described in § 3, we construct Squeezeformer variants with differentsize and FLOPs: we apply the macro and micro-architecture changes in § 3.1 and § 3.2, respectively,to construct Squeezeformer-XS, SM, and ML from Conformer-S, M, and L retaining the model size.
Afterwards, we construct Squeezeformer-S, M, and L by scaling up each model to match the FLOPsof the corresponding Conformer.
The detailed architecture configurations are described in Tab. 2.
[Table] While there are multiple options available for the decoder such as RNN-Transducer (RNN-T) [15] and
Connectionist Temporal Classification (CTC) [16], we use a CTC decoder whose non-autoregressivedecoding method benefits training and inference latency [36].
However, the main focus of this workis the model architecture design of the encoder, which can be orthogonal to the decoder type.
Another subtlety when evaluating models is the use of external language models (LM).
In many priorworks [19, 25, 35, 40, 54, 56], decoders are often augmented with external LMs such as pre-trained4-gram or Transformer, which boosts the final WER by re-scoring the outputs in a more lexicallyaccurate manner.
However, we compare the results without external LMs to fairly compare the truerepresentation power of the model architectures alone − external LMs can be incorporated as anorthogonal optimization afterward.
Training Details.
Because the training recipes and codes for Conformer have not been open-sourced,we train it to reproduce the best performance numbers as possible.
We train both Conformer-CTC and
Squeezeformer on the LibriSpeech-960hr [41] for 500 epochs on Google’s cloud TPUs v3 with batchsize 1024 for the small and medium variants and 2048 for the large variants.
We use AdamW [33]optimizer with weight decay 5e-4 for all models.
More details for the training and evaluation setupare given in § A.2 and § A.3.
4.2 Main Results
In Tab.
3 we compare the WER of Squeezeformer with Conformer-CTC and other state-of-the-art
CTC-based ASR models including QuartzNet [27], CitriNet [36], Transformer [31], and Efficient
Conformer [5] on the clean and other datasets.
Please note that the numbers for Conformer-CTC 2are based on our own reproduction to the best performance as possible due to the absence ofpublic training recipes or codes.
For simplicity, we denote WER as test-clean/test-other without %throughout the section.
Squeezeformer vs. Conformer.
Our smallest model Squeezeformer-XS outperforms ConformerCTC-S by 0.32/1.49 (3.74/9.09 vs.
with 1.66 × FLOPs reduction.
Comparedwith Conformer-CTC-M, Squeezeformer-S achieves 0.12/0.43 WER improvement (3.08/7.47 vs.3.20/7.90) with 1.47 × smaller size and 2.73 × less FLOPs, and Squeezeformer-SM further improves
WER by 0.41/1.01 (2.79/6.89 vs. 3.20/7.90) with a comparable size and 1.70 × less FLOPs.
Compared with Conformer-CTC-L, Squeezeformer-M shows 0.24/0.05 WER improvement (2.56/6.50vs. 2.80/6.55) with significant size and FLOPs reductions of 2.18 × and 3.90 × , respectively, and
Squeezeformer-ML shows 0.19/0.50 WER improvement (2.61/6.05 vs. 2.80/6.55) with a similar sizeand 1.66 × less FLOPs.
Finally, our largest model Squeezeformer-L improves WER by 0.33/0.58 upon
Conformer-CTC-L with the same FLOPs count, achieving the state-of-the-art result of 2.47/5.97.
Squeezeformer vs. Other ASR Models.
As can be seen in Tab. 3, our model consistently outperforms
QuartzNet, CitriNet, and Transformer with comparable or smaller model sizes and FLOPs counts.
Anotable result is a comparison against Efficient-Conformer: our model outperforms the efficientlydesigned Efficient Conformer by a large margin of 0.49/1.52 (2.79/3.08 vs. 3.57/8.99) with the same
FLOPs count.
The overall results are summarized as a plot in Fig.
1 (Right) where Squeezeformerconsistently outperforms other models across all FLOPs regimes.
4.3 Ablation Studies
In this section, we provide additional ablation studies for the design choices made for individualarchitecture components using Squeezeformer-M as the base model.
Unless specified, we use thesame hyperparameter settings as in the main experiment.
Temporal U-Net.
In the 2nd row of Tab. A.1, the model clearly under performs by 0.35/0.87 withoutthe skip connection from the downsampling layer to the upsampling layer.
This shows that thehigh-resolution information collected in the early layers is critical for successful decoding.
The 3rdrow in Tab.
A.1 shows that our model completely fails to converge without the upsampling layer dueto training stability, even with several different peak learning rates of {0.5, 1.0, 1.5}e-3.
LayerNorm.
In the 4th line of Tab. A.1, we show that WER drops significantly by 3.17/7.49 when weapply the PostLN-only scheme without the learned scaling layer.
Another alternative design choiceis to apply the PreLN-only scheme without the learned scaling, which also results in a noticeable
WER degradation of 0.59/1.76 as shown in the 5th line of Tab. A.1.
In both cases, the model fails toconverge, so we report the best WER before divergence.
The results suggest that the learned scalinglayer plays a key role for training stabilization and better WER.
Convolution Module.
When ablating the GLU activation in the convolution modules, anotherpossible design choice is to drop it without replacing it with the Swish activation.
This, however,results in 0.10/0.22 worse WER as shown in the last line of Tab. A.1.
Conclusions
In this work, we performed a series of systematic ablation studies on the macro and micro architectureof the Conformer architecture and proposed a novel hybrid attention-convolution architecture thatis simpler, and consistently achieves better performance than other models for a wide range ofcomputational budgets.
The key novel components of Squeezeformer’s macro-architecture is theincorporation of the Temporal U-Net structure which downsamples audio signals in the second halfof the network to reduce the temporal redundancy between adjacent features and save compute, aswell reorder the block order to MFCF be more similar to the standard Transformer-style MF/CFblock structure which simplifies the architecture and improves performance.
Furthermore, the microarchitecture of Squeezeformer simplifies the activations throughout the model and removes redundant
LayerNorms with the Scaled postLN, which is more efficient and leads to better accuracy.
We alsodrastically reduce the subsampling cost at the beginning of the model by incorporating a depthwiseseparable convolution.
We perform extensive testing of the proposed architecture and find that
Squeezeformer scales very well across different model sizes and FLOP regimes, surpassing priormodel architectures when trained under the same settings.
Our code along with the checkpoints forall of the trained models is open sourced and available online [1].