EfﬁcientFormer: Vision Transformers at MobileNet Speed
Abstract
Vision Transformers (ViT) have shown rapid progress in computer vision tasks, achieving promising results on various benchmarks.
However, due to the massive number of parameters and model design, e.g. , attention mechanism, ViT-based models are generally times slower than lightweight convolutional networks.
Therefore, the deployment of ViT for real-time applications is particularly challenging, especially on resource-constrained hardware such as mobile devices.
Recent efforts try to reduce the computation complexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still unsatisfactory.
This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance?
To answer this, we ﬁrst revisit the network architecture and operators used in ViT-based models and identify inefﬁcient designs.
Then we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm.
Finally, we perform latencydriven slimming to get a series of ﬁnal models dubbed EfﬁcientFormer.
Extensive experiments show the superiority of EfﬁcientFormer in performance and speed on mobile devices.
Our fastest model, EfﬁcientFormer-L1, achieves 79 .
2% top-1 accuracy on ImageNet-1K with only 1 .
6 ms inference latency on iPhone 12 (compiled with CoreML), which runs as fast as MobileNetV2 × 1 .
( 1 . 6 ms, 74 .
7% top-1), and our largest model, EfﬁcientFormer-L7, obtains 83 .
3% accuracy with only 7 .
ms latency.
Our work proves that properly designed transformers can reach extremely low latency on mobile devices while maintaining high performance 1 .
Introduction
The transformer architecture [1], initially designed for Natural Language Processing (NLP) tasks, introduces the Multi-Head Self Attention (MHSA) mechanism that allows the network to model long-term dependencies and is easy to parallelize.
In this context, Dosovitskiy et al.
[2] adapt the attention mechanism to 2D images and propose Vision Transformer (ViT): the input image is divided into non-overlapping patches, and the inter-patch representations are learned through
MHSA without inductive bias.
ViTs demonstrate promising results compared to convolutional neural networks (CNNs) on computer vision tasks.
Following this success, several efforts explore the potential of ViT by improving training strategies [3, 4, 5], introducing architecture changes [6, 7], redesigning attention mechanisms [8, 9], and elevating the performance of various vision tasks such as classiﬁcation [10, 11, 12], segmentation [13, 14], and detection [15, 16].
On the downside, transformer models are usually times slower than competitive CNNs [17, 18].
There are many factors that limit the inference speed of ViT, including the massive number of
Resnet
MobileNetV
MobileNetV2×1.
LeViTMobileViT-XS
Deit-Tiny Deit-Small
PoolFormer-S PoolFormer-S
PoolFormer-S
ResMLP-S
ConvmixerConvmixer-
EfficientNet-B
EfficientNet-B
EfficientFormer-L EfficientFormer-L
EfficientFormer-L
I m age N e tK T op - 1 ( % )
Latency (ms) parameters, quadratic-increasing computation complexity with respect to token length, non-foldable normalization layers, and lack of compiler level optimizations ( e.g. , Winograd for CNN [19]).
The high latency makes transformers impractical for real-world applications on resource-constrained hardware, such as augmented or virtual reality applications on mobile devices and wearables.
As a result, lightweight CNNs [20, 21, 22] remain the default choice for real-time inference.
To alleviate the latency bottleneck of transformers, many approaches have been proposed.
For instance, some efforts consider designing new architectures or operations by changing the linear layers with convolutional layers (CONV) [23], combining self-attention with MobileNet blocks [24], or introducing sparse attention [25, 26, 27], to reduce the computational cost, while other efforts leverage network searching algorithm [
28] or pruning [
] to
improve efﬁciency
Although
the computation-performance trade-off has been improved by existing works, the fundamental question that relates to the applicability of transformer models remains unanswered: Can powerful vision transformers run at MobileNet speed and become a default option for edge applications?
This work provides a study towards the answer through the following contributions:
• First, we revisit the design principles of ViT and its variants through latency analysis (Sec. 3).
Following existing work [18], we utilize iPhone 12 as the testbed and publicly available CoreML [30] as the compiler, since the mobile device is widely used and the results can be easily reproduced.
• Second, based on our analysis, we identify inefﬁcient designs and operators in ViT and propose a new dimension-consistent design paradigm for vision transformers (Sec. 4.1).
• Third, starting from a supernet with the new design paradigm, we propose a simple yet effective latency-driven slimming method to obtain a new family of models, namely, EfﬁcientFormers (Sec. 4.2).
We directly optimize for inference speed instead of MACs or number of parameters [31, 32, 33].
Our fastest model, EfﬁcientFormer-L1, achieves 79 .
2% top-1 accuracy on ImageNet-1K [34] classiﬁcation task with only 1 . 6 ms inference time (averaged over 1 , 000 runs), which runs as fast as
MobileNetV2 × 1 . 4 and wields 4 .
5% higher top-1 accuracy (more results in Fig.
1 and Tab. 1).
The promising results demonstrate that latency is no longer an obstacle for the widespread adoption of vision transformers.
Our largest model, EfﬁcientFormer-L7, achieves 83 .
3% accuracy with only 7 .
ms latency, outperforms ViT ×
MobileNet hybrid designs (MobileViT-XS, 74 .
8% , 7 . 2 ms) by a large margin.
Additionally, we observe superior performance by employing EfﬁcientFormer as the backbone in image detection and segmentation benchmarks (Tab. 2).
We provide a preliminary answer to the aforementioned question, ViTs can achieve ultra fast inference speed and wield powerful performance at the same time.
We hope our EfﬁcientFormer can serve as a strong baseline and inspire followup works on the edge deployment of vision transformers.
Related Work
Transformers are initially proposed to handle the learning of long sequences in NLP tasks [1].
Dosovitskiy et al.
[2] and Carion et al.
[15] adapt the transformer architecture to classiﬁcation and detection, respectively, and achieve competitive performance against CNN counterparts with stronger training techniques and larger-scale datasets.
DeiT [3] further improves the training pipeline with the aid of distillation, eliminating the need for large-scale pretraining [35].
Inspired by the competitive performance and global receptive ﬁeld of transformer models, follow-up works are proposed to reﬁne the architecture [36, 37], explore the relationship between CONV nets and ViT [38, 39, 40], and adapt ViT to different computer vision tasks [13, 41, 42, 43, 44, 45, 46].
Other research efforts explore the essence of attention mechanism and propose insightful variants of token mixer, e.g. , local attention [8], spatial MLP [47, 48], and pooling-mixer [6].
Despite the success in most vision tasks, ViT-based models cannot compete with the well-studied lightweight CNNs [21, 49] when the inference speed is the major concern [50, 51, 52], especially on resource-constrained edge devices [17].
To accelerate ViT, many approaches have been introduced with different methodologies, such as proposing new architectures or modules [53, 54, 55, 56, 57, 58], re-thinking self-attention and sparse-attention mechanisms [59, 60, 61, 62, 63, 64, 65], and utilizing search algorithms that are widely explored in CNNs to ﬁnd smaller and faster ViTs [66, 28, 29, 67].
Recently, LeViT [23] proposes a CONV-clothing design to accelerate vision transformer.
However, in order to perform MHSA, the 4 D features need to be frequently reshaped into ﬂat patches, which is still expensive to compute on edge resources (Fig. 2).
Likewise, MobileViT [18] introduces a hybrid architecture that combines lightweight MobileNet blocks (with point-wise and depth-wise CONV) and MHSA blocks; the former is placed at early stages in the network pipeline to extract low-level features, while the latter is placed in late stages to enjoy the global receptive ﬁeld.
Similar approach has been explored by several works [24, 28] as a straightforward strategy to reduce computation.
Different from existing works, we aim at pushing the latency-performance boundary of pure vision transformers instead of relying on hybrid designs, and directly optimize for mobile latency.
Through our detailed analysis (Sec. 3), we propose a new design paradigm (Sec. 4.1), which can be further elevated through architecture search (Sec. 4.2).
On-Device Latency Analysis of Vision Transformers
Most existing approaches optimize the inference speed of transformers through computation complexity (MACs) or throughput (images/sec) obtained from server GPU [23, 28].
While such metrics do not reﬂect the real on-device latency.
To have a clear understanding of which operations and design choices slow down the inference of ViTs on edge devices, we perform a comprehensive latency analysis over a number of models and operations, as shown in Fig.
2, whereby the following observations are drawn.
Observation 1 : Patch embedding with large kernel and stride is a speed bottleneck on mobile devices.
Patch embedding is often implemented with a non-overlapping convolution layer that has large kernel size and stride [3, 55].
A common belief is that the computation cost of the patch embedding layer in a transformer network is unremarkable or negligible [2, 6].
However, our comparison in Fig.
2 between models with large kernel and stride for patch embedding, i.e. , DeiT-S [3] and PoolFormer-S24 [6], and the models without it, i.e. , LeViT-256 [23] and EfﬁcientFormer, shows that patch embedding is instead a speed bottleneck on mobile devices.
Large-kernel convolutions are not well supported by most compilers and cannot be accelerated through existing algorithms like Winograd [19].
Alternatively, the non-overlapping patch embedding can be replaced by a convolution stem with fast downsampling [68, 69, 23] that consists of several hardware-efﬁcient 3 × 3 convolutions (Fig. 3).
EfficientFormer-L EfficientFormer-L
PoolFormer-S
LeViT-256†
DeiT-Small
MobileViT-XS
ResMLP-S
ResNet
EfficientNet-B
MobileNetV2×1.
Latency (ms)
Patch Embedding Attention Linear (Proj.
+ MLP)
Reshape Norm Activation Others Total (74.7%) (77.1%) (78.5%) (79.4%) (74.8%) (81.2%) (81.6%) (80.3%) (82.4%) (79.2%)
†LeViT uses HardSwish which is not well supported by CoreML, we replace it with GeLU for
Observation 2 : Consistent feature dimension is important for the choice of token mixer.
MHSA is not necessarily a speed bottleneck.
Recent work extends ViT-based models to the MetaFormer architecture [6] consisting of MLP blocks and unspeciﬁed token mixers.
Selecting a token mixer is an essential design choice when building
ViT-based models.
The options are many—the conventional MHSA mixer with a global receptive ﬁeld, more sophisticated shifted window attention [8], or a non-parametric operator like pooling [6].
We narrow the comparison to the two token mixers, pooling and MHSA, where we choose the former for its simplicity and efﬁciency, while the latter for better performance.
More complicated token mixers like shifted window [8] are currently not supported by most public mobile compilers and we leave them outside our scope.
Furthermore, we do not use depth-wise convolution to replace pooling [70] as we focus on building architecture without the aid of lightweight convolutions.
To understand the latency of the two token mixers, we perform the following two comparisons:
• First, by comparing PoolFormer-s24 [6] and LeViT-256 [23], we observe that the Reshape operation is a bottleneck for LeViT-256.
The majority of LeViT-256 is implemented with CONV on 4 D tensor, requiring frequent reshaping operations when forwarding features into MHSA since the attention has to be performed on patchiﬁed
D tensor (discarding the extra dimension of attention heads).
The extensive usage of Reshape limits the speed of LeViT on mobile devices (Fig. 2).
On the other hand, pooling naturally suits the 4 D tensor when the network primarily consists of CONV-based implementations, e.g. , CONV 1 × 1 as MLP implementation and CONV stem for downsampling.
As a result, PoolFormer exhibits faster inference speed.
• Second, by comparing DeiT-Small [3] and LeViT-256 [23], we ﬁnd that MHSA does not bring signiﬁcant overhead on mobiles if the feature dimensions are consistent and Reshape is not required.
Though much more computation intensive, DeiT-Small with a consistent 3D feature can achieve comparable speed to the new ViT variant, i.e. , LeViT-256.
In this work, we propose a dimension-consistent network (Sec. 4.1) with both 4 D feature implementation and 3 D MHSA but the inefﬁcient frequent Reshape operations are eliminated.
Observation 3 : CONV-BN is more latency-favorable than LN (GN)-Linear and the accuracy drawback is generally acceptable.
4D 3D + +
[Figure]
+
Choosing the MLP implementation is another essential design choice.
Usually, one of the two options is selected: layer normalization (LN) with 3D linear projection (proj.) and CONV 1 × 1 with batch normalization (BN).
CONV-BN is more latency favorable because BN can be folded into the preceding convolution for inference speedup, while dynamic normalizations, such as LN and GN, still collects running statistics at the inference phase, thus contributing to latency.
From the analysis of DeiT-Small and PoolFormer-S24 in Fig.
2 and previous work [17], the latency introduced by LN constitutes around 10% − 20% latency of the whole network.
Based on our ablation study in Appendix Tab. 3, CONV-BN only slightly downgrades performance compared to GN and achieves comparable results to channel-wise LN.
In this work, we apply CONVBN as much as possible (in all latent 4 D features) for the latency gain with a negligible performance drop, while using LN for the 3 D features, which aligns with the original MHSA design in ViT and yields better accuracy.
Observation 4 : The latency of nonlinearity is hardware and compiler dependent.
Lastly, we study nonlinearity, including GeLU, ReLU, and HardSwish.
Previous work [17] suggests
GeLU is not efﬁcient on hardware and slows down inference.
However, we observe GeLU is well supported by iPhone 12 and hardly slower than its counterpart, ReLU.
On the contrary, HardSwish is surprisingly slow in our experiments and may not be well supported by the compiler (LeViTlatency with HardSwish is 44 .
5 ms while with GeLU 11 .
9 ms).
We conclude that nonlinearity should be determined on a case-by-case basis given speciﬁc hardware and compiler at hand.
We believe that most of the activations will be supported in the future.
In this work, we employ GeLU activations.
Design of EfﬁcientFormer
Based on the latency analysis, we propose the design of EfﬁcientFormer, demonstrated in Fig.
The network consists of a patch embedding ( PatchEmbed ) and stack of meta transformer blocks, denoted as MB :
[Equation]
where X 0 is the input image with batch size as B and spatial size as [ H,W ] , Y is the desired output, and m is the total number of blocks (depth).
MB consists of unspeciﬁed token mixer ( TokenMixer ) followed by a MLP block and can be expressed as follows:
[Equation]
where X i ∣ i > 0 is the intermediate feature that forwarded into the i th MB .
We further deﬁne Stage (or S ) as the stack of several MetaBlocks that processes the features with the same spatial size, such as N 1 × in Fig.
3 denoting S 1 has N 1 MetaBlocks.
The network includes 4 Stages.
Among each Stage, there is an embedding operation to project embedding dimension and downsample token length, denoted as Embedding in Fig.
With the above architecture, EfﬁcientFormer is a fully transformer-based model without integrating MobileNet structures.
Next, we dive into the details of the network design, speciﬁcally, the architecture details and the search algorithm.
4.1 Dimension-Consistent Design
With the observations in Sec. 3, we propose a dimension consistent design which splits the network into a 4D partition where operators are implemented in CONV-net style ( MB 4 D ), and a 3D partition where linear projections and attentions are performed over 3D tensor to enjoy the global modeling power of MHSA without sacriﬁcing efﬁciency ( MB 3 D ), as shown in Fig.
Speciﬁcally, the network starts with 4D partition, while 3D partition is applied in the last stages.
Note that Fig.
3 is just an instance, the actual length of 4D and 3D partition is speciﬁed later through architecture search.
First, input images are processed by a CONV stem with two 3 × 3 convolutions with stride 2 as patch embedding,
[Equation]
where C j is the channel number (width) of the j th stage.
Then the network starts with MB 4 D with a simple Pool mixer to extract low level features,
[Equation]
where Conv B,G refers to whether the convolution is followed by BN and GeLU, respectively.
Note here we do not employ Group or Layer Normalization ( LN ) before the Pool mixer as in [6], since the 4D partition is CONV-BN based design, thus there exists a BN in front of each Pool mixer.
After processing all the MB 4 D blocks, we perform a one-time reshaping to transform the features size and enter 3D partition.
MB 3 D follows conventional ViT structure, as in Fig.
Formally,
[Equation]
where Linear G denotes the Linear followed by GeLU, and
[Equation]
where Q,K,V represents query, key, and values learned by the linear projection, and b is parameterized attention bias as position encodings.
4.2 Latency Driven Slimming
Design of Supernet.
Based on the dimension-consistent design, we build a supernet for searching efﬁcient models of the network architecture shown in Fig.
3 (Fig.
3 shows an example of searched ﬁnal network).
In order to represent such a supernet, we deﬁne the MetaPath ( MP ), which is the collection of possible blocks:
[Equation]
[Equation]
where I represents identity path, j denotes the j th Stage, and i denotes the i th block.
The supernet can be illustrated by replacing MB in Fig.
3 with MP .
As in Eqn. 7, in S 1 and S 2 of the supernet, each block can select from MB 4 D
or I , while in S 3 and
S 4 , the block can be MB 3 D , MB 4 D , or I .
We only enable MB 3 D in the last two Stages for two reasons.
First, since the computation of MHSA grows quadratically with respect to token length, integrating it in early Stages would largely increase the computation cost.
Second, applying the global MHSA to the last Stages aligns with the intuition that early stages in the networks capture low-level features, while late layers learn long-term dependencies.
Searching Space.
Our searching space includes C j (the width of each Stage), N j (the number of blocks in each Stage, i.e. , depth), and last N blocks to apply MB 3 D .
Searching Algorithm.
Previous hardware-aware network searching methods generally rely on hardware deployment of each candidate in search space to obtain the latency, which is time consuming [71].
In this work, we propose a simple, fast yet effective gradient-based search algorithm to obtain a candidate network that just needs to train the supernet for once.
The algorithm has three major steps.
First, we train the supernet with Gumbel Softmax sampling [72] to get the importance score for the blocks within each MP , which can be expressed as
[Equation]
where α evaluates the importance of each block in MP as it represents the probability to select a block, e.g. , MB 4 D or MB 3 D for the i th block.
(cid:15) ∼ U ( 0 , 1 ) ensures exploration, τ is the temperature, and n represents the type of blocks in MP , i.e. , n ∈ { 4 D,I } for S 1 and S 2 , and n ∈ { 4 D, 3 D,I } for S 3 and
S 4 .
By using Eqn. 8, the derivatives with respect to network weights and α can be computed easily.
The training follows the standard recipe (see Sec. 5.1) to obtain the trained weights and architecture parameter α .
Second, we build a latency lookup table by collecting the on-device latency of MB 4 D and MB 3 D with different widths (multiples of 16 ).
Finally, we perform network slimming on the supernet obtained from the ﬁrst step through latency evaluation using the lookup table.
Note that a typical gradient-based searching algorithm simply select the block with largest α [72], which does not ﬁt our scope as it cannot search the width C j .
In fact, constructing a multiple-width supernet is memory-consuming and even unrealistic given that each MP has several branches in our design.
Instead of directly searching on the complex searching space, we perform a gradual slimming on the single-width supernet as follows.
We ﬁrst deﬁne the importance score for MP i as α 4 Di α Ii and α 3 Di + α 4 Di α Ii for S 1 , 2 and S 3 , 4 , respectively.
Similarly, the importance score for each Stage can be obtained by summing up the scores for all MP within the Stage.
With the importance score, we deﬁne the action space that includes three options: 1) select I for the least important MP , 2) remove the ﬁrst MB 3 D , and 3) reduce the width of the least important Stage (by multiples of 16 ).
Then, we calculate the resulting latency of each action through lookup table, and evaluate the accuracy drop of each action.
Lastly, we choose the action based on per-latency accuracy drop ( − % ms ) .
This process is performed iteratively until target latency is achieved.
We show more details of the algorithm in Appendix.
Experiments and Discussion
We implement EfﬁcientFormer through PyTorch 1.11 [73] and Timm library [74], which is the common practice in recent arts [18, 6].
Our models are trained on a cluster with NVIDIA A100 and
V100 GPUs.
The inference speed on iPhone 12 (A14 bionic chip) is measured with iOS version 15 and averaged over 1 , 000 runs, with all available computing resources (NPU), or CPU only.
CoreMLTools is used to deploy the run-time model.
In addition, we provide latency analysis on
Nvidia A100 GPU with batch size 64 to exploit hardware rooﬂine.
The trained PyTorch models are deployed in ONNX format and are compiled with TensorRT.
We report GPU runtime that excludes preprocessing.
We provide the detailed network architecture and more ablation studies in Appendix 6.
5.1 Image Classiﬁcation
All EfﬁcientFormer models are trained from scratch on ImageNet-1K dataset [34] to perform the image classiﬁcation task.
We employ standard image size ( 224 × 224 ) for both training and testing.
We follow the training recipe from DeiT [3] but mainly report results with 300 training epochs to have the comparison with other ViT-based models.
We use AdamW optimizer [75, 76], warm-up training with 5 epochs, and a cosine annealing learning rate schedule.
The initial learning rate is set as 10 − 3 × ( batch size / 1024 ) and the minimum learning rate is 10 − 5 .
The teacher model for distillation
[Table]
is RegNetY-16GF [77] pretrained on ImageNet with 82 .
9% top-1 accuracy.
Results are demonstrated in Tab. 1 and Fig.
Comparison to CNNs.
Compared with the widely used CNN-based models, EfﬁcientFormer achieves a better trade-off between accuracy and latency.
On iPhone Neural Engine, EfﬁcientFormerL1 runs at MobileNetV2 × 1 .
4 speed while achieving 4 .
5% higher top-1 accuracy.
In addition,
EfﬁcientFormer-L3 runs at a similar speed to EfﬁcientNet-B0 while achieving relative 5 .
3% higher top-1 accuracy.
For the models with high performance ( > 83% top-1), EfﬁcientFormer-L7 runs more than 3 × faster than EfﬁcientNet-B5, demonstrating the advantageous performance of our models.
Moreover on desktop GPU (A100), EfﬁcientFormer-L1 runs 38% faster than EfﬁcientNet-B0 while achieving 2.1% higher top-1 accuracy.
EfﬁcientFormer-L7 runs 4.6 × faster than EfﬁcientNet-B5.
These results allow us to answer the central question raised earlier; ViTs do not need to sacriﬁce latency to achieve good performance, and an accurate ViT can still have ultra-fast inference speed as lightweight CNNs do.
Comparison to ViTs.
Conventional ViTs are still under-performing CNNs in terms of latency.
For instance, DeiT-Tiny achieves similar accuracy to EfﬁcientNet-B0 while it runs 3 .
4 × slower.
However,
EfﬁcientFormer performs like other transformer models while running times faster.
EfﬁcientFormerL3 achieves higher accuracy than DeiT-Small ( 82 . 4% vs. 81 .
2% ) while being 4 × faster.
It is notable that though the recent transformer variant, PoolFormer [6], naturally has a consistent 4 D architecture and runs faster compared to typical ViTs, the absence of global MHSA greatly limits the performance upper-bound.
EfﬁcientFormer-L3 achieves 1% higher top-1 accuracy than PoolFormer-S36, while being 3 × faster on Nvidia A100 GPU, 2.2 × faster on iPhone NPU and 6.8 × faster on iPhone CPU.
Comparison to Hybrid Designs.
Existing hybrid designs, e.g. , LeViT-256 and MobileViT, still struggle with the latency bottleneck of ViTs and can hardly outperform lightweight CNNs.
For example, LeViT-256 runs slower than DeiT-Small while having 1% lower top-1 accuracy.
For
MobileViT, which is a hybrid model with both MHSA and MobileNet blocks, we observe that it
[Table]
is signiﬁcantly slower than CNN counterparts, e.g. , MobileNetV2 and EfﬁcientNet-B0, while the accuracy is not satisfactory either ( 2 . 3% lower than EfﬁcientNet-B0).
Thus, simply trading-off
MHSA with MobileNet blocks can hardly push forward the Pareto curve, as in Fig.
In contrast,
EfﬁcientFormer, as pure transformer-based model
[Table]
can maintain high performance while achieving ultra-fast inference speed.
EfﬁcientFormer-L1 has 4.4% higher top-1 accuracy than MobileViT-XS and runs much faster across different hardware and compilers (1.9 × faster on Nvidia A100 GPU
Computing, 2.3 × faster on iPhone CPU, and 4.5 × faster on iPhone NPU).
At a similar inference time,
EfﬁcientFormer-L7 outperforms MobileViT-XS by 8 .
5% top-1 accuracy on ImageNet, demonstrating the superiority of our design.
5.2 EfﬁcientFormer as Backbone
Object Detection and Instance Segmentation.
We follow the implementation of Mask-RCNN [78] to integrate EfﬁcientFormer as the backbone and verify performance.
We experiment over COCO2017 [79] which contains training and validations sets of 118 K and 5 K images, respectively.
The
EfﬁcientFormer backbone is initialized with ImageNet-1K pretrained weights.
Similar to prior work [6], we use AdamW optimizer [75, 76] with initial learning rate of 2 × 10 − 4 , and train the model for 12 epochs.
We set the input size as 1333 × 800 .
The results for detection and instance segmentation are shown in Tab. 2.
EfﬁcientFormers consistently outperform CNN (ResNet) and transformer (PoolFormer) backbones.
With similar computation cost,
EfﬁcientFormer-L3 outperforms ResNet50 backbone by 3 . 4 box AP and 3 .
7 mask AP , and outperforms PoolFormer-S24 backbone with 1 . 3 box AP and 1 . 1 mask AP , proving that EfﬁcientFormer generalizes well as a strong backbone in vision tasks.
Semantic Segmentation.
We further validate the performance of EfﬁcientFormer on the semantic segmentation task.
We use the challenging scene parsing dataset, ADE20K [80, 81], which contains 20 K training images and 2 K validation ones covering 150 class categories.
Similar to existing work [6], we build EfﬁcientFormer as backbone along with Semantic FPN [82] as segmentation decoder for fair comparison.
The backbone is initialized with pretrained weights on ImageNet-1K and the model is trained for 40 K iterations with a total batch size of 32 over 8 GPUs.
We follow the common practice in segmentation [6, 13], use AdamW optimizer [75, 76], and apply a poly learning rate schedule with power 0 .
9 , starting from a initial learning rate 2 ×
10 − 4 .
We resize and crop input images to 512 × 512 for training and shorter side as 512 for testing (on validation set).
As shown in Tab. 2, EfﬁcientFormer consistently outperforms CNN- and transformer-based backbones by a large margin under a similar computation budget.
For example, EfﬁcientFormer-L3 outperforms
PoolFormer-S24 by 3 .
2 mIoU.
We show that with global attention, EfﬁcientFormer learns better long-term dependencies, which is beneﬁcial in high-resolution dense prediction tasks.
5.3 Discussion
Relations to MetaFormer.
The design of EfﬁcientFormer is partly inspired by the MetaFormer concept [6].
Compared to PoolFormer, EfﬁcientFormer addresses the dimension mismatch problem, which is a root cause of inefﬁcient edge inference, thus being capable of utilizing global MHSA without sacriﬁcing speed.
Consequently, EfﬁcientFormer exhibits advantageous accuracy performance over PoolFormer.
In spite of its fully 4 D design, PoolFormer employs inefﬁcient patch embedding and group normalization (Fig. 2), leading to increased latency.
Instead, our redesigned 4 D partition of EfﬁcientFormer (Fig. 3) is more hardware friendly and exhibits better performance across several tasks.
Limitations.
(i)
Though most designs in EfﬁcientFormer are general-purposed, e.g. , dimensionconsistent design and 4 D block with CONV-BN fusion, the actual speed of EfﬁcientFormer may vary on other platforms.
For instance, if GeLU is not well supported while HardSwish is efﬁciently implemented on speciﬁc hardware and compiler, the operator may need to be modiﬁed accordingly.
(ii)
The proposed latency-driven slimming is simple and fast.
However, better results may be achieved if search cost is not a concern and an enumeration-based brute search is performed.
Conclusion
In this work, we show that Vision Transformer can operate at MobileNet speed on mobile devices.
Starting from a comprehensive latency analysis, we identify inefﬁcient operators in a series of ViTbased architectures, whereby we draw important observations that guide our new design paradigm.
The proposed EfﬁcientFormer complies with a dimension consistent design that smoothly leverages hardware-friendly 4 D MetaBlocks and powerful 3 D MHSA blocks.
We further propose a fast latencydriven slimming method to derive optimized conﬁgurations based on our design space.
Extensive experiments on image classiﬁcation, object detection, and segmentation tasks show that EfﬁcientFormer models outperform existing transformer models while being faster than most competitive
CNNs.
The latency-driven analysis of ViT architecture and the experimental results validate our claim: powerful vision transformers can achieve ultra-fast inference speed on the edge.
Future research will further explore the potential of EfﬁcientFormer on several resource-constrained devices
. Acknowledgment
This work is supported in part by National Science Foundation CCF-
Appendix A Latency-Driven Slimming Algorithm
We provide the details of the proposed latency-driven fast slimming in Alg. 1.
Formulations of the algorithm can be found in Sec.
The proposed latency-driven slimming is speed-oriented, which does not require retraining for each sub-network.
The importance score for each design choice is estimated based on the trainable architecture parameter α .
Algorithm
Require: Latency lookup table T =
{ MB 4 D { dim = 16 ×} , MB 3 D { dim = 16 ×}}
Ensure: Final latency satisfy budget ∑ T ≈ T
Super-net Pretraining : for epoch do for each iter do for MP i,j do
X i +
[Equation]
=
∑ n
[Equation]
( αni
[Equation]
(cid:15)ni )/
[Equation]
∑ n e ( αn i
[Equation]
(cid:15)n i )/
[Equation]
⋅ MP i,
[Equation]
(X i ) , end for
L ← criterion (Y ,label ) backpropagate ( L ), update parameters end for end for ▷ get super-net
Latency-driven slimming:
Initialize action space
A ∈ {Depth Reduction (DR), Width Reduction (WR), MB 3 D Reduction (MR)}
Compute importance of MP i,j by I i,j = α 4 Di α Ii ,or α 3 Di + α 4 Di α Ii while ∑ T > T do
DR ← argmin I
i,j ( MP i,j ) ,
WR ← argmin ∑ j I i,j ( MP i,j ) ,
MR ← First − MB 3 D ,
Execute Action = argmin accuracydrop
Ti,j ( A ) end while ▷ get sub-net with target latency
Train the searched architecture from scratch :
Similar to super-net training.
▷ get ﬁnal model
B Ablation Analysis
Our major conclusions and speed analysis can be found in Sec. 3 and Fig.
Here we include more ablation studies for different design choices, provided in Tab. 3, taking the EfﬁcientFormer-L3 as an example.
The latency is measured on iPhone 12 with CoreML, and the top-1 accuracy is obtained from the ImageNet-1K dataset.
Patch Embedding.
Compared to non-overlap large-kernel patch embedding (V5 in Tab. 3), the proposed convolution stem in EfﬁcientFormer (V1 in Tab. 3) greatly reduces inference latency by 48% , while provides 0 .
7% higher accuracy.
We demonstrate that convolution stem [23] is not only beneﬁcial to model convergence and accuracy but also boosts inference speed on the mobile device by a large margin, thus can serve as a good alternative to non-overlapping patch embedding implementations.
MHSA and Latency-Driven Search.
Without the proposed 3 D MHSA and latency-driven search,
EfﬁcientFormer downgrades to a pure 4 D design with pool mixer, which is similar to PoolFormer [6] (the patch embeddings and normalizations are different).
By comparing EfﬁcientFormer with V in Tab. 3, we can observe that the integration of 3 D MHSA and latency-driven search greatly boost top-1 accuracy by 2 .
1% with minimal impact on the inference speed ( 0 .
5 ms).
The results prove that
MHSA with the global receptive ﬁeld is an essential contribution to model performance.
As a result, though enjoying faster inference speed, simply removing MHSA [6] greatly limits the performance upper bound.
In EfﬁcientFormer, we smoothly integrate MHSA in a dimension consistent manner, obtaining better performance while simultaneously achieving ultra fast inference speed.
Normalization.
Apart from the CONV-BN structure in the 4 D partition of EfﬁcientFormer, we explore Group Normalization (GN) and channel-wise Layer Normalization (LN) in the 4 D partition as employed in the prior work [6].
By comparing V1 and V2 in Tab. 3, we can observe that the GN (V2-GN) can only slightly improve accuracy ( 0 .
3% top-1)
but incurs latency overhead as it can not be folded at the inference stage.
Similarly, applying LN (V2-LN) gets higher latency than BN while the performance improvement is negligible.
As a result, we apply the CONV-BN structure in the entire 4 D partition in EfﬁcientFormer.
Activation Functions.
We explore ReLU and HardSwish (V3 and V4 in Tab. 3) in addition to GeLU employed in this work (V1 in Tab. 3).
It is widely agreed that ReLU is the simplest and fastest activation function, while GeLU and HardSwish wield better performance.
We observe that ReLU can hardly provide any speedup over GeLU on iPhone 12 with CoreMLTools, while HardSwish is signiﬁcantly slower than ReLU and GeLU.
We draw a conclusion that the activation function can be selected on a case-by-case basis depending on the speciﬁc hardware and compiler.
In this work, we use GeLU to provide better performance than ReLU while executing faster.
For a fair comparison, we modify inefﬁcient operators in other works according to the supports from iPhoneand CoreMLTools, e.g. , report LeViT latency by changing HardSwish to GeLU.
Dimension Consistent Design.
We perform the latency analysis on the Nvidia A100 GPU to show that the proposed dimension-consistent (D-C) design is beneﬁcial besides the iPhone.
We deploy the
PyTorch models with batch size 64 as ONNX format and use TensorRT to compile and benchmark the latency.
We report the latency results averaged over 1 , 000 runs in Tab. 4.
For the non-D-C design, we revert the proposed Meta3D block into 4D implementation, where linear projections and MLPs are all implemented with CONV1 × 1-BN instead of 3D-Linear layers, and reshaping operations become necessary in order to perform multi-head self-attention.
With this conﬁguration, attention blocks can be arbitrarily placed along with Meta4D blocks without following dimension-consistent design, while frequent reshaping is introduced.
We conduct the comparison on the following two models, both with a D-C version and a non-D-C one with the exact same computation complexity:
• EfﬁcientFormer-L7, which has 8 attention blocks.
• DummyNet, a handcrafted dummy model with a total of 16 attention blocks.
As can be seen from Tab. 4, the proposed dimension-consistent design achieves faster inference speed than the non-dimension-consistent design for both EfﬁcientFormer-L7 and the DummyNet.
Latency Driven Slimming.
Besides the hardware-efﬁcient architecture design, it is still crucial to ﬁnd appropriate depth and width conﬁgurations for the model to achieve satisfactory performance.
To understand the beneﬁts of our latency driven slimming, we randomly sample networks from our search space that have the same computation, i.e. , 1.3 GMACs, as our searched model EfﬁcientFormer-L1.
The sampled networks are denoted as Random 1 to Random 5, which are either deeper and narrower, or shallower and wider than EfﬁcientFormer-L1.
We train the sampled models on ImageNet-1K with the same training recipe as EfﬁcientFormer-L1.
The comparison between these models is shown in
Tab. 5.
As can be seen, our searched EfﬁcientFormer-L1 has better latency or higher top-1 accuracy on ImageNet-1K than the randomly sampled networks, proving the advantages of our proposed latency driven slimming.
C Analysis of Hardware Utilization
EfﬁcientFormer improves the latency vs. accuracy trade-off through better architecture design so that higher hardware utilization is achieved.
To understand hardware utilization, we employ throughput in
TFLOPS (Tera FLOPs per Second) as the evaluation metric, which is calculated by model computation cost (FLOPs) divided by execution time.
Models with higher throughput (TFLOPS) better exploit the computation power of the hardware.
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
To fairly compare with baseline models under different computation complexity, we L inearly S cale the depth and width of EfﬁcientFormer-L1 to obtain a series of models (EfﬁcientFormer-LS-1 to
EfﬁcientFormer-LS-14), with the number of parameters ranging from 1.1M to 31.3M and MACs from 0.09G to 3.9G, and benchmark the latency and utilization on iPhone 12 NPU.
As in Fig.
4, super-tiny models still run at about 1ms, such as EfﬁcientFormer-LS-{1, 2, 3}, where the throughput is low and the hardware is not fully exploited.
Data processing and transferring become the bottleneck.
As a result, making the model super small with sacriﬁced accuracy is less valuable.
In contrast, our 1.3GMACs model, EfﬁcientFormer-L1 lies at a sweet point, enjoying fast inference speed (1.6ms) while maintaining high accuracy.
Furthermore, we can observe that EfﬁcientFormer variants outperform both CNNs and ViTs in hardware utilization across different computation complexity levels.
For instance, at 4-GMACs level,
EfﬁcientFormer-LS-14 outperforms DeiT-S by 3.3 × higher TFLOPS and outperforms PoolFormer by 2.2 × , achieving comparable throughput to ResNet50.
In the lightweight domain, EfﬁcientFormerLS-4 has 2.2 × higher TFLOPS than EfﬁcientNet-B0.
We demonstrate that with the proposed hardware-friendly design, EfﬁcientFormer naturally has better hardware utilization.
D Architecture of EfﬁcientFormers
The detailed network archiecture for EfﬁcientFormer-L1, EfﬁcientFormer-L3, and EfﬁcientFormer-L is provided in Tab. 6.
We report the resolution and number of blocks for each stage.
In addition, the width of EfﬁcientFormer is speciﬁed as the embedding dimension (Embed.
Dim.).
As for the
MHSA block, the dimension of Query and Key is provided, and we employ eight heads for all
EfﬁcientFormer variants.
MLP expansion ratio is set as default (4), as in most ViT arts [3].