Fast Vision Transformers with HiLo Attention
Anonymous Author(s)
AffiliationAddress email
Abstract
Vision Transformers (ViTs) have triggered the most recent and significant breakthroughs in computer vision.
Their efficient designs are mostly guided by theindirect metric of computational complexity, i .
e ., FLOPs, which however has aclear gap with the direct metric such as throughput.
Thus, we propose to use thedirect speed evaluation on the target platform as the design principle for efficient
ViTs.
Particularly, we introduce LITv2, a simple and effective ViT which performsfavourably against the existing state-of-the-art methods across a spectrum of different model sizes with faster speed.
At the core of LITv2 is a novel self-attentionmechanism, which we dub HiLo .
HiLo is inspired by the insight that high frequencies in an image capture local fine details and low frequencies focus on globalstructures, whereas a multi-head self-attention layer neglects the characteristic ofdifferent frequencies.
Therefore, we propose to disentangle the high/low frequencypatterns in an attention layer by separating the heads into two groups, where onegroup encodes high frequencies via self-attention within each local window, andanother group performs the attention to model the global relationship between theaverage-pooled low-frequency keys from each window and each query position inthe input feature map.
Benefit from the efficient design for both groups, we showthat HiLo is superior to the existing attention mechanisms by comprehensivelybenchmarking on FLOPs, speed and memory consumption on GPUs.
Powered by
HiLo, LITv2 serves as a strong backbone for mainstream vision tasks includingimage classification, dense detection and segmentation.
Introduction
Real-world applications usually require a model to have an optimal speed and accuracy trade-offunder limited computational budget, such as UAV and autonomous driving.
This motivates substantialworks toward efficient vision Transformer (ViT) design, such as PVT [45], Swin [29] and Focal
Transformer [53], among others.
To measure the computational complexity, a widely adopted metricin recent ViT design is the number of float-point operations, i .
e ., FLOPs.
However, FLOPs is anindirect metric, which can not directly reflect the real speed on the target platform.
For example,
Focal-Tiny is much slower than Swin-Ti on GPUs although their FLOPs are comparable.
In general, the discrepancy between the indirect metric (FLOPs) and the direct metric (speed) inrecent ViTs can be attributed to two main reasons.
First, although self-attention is efficient onlow-resolution feature maps, the quadratic complexity in both memory and time makes it muchslower on high-resolution images due to intensive memory access cost [31], where fetching data fromoff-chip DRAM can be speed-consuming.
Second, some efficient attention mechanisms in ViTs havelow theoretical complexity guarantee but are actually slow on GPUs due to particular operations
that
[Figure]
C on c a t
P r o j e c t i on
QKV
[Figure]
Dot
[Figure]
Attention
C on c a t
P r o j e c t i on
High Frequency Attention (Hi-Fi) are not hardware-friendly or cannot be parallelized, such as the multi-scale window partition [53],recursion [40] and dilated window [17].
With these observations, in this paper we propose to evaluate ViT by the direct metric, i .
e ., throughput,not only FLOPs.
Based on this principle, we introduce LITv2, a novel efficient and accurate vision
Transformer that outperforms most state-of-the-art (SoTA) ViTs on standard benchmarks while beingpractically faster on GPUs.
LITv2 is bulit upon LITv1 [33], a simple ViT baseline which removesall multi-head self-attention layers (MSAs) in the early stages while applying standard MSAs inthe later stages.
Benefit from this design, LITv1 is faster than many existing works on ImageNetclassification due to no computational cost from the early MSAs while the later MSAs only need toprocess downsampled low-resolution feature maps.
However, the standard MSA still suffers fromhuge computational cost on high-resolution images, especially for dense prediction tasks.
To address this problem, we propose a novel efficient attention mechanism, termed HiLo .
HiLo ismotivated by the fact that natural images contain rich frequencies where high/low frequencies playdifferent roles in encoding image patterns, i .
e ., local fine details and global structures, respectively.
Atypical MSA layer enforces the same global attention across all image patches without consideringthe characteristics of different underlying frequencies.
This motivates us to propose to separate an
MSA layer into two paths where one path encodes high-frequency interactions via local self-attentionwith relatively high-resolution feature maps while the other path encodes low-frequency interactionsvia global attention with down-sampled feature maps, which leads to a great efficiency improvement.
Specifically, HiLo employs two efficient attentions to disentangle Hi gh/ Lo w frequencies in featuremaps.
As shown in Figure 1, in the upper path, we allocate a few heads to the high frequency attention(Hi-Fi) to capture fine-grained high frequencies by local window self-attention ( e .
g ., 2 × 2 windows),which is much more efficient than standard MSAs.
The lower path, implementing the low-frequencyattention (Lo-Fi), first applies average pooling to each window to obtain low-frequency signals.
Then,we allocate the remaining heads for Lo-Fi to model the relationship between each query positionin the input feature map and the average-pooled low-frequency keys from each window.
Benefitfrom the reduced length of keys and values, Lo-Fi also achieves significant complexity reduction.
Finally, we concatenate the refined high/low-frequency features and forward the resulting output intosubsequent layers.
Since both Hi-Fi and Lo-Fi are not equipped with time-consuming operationssuch as dilated windows and recursion, the overall framework of HiLo is fast on GPUs.
We show bycomprehensive benchmarks that HiLo achieves advantage over the existing attention mechanisms interms of performance, FLOPs, throughput and memory consumption.
Besides, we find the fixed relative positional encoding in LITv1 dramatically slows down its speed ondense prediction tasks due to the interpolation for different image resolutions.
For better efficiency,we propose to adopt one 3 × 3 depthwise convolutional layer with zero-padding in each FFN toincorporate
the implicitly learned position information from zero-padding [24].
Moreover, the3 × 3 convolutional filters simultaneously help to enlarge the receptive field of the early multi-layerperceptron (MLP) blocks in LITv1.
Finally, we conduct extensive experiments on ImageNet, COCOand ADE20K to evaluate the performance of LITv2.
Comprehensive comparisons with SoTA modelsshow that our architecture achieves competitive performance with
Related Work
Vision Transformers.
Vision Transformers are neural networks that adopt self-attention mechanismsinto computer vision tasks.
In [15], Dosovitskiy et al .
propose a ViT for image classification, whichinherits the similar architecture from a standard Transformer [42] in natural language processing(NLP) tasks.
Since then, subsequent works have been proposed to improve ViT by incorporating moreconvolutional layers [48, 54], introducing pyramid feature maps [45, 29], enhancing the locality [55],as well as automatically searching a well-performed architecture [4, 2] with neural architecturesearch (NAS).
Some others also seek for token pruning to accelerate the inference speed of ViTs [34].
Compared to existing works, this paper focuses on a general ViT-based backbone for computer vision(CV) tasks and aims to achieve better efficiency on GPUs while maintaining competitive performance.
Efficient attention mechanisms.
Efficient attention mechanisms aim to reduce the quadratic complexity of standard MSAs.
Existing efforts in NLP can be roughly categories into low-rank decomposition [44], kernelization [25, 36], memory [37] and sparsity mechanism [9].
However, simplyadopting these method usually performs suboptimally in CV tasks [29, 56].
In CV, representativeefficient self-attention mechanisms includes spatial reduction attention (SRA)
[45], local windowattention [29, 23] and Twins attention [11].
However, they only focus on either local or globalattention at the same layer.
To address this problem, TNT [18] introduced additional global tokensand MixFormer [5] mixed local window attention with depthwise convolutional layers.
Some otherattention mechanisms consider both simultaneously, such as Focal [53] and QuadTree [40].
However,due to the inefficient operations which are not hardware-friendly and cannot be reflected in FLOPs( e . g ., multi-scale window partition, recursion), they are slow on GPUs even compared to standard
MSA. To this end, the proposed HiLo attention simultaneously captures rich local-global informationat the same MSA layer and is faster and more memory-efficient compared to the existing works.
Frequency domain analysis in vision.
The frequency domain analysis in CV has been well studied inthe literature.
According to [12, 14], the low frequencies in an image usually capture global structuresand color information while the high frequencies contain fine details of objects ( e .
g ., sharp edges).
Based on this insight, a plethora of solutions have been proposed for image super-resolution [58, 16],generalization [22], image re-scaling [49] and neural network compression [52, 6].
Furthermore,
Octave convolution [8] targeted convolutional layers and proposed to locally applies convolution onhigh/low-resolution feature maps, separately. Different from it, the proposed HiLo is a novel attentionmechanism that captures both local and global relationships with self-attention.
Background
Multi-head self-attention.
Transformers are built upon multi-head self-attention, which enables tocapture long-range relationships for tokens at different positions.
Specifically, let X ∈ R N × D be theinput sequence into a standard MSA layer, where N is the length of the input sequence and D refersto the number of hidden dimensions.
Each self-attention head calculates the query Q , key K andvalue V matrices with a linear transformation from X ,
[Equation]
where W q , W k , W v ∈ R D
× D h are learnable parameters and D h is the number of hidden dimensionsfor a head.
Next, the output of a self-attention head is a weighted sum over N value vectors,
[Equation]
For an MSA layer with N h heads, the final output is computed by a linear projection of the concatenated outputs from each self-attention head, which can be formulated by
[Equation]
where W o ∈ R ( N h × D h ) × D is a learnable parameter.
In practice, D is usually h
D
Transformer blocks.
A standard vision Transformer as described in [15] consists of a patchembedding layer, several blocks and a prediction head.
Let l be the index of a block.
Then each block contains an MSA layer and a position-wise feed-forward network (FFN), which can expressed as
[Equation]
where LN denotes the LayerNorm [1] and an FFN consists of two FC layers with GELU [21] nonlinearity in between.
Recent works on ViT have proposed to divide the blocks into several stages(typically 4 stages) to generate pyramid feature maps for dense prediction tasks.
Furthermore, toreduce the computational cost on high-resolution feature maps in the early stages, the MSA in Eq.
(4)has been replaced with efficient alternatives, such as SRA [45] and W-MSA [29].
Bottlenecks of LITv1.
Recent studies have shown that the MSA layers in the early stages in amodel still focus on local patterns [13].
With the same observation, LITv1 [33] removes all early
MSAs ( i .
e ., exclude Eq.
(4) in each block) while applying standard MSAs at the later stages.
Thisdesign principle has achieved better efficiency with competitive performance on ImageNet comparedto PVT [45] and Swin [29].
However, LITv1 still has two main bottlenecks in speed: 1) Given ahigh-resolution image, the standard MSAs in the later stages still result in huge computational cost.2)
The fixed relative positional encoding [29] dramatically slows down the speed when dealing withdifferent image resolutions.
This is due to interpolating the fixed-size positional encoding for eachdifferent image resolution.
In the next section, we describe a novel attention mechanism with zeropadding positional encoding to comprehensively accelerate LITv1.
Method
[Figure] 4.1 HiLo Attention
[Figure]
We propose to separately process high/low frequencies in a feature map at an attention layer.
We namethe new attention mechanism as HiLo, which is depicted in Figure 1.
Essentially, the low-frequencyattention branch (Lo-Fi) is to capture the global dependencies of the input (image/features), whichdoes not need a high-resolution feature map but requires global attention. On the other hand, thehigh-frequency attention branch (Hi-Fi) is to capture the fine detailed local dependency, whichrequires a high-resolution feature map but can be done via local attention.
In the next, we describethe two attentions in detail.
High-frequency attention.
Intuitively, as high frequencies encode local details of objects, it can beredundant and computationally expensive to apply global attention on a feature map.
Therefore, wepropose to design Hi-Fi to capture fine-grained high frequencies with local window self-attention( e .
g .,
2 × 2 windows), which saves significant computational complexity.
Furthermore, we employthe simple non-overlapping window partition in Hi-Fi, which is more hardware-friendly compared tothe time-consuming operations such as window shifting [29] or multi-scale window partition [53].
Low-frequency attention.
Recent studies have shown that the global attention in MSA helps tocapture low frequencies [35].
However, directly applying MSA to high-resolution feature mapsrequires huge computational cost.
As averaging is a low-pass filter [43], Lo-Fi firstly applies averagepooling to each window to get low-frequency signals in the input X .
Next, the average-pooled featuremaps are projected into keys K ∈ R N/s 2 × D h and values V ∈ R N/s 2 × D h , where s is the windowsize.
The queries Q in Lo-Fi still comes from the original feature map X .
We then apply the standardattention to capture the rich low-frequency information in feature maps.
Note that due to the spatialreduction of K and V , Lo-Fi simultaneously reduces the complexity for both Eq.
(1) and Eq.
Head splitting.
A naive solution for head assignment is to allocate both Hi-Fi and Lo-Fi the samenumber of heads as the standard MSA layer.
However, doubling heads results in more computationalcost.
In order to achieve better efficiency, HiLo separates the same number of heads in an MSA intotwo groups with a split ratio α , where (1 − α ) N h heads will be employed for Hi-Fi and the other αN hheads are used for Lo-Fi.
By doing so, as each attention has a lower complexity than a standard MSA,the entire framework of HiLo guarantees a low complexity and ensures high throughput on GPUs.
Moreover, another benefit of head splitting is that the learnable parameter W o can be decomposedinto two smaller matrices, which helps to reduce model parameters.
Finally, the output of HiLo is aconcatenation of the outputs from each attention
[Equation]
(6) where [ · ] denotes the concatenation operation.
[Figure]
Complexity Analysis.
Without loss of generality, we assume Hi-Fi and Lo-Fi have an equal numberof heads ( i .
e .,
α = 0 .
and the feature map has equal width and height.
Then, Hi-Fi and Lo-Fi havea computational cost of 74 ND 2 + s 2 ND and ( 34 + 1 s 2 ) ND
+ 1 s 2 N 2 D , respectively.
Derivation forthis result can be found in the supplementary material.
As shown in Figure 2-(a) and (b), under asmall input image resolution and a small value of s ( e .
g .
, s = 2 ), both Hi-Fi and Lo-Fi are comparablyefficient.
However, with a much higher resolution, Lo-Fi will result in a huge computational cost as itstill has a quadratic complexity in terms of N in Eq.
(2), i .
e ., 1 s 2 N 2 D .
In this case, slightly increasings ( e .
g ., s = 4 ) helps Lo-Fi achieve better efficiency while preserving the accuracy.
Combining thetwo attentions together, a larger window size also helps the overall framework of HiLo to reducemore FLOPs on high-resolution images, as shown in Figure 2-(c).
Thus, we suggest a practicalguideline for adopting HiLo into existing frameworks: increasing the window size in order to getbetter efficiency on high-resolution images.
We further show in Section 5.2 that this principle helps
LITv2 achieve a better speed and accuracy trade-off on downstream tasks, e .
g ., dense object detection.
4.2 Positional Encoding
Positional encoding is essential to self-attention due to its permutation-invariant property.
In LITv1,the later MSAs adopt the same relative positional encoding (RPE) scheme as Swin [29].
Thisapproach has significantly improves Swin by 0.7% in Top-1 accuracy on ImageNet compared tousing absolute positional encoding [29].
However, on dense prediction tasks, the fixed RPE has to beinterpolated for different image resolutions, which dramatically slows down the training/inferencespeed of LITv1.
As a recent study [24] has shown that position information can be implicitly learnedfrom zero-padding in CNNs, we propose to adopt one layer of 3 × 3 depthwise convolutional layerwith zero-padding in each FFN to replace the time-consuming RPE.
Notably, due to the eliminationof early MSAs, the early blocks in LITv1 only have FFNs left, which results in a tiny receptive fieldof 1 × 1 .
To this end, we show in Section 5.4 that the 3 × 3 convolutional filters adopted in each FFNalso improve LITv2 by simultaneously enlarging the receptive field in the early stages.
4.3 Model Architecture
LITv2 has three variants: LITv2-S, LITv2-M and LITv2-B, corresponding to the small, medium andbase settings in LITv1, respectively.
For a fair comparison, we keep the network width and depthas the same as LITv1.
The overall modifications are simply in two steps: 1) Adding one layer ofdepthwise convolution with zero-padding in each FFN and removing all relative positional encodingsin all MSAs.
2) Replacing all attention layers with the proposed HiLo attention.
Detailed architectureconfigurations can be found in the supplementary material.
Experiment
In this section we conduct experiments to validate the effectiveness of the proposed LITv2.
Followingcommon practice [45, 29, 11, 53], we experiment LITv2 on three tasks, including image classificationon ImageNet-1K [39], object detection and instance segmentation on COCO [28] and semanticsegmentation on ADE20K [57].
[Table] [Table]
5.1 Image Classification on ImageNet-1K
We conduct image classification experiments on ImageNet-1K [39], a large-scale image dataset whichcontains ∼ 1.2M training images and 50K validation images from 1K categories.
We measure themodel performance by Top-1 accuracy.
Furthermore, we report the FLOPs, throughput, as well astraining/test memory consumption on GPUs.
We compare with two CNN-based models [47, 30]and several representative SoTA ViTs [45, 29, 48, 53, 11].
Note that this paper does not considermobile-level architectures [7, 32].
Instead, we focus on models with the similar model size.
Besides,we are also not directly comparable with NAS-based methods [2, 4] as LITv2 is manually designed.
Implementation details.
All models are trained for 300 epochs from scratch on 8 V100 GPUs.
Attraining time, we set the total batch size as 1,024.
The input images are resized and randomly croppedinto 224 × 224 .
The initial learning rate is set to 1 × 10 − 3 and the weight decay is set to 5 × 10 − 2 .
Weuse AdamW optimizer with a cosine decay learning rate scheduler.
All training strategies includingthe data augmentation are same as in LITv1.
For HiLo, the window size s is set to 2.
The splitratio α is set to 0.9, which is chosen from a simple grid search on ImageNet-1K. The depthwiseconvolutional layers in FFNs are set with a kernel size of 3 × 3 , stride of 1 and zero padding size of 1 .
Results.
In Table 1, we report the experiment results on ImageNet-1K. First, compared to LITv1 baselines, LITv2 achieves consistent improvement on Top-1 accuracy while using less FLOPs.
Moreover,benefit from HiLo, LITv2 achieves faster throughput and significant training time memory reduction( e .
g ., 13% , 27% , 36% inference speedup for the small, medium and base settings, respectively)compared to LITv1.
Second, compared to CNNs, LITv2 models outperform all counterparts of
ResNet and ConvNext in terms of FLOPs, throughput and memory consumption while achievingcomparable performance.
Last, compared to SoTA ViTs, LITv2 surpasses many models in termsof throughput and memory consumption with competitive performance.
For example, under thesimilar amount of FLOPs, LITv2-S achieves faster inference speed than PVT-S and Twins-PCPVT-S
[Table]
with better performance.
Although Focal-Tiny achieves better Top-1 accuracy than LITv2-S [Table] it runsmuch slower ( i [Table]
e ., 384 vs. 1,471 images/s) and requires a large amount of memory to train.
Besides,when finetuning on a higher resolution [Table] LITv2-B outperforms both DeiT-B and Swin-B with a fasterthroughput and lower complexity.
5.2 Object Detection and Instance Segmentation on COCO
In this section [Table] we conduct experiments on COCO 2017, a common benchmark for object detectionand instance segmentation which contains ∼ 118K images for the training set and ∼ 5K images for thevalidation set.
Following common practice [11, 45], we experiment with two detection frameworks:
RetinaNet [27] and Mask R-CNN [19].
We measure model performance by Average Precision (AP).
Implementation details.
All backbones are initialized with pretrained weights on ImageNet-1K.
We train each model on 8 GPUs with 1 × schedule (12 epochs) and a total batch size of 16.
For afair comparison, we adopt the same training strategy and hyperparameter settings as in LITv1 [33].
Note that we pretrain LITv2 with a local window size of 2 and α = 0 .
9 on ImageNet-1K.
Underthe same α , a larger window size helps to achieve lower complexity and thus improves the speed athigh resolution, as explained in Section 4.1.
In this case, we also train models with a slightly largerwindow size of s = 4 for better efficiency, which we denote with “*”.
By default, FLOPs is evaluatedbased on the input resolution of 1280 × 800 .
FPS is measured on one RTX 3090 GPU based on themmdetection [3] framework.
Results.
In Table 2, we report the experimental results on COCO.
In general, LITv2 outperforms
LITv1 by a large margin in almost all metrics.
Besides, our LITv2 significantly surpasses ResNetin terms of AP, though it runs slightly slower in some cases.
More importantly, our LITv2 beats allthe compared SoTA ViTs, achieving the best AP with compelling fast inference speed.
Furthermore,by adopting a larger window size ( i . e ., s = 4 ), LITv2 achieves better efficiency with a slightlyperformance drop.
5.3 Semantic Segmentation on ADE20K
In this section, we evaluate LITv2 on the semantic segmentation task.
We conduct experiments on
ADE20K [57], a widely adopted dataset for semantic segmentation which has ∼ 20K training images,∼ 2K validation images and ∼ 3K test images.
Following prior works, we adopt the framework of
Semantic FPN [26] and measure the model performance by mIoU.
We train each model on 8 GPUswith a total batch size of 16 with 80K iterations.
All backbones are initialized with pretrained weightson ImageNet-1K.
The stochastic depth for the small, medium and base models of LITv2 are 0.2, 0.2and 0.3, respectively.
All other training strategies are the same as in LITv1 [33].
[Figure]
[Table]
[Figure]
[Table]
[Figure]
segmentation performance of different backbones on the ADE20K validation set.
FLOPs is evaluated based on the image resolution of 512 × 512 .
[Table]
Results.
In Table 3, we compare LITv2 with
ResNet and representative ViTs on ADE20K. Ingeneral, LITv2 achieves fast speed while outperforming many SoTA models.
For example, our
LITv2-S, LITv2-M and LITv2-B surpass SwinTi, Swin-S and Swin-B by 2.8%, 0.5% and 1.2%in mIoU with higher FPS, respectively.
In this section, we provide ablation studies for
LITv2, including the comparison with other efficient attention variants, the effect of α in HiLo,as well as the effect of architecture modifications.
By default, the throughput and memoryconsumption are measured on one RTX
GPU with a batch size of 64 under the resolutionof 224 × 224 .
Comparing HiLo with other attention mechanisms.
Based on LITv2-S, we compare theperformance of HiLo with other efficient attention mechanisms on ImageNet-1K, includingspatial reduction attention (SRA) in PVT [45],shifted-window based attention (W-MSA) in Swin [29] and alternated local and global attention(T-MSA) in Twins [11].
In our implementation, we directly replace HiLo with each compared method.
The results are reported in Table 4.
In general, HiLo reduces more FLOPs while achieving betterperformance and faster speed than the compared methods.
Furthermore, in Figure 3, we providecomprehensive benchmarks for more attention mechanisms based on different image resolutions,including Focal [53], QuadTree [40] and Performer [10].
Suffering from weak parallelizability, theyare even slower than that of using standard MSAs on GPUs.
Compared to them, HiLo achievescompetitive results in terms of the FLOPs, throughput and memory consumption.
Moreover, weconduct experiments based on ADE20K and Semantic FPN and show that HiLo achieves moreperformance gain than other attention mechanisms on the downstream dense prediction task.
Effect of α .
As shown in Figure 4, since the complexity of Lo-Fi is lower than Hi-Fi under theresolution of 224 × 224 and the window size of 2, a larger α helps to reduce more FLOPs as weallocate more heads to Lo-Fi.
Moreover, we found HiLo performs badly with α = 0 , in which case
[Figure] [Figure]
opA cc [Figure] cy ( % ) 3.
F L O P s ( G )
“RPE”
Name
[Table]
[Figure]
The magnitude is averaged over the the only the Hi-Fi is left and HiLo only focuses on high frequencies.
We speculate that low frequenciesplay an important role in self-attention.
For other values of α , we find the performance difference isaround 0.2%, where α = 0 .
9 achieves the best performance.
However, it is worth noting that althoughthe pure Lo-Fi branch ( α = 1 . 0 ) can achieve competitive results on ImageNet-1K, high-frequencysignals play an important role in capturing fine object details, which is particularly important fordense prediction tasks such as semantic segmentation.
For example, with α = 0 . 9 , LITv2-S based
{+Semantic FPN achieves more performance gain (+0.6%) than that of using α = 1 . 
+}
Effect of architecture modifications.
Based on LITv2-S, we explore the effect of architecturemodifications.
As shown in Table 5, benefit from the enlarged receptive field in the early stages, theadoption of depthwise convolutions improves the performance on both ImageNet and COCO.
Next,by removing the relative positional encoding, we significantly improve FPS on dense prediction taskswith a slightly performance drop on both datasets.
Also note that since depthwise convolutions haveencoded positional information by zero paddings [24], the elimination of RPE does not result in asignificant performance drop compared to prior works [29].
Finally, benefit from HiLo, we achievemore gains in model efficiency on both ImageNet and COCO.
Spectrum analysis of HiLo.
In Figure 5, we visualize the magnitude of frequency component [38]by applying Fast Fourier Transform (FFT) to the output feature maps from Hi-Fi and Lo-Fi attentions,respectively.
The visualisation indicates that Hi-Fi captures more high frequencies and Lo-Fi mainlyfocuses on low frequencies.
This strongly aligns with our aim of disentangling high and lowfrequencies in feature maps at a single attention layer.
Conclusion and Future Work
In this paper, we have introduced LITv2, a novel efficient vision Transformer backbone with fastspeed on GPUs and outperforms most SoTA models on ImageNet and downstream tasks.
We havealso presented HiLo attention, the core of LITv2 which helps to achieve better efficiency especiallyon high-resolution images.
With competitive performance, HiLo achieves great advantage over theexisting attention mechanisms across FLOPs, throughput and memory consumption.
Future workmay include incorporating convolutional stem [50] and overlapping patch embedding [46] for betterperformance, or extending HiLo on more tasks such as speech recognition and video processing.
Limitations and societal impact.
HiLo adopts a head splitting ratio to assign different numbersof heads into Hi-Fi and Lo-Fi.
In our experiments, this ratio is determined by a grid search on
ImageNet ( i .
e .,
α = 0 . 9 )
However, different tasks may have different importance on high and lowfrequencies.
Thus, the optimal value of α is task-specific and needs to be set manually.
Besides, ourwork potentially brings some negative societal impacts, such as the huge energy consumption andcarbon emissions from large-scale training on GPU clusters.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately
reflect the
paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Code and pretrained models are included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] We use the same random seed as in recent works for fair
comparison.
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5.
(b) Did you mention the license of the assets?
[No] The license of the public datasets canbe found in their websites.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Code and pretrained models are included in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We use public datasets ( e . g .
, ImageNet [39] and ADE20K [57]).
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] No such concerns as we are using widely
adopted public datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]