Focal Modulation Networks
Abstract
We propose focal modulation networks ( FocalNets in short), where self-attention (SA) is completely replaced by a focal modulation module for modeling token interactions in vision.
Focal modulation comprises three components: ( i ) hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, ( ii ) gated aggregation to selectively gather contexts for each query token based on its content, and ( iii ) element-wise modulation or affine transformation to fuse the aggregated context into the query.
Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts ( e.g. , Swin and Focal Transformers) with similar computational cost on the tasks of image classification, object detection, and semantic segmentation.
Specifically, FocalNets with tiny and base size achieve 82.3 % and 83.9 % top-1 accuracy on ImageNet-1K. After pretrained on ImageNet22K, it attains 86.5 % and 87.3 % top-1 accuracy when finetuned with resolution 224 2 and 384 2 , respectively.
When transferred to downstream tasks, FocalNets exhibit clear superiority.
For object detection with Mask R-CNN, FocalNet base trained with 1 × outperforms the Swin counterpart by 2.1 points and already surpasses Swin trained with 3 × schedule ( 49.0 v.s. 48.5).
For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4 , and beats
Swin at multi-scale ( 50.5 v.s. 49.7).
Using large FocalNet and mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO
Panoptic Segmentation.
These results render focal modulation a favorable alternative to SA for effective and efficient visual modeling.
Code is available at: https://github.com/microsoft/FocalNet .
Introduction
Transformers [66], originally proposed for natural language processing (NLP), have become a prevalent architecture in computer vision since the seminal work of Vision Transformer (ViT) [19].
Its promise has been demonstrated in various vision tasks including image classification [63, 69, 74, 46, 87, 65], object detection [3, 97, 93, 15], segmentation [67, 72, 13], and beyond [38, 91, 4, 9, 68, 36].
In Transformers, the self-attention (SA) is arguably the key to its success which enables inputdependent global interactions, in contrast to convolution operation which constrains interactions in a local region with a shared kernel.
Despite this advantages, the efficiency of SA has been a concern due to its quadratic complexity over the number of visual tokens, especially for high-resolution inputs.
To address this, many works have proposed SA variants through token coarsening [69], window attention [46, 65, 87], dynamic token selection [51, 81, 50], or the hybrid [79, 14].
Meanwhile, a number of models have been proposed by augmenting SA with (depth-wise) convolutions to capture long-range dependencies with a good awareness of local structures [74, 22, 78, 20, 18, 35, 7, 17].
In this work, we aim at answering the fundamental question: Is there a better way than (hybrid) SA to model input-dependent long-range interactions?
We start with an analysis on the current advanced designs for SA.
In Fig. 1(a), we show a window-wise attention between the red query token and the surrounding orange tokens proposed in Swin Transformer [46].
With a simple window-shift
[Figure]
strategy, Swin attains superior performance to ResNets across various vision tasks.
To enlarge the receptive field, focal attention [79] is proposed to additionally aggregate summarized visual tokens far away to capture coarse-grained, long-range visual dependencies, as shown in Fig. 1(b).
To produce the outputs, both methods involve heavy interactions (green arrows) followed by equally heavy aggregations (purple arrows) between the query and a large number of spatially distributed tokens (context features), which are extracted via either window partition or unfolding.
In this work, we take an alternative way by first aggregating contexts around each query and then modulating the query with the aggregated context .
This alteration still enables input-dependent token interaction, but significantly eases the process by decoupling the aggregation from individual queries, hence making the interactions light-weight upon a couple of features.
As shown in Fig.
1(c), we can simply apply query-agnostic aggregations ( e.g. , depth-wise convolution) to generate summarized tokens at different levels of granularity.
Afterwards, these summarized contexts are selectively aggregated depending on the query content, and finally fused into the query vector.
We call this new token interaction mechanism focal modulation , with which we replace SA in Transformers to build a simpler and attention-free architecture, called Focal Modulation Network , or FocalNet in short.
Finally, extensive experiments on image classification, object detection and segmentation, show that our FocalNets consistently and significantly outperform the SoTA SA counterparts with comparable costs.
Notably, our FocalNet achieves 82.3 % and 83.9 % top-1 accuracy using tiny and base model size, but with comparable and doubled throughput than Swin and Focal Transformer, respectively.
When pretrained on ImageNet-22K, our FocalNets achieve 86.5 % and 87.3 % in 224 2 and 384resolution, respectively, which are comparable or better than Swin at similar cost.
The advantage is particularly significant when transferred to dense prediction tasks.
For object detection on COCO [42], our FocalNets with tiny and base model size achieve 46.1 and 49.0 box mAP on Mask R-CNN 1 × , surpassing Swin with 3 × schedule (46.0 and 48.5 box mAP).
For semantic segmentation on
ADE20k [95], our FocalNet with base model size achieves 50.5 mIoU at single-scale evaluation, outperforming Swin at multi-scale evaluation (49.7 mIoU).
Using the pretrained large FocalNet, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic
Segmentation based on Mask2former [12].
Furthermore, we apply our focal modulation to monolithic ViT and clearly demonstrate superior performance across different model sizes.
Related Work
Self-attentions .
Transformer [66] is first introduced to vision in Vision Transformer (ViT) [19] by splitting an image into a sequence of visual tokens.
The self-attention (SA) strategy in ViTs has demonstrated superior performance to modern convolutional neural networks (ConvNets) such as ResNet [27] when trained with optimized recipes [19, 63].
Afterwards, multi-scale architectures [5, 69, 78], light-weight convolution layers [74, 22, 39], local self-attention mechanisms [46, 87, 14, 79] and learnable attention weights [84] have been proposed to boost the performance and support high-resolution input.
More comprehensive surveys are covered in [34, 24, 34].
Our focal modulation significantly differs from SA by first aggregating the contexts from different levels of granularity and then modulating individual query tokens, rendering an attention-free mechanism for token interactions.
For context aggregation, our method is inspired by focal attention proposed in [79].
However, the context aggregation for focal modulation is performed at each query location instead of target location, followed by a modulation rather than an attention.
These differences in mechanism lead to significant improvement of efficiency and performance as well.
Another closely related work is Poolformer [83] which uses a pooling to summarize the local context and a simple subtraction to adjust the individual inputs.
Though achieving decent efficiency, Poolformer lags behind popular vision transformers like Swin on performance.
As we will show later, capturing local structures at different levels is essential for performance.
MLP architectures .
Visual MLPs can be categorized into two groups: ( i ) Global-mixing MLPs, such as MLP-Mixer [60] and ResMLP [62], perform global communication among visual tokens through spatial-wise projections augmented by various techniques, such as gating, routing, and
Fourier transforms [44, 49, 58, 59].
( ii ) Local-mixing MLPs sample nearby tokens for interactions, using spatial shifting, permutation, and pseudo-kernel mixing [82, 29, 41, 8, 23].
Recently, MixShift-MLP [92] exploits both local and global interactions with MLPs, in a similar spirit of focal attention [79].
Both MLP architectures and our focal modulation network are attention-free.
However, focal modulation with multi-level context aggregation naturally captures the structures in both shortand long-range, and thus achieves much better accuracy-efficiency trade-off.
Convolutions .
ConvNets have been the primary driver of the renaissance of deep neural networks in computer vision.
The field has evolved rapidly since the emerge of VGG [52], InceptionNet [56] and ResNet [27].
Representative works that focus on the efficiency of ConvNets are MobileNet [30],
ShuffleNet [90] and EfficientNet [57].
Another line of works aimed at integrating global context to compensate ConvNets such as SE-Net [32], Non-local Network [71], GCNet [2], LR-Net [31] and
C3Net [80], etc .
Introducing dynamic operation is another way to augment ConvNets as demonstrated in Involution [37] and DyConv [10].
Recently, ConvNets strike back from two aspects: ( i ) convolution layers are integrated to SA and bring significant gains [74, 22, 39, 20] or the vice versa [64]; ( ii )
ResNets have closed the gap to ViTs using similar data augmentation and regularization strategies [73], and replacing SA with (dynamic) depth-wise convolution [25, 47] can also slightly surpass Swin.
Our focal modulation network also exploits depth-wise convolution as the micro-architecture but goes beyond by introducing a multi-level context aggregation and input-dependent modulation.
We will show this new module significantly outperforms raw convolution networks.
Focal Modulation Network 3.1 From Self-Attention to Focal Modulation
Given a visual feature map X ∈ R H × W × C as input, a generic encoding process generates for each visual token (query) x i ∈ R C a feature representation y i ∈ R C via the interaction T with its surroundings X ( e.g., neighboring tokens) and aggregation M over the contexts. Self-attention .
The self-attention modules use a late aggregation procedure formulated as
[Equation]
where the aggregation M 1 over the contexts X is performed after the attention scores between query and target are computed via interaction T 1 .
Focal modulation . In contrast, focal modulation generates refined representation
y i using an early aggregation procedure formulated as
[Equation]
where the context features are first aggregated using M 2 at each location i , then the query interacts with the aggregated feature based on T 2 to form y i . Comparing Eq.
(1) and Eq. (2), we see that ( i ) the context aggregation of focal modulation Mamortizes the computation of contexts via a shared operator ( e.g. , depth-wise convolution), while Min SA is more computationally expensive as it requires summing over non-shareable attention scores for different queries; ( ii ) the interaction T 2 is a lightweight operator between a token and its context, while T 1 involves computing token-to-token attention scores, which has quadratic complexity.
Based on Eq.
(2), we instantiate our focal modulation to
[Equation]
Input k q v Attention
Output
Interact
Aggregate
Linear
Linear Linear
Input q
Output
Aggregate
Linear
Linear modulator
Linear
Interact (a) [Figure] -Attention (b) Focal-Modulation
Input
Linear
Light-weight Linear
Hierarchical Contextualization
Gated Aggregation + + + (c) Context Aggregation 𝑍 𝑜𝑢𝑡 gate gate gate gate where q ( · ) is a query projection function and ⊙ is the element-wise multiplication. m ( · ) is a context aggregation function, whose output is called modulator . Fig. 2(a) and (b) compare self-attention and focal modulation.
The proposed focal modulation has the following favorable properties:
• Translation invariance .
Since q ( · ) and m ( · ) are always centered at the query token i and no positional embedding is used, the modulation is invariant to translation of input feature map X .
• Explicit input-dependency .
The modulator is computed via m ( · ) by aggregating the local features around target location i , hence our focal modulation is explicitly input-dependent.
• Spatial- and channel-specific .
The target location i as a pointer for m ( · ) enables spatial-specific modulation.
The element-wise multiplication enables channel-specific modulation.
• Decoupled feature granularity .
q ( · ) preserve the finest information for individual tokens, while m ( · ) extracts the coarser context.
They are decoupled but combined through modulation.
In what follows, we describe in detail the implementation of m ( · ) in Eq.
3.2 Context Aggregation via m ( · )
It has been proved that both short- and long-range contexts are important for visual modeling [79, 18, 47].
However, a single aggregation with larger receptive field is not only computationally expensive in time and memory, but also undermines the local fine-grained structures which are particularly useful for dense prediction tasks.
Inspired by [79], we propose a multi-scale hierarchical context aggregation.
As depicted in Fig.
2 (c), the aggregation procedure consists of two steps: hierarchical contextualization to extract contexts from local to global ranges at different levels of granularity and gated aggregation to condense all context features at different granularity levels into the modulator.
Step 1: Hierarchical Contextualization.
Given input feature map X , we first project it into a new feature space with a linear layer Z 0 = f
z ( X ) ∈
R H × W × C .
Then, a hierarchical presentation of contexts is obtained using a stack of L depth-wise convolutions.
At focal level ℓ ∈ { 1 , ..., L } , the output Z ℓ is derived by:
[Equation]
where f ℓa is the contextualization function at the ℓ -th level, implemented via a depth-wise convolution
DW-Conv with kernel size k ℓ followed by a GeLU activation function [28].
The use of depth-wise convolution for hierarchical contextualization of Eq.
(4) is motivated by its desirable properties.
Compared to pooling [83, 32], depth-wise convolution is learnable and structure-aware.
In contrast to regular convolution, it is channel-wise and thus computationally much cheaper.
Hierarchical contextualization of Eq.
(4) generates L levels of feature maps.
At level ℓ , the effective receptive field is r ℓ = 1 + (cid:80) ℓi
=1 ( k ℓ − 1) , which is much larger than the kernel size k ℓ .
To capture global context of the whole input, which could be high-resolution, we apply a global average pooling on the L -th level feature map Z L +1 =
Avg-Pool ( Z L ) .
Thus, we obtain in total ( L + 1) feature maps { Z ℓ } L +1 ℓ =1 , which collectively capture short- and long-range contexts at different levels of granularity.
Step 2: Gated Aggregation.
In this step, the ( L + 1) feature maps obtained via hierarchical contextualization are condensed into a modulator.
In an image, the relation between a visual token (query) and its surrounding contexts often depends on the content itself.
For example, the model might rely on local fine-grained features for encoding the queries of salient visual objects, but mainly global coarse-grained features for the queries of background scenes.
Based on this intuition, we use a gating mechanism to control how much to aggregate from different levels for each query.
Specifically, we use a linear layer to obtain a spatial- and level-aware gating weights G = f g ( X ) ∈
R H
× W × ( L +1) .
Then, we perform a weighted sum through an element-wise multiplication to obtain a single feature map Z out which has the same size as the input X ,
[Equation]
5) where G ℓ ∈ R H × W × 1 is a slice of G for the level ℓ .
When visualizing these gating maps in Fig.
3, we surprisingly find our FocalNet indeed learns gathering the context from different focal levels adaptively as we expect.
As we can see, for a token on a small object, it focuses more on the fine-grained local structure at low focal level, while a token in a uniform background needs to be aware of much larger contexts from higher levels.
Until now, all the aggregation is spatial.
To enable the communication across different channels, we use another linear layer h ( . ) to obtain the modulator map M = h ( Z out ) ∈
R H × W × C .
In Fig.
4, we visualize the magnitude of modulator M at the last layer of our FocalNet.
Interestingly, the modulators automatically pay more attention to the objects inducing the category, which implies a simple way of interpreting FocalNets.
Focal Modulation.
Given the implementation of m ( · ) as described above, focal modulation of Eq.(3) can be rewritten at the token level as
[Equation]
where g ℓi and z ℓi are the gating value and visual feature at location i of G ℓ and Z ℓ , respectively.
We summarize the proposed focal modulation in Pytorch-style pseudo code in Algorithm 1. As we can see, it can be easily implemented with a few convolution and linear layers.
3.3 Complexity
In focal modulation as Eq.
(6), there are mainly three linear projections q ( · ) , h ( · ) , and f z ( · ) for
Z 0 .
Besides, it requires a lightweight linear function f g ( · ) for gating and L depth-wise convolution f { 1 ,...,L } a for hierarchical contextualization.
Therefore, the overall number of learnable parameters is 3 C 2 + C ( L + 1) + C (cid:80) ℓ ( k ℓ ) 2 .
Since L and ( k ℓ ) 2 are typically much smaller than C , the model size is mainly determined by the first term as we will show in Sec. 4.
Regarding the time complexity, besides the linear projections and the depth-wise convolution layers, the element-wise multiplications introduce O ( C ( L + 2)) for each visual token.
Hence, the total complexity for a feature map is
O ( HW × (3 C 2 + C (2 L + 3) + C (cid:80) ℓ ( k ℓ ) 2 )) .
For comparison, a window-wise attention in Swin
Transformer with window size w is O
( HW × (3 C 2 + 2 Cw 2 )) , where w is the window size. [Table]
3.4 Network Architectures
We use the same stage layouts and hidden dimensions as in Swin [46] and Focal Transformers [79], but replace the SA modules with the focal modulation modules.
We thus construct a series of Focal Modulation Network (FocalNet) variants.
In FocalNets, we only need to specify the number of focal levels ( L ) and the kernel size ( k ℓ ) at each level.
For simplicity, we gradually increase the kernel size by 2 from lower focal levels to higher ones, i.e. , k ℓ = k ℓ − 1 + 2 .
To match the complexities of Swin and Focal Transformers, we design a small receptive field (SRF) and a large receptive field (LRF) version for each of the four layouts by using 2 and 3 focal levels, respectively.
We use non-overlapping convolution layers for patch embedding at the beginning (kernel size= 4 × 4 , stride=4) and between two stages (kernel size= 2 × 2 , stride=2), respectively.
Experiment 4.1 Image Classification
We compare different methods on ImageNet-1K classification [16].
Following the recipes in [63, 46, 79], we train FocalNet-T, FocalNet-S and FocalNet-B with ImageNet-1K training set and report
Top-1 accuracy (%) on the validation set.
Training details are described in the appendix.
To verify the effectiveness of FocalNet, we compare it with three groups of methods based on
ConvNets, Transformers and MLPs.
The results are reported in Table 1.
We see that FocalNets outperform the conventional CNNs ( e.g. , ResNet [27] and the augmented version [73]), MLP architectures such as MLP-Mixer [61] and gMLP [43], and Transformer architectures DeiT [63] and PVT [69].
In particular, we compare FocalNets against Swin and Focal Transformers which use the same architecture to verify FocalNet’s stand-alone effectiveness at the bottom part.
We see that FocalNets with small receptive fields (SRF) achieve consistently better performance than Swin
Transformer but with similar model size, FLOPs and throughput.
For example, the tiny FocalNet improves Top-1 accuracy by 0 .
9% over Swin-Tiny.
To compare with Focal Transformers (FocalAtt), we change to large receptive fields (LRF) though it is still much smaller than the one used in FocalAtt.
Focal modulation outperforms the strong and sophisticatedly designed focal attention across all model sizes.
More importantly, its run-time speed is much higher than FocalAtt by getting rid of many time-consuming operations like rolling and unfolding.
Model augmentation.
We investigate whether some commonly used techniques for vision transformers can also improve our FocalNets.
First, we study the effect of using overlapped patch embedding for downsampling [22].
Following [74], we change the kernel size and stride from (4 , 4) to (7 , 4) for patch embedding at the beginning, and (2 , 2) to (3 , 2) for later stages.
The comparisons are reported in Table 2.
Overlapped patch embedding improves the performance for models of all sizes, with slightly increased computational complexity and time cost.
Second, we make our FocalNets deeper but thinner as in [18, 96].
In Table 3, we change the depth layout of our FocalNet-T from 2-2-6-2 to 3-3-16-3, and FocalNet-S/B from 2-2-18-2 to 4-4-28-4.
Meanwhile, the hidden dimension at first stage is reduced from 96, 128 to 64, 96, respectively.
These changes lead to smaller model sizes and fewer FLOPs, but higher time cost due to the increased number of sequential blocks.
It turns out that
[Table]
[Table]
[Table]
[Table]
[Table]
going deeper improves the performance of FocalNets significantly.
These results demonstrate that the commonly used model augmentation techniques developed for vision transformers can be easily adopted to improve the performance of FocalNets.
ImageNet-22K pretraining.
We investigate the effectiveness of FocalNets when pretrained on
ImageNet-22K which contains 14.2M images and 21K categories.
Training details are described in the appendix.
We report the results in Table 4.
Though FocalNet-B/L are both pretrained with 224 × 224 resolution and directly transferred to target domain with 384 × 384 image size, we can see that they consistently outperform Swin Transformers.
4.2 Detection and Segmentation
Object detection and instance segmentation .
We make comparisons on object detection with
COCO 2017 [42].
We choose Mask R-CNN [26] as the detection method and use FocalNet-T/S/B pretrained on ImageNet-1K as the backbones.
All models are trained on the 118k training images and evaluated on 5K validation images.
We use two standard training recipes, 1 × schedule with 12 epochs and 3 × schedule with 36 epochs.
Following [46], we use the same multi-scale training strategy by randomly resizing the shorter side of an image to [480 , 800] .
Similar to [79], we increase the kernel size k ℓ by 6 for context aggregation at all focal levels to adapt to higher input resolutions.
Instead of up-sampling the relative position biases as in [79], FocalNets uses simple zero-padding for the extra kernel parameters.
This expanding introduces negligible overhead but helps extract longer range contexts.
For training, we use AdamW [48] as the optimizer with initial learning rate 10 − 4 and weight decay 0 . 05 .
All models are trained with batch size 16.
We set the stochastic drop rates to 0 .
3 in 1 × and 0 .
5 in 3 × training schedule for FocalNet-T/S/B, respectively.
The results are shown in Table 5.
We measure both box and mask mAP, and report the results for both small and large receptive field models.
Comparing with Swin Transformer, FocalNets improve the box mAP (AP b ) by 2 . 2 , 1 . 5 and 1 . 9 in 1 × schedule for tiny, small and base models, respectively.
In 3 × schedule, the improvements are still consistent and significant.
Remarkably, the 1 × performance of FocalNet-T/B (45.9/48.8) rivals Swin-T/B (46.0/48.5) trained with 3 × schedule.
When comparing with FocalAtt [79], FocalNets with large receptive fields consistently outperform under all settings [Table] [Table] [Table]
[Table] [Table] and cost much less FLOPs. For instance segmentation, we observe the similar trend as that of object detection for FocalNets.
To further verify the generality of FocalNets, we train three detection models,
Cascade Mask R-CNN [1], Sparse RCNN [55] and ATSS [88] with FocalNet-T as the backbone.
We train all models with 3 × schedule, and report the box mAPs in Table 6.
As we can see, FocalNets bring clear gains to all three detection methods over the previous SoTA methods.
Semantic segmentation .
We benchmark FocalNets on semantic segmentation, a dense prediction task that requires fine-grained understanding and long-range interactions.
We use ADE20K [95] for our experiments and follow [46] to use UperNet [75] as the segmentation method.
With FocalNetT/S/B trained on ImageNet-1K as the backbones, we train UperNet for 160k iterations with input resolution 512 × 512 and batch size 16.
For comparisons, we report both single- and multi-scale (MS) mIoU. Table 7 shows the results with different backbones.
FocalNet outperforms Swin and Focal
Transformer significantly under all settings.
Even for the base models, FocalNet (SRF) exceeds Swin
Transformer by 2 . 1 and 1 . 4 at single- and multi-scale, respectively.
Compared with Focal Transformer,
FocalNets outperform Focal Transformer, with a larger gain than that of Swin Transformer, and consume much less FLOPs.
These results demonstrate the superiority of FocalNets on the pixel-level dense prediction tasks, in addition to the instance-level object detection task.
Given the superior results for FocalNets on segmentation tasks shown in Table 7, we further investigate its effectiveness while scaling up.
Particularly, to fairly compare with Swin-L pretrained on ImageNet22K with 384 × 384, we also pretrain our FocalNet-L on ImageNet-22K with 384 × 384 with 3 focal levels and kernel sizes [3 , 5 , 7] . We use Mask2former [12] for semantic segmentation on ADE20K and panoptic segmentation on COCO. As shown in Table 8, FocalNet-L achieves superior performance to [Table]
[Table]
the panoptic segmentation, instance segmentation and semantic segmentation, respectively.
[Table]
Swin-L with similar model size and same pretraining data.
We note that the methods in gray font like
Swinv2-G and ViT-Adapter-L achieve better performance but use much more parameters and training data.
We leave the further scaling-up of our FocalNets as future work.
In Table 9, we compare different models for panoptic segmentation on COCO with 133 categories.
Our FocalNet-L slightly outperforms Swin-L on PQ, rendering a new state-of-the-art for panoptic segmentation.
These results clearly demonstrate the effectiveness of our FocalNets for various segmentation tasks.
4.3 Network Inspection
Model Variants .
We compare in Table 10 six different model variants derived from FocalNet.
• Depth-wise ConvNet.
It feeds the feature vectors at the top level L to a two-layer MLP.
The resultant model is close to DW-Net [25].
Although it can achieve 81.6%, surpassing Swin (81.3%), it underperforms FocalNet by 0.7%.
FocalNet uses depth-wise convolutions as a component but differently for aggregating contexts, which is then used to modulate each individual tokens. • Pooling Aggregator.
It replaces the depth-wise convolution module with average pooling, and is similar to MetaFormer [83] in terms of token aggregation.
Average pooling has slightly lower complexity but leads to a significant drop of accuracy by 1.8%.
Compared with depth-wise convolution, pooling is permutation-invariant and thus incapable of capturing visual structures.
• Global Pooling Aggregator.
It removes local aggregations at all levels and only keeps the global one ( Z L +1 ).
This variant resembles SENet [32].
It turns out that global context alone is insufficient for visual modeling, leading to a significant 6.7% drop. • Multi-scale Self-Attention.
Given the summarized tokens at different levels, a straightforward way to combine them is performing a SA among all of them.
We have developed two SA methods: computing q, k, v before and after aggregation, respectively.
Both methods result in visible performance drop and increase the run time latency, compared to FocalNet.
• Sliding-window Self-Attention.
Finally, we apply a sliding-window SA for each visual token within a window.
Since it involves dense interactions for each fine-grained tokens, the time and memory cost explodes, and the performance is worse than FocalNet.
[Table]
[Table]
Component Analysis .
Here we ablate FocalNet to study the relative contribution of each component
The result is reported in Table 11, where we investigate the impact of the following model architecture changes on model performance:
• Replacing Multiplication with Addition : we change the element-wise multiplication to addition in Eq.
(6), which converts the modulator into a bias term.
This leads to 0.7% accuracy drop, which indicates that element-wise multiplication is a more powerful way of modulation than addition.
• No Global Aggregation : we remove the top global average pooling in focal modulation.
It hurts the performance by 0 .
Even though the hierarchical aggregation already covers a relatively large receptive field, global information ( Z L +1 ) is still useful for capturing global context.
• Top-only Aggregation : Instead of aggregating the feature maps from all focal levels, we only use the top level map.
In this case, the features at lower levels that are more “local” and “finegrained” are completely discarded.
This change leads to 0.4% performance drop, which verifies our hypothesis that features at different levels and spatial scopes compensate each other. • None-gating Aggregation : We remove the gating mechanism when aggregating the multiple levels of feature maps.
This causes 0.4% drop.
As we discussed earlier, the dependencies between visual token (query) and its surroundings differ based on the query content.
The proposed gating mechanism helps the model to adaptively learn where and how much to gather.
In Table 12, we study the effect of varying the focal level ( i.e. the number of depth-wise convolution layers L ).
In our experiments reported above, the results show that large receptive field in general achieves better performance (LRF v.s. SRF).
Here, we investigate by further altering L .
In additional to setting L = 2 and 3 , we also try L = 0 , L = 1 , and L = 4 .
Accordingly, increasing L brings slight improvement and finally reaches a plateau.
Surprisingly, a single level with kernel size 3 can already obtain a decent performance.
When we increase the single-level kernel size from 3 to 13, there is a slight 0.1% drop, and a 0.4% gap to the one with three levels but same size of receptive field (second row).
This indicates that simply increasing the receptive field does not necessarily improve the performance, and a hierarchical aggregation for both fine- and coarse-grained contexts is crucial.
[Table]
At last, we study whether our focal modulation can fit the monolithic architectures like ViTs.
We replace all SA modules in ViTs with focal modulation to construct monolithic FocalNet-T/S/B.
We use patch size 16 and three focal levels with kernel sizes 3,5 and 7, so that the effective receptive field is close to the global SA in ViT. As shown in Table 13, FocalNets consistently outperform ViTs, with comparable FLOPs and inference speed.
Conclusion
We have proposed focal modulation , a new mechanism that enables input-dependent token interactions for visual modeling.
It consists of a hierarchical contextualization to gather for each query token its contexts from short- to long-ranges, a gated aggregation to adaptively gather contexts based on the query content, followed by a simple modulation.
With focal modulation , we built a series of simple attention-free Focal Modulation Networks (FocalNets).
Extensive experiments show that FocalNets significantly outperform the SoTA SA counterparts ( e.g., Swin and Focal Transformer) with similar time-/memory-cost on the tasks of image classification, object detection and semantic segmentation.