FACS: F AST A DAPTIVE C HANNEL S QUEEZING
Anonymous authors
Paper under double-blind review
A BSTRACT
Channel squeezing is one of the central operations performed in CNN bottlenecks to reduce the number of channels in a feature map.
This operation is carried out by using a 1 × 1 pointwise convolution which constitutes a signiﬁcant amount of computations and parameters in a given network.
ResNet- 50 for instance, consists of 16 such layers which form 33% of total layers and 25% ( 1 .
05 B / 4 .
12 B) of total FLOPs or computations.
In the light of their predominance, we propose a novel “Fast Adaptive Channel Squeezing” module which carries out the squeezing operation in a computationally efﬁcient manner.
The key beneﬁt of FACS is that it neither alters the number of parameters nor affects the accuracy of a given network.
When plugged into diverse CNNs architectures, namely ResNet, VGG, and MobileNet-v 2 , FACS achieves state-of-the-art performance on ImageNet and
CIFAR datasets at dramatically reduced FLOPs.
FACS also cuts the training time signiﬁcantly, and lowers the latency which is particularly advantageous for fast inference on edge devices.
The source-code will be made publicly available.
I NTRODUCTION
Introduced by ResNet He et al. (2016), squeeze-and expand units are the de-facto in state-of-the-art CNNs.
It is essentially a three layered unit in which the ﬁrst layer ( 1 × 1 ) performs channel squeezing while the third layer ( 1 × 1 ) performs channel expansion.
The middle layer ( 3 × 3 ), on the other hand maintains the channel count, and governs the network’s receptive ﬁeld.
Interestingly, in ResNet like designs, our key structural observation suggests that the squeeze and expand layers dominate in both the number as well as in the computations, while do not contribute to a network’s receptive ﬁeld due to their pointwise nature.
For instance, ResNet- 50 consists of 32 such layers out of 50 total layers, accounting for ∼ 54% ( 2 .
23 B / 4 .
12 B) of overall FLOPs, whereas ResNet- 101 consists of 66 of them out of 101 , accounting for ∼ 50% ( 3 .
98 B / 7 . 85 B) of total FLOPs.
As ResNet, VGG style networks are still dominant in both academia and industry Kumar & Behera (2019); Ding et al. ; Kumar et al. (2020) due to their architectural simplicity, customizability, and high representation power in contrast to newer complex networks Tan & Le (2019), the preceding observations compel us to ask a naturally arising question: can computations in 1 × 1 layers be reduced without sacriﬁcing network accuracy and parameters ?
If so, the inference of such networks can be signiﬁcantly accelerated on edge computing devices, beneﬁting a whole spectrum of applications such as autonomous driving, robotics, and so on.
To the best of our knowledge, the above problem is of great practical importance but not addressed previously.
Hence, in this paper, we show that indeed it is possible to achieve the desired objective by examining the functioning of squeeze layer through the lens of computational complexity.
To this end, we propose a novel “fast adaptive channel squeezing” (FACS) module that transforms a feature map X ∈ R C × H × W into another map Y ∈ R C R
× H × W , mimicking the functionality of a squeeze layer but with fewer computations, while retaining a network’s parameters, accuracy, and non-linearity.
FACS has been crafted by leveraging many network design concepts which may individually be found in the literature, however, their aggregation in FACS is entirely novel.
We evaluate FACS by embedding it into three CNN architectures: plain (VGG), residual (ResNet), and mobile-series (MobileNet-v 2 ), and on three datasets: ImageNet, Deng et al.
(2009), and CIFAR10 , CIFAR- 100 .
FACS is backed with comprehensive ablation study.
Since FACS is novel, we demonstrate intermediate data representations learnt by FACS using GradCAM Selvaraju et al. (2017), and show that they are better relative to the baselines.
FACS brings huge improvements e.g.
ResNet- 50 becomes ∼ 23% faster with 0 .
47% improved accuracy on ImageNet Deng et al.
(2009), whereas VGG becomes faster by ∼ 6% with 0 .
23% improved accuracy, without bells and whistles.
Next section deals with the related work.
Sec. 3 describes the FACS module, and its integration into different CNNs.
Sec. 4 carries solid experiments in support of FACS.
Sec. 5, concludes our ﬁndings.
R ELATED W ORK 2. C ONV N ETS
The earlier CNNs Krizhevsky et al. ; Simonyan & Zisserman (2014)
; He et al. (2016) are accuracy oriented but have grown complex in terms of branching Huang et al. despite achieving higher accuracy.
Mobile networks Sandler et al. ; Zhang et al.
(2018) focus on lower FLOPs for high speed inference by using depthwise convolutions Sifre & Mallat.
However they suffer from low representation power, need prolonged training schedule of 200 − 400 epochs in contrast to earlier ones which are typically trained for 90 − 120 epochs.
The limited representation power hinders their performance on downstream tasks e.g. Howard et al. (2017) needs 200 epochs on ImageNet to perform similar to Simonyan & Zisserman (2014) (trained for 75 epochs), but performs poorly on object detection.
Tan & Le (2019) tackles this issue by being efﬁcient in parameters and FLOPs, however ends-up being highly branched and deeper, memory hungry, and sluggish Ding et al.
Recent Transformer based methods are computationally hungry due to their attention mechanisms Vaswani et al. ; Dosovitskiy et al. (2020) which put them out of the reach of edge-devices.
ResNet design space exploration Schneider et al.
(2017) provides several variants competitive to Tan & Le (2019) like designs while being simpler, quite advantageous for edge devices and real-time applications.
Similarly, Ding et al. (2021) improves the speeds of VGG however its training time design has many branches which is still inferior to ResNet.
Furthermore, Hu et al. ; Woo et al.
(2018) propose novel units which yield improved accuracy in ResNet at the expense of increased parameters because of additional convolutions, and marginal computational overhead.
The above discussion shows that the older architectures still have a room for improvement, and real-world applications can beneﬁt by focusing on these areas.
2. S PEEDED -U P I NFERENCE
Fast inference of CNNs has been an active research area which mainly includes static pruning, network compression / quantization.
These methods are generally employed post training and are agnostic to any given network. More recently, dynamic pruning methods such as Gao et al.
(2018) have become state-of-the-art in this area, which deals with the limitation of static pruning methods.
Pruning works mainly by disabling or suppressing a set of channels in a feature maps, and for which convolution computations are inhibited. For example, Gao et al. (2018) consumes a tensor
X ∈ R C × H × W and outputs another tensor Y ∈ R C × H × W , but some of the channels in Y have zero values for which convolution operations are inhibited, thus saving computations. Gao et al.
(2018) achieves its goal by introducing new convolution layers into a network. FACS is orthogonal to the above approaches.
It consumes a tensor X ∈ R C × H × W , and performs computationally efﬁcient channel squeezing, resulting in a tensor Y ∈ R C R × H × W .
In addition, FACS preserves a network’s parameters and accuracy, and does not involve new convolution layers, rather reuses the parameters of the original squeeze layer while redeﬁning the information ﬂow.
Hence,
FACS and pruning can not complement each other. However as pruning is applicable to any network architecture, and as FACS is an architectural enhancement, thus it can be seamlessly integrated with any pruning method to speed-up the overall network. However, in this paper, we have limited ourself to the study of FACS in deep networks.
FACS
Given an input feature map X ∈ R C × H × W , FACS progressively infers intermediate 1 D descriptors z ∈ R C × 1 × 1 , p ∈ R C R × 1 × 1 , and an output feature map Y ∈ R C R
× H × W of reduced dimensionality.
FACS can primarily be sectioned into three stages: First, global context aggregation which provides channel-wise global spatial context in form of a 1 D descriptor z .
Second, cross channel information blending which transforms z into another descriptor p , referred as adaptive channel fusion probability.
Third, channel fusion which utilizes p and X in order to produce Y .
The overall structure of FACS module is illustrated in Figure 2.
For reference, we have also shown the baseline channel squeezing method in Figure 1.
It can be noticed that FACS is substantially different from the baseline counterpart in term of structure.
The only component shared between them is the convolution layer, yet the tensors on which it operates are different.
Further, FACS commonly employs global pooling, maximum or averaging operations which are the most fundamental operation in neural networks, however their use in FACS conﬁguration is entirely novel.
For this reason, we term FACS as a novel module to perform fast channel squeezing.
X ∈ R C × H × W
Conv 1 × 1 , W ∈ R C R × C
Y ∈ R C R × H × W
X ∈ R C × H × W z ∈ R C × 1 ×
GlobalPooling
Conv 1 × 1 , W ∈ R C R × C p ∈ R C R × 1 ×σ ( · )
Y ∈ R C R × H × W
ChannelFusion
3. G LOBAL C ONTEXT A GGREGATION
The ﬁrst and foremost step of FACS is to gather global spatial context.
For this purpose, we employ global average pooling over the per-channel activations of the input feature map X .
We apply the operator on X which results in channel-wise global contextual information in form of a 1 D descriptor z ∈ R C × 1 × 1 .
In the computations, each element z i corresponding to i th channel of z is given by:
[Equation]
The use of global pooling in our case is inspired by its success to aggregate global statistics in a computationally efﬁcient manner e.g. at the end of the classiﬁcation networks He et al.
; Hu et al.
; Woo et al.
3. C ROSS C HANNEL I NFORMATION B LENDING
From the previous step, we obtain global context which is limited to a single channel only, therefore the descriptor z is devoid of cross-channel information.
To mitigate the issue, the per-channel global information in z is now blended with the other channels by using a 1 × 1 convolution operator.
In addition to the blending, the operator also reduces the input number of channels by a factor of R i.e. number of input channels are reduced from C to C/ R .
It is important to note that the convolution operator is same as the squeeze layer in Figure 1, but operates only upon a 1 D descriptor instead of a 3 D tensor.
This is where the majority of computation savings occur in FACS relative to the baseline.
Next, the output of the convolution operator is squeezed into a range ∈ [0 , 1] using sigmoidal activation, resulting in a 1 D descriptor p ∈ R C/ R .
We refer the descriptor p as adaptive channel fusion probability which accommodates both the spatial, and the channel context.
Now both the p and X are forwarded to the channel fusion stage which yield the ﬁnal output Y of the FACS module.
3. C HANNEL F USION
In this stage, the task of channel squeezing of X is performed in an adaptive and computationally efﬁcient manner.
We deﬁne two reduction operations: maximum (Max), and average (Avg).
During fusion, the input feature map is ﬁrst virtually partitioned into chunks of R channels, and then the channels of each chunk are adaptively fused to produce a single channel.
In other words, fusion operation results in C/ R output channels, if the number of input channels were C .
We devise two strategies to perform this task, as discussed below.
3.3. T HRESHOLD B ASED F USION
In this strategy, the channels are fused by applying the reduction operator (Max or Avg) at each of the spatial location ∈ H × W across R channels.
For any spatial location ( h, w ) ∈ H × W , there exist R scalar values which are reduced to a single scalar by using one of the reduction operator.
The selection of the reduction operator is entirely adaptive because it is determined by the fusion probability p , which in turn is learnt by the network itself during the training.
In short, as soon as the reduction operator is selected, the values at any given location are fused via the following equations:
[Equation]
where, p i is the i th element of p , and T cf is channel fusion threshold.
This is a hyperparameter which is set to 0 .
5 based on our ablation study (See Sec. 4).
Y i ( h, w ) is the value at the ( h, w ) spatial location of the i th channel in the output feature map Y , i ∈ [0 , C/ R ] and j = R ∗ i .
A pictorial represntation of this fusion strategy is provided in Figure 3a.
X ∈ R C × H × W p ∈ R C R × 1 ×
Max
Avg × ×
Y ∈ R C R × H × W p i ≤ T cf p i > T cf p
i p i p i ≤ T cf p
i > T cf
X ∈ R C × H × W p ∈ R C R × 1 ×
Max
Avg × ×
Y ∈ R C R ×
H ×
W p i 1 − p
i +
[Figure]
[Figure]
W EIGHTED F USION
In contrast to the threshold based fusion, this strategy is free of T cf .
In this strategy, for any spatial location ( h, w ) ∈ H × W , all the R scalar values are ﬁrst reduced by using both of the reduction operators separately, and then the resulting values are combined using a weighted sum as given below.
[Equation]
In this strategy, the network itself learns to determine how much contribution should be given to a reduction operator.
Due to this ﬂexibility, each location in a chunk can enjoy different combinations of the reduction operators.
This is not the case with threshold based fusion.
Infact, threshold based fusion becomes a special case of weighted fusion when p i approaches the two extremas i.e. { 0 , 1 } .
Figure 3b illustrates the weighted fusion strategy.
In both of the above techniques, the use of p helps choosing or weighting different reduction operators adaptively based on the input.
Without that, all the squeeze layers shall have to be eliminated which will reduce to the case of Hussain & Hesheng (2019), leading to a heavy accuracy loss (Table 2).
3. A T HEORETICAL V IEWPOINT
Theoretically, FACS does not introduce any parameter overhead instead reuse the parameters of the squeeze layer.
As squeeze layer has learnable parameters, therefore it also contributes in the non-linearity of the network.
Hence, FACS [-does not introduce any memory overhead. The total memory required by the baseline squeeze operation can be given by: #M = C/ 
 × H × W .-]deals with two problems.
[-On the other hand, the memory required for FACS is given by: #M = C + C/ 
 + C/ 4 × H × W . We can see that there is negligible increment in the memory footprint i.e. from 0 .-]First , preserve non-linearity, and Second , reduce computations.
The computation reduction primarily occurs due to collapsing the input tensor X ∈ R C × H × W into a 1 D descriptor z ∈ R C .
Global pooling can potentially introduce information loss, therefore loss of non-linearity.
However, the proposed learnable cross channel information blending p and the fusion operator works in such a way that non-linearity is maintained.
In this process, role of p is crucial which is used for aggregating knowledge across a chunk of R channels at a given spatial location in a feature map to dynamically decide the choice of fusion operator.
In case of max fusion, only one channel out of R goes out while in the AVG pooling, the output have an effect of all the channels in the chunk.
Interestingly, each chunk of R channels can dynamically undergo either Max or Avg fusion, depending on p .
Hence, each spatial location has ﬂexibility to choose among Max or AVG.
For this reason, at any given spatial location ∈ R H × W , any one of the R channels make it to the output in max pooling, while at another location, some other channel wins.
This degree of freedom introduces a high level of non-linearity into the network which helps achieving FACS slightly better accuracy even at signiﬁcantly lower FLOP requirements.
These improvements brings huge training and inference speed-ups and are empirically veriﬁed in the Sec. 4.
3. FACS INSTANTIATION
FACS-ResNet.
ResNet is enhanced with FACS by replacing all the residual squeeze-expand units (Figure 4a) with FACS ones (Figure 4b).
We experiment with ResNet- 50 , ResNet- 101 , and ResNet152 , and add “FACS” preﬁx to the resulting networks.
FACS-VGG.
Although VGG Simonyan & Zisserman (2014) does not have a provision of squeeze layers since it consists only of 3 × 3 convolution (Figure 4c), we also pave a way to enhance them with FACS, due to its popularity in robotics and other real-time applications.
To achieve that, we insert a 1 × 1 convolution before each 3 × 3 convolution layer except the ﬁrst (Figure 4d)).
The purpose of this layer is to function as a squeeze layer which reduces the input channels by a factor of R .
We name the resulting model as VGG- 1 × 1 which is enhanced with FACS as shown in Figure 4e.
FACS-MobileNet-v 2 .
We also study FACS for mobile networks MobileNet-v2 Sandler et al. (2018) which employs inverted residual units with depthwise separable convolutions.
The baseline inverted residual unit and its FACS enhanced counterpart is illustrated in Figure 4f, 4g. [Table] ∈ [Table]
∈
[Table] ∈ [Table] FACS
Conv
[Table]
∈
[Table]
+
[Table]
∈
[Table]
T ∈ R
[Table]
R
[Table]
T ∈
[Table]
T ∈
[Table]
[Figure]
E XPERIMENTS
We evaluate FACS on ILSVRC’ 2012 ImageNet Deng et al. (2009) having 1 .
M training and 50 K validation images over 1000 categories.
For illustration of transfer learning, we use CIFAR- 10 and CIFAR- 100 image classiﬁcation datasets.
We also evaluate FACS onto a downstream task of semantic segmentation against a popular pipeline Zhao et al.
We ﬁrst show a comprehensive ablation study of FACS design, and then examine impact of FACS into previously instantiated networks.
We also provide crucial throughput and latency improvements over a ﬁve different GPU hardware including edge devices.
All the baselines are retrained in PyTorch Paszke et al.
Training Speciﬁcations.
The training procedure is kept straightforward to ensure reproducibility.
We use a batch size of 256 which is split across 8 GPUs.
We use a RandomResized crop of 224 ×along with horizontal ﬂip.
We use SGD with Nesterov momentum of 0 .
9 , base_lr = 0 .
1 with CosineAnnealing Loshchilov & Hutter (2016) learning rate scheduler, and a weight decay of 0 . 0001 .
All models are trained for 120 epochs following ResNet He et al.
(2016), unless otherwise stated.
4. A BLATION S TUDY
We empirically validate FACS design practices by performing the most pertinent ablations possible.
ResNet- 50 is adopted as the baseline for this purpose.
To begin with, we ﬁrst analyze the effect of changing the activation function in the cross-channel information blending stage, and then examine the effect of placing a BatchNorm prior to the sigmoidal activation.
Further, we verify the behavior of proposed channel fusion strategies, and also the effect of varying fusion threshold T cf .
Fusion Activation.
The channel fusion stage utilizes the fusion probability p produced by the sigmoidal layer.
Given that the value of p lies in the interval [0 , 1] , we wished to examine the behavior of FACS if this range is achieved via a different activation function.
For this purpose, we select
TanH function which natively squeezes the input into a range [ − 1 , 1] .
Therefore, we rewrite the mathematical expression to 0 .
5 ∗ (1+ TanH ) in order to place the output of TanH into the desired range of [0 , 1] .
We replace the sigmoidal activation with the above expression and retrain the network.
The corresponding results are shown in the top section of Table 1.
The measures reveal that sigmoidal activation is superior to TanH activation for the case of FACS.
BatchNorm in Global Context Aggregation.
Out of curiosity, we also analyze the behavior of
FACS module by placing a BatchNorm Ioffe & Szegedy (2015) after the 1 × 1 convolution, because the squeeze layer in the baseline method is also followed by a BatchNorm layer.
We observe that BatchNorm negatively impacts the performance (second section, Table 1).
Effect of Fusion Threshold ( T cf ).
The hyperparameter T cf is evaluated against three values ∈ { 0 . 0 , 0 . 5 , 1 . 0 } .
In accordance with the Eq. 2, T cf = 0 corresponds to Avg operator, T cf = 1 .corresponds to Max operator regardless of the value of p .
Whereas T cf = 0 .
5 offers equal opportunity to the Max and Avg reduction operators which is adaptively taken care of by the value of p .
We present an ablation over the aforementioned three values of T cf .
The third section of Table 1 holds the corresponding results.
It is observable that T cf
=
5 attains the best performance, which is the case when the network has the ﬂexibility to choose between the two reduction operators adaptively.
Hence, in the subsequent experiments, we use T cf = 0 .
5 for threshold based fusion.
One may argue that there is marginal accuracy difference for different T cf values, so why not use a preﬁxed operator?
As stated earlier in Sec 3.3, in that case p becomes of no use, and corresponding conv layer will have to be removed, which translates to Hussain & Hesheng (2019), resulting in loss of parameters and accuracy (Table-2). [Table]
Channel Fusion Strategy.
We train the network for both of the fusion strategies.
Our ﬁndings suggest that threshold based fusion is superior to the weighted fusion, as shown in the bottom section of Table 1.
Although, the effect is marginal, in the context of implementation, Eq.
(2) is easier and faster as compared to Eq.
It is because, Eq.
(2) requires atmost one operator to be computed based on the value of T cf , whereas Eq.
(3) requires both of the operators to be computed.
4. I MAGE N ET C LASSIFICATION
To the best of our knowledge, computationally efﬁcient squeeze operation has not been discussed in the literature, hence due to the lack of existing algorithms on this problem, the most suitable evaluation possible is with the baselines only.
Nonetheless, we could ﬁnd Hussain & Hesheng (2019) which is only the aligned work.
We have compared our approach with this approach.
Table 2 shows evaluation of FACS when plugged into ResNet.
FACS indeed achieves computationally efﬁcient squeezing, as indicated by the ∼ 23% reduction in FLOPs of all the FACS variants.
Though improving the accuracy is not our goal, FACS also boosts the accuracy signiﬁcantly i.e. 0 .
44% for FACS- 50 , 101 , and 152 respectively.
To our surprise, FACS-ResNet- 101 surpasses the baseline ResNet- 152 by 0 .
18% in FACS-ResNet vs Depthwise/Channel Pooling. In terms of comparison, a closely matching approach is depthwise pooling (DWP) Hussain & Hesheng (2019) which is a parameterless method. We endow ResNet- 50 with DWP and retrain the network with identical settings of FACS-ResNet50 .
Table 2 shows the corresponding comparison.
It can be noticed that DWP drastically reduces the number of parameters of the baseline ResNet which results in a huge accuracy drop of 1 . 68% .
Although, there is signiﬁcant parameter gap between FACS/ResNet and DWP, therefore, at a ﬁrst glance the comparison might seem unfair. However, our intention to conduct this experiment is to demonstrate that FACS does not incur any reduction in parameter while maintains the accuracy. Reduction Ratio ( R ). It is important to note that the reduction ratio R is not a hyperparameter of
FACS.
It can be a potential source of confusion when discussing Hu et al.
; Woo et al.
(2018) which employ two reduction ratios: one for the squeeze layer ( R ) and one for the SE or CBAM modules ( r ).
The value of R is independent of the FACS, SE or CBAM since it is the hyperparameter of the baseline ResNet where it is set to 4 .
Likewise, R is also set to 4 in the FACS variants.
On the other hand, the reduction ratio r is set to 16 in the SE unit of Hu et al.
(2018) and the shared MLP of the Woo et al.
(2018), but is irrelevant to FACS.
Although ablation on R is out of scope of this paper, yet we examine the impact of varying R on the performance of ResNet and FACS variants.
To evaluate, we perform an additional experiment by setting R = 8 whose outcomes are displayed in Table 2.
It can be noticed that with R = 8 ,
FACS-ResNet performs better by ∼ 0 .
81% with almost 25% lesser FLOPs which is huge in itself.
In addition, the accuracy gap between ResNet for R = 4 and R = 8 is 2 .
64% , while this gap reduces to 2 .
30% for FACS at signiﬁcant 56% reduction in the computation cost.
It shows the robustness of
FACS towards parameter reduction and improved generalization over large scale ImageNet dataset by learning richer data representations.
It further signiﬁes the vast scope of FACS in edge devices, especially during network customization for a required purpose.
FACS in Conjunction With SE.
When used in conjunction with SE Hu et al.
(2018) in ResNet- 50 ,
FACS reduces the computational demands of baseline SE-ResNet- 50 while performing better, Table 2.
[Table]
×
FP 32 precision
25 runs [Table] Throughput & Latency. The ∼ 23% FLOPs reduction directly reﬂects increased throughput as training time reduces from 4 to 2 .
5 days for ResNet- 50 , 4 to 3 days for ResNet- 101 , and 6 to 4 days for ResNet- 152 on an 8 × NVIDIA 1080 Ti system.
Additionally, huge latency improvements are also seen on edge computing devices.
Table 3 shows the latency analysis on a number of GPUs with varying compute capabilities.
It is done in order to demonstrate the portability of FACS across devices with varying computational power.
The ﬁrst three are high power GPUs among which the ﬁrst is compute specialized, second and third are gaming GPUs with former powerful than the latter.
On the other hand, the last two are edge computing devices which are far less powerful.
The difference in latency is mostly attributed to the variation in the number of computing elements or cores.
Theoretically, more cores should run a network faster, however owing to the sequential linking of layers, a layer must wait until preceding ones ﬁnish.
This causes similar latency for the ﬁrst three
GPUs.
However, for such devices, considerable gains are seen during batched inference and training, which is a measure of throughput and, is reﬂected via days-long reduction in training time.
In contrast, the impact of FACS is more pronounced on edge devices, where the cores are a scarce resource.
On Jetson-NX, FACS- 50 is 16% faster then ResNet- 50 , FACS- 101 is 14% faster, and
FACS- 152 is 15% faster.
On Jetson-Nano, FACS- 50 is 7% faster then ResNet- 50 , FACS- 101 is 13% faster, and FACS- 152 is 12% faster.
Notably, these speed-ups are huge which can be further enhanced via half-precision (FP 16 ) inference, roughly doubling the speed.
Considering the extensive usage of edge devices in real-time applications, the aforementioned enhancements are quite advantageous.
Dynamic Pruning Gao et al.
(2018) vs FACS Although dynamic pruning is not applicable to channel squeezing, we customize a recent approach “FBS” Gao et al.
(2018) for this purpose.
FBS works by subsampling channels from the input via global pooling followed by a convolution to compute saliency, and then keeping top-k channels based on that, while turning off the remaining ones by setting all the values to zero. Notice that, in FBS, both input and output tensors lie in R C × H × W .
To adapt it for our use, we keep the top-k channels, where k equals to C / R .
In this case,
FBS outputs a tensor ∈ R C R × H × W instead of ∈ R C × H × W which is its default behavior.
During training, we see convergence issues.
This happens because when keeping only top-k channels, the intermediate channels are dropped based on the input which can be different for different input.
When this tensor is passed to the subsequent convolution, due to dynamically changing position of the channels in the output, the convolution struggles to learn the relation between the channels, because channel indexing is lost during choosing top-k. This problem does not occur in naive FBS.
Secondly,
FBS requires an additional convolution to compute channel saliency which has more parameters in contrast to of FACS. It leads to roughly 48% rise in the parameters i.e. from 24 .
4 M to 37 M, which is not desirable.
This clearly indicates that pruning methods can not complement FACS, which further signiﬁes novelty and utility of FACS.
[Table] [Table]
4.2. FACS-VGG
We select the popular VGG- 16 variant of VGG series and upgrade it to VGG- 16 - 1 × 1 model with
BatchNorm Ioffe & Szegedy (2015) in all the layers.
We compare this model with its FACS variant.
The experiments are conducted for two values of reduction ratio R i.e. { 2 , 4 } , the results for which are shown in Table 4.
It can be observed that the FACS is consistent as computation reduction is ∼ 6% while accuracy is improved signiﬁcantly for different values of R .
Noticeably, relative gain w.r.t.
ResNet is smaller ( 23% vs 6% ), it is so because in this architecture most computations are packed into 3 × 3 convolutions ( 90% ) instead of 1 × 1 ( 6% ).
Nonetheless, potential of FACS is huge in VGG like architecture, especially for low power applications and edge devices because narrower models of VGG are quite popular for real-time applications Kumar et al.
4.2. FACS-M OBILE N ET - V 2 .
Mobile networks are not our point of consideration because of their limited representation power and complexity, as mentioned in Sec. 2.
However, we analyze how FACS behaves with mobilenets which employ inverted-residual bottlenecks consisting of depthwise separable convolution networks.
We choose MobileNet-v 2 Sandler et al. (2018) for the evaluation purpose and train for 120 epochs.
The experimental results are reported in Table 5.
We observe that for depthwise separable convolution,
FACS performs slightly inferior.
We hypothesis that the accuracy drop is due to the lack of interchannel communication in depth-wise separable convolutions.
In MobileNet-v 2 , this communication is performed by the squeeze layer in addition to performing the task of reducing the number of channels.
However in FACS, this is performed by global pooling and the convolution layer which operates on a descriptor (less information) instead of the tensor (more information) which according to our hypothesis becomes the reason of slightly reduced accuracy.
To validate our hypothesis, we replace all the depthwise separable convolution by fully connected convolutions in MobileNet-v 2 .
In this case, we observe improvements in the accuracy, as shown in the second section of Table 5.
This veriﬁes our previously made hypothesis that devoid of inter-channel communication limits the performance of FACS.
However, in this case the FLOPs improvement drops which is expected.
It is so because, inverted residual design increases the number of channels which when fed to a fully connected 3 × 3 convolution, leads to most of the network computations packed into these layers ( 95% ).
Hence, effect of FACS diminishes.
This experiment is for hypothesis veriﬁcation only because inverted bottlenecks with fully connected 3 × 3 are not used in practice due to exponential rise in computations.
In addition to the above, we also performed ablations on MobileNet-v 2 by placing BatchNorm Ioffe & Szegedy (2015) at the input and output of FACS.
We observe that BatchNorm slightly lowers the performance of depthwise convolutions while for the fully connected case, pre-BatchNorm marginally improves the performance.
Conclusively, we say that FACS performs on a par for depthwise separable convolutions, but 21% reduction in FLOPs can be traded for an accuracy drop of 0 . 67% which is negligible. While FACS performs better for fully connected convolutions in inverted-residual units.
[Table]
[Figure]
4. T RANSFER L EARNING
To analyze the generalization of FACS across datasets, we perform transfer learning from ImageNet to CIFAR- 10 and CIFAR- 100 .
Each of the dataset consists of 50 K training and 10 K test images.
For training, we ﬁnetune FACS-ResNet- 50 and ResNet- 50 pretrained over ImageNet.
The training protocol for both the datasets remain identical to that of ImageNet except 200 epochs.
Table 6 shows the classiﬁcation results.
It can be seen that, surprisingly, FACS-ResNet- 50 outperforms the baseline by 0 .
10% on CIFAR- 10 and by 0 .
62% on CIFAR- 100 at a lower FLOP requirements.
In addition, we evaluate FACS for a challenging downstream task of semantic segmentation.
We use the popular approach Zhao et al. and replace the backbone with FACS-ResNet- 101 .
Consistently, FACS variant outperforms the baseline both in terms of FLOPs efﬁciency and accuracy by 0 .
7% units mIoU (mean intersection over union), indicating that FACS indeed transfers better across tasks and datasets.
4. G RAD CAM V ISUALIZATION
As FACS employs global context in one of its stages and also choose the reduction operators adaptively, it encouraged us to analyze the behavior of layer activations in the network corresponding to a groundthruth class.
To do this, we use GradCAM Selvaraju et al. (2017) that utilizes gradients to visualize the importance of spatial activations.
GradCAM is given a label for which the attended regions in the feature maps are computed.
This feature of GradCAM is utilized to visualize and understand that how FACS attends the spatial regions in comparison to the baseline networks.
It gives us the qualitative explanation that despite the reduction in the computations, why FACS is performing
Figure 5 shows the GradCAM analysis for ResNet and VGG.
It can be seen that FACS signiﬁcantly improves the attended regions of the target class relative to the baselines.
Also, in case of images with multiple instances of an object, FACS focuses on each instance strongly.
It indicates that FACS enhances a network’s ability to generalize better by learning to emphasize class-speciﬁc parts.
C ONCLUSION
In this work, we introduce a novel module called “Fast Adaptive Channel Squeezing” (FACS) which redeﬁnes the manner in which channel squeezing is accomplished in CNNs.
It functions by exploiting global spatial context and blending cross-channel information in order to carry out computationally efﬁcient channel squeezing.
We plugged the FACS module into three distinct CNN architectures:
ResNet, VGG, and MobileNet-v 2 , and presented a comprehensive ablation study including ImageNet classiﬁcation results and transfer learning on semantic segmentation task.
Our experimental study suggests that FACS-enhanced CNNs are roughly 23% faster (lower FLOPs) while being more accurate in the same number of parameters while generalizing well across datasets and tasks.
We show that the reduction in FLOPs cuts a network’s training time signiﬁcantly, and also demonstrate that FACS results in lower latency of a network which speeds-up inference on edge devices. A PPENDIX A C OMPUTATIONAL COMPLEXITY The primary intent of the FACS module has been to reduce the computational complexity of squeeze operation. Therefore, it is necessary to discuss the computational requirement associated with FACS.
However for better understanding, we ﬁrst discuss the FLOPs of different kinds of layers.
A. C ONVOLUTION
Consider a convolution layer having N kernels and an input feature map X ∈ R C
× H × W .
The size of each kernel can be given by C × k × k .
FLOPs for convolution operation are determined in terms of Fusion-Multi-Addition (FMA) instructions, therefore the computational demands of a convolution layer can be given as:
[Equation]
The BatchNorm Ioffe & Szegedy (2015) operation is performed per spatial location and can be given as ˆX =
( X − µ ) γσ + β .
It can be implemented in three FLOPs i.e. ﬁrst for computing X − µ , second for γ / σ and last as FMA with β .
In general, σ is stored as σ 2 , therefore, it requires to compute square-root of σ 2 to obtain σ .
Overall, it takes four FLOPs to implement a BatchNorm operation per spatial location.
Thus, total number of FLOPs for a BatchNorm layer can be given as:
[Equation]
Optionally, during inference, BN can be fused with a Conv operation where a convolution is followed by BN, but we remain agnostic to such cases to account for the training phase and other architectures.
A. R E LU
A ReLU operation is given by Y
{+= X for X ≥ 0 and Y = 
+}
for X < 0 .
It simply requires a comparison instruction, leading to total number of FLOPs given by:
[Equation]
A. S IGMOID
A Sigmoid operation is given by Y = 1 / 1+exp − X .
It can be implemented in four FLOPs.
Therefore, the total FLOPs for a Sigmoid layer can be given by:
[Equation]
A. G LOBAL POOLING
Apart from the above layers, in an FACS module, a global pooling operation is also performed.
There are several ways to implement a global pooling operation, however, the most common is by using matrix multiplication routines and FMA instructions.
The whole channel of a feature map can be considered as a vector of size H × W which can be reduced to a scalar by taking its dot product with a vector whose all of the elements equals to one.
Hence, the total number of FLOPs for the global pooling operation can be given by:
[Equation]
A. C HANNEL FUSION
Channel fusion operates on a chunk of k channels.
For a Max operation, ( k − 1 ) compare instructions, while for an Avg operation, ( k − 1 ) FMA instructions are required.
Thus, the total number of FLOPs for channel fusion can be given by:
[Equation]
Based on the several equations developed above, the computational complexity of residual bottleneck and the FACS block can be calculated.
A. E FFECT OF FACS ON C OMPUTATIONAL C OMPLEXITY
In the baseline method, the squeeze layer operates upon X ∈ R C × H × W which requires C / R ×
C × H × W FLOPs.
Whereas in FACS, the global context aggregation requires C × H × W
FLOPs, cross-channel information blending requires C / R × C FLOPs.
and channel fusion requires C / R ×
( R − 1) × H × W FLOPs.
As an example, consider an input tensor X ∈ R 12 × 5 × 5 to a squeeze layer kernels of size 1 × 1 .
With
R = 4 , the N = 3 .
From the equations below, total number of FLOPs for a squeeze layer equals to 8475 .
[Equation]
On the other hand, based on the following equations, the FLOPs for the FACS module with k =equals to 811 .
[Equation]
In the above example, the baseline squeezing method requires 1275 FLOPs, whereas FACS requires only 523 and 748 FLOPs for threshold based and weight fusion respectively.
In the similar manner, we achieve huge gains when FACS is plugged into the existing networks which are discussed in the experiments.
For the calculation of FLOPs, kindly see above.
A. E FFECT OF FACS ON M EMORY
Despite the computational beneﬁts, FACS does not introduce any memory overhead.
The total memory required by the baseline squeeze operation can be given by:
{+#M = C/ 
+}
× H × W .
On the other hand, the memory required for FACS is given by: #M = C +
{+C/ 
+}
+ C/ 4 × H × W .
We can see that there is negligible increment in the memory footprint i.e. from 0 .
75 × C × H × W to 0 .
75 × C × H × W + 1 .
25 C .
For an FP 32 precision, the raw memory footprint will be 4 × M .
B C ODES AND I MPLEMENTATION
We will opensource the codes and all the pretrained models for the framework in PyTorch Paszke et al.
In order to better understand the concept, we have included our GPU implementation of channel fusion layer in the supplementary material.
The corresponding ﬁle is named as facs_layer.cu which consists of GPU implementation of the fusion formulations described in the paper.
Other ﬁles facs_layer.cpp , facs_layer.py are also provided which holds pytorch class implementation of FACS.
We have not provided the network training and inference code which will be released later.
It also contains code for the new experiment related to FBS in three ﬁles: facs_fbs_layer.cu which consists of GPU implementationa along with helper ﬁles facs_fbs_layer.cpp , facs_fbs_layer.py
C GPU C ONSIDERATIONS
The implementation of FACS operator is quite straightforward and fully parallelizable.
For reference, we also have provided the GPU implementation in the supplementary material.
The computations of fusion probability and the fused feature map are entirely parallelizable because they are pointwise operations.
The whole code can be implemented in merely 10 − 15 lines of a CUDA kernel and other parallelization device.