Original: L EARNING WITH P ARAMETER -F REE L AYERS
Revised: L EARNING WITH P ARAMETER -F REE L AYERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Trainable layers such as convolutional building blocks are the standard network design choices by learning parameters to capture the global context through successive spatial operations.
Revised: Trainable layers such as convolutional building blocks are the standard network design choices by learning parameters to capture the global context through successive spatial operations.

Original: When designing an efﬁcient network, trainable layers such as the depthwise convolution is the source of efﬁciency in the number of parameters and FLOPs, but there was little improvement to the model speed in practice.
Revised: When designing an efﬁcient network, trainable layers such as the depthwise convolution is the source of efﬁciency in the number of parameters and FLOPs, but there was little improvement to the model speed in practice.

Original: This paper argues that simple built-in parameter-free operations can be a favorable alternative to the efﬁcient trainable layers replacing spatial operations in a network architecture.
Revised: This paper argues that simple built-in parameter-free operations can be a favorable alternative to the efﬁcient trainable layers replacing spatial operations in a network architecture.

Original: We aim to break the stereotype of organizing the spatial operations of building blocks into trainable layers.
Revised: We aim to break the stereotype of organizing the spatial operations of building blocks into trainable layers.

Original: Extensive experimental analyses based on layer-level studies with fully-trained models and network architecture searches are provided to investigate whether parameter-free operations  are functional.
Revised: Extensive experimental analyses based on layer-level studies with fully-trained models and network architecture searches are provided to investigate whether parameter-free operations such as the max-pool are functional.

Original: The studies eventually give us a simple yet effective idea for redesigning network architectures, where the parameter-free operations are heavily used as the main building block without sacriﬁcing the model accuracy as much.
Revised: The studies eventually give us a simple yet effective idea for redesigning network architectures, where the parameter-free operations are heavily used as the main building block without sacriﬁcing the model accuracy as much.

Original: Experimental results on the ImageNet dataset demonstrate that the network architectures with parameter-free operations could enjoy the advantages of further efﬁciency in terms of model speed, the number of the parameters, and FLOPs.
Revised: Experimental results on the ImageNet dataset demonstrate that the network architectures with parameter-free operations could enjoy the advantages of further efﬁciency in terms of model speed, the number of the parameters, and FLOPs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image classiﬁcation has been advanced with deep convolutional neural networks (Simonyan & Zisserman, 2015; Huang et al., 2017;
Revised: Image classiﬁcation has been advanced with deep convolutional neural networks (Simonyan & Zisserman, 2015; Huang et al., 2017;

Original: He et al., 2016b) with the common design paradigm of the network building blocks with trainable spatial convolutions inside.
Revised: He et al., 2016b) with the common design paradigm of the network building blocks with trainable spatial convolutions inside.

Original: Such trainable layers with learnable parameters effectively grasp attentive signals to distinguish input but are computationally heavy.
Revised: Such trainable layers with learnable parameters effectively grasp attentive signals to distinguish input but are computationally heavy.

Original: Rather than applying pruning or distillation techniques to reduce the computational cost, developing new efﬁcient operations has been another underlying strategy.
Revised: Rather than applying pruning or distillation techniques to reduce the computational cost, developing new efﬁcient operations has been another underlying strategy.

Original: For example, a variant of the regular convolution, such as depthwise convolution (Howard et al., 2017) has been proposed to bring more efﬁciency by reducing the inter-channel computations.
Revised: For example, a variant of the regular convolution, such as depthwise convolution (Howard et al., 2017) has been proposed to bring more efﬁciency by reducing the inter-channel computations.

Original: The operation has beneﬁts in the computational budgets, including the number of parameters and FLOPs.
Revised: The operation has beneﬁts in the computational budgets, including the number of parameters and FLOPs.

Original: However, the networks heavily using the depthwise convolution (Howard et al., 2017; Sandler et al., 2018; Tan & Le, 2019) have an inherent downside of the latency, which generally do not reach the speed of the regular convolution.
Revised: However, the networks heavily using the depthwise convolution (Howard et al., 2017; Sandler et al., 2018; Tan & Le, 2019) have an inherent downside of the latency, which generally do not reach the speed of the regular convolution.

Original: In a line of study of efﬁcient operations, there have been many works (Wang et al., 2018; Wu et al., 2018; Han et al., 2020; Tan & Le, 2021) based on the regular convolution and the depthwise convolution.
Revised: In a line of study of efﬁcient operations, there have been many works (Wang et al., 2018; Wu et al., 2018; Han et al., 2020; Tan & Le, 2021) based on the regular convolution and the depthwise convolution.

Original: Most methods utilize the depthwise convolution’s efﬁciency or target FLOPs-efﬁciency but are slow in the computation.
Revised: Most methods utilize the depthwise convolution’s efﬁciency or target FLOPs-efﬁciency but are slow in the computation.

Original: Meanwhile, parameter-free operations were proposed; a representative work is the Shift operation (Wu et al., 2018).
Revised: Meanwhile, parameter-free operations were proposed; a representative work is the Shift operation (Wu et al., 2018).

Original: Its efﬁciency stems from the novel operation without learning spatial parameters by letting the feature mixing convolutions learn from the shifted features.
Revised: Its efﬁciency stems from the novel operation without learning spatial parameters by letting the feature mixing convolutions learn from the shifted features.

Original: However, the implementation does not bring about the actual speed-up as expected.
Revised: However, the implementation does not bring about the actual speed-up as expected.

Original: This is because the operation-level optimization is still demanding compared to the regular convolution with highly optimized performance.
Revised: This is because the operation-level optimization is still demanding compared to the regular convolution with highly optimized performance.

Original: Another parameter-free operation feature shufﬂing (Zhang et al., 2018) is a seminal operation that reduces the computational cost of the feature mixing layer.
Revised: Another parameter-free operation feature shufﬂing (Zhang et al., 2018) is a seminal operation that reduces the computational cost of the feature mixing layer.

Original: However, it hardly plays the role of a spatial operation.
Revised: However, it hardly plays the role of a spatial operation.

Original: In this paper, we focus on efﬁcient parameter-free operations that actually replace trainable layers for network design.
Revised: In this paper, we focus on efﬁcient parameter-free operations that actually replace trainable layers for network design.

Original: We revisit the popular parameter-free operations, the max-pool and the avg-pool operations ( i.e., layers), which are used in many deep neural networks (Simonyan & Zisserman, 2015; 
Revised: We revisit the popular parameter-free operations, the max-pool and the avgpool operations ( i.e., layers), which are used in many deep neural networks (Simonyan & Zisserman, 2015; Huang et al., 2017) restricted to perform downsampling (Simonyan & Zisserman, 2015;

Original: Huang et al., 2017) restricted to perform downsampling (Simonyan & Zisserman, 2015; Howard et al., 2017; Sandler et al., 2018;
Revised: Howard et al., 2017; Sandler et al., 2018;

Original: He et al., 2016b).
Revised: He et al., 2016b).

Original: Can those simple parameter-free operations be used as the main network building block?
Revised: Can those simple parameter-free op- erations be used as the main network building block?

Original: If so, one can reduce a large portion of the parameters and the overall computational budget required during training and inference.
Revised: If so, one can reduce a large portion of the parameters and the overall computational budget required during training and inference.

Original: To answer the question, we conduct comprehensive studies on the layer replacements of the regular convolutions inside networks searched upon the baseline models with model training.
Revised: To answer the question, the max-pool and the avg-pool operations are chosen to be demonstrated as representative simple parameter-free operations. We ﬁrst conduct comprehensive studies on the layer replacements of the regular convolutions inside networks searched upon the baseline models with model training.

Original: Additionally, we incorporate a network architecture search (Liu et al., 2019) to explore effective architectures based on the operation list with  parameter-free operations and convolutional operations.
Revised: Additionally, we incorporate a network architecture search (Liu et al., 2019) to explore effective architectures based on the operation list with the parameter-free operations and convolutional operations.

Original: Based on the investigations, we provide a simple rule of thumb to design an efﬁcient architecture using  parameter-free operations upon primitive network architectures.
Revised: Based on the investigations, we provide a simple rule of thumb to design an efﬁcient architecture using the parameter-free operations upon primitive network architectures.

Original: The design guide is applied to popular heavy networks and validated by the performance trained on ImageNet (Russakovsky et al., 2015).
Revised: The design guide is applied to popular heavy networks and validated by the performance trained on ImageNet (Russakovsky et al., 2015).

Original: It turns out that our models have apparent beneﬁts to the computational costs, particularly the faster model speeds.
Revised: It turns out that our models have apparent beneﬁts to the computational costs, particularly the faster model speeds.

Original: In addition, ImageNet-C (Hendrycks &Dietterich, 2019) and ImageNet-O
Revised: In addition, ImageNet-C (Hendrycks &

Original:  (Hendrycks et al., 2021) results show our models are less prone to be overﬁtting.
Revised: Dietterich, 2019) and ImageNet-O (Hendrycks et al., 2021) results show our models are less prone to be overﬁtting.

Original: We further propose a novel deformable parameter-free operation  to demonstrate the future of a parameter-free operation.
Revised: We further propose a novel deformable parameter-free operation based on the maxpool and the avg-pool to demonstrate the future of a parameter-free operation.

Original: Finally, we show that the parameter-free operations can successfully replace the self-attention layer (Vaswani et al., 2017), thus attaining further efﬁciency and speed up.
Revised: Finally, we show that the parameter-free operations can successfully replace the self-attention layer (Vaswani et al., 2017), thus attaining further efﬁciency and speed up.

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows

Original:  • We study whether parameter-free operations can replace trainable layers as a network building block.
Revised: : • We study whether parameter-free operations can replace trainable layers as a network building block.

Original: To our knowledge, this is the ﬁrst attempt to investigate a simple, builtin parameter-free layer as a building block for further efﬁciency (§3).
Revised: To our knowledge, this is the ﬁrst attempt to investigate a simple, builtin parameter-free layer as a building block for further efﬁciency (§3).

Original: • We provide a rule of thumb for designing a deep neural network including convolutional neural networks and vision transformers with parameter-free layers (§4).
Revised: • We provide a rule of thumb for designing a deep neural network including convolutional neural networks and vision transformers with parameter-free layers (§4).

Original: • Experimental results show that our efﬁcient models outperform the previous efﬁcient models and yield faster model speeds with further robustness(§5).
Revised: • Experimental results show that our efﬁcient models outperform the previous efﬁcient models and yield faster model speeds with further robustness

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: A network building block where trainable layers are inside is a fundamental element of modularized networks (Xie et al., 2017; Sandler et al., 2018; Tan & Le, 2019).
Revised: A network building block where trainable layers are inside is a fundamental element of modularized networks (Xie et al., 2017; Sandler et al., 2018; Tan & Le, 2019).

Original: We start a discussion with the elementary building block and then explore more efﬁcient ones.
Revised: We start a discussion with the elementary building block and then explore more efﬁcient ones.

Original: 2. B ASIC B UILDING B LOCKS
Revised: 2. B ASIC B UILDING B LOCKS

Original: Convolution Layer.
Revised: Convolution Layer.

Original: We recall the regular convolutional operation by formulating with matrix multiplication ﬁrst.
Revised: We recall the regular convolutional operation by formulating with matrix multiplication ﬁrst.

Original: Let f ∈ R c in × H × W as the input feature, the regular convolution of the feature f with kernel size k and stride r is given as
Revised: Let f ∈ R c in × H × W as the input feature, the regular convolution of the feature f with kernel size k and stride r is given as

Original: [Equation]
Revised: [Equation]

Original: where W denotes the weight matrix, and the function σ ( · ) denotes an activation function such as
Revised: where W denotes the weight matrix, and the function σ ( · ) denotes an activation function such as

Original: ReLU (Nair & Hinton, 2010) with or without batch normalization (BN) (Ioffe & Szegedy, 2015).
Revised: ReLU (Nair & Hinton, 2010) with or without batch normalization (BN) (Ioffe & Szegedy, 2015).

Original: The convolution itself has been a building block due to the expressiveness and design ﬂexibility.
Revised: The convolution itself has been a building block due to the expressiveness and design ﬂexibility.

Original: VGG (Simonyan & Zisserman, 2015) network was designed by accumulating the 3 × 3 convolution to substitute the convolutions with larger kernel sizes but still have high computational costs.
Revised: VGG (Simonyan & Zisserman, 2015) network was designed by accumulating the 3 × 3 convolution to substitute the convolutions with larger kernel sizes but still have high computational costs.

Original: Bottleneck Block.
Revised: Bottleneck Block.

Original: We now recall the bottleneck block, which primarily aimed at efﬁciency.
Revised: We now recall the bottleneck block, which primarily aimed at efﬁciency.

Original: We represent the bottleneck block by the matrix multiplication as equation 1:
Revised: We represent the bottleneck block by the matrix multiplication as equation 1:

Original: [Equation]
Revised: [Equation]

Original: where g o,i,j = σ ( (cid:80) c in u =1 Q o,u · f u,i,j ) , and the matrix P and Q denote the weights of 1 × 1 convolutions with the inner dimensions ρc in .
Revised: where g o,i,j = σ ( (cid:80) c in u =1 Q o,u · f u,i,j ) , and the matrix P and Q denote the weights of 1 × 1 convolutions with the inner dimensions ρc in .

Original: This design regime is efﬁcient in terms of the computational budgets and even proven to be effective in the generalization ability when stacking up the bottleneck blocks compared with the basic blocks.
Revised: This design regime is efﬁcient in terms of the computational budgets and even proven to be effective in the generalization ability when stacking up the bottleneck blocks compared with the basic blocks.

Original: Albeit a 3 × 3 convolution is replaced with two feature mixing layers ( i.e., 1 × 1 convolutions), the expressiveness is still high enough with a low channel expansion ratio ρ ( i.e., ρ = 1 / 4 in ResNet (He et al., 2016b; Xie et al., 2017)).
Revised: Albeit a 3 × 3 convolution is replaced with two feature mixing layers ( i.e., 1 × 1 convolutions), the expressiveness is still high enough with a low channel expansion ratio ρ ( i.e., ρ = 1 / 4 in ResNet (He et al., 2016b; Xie et al., 2017)).

Original: However, due to the presence of the regular 3 × 3 convolution, only adjusting ρ hardly achieves further efﬁciency.
Revised: However, due to the presence of the regular 3 × 3 convolution, only adjusting ρ hardly achieves further efﬁciency.

Original: 2. E FFICIENT B UILDING B LOCKS
Revised: 2. E FFICIENT B UILDING B LOCKS

Original: Inverted Bottleneck.
Revised: Inverted Bottleneck.

Original: The grouped operations, including the group convolution (Xie et al., 2017) and the depthwise convolution (Howard et al., 2017)) have emerged as more efﬁcient building blocks.
Revised: The grouped operations, including the group convolution (Xie et al., 2017) and the depthwise convolution (Howard et al., 2017)) have emerged as more efﬁcient building blocks.

Original: Using the depthwise convolution inside a bottleneck (Sandler et al., 2018) is called the inverted bottleneck, which is represented as
Revised: Using the depthwise convolution inside a bottleneck (Sandler et al., 2018) is called the inverted bottleneck, which is represented as

Original: [Equation]
Revised: [Equation]

Original: where the summation over the channels in equation 2 has vanished.
Revised: where the summation over the channels in equation 2 has vanished.

Original: This slight modiﬁcation takes advantage of further generalization ability, and therefore stacking of the blocks leads to outperform
Revised: This slight modiﬁcation takes advantage of further generalization ability, and therefore stacking of the blocks leads to outperform

Original: ResNet with a better trade-off between accuracy and efﬁciency as proven in many architectures (Tan & Le, 2019; Han et al., 2021).
Revised: ResNet with a better trade-off between accuracy and efﬁciency as proven in many architectures (Tan & Le, 2019; Han et al., 2021).

Original: The feature mixing operation in equation 2 with the following pointwise activation function ( i.e., ReLU) may offer a sufﬁcient rank of the feature with the efﬁcient operation.
Revised: The feature mixing operation in equation 2 with the following pointwise activation function ( i.e., ReLU) may offer a sufﬁcient rank of the feature with the efﬁcient operation.

Original: However, the inverted bottleneck and the variants below usually need a large expansion ratio ρ > 1 to secure the expressiveness (Sandler et al., 2018; Howard et al., 2019; Tan & Le, 2019), so the actual speed is hampered by the grouped operation that requires more optimization on
Revised: However, the inverted bottleneck and the variants below usually need a large expansion ratio ρ > 1 to secure the expressiveness (Sandler et al., 2018; Howard et al., 2019; Tan & Le, 2019), so the actual speed is hampered by the grouped operation that requires more optimization on

Original: GPU (Gibson et al., 2020; Lu et al., 2021).
Revised: GPU (Gibson et al., 2020; Lu et al., 2021).

Original: Variants of Inverted Bottleneck More reﬁnements on the bottleneck block could bring further efﬁciency; the prior works (Wang et al., 2018; Han et al., 2020; Tan & Le, 2021; Wu et al., 2018) fully or partially redesigned the layers in the bottleneck with new operations, and therefore theoretical computational costs have been decreased.
Revised: More reﬁnements on the bottleneck block could bring further efﬁciency; the prior works (Wang et al., 2018; Han et al., 2020; Tan & Le, 2021; Wu et al., 2018) fully or partially redesigned the layers in the bottleneck with new operations, and therefore theoretical computational costs have been decreased.

Original: VersatileNet (Wang et al., 2018) replaced the convolutions with the proposed ﬁlters consisting of multiple convolutional ﬁlters; GhostNet (Han et al., 2020) similarly replaced the layers with the proposed module that concatenates a preceding regular convolution and additional depthwise convolution.
Revised: VersatileNet (Wang et al., 2018) replaced the convolutions with the proposed ﬁlters consisting of multiple convolutional ﬁlters; GhostNet (Han et al., 2020) similarly replaced the layers with the proposed module that concatenates a preceding regular convolution and additional depthwise convolution.

Original: Rather than involving new operations, a simple replacement by simplifying the operations has been proposed instead.
Revised: Rather than involving new operations, a simple replacement by simplifying the operations has been proposed instead.

Original: EfﬁcientNetV2 (Tan &
Revised: EfﬁcientNetV2 (Tan &

Original: Le, 2021) improved the training speed by fusing the pointwise and the depthwise convolutions to a single regular convolution.
Revised: Le, 2021) improved the training speed by fusing the pointwise and the depthwise convolutions to a single regular convolution.

Original: This takes advantage of using the highly-optimized operation rather than using new unoptimized operations.
Revised: This takes advantage of using the highly-optimized operation rather than using new unoptimized operations.

Original: ShiftNet (Wu et al., 2018) simplify the depthwise convolution to the shift operation, where the formulation is very similar to equation 3:
Revised: ShiftNet (Wu et al., 2018) simplify the depthwise convolution to the shift operation, where the formulation is very similar to equation 3:

Original: [Equation]
Revised: [Equation]

Original: where W v, : , : is simpliﬁed to have one 1 and 0 for the rest all h, w for each v .
Revised: where W v, : , : is simpliﬁed to have one 1 and 0 for the rest all h, w for each v .

Original: This change involves a new operation, so-called Shift, that looks like shifting the features after the computation, and the preceding and the following pointwise convoltuions mix the shifted output.
Revised: This change involves a new operation, so-called Shift, that looks like shifting the features after the computation, and the preceding and the following pointwise convoltuions mix the shifted output.

Original: The shift operation is actually classiﬁed as a parameter-free operation, but a large expansion ratio is yet needed to hold the expressiveness.
Revised: The shift operation is actually classiﬁed as a parameter-free operation, but a large expansion ratio is yet needed to hold the expressiveness.

Original: Furthermore, the implementation is not as readily optimized as regular operations, even in CUDA implementation (He et al., 2019; Chen et al., 2019a).
Revised: Furthermore, the implementation is not as readily optimized as regular operations, even in CUDA implementation (He et al., 2019; Chen et al., 2019a).

Original:  E FFICIENT B UILDING B LOCK WITH P ARAMETER - FREE O PERATIONS
Revised:  E FFICIENT B UILDING B LOCK WITH P ARAMETER - FREE O PERATIONS

Original: In this section, we extend an efﬁcient building block by incorporating parameter-free operations.
Revised: In this section, we extend an efﬁcient building block by incorporating parameter-free operations.

Original: We empirically study whether the parameter-free operations can be used as the main building block in deep neural networks like regular convolution.
Revised: We empirically study whether the parameter-free operations can be used as the main building block in deep neural networks like regular convolution.

Original: 3. M OTIVATION
Revised: 3. M OTIVATION

Original: The learning mechanism of ResNets (He et al., 2016b;a) has been analyzed as the iterative unrolled estimation (Greff et al., 2017; Jastrzebski et al., 2018) which iteratively reﬁne the features owing to the presence of skip connections.
Revised: The learning mechanism of ResNets (He et al., 2016b;a) has been analyzed as the iterative unrolled estimation (Greff et al., 2017; Jastrzebski et al., 2018) which iteratively reﬁne the features owing to the presence of skip connections.

Original: This indicates that some layers do not contribute much to learning, which has also been revealed in layer-drop experiments (Veit et al., 2016; Huang et al., 2016).
Revised: This indicates that some layers do not contribute much to learning, which has also been revealed in layer-drop experiments (Veit et al., 2016; Huang et al., 2016).

Original: In this light, we argue some building blocks in a residual network can be replaced with parameterfree operations.
Revised: In this light, we argue some building blocks in a residual network can be replaced with parameterfree operations.

Original: We investigate the replacement of the spatial operation in popular networks with parameter-free operations to rethink the common practice of network design.
Revised: We investigate the replacement of the spatial operation in popular networks with parameter-free operations to rethink the common practice of network design.

Original: [Figure]
Revised: [Figure]

Original: 3. R ETHINKING P ARAMETER -F REE O PERATIONS
Revised: 3. R ETHINKING P ARAMETER -F REE O PERATIONS

Original: Our goal is to design an efﬁcient building block that booststhe model speed in practice while maintainingthe model accuracy.
Revised: Our goal is to design an efﬁcient building block that boosts model speed in practice while maintaining model accuracy.

Original: Based on equation 2, we remove the inter-channel operation similarly done in equation 3 and equation 4.
Revised: Based on equation 2, we remove the inter-channel operation similarly done in equation 3 and equation 4.

Original: Then, instead of assigning W v,h,w to be a single value, as in equation 4, we let Wv,h,w have a dependency on the feature g
Revised: Then, instead of assigning W v,h,w to be a single value, as in equation 4, we let W have a dependency on the feature g

Original: ( i e., W v, : , : = 
Revised: ( i e., W v,h,w = s ( g v,r ∗

Original: s ( g ) ) by introducing a function s ( · ) .
Revised: i + h,r ∗ j + w ) ) by introducing a function s ( · ) .

Original: Then, the layer would have different learning dynamics interacting with P and Q .
Revised: Then, the layer would have different learning dynamics interacting with P and Q .

Original: Among many candidates to formalize s ( · ) , we allocate the function with a simple one that does not have trainable parameters.
Revised: Among many candidates to formalize s ( · ) , we allocate the function with a simple one that does not have trainable parameters.

Original: We let s ( · ) pick some large values of g in the range of all h, w per each v like the impulse function.
Revised: We let s ( · ) pick some large values of g in the range of all h, w per each v like the impulse function.

Original: Here, we simplify the function s ( · ) again to use only the largest values of g , where we have the representation of W v,i,j =1 , ( i, j )=
Revised: Here, we simplify the function s ( · ) again to use only the largest values of g , where we have the representation of W v,h ∗ ,w ∗ =1 , ( h ∗ , w ∗ ) =

Original: argmax ( h,w ) g v,r ∗ 
Revised: argmax ( h,w ) g v,r ∗ i

Original: i + h,r ∗ j + w  in equation 3.
Revised: + h,r ∗ j + w and other W v,h,w to be 0 in equation 3.

Original: One may come up with another simple option: let s ( · ) compute the moments such as mean or variance of g .
Revised: One may come up with another simple option: let s ( · ) compute the moments such as mean or variance of g .

Original: In practice, those operations can be replaced with built-in spatial pooling operations, which are highly optimized at the operation level.
Revised: In practice, those operations can be replaced with built-in spatial pooling operations, which are highly optimized at the operation level.

Original: In the following section, we empirically study the effectivness of efﬁcient parameter-free operations in network designs.
Revised: In the following section, we empirically study the effectivness of efﬁcient parameter-free operations in network designs.

Original: 3. E MPIRICAL S TUDIES
Revised: 3. E MPIRICAL S TUDIES

Original: On a Single Bottleneck.
Revised: On a Single Bottleneck.

Original: Our study begins with a single bottleneck block of ResNet (He et al., 2016b).
Revised: Our study begins with a single bottleneck block of ResNet (He et al., 2016b).

Original: We identify such a parameter-free operation can replace the depthwise convolution which is a strong alternative of the regular convolution.
Revised: We identify such a parameter-free operation can replace the depthwise convolution which is a strong alternative of the regular convolution.

Original: We train a large number of different architectures on CIFAR10 and observe the accuracy trend of trained models.
Revised: We train a large number of different architectures on CIFAR10 and observe the accuracy trend of trained models.

Original: The models consist of a single bottleneck, but the channel expansion ratio inside a bottleneck varies from 0.25 to 2, and there are two options for the base-channel width (32 and 64).
Revised: The models consist of a single bottleneck, but the channel expansion ratio inside a bottleneck varies from 0.25 to 2, and there are two options for the base-channel width (32 and 64).

Original: Training is done with three different optimizers of
Revised: Training is done with three different optimizers of

Original: SGD, AdamW (Loshchilov & Hutter, 2017b), and AdamP (Heo et al., 2021a).
Revised: SGD, AdamW (Loshchilov & Hutter, 2017b), and AdamP (Heo et al., 2021a).

Original: Finally, we have total 4 × 2 × 3 × 3=72 models for the study.
Revised: Finally, we have total 4 × 2 × 3 × 3=72 models for the study.

Original: Fig.1 exhibit that the parameter-free operation consistently outperforms the depthwise convolution in a bottleneck 1 .
Revised: Fig.1 exhibit that the parameter-free operation consistently outperforms the depthwise convolution in a bottleneck 1 .

Original: The regular convolutions clearly achieved higher absolute accuracy, but the efﬁciency (the accuracy over the number of parameters) is getting worse for larger expansion ratios.
Revised: The regular convolutions clearly achieved higher absolute accuracy, but the efﬁciency (the accuracy over the number of parameters) is getting worse for larger expansion ratios.

Original: We observe that the accuracy gaps between the regular convolution and the parameter-free operation are low for small channel expansion ratios (especially at the ratio=1/4 where the original bottleneck (He et al., 2016b; Xie et al., 2017) adopts).
Revised: We observe that the accuracy gaps between the regular convolution and the parameter-free operation are low for small channel expansion ratios (especially at the ratio=1/4 where the original bottleneck (He et al., 2016b; Xie et al., 2017) adopts).

Original: It means that the regular convolution in the original bottleneck block with the expansion ratio of 1/4 can be replaced by the parameter-free operation.
Revised: It means that the regular convolution in the original bottleneck block with the expansion ratio of 1/4 can be replaced by the parameter-free operation.

Original: On Multiple Bottlenecks.

Original: We extend the single bottleneck study to deeper networks having multiple bottleneck blocks.

Original: We similarly study the bottleneck with different spatial operations, includ-

Original: 3.2 Speed (ms) 91.
Revised: 3.2 Speed (ms) 91.

Original: T o p - 1 a cc .
Revised: T o p - 1 a cc .

Original: ( % ) max dwconv (a) Speed (1-image) 18.
Revised: ( % ) max dwconv (a) Speed (1-image) 18.

Original: 19.5 Speed (ms) 91.
Revised: 19.5 Speed (ms) 91.

Original: T o p - 1 a cc .
Revised: T o p - 1 a cc .

Original: ( % ) max dwconv (b) Speed (256-images) 0.17 0.18 0.19 0.20 0.21 0.

Revised: ( % ) max dwconv (b) Speed (256-images) 0.17 0.18 0.19 0.20 0.21 0.

Original: # of params.
Revised: # of params.

Original: (M) 91.
Revised: (M) 91.

Original: T o p - 1 a cc .
Revised: T o p - 1 a cc .

Original: ( % ) max dwconv (c) # parameters    34 FLOPs (M) 91.
Revised: ( % ) max dwconv (c) # parameters    34 FLOPs (M) 91.

Original: T o p - 1 a cc .
Revised: T o p - 1 a cc .

Original: ( % ) max dwconv (d) FLOPs ing the depthwise convolution and parameter-free operations, replacing the regular convolution.
Revised: ( % ) max dwconv (d) FLOPs On Multiple Bottlenecks. We extend the single bottleneck study to deeper networks having multiple bottleneck blocks. We similarly study the bottleneck with different spatial operations, including the depthwise convolution and parameter-free operations, replacing the regular convolution.

Original: We choose the max-pool operation again as the parameter-free operation and compare it with the depthwise convolution.
Revised: We choose the max-pool operation again as the parameter-free operation and compare it with the depthwise convolution.

Original: To study the impact of repeatedly using parameter-free operations in-depth, we exhaustively perform the layer replacements at layer-level upon the baseline network.
Revised: To study the impact of repeatedly using parameter-free operations in-depth, we exhaustively perform the layer replacements at layer-level upon the baseline network.

Original: We report the trade-off between accuracy and speed by replacing the regular convolutions with the max-pool operation and the depthwise convolution.
Revised: We report the trade-off between accuracy and speed by replacing the regular convolutions with the max-pool operation and the depthwise convolution.

Original: We use the baseline network, which has eight bottleneck blocks in a  ResNet architecture, and the study has done by letting each block have efﬁcient operations.
Revised: We use the baseline network, which has eight bottleneck blocks in a standard ResNet architecture (i.e., ResNet-26), and the study has done by replacing each block have efﬁcient operations (i e., the efﬁcient bottleneck and the bottleneck with the depthwise convolution).

Original: We train all the networks with large epochs (300 epochs) due to the depthwise convolution.
Revised: We train all the networks with large epochs (300 epochs) due to the depthwise convolution.

Original: Fig.2 illustrates the trade-offs of top-1 accuracy and the actual model speed; we observe the models with the parameter-free operations are faster than those with the depthwise convolution as the model accuracies are comparable
Revised: Fig.2 illustrates the trade-offs of top-1 accuracy and the actual model speed; we observe the models with the parameter-free operations are faster than those with the depthwise convolution as the model accuracies are comparable. [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: On Neural Architecture Searches.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  We further investigate whether parameter-free operations are likely to be chosen for model accuracy along with trainable operations in a neural architecture search (NAS).
Revised: [Table] [Table] On Neural Architecture Searches. We further investigate whether parameter-free operations are likely to be chosen for model accuracy along with trainable operations in a neural architecture search (NAS).

Original: We choose DARTS (Liu et al., 2019) with the default architecture of eight cells and perform searches on CIFAR-10 (Krizhevsky, 2009) 2 .
Revised: We choose DARTS (Liu et al., 2019) with the default architecture of eight cells and perform searches on CIFAR-10 (Krizhevsky, 2009) 2 .

Original: Towards a more sophisticated study, we reﬁne the search conﬁguration as follows.
Revised: Towards a more sophisticated study, we reﬁne the search conﬁguration as follows.

Original: First, we force the entire normal cells to be searched individually rather than following the default setting searching one uniﬁed normal cell.
Revised: First, we force the entire normal cells to be searched individually rather than following the default setting searching one uniﬁed normal cell.

Original: This is because when searching with the uniﬁed normal cell, it is more likely to be searched without the parameter-free operations to secure the expressiveness of the entire network (see the search result at the ﬁrst row in Table 1).
Revised: This is because when searching with the uniﬁed normal cell, it is more likely to be searched without the parameter-free operations to secure the expressiveness of the entire network (see the search result at the ﬁrst row in Table 1).

Original: Therefore we prevent this with the more natural conﬁguration.
Revised: Therefore we prevent this with the more natural conﬁguration.

Original: Second, the operation list is changed to ensure fairness of the expressiveness among the operations; since the primitive DARTS has the default operation list with the separable convolutions ( SEP CONV 3 × 3,
Revised: Second, the operation list is changed to ensure fairness of the expressiveness among the operations; since the primitive DARTS has the default operation list with the separable convolutions ( SEP CONV 3 × 3,

Original: SEP CONV 5 × 5) and the dilated separable convolutions ( DIL CONV 3 × 3, DIL CONV 5 × 5) which consist of multiple convolutions and ReLU with BN.
Revised: SEP CONV 5 × 5) and the dilated separable convolutions ( DIL CONV 3 × 3, DIL CONV 5 × 5) which consist of multiple convolutions and ReLU with BN.

Original: Thus, we need to simplify the primitive operations to [ MAX POOL 3 × 3, AVG POOL 3 × 3, CONV 1 × 1, CONV 3 × 3, DW CONV 3 × 3, ZERO ,
Revised: Thus, we need to simplify the primitive operations to [ MAX POOL 3 × 3, AVG POOL 3 × 3, CONV 1 × 1, CONV 3 × 3, DW CONV 3 × 3, ZERO ,

Original: SKIP CONNECT ].
Revised: SKIP CONNECT ].

Original: Finally, we further perform searches with different numbers of nodes inside each cell to conﬁrm the trend of search results regarding the architectural constraint.
Revised: Finally, we further perform searches with different numbers of nodes inside each cell to conﬁrm the trend of search results regarding the architectural constraint.

Original: All the searches are repeated with three different seed numbers.
Revised: All the searches are repeated with three different seed numbers.

Original: As shown in Table 1 and Fig.3, we observe the parameter-free operations are frequently searched in the normal cells, which is different from the original search result (at the ﬁrst row) similarly shown in
Revised: As shown in Table 1 and Fig.3, we observe the parameter-free operations are frequently searched in the normal cells, which is different from the original search result (at the ﬁrst row) similarly shown in

Original: [Figure]
Revised: [Figure]

Original: the previous works (Liu et al., 2019; Chen et al., 2019b et al.,  et al
Revised: the previous works (Liu et al., 2019; Chen et al., 2019b; Liang et al., 2019; Chu et al., 2020).

Original: Moreover, the accuracies are similar to the original setting with two uniﬁed cells, where the parameter-free operations are not searched in the normal cell (see the ﬁrst and the last rows).
Revised: Moreover, the accuracies are similar to the original setting with two uniﬁed cells, where the parameter-free operations are not searched in the normal cell (see the ﬁrst and the last rows).

Original: Interestingly, the number of the picked parameter-free operations increases as the number of nodes increases; they do not dominate the reduction cells.
Revised: Interestingly, the number of the picked parameter-free operations increases as the number of nodes increases; they do not dominate the reduction cells.

Original: As a result, it turns out that the accuracy objective in searches lets the parameter-free operations become chosen, and therefore the operations can be used at certain positions.
Revised: As a result, it turns out that the accuracy objective in searches lets the parameter-free operations become chosen, and therefore the operations can be used at certain positions which looks similar to the Inception blocks (Szegedy et al., 2015; Ioffe & Szegedy, 2015; Szegedy et al., 2016).

Original: Notice that similar results are produced with different settings and seeds.
Revised: Notice that similar results are produced with different settings and seeds.

Original:  D ESIGNING E FFICIENT D EEP N EURAL N ETWORKS
Revised:  D ESIGNING E FFICIENT D EEP N EURAL N ETWORKS

Original: Based on the studies above, parameter-free operations can be a building block of designing a network architecture.
Revised: Based on the studies above, parameter-free operations can be a building block of designing a network architecture.

Original: We apply the parameter-free operations to redesign a deeper neural network architecture, including a convolution neural network (CNN) and a vision transformer (ViT).
Revised: We apply the parameter-free operations to redesign a deeper neural network architecture, including a convoluption neural network (CNN) and a vision transformer (ViT).

Original: 4. E FFICIENT CNN A RCHITECTURES
Revised: 4. E FFICIENT CNN A RCHITECTURES

Original: Hybrid Architecture with Efﬁcient Bottlenecks.
Revised: Hybrid Architecture with Efﬁcient Bottlenecks.

Original: We propose an efﬁcient bottleneck, the building block of efﬁcient CNN architectures.
Revised: We employ the proposed efﬁcient bottleneck to design deeper efﬁcient CNNs.

Original: We involve a parameter-free operationthat plays a role of the spatial operationreplacing the regular convolutions in the regular bottleneck; namely, the 3 ×convolution, BN, and ReLU triplet are replaced with asingle parameter-free operation. We choose the max-pool operationas the spatial operation in the efﬁcient bottleneck 3 .
Revised: The max-pool operation plays a role of the spatial operation in the efﬁcient bottleneck; speciﬁcally, the 3 × 3 convolution, BN, and ReLU triplet are replaced with a max-pool operation 3 .

Original: To design the entire network architecture, fully utilizing the efﬁcient bottlenecks would yield a much faster inference speed model, but a combination of the efﬁcient bottleneck and the regular bottleneck has a better trade-off between accuracy and speed.
Revised: To design the entire network architecture, fully utilizing the efﬁcient bottlenecks would yield a much faster inference speed model, but a combination of the efﬁcient bottleneck and the regular bottleneck has a better trade-off between accuracy and speed.

Original: We use ResNet50 (He et al., 2016b) as the baseline to equip a simple heuristic of building a hybrid architecture of the efﬁcient bottleneck and the regular bottleneck.
Revised: We use ResNet50 (He et al., 2016b) as the baseline to equip a simple heuristic of building a hybrid architecture of the efﬁcient bottleneck and the regular bottleneck.

Original: The observation from the NAS searches in §3.3, the parameter-free operations are not explored much at the downsampling blocks but normal blocks.
Revised: The observation from the NAS searches in §3.3, the parameter-free operations are not explored much at the downsampling blocks but normal blocks.

Original: Thus, we similarly design a new efﬁcient network architecture using the efﬁcient bottlenecks as the fundamental building block except for the downsampling positions with the regular bottlenecks.
Revised: Thus, we similarly design a new efﬁcient network architecture using the efﬁcient bottlenecks as the fundamental building block except for the downsampling positions with the regular bottlenecks.

Original: We call this model hybrid architecture.
Revised: We call this model hybrid architecture.

Original: The details of the architectures are illustrated in Appendix A.
Revised: The details of the architectures are illustrated in Appendix A.

Original: Architectural Study.
Revised: Architectural Study.

Original: The architectural study is further conducted with the replacements of the regular spatial convolutions in the regular bottlenecks.
Revised: The architectural study is further conducted with the replacements of the regular spatial convolutions in the regular bottlenecks.

Original: We involve the efﬁcient bottlenecks into the stages of ResNet50, replacing the existing regular bottlenecks exhaustively to cover the design space as much as possible.
Revised: We involve the efﬁcient bottlenecks into the stages of ResNet50, replacing the existing regular bottlenecks exhaustively to cover the design space as much as possible.

Original: It is worth noting that performing a neural architecture search would yield a precise design guide, but a direct search on ImageNet is costly.
Revised: It is worth noting that performing a neural architecture search would yield a precise design guide, but a direct search on ImageNet is costly.

Original: We use the ImageNet dataset to provide a more convincing result of the investigation.
Revised: We use the ImageNet dataset to provide a more convincing result of the investigation.

Original: We also report the performance of the competitivethe max-pool operation without following BN and ReLU do not degrade accuracy. Mathematically, using ReLU after the max-pool operation (without BN) does not give nonlinearity due to the preceding ReLU. Empirically, involving BN after a parameter-free operation could stabilize the training initially, but model accuracy was not improved in the end.
Revised: We also report the performance of the competitive

Original: [Table]
Revised: [Table]

Original: architectures replacing the spatial operations (  the 3 × 3 convolutions) with 1) the  × convolutions
Revised: architectures replacing the spatial operations ( i e., the 3 × 3 convolutions) with 1) the 1 × 1 convolutions; 2) the 3 × 3 depthwise convolution in entire stages.

Original: ; 2) the 3 × 3 depthwise convolution in entire stages

Original: Table 2 shows the trade-off between accuracy and speed of the different models mixing the regular and the efﬁcient bottlenecks.
Revised: Table 2 shows the trade-off between accuracy and speed of the different models mixing the regular and the efﬁcient bottlenecks.

Original: B → B → B → B denotes the baseline ResNet50.
Revised: B → B → B → B denotes the baseline ResNet50; E / B denotes the model using the regular bottleneck and the efﬁcient block alternately.

Original: Among the different models, we highlight  E → E → B→ B and hybrid models due to the promising performance.
Revised: Among the different models, we highlight hybrid and E / B models due to the promising performance.

Original: This shows that only a simple module-based design regime can achieve improved performance over the baseline.
Revised: This shows that only a simple module-based design regime can achieve improved performance over the baseline.

Original: E → E → E → E is the model using the efﬁcient bottlenecks only and shows the fastest speed as we expected.
Revised: E → E → E → E is the model using the efﬁcient bottlenecks only and shows the fastest speed as we expected.

Original: The model with the 1 × 1 convolution replacements cannot reach the accuracy standard presumably due to the lack of the receptive ﬁeld; this is why a sufﬁcient number of spatial operations are required.
Revised: The model with the 1 ×convolution replacements cannot reach the accuracy standard presumably due to the lack of the receptive ﬁeld; this is why a sufﬁcient number of spatial operations are required.

Original: We observe that the model with the depthwise convolutions shows a promising performance, but our models surpass it as we similarly observed in §3.3.
Revised: We observe that the model with the depthwise convolutions shows a promising performance, but our models surpass it as we similarly observed in §3.3.

Original: All the models are trained on ImageNet with the standard 90-epochs training setup 4 (He et al., 2016a) to report the performance.
Revised: All the models are trained on ImageNet with the standard 90-epochs training setup 4 (He et al., 2016a) to report the performance.

Original: 4. E FFICIENT V I T A RCHITECTURES
Revised: 4. E FFICIENT V I T A RCHITECTURES

Original: We provide a further use case of the parameter-free operations in a totally different architecture from CNN.
Revised: We provide a further use case of the parameter-free operations in a totally different architecture from CNN.

Original: We again choose the max-pool operation as the parameter-free operation and apply it to replace the self-attention layer (Vaswani et al., 2017) in vision transformer (ViT)
Revised: We again choose the max-pool operation as the parameter-free operation and apply it to replace the self-attention layer (Vaswani et al., 2017) in vision transformer (ViT)

Original: (Dosovitskiy et al., 2021) to observe how the efﬁcient operation can replace a complicated layer.
Revised: (Dosovitskiy et al., 2021) to observe how the efﬁcient operation can replace a complicated layer.

Original: We do not change the MLP in a transformer block but replace the self-attention module with the max-pool operation.
Revised: We do not change the MLP in a transformer block but replace the self-attention module with the max-pool operation.

Original: Speciﬁcally, after the linear projection with the ratio of 3 ( i.e., identical to the  projection for query, key, and value in the self-attention layer in ViT), the projected 1d-feature is reshaped to 2d for the input of the spatial operation.
Revised: Speciﬁcally, after the linear projection with the ratio of 3 ( i.e., identical to the concatenation of the projection of query, key, and value in the self-attention layer in ViT), the projected 1d-features are reshaped to 2d-features for the input of the spatial operation.

Original: Consequently, the self-attention layer is replaced with a cheaper parameter-free operation and will bring clear efﬁciency.
Revised: Consequently, the self-attention layer is replaced with a cheaper parameter-free operation and will bring clear efﬁciency.

Original: We use the global average pooling (GAP) layer instead of using the classiﬁcation token of ViT since the classiﬁcation token is hardly available without the self-attention layer.
Revised: We use the global average pooling (GAP) layer instead of using the classiﬁcation token of ViT since the classiﬁcation token is hardly available without the self-attention layer.

Original: Many transformer-based ViTs can be a baseline; we additionally adopt a strong baseline Pooling-based Vision Transformer (PiT) (Heo et al., 2021b) to show applicability.
Revised: Many transformer-based ViTs can be a baseline; we additionally adopt a strong baseline Pooling-based Vision Transformer (PiT) (Heo et al., 2021b) to show applicability.

Original: More details are elaborated in Appendix A.
Revised: More details are elaborated in Appendix A.

Original:  E XPERIMENTS 5. I MAGE N ET C LASSIFICATION
Revised:  E XPERIMENTS 5. I MAGE N ET C LASSIFICATION

Original: Efﬁcient ResNets.
Revised: Efﬁcient ResNets.

Original: We perform ImageNet (Russakovsky et al., 2015) trainings to validate the model performance.
Revised: We perform ImageNet (Russakovsky et al., 2015) trainings to validate the model performance.

Original: We adopt the standard architecture ResNet

Revised: We adopt the standard architecture ResNet

Original: (He et al., 2016b) as the baseline and train our models with the aforementioned standard 90-epochs training setting to fairly compare with the competitors (Luo et al., 2017; Huang & Wang, 2018; Wu et al., 2018; Wang et al., 2018;
Revised: (He et al., 2016b) as the baseline and train our models with the aforementioned standard 90-epochs training setting to fairly compare with the competitors (Luo et al., 2017; Huang & Wang, 2018; Wu et al., 2018; Wang et al., 2018;

Original: Han et al., 2020; Yu et al., 2019; Qiu et al., 2021), where the efﬁcient operators were proposed or the networks were pruned.
Revised: Han et al., 2020; Yu et al., 2019; Qiu et al., 2021), where the efﬁcient operators were proposed or the networks were pruned.

Original: We report each averaged speed of the models with the publicly released codes on a V100 GPU.
Revised: We report each averaged speed of the models with the publicly released codes on a V100 GPU.

Original: Table 3 shows our networks acheieve faster inference speeds than those of the competitors with the comparable accuracies.
Revised: Table 3 shows our networks acheieve faster inference speeds than those of the competitors with the comparable accuracies.

Original: The channel-width pruned models (Slimmable-R50 0.5 × , 0.75 × ) and the models with new operations (Veratile-R50, GhostNet-R50, and SlimConv-R50) can-
Revised: The channel-width pruned models (Slimmable-R50 0.5 × , 0.75 × ) and the models with new operations (Veratile-R50, GhostNet-R50, and SlimConv-R50) can-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: not reach the model speed to ours.
Revised: not reach the model speed to ours.

Original: Additionally, we report the improved model accuracy with further training tricks (see the details in Appendix C).
Revised: Additionally, we report the improved model accuracy with further training tricks (see the details in Appendix C).

Original: This aims to show the maximal capacity of our model even using many parameter-free operations inside.
Revised: This aims to show the maximal capacity of our model even using many parameter-free operations inside.

Original: The last rows in the table present our model can follow the baseline accuracy well, and we see the gap between the baseline and ours has diminished; this show a potential of using parameter-free operations for a network design.
Revised: The last rows in the table present our model can follow the baseline accuracy well, and we see the gap between the baseline and ours has diminished [Table]

Original: Bigger Network Architectures.
Revised: Bigger Network Architectures.

Original: Our design regime is applied to complicated network architectures such as ResNet50-SE (Hu et al., 2017) and Wide ResNet-101-2 (WRN101-2) (Zagoruyko & Komodakis, 2016).
Revised: Our design regime is applied to complicated network architectures such as ResNet50-SE (Hu et al., 2017) and Wide ResNet-101-2 (WRN101-2) (Zagoruyko & Komodakis, 2016).

Original: We report the ImageNet performance plus mean Corruption Error (mCE) onImageNet-C
Revised: We report the ImageNet performance plus mean Corruption Error (mCE) on

Original:  (Hendrycks & Dietterich, 2019) and area under the precision-recall curve (AUC) the measure of out-of-distribution detection performance on ImageNet-O (Hendrycks et al., 2021).
Revised: ImageNet-C (Hendrycks & Dietterich, 2019) and Area Under the precision-recall Curve (AUC) the measure of out-of-distribution detection performance on ImageNet-O (Hendrycks et al., 2021).

Original: Table 4 indicates the models redesigned with the parameter-free operations work well; bigger models are more signiﬁcantly compressed in the computational costs.
Revised: Table 4 indicates the models redesigned with the parameter-free operations work well; bigger models are more signiﬁcantly compressed in the computational costs.

Original: Additionally, we notice all the mCEs and AUC of our models are remarkably improved, overtaking the accuracy degradation, which means the models with parameter-free operations suffer less from overﬁtting.
Revised: Additionally, we notice all the mCEs and AUC of our models are remarkably improved, overtaking the accuracy degradation, which means the models with parameter-free operations suffer less from overﬁtting.

Original: Deformable Max-pool Operation.
Revised: Deformable Max-pool Operation.

Original: We manifest a future direction of utilizing a parameter-free operation.
Revised: We manifest a future direction of utilizing a parameter-free operation.

Original: We borrow a similar idea of the deformable convolution 5 (Dai et al.,  2017).
Revised: We borrow a similar idea of the deformable convolution 5 (Dai et al., 2017; Zhou & Feng, 2017).

Original: Speciﬁcally, we involve a convolution to interpolate the features predicted by itself to perform a parameter-free operation on.
Revised: Speciﬁcally, we involve a convolution to interpolate the features predicted by itself to perform a parameter-free operation on.

Original: The max-pool operation still covers the spatial operation.
Revised: The max-pool operation still covers the spatial operation; only the offset predictor has weights to predict the locations.

Original: We regard this operator as to how the performance of the max-pool operation can be stretched when involving few numbers of parameters.
Revised: We regard this operator as to how the performance of the max-pool operation can be stretched when involving few numbers of parameters.

Original: Only performing computation on predicted locations can improve the accuracy over the 
Revised: Only performing computation on predicted locations can improve the accuracy over the vanilla computa-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: vanilla computation with a few extra costs, as shown in Table 3.
Revised: tion with a few extra costs, as shown in Table 3.

Original: Note that there is room for faster operation speed as the implementation can be further optimized.
Revised: Note that there is room for faster operation speed as the implementation can be further optimized.

Original: 5. COCO O BJECT D ETECTION
Revised: 5. COCO O BJECT D ETECTION

Original: We verify the transferability of our efﬁcient backbones on the COCO2017 dataset (Lin et al., 2014).
Revised: We verify the transferability of our efﬁcient backbones on the COCO2017 dataset (Lin et al., 2014).

Original: We adopt the standard Faster RCNN (Ren et al., 2015) with FPN (Lin et al., 2017) without any bells and whistles to ﬁnetune the backbones following the original training settings (Ren et al., 2015; Lin et al., 2017)
Revised: We adopt the standard Faster RCNN (Ren et al., 2015) with FPN (Lin et al., 2017) without any bells and whistles to ﬁnetune the backbones following the original training settings (Ren et al., 2015; Lin et al., 2017)

Original: Table 5 shows our model achieves a better trade-off between the AP scores and the computational costs, including the model speed, and does not signiﬁcantly degrade the AP scores even inside massive parameter-free operations. 
Revised: Table 5 shows our models achieve a better trade-off between the AP scores and the computational costs, including the model speed, and does not signiﬁcantly degrade the AP scores even inside massive parameter-free operations. We further validate the backbone with the deformable max-pool operations in Table 5.

Revised: Strikingly, AP scores are improved over ResNet50 even with the low ImageNet accuracy; this shows the effectiveness of the operation in a localization task.

Original: 5. I MAGE N ET C LASSIFICATION WITH E FFICIENT V ISION T RANSFORMERS
Revised: 5. I MAGE N ET C LASSIFICATION WITH E FFICIENT V ISION T RANSFORMERS

Original: We demonstrate using parameter-free operations in ViT (Dosovitskiy et al., 2021) in a novel way.
Revised: We demonstrate using parameter-free operations in ViT (Dosovitskiy et al., 2021) in a novel way.

Original: We follow the aforementioned architectural modiﬁcations.
Revised: We follow the aforementioned architectural modiﬁcations.

Original: Two vision transformer models ViTS and PiT-S (Heo et al., 2021b) are trained on ImageNet with three different training settings:
Revised: Two vision transformer models ViTS and PiT-S (Heo et al., 2021b) are trained on ImageNet with three different training settings:

Original: Vanilla (Dosovitskiy et al., 2021), with CutMix (Yun et al., 2019) and DeiT (Touvron et al., 2021) settings in Heo et al.
Revised: Vanilla (Dosovitskiy et al., 2021), with CutMix (Yun et al., 2019) and DeiT (Touvron et al., 2021) settings in Heo et al.

Original: (2021b).
Revised: (2021b).

Original: We similarly apply the depthwise convolution into the self-attention layer, which can be a strong competitor to make a comparison.
Revised: We similarly apply the depthwise convolution into the self-attention layer, which can be a strong competitor to make a comparison.

Original: We report the performance of the models in Table 6; we report throughput (images/sec) as the speed measure following the ViT papers (Dosovitskiy et al., 2021; Heo et al., 2021b), which is measured on 256 batch-size and a single batch-size both.
Revised: We report the performance of the models in Table 6; we report throughput (images/sec) as the speed measure following the ViT papers (Dosovitskiy et al., 2021; Heo et al., 2021b), which is measured on 256 batch-size and a single batch-size both.

Original: The result demonstrates that ViT and PiT with the max-pool operation have faster throughput without signiﬁcant accuracy degradation compared with the baselines; the depthwise convolution is a promising alternative, but the throughput is a matter compared with the max-pool operation for both architectures.
Revised: The result demonstrates that ViT and PiT with the max-pool operation have faster throughput without signiﬁcant accuracy degradation compared with the baselines; the depthwise convolution is a promising alternative, but the throughput is a matter compared with the max-pool operation for both architectures.

Original: Interestingly, PiT takes advantage of using the parameter-free operation, which presumably comes from larger features in early layers.
Revised: Interestingly, PiT takes advantage of using the parameter-free operation, which presumably comes from larger features in early layers.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we rethink parameter-free operations as the building block of learning spatial information to explore a novel way of designing network architecture.
Revised: In this paper, we rethink parameter-free operations as the building block of learning spatial information to explore a novel way of designing network architecture.

Original: We have experimentally studied the applicability of the parameter-free operations in network design and rebuild network architectures, including convolutional neural networks and vision transformers, towards more efﬁcient ones.
Revised: We have experimentally studied the applicability of the parameter-free operations in network design and rebuild network architectures, including convolutional neural networks and vision transformers, towards more efﬁcient ones.

Original: Extensive results on a large-scale dataset  ImageNet  with diverse network architectures have demonstrated the effectiveness over the existing efﬁcient architectures and the use case of the parameterfree operations as the main building block.
Revised: Extensive results on a large-scale dataset including ImageNet and COCO with diverse network architectures have demonstrated the effectiveness over the existing efﬁcient architectures and the use case of the parameter-free operations as the main building block.

Original: We believe our work highlighted a new design paradigm for future research beyond conventional efﬁcient architecture designs.
Revised: We believe our work highlighted a new design paradigm for future research beyond conventional efﬁcient architecture designs.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This paper studies a general topic in computer vision which is designing an efﬁcient network architecture.
Revised: This paper studies a general topic in computer vision which is designing an efﬁcient network architecture.

Original: Therefore, our work does not be expected to have any potential negative social impact but would contribute to the computer vision ﬁeld by providing pretrained models.
Revised: Therefore, our work does not be expected to have any potential negative social impact but would contribute to the computer vision ﬁeld by providing pretrained models.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We provide detailed information of all the experiments in the paper.
Revised: We provide detailed information of all the experiments in the paper.

Original: Furthermore, the details of our models with speciﬁc training hyper-paramaters are clearly announced for those who would like to design or train our proposed models
Revised: Furthermore, the details of our models with speciﬁc training hyper-paramaters are clearly announced for those who would like to design or train our proposed models

Original: A PPENDIX A D ETAILS OF E FFICIENT A RCHITECTURES
Revised: A PPENDIX A D ETAILS OF E FFICIENT A RCHITECTURES

Original: We elaborate on the efﬁcient building blocks used in the experiments above.
Revised: We elaborate on the efﬁcient building blocks used in the experiments above.

Original: Fig.
Revised: Fig.

Original: A.1 shows the schematic illustration of the proposed blocks compared with the original ones.
Revised: A.1 shows the schematic illustration of the proposed blocks compared with the original ones.

Original: We observe that the modiﬁcation is simple and readily be applied to any network architecture.
Revised: We observe that the modiﬁcation is simple and readily be applied to any network architecture.

Original: E ff - l a y e r N o r m
Revised: E ff - l a y e r N o r m

Original: N o r m
Revised: N o r m

Original: M L P + +
Revised: M L P + +

Original: N o r m + R e L U
Revised: N o r m + R e L U

Original: C o n v 1 x
Revised: C o n v 1 x

Original: C o n v 1 x
Revised: C o n v 1 x

Original: N o r m + E ff - l a y e r + + + E ff - l a y e r
Revised: N o r m + E ff - l a y e r + + + E ff - l a y e r

Original: E ff - l a y e r
Revised: E ff - l a y e r

Original: C
Revised: C

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: C
Revised: C

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: x
Revised: x

Original: N
Revised: N

Original: [Table]
Revised: [Table]

Original: r m +
Revised: r m +

Original: R e
Revised: R e

Original: [Table]
Revised: [Table]

Original: N
Revised: N

Original: [Table]
Revised: [Table]

Original: r m
Revised: r m

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: m
Revised: m

Original: N
Revised: N

Original: [Table]
Revised: [Table]

Original: r m + + +
Revised: r m + + +

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Table]
Revised: [Table]

Original: x
Revised: x

Original: [Table]
Revised: [Table]

Original: l fa tt e n t i o n
Revised: l fa tt e n t i o n

Original: [Table]
Revised: [Table]

Original: x
Revised: x

Original: [Table]
Revised: [Table]

Original: x
Revised: x

Original: [Table]
Revised: [Table]

Original: N
Revised: N

Original: [Table]
Revised: [Table]

Original: m +
Revised: m +

Original: [Table]
Revised: [Table]

Original: E ff - l a y e r
Revised: E ff - l a y e r

Original: N
Revised: N

Original: [Table]
Revised: [Table]

Original: r m
Revised: r m

Original: [Table]
Revised: [Table]

Original: r m
Revised: r m

Original: M L P + +
Revised: M L P + +

Original: N o r m +
Revised: N o r m +

Original: [Table]
Revised: [Table]

Original: n v
Revised: n v

Original: [Table]
Revised: [Table]

Original: x
Revised: x

Original: N o r m +
Revised: N o r m +

Original: E ff
Revised: E ff

Original: [Table]
Revised: [Table]

Original: l a y e r + + +
Revised: l a y e r + + +

Original: E ff
Revised: E ff

Original: [Table]
Revised: [Table]

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: r
Revised: r

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: m
Revised: m

Original: [Table]
Revised: [Table]

Original: m + + +
Revised: m + + +

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Efﬁcient Bottleneck.
Revised: Efﬁcient Bottleneck.

Original: We replace the triplet of 3 × 3 convolution, BN (Ioffe & Szegedy, 2015), and ReLU (Nair & Hinton, 2010) in the regular bottleneck with a single spatial parameter-free operation (denoted as eff-layer in Fig.
Revised: We replace the triplet of 3 × 3 convolution, BN (Ioffe & Szegedy, 2015), and ReLU (Nair & Hinton, 2010) in the regular bottleneck with a single spatial parameter-free operation (denoted as eff-layer in Fig.

Original: A.1c).
Revised: A.1c).

Original: The fastest architecture in Table 2 and Table 3 fully replace the regular bottlenecks, including the downsampling blocks with the efﬁcient bottlenecks; therefore, the parameter-free operation (here the max-pool operation) plays a role of spatially aggregating the features by reducing the resolution.
Revised: The fastest architecture in Table 2 and Table 3 fully replace the regular bottlenecks, including the downsampling blocks with the efﬁcient bottlenecks; therefore, the parameter-free operation (here the max-pool operation) plays a role of spatially aggregating the features by reducing the resolution.

Original: The hybrid architecture has an almost identical design regime with the most efﬁcient architecture ( i.e., E → E → E → E in Table 2) except for the downsampling blocks.
Revised: The hybrid architecture has an almost identical design regime with the most efﬁcient architecture ( i.e., E → E → E → E in Table 2) except for the downsampling blocks.

Original: We remain each downsampling block with the regular bottleneck blocks and involve an efﬁcient parameter-free operation before the spatial operation (we assign the avg-pool as the parameter-free operation).
Revised: We remain each downsampling block with the regular bottleneck blocks and involve an efﬁcient parameter-free operation before the spatial operation (we assign the avg-pool as the parameter-free operation).

Original: We apply the identical bottleneck conﬁguration of the hybrid architecture to the diverse deep convolutional neural networks (CNNs) in Table 4.
Revised: We apply the identical bottleneck conﬁguration of the hybrid architecture to the diverse deep convolutional neural networks (CNNs) in Table 4.

Original: Note that we do not modify the depth and the width of network architecture, the stem of CNNs that is the ﬁrst set of layers before the ﬁrst bottleneck, and the output layer having a fully connected layer with 1000-d output dimension.
Revised: Note that we do not modify the depth and the width of network architecture, the stem of CNNs that is the ﬁrst set of layers before the ﬁrst bottleneck, and the output layer having a fully connected layer with 1000-d output dimension.

Original: Furthermore, we do not modify speciﬁc architectural elements such as the SE-block of ResNet50-SE (Hu et al., 2017) inside each regular bottleneck but use it in each efﬁcient bottleneck at the same positions.
Revised: Furthermore, we do not modify speciﬁc architectural elements such as the SE-block of ResNet50SE (Hu et al., 2017) inside each regular bottleneck but use it in each efﬁcient bottleneck at the same positions.

Original: Efﬁcient Transformer.
Revised: Efﬁcient Transformer.

Original: Designing the efﬁcient vision transformer (ViT) architecture is done by replacing half of the self-attention layers with the spatial parameter-free operation (denoted as efflayer in Fig.
Revised: Designing the efﬁcient vision transformer (ViT) architecture is done by replacing half of the self-attention layers with the spatial parameter-free operation (denoted as efflayer in Fig.

Original: A.1d).
Revised: A.1d).

Original: In other words, the original transformer block and the efﬁcient transformer block are used alternately.
Revised: In other words, the original transformer block and the efﬁcient transformer block are used alternately.

Original: We use the 3 × 3 max-pool operation for eff-layer in the efﬁcient transformer block and compare with the variant using the 5 × 5 depth-wise convolution for eff-layer.
Revised: We use the 3 × 3 max-pool operation for eff-layer in the efﬁcient transformer block and compare with the variant using the 5 × 5 depth-wise convolution for eff-layer.

Original: ReLU is added after eff-layer and the ﬁrst linear layer of the efﬁcient transformer block to give additional non-linearity.
Revised: ReLU is added after eff-layer and the ﬁrst linear layer of the efﬁcient transformer block to give additional non-linearity.

Original: For Pooling-based Vision Transformer (PiT) (Heo et al., 2021a) and ViT (Dosovitskiy et al., 2021), we do not modify the architectural elements including 1) the patch size; 2) the stem that patchiﬁes the input for the following transformer; 3) the number of transformers.
Revised: For Pooling-based Vision Transformer (PiT) (Heo et al., 2021a) and ViT (Dosovitskiy et al., 2021), we do not modify the architectural elements including 1) the patch size; 2) the stem that patchiﬁes the input for the following transformer; 3) the number of transformers.

Original: We only change the classiﬁcation head position from the classiﬁcation token to the Global Average Pooling (GAP) at the last layer since the classiﬁcation token is not compatible with convolutional operations.
Revised: We only change the classiﬁcation head position from the classiﬁcation token to the Global Average Pooling (GAP) at the last layer since the classiﬁcation token is not compatible with convolutional operations.

Original: As reported in Zhai et al.
Revised: As reported in Zhai et al.

Original: (2021), transformer with GAP shows comparable performance to ViT with the classiﬁcation token.
Revised: (2021), transformer with GAP shows comparable performance to ViT with the classiﬁcation token.

Original: We also evaluate a more efﬁcient network architecture which is fully equipped with the efﬁcient transformer; it achieves faster speed, but slightly less accurate.
Revised: We also evaluate a more efﬁcient network architecture which is fully equipped with the efﬁcient transformer; it achieves faster speed, but slightly less accurate.

Original: We leave further improvements with a parameter-free operation inside the efﬁcient transformer to reach the accuracy of the original transformer as future work.
Revised: We leave further improvements with a parameter-free operation inside the efﬁcient transformer to reach the accuracy of the original transformer as future work.

Original: [Figure]
Revised: [Figure]

Original: B O N P ARAMETER - FREE O PERATIONS
Revised: B O N P ARAMETER - FREE O PERATIONS

Original: Can We Use Avg-pool as the Spatial Parameter-free Operation?
Revised: Can We Use Avg-pool as the Spatial Parameter-free Operation?

Original: We mainly used the max-pool operation as the spatial parameter-free operation that replaces the regular convolutions and the selfattention layer in experiments.
Revised: We mainly used the max-pool operation as the spatial parameter-free operation that replaces the regular convolutions and the selfattention layer in experiments.

Original: The max-pool operation is expected to have higher expressiveness compared with that of the avg-pool operation.
Revised: The max-pool operation is expected to have higher expressiveness compared with that of the avg-pool operation.

Original: Because the avg-pool operation is a conceptually smoothing operation, and the max-pool operation contains a nonlinearity similar to what ReLU has ( i.e., max ( x ) and max ( x, 0) ).
Revised: Because the avg-pool operation is a conceptually smoothing operation, and the max-pool operation contains a nonlinearity similar to what ReLU has ( i.e., max ( x ) and max ( x, 0) ).

Original: Experimentally, we found low expressiveness of the avg-pool operation in the shallow network study.
Revised: Experimentally, we found low expressiveness of the avg-pool operation in the shallow network study.

Original: Fig.
Revised: Fig.

Original: B.1 shows the comparison of the max-pool and the avgpool operations trained in a single bottleneck block.
Revised: B.1 shows the comparison of the max-pool and the avgpool operations trained in a single bottleneck block.

Original: The setting is identical to the previous study in §3, and we only visualize the case of the channel width of 32 because a similar trend with the channel width of 64 was observed.
Revised: The setting is identical to the previous study in §3, and we only visualize the case of the channel width of 32 because a similar trend with the channel width of 64 was observed.

Original: We observe the large accuracy gaps between the two operations.
Revised: We observe the large accuracy gaps between the two operations.

Original: This may be a ground of the search results where few avg-pool operations are chosen in normal cells in the previous NAS experiments in §3.3. 
Revised: This may be a ground of the search results where few avg-pool operations are chosen in normal cells in the previous NAS experiments in §3.3. ㅇ

Original: Deformable Avg-pool Operation.
Revised: Deformable Avg-pool Operation.

Original: We here provide an interesting idea of using the avg-pool operation, which shows inferior outcomes than the max-pool operation in the operation expressiveness.
Revised: We here provide an interesting idea of using the avg-pool operation, which shows inferior outcomes than the max-pool operation in the operation expressiveness.

Original: We have proposed the deformable max-pool operation involving a few parameters to improve the parameter-free operation’s discriminative power signiﬁcantly.
Revised: We have proposed the deformable max-pool operation involving a few parameters to improve the parameter-free operation’s discriminative power signiﬁcantly.

Original: Using the same idea into the avg-pool operation is available; surprisingly, a downside of the low expressiveness of the avg-pool operation has been vanished, as shown in Table B.1.
Revised: Using the same idea into the avg-pool operation is available; surprisingly, a downside of the low expressiveness of the avg-pool operation has been vanished, as shown in Table B.1.

Original: This result exhibits that even a smoothing operation can be employed to learn discriminative features in a deep neural network.
Revised: This result exhibits that even a smoothing operation can be employed to learn discriminative features in a deep neural network.

Original: [Table]
Revised: [Table]

Original: C D ETAILED T RAINING R ECIPES
Revised: C D ETAILED T RAINING R ECIPES

Original: We use the following training recipes to maximize the model accuracy.
Revised: We use the following training recipes to maximize the model accuracy.

Original: We use the cosine learning rate scheduling (Loshchilov & Hutter, 2017a) with the initial learning rate of 0.5 using four V
Revised: We use the cosine learning rate scheduling (Loshchilov & Hutter, 2017a) with the initial learning rate of 0.5 using four V

Original: GPUs with a total batch size of 512.
Revised: GPUs with a total batch size of 512.

Original: Exponential moving average (Tarvainen & Valpola, 2017) over the network weights is used during training.
Revised: Exponential moving average (Tarvainen & Valpola, 2017) over the network weights is used during training.

Original: We use the regularization techniques and data augmentations including label smoothing (Szegedy et al., 2016) (0.1), RandAug (Cubuk et al., 2019)
Revised: We use the regularization techniques and data augmentations including label smoothing (Szegedy et al., 2016) (0.1), RandAug (Cubuk et al., 2019)

Original: (magnitude of 9),
Revised: (magnitude of 9),

Original: Random Erasing (Hermans et al., 2017) with pixels (0.2), lowered weight decay (1e-5), and a large training epochs (400 epochs).
Revised: Random Erasing (Hermans et al., 2017) with pixels (0.2), lowered weight decay (1e-5), and a large training epochs (400 epochs).

Original: We use the code baseline in the renowned repository 6 for our ImageNet training.
Revised: We use the code baseline in the renowned repository 6 for our ImageNet training.

Original: The accuracy of the baseline models could reach the known improved accuracy reported in such a paper (Bello et al., 2021) which presents the training recipes for highly improved models. 
Revised: The accuracy of the baseline models could reach the known improved accuracy reported in such a paper (Bello et al., 2021) which presents the training recipes for highly improved models. I npu t R e s N e t O u r s

Revised: D U NDERSTANDING M AX - POOL O PERATION

Revised: This section provides intuitive explanations how parameter-free operations such as the max-pool operation could work as replacing a trainable layers in a network.

Revised: D. C ONNECTION WITH M AXOUT

Revised: Maxout (Goodfellow et al., 2013) performs the max operation to a set of the inputs like an activation function with the multiple inputs.

Revised: Maxout is designed to use after a bunch of linear or convolution layers to ensemble the outputs in a nonlinear manner, which leads to the increase of model capacity.

Revised: The accuracy improvement in the original paper (Goodfellow et al., 2013) results from the claim that Maxout can approximate any operations, namely perform as a universal approximator.

Revised: From an architectural point of view, the success of Maxout is probably due to the enhanced model capacity by the increased input dimension with the nonlinearity imposed by the max operation.

Revised: For the max-pool operation with the efﬁcient bottleneck mainly used throughout the paper, the relationship between the Maxout operation and ours is worth discussing.

Revised: Maxout outputs through the max operation of the outputs of multiple linear layers computed from a single input.

Revised: Otherwise, the max-pool operation in the efﬁcient bottleneck performs the max operation with multiple transformed inputs (i.e., transformed pixels spatially adjacent to each point) by the preceding 1 × 1 convolution, which acts as a single linear layer for a given channel.

Revised: However, if we replace the small variations in neighboring pixels with the perturbations in the weights of the linear layer as claimed in Seong et al.

Revised: (2018), then we may interpret the max-pool operation performs like Maxout.

Revised: D. R ECEPTIVE F IELD M ATTER

Revised: Since the spatial parameter-free operations have the receptive ﬁeld (Luo et al., 2016) like convolutions, we conjecture such simple operations can capture striking pixels (or patches).

Revised: We empirically support the conjecture by providing a visualization of the ﬁnal output features of the ImageNetpretrained ResNet50 and our ResNet50 ( hybrid ) in Fig.

Revised: D.1.

Revised: We observe that two models successfully localize the foreground object in common for all the images.

Revised: A noticeable difference is that

Revised: ResNet50 tends to focus more on speciﬁc crucial regions but ResNet50 ( hybrid ) on wider regions (a similar trend will be observed in Fig.D.2).

Revised: Note that the area of the highlighted region does not directly link to a model’s accuracy (in fact, ResNet50 has higher accuracy).

Revised: We believe that this indicates how much a model learns localizable features and can be utilized to understand the learning dynamics of various models.

Revised: We further conjecture that a 1 × 1 convolution can complement the expressiveness of a parameter-free operation such as the max-pool operation or avg-pool operation by nonlinearly mixing the features, which is computed by the parameter-free operation.

Revised: To make a ground for the conjecture, we would like to visualize the intermediate features extracted from 1) each of two 1 × 1 convolutions; 2) the

Revised: [Figure]

Revised: e

Revised: [Figure]

Revised: e t

Revised: O u r s

Revised: R e s N e t

Revised: O u r s

Revised: R e s N e t

Revised: O u r s spatial operations including the 3 × 3 convolution and the max-pool operation in a bottleneck block.

Revised: We choose the last two bottlenecks in the ﬁnal stage conv5 and visualize the output features of the aforementioned layers.

Revised: Fig.

Revised: D.2 shows the highlighted features produced by the three different input images randomly picked from the ImageNet validation set.

Revised: We use the identical models employed to visualize the ﬁnal features shown in Fig.

Revised: D.1.

Revised: We let conv5 x y denote the output features of the y-th layer in the x-th bottleneck of a speciﬁc model; for example, each visualized feature of conv5 2 2 in ResNet50 and our model (ResNet50 ( hybrid ) indicates the output features of the 3 × 3 convolution and the spatial max-pool operation, respectively.

Revised: First, we observe the 1 × 1 convolutions reﬁne the features to be more discriminative; when comparing the features in conv5 x 2 with those of conv5 x 3, the output features usually get more highlighted on the crucial region in the foreground objects.

Revised: Furthermore, it seems that the 1 × 1 convolutions change more of the previous features in our model, which may come from the different output features by the 3 × 3 convolutions and the max-pool operations (compare the output features of ResNet50 with ours in conv5 x 2).

Revised: Therefore, based on the observations, the 1 × 1 convolution performs to make features more discriminative, and they do more with the parameter-free operations like the max-pool operation.

Revised: E M ORE V ISUALIZATIONS

Revised: In this section, we further discuss the studied materials with the additional graphs and ﬁgures to provide more detailed information.

Revised: §E.1 contains extended trade-off graphs of our study in §3.3; §E.2 discusses predicted offsets by the proposed deformable max-pool operation; ﬁnally in §E.3, we discuss the performance trade-offs of the models on the ImageNet datasets in §5.1.

Revised: [Figure]

Revised: (d)

Revised: [Figure]

Revised: Fig.

Revised: E.1 shows the performance of the entire models which are trained in §3.3.

Revised: The graphs includes the 20% best-performing models shown in Fig.2.

Revised: As expected, the models using many max-pool operations have degraded accuracies but show clear speed beneﬁts; there is about a 1.5% accuracy gap between the fastest models using each operation yet more than a 5ms speed gap between them.

Revised: E. P REDICTED O FFSETS BY D EFORMABLE M AX - POOL O PERATION

Revised: We visualize the predicted offsets by the proposed deformable max-pool operation (deform max) with the sample images in Fig.

Revised: E.2.

Revised: We plot the aggregated offsets of the last two deform maxs in the ﬁnal stage of ResNet50 with the initial points, which look like the grid-like offset points of the two successive regular convolutions.

Revised: We observe the offsets are concentrated on the objects when the center is on each foreground object.

Revised: The offsets spread widely when the center is on the background, which is similarly observed in the deformable convolution paper (Dai et al., 2017; Zhu et al., 2019).

Revised: It is surprising that our models only trained with the ImageNet’s class labels without strong supervisions (i.e., detection boxes or segmentation masks) show the improved localization capability.

Revised: Furthermore, albeit our model does not be trained with the background class, each shape of the predicted offsets on foreground and background looks quite different as shown in Fig.

Revised: E.2.

Revised: We would like to stress the main differences of the experimental settings with those in the deformable convolution papers.

Revised: First, our model is trained on ImageNet only with the class labels without any strong supervision so that the models may have a weaker localization capability than the models trained with more supervision, such as on detection and segmentation tasks.

Revised: Second, our model incorporates much more deformable operations in the entire stages of a ResNet, so the dynamics of predicting offsets would be different from the deformable convolution where the only later stages have the deformable convolutions.

Revised: E. P ERFORMANCE T RADE - OFFS

Revised: This work presents the potential usefulness of parameter-free operations such as the max-pool as a building block rather than pushing the performance to the limit.

Revised: We ﬁrst visualize the ImageNet performance comparison with many efﬁcient models shown in Table 3 in Fig.

Revised: E.3.

Revised: Fig.

Revised: E.3 shows our models have competitive trade-offs between accuracy and speed.

Revised: Since we do not modify ResNet but just replace the spatial operation into parameter-free operations such as the max-pool, our models do not have much beneﬁts in the number of parameters and FLOPs.

Revised: Moreover, we visualize the ImageNet results including 1) top-1 accuracy on ImageNet; 2) mean Corruption Error (mCE) on ImageNet-C; 3) Area Under the precision-recall Curve (AUC) on ImageNetO applying the efﬁcient bottleneck into existing big CNNs, which is shown in Table 4, in Fig.

Revised: E.4.

Revised: Fig.

Revised: E.4 shows our models achieve large improvements on the entire computational costs but show similar trade-offs of the baseline models in top-1 accuracy.

Revised: This experiment originally aims to investigate the redundancy of using standard building blocks inside heavy CNN models.

Revised: However, a simple replacement of the operations in existing models achieves signiﬁcant efﬁciency; namely, in terms of the model speeds, the number of parameters, and FLOPs, our models reduce meaningful amounts without much accuracy loss.

Revised: Furthermore, our models signiﬁcantly outperform the baseline models with efﬁciency in mCE and AUC measures.

Revised: Improving the ImageNet top-1 accuracy while having the advantages of the parameters-free operations will be our future work.

Revised: [Figure]

Revised: [Figure]