Original: R ETHINKING C ONVOLUTION : T OWARDS AN O PTIMAL E FFICIENCY
Revised: R ETHINKING C ONVOLUTION : T OWARDS AN O PTIMAL E FFICIENCY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we present our recent research about the computational efﬁciency in convolution.
Revised: In this paper, we present our recent research about the computational efﬁciency in convolution.

Original: Convolution operation is the most critical component in recent surge of deep learning research.
Revised: Convolution operation is the most critical component in recent surge of deep learning research.

Original: Conventional 2D convolution takes O ( C 2 K 2 HW ) to calculate, where C is the channel size, K is the kernel size, while H and W are the output height and width.
Revised: Conventional 2D convolution takes O ( C 2 K 2 HW ) to calculate, where C is the channel size, K is the kernel size, while H and W are the output height and width.

Original: Such computation has become really costly considering that these parameters increased over the past few years to meet the needs of demanding applications.
Revised: Such computation has become really costly considering that these parameters increased over the past few years to meet the needs of demanding applications.

Original: Among various implementations of the convolution, separable convolution has been proven to be more efﬁcient in reducing the computational demand.
Revised: Among various implementations of the convolution, separable convolution has been proven to be more efﬁcient in reducing the computational demand.

Original: For example, depth separable convolution reduces the complexity to O ( CHW · ( C + K 2 )) while spatial separable convolution reduces the complexity to O
Revised: For example, depth separable convolution reduces the complexity to O ( CHW · ( C + K 2 )) while spatial separable convolution reduces the complexity to O

Original: ( C 2 KHW ) .
Revised: ( C 2 KHW ) .

Original: However, these are considered an ad hoc design which cannot ensure that they can in general achieve optimal separation.
Revised: However, these are considered an ad hoc design which cannot ensure that they can in general achieve optimal separation.

Original: In this research, we propose a novel operator called optimal separable convolution which can be calculated at O ( C 32 KHW ) by optimal design for the internal number of groups and kernel sizes for general separable convolutions.
Revised: In this research, we propose a novel operator called optimal separable convolution which can be calculated at O ( C 32 KHW ) by optimal design for the internal number of groups and kernel sizes for general separable convolutions.

Original: When there is no restriction in the number of separated convolutions, an even lower complexity at
Revised: When there is no restriction in the number of separated convolutions, an even lower complexity at

Original: O ( CHW · log( CK 2 )) can be achieved.
Revised: O ( CHW · log( CK 2 )) can be achieved.

Original: Experimental results demonstrate that the proposed optimal separable convolution is able to achieve an improved accuracyFLOPs and accuracy-#Params trade-offs over both conventional and depth/spatial separable convolutions.
Revised: Experimental results demonstrate that the proposed optimal separable convolution is able to achieve an improved accuracyFLOPs and accuracy-#Params trade-offs over both conventional and depth/spatial separable convolutions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Tremendous progresses have been made in recent years towards more accurate image analysis tasks, such as image classiﬁcation, with deep convolutional neural networks (DCNNs) (Krizhevsky et al., 2012; Srivastava et al., 2015;
Revised: Tremendous progresses have been made in recent years towards more accurate image analysis tasks, such as image classiﬁcation, with deep convolutional neural networks (DCNNs) (Krizhevsky et al., 2012; Srivastava et al., 2015;

Original: He et al., 2016; Real et al., 2019; Tan & Le, 2019; Dai et al., 2020).
Revised: He et al., 2016; Real et al., 2019; Tan & Le, 2019; Dai et al., 2020).

Original: However, the computational complexity for state-of-the-art DCNN models has also become increasingly high and computationally expensive.
Revised: However, the computational complexity for state-of-the-art DCNN models has also become increasingly high and computationally expensive.

Original: This can signiﬁcantly defer their deployment to real-world applications, such as mobile platforms and robotics, where the resources are highly constrained (Howard et al., 2017; Dai et al., 2020).
Revised: This can signiﬁcantly defer their deployment to real-world applications, such as mobile platforms and robotics, where the resources are highly constrained (Howard et al., 2017; Dai et al., 2020).

Original: It is very much desired that a DCNN could achieve better performance with less computation and fewer model parameters.
Revised: It is very much desired that a DCNN could achieve better performance with less computation and fewer model parameters.

Original: The most time-consuming building block of a DCNN is the convolutional layer.
Revised: The most time-consuming building block of a DCNN is the convolutional layer.

Original: There have been many previous works aiming at reducing the amount of computation in the convolutional layer.
Revised: There have been many previous works aiming at reducing the amount of computation in the convolutional layer.

Original: Historically, researchers apply Fast Fourier Transform (FFT) (Nussbaumer, 1981; Quarteroni et al., 2010) to implement convolution and they gain great speed up for large convolutional kernels.
Revised: Historically, researchers apply Fast Fourier Transform (FFT) (Nussbaumer, 1981; Quarteroni et al., 2010) to implement convolution and they gain great speed up for large convolutional kernels.

Original: For small convolutional kernels, a direct application is often still cheaper (Podlozhnyuk, 2007).
Revised: For small convolutional kernels, a direct application is often still cheaper (Podlozhnyuk, 2007).

Original: Researchers also explore low rank approximation (Jaderberg et al., 2014; Ioannou et al., 2015) to implement convolutions.
Revised: Researchers also explore low rank approximation (Jaderberg et al., 2014; Ioannou et al., 2015) to implement convolutions.

Original: However, most of the existing methods start from a pre-trained model and mainly focus on network pruning and compression.
Revised: However, most of the existing methods start from a pre-trained model and mainly focus on network pruning and compression.

Original: In this research, we study how to design a separable convolution to achieve an optimal implementation in terms of computational complexity.
Revised: In this research, we study how to design a separable convolution to achieve an optimal implementation in terms of computational complexity.

Original: Enabling convolution separable has been proven to be an efﬁcient way to reduce the computational complexity (Sifre & Mallat, 2014;
Revised: Enabling convolution separable has been proven to be an efﬁcient way to reduce the computational complexity (Sifre & Mallat, 2014;

Original: Howard et al., 2017; Szegedy et al., 2016).
Revised: Howard et al., 2017; Szegedy et al., 2016).

Original: Comparing to the FFT and low rank approximation approaches, a welldesigned separable convolution shall be efﬁcient for both small and large kernel sizes and shall not require a pre-trained model to operate on.
Revised: Comparing to the FFT and low rank approximation approaches, a welldesigned separable convolution shall be efﬁcient for both small and large kernel sizes and shall not require a pre-trained model to operate on.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: In the DCNN research, two most well-known separable convolutions are depth separable (Sifre &
Revised: In the DCNN research, two most well-known separable convolutions are depth separable (Sifre &

Original: Mallat, 2014) and spatial separable (Szegedy et al., 2016) convolutions.
Revised: Mallat, 2014) and spatial separable (Szegedy et al., 2016) convolutions.

Original: Both are able to reduce the computational complexity of a convolution.
Revised: Both are able to reduce the computational complexity of a convolution.

Original: The complexity of a conventional 2D convolution is quadratic with three hyper-parameters: number of channels ( C ), kernel size ( K ), and spatial dimensions ( H or W ), and its computational complexity is actually O ( C 2 K 2 HW ) .
Revised: The complexity of a conventional 2D convolution is quadratic with three hyper-parameters: number of channels ( C ), kernel size ( K ), and spatial dimensions ( H or W ), and its computational complexity is actually O ( C 2 K 2 HW ) .

Original: Depth separable convolution is constructed as a depth-wise convolution followed by a point-wise convolution, where depth-wise convolution is a group convolution with its number of groups g = C and point-wise convolution is a 1 × 1 convolution.
Revised: Depth separable convolution is constructed as a depth-wise convolution followed by a point-wise convolution, where depth-wise convolution is a group convolution with its number of groups g = C and point-wise convolution is a 1 × 1 convolution.

Original: Spatial separable convolution replaces a K × K kernel to K ×and 1 × K kernels.
Revised: Spatial separable convolution replaces a K × K kernel to K ×and 1 × K kernels.

Original: Different types of convolutions and their computational costs are summarized in Table 1.
Revised: Different types of convolutions and their computational costs are summarized in Table 1.

Original: From this table, we can easily verify that depth separable convolution has a complexity of O ( CHW · ( C + K 2 )) and spatial separable convolution has a complexity of O ( C 2 KHW ) .
Revised: From this table, we can easily verify that depth separable convolution has a complexity of O ( CHW · ( C + K 2 )) and spatial separable convolution has a complexity of O ( C 2 KHW ) .

Original: Both depth separable and spatial separable convolutions follow an ad hoc design.
Revised: Both depth separable and spatial separable convolutions follow an ad hoc design.

Original: They are able to reduce the computational cost to some degree but normally will not achieve an optimal separation.
Revised: They are able to reduce the computational cost to some degree but normally will not achieve an optimal separation.

Original: A separable convolution in general has three sets of parameters: the internal number of groups, channel size, and kernel size of each separated convolution.
Revised: A separable convolution in general has three sets of parameters: the internal number of groups, channel size, and kernel size of each separated convolution.

Original: Instead of setting these parameters in an ad hoc fashion, we design a scheme to achieve an optimal separation.
Revised: Instead of setting these parameters in an ad hoc fashion, we design a scheme to achieve an optimal separation.

Original: The resulting separable convolution is called optimal separable convolution in this research.
Revised: The resulting separable convolution is called optimal separable convolution in this research.

Original: To prevent the proposed optimal separable convolution from being degenerated, we assume that the internal channel size is in an order of O ( C ) and propose the following volumetric receptive ﬁeld condition .
Revised: To prevent the proposed optimal separable convolution from being degenerated, we assume that the internal channel size is in an order of O ( C ) and propose the following volumetric receptive ﬁeld condition .

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 1a, similar to the receptive ﬁeld (RF) of a convolution which is deﬁned as the region in the input space that a particular CNN’s feature is looking at (or affected by) (Lindeberg, 2013), we deﬁne the volumetric RF of a convolution to be the volume in the input space that affects CNN’s output.
Revised: 1a, similar to the receptive ﬁeld (RF) of a convolution which is deﬁned as the region in the input space that a particular CNN’s feature is looking at (or affected by) (Lindeberg, 2013), we deﬁne the volumetric RF of a convolution to be the volume in the input space that affects CNN’s output.

Original: The volumetric RF condition requires that a properly decomposed separable convolution maintains the same volumetric RF as the original convolution before decomposition.
Revised: The volumetric RF condition requires that a properly decomposed separable convolution maintains the same volumetric RF as the original convolution before decomposition.

Original: Hence, the proposed optimal separable convolution will be equivalent to optimizing the internal number of groups and kernel sizes to achieve the computational objective (measured in
Revised: Hence, the proposed optimal separable convolution will be equivalent to optimizing the internal number of groups and kernel sizes to achieve the computational objective (measured in

Original: FLOPs 1 ) while satisfying the volumetric RF condition.
Revised: FLOPs 1 ) while satisfying the volumetric RF condition.

Original: Formally, the objective function is deﬁned by Equation (2) under the constraints deﬁned by Equations (3)-(6).
Revised: Formally, the objective function is deﬁned by Equation (2) under the constraints deﬁned by Equations (3)-(6).

Original: The solution to this optimization problem will be described in detail in Section 2.
Revised: The solution to this optimization problem will be described in detail in Section 2.

Original: We shall show that the proposed optimal separable convolution can be calculated at the order of
Revised: We shall show that the proposed optimal separable convolution can be calculated at the order of

Original: O ( C 32 KHW ) .
Revised: O ( C 32 KHW ) .

Original: This is at least a factor of √ C more efﬁcient than the depth separable and spatial separable convolutions.
Revised: This is at least a factor of √ C more efﬁcient than the depth separable and spatial separable convolutions.

Original: The proposed optimal separable convolution is able to be easily generalized into an N -separable case, where the number of separated convolutions N can be optimized further.
Revised: The proposed optimal separable convolution is able to be easily generalized into an N -separable case, where the number of separated convolutions N can be optimized further.

Original: In such a generalized case, an even lower complexity at O ( CHW · log( CK 2 )) can be achieved.
Revised: In such a generalized case, an even lower complexity at O ( CHW · log( CK 2 )) can be achieved.

Original: Extensive experiments are carried out to demonstrate the effectiveness of the proposed optimal separable convolution.
Revised: Extensive experiments are carried out to demonstrate the effectiveness of the proposed optimal separable convolution.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 3, on the CIFAR10 dataset (Krizhevsky et al., 2009), the proposed optimal separable convolution achieves a better Pareto-frontier 2 than both conventional and depth separable convolutions using the ResNet (He et al., 2016) architecture.
Revised: 3, on the CIFAR10 dataset (Krizhevsky et al., 2009), the proposed optimal separable convolution achieves a better Pareto-frontier 2 than both conventional and depth separable convolutions using the ResNet (He et al., 2016) architecture.

Original: To demonstrate that the proposed optimal separable convolution generalizes well to other DCNN architectures, we adopt the DARTS (Liu et al., 2018) architecture by replacing the depth separable convolution with the proposed optimal separable convolution.
Revised: To demonstrate that the proposed optimal separable convolution generalizes well to other DCNN architectures, we adopt the DARTS (Liu et al., 2018) architecture by replacing the depth separable convolution with the proposed optimal separable convolution.

Original: The accuracy is improved from 97.24% to 97.67% with fewer parameters.
Revised: The accuracy is improved from 97.24% to 97.67% with fewer parameters.

Original: On the ImageNet dataset (Deng et al., 2009), the proposed optimal separable convolution also achieves an improved performance.
Revised: On the ImageNet dataset (Deng et al., 2009), the proposed optimal separable convolution also achieves an improved performance.

Original: For the DARTS architecture, the proposed approach achieves 74.2% top1 accuracy with only 4.5 million parameters.
Revised: For the DARTS architecture, the proposed approach achieves 74.2% top1 accuracy with only 4.5 million parameters.

Original:  T HE P ROPOSED A PPROACH 2. C ONVOLUTION AND ITS C OMPUTATIONAL C OMPLEXITY
Revised:  T HE P ROPOSED A PPROACH 2. C ONVOLUTION AND ITS C OMPUTATIONAL C OMPLEXITY

Original: A convolutional layer takes an input tensor B l − 1 of shape ( C l − 1 , H l − 1 , W l − 1 ) and produces an output tensor B l of shape ( C l , H l , W l ) , where C ∗ , H ∗ , W ∗ are input and output channels, feature heights and widths.
Revised: A convolutional layer takes an input tensor B l − 1 of shape ( C l − 1 , H l − 1 , W l − 1 ) and produces an output tensor B l of shape ( C l , H l , W l ) , where C ∗ , H ∗ , W ∗ are input and output channels, feature heights and widths.

Original: The convolutional layer is parameterized with a convolutional kernel of shape ( C l , C l − 1 , K Hl , K Wl ) , where K ∗ l are the kernel sizes, and the superscript indicates whether it is aligned with the features in height or width.
Revised: The convolutional layer is parameterized with a convolutional kernel of shape ( C l , C l − 1 , K Hl , K Wl ) , where K ∗ l are the kernel sizes, and the superscript indicates whether it is aligned with the features in height or width.

Original: In this research, we take C ∗ =
Revised: In this research, we take C ∗ =

Original: O ( C ) , H ∗ =
Revised: O ( C ) , H ∗ =

Original: O ( H ) ,
Revised: O ( H ) ,

Original: W ∗ =
Revised: W ∗ =

Original: O ( W ) , and K H | W ∗
Revised: O ( W ) , and K H | W ∗

Original: =
Revised: =

Original: O ( K ) for complexity analysis.
Revised: O ( K ) for complexity analysis.

Original: Formally, we have
Revised: Formally, we have

Original: [Equation]
Revised: [Equation]

Original: where h l = h l − 1 + k Hl and w l = w l − 1 + k Wl .
Revised: where h l = h l − 1 + k Hl and w l = w l − 1 + k Wl .

Original: Hence, the number of FLOPs (multiply-adds) for convolution is C l H l W l · C l −
Revised: Hence, the number of FLOPs (multiply-adds) for convolution is C l H l W l · C l −

Original: 1 K
Revised: 1 K

Original: Hl K Wl
Revised: Hl K Wl

Original: = O ( C 2 K 2 HW ) and the number of parameters is
Revised: = O ( C 2 K 2 HW ) and the number of parameters is

Original: C l C l − 1 K Hl K Wl
Revised: C l C l − 1 K Hl K Wl

Original: = O ( C 2 K 2 ) .
Revised: = O ( C 2 K 2 ) .

Original: For a group convolution, we have g convolutions with kernels of shape ( C l /g, C l − 1 /g, K Hl , K Wl ) .
Revised: For a group convolution, we have g convolutions with kernels of shape ( C l /g, C l − 1 /g, K Hl , K Wl ) .

Original: Hence, it has O ( C 2 K 2 HW/g ) FLOPs and O ( C 2 K 2 /g ) parameters, where g is the number of groups.
Revised: Hence, it has O ( C 2 K 2 HW/g ) FLOPs and O ( C 2 K 2 /g ) parameters, where g is the number of groups.

Original: A depth-wise convolution is equivalent to a group convolution with g = C ∗ =
Revised: A depth-wise convolution is equivalent to a group convolution with g = C ∗ =

Original: C .
Revised: C .

Original: A pointwise convolution is a 1 × 1 convolution.
Revised: A pointwise convolution is a 1 × 1 convolution.

Original: A depth separable convolution is composed of a depth-wise convolution and a point-wise convolution.
Revised: A depth separable convolution is composed of a depth-wise convolution and a point-wise convolution.

Original: A spatial separable convolution replaces a K × K kernel with K × 1 and 1 × K kernels.
Revised: A spatial separable convolution replaces a K × K kernel with K × 1 and 1 × K kernels.

Original: Different types of convolutions are summarized in Table 1.
Revised: Different types of convolutions are summarized in Table 1.

Original: From this table, their FLOPs and number of parameters can be easily veriﬁed.
Revised: From this table, their FLOPs and number of parameters can be easily veriﬁed.

Original: 2. R ETHINKING C ONVOLUTION AND THE V OLUMETRIC R ECEPTIVE F IELD C ONDITION
Revised: 2. R ETHINKING C ONVOLUTION AND THE V OLUMETRIC R ECEPTIVE F IELD C ONDITION

Original: Separable convolution has been proven to be efﬁcient in reducing the computational demand in convolution.
Revised: Separable convolution has been proven to be efﬁcient in reducing the computational demand in convolution.

Original: However, existing approaches including both depth separable and spatial separable convolutions follow an ad hoc design.
Revised: However, existing approaches including both depth separable and spatial separable convolutions follow an ad hoc design.

Original: They are able to reduce the computational cost to some extent but will not normally achieve an optimal separation.
Revised: They are able to reduce the computational cost to some extent but will not normally achieve an optimal separation.

Original: In this research, we shall design an efﬁcient convolution operator achieving the computational objective by optimal design of its internal hyperparameters.
Revised: In this research, we shall design an efﬁcient convolution operator achieving the computational objective by optimal design of its internal hyperparameters.

Original: The resulting operator is called optimal separable convolution .
Revised: The resulting operator is called optimal separable convolution .

Original: One difﬁculty is that if we do not pose any restriction to a separable convolution, optimizing the
Revised: One difﬁculty is that if we do not pose any restriction to a separable convolution, optimizing the

Original: FLOPs target will resulting in a separable convolution being equivalent to a degenerated channel scaling operator 3 .
Revised: FLOPs target will resulting in a separable convolution being equivalent to a degenerated channel scaling operator 3 .

Original: Hence, we propose the following volumetric receptive ﬁeld condition
Revised: Hence, we propose the following volumetric receptive ﬁeld condition

Original: From Table 1, let g = C and K = 1 , a convolution will have C parameters and CHW FLOPs.
Revised: From Table 1, let g = C and K = 1 , a convolution will have C parameters and CHW FLOPs.

Original: This is in fact a channel scaling operator.
Revised: This is in fact a channel scaling operator.

Original: Composition of such operators is not meaningful because the composition itself is equivalent to a single channel scaling operator.
Revised: Composition of such operators is not meaningful because the composition itself is equivalent to a single channel scaling operator.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 1a, the receptive ﬁeld (RF) of a convolution is deﬁned to be the region in the input space that a particular CNN’s feature is affected by (Lindeberg, 2013).
Revised: 1a, the receptive ﬁeld (RF) of a convolution is deﬁned to be the region in the input space that a particular CNN’s feature is affected by (Lindeberg, 2013).

Original: We deﬁne the channel
Revised: We deﬁne the channel

Original: RF to be the channels that affect CNN’s output and deﬁne the volumetric RF to be the Cartesian product of the RF and channel RF of this convolution.
Revised: RF to be the channels that affect CNN’s output and deﬁne the volumetric RF to be the Cartesian product of the RF and channel RF of this convolution.

Original: The volumetric RF of a convolution is actually the volume in the input space that affects CNN’s output.
Revised: The volumetric RF of a convolution is actually the volume in the input space that affects CNN’s output.

Original: The volumetric RF condition requires that a properly decomposed separable convolution maintains (at least) the same volumetric RF as the original convolution before decomposition.
Revised: The volumetric RF condition requires that a properly decomposed separable convolution maintains (at least) the same volumetric RF as the original convolution before decomposition.

Original: Hence, the proposed optimal separable convolution will be equivalent to optimizing its internal parameters while satisfying the volumetric RF condition.
Revised: Hence, the proposed optimal separable convolution will be equivalent to optimizing its internal parameters while satisfying the volumetric RF condition.

Original: 2. O PTIMAL S EPARABLE C ONVOLUTION
Revised: 2. O PTIMAL S EPARABLE C ONVOLUTION

Original: In this section, we discuss the case of two-separable convolution.
Revised: In this section, we discuss the case of two-separable convolution.

Original: We present the discussion informally to gain intuition into the proposed approach.
Revised: We present the discussion informally to gain intuition into the proposed approach.

Original: In the next section, we shall provide a formal proof.
Revised: In the next section, we shall provide a formal proof.

Original: Suppose that the shape of the original convolutional kernel is ( C out , C in , K H , K W ) , where
Revised: Suppose that the shape of the original convolutional kernel is ( C out , C in , K H , K W ) , where

Original: C in , C out are the input and output channels, and ( K H , K W ) is the kernel size.
Revised: C in , C out are the input and output channels, and ( K H , K W ) is the kernel size.

Original: Let C 1 = C in , and
Revised: Let C 1 = C in , and

Original: C 3 = C out .
Revised: C 3 = C out .

Original: For the proposed optimal separable convolution, we optimize the FLOPs as computational objective while maintaining the original convolution’s volumetric RF.
Revised: For the proposed optimal separable convolution, we optimize the FLOPs as computational objective while maintaining the original convolution’s volumetric RF.

Original: Formally, the computational demand of the proposed separable convolution is
Revised: Formally, the computational demand of the proposed separable convolution is

Original: [Equation]
Revised: [Equation]

Original: In order to satisfy the volumetric RF condition, the following three conditions need to be satisﬁed 4 :
Revised: In order to satisfy the volumetric RF condition, the following three conditions need to be satisﬁed 4 :

Original: [Equation]
Revised: [Equation]

Original: We have three sets of parameters: the number of groups g 1 , g 2 , the internal channel size C 2 , and the internal kernel sizes K H | W ∗ .
Revised: We have three sets of parameters: the number of groups g 1 , g 2 , the internal channel size C 2 , and the internal kernel sizes K H | W ∗ .

Original: In this research, we shall assume that the internal channel size C 2 is in an order of O ( C ) and is preset according to a given policy.
Revised: In this research, we shall assume that the internal channel size C 2 is in an order of O ( C ) and is preset according to a given policy.

Original: Otherwise, g 1 = g 2 =
Revised: Otherwise, g 1 = g 2 =

Original: C 2 = 1 will be a trivial solution.
Revised: C 2 = 1 will be a trivial solution.

Original: This could lead the separable convolution to be over-simpliﬁed and not applicable in practice.
Revised: This could lead the separable convolution to be over-simpliﬁed and not applicable in practice.

Original: Typical policies of presetting C 2 include C 2 = min( C 1 , C 3 ) (normal architecture),
Revised: Typical policies of presetting C 2 include C 2 = min( C 1 , C 3 ) (normal architecture),

Original: C 

Revised: C

Original: =
Revised: =

Original: ( C 1 + C 3 ) / 2 (linear architecture),
Revised: ( C 1 + C 3 ) / 2 (linear architecture),

Original: C 2 = max( C 1 , C 3 ) / 4 (bottleneck architecture (He et al., 2016)), or C 2 = 4 min( C 1 , C 3 ) (inverted residual architecture (Sandler et al., 2018)).
Revised: C 2 = max( C 1 , C 3 ) / 4 (bottleneck architecture (He et al., 2016)), or C 2 = 4 min( C 1 , C 3 ) (inverted residual architecture (Sandler et al., 2018)).

Original: [Figure]
Revised: [Figure]

Original: The proposed problem is a constrained optimization.
Revised: The proposed problem is a constrained optimization.

Original: It is usually hard to solve it directly .
Revised: It is usually hard to solve it directly .

Original: However, we shall show that the optimal solution shall be g 1 , g 2 ∼ √ C .
Revised: However, we shall show that the optimal solution shall be g 1 , g 2 ∼ √ C .

Original: For large channel sizes, the optimal solution is usually an interior point in the solution space rather than on the boundary.
Revised: For large channel sizes, the optimal solution is usually an interior point in the solution space rather than on the boundary.

Original: Let K H | W ∗ be constants, by substituting g 

Revised: Let K H | W ∗ be constants, by substituting g

Original: = C 2 /g 1and setting f (cid:48) ( g 1 ) = 0 , one can derive that
Revised: = C 2 /g 1and setting f (cid:48) ( g 1 ) = 0 , one can derive that

Original: [Equation]
Revised: [Equation]

Original: and min
Revised: and min

Original: [Equation]
Revised: [Equation]

Original: if we set K H | W= K H | W and K H |
Revised: if we set K H | W= K H | W and K H |

Original: [Equation]
Revised: [Equation]

Original: = 1 .
Revised: = 1 .

Original: One interesting fact is that we can optimize the internal number of groups g 1 , g 2 and internal kernel sizes K H | W ∗ simultaneously.
Revised: One interesting fact is that we can optimize the internal number of groups g 1 , g 2 and internal kernel sizes K H | W ∗ simultaneously.

Original: For simplicity, we assume that kernel sizes aligned in height and width are equal.
Revised: For simplicity, we assume that kernel sizes aligned in height and width are equal.

Original: By setting f (cid:48) ( g 1 ) = 0 and f (cid:48)
Revised: By setting f (cid:48) ( g 1 ) = 0 and f (cid:48)

Original: ( K 1 ) = 0 , one can derive that g 1 is the same as
Revised: ( K 1 ) = 0 , one can derive that g 1 is the same as

Original: Equation (7) and K 1 = K 2 = K +12 , substituting them into Equation (8), one can get f ( g 1 , K 1 ) =
Revised: Equation (7) and K 1 = K 2 = K +12 , substituting them into Equation (8), one can get f ( g 1 , K 1 ) =

Original: O ( C 32 K 2 HW ) .
Revised: O ( C 32 K 2 HW ) .

Original: This results in a higher complexity than O ( C 32 KHW ) .
Revised: This results in a higher complexity than O ( C 32 KHW ) .

Original: In fact, the solution to f (cid:48) ( g 1 ) = 

Revised: In fact, the solution to f (cid:48) ( g 1 ) =

Original: and f (cid:48) ( K 1 ) = 0 is a saddle point.
Revised: and f (cid:48) ( K 1 ) = 0 is a saddle point.

Original: As illustrated in Fig. 2, given the input channel C 1 = 64 and output channel C 3 = 64 , kernel size ( K H , K W ) = (5 , 5) , we take C 2 =
Revised: As illustrated in Fig. 2, given the input channel C 1 = 64 and output channel C 3 = 64 , kernel size ( K H , K W ) = (5 , 5) , we take C 2 =

Original: min( C 1 , C 3 ) = 64 .
Revised: min( C 1 , C 3 ) = 64 .

Original: By setting f (cid:48) ( g 1 ) = 0 , f (cid:48)
Revised: By setting f (cid:48) ( g 1 ) = 0 , f (cid:48)

Original: ( K 1 ) = 0 , the solution g 1 = √ 64 = 8 , K 1 = 5+12 = 3 is a saddle point.
Revised: ( K 1 ) = 0 , the solution g 1 = √ 64 = 8 , K 1 = 5+12 = 3 is a saddle point.

Original: 2. O PTIMAL S EPARABLE C ONVOLUTION (G ENERAL C ASE )
Revised: 2. O PTIMAL S EPARABLE C ONVOLUTION (G ENERAL C ASE )

Original: In this section, we shall generalize the proposed optimal separable convolution from N = 2 to an optimal N and shall provide a formal proof.
Revised: In this section, we shall generalize the proposed optimal separable convolution from N = 2 to an optimal N and shall provide a formal proof.

Original: Suppose that the shape of the original convolutional kernel is ( C out , C in , K H , K W ) .
Revised: Suppose that the shape of the original convolutional kernel is ( C out , C in , K H , K W ) .

Original: Let C 1 = C in , and C N
Revised: Let C 1 = C in , and C N

Original: +

Revised: +

Original: =
Revised: =

Original: C out ( C N +

Revised: C out ( C N +

Original: = C 3 for N = 2 ).
Revised: = C 3 for N = 2 ).

Original: The computational demand of an N -separable convolution is
Revised: The computational demand of an N -separable convolution is

Original: [Equation]
Revised: [Equation]

Original: For ease of analysis, we ﬁrst introduce the notation channels per group n l = C l g l , which simply means: channels per group × number of groups =
Revised: For ease of analysis, we ﬁrst introduce the notation channels per group n l = C l g l , which simply means: channels per group × number of groups =

Original: the number of channels.
Revised: the number of channels.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: satisfying the volumetric RF condition
Revised: satisfying the volumetric RF condition

Original: [Equation]
Revised: [Equation]

Original: We keep both notations g l and n l .
Revised: We keep both notations g l and n l .

Original: This is because, for the channel condition, it is intuitive that n 1 · · · n N ≥ C 1 means that the product of n 1 · · · n N needs to occupy each node in the input channel
Revised: This is because, for the channel condition, it is intuitive that n 1 · · · n N ≥ C 1 means that the product of n 1 · · · n N needs to occupy each node in the input channel

Original: C 1 = C in .
Revised: C 1 = C in .

Original: This is equivalent to the less intuitive condition g 1 · · · g N ≤ C 2 · · · C N .
Revised: This is equivalent to the less intuitive condition g 1 · · · g N ≤ C 2 · · · C N .

Original: Similarly, for the group convolution condition, g l ≤ min( C l , C l +1 ) means the number of groups can not exceed the input and output channels of this group convolution, while n l ≥ max(1 , C l +1 C l ) is less intuitive.
Revised: Similarly, for the group convolution condition, g l ≤ min( C l , C l +1 ) means the number of groups can not exceed the input and output channels of this group convolution, while n l ≥ max(1 , C l +1 C l ) is less intuitive.

Original: For Equation (10), apply an arithmetic-geometric mean inequality, we can get
Revised: For Equation (10), apply an arithmetic-geometric mean inequality, we can get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The equality holds if and only if C 2 n 1 K H 1 K W
Revised: The equality holds if and only if C 2 n 1 K H 1 K W

Original: [Equation]
Revised: [Equation]

Original: = · · · =
Revised: = · · · =

Original: [Equation]
Revised: [Equation]

Original: N +1 n N K HN K WN .
Revised: N +1 n N K HN K WN .

Original: Let n l = β l n 1 , where β l = C 2 K H 

Revised: Let n l = β l n 1 , where β l = C 2 K H

Original: K W 1 C l +1 K Hl K Wl .
Revised: K W 1 C l +1 K Hl K Wl .

Original: Let β
Revised: Let β

Original: = Π β i , we can solve n 1 = N (cid:113)
Revised: = Π β i , we can solve n 1 = N (cid:113)

Original: [Equation]
Revised: [Equation]

Original: 1 β = N
Revised: 1 β = N

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Note that the inequality (16) holds for arbitrary K H | W ∗ .
Revised: Note that the inequality (16) holds for arbitrary K H | W ∗ .

Original: We need to further optimize K H | W ∗ .
Revised: We need to further optimize K H | W ∗ .

Original: From the arithmetic-geometric mean inequality again, we can get K H 1 · · · K HN ≤ ( K H 1 + ··· + K HN N ) N =
Revised: From the arithmetic-geometric mean inequality again, we can get K H 1 · · · K HN ≤ ( K H 1 + ··· + K HN N ) N =

Original: ( K H + N − 1 N ) N and the equality holds if and only if K H 1 = · · · = K HN =
Revised: ( K H + N − 1 N ) N and the equality holds if and only if K H 1 = · · · = K HN =

Original: K H + N − 1 N .
Revised: K H + N − 1 N .

Original: However, we want the inequality reversed, instead of ﬁnding the maximum of this product, we expect to ﬁnd its minimum.
Revised: However, we want the inequality reversed, instead of ﬁnding the maximum of this product, we expect to ﬁnd its minimum.

Original: This still gives us a hint, the maximum is achieved when the internal kernel sizes are as even as possible, so the minimum should be achieved when the internal kernel sizes are as diverse as possible.
Revised: This still gives us a hint, the maximum is achieved when the internal kernel sizes are as even as possible, so the minimum should be achieved when the internal kernel sizes are as diverse as possible.

Original: In the extreme case, one of the internal kernel sizes should take K H and all the rest takes 1 .
Revised: In the extreme case, one of the internal kernel sizes should take K H and all the rest takes 1 .

Original: A formal proof of this claim can be derived.
Revised: A formal proof of this claim can be derived.

Original: Hence, we have
Revised: Hence, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: It can be veriﬁed that, for N = 2 , Equation (8) and Equation (18) match with the same complexity.
Revised: It can be veriﬁed that, for N = 2 , Equation (8) and Equation (18) match with the same complexity.

Original: By setting f (cid:48) ( N ) = 0 , we can derive that N = log( CK 2 ) , and min f ( { n ∗ } , { K H | W ∗ } ) =
Revised: By setting f (cid:48) ( N ) = 0 , we can derive that N = log( CK 2 ) , and min f ( { n ∗ } , { K H | W ∗ } ) =

Original: eCHW · log( CK 2 )
Revised: eCHW · log( CK 2 )

Original: =
Revised: =

Original: O ( CHW · log( CK 2 ))
Revised: O ( CHW · log( CK 2 ))

Original: where e = 2 .
Revised: where e = 2 .

Original: 71828 ... is the natural logarithm constant.
Revised: 71828 ... is the natural logarithm constant.

Original: The proposed optimal separable convolution can have a spatial separable conﬁguration: a single kernel takes ( K H , K W ) or two kernels take ( K H , 1) and (1 , K W ) .
Revised: The proposed optimal separable convolution can have a spatial separable conﬁguration: a single kernel takes ( K H , K W ) or two kernels take ( K H , 1) and (1 , K W ) .

Original: Besides, the proposed optimal separable convolution allows using a mask of the internal number of groups and solve for an Mseparable sub-problem ( M < N ) .
Revised: Besides, the proposed optimal separable convolution allows using a mask of the internal number of groups and solve for an Mseparable sub-problem ( M < N ) .

Original: Details are discussed in Appendix A, where the implementation of the proposed optimal separable convolution is also presented in Algorithm 1.
Revised: Details are discussed in Appendix A, where the implementation of the proposed optimal separable convolution is also presented in Algorithm 1.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we carry out extensive experiments on benchmark datasets to demonstrate the effectiveness of the proposed optimal separable convolution scheme.
Revised: In this section, we carry out extensive experiments on benchmark datasets to demonstrate the effectiveness of the proposed optimal separable convolution scheme.

Original: In the proposed experiments, we use a preﬁx d - or o - to indicate that the conventional or depth separable convolutions in the baseline networks are replaced with depth separable ( dsep ) or the proposed optimal separable ( osep ) convolutions.
Revised: In the proposed experiments, we use a preﬁx d - or o - to indicate that the conventional or depth separable convolutions in the baseline networks are replaced with depth separable ( dsep ) or the proposed optimal separable ( osep ) convolutions.

Original: In this research, we set the number of separated convolutions N = 2 .
Revised: In this research, we set the number of separated convolutions N = 2 .

Original: The details of the training settings for the proposed experiments are described in Appendix B.
Revised: The details of the training settings for the proposed experiments are described in Appendix B.

Original: 3. E XPERIMENTAL R ESULTS ON CIFAR 
Revised: 3. E XPERIMENTAL R ESULTS ON CIFAR

Original: CIFAR10 (Krizhevsky et al., 2009) is a dataset consist of 50,000 training images and 10,000 testing images.
Revised: CIFAR10 (Krizhevsky et al., 2009) is a dataset consist of 50,000 training images and 10,000 testing images.

Original: These images are with a resolution of 32 × 32 and are categorized into 10 object classes.
Revised: These images are with a resolution of 32 × 32 and are categorized into 10 object classes.

Original: In the proposed experiments, we use ResNet (He et al., 2016) as baselines and replace the conventional convolutions in ResNet with dsep and osep convolutions, resulting in d -ResNet and o -ResNet.
Revised: In the proposed experiments, we use ResNet (He et al., 2016) as baselines and replace the conventional convolutions in ResNet with dsep and osep convolutions, resulting in d -ResNet and o -ResNet.

Original: The proposed osep scheme can signiﬁcantly reduce FLOPs/#Params.
Revised: The proposed osep scheme can signiﬁcantly reduce FLOPs/#Params.

Original: In Section 2, we proved that this reduction factor can be √ CK in theory 6 .
Revised: In Section 2, we proved that this reduction factor can be √ CK in theory 6 .

Original: As illustrated by the solid lines in Fig.
Revised: As illustrated by the solid lines in Fig.

Original: 3 (a) and (b), the orange solid curve lies in a region with signiﬁcantly smaller x -values than the blue solid curve.
Revised: 3 (a) and (b), the orange solid curve lies in a region with signiﬁcantly smaller x -values than the blue solid curve.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This indicates that o -ResNet shall have signiﬁcantly smaller FLOPs and fewer parameters than the
Revised: This indicates that o -ResNet shall have signiﬁcantly smaller FLOPs and fewer parameters than the

Original: ResNet baseline.
Revised: ResNet baseline.

Original: For example, o -ResNet 110 has even lower FLOPs (0.033 billion vs 0.041 billion) and fewer parameters (0.177 million vs 0.270 million) than ResNet 20 , yet with noticeable higher accuracy (92.12% vs 91.25%).
Revised: For example, o -ResNet 110 has even lower FLOPs (0.033 billion vs 0.041 billion) and fewer parameters (0.177 million vs 0.270 million) than ResNet 20 , yet with noticeable higher accuracy (92.12% vs 91.25%).

Original: This demonstrates that the proposed osep scheme could signiﬁcantly reduce both computational cost and number of parameters for conventional convolutions.
Revised: This demonstrates that the proposed osep scheme could signiﬁcantly reduce both computational cost and number of parameters for conventional convolutions.

Original: For dsep , this reduction factor is 1 1 /K 2 +1 /C , which is bounded by K 2 .
Revised: For dsep , this reduction factor is 1 1 /K 2 +1 /C , which is bounded by K 2 .

Original: For 3 × 3 kernels, this reduction can be at most 9.
Revised: For 3 × 3 kernels, this reduction can be at most 9.

Original: Whereas for the proposed osep scheme, no such bounds exist.
Revised: Whereas for the proposed osep scheme, no such bounds exist.

Original: The advantage of the proposed osep scheme over dsep is illustrated in Fig.
Revised: The advantage of the proposed osep scheme over dsep is illustrated in Fig.

Original: 3 (a) and (b) by the orange and green solid curves.
Revised: 3 (a) and (b) by the orange and green solid curves.

Original: From which, we can see the proposed osep scheme is more efﬁcient 7 with smaller x -values.
Revised: From which, we can see the proposed osep scheme is more efﬁcient 7 with smaller x -values.

Original: The proposed o -ResNets can have 8x-16x smaller FLOPs and 10x-18x fewer parameters than the
Revised: The proposed o -ResNets can have 8x-16x smaller FLOPs and 10x-18x fewer parameters than the

Original: ResNet baselines in the proposed experiments.
Revised: ResNet baselines in the proposed experiments.

Original: For fair comparisons, we introduce the channel multiplier in order to approximately match the FLOPs.
Revised: For fair comparisons, we introduce the channel multiplier in order to approximately match the FLOPs.

Original: We use the sufﬁx “ m < multiplier > ” 8 to indicate the channel multiplier.
Revised: We use the sufﬁx “ m < multiplier > ” 8 to indicate the channel multiplier.

Original: Note that FLOPs/#Params is proportional to channel multiplier 3 /for osep .
Revised: Note that FLOPs/#Params is proportional to channel multiplier 3 /for osep .

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 3, from which we can see, the proposed optimal separable convolution scheme is much more efﬁcient than conventional convolutions.
Revised: 3, from which we can see, the proposed optimal separable convolution scheme is much more efﬁcient than conventional convolutions.

Original: The orange curve, including both solid and dashed parts, achieved a better accuracy-FLOPs Pareto-frontier than the blue curve.
Revised: The orange curve, including both solid and dashed parts, achieved a better accuracy-FLOPs Pareto-frontier than the blue curve.

Original: It is worth noting that even under the same FLOPs, the number of o -ResNet parameters is also smaller than that of ResNet by a large margin.
Revised: It is worth noting that even under the same FLOPs, the number of o -ResNet parameters is also smaller than that of ResNet by a large margin.

Original: This could result in a more regularized network with fewer parameters to prevent over-ﬁtting and possibly contribute to the ﬁnal performance.
Revised: This could result in a more regularized network with fewer parameters to prevent over-ﬁtting and possibly contribute to the ﬁnal performance.

Original: In Fig.
Revised: In Fig.

Original: 3, we also present the d -ResNet curves in dashed green by replacing the conventional convolutions with depth separable convolutions.
Revised: 3, we also present the d -ResNet curves in dashed green by replacing the conventional convolutions with depth separable convolutions.

Original: As can be seen, d -ResNet achieves good accuracy-FLOPs balances for small networks (e.g. d -ResNet20 and d -ResNet32), but performs comparable or no better than conventional convolutions for large ones (e.g. d -ResNet56 and d -ResNet110).
Revised: As can be seen, d -ResNet achieves good accuracy-FLOPs balances for small networks (e.g. d -ResNet20 and d -ResNet32), but performs comparable or no better than conventional convolutions for large ones (e.g. d -ResNet56 and d -ResNet110).

Original: In summary, the proposed optimal separable convolution achieves better accuracy-FLOPs and accuracy-#Params
Revised: In summary, the proposed optimal separable convolution achieves better accuracy-FLOPs and accuracy-#Params

Original: Pareto-frontiers than both conventional and depth separable convolutions.
Revised: Pareto-frontiers than both conventional and depth separable convolutions.

Original: To demonstrate that the proposed osep scheme generalizes well to other DCNN architectures, we adopt the DARTS (V2) (Liu et al., 2018) network as the baseline.
Revised: To demonstrate that the proposed osep scheme generalizes well to other DCNN architectures, we adopt the DARTS (V2) (Liu et al., 2018) network as the baseline.

Original: The DARTS evaluation network has 20 cells and 36 initial channels, we increase the initial channels to 42 to match the FLOPs.
Revised: The DARTS evaluation network has 20 cells and 36 initial channels, we increase the initial channels to 42 to match the FLOPs.

Original: By replacing the dsep convolutions in DARTS with the proposed osep convolutions, as illustrated in
Revised: By replacing the dsep convolutions in DARTS with the proposed osep convolutions, as illustrated in

Original: Table 2, the resulting o -DARTS improved the accuracy from 97.24% to 97.67% with fewer parameters (3.25 million vs 3.35 million).
Revised: Table 2, the resulting o -DARTS improved the accuracy from 97.24% to 97.67% with fewer parameters (3.25 million vs 3.35 million).

Original: It is worth noting that it is very hard to signiﬁcantly improve the
Revised: It is worth noting that it is very hard to signiﬁcantly improve the

Original: DARTS search space.
Revised: DARTS search space.

Original: In Table 2, we also include three variants of DARTS, i.e. P-DARTS (Chen et al., 2019),
Revised: In Table 2, we also include three variants of DARTS, i.e. P-DARTS (Chen et al., 2019),

Original: PC-DARTS (Xu et al., 2019), and GOLD-DARTS (Bi et al., 2020), with more advanced search strategies for comparison.
Revised: PC-DARTS (Xu et al., 2019), and GOLD-DARTS (Bi et al., 2020), with more advanced search strategies for comparison.

Original: As can be seen, o -DARTS even achieved higher accuracies than these advanced network architectures.
Revised: As can be seen, o -DARTS even achieved higher accuracies than these advanced network architectures.

Original: Ablation Studies We carry out ablation studies on the effects of the internal BatchNorms and nonlinearities, and the spatial separable conﬁguration.
Revised: Ablation Studies We carry out ablation studies on the effects of the internal BatchNorms and nonlinearities, and the spatial separable conﬁguration.

Original: We conclude that internal BatchNorms and nonlinearities have no effects on the results yet introduce extra computation and parameters, while the spatial separable conﬁguration leads to a slightly worse performance.
Revised: We conclude that internal BatchNorms and nonlinearities have no effects on the results yet introduce extra computation and parameters, while the spatial separable conﬁguration leads to a slightly worse performance.

Original: Hence, they are not adopted in this research.
Revised: Hence, they are not adopted in this research.

Original: Details are presented in Table 5 with discussion in Appendix C.
Revised: Details are presented in Table 5 with discussion in Appendix C.

Original: 3. E XPERIMENTAL R ESULTS ON I MAGE N ET
Revised: 3. E XPERIMENTAL R ESULTS ON I MAGE N ET

Original: We evaluate the proposed optimal separable convolution scheme on the benchmark ImageNet (Deng et al., 2009) dataset, which contains 1.28 million training images and 50,000 testing images.
Revised: We evaluate the proposed optimal separable convolution scheme on the benchmark ImageNet (Deng et al., 2009) dataset, which contains 1.28 million training images and 50,000 testing images.

Original: 3.2. I MAGE N ET
Revised: 3.2. I MAGE N ET

Original: Because carrying out experiments directly on the ImageNet dataset can be resource- and timeconsuming, we resized all the images into 40 × 40 pixels.
Revised: Because carrying out experiments directly on the ImageNet dataset can be resource- and timeconsuming, we resized all the images into 40 × 40 pixels.

Original: A 32 × 32 patch is randomly cropped and a random horizontal ﬂip with a probability of 0.5 is applied before feeding into the network.
Revised: A 32 × 32 patch is randomly cropped and a random horizontal ﬂip with a probability of 0.5 is applied before feeding into the network.

Original: No extra data augmentation strategies are used.
Revised: No extra data augmentation strategies are used.

Original: The baseline ResNet architecture is a modiﬁed version of that used on the CIFAR10 dataset, except that the channel sizes are set to be 4 × larger, the features are
Revised: The baseline ResNet architecture is a modiﬁed version of that used on the CIFAR10 dataset, except that the channel sizes are set to be 4 × larger, the features are

Original: We usually
Revised: We usually

Original: [Table] [Table]
Revised: [Table] [Table]

Original: calculated on scales of [16, 8, 4], and the last fully-connected (FC) layer outputs 1000 categories for classiﬁcation.
Revised: calculated on scales of [16, 8, 4], and the last fully-connected (FC) layer outputs 1000 categories for classiﬁcation.

Original: We make this modiﬁcation because the ImageNet dataset has signiﬁcantly more training samples than the CIFAR10 dataset.
Revised: We make this modiﬁcation because the ImageNet dataset has signiﬁcantly more training samples than the CIFAR10 dataset.

Original: Experimental results are illustrated in Table 3, as can be seen, by substituting conventional convolutions with the proposed optimal separable convolutions, the resulting o -ResNet achieved 4-5% (e.g. 49.97% vs 44.93% for 56-layer and 50.72% vs 46.74% for 110-layer) performance gains comparing against the ResNet baselines.
Revised: Experimental results are illustrated in Table 3, as can be seen, by substituting conventional convolutions with the proposed optimal separable convolutions, the resulting o -ResNet achieved 4-5% (e.g. 49.97% vs 44.93% for 56-layer and 50.72% vs 46.74% for 110-layer) performance gains comparing against the ResNet baselines.

Original: This demonstrates that the proposed optimal separable convolution scheme is much more efﬁcient.
Revised: This demonstrates that the proposed optimal separable convolution scheme is much more efﬁcient.

Original: For o - ResNet56 and oResNet110, they also have fewer parameters that could contribute to a more regularized model.
Revised: For o - ResNet56 and oResNet110, they also have fewer parameters that could contribute to a more regularized model.

Original: For o - ResNet20 and o - ResNet32, they have slightly more parameters because the last FC layer accounts for a great portion of overhead for 1000 classes.
Revised: For o - ResNet20 and o - ResNet32, they have slightly more parameters because the last FC layer accounts for a great portion of overhead for 1000 classes.

Original: 3.2. F ULL I MAGE N ET
Revised: 3.2. F ULL I MAGE N ET

Original: Similar to the experiments on CIFAR10, we replace the dsep convolutions in the DARTS (V2) network with the proposed osep convolutions to demonstrate that the proposed approach is able to generalize to other network architectures.
Revised: Similar to the experiments on CIFAR10, we replace the dsep convolutions in the DARTS (V2) network with the proposed osep convolutions to demonstrate that the proposed approach is able to generalize to other network architectures.

Original: The experiment is carried out on the full ImageNet dataset.
Revised: The experiment is carried out on the full ImageNet dataset.

Original: The DARTS evaluation network has 14 cells and 48 initial channels, we increase the initial channel size to 56 to match the FLOPs.
Revised: The DARTS evaluation network has 14 cells and 48 initial channels, we increase the initial channel size to 56 to match the FLOPs.

Original: The resulting network is called o -DARTS.
Revised: The resulting network is called o -DARTS.

Original: Experimental results are illustrated in Table 4.
Revised: Experimental results are illustrated in Table 4.

Original: It can be seen that, with fewer parameters (4.50 million vs 4.72 million), the proposed o -DARTS network achieved higher accuracies in both top1 (74.2% vs 73.3%) and top (91.9% vs 91.3%) accuracies than the DARTS baseline.
Revised: It can be seen that, with fewer parameters (4.50 million vs 4.72 million), the proposed o -DARTS network achieved higher accuracies in both top1 (74.2% vs 73.3%) and top (91.9% vs 91.3%) accuracies than the DARTS baseline.

Original: This indicates that the proposed osep is able to achieve better accuracy-FLOPs and accuracy-#Params balances than dsep convolutions.
Revised: This indicates that the proposed osep is able to achieve better accuracy-FLOPs and accuracy-#Params balances than dsep convolutions.

Original: It is worth noting that in the proposed experiments, we adopt ResNet and DARTS as the baselines because these are two most well-known architectures.
Revised: It is worth noting that in the proposed experiments, we adopt ResNet and DARTS as the baselines because these are two most well-known architectures.

Original: In practice, one may simply replace the conventional or depth separable convolutions with the proposed optimal separable convolutions in a DCNN to reduce the computation and model parameters.
Revised: In practice, one may simply replace the conventional or depth separable convolutions with the proposed optimal separable convolutions in a DCNN to reduce the computation and model parameters.

Original: By increasing the channel sizes, better-performing models can be expected.
Revised: By increasing the channel sizes, better-performing models can be expected.

Original: The proposed optimal separable convolution achieved improved accuracy-FLOPs and accuracy-#Params Pareto-frontiers than both conventional and depth separable convolutions.
Revised: The proposed optimal separable convolution achieved improved accuracy-FLOPs and accuracy-#Params Pareto-frontiers than both conventional and depth separable convolutions.

Original: Hence, one can either match the accuracy to get a smaller model with reduced computation and model parameters, or match the FLOPs to get a better-performing model.
Revised: Hence, one can either match the accuracy to get a smaller model with reduced computation and model parameters, or match the FLOPs to get a better-performing model.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we have presented a novel scheme called optimal separable convolution to improve the computational efﬁciency in convolution.
Revised: In this paper, we have presented a novel scheme called optimal separable convolution to improve the computational efﬁciency in convolution.

Original: Conventional convolution took a costly complexity at O
Revised: Conventional convolution took a costly complexity at O

Original: ( C 2 K 2 HW ) .
Revised: ( C 2 K 2 HW ) .

Original: The proposed optimal separable convolution scheme is able to achieve its complexity at O
Revised: The proposed optimal separable convolution scheme is able to achieve its complexity at O

Original: ( C 32 KHW ) , which is even lower than that of depth separable convolution at
Revised: ( C 32 KHW ) , which is even lower than that of depth separable convolution at

Original: O ( CHW · ( C + K 2 )) .
Revised: O ( CHW · ( C + K 2 )) .

Original: Hence, the proposed optimal separable convolution has the full potential to replace the usage of depth separable convolutions in a DCNN .
Revised: Hence, the proposed optimal separable convolution has the full potential to replace the usage of depth separable convolutions in a DCNN .

Original: Examples include but are not limited to the ResNet and DARTS architectures.
Revised: Examples include but are not limited to the ResNet and DARTS architectures.

Original: The proposed optimal separable convolution also has a spatial separable conﬁguration.
Revised: The proposed optimal separable convolution also has a spatial separable conﬁguration.

Original: A generalized N -separable case can achieve better performance at
Revised: A generalized N -separable case can achieve better performance at

Original: O ( CHW · log( CK 2 )) .
Revised: O ( CHW · log( CK 2 )) .

Original: Another potential impact of the proposed optimal separable convolution is for the AutoML community.
Revised: Another potential impact of the proposed optimal separable convolution is for the AutoML community.

Original: The proposed novel operator is able to increase the neural architecture search space.
Revised: The proposed novel operator is able to increase the neural architecture search space.

Original: In a multi-objective optimization formulation, where both accuracy and FLOPs are optimized, we expect a more efﬁcient network architecture can be discovered in the future using the proposed optimal separable convolution operator
Revised: In a multi-objective optimization formulation, where both accuracy and FLOPs are optimized, we expect a more efﬁcient network architecture can be discovered in the future using the proposed optimal separable convolution operator

Original: A A LGORITHMIC D ETAILS OF THE P ROPOSED O PTIMAL S EPARABLE C ONVOLUTION
Revised: A A LGORITHMIC D ETAILS OF THE P ROPOSED O PTIMAL S EPARABLE C ONVOLUTION

Original: For the proposed optimal separable convolution, one of the internal kernel sizes can take K H | W , while the rest takes 1 .
Revised: For the proposed optimal separable convolution, one of the internal kernel sizes can take K H | W , while the rest takes 1 .

Original: In this research, we simply select the middle kernel size as ( K H , K W ) .
Revised: In this research, we simply select the middle kernel size as ( K H , K W ) .

Original: In a spatial separable conﬁguration, we select the middle two to have their kernel sizes as ( K H , 1) and (1 , K W ) .
Revised: In a spatial separable conﬁguration, we select the middle two to have their kernel sizes as ( K H , 1) and (1 , K W ) .

Original: It is worth noting that all these conﬁgurations have the same FLOPs.
Revised: It is worth noting that all these conﬁgurations have the same FLOPs.

Original: Unlike the spatial separable convolution where the spatial separable conﬁguration is able to reduce the complexity from K 2 to 2 K .
Revised: Unlike the spatial separable convolution where the spatial separable conﬁguration is able to reduce the complexity from K 2 to 2 K .

Original: This is because the complexity has already been reduced to O ( K ) for the proposed optimal separable convolution.
Revised: This is because the complexity has already been reduced to O ( K ) for the proposed optimal separable convolution.

Original: Another interesting property of the proposed optimal separable convolution is that it prefers large kernel sizes over small ones.
Revised: Another interesting property of the proposed optimal separable convolution is that it prefers large kernel sizes over small ones.

Original: For the proposed optimal separable convolution, we are able to preset the internal convolutional kernel sizes according to a custom policy, and optimize the internal number of groups only.
Revised: For the proposed optimal separable convolution, we are able to preset the internal convolutional kernel sizes according to a custom policy, and optimize the internal number of groups only.

Original: Furthermore, we are able to preset a portion of the internal number of groups to certain values, and optimize only the remaining internal number of groups.
Revised: Furthermore, we are able to preset a portion of the internal number of groups to certain values, and optimize only the remaining internal number of groups.

Original: Suppose that the internal channel and kernel sizes are given.
Revised: Suppose that the internal channel and kernel sizes are given.

Original: Without loss of generality, we assume that g M +1 , · · · , g N are preset.
Revised: Without loss of generality, we assume that g M +1 , · · · , g N are preset.

Original: The proposed optimal separable problem will be an M -separable convolution sub-problem ( M < N ):
Revised: The proposed optimal separable problem will be an M -separable convolution sub-problem ( M < N ):

Original: [Equation]
Revised: [Equation]

Original: satisfying the volumetric RF condition
Revised: satisfying the volumetric RF condition

Original: [Equation]
Revised: [Equation]

Original: This M -separable sub-problem can be solved by the same algorithm.
Revised: This M -separable sub-problem can be solved by the same algorithm.

Original: A detailed implementation of the proposed optimal separable convolution is described by Algorithm 1.
Revised: A detailed implementation of the proposed optimal separable convolution is described by Algorithm 1.

Original: [Table]
Revised: [Table]

Original: B T RAINING S ETTINGS
Revised: B T RAINING S ETTINGS

Original: Experiments on CIFAR10 for the ResNet architecture The images are padded with 4 pixels and randomly cropped into 32 × 32 to feed into the network.
Revised: Experiments on CIFAR10 for the ResNet architecture The images are padded with 4 pixels and randomly cropped into 32 × 32 to feed into the network.

Original: A random horizontal ﬂip with a probability of 0.5 is also applied.
Revised: A random horizontal ﬂip with a probability of 0.5 is also applied.

Original: All the networks are trained with a standard SGD optimizer for 200 epochs.
Revised: All the networks are trained with a standard SGD optimizer for 200 epochs.

Original: The initial learning rate is set to 0.1, with a decay of 0.1 at the 100 and 150 epochs.
Revised: The initial learning rate is set to 0.1, with a decay of 0.1 at the 100 and 150 epochs.

Original: The batch size is 128.
Revised: The batch size is 128.

Original: A weight decay of 0.0001 and a momentum of 0.9 are used.
Revised: A weight decay of 0.0001 and a momentum of 0.9 are used.

Original: Experiments on CIFAR10 for the DARTS architecture We follow the same training settings in (Liu et al., 2018): the network is trained with a standard SGD optimizer for 600 epochs with a batch size of 96.
Revised: Experiments on CIFAR10 for the DARTS architecture We follow the same training settings in (Liu et al., 2018): the network is trained with a standard SGD optimizer for 600 epochs with a batch size of 96.

Original: The initial learning rate is set to 0.025 with a cosine learning rate scheduler.
Revised: The initial learning rate is set to 0.025 with a cosine learning rate scheduler.

Original: A weight decay of 0.0003 and a momentum of 0.9 are used.
Revised: A weight decay of 0.0003 and a momentum of 0.9 are used.

Original: Additional enhancements include cutout, path dropout of probability 0.2, and auxiliary towers with weight 0.4.
Revised: Additional enhancements include cutout, path dropout of probability 0.2, and auxiliary towers with weight 0.4.

Original: Experiments on ImageNet40 for the ResNet architecture Each network is trained with a standard SGD optimizer for 20 epochs with the initial learning rate set to 0.1, and a decay of 0.1 at the 10 and 15 epochs.
Revised: Experiments on ImageNet40 for the ResNet architecture Each network is trained with a standard SGD optimizer for 20 epochs with the initial learning rate set to 0.1, and a decay of 0.1 at the 10 and 15 epochs.

Original: The batch size is 256, the weight decay is 0.0001 and the momentum is 0.9.
Revised: The batch size is 256, the weight decay is 0.0001 and the momentum is 0.9.

Original: Experiments on full ImageNet for the DARTS architecture We follow the training settings in (Chen et al., 2019) for multi-GPU training: the images are random resized crop into 224 ×patches with a random scale in [0.08, 1.0] and a random aspect ratio in [0.75, 1.33].
Revised: Experiments on full ImageNet for the DARTS architecture We follow the training settings in (Chen et al., 2019) for multi-GPU training: the images are random resized crop into 224 ×patches with a random scale in [0.08, 1.0] and a random aspect ratio in [0.75, 1.33].

Original: Random horizontal ﬂip and color jitter are also applied.
Revised: Random horizontal ﬂip and color jitter are also applied.

Original: The network is trained from scratch for 250 epochs with batch size 1024 on 8 GPUs.
Revised: The network is trained from scratch for 250 epochs with batch size 1024 on 8 GPUs.

Original: An SGD optimizer with an initial learning rate of 0.5, a momentum of 0.9, and a weight decay of 3e-5.
Revised: An SGD optimizer with an initial learning rate of 0.5, a momentum of 0.9, and a weight decay of 3e-5.

Original: The learning rate is decayed linearly after each epoch.
Revised: The learning rate is decayed linearly after each epoch.

Original: Additional enhancements include label smoothing with weight 0.1 and auxiliary towers with weight 0.4.
Revised: Additional enhancements include label smoothing with weight 0.1 and auxiliary towers with weight 0.4.

Original: C A BLATION S TUDIES
Revised: C A BLATION S TUDIES

Original: Internal BatchNorm and Non-linearity For a DCNN, it is generally a good practice to add a
Revised: Internal BatchNorm and Non-linearity For a DCNN, it is generally a good practice to add a

Original: BatchNorm (BN) (Ioffe & Szegedy, 2015) and a non-linearity after each convolution.
Revised: BatchNorm (BN) (Ioffe & Szegedy, 2015) and a non-linearity after each convolution.

Original: For the proposed optimal separable convolution, we wonder if it is still necessary to add such a BN and a non-linearity after each of the internal separated convolutions.
Revised: For the proposed optimal separable convolution, we wonder if it is still necessary to add such a BN and a non-linearity after each of the internal separated convolutions.

Original: Experimental results are illustrated in Table 5.
Revised: Experimental results are illustrated in Table 5.

Original: Comparing the “Internal BN and Non-linearity” column against the “Accuracy” column, we are able to conclude that with or without internal BN and non-linearity, similar results with only statistical variances can be generated.
Revised: Comparing the “Internal BN and Non-linearity” column against the “Accuracy” column, we are able to conclude that with or without internal BN and non-linearity, similar results with only statistical variances can be generated.

Original: This is reasonable because the network has already been regularized by outer BN and non-linearity layers from the macro architecture.
Revised: This is reasonable because the network has already been regularized by outer BN and non-linearity layers from the macro architecture.

Original: Internal ones shall offer little to no additional improvements.
Revised: Internal ones shall offer little to no additional improvements.

Original: Because internal BN and non-linearity could introduce extra computation and parameters, in the proposed research, we shall not use internal BN and non-linearity.
Revised: Because internal BN and non-linearity could introduce extra computation and parameters, in the proposed research, we shall not use internal BN and non-linearity.

Original: Spatial Separable
Revised: Spatial Separable

Original: Another variation of the proposed optimal separable convolution scheme is the spatial separable conﬁguration.
Revised: Another variation of the proposed optimal separable convolution scheme is the spatial separable conﬁguration.

Original: For Equation (16), the optimal solution is achieved when one of  the internal kernel sizes takes K H | W and all the rest takes 1 .
Revised: For Equation (16), the optimal solution is achieved when one of [Table] the internal kernel sizes takes K H | W and all the rest takes 1 .

Original: It does not matter which one of the internal kernel sizes takes K H | W .
Revised: It does not matter which one of the internal kernel sizes takes K H | W .

Original: Hence, we have this spatial separable variant: a single kernel takes ( K H , K W ) or two kernels take ( K H , 1) and (1 , K W ) .
Revised: Hence, we have this spatial separable variant: a single kernel takes ( K H , K W ) or two kernels take ( K H , 1) and (1 , K W ) .

Original: The detailed implementation is illustrated in Algorithm 1.
Revised: The detailed implementation is illustrated in Algorithm 1.

Original: While spatial separable or not affects neither the FLOPs nor the number of parameters for the proposed optimal separable convolution, the results could be slightly different.
Revised: While spatial separable or not affects neither the FLOPs nor the number of parameters for the proposed optimal separable convolution, the results could be slightly different.

Original: As illustrated by the column “Spatial Separable” in Table 5, the spatial separable conﬁguration leads to slightly worse performances.
Revised: As illustrated by the column “Spatial Separable” in Table 5, the spatial separable conﬁguration leads to slightly worse performances.

Original: The reason might be that spatial separation fuses horizontal and vertical features separately, which could be less efﬁcient than fusing them simultaneously.
Revised: The reason might be that spatial separation fuses horizontal and vertical features separately, which could be less efﬁcient than fusing them simultaneously.

Revised: D I NFERENCE T IME FOR THE P ROPOSED O PTIMAL S EPARABLE C ONVOLUTION

Revised: FLOPs measures the best possible theoretical speed we are able to achieve.

Revised: In this Section, we further report the wall-clock inference time of the proposed optimal separable convolution scheme.

Revised: The inference time is measured on a laptop computer with Windows 10 operating system and Intel i5-8250 CPU, which we use to simulate a mobile platform.

Revised: The results are illustrated in Table 6.

Original: D R ELATED W ORK{+As can be seen, for ResNet20, o -ResNet20 m3.875, and d -ResNet20 m2.75, they have a similar FLOPs ( ≈ 0.0405 billion), yet o -ResNet
 m3.875 and d -ResNet20 m2.75 take a slightly longer inference time (0.0468s vs 0.0310s). This is because current implementation of grouped convolution in PyTorch is not optimized. From Table 6, we can also conclude that under the same FLOPs, o -ResNet has slightly better performance than d -ResNet (e.g. o -ResNet32 takes 0.0937s while d -ResNet takes 0.1154s).+}
Revised: {+As can be seen, for ResNet20, o -ResNet20 m3.875, and d -ResNet20 m2.75, they have a similar FLOPs ( ≈ 0.0405 billion), yet o -ResNet
 m3.875 and d -ResNet20 m2.75 take a slightly longer inference time (0.0468s vs 0.0310s). This is because current implementation of grouped convolution in PyTorch is not optimized. From Table 6, we can also conclude that under the same FLOPs, o -ResNet has slightly better performance than d -ResNet (e.g. o -ResNet32 takes 0.0937s while d -ResNet takes 0.1154s).+}

Revised: This might because the proposed optimal separable convolution scheme usually has smaller “groups” hyper-parameters than depth separable convolution.

Revised: Similar conclusions can also be drawn for deeper ResNet32, ResNet56, and ResNet110 networks.

Revised: E R ELATED W ORK

Original: There have been many previous works aiming at reducing the amount of computation in convolution.
Revised: There have been many previous works aiming at reducing the amount of computation in convolution.

Original: Historically, researchers apply Fast Fourier Transform (FFT) (Nussbaumer, 1981; Quarteroni et al., 2010) to implement convolution.
Revised: Historically, researchers apply Fast Fourier Transform (FFT) (Nussbaumer, 1981; Quarteroni et al., 2010) to implement convolution.

Original: For 1D convolution, FFT reduces the number of computations for H points from O ( H 2 ) to O
Revised: For 1D convolution, FFT reduces the number of computations for H points from O ( H 2 ) to O

Original: ( H log H ) .
Revised: ( H log H ) .

Original: For 2D convolution, FFT-2D reduces the computational complexity from O ( HW · K 2 ) to O ( HW · (log H +log W ))
Revised: For 2D convolution, FFT-2D reduces the computational complexity from O ( HW · K 2 ) to O ( HW · (log H +log W ))

Original: (Podlozhnyuk, 2007).
Revised: (Podlozhnyuk, 2007).

Original: Hence, it can be easily concluded that FFT gains great speed up for large convolutional kernels.
Revised: Hence, it can be easily concluded that FFT gains great speed up for large convolutional kernels.

Original: For small convolutional kernels ( K << H or W ), a direct application is often still cheaper.
Revised: For small convolutional kernels ( K << H or W ), a direct application is often still cheaper.

Original: Researchers also explore low rank approximation (Jaderberg et al., 2014; Ioannou et al., 2015) to implement convolutions.
Revised: Researchers also explore low rank approximation (Jaderberg et al., 2014; Ioannou et al., 2015) to implement convolutions.

Original: However, most of the existing methods obtain moderate efﬁciency improvements, and they usually require a pre-trained model and mainly focus on network pruning and compression.
Revised: However, most of the existing methods obtain moderate efﬁciency improvements, and they usually require a pre-trained model and mainly focus on network pruning and compression.

Original: In recent state-of-the-art deep CNN models, several heuristics are adopted to reduce the heavy computation in convolution.
Revised: In recent state-of-the-art deep CNN models, several heuristics are adopted to reduce the heavy computation in convolution.

Original: For example, in (He et al., 2016), the authors use a bottleneck structure.
Revised: For example, in (He et al., 2016), the authors use a bottleneck structure.

Original: Yet in (Sandler et al., 2018), the authors adopt an inverted bottleneck structure.
Revised: Yet in (Sandler et al., 2018), the authors adopt an inverted bottleneck structure.

Original: Such heuristics may require further ad hoc design to work in practice, however, they are not solid and shall become less convincing.
Revised: Such heuristics may require further ad hoc design to work in practice, however, they are not solid and shall become less convincing.

Original: Among various implementations of convolution, separable convolution has been proven to be more efﬁcient in reducing the computational demand.
Revised: Among various implementations of convolution, separable convolution has been proven to be more efﬁcient in reducing the computational demand.

Original: Depth separable convolution is explored extensively in modern DCNNs (Howard et al., 2017;
Revised: Depth separable convolution is explored extensively in modern DCNNs (Howard et al., 2017;

Original: Sandler et al., 2018; Howard et al., 2019;
Revised: Sandler et al., 2018; Howard et al., 2019;

Original: Liu et al., 2018; Tan & Le, 2019).
Revised: Liu et al., 2018; Tan & Le, 2019).

Original: It reduces the computational cost of a conventional convolution from O ( C 2 K 2 HW ) to O ( CHW · ( C + K 2 )) .
Revised: It reduces the computational cost of a conventional convolution from O ( C 2 K 2 HW ) to O ( CHW · ( C + K 2 )) .

Original: However, the proposed optimal separable convolution is even more efﬁcient than depth separable convolution.
Revised: However, the proposed optimal separable convolution is even more efﬁcient than depth separable convolution.

Original: It can be calculated at O ( C 32 KHW ) and has the full potential to replace the usage of depth separable convolutions.
Revised: It can be calculated at O ( C 32 KHW ) and has the full potential to replace the usage of depth separable convolutions.

Original: A second advantage of the proposed optimal separable convolution is that it can be applied to fully connected layers if we view them as 1 × 1 convolutional layers, whereas depth separable convolution cannot.
Revised: A second advantage of the proposed optimal separable convolution is that it can be applied to fully connected layers if we view them as 1 × 1 convolutional layers, whereas depth separable convolution cannot.

Original: Further, depth separable convolution requires the middle channel size to be equal to the input channel size, whereas for the proposed optimal separable convolution, the middle channel size can be freely set.
Revised: Further, depth separable convolution requires the middle channel size to be equal to the input channel size, whereas for the proposed optimal separable convolution, the middle channel size can be freely set.

Original: Spatial separable convolution was originally developed to speed up image processing operations.
Revised: Spatial separable convolution was originally developed to speed up image processing operations.

Original: For example, a Sobel kernel is a 3 × 3 kernel and can be written as (1 , 2 , 1) T · ( − 1 , 0 , 1) .
Revised: For example, a Sobel kernel is a 3 × 3 kernel and can be written as (1 , 2 , 1) T · ( − 1 , 0 , 1) .

Original: Spatial separable will require 6 instead of 9 parameters while doing the same operation.
Revised: Spatial separable will require 6 instead of 9 parameters while doing the same operation.

Original: Spatial separable convolution is also adopted in the design of modern DCNNs.
Revised: Spatial separable convolution is also adopted in the design of modern DCNNs.

Original: For example, in (Szegedy et al., 2016), the authors introduce spatial separation to the GoogLeNet (Szegedy et al., 2015) architecture.
Revised: For example, in (Szegedy et al., 2016), the authors introduce spatial separation to the GoogLeNet (Szegedy et al., 2015) architecture.

Original: For the proposed optimal separable convolution, there is also a spatial separable conﬁguration.
Revised: For the proposed optimal separable convolution, there is also a spatial separable conﬁguration.

Original: In the body of literature, separable convolution is also referred to factorized convolution or convolution decomposition .
Revised: In the body of literature, separable convolution is also referred to factorized convolution or convolution decomposition .

Original: In this research, the proposed scheme is called optimal separable convolution following the naming conventions of depth and spatial separable convolutions.
Revised: In this research, the proposed scheme is called optimal separable convolution following the naming conventions of depth and spatial separable convolutions.