Original: Block-wise Separable Convolutions: An Alternative Way to Factorize Standard Convolutions
Revised: Block-wise Separable Convolutions: An Alternative Way to Factorize Standard Convolutions

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Convolutional neural networks (CNNs) have demonstrated great capability of solving various computer vision tasks with nice prediction performance.
Revised: Convolutional neural networks (CNNs) have demonstrated great capability of solving various computer vision tasks with nice prediction performance.

Original: Nevertheless,the higher accuracy often comes with an increasing number of model parametersand large computational cost.
Revised: Nevertheless,the higher accuracy often comes with an increasing number of model parametersand large computational cost.

Original: This raises challenges in deploying them in resourcelimited devices.
Revised: This raises challenges in deploying them in resourcelimited devices.

Original: In this paper, we introduce block-wise separable convolutions(BlkSConv) to replace the standard convolutions in order to compress deep CNNmodels.
Revised: In this paper, we introduce block-wise separable convolutions(BlkSConv) to replace the standard convolutions in order to compress deep CNNmodels.

Original: First, BlkSConv expresses the standard convolutional kernel as an orderedset of block vectors each of which is a linear combination of fixed basis blockvectors.
Revised: First, BlkSConv expresses the standard convolutional kernel as an orderedset of block vectors each of which is a linear combination of fixed basis blockvectors.

Original: Then it eliminates most basis block vectors and their corresponding coefficients to obtain an approximated convolutional kernel.
Revised: Then it eliminates most basis block vectors and their corresponding coefficients to obtain an approximated convolutional kernel.

Original: Moreover, the proposed
Revised: Moreover, the proposed

Original: BlkSConv operation can be efficiently realized via a combination of pointwise andgroup-wise convolutions.
Revised: BlkSConv operation can be efficiently realized via a combination of pointwise andgroup-wise convolutions.

Original: Thus the constructed networks have smaller model sizeand fewer multiply-adds operations while keeping comparable prediction accuracy.
Revised: Thus the constructed networks have smaller model sizeand fewer multiply-adds operations while keeping comparable prediction accuracy.

Original: However, it is unknown how to search a qualified hyperparameter settingof the block depth and number of basis block vectors.
Revised: However, it is unknown how to search a qualified hyperparameter settingof the block depth and number of basis block vectors.

Original: To address this problem,we develop a hyperparameter search framework based on principal componentanalysis (PCA) to help determine these two hyperparameters such that the corresponding network achieves nice prediction performance while simultaneouslysatisfying the constraints of model size and model efficiency.
Revised: To address this problem,we develop a hyperparameter search framework based on principal componentanalysis (PCA) to help determine these two hyperparameters such that the corresponding network achieves nice prediction performance while simultaneouslysatisfying the constraints of model size and model efficiency.

Original: Experimental resultsdemonstrate the prediction performance of constructed BlkSConv-based CNNswhere several convolutional layers are replaced by BlkSConv layers suggestedby the proposed PCA-based hyperparameter search algorithm.
Revised: Experimental resultsdemonstrate the prediction performance of constructed BlkSConv-based CNNswhere several convolutional layers are replaced by BlkSConv layers suggestedby the proposed PCA-based hyperparameter search algorithm.

Original: Our results showthat BlkSConv-based CNNs achieve competitive performance compared with thestandard convolutional models for the datasets including ImageNet, CIFAR-10/100,
Revised: Our results showthat BlkSConv-based CNNs achieve competitive performance compared with thestandard convolutional models for the datasets including ImageNet, CIFAR-10/100,

Original: Stanford Dogs, and Oxford Flowers.
Revised: Stanford Dogs, and Oxford Flowers.

Original:  Introduction
Revised:  Introduction

Original: In the past decade, Deep Learning (DL) has been the basis of many successes in artificial intelligence,including a variety of applications in computer vision, reinforcement learning, and natural languageprocessing.
Revised: In the past decade, Deep Learning (DL) has been the basis of many successes in artificial intelligence,including a variety of applications in computer vision, reinforcement learning, and natural languageprocessing.

Original: One of the most popular deep neural networks is Convolutional Neural Network (CNN).
Revised: One of the most popular deep neural networks is Convolutional Neural Network (CNN).

Original: With the help of various techniques such as residual connections and batch normalization, it is easy totrain deep CNNs with many layers on powerful GPUs.
Revised: With the help of various techniques such as residual connections and batch normalization, it is easy totrain deep CNNs with many layers on powerful GPUs.

Original: While large-scale CNN models have achievedgreat successes, they require huge computational complexity and massive storage.
Revised: While large-scale CNN models have achievedgreat successes, they require huge computational complexity and massive storage.

Original: For example,
Revised: For example,

Original: VGG16 (27) has 138 million parameters and requires 154700 million multiply-add operations(MAdds) to classify an image.
Revised: VGG16 (27) has 138 million parameters and requires 154700 million multiply-add operations(MAdds) to classify an image.

Original: It is a great challenge to deploy them in real-time applications,especially on devices with limited resources such as mobile phones and embedded systems.
Revised: It is a great challenge to deploy them in real-time applications,especially on devices with limited resources such as mobile phones and embedded systems.

Original: Thus,the prediction models are required be compact and fast while keeping acceptable accuracy.
Revised: Thus,the prediction models are required be compact and fast while keeping acceptable accuracy.

Original: The mainapproach to be compact is the model compression which aims at establishing a tradeoff between model efficiency and accuracy.
Revised: The mainapproach to be compact is the model compression which aims at establishing a tradeoff between model efficiency and accuracy.

Original: In the area of model compression, methods to construct efficient andcompact CNNs are mainly divided into two approaches: one approach is to compress trained CNNsand the other approach is to design new compact CNNs and train them from scratch.
Revised: In the area of model compression, methods to construct efficient andcompact CNNs are mainly divided into two approaches: one approach is to compress trained CNNsand the other approach is to design new compact CNNs and train them from scratch.

Original: Many worksbased on the first approach suggested several techniques such as quantization (33), model pruning(6; 24), Huffman coding (6), and low rank factorization (12).
Revised: Many worksbased on the first approach suggested several techniques such as quantization (33), model pruning(6; 24), Huffman coding (6), and low rank factorization (12).

Original: Studies in the second approach mainly explored many ways for factorizing convolutions.
Revised: Studies in the second approach mainly explored many ways for factorizing convolutions.

Original: Forinstance, Szegedy et al.
Revised: Forinstance, Szegedy et al.

Original: (30) improved GoogLeNet (29) through factorizing convolutions with largerspatial filters by a two-layer convolutional architecture with smaller spatial filters.
Revised: (30) improved GoogLeNet (29) through factorizing convolutions with largerspatial filters by a two-layer convolutional architecture with smaller spatial filters.

Original: At present, mostfactorizing methods are usually performed via a combination of depthwise convolution, pointwiseconvolution, and groupwise convolution.
Revised: At present, mostfactorizing methods are usually performed via a combination of depthwise convolution, pointwiseconvolution, and groupwise convolution.

Original: For example, in (25), the depth-wise separable convolutions(DSCs) were proposed where the standard convolution is decomposed into a depth-wise convolutionand a pointwise convolution.
Revised: For example, in (25), the depth-wise separable convolutions(DSCs) were proposed where the standard convolution is decomposed into a depth-wise convolutionand a pointwise convolution.

Original: The ShuffleNets (36; 18) utilizes pointwise group convolution withchannel shuffle to decompose the standard convolution.
Revised: The ShuffleNets (36; 18) utilizes pointwise group convolution withchannel shuffle to decompose the standard convolution.

Original: Moreover, many lightweight models basedon DSCs or groupwise convolutions such as MobileNets (8; 23; 7) and ShuffleNets (36; 18) wereproposed to greatly reduce computation cost while maintaining accuracy.
Revised: Moreover, many lightweight models basedon DSCs or groupwise convolutions such as MobileNets (8; 23; 7) and ShuffleNets (36; 18) wereproposed to greatly reduce computation cost while maintaining accuracy.

Original: In this paper, we follow the research path of the second approach and propose block-wise separableconvolutions ( BlkSConv ) to replace standard convolutions.
Revised: In this paper, we follow the research path of the second approach and propose block-wise separableconvolutions ( BlkSConv ) to replace standard convolutions.

Original: BlkSConv approximates a standardconvolution as follows.
Revised: BlkSConv approximates a standardconvolution as follows.

Original: A standard k × k × M convolutional kernel can be represented as an orderedset of block vectors of size k × k × t .
Revised: A standard k × k × M convolutional kernel can be represented as an orderedset of block vectors of size k × k × t .

Original: Since each block vector can be written as a linear combinationof k 2 t basis vectors of size k × k × t , this standard convolutional kernel can be viewed as an orderedset of block vectors each of which is a linear combination of k 2 t basis block vectors.
Revised: Since each block vector can be written as a linear combinationof k 2 t basis vectors of size k × k × t , this standard convolutional kernel can be viewed as an orderedset of block vectors each of which is a linear combination of k 2 t basis block vectors.

Original: Then BlkSConveliminates most basis block vectors and their corresponding coefficients to obtain an approximatedconvolutional kernel.
Revised: Then BlkSConveliminates most basis block vectors and their corresponding coefficients to obtain an approximatedconvolutional kernel.

Original: As shown on the left of Figure 1, the extreme version of BlkSConv is called thebasic BlkSConv where only one basis block vector is used.
Revised: As shown on the left of Figure 1, the extreme version of BlkSConv is called thebasic BlkSConv where only one basis block vector is used.

Original: When carefully setting the depth of theblock vector, that is the parameter t , an approximated convolution of fewer parameters can be obtainedand the corresponding compact CNN has acceptable prediction performance compared to the standardconvolutions.
Revised: When carefully setting the depth of theblock vector, that is the parameter t , an approximated convolution of fewer parameters can be obtainedand the corresponding compact CNN has acceptable prediction performance compared to the standardconvolutions.

Original: To increase the prediction accuracy of the basic BlkSConv, an enhanced version isproposed by increasing the number of basis block vectors, that is the parameter s , as shown on theright of Figure 1.
Revised: To increase the prediction accuracy of the basic BlkSConv, an enhanced version isproposed by increasing the number of basis block vectors, that is the parameter s , as shown on theright of Figure 1.

Original: However, adding too many basis block vectors will significantly increase the modelsize and computational cost.
Revised: However, adding too many basis block vectors will significantly increase the modelsize and computational cost.

Original: Thus there is a tradeoff between model efficiency/size and accuracy.
Revised: Thus there is a tradeoff between model efficiency/size and accuracy.

Original: Torealize the full potential of the enhanced BlkSConv in trading-off model efficiency/size and accuracy,we propose a framework based on the principal component analysis to search for the hyperparameterst and s of each BlkSConv layer for the given standard convolutional network.
Revised: Torealize the full potential of the enhanced BlkSConv in trading-off model efficiency/size and accuracy,we propose a framework based on the principal component analysis to search for the hyperparameterst and s of each BlkSConv layer for the given standard convolutional network.

Original: The proposed searchframework suggests a possible setting of parameters t and s such that the constructed model based onthese selected hyperparameters may achieve high prediction accuracy while simultaneously satisfyingthe constraints of model size and model efficiency in terms of MAdds.
Revised: The proposed searchframework suggests a possible setting of parameters t and s such that the constructed model based onthese selected hyperparameters may achieve high prediction accuracy while simultaneously satisfyingthe constraints of model size and model efficiency in terms of MAdds.

Original: To summarize, our main contributions are as follows.
Revised: To summarize, our main contributions are as follows.

Original: First, we develop a new convolutional layercalled BlkSConv to approximate the standard convolutional layer.
Revised: First, we develop a new convolutional layercalled BlkSConv to approximate the standard convolutional layer.

Original: To approximate a standard convolutional kernel, BlkSConv divides the kernel into blocks and approximates each block by a linearcombination of several fixed basis block vectors.
Revised: To approximate a standard convolutional kernel, BlkSConv divides the kernel into blocks and approximates each block by a linearcombination of several fixed basis block vectors.

Original: The constructed networks have small model sizeand cost fewer multiply-adds operations while maintaining acceptable prediction accuracy.
Revised: The constructed networks have small model sizeand cost fewer multiply-adds operations while maintaining acceptable prediction accuracy.

Original: Then, wealso develop a search framework to determine the block depth and the number of basis block vectorssuch that the corresponding networks with selected hyperparameters achieve comparable predictionperformance while simultaneously satisfying the constraints of model size and model efficiency.
Revised: Then, wealso develop a search framework to determine the block depth and the number of basis block vectorssuch that the corresponding networks with selected hyperparameters achieve comparable predictionperformance while simultaneously satisfying the constraints of model size and model efficiency.

Original: We also present experimental results to demonstrate the performance of selected BlkSConv-based
Revised: We also present experimental results to demonstrate the performance of selected BlkSConv-based

Original: CNNs based on our proposed hyperparameter search algorithm.
Revised: CNNs based on our proposed hyperparameter search algorithm.

Original: Our results show that selected
Revised: Our results show that selected

Original: BlkSConv-based CNNs achieve competitive performance compared with the standard convolutionalmodels for the datasets including ImageNet, CIFAR-10/100, Stanford Dogs, and Oxford Flowers.
Revised: BlkSConv-based CNNs achieve competitive performance compared with the standard convolutionalmodels for the datasets including ImageNet, CIFAR-10/100, Stanford Dogs, and Oxford Flowers.

Original:  Related Work
Revised:  Related Work

Original: Many efforts have been devoted to improve the efficiency of CNNs which could be roughly dividedinto three categories.
Revised: Many efforts have been devoted to improve the efficiency of CNNs which could be roughly dividedinto three categories.

Original: First, model pruning is a popular method to improve efficiency of CNNs.
Revised: First, model pruning is a popular method to improve efficiency of CNNs.

Original: In(6; 37), their methods remove redundancy in the trained CNN model by pruning connection.
Revised: In(6; 37), their methods remove redundancy in the trained CNN model by pruning connection.

Original: In(6; 21; 20; 35), the calculation amount of the trained model is compressed via quantization.
Revised: In(6; 21; 20; 35), the calculation amount of the trained model is compressed via quantization.

Original: In(17; 11; 16; 9; 28), model filters that have small contributions are removed and the correspondingtrained model is fine-tuned to preserve the performance.
Revised: In(17; 11; 16; 9; 28), model filters that have small contributions are removed and the correspondingtrained model is fine-tuned to preserve the performance.

Original: H
Revised: H

Original: W
Revised: W

Original: M = k k t rr M/t k k tr’r’ M/t ….
Revised: M = k k t 𝑃 ! ⁄ ($ %) (1) k k t ….

Original: Conv.
Revised: Conv.

Original: with
Revised: with

Original: N k k tr s, r s,M/t k k tr’ s, r’ s,M/t ….
Revised: N k k t k k t ….

Original: Conv.
Revised: Conv.

Original: with
Revised: with

Original: Nr 1, r 1,M/t r’ 1, r’ 1,M/t … .
Revised: N … .

Original: + + + +
Revised: + + + +

Original: [Figure]
Revised: [Figure]

Original:  layer
Revised: convolution layer

Revised: Enhanced block-wise separable convolution layer

Original: H
Revised: H

Original: W
Revised: W

Original: N 
Revised: N 𝑃 ! (!) (1) 𝑃 ’(!) (1) 𝑃 ’ ⁄ ($ %) (1) 𝐵 ! 𝑃 ! ⁄ ($ %) (𝑠) 𝑃 ’ ⁄ ($ %)

Revised: (𝑠) 𝑃 ’(!)

Revised: (𝑠) 𝑃 !(!)

Revised: (𝑠) 𝑃 ! ⁄

Revised: ($ %) (1) 𝑃 ’ ⁄ ($ %) (1) 𝑃 ’(!)

Revised: (1) 𝑃 !(!)

Revised: (1) 𝐵 !

Revised: 𝐵 (

Original: Second, many techniques are developed to factorize the standard convolutions.
Revised: Second, many techniques are developed to factorize the standard convolutions.

Original: In (30), convolutionswith larger spatial filters are factorized into two-layer convolutional architectures with smallerspatial filters.
Revised: In (30), convolutionswith larger spatial filters are factorized into two-layer convolutional architectures with smallerspatial filters.

Original: Through different combinations of depthwise convolution, pointwise convolution, andgroupwise convolution, many well-known factorizing frameworks were developed.
Revised: Through different combinations of depthwise convolution, pointwise convolution, andgroupwise convolution, many well-known factorizing frameworks were developed.

Original: In (25), the depthwise separable convolutions (DSCs) were proposed where the standard convolution is decomposedinto a depth-wise convolution and a pointwise convolution.
Revised: In (25), the depthwise separable convolutions (DSCs) were proposed where the standard convolution is decomposedinto a depth-wise convolution and a pointwise convolution.

Original: The ShuffleNets (36; 18) uses pointwisegroup convolution with channel shuffle to decompose the standard convolution.
Revised: The ShuffleNets (36; 18) uses pointwisegroup convolution with channel shuffle to decompose the standard convolution.

Original: Moreover, manylightweight models based on DSCs or groupwise convolutions such as MobileNets (8; 23; 7) and
Revised: Moreover, manylightweight models based on DSCs or groupwise convolutions such as MobileNets (8; 23; 7) and

Original: ShuffleNets (36; 18) were proposed to greatly reduce computation cost while maintaining accuracy.
Revised: ShuffleNets (36; 18) were proposed to greatly reduce computation cost while maintaining accuracy.

Original: Recently, neural architecture search-based methods (34; 32; 38; 39; 31) have been proposed toautomatically construct network architectures.
Revised: Recently, neural architecture search-based methods (34; 32; 38; 39; 31) have been proposed toautomatically construct network architectures.

Original: These methods search over a set of network hyperparameters including different types of convolutional layers and kernel sizes, to find a network structurewhich satisfies optimization constraints such as inference speed.
Revised: These methods search over a set of network hyperparameters including different types of convolutional layers and kernel sizes, to find a network structurewhich satisfies optimization constraints such as inference speed.

Original: Major search frameworks includegenetic-based methods (34) and reinforcement learning based methods (38).
Revised: Major search frameworks includegenetic-based methods (34) and reinforcement learning based methods (38).

Original: These techniques were [Figure] used in state-of-the-art CNN architectures such as MnasNet (31) and MobileNetV3 (7).
Revised: These techniques wereused in state-of-the-art CNN architectures such as MnasNet (31) and MobileNetV3 (7).

Original: Convolution weights of trained CNNs are also analyzed in (1; 3; 26; 4).
Revised: Convolution weights of trained CNNs are also analyzed in (1; 3; 26; 4).

Original: Following their analysis, several approaches toward reducing redundant weights were proposed.
Revised: Following their analysis, several approaches toward reducing redundant weights were proposed.

Original: In (2; 12; 13), the convolutionalkernels are approximated via low-rank factorization.
Revised: In (2; 12; 13), the convolutionalkernels are approximated via low-rank factorization.

Original: In (4), the kernels are analyzed via principalcomponent analysis.
Revised: In (4), the kernels are analyzed via principalcomponent analysis.

Original:  Block-wise Separable Convolutions (BlkSConv)
Revised:  Block-wise Separable Convolutions (BlkSConv)

Original: For any natural number n , let [ n ] denote the set { 1 , 2 , . .
Revised: For any natural number n , let [ n ] denote the set { 1 , 2 , . .

Original: . , n } .
Revised: . , n } .

Original: In a standard CNN, each convolutionallayer converts an input tensor I of size M ×
Revised: In a standard CNN, each convolutionallayer converts an input tensor I of size M ×

Original: X × Y into an output tensor O of size N × X ×
Revised: X × Y into an output tensor O of size N × X ×

Original: Yby applying the filter kernels F 1 , F 2 , . .
Revised: Yby applying the filter kernels F 1 , F 2 , . .

Original: F N , each of size M × ℓ × ℓ with odd ℓ such that, for anyx, y, j ∈
Revised: F N , each of size M × ℓ × ℓ with odd ℓ such that, for anyx, y, j ∈

Original: [ X ] × [ Y ] × [ N ] ,
Revised: [ X ] × [ Y ] × [ N ] ,

Revised: O ( x, y, j ) =

Original: [Equation]
Revised: ( ℓ − 1) / 2 (cid:88) s 1 = − ( ℓ − 1) / ( ℓ − 1) / 2 (cid:88) s 2 = − ( ℓ − 1) / M (cid:88) s 3 = I ( x + s 1 , y + s 2 , s 3 ) · F j ( s 1 , s 2 , s 3 ) .

Original: During training, the weights of each kernel F j are optimized via backpropagation.
Revised: During training, the weights of each kernel F j are optimized via backpropagation.

Original: The total numberof weight parameters to be optimized in each kernel F j is ℓ 2 · M .
Revised: The total numberof weight parameters to be optimized in each kernel F j is ℓ 2 · M .

Original: In the subsequent work, we proposea framework to reduce the number of parameters of the standard convolutions while preserving itsprediction performance.
Revised: In the subsequent work, we proposea framework to reduce the number of parameters of the standard convolutions while preserving itsprediction performance.

Original: Then, in order to implement our new framework, we adopt a combination ofpointwise and group-wise convolutions to efficiently realize the reduced convolutions.
Revised: Then, in order to implement our new framework, we adopt a combination ofpointwise and group-wise convolutions to efficiently realize the reduced convolutions.

Original: Combiningthese ideas, we introduce block-wise separable convolutions, denoted by BlkSConv .
Revised: Combiningthese ideas, we introduce block-wise separable convolutions, denoted by BlkSConv .

Original: However,to generate a BlkSConv-based models, many hyperparameters should be determined for keepingprediction performance, model size, and model efficiency.
Revised: However,to generate a BlkSConv-based models, many hyperparameters should be determined for keepingprediction performance, model size, and model efficiency.

Original: Thus, we also propose an efficienthyperparameter search algorithm to select hyperparameters satisfying the given model constraints.
Revised: Thus, we also propose an efficienthyperparameter search algorithm to select hyperparameters satisfying the given model constraints.

Original: 3.1 Expressing a standard convolution via a linear combination of block vectors
Revised: 3.1 Expressing a standard convolution via a linear combination of block vectors

Original: In this section, we propose block-wise separable convolutions.
Revised: In this section, we propose block-wise separable convolutions.

Original: First, each convolutional kernel F jof size M × ℓ × ℓ can be expressed as a concatenation of M/t blocks Q (1) j , Q (2) j , . . .
Revised: First, each convolutional kernel F jof size M × ℓ × ℓ can be expressed as a concatenation of M/t blocks Q (1) j , Q (2) j , . . .

Original: , Q ( M/t )
Revised: , Q ( M/t )

Original: j eachof size ℓ × ℓ × t where Q ( k ) j ( x, y, z ) =
Revised: j eachof size ℓ × ℓ × t where Q ( k ) j ( x, y, z ) =

Original: F j ( x, y, z + ( k − 1) t ) for any x, y, z ∈
Revised: F j ( x, y, z + ( k − 1) t ) for any x, y, z ∈

Original: [ X ] × [ Y ] × [ t ] .
Revised: [ X ] × [ Y ] × [ t ] .

Original: We call t the block depth.
Revised: We call t the block depth.

Original: Let { B 1 , B 2 , . . .
Revised: Let { B 1 , B 2 , . . .

Original: , B tℓ 2 } be a set of basis block vectors.
Revised: , B tℓ 2 } be a set of basis block vectors.

Original: Each Q ( k ) j canbe expressed uniquely as a linear combination of B 1 , B 2 , . . .
Revised: Each Q ( k ) j canbe expressed uniquely as a linear combination of B 1 , B 2 , . . .

Original: , B tℓ 2 , that is, there exist tℓ 2 values
Revised: , B tℓ 2 , that is, there exist tℓ 2 values

Original: P ( k ) j ( i ) ∈
Revised: P ( k ) j ( i ) ∈

Original: R such that Q ( k ) j = (cid:80)
Revised: R such that Q ( k ) j = (cid:80)

Original: tℓ 2 i =1 P ( k ) j
Revised: tℓ 2 i =1 P ( k ) j

Original: ( i ) · B i .
Revised: ( i ) · B i .

Original: In practice, tℓ 2 may be large.
Revised: In practice, tℓ 2 may be large.

Original: In order to reducethe model size, we require the number of basis block vectors is fewer than or equal to a fixed numbers with s < tℓ 2 .
Revised: In order to reducethe model size, we require the number of basis block vectors is fewer than or equal to a fixed numbers with s < tℓ 2 .

Original: Now each Q ( k ) j is replaced by the following linear combination of B 1 , . . .
Revised: Now each Q ( k ) j is replaced by the following linear combination of B 1 , . . .

Original: , B s , thatis (cid:98) Q ( k )
Revised: , B s , thatis (cid:98) Q ( k )

Original: j = (cid:80) si =1 P ( k ) j ( i ) · B i .
Revised: j = (cid:80) si =1 P ( k ) j ( i ) · B i .

Original: The corresponding convolutional kernel (cid:98) F j is the concatenation of
Revised: The corresponding convolutional kernel (cid:98) F j is the concatenation of

Original: M/t blocks (cid:98) Q (1) j , . . .
Revised: M/t blocks (cid:98) Q (1) j , . . .

Original: , (cid:98) Q ( M/t ) j .
Revised: , (cid:98) Q ( M/t ) j .

Original: Therefore, the corresponding output tensor is(cid:98)
Revised: Therefore, the corresponding output tensor is(cid:98)

Original: O ( x, y, j ) =
Revised: O ( x, y, j ) =

Original: ( ℓ − 1) / 2 (cid:88) s 1 ,s 2 = − ( ℓ − 1) /
Revised: ( ℓ − 1) / 2 (cid:88) s 1 ,s 2 = − ( ℓ − 1) /

Original: M (cid:88) s 3 =
Revised: M (cid:88) s 3 =

Original: I ( x + s 1 , y +
Revised: I ( x + s 1 , y +

Original: [Equation]
Revised: [Equation]

Original: By Equation 2, the number of weight parameters in BlkSConv is s · ( t · ℓ 2 + Mt ) .
Revised: By Equation 2, the number of weight parameters in BlkSConv is s · ( t · ℓ 2 + Mt ) .

Original: To significantly reducemodel size, we set s = 1 .
Revised: To significantlyreduce model size, we set s = 1 .

Original: The top of Figure 1  illustrates the operation of BlkSConv when s = 1 .
Revised: Figure 1 left illustrates the operation of BlkSConv when s = 1 .

Original: In order to achieve the minimal model size, t can be set as √ M/ℓ and the number of parametersbecomes 2 ℓ √ M while the parameter number of the standard and 1 × 1 pointwise convolutions are
Revised: In order to achieve the minimal model size, t can be set as √ M/ℓ and the number of parametersbecomes 2 ℓ √ M while the parameter number of the standard and 1 × 1 pointwise convolutions are

Original: Mℓ 2 and M , respectively.
Revised: Mℓ 2 and M , respectively.

Original: Thus, the constructed BlkSConv-based CNNs have smaller model sizethan existing lightweight CNN models.
Revised: Thus, the constructed BlkSConv-based CNNs have smaller model sizethan existing lightweight CNN models.

Original: Take the ResNet34 (10) as an example where, in the laststage of the ResNet-34, the convolutional kernel size is 3 × 3 and the channel size is 512, that is ℓ =
Revised: Take the ResNet34 (10) as an example where, in the laststage of the ResNet-34, the convolutional kernel size is 3 × 3 and the channel size is 512, that is ℓ =

Original: 3and M = 512 .
Revised: 3and M = 512 .

Original: In this case, the ratio between the parameter size of the BlkSConv-based convolutionsand the standard convolutions is approximately 0.0295.
Revised: In this case, the ratio between the parameter size of the BlkSConv-based convolutionsand the standard convolutions is approximately 0.0295.

Original: However, the prediction performance of the BlkSConv-based CNN with the smallest model size isusually worse than the standard CNNs.
Revised: However, the prediction performance of the BlkSConv-based CNN with the smallest model size isusually worse than the standard CNNs.

Original: To increase accuracy, the number of basis block vectorsshould be increased, that is s > 1 .
Revised: To increase accuracy, the number of basis block vectorsshould be increased, that is s > 1 .

Original: The bottom of Figure 1  illustrates the operation of BlkSConvwhen s > 1 .
Revised: Figure 1 right illustrates the operation of BlkSConv when s > 1 .

Original: In this case, the number of parameters becomes 2 sℓ √ M .
Revised: In this case, the number of parameters becomes 2 sℓ √ M .

Original: Let us take convolutions in thelast stage of ResNet-34 as examples.
Revised: Let us take convolutions in the last stageof ResNet-34 as examples.

Original: Let us set t = 4 in the BlkSConv.
Revised: Let us set t = 4 in the BlkSConv.

Original: Now the ratio between theparameter size of the BlkSConv-based convolutions and the standard convolutions is approximately0.0356.
Revised: Now the ratio between the parametersize of the BlkSConv-based convolutions and the standard convolutions is approximately 0.0356.

Original: Thus we can add at least 5 basis block vectors to increase prediction accuracy.
Revised: Thus we can add at least 5 basis block vectors to increase prediction accuracy.

Original: In this case,the ratio between the parameter size of the BlkSConv-based convolutions with 5 basis block vectorsand the standard convolutions is approximately 0.178.
Revised: In this case, the ratiobetween the parameter size of the BlkSConv-based convolutions with 5 basis block vectors and thestandard convolutions is approximately 0.178.

Original: In the experimental section, we demonstratethat the BlkSConv-based convolutions with few basis block vectors have prediction performanceas well as  the standard convolutions on ImageNet or even outperform the standard convolutions onseveral datasets when the backbone CNNs are ResNets.
Revised: In the experimental section, we demonstrate that the BlkSConv-based convolutions with few basis block vectors have prediction performance as wellas the standard convolutions on ImageNet or even outperform the standard convolutions on severaldatasets when the backbone CNNs are ResNets.

Original: The next problem is the computational efficiency of BlkSConv.
Revised: The next problem is the computational efficiency of BlkSConv.

Original: If we compute the kernel (cid:98) F j first andperform a regular convolution according to the kernel (cid:98) F j , then it is obvious that the computationalcost is larger than the cost for just performing a standard convolution.
Revised: If we compute the kernel (cid:98) F j first andperform a regular convolution according to the kernel (cid:98) F j , then it is obvious that the computationalcost is larger than the cost for just performing a standard convolution.

Original: We will address this problemin the subsequent section.
Revised: We will address this problemin the subsequent section.

Original: 3.2 Implementation of BlkSConv via a combination of pointwise and group-wise convolutions
Revised: 3.2 Implementation of BlkSConv via a combination of pointwise and group-wise convolutions

Original: In this section, we propose an efficient implementation method to realize BlkSConv.
Revised: In this section, we propose an efficient implementation method to realize BlkSConv.

Original: The flowchartof the proposed implementation is illustrated in Figure 2.
Revised: The flowchartof the proposed implementation is illustrated in Figure 2.

Original: To derive an efficient implementation for
Revised: To derive an efficient implementation for

Original: BlkSConv operation, we rewrite Equation 2 as follows.
Revised: BlkSConv operation, we rewrite Equation 2 as follows.

Original: I
Revised: I

Original: X
Revised: X

Original: Y
Revised: Y

Original: I(  ,  ,z)I(  ,  ,z+t) I(  ,  ,z+2t)
Revised: I(  ,  ,z)I(  ,  ,z+t) I(  ,  ,z+2t)

Original: I(  ,  ,z+(k-1)t) ሚ𝐼 𝑧
Revised: I(  ,  ,z+(k-1)t) ሚ𝐼 𝑧

Original: X
Revised: X

Original: M
Revised: M

Original: Concatenated tensor with t tensors ሚ𝐼 𝑧 ሚ𝐼 𝑡 ሚ𝐼 𝑡− … .
Revised: Concatenated tensor with t tensors ሚ𝐼 𝑧 ሚ𝐼 𝑡 ሚ𝐼 𝑡− … .

Original: M/t ሚ𝐼ሚ𝐼s 1  1 conv.
Revised: M/t ሚ𝐼ሚ𝐼s 1  1 conv.

Original: Step a 𝐽 (𝑡) 𝐽 (𝑡−1) … .
Revised: Step a 𝐽 (𝑡) 𝐽 (𝑡−1) … .

Original: 𝐽 (2) 𝐽 (1) tY
Revised: 𝐽 (2) 𝐽 (1) tY

Original: X tY
Revised: X tY

Original: X s
Revised: X s

Original: Rearrange 𝐽 𝑠
Revised: Rearrange 𝐽 𝑠

Original: 𝐽 𝑠−1(.) … .
Revised: 𝐽 𝑠−1(.) … .

Original: 𝐽
Revised: 𝐽

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: (.) sY
Revised: (.) sY

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: M/t
Revised: M/t

Original: Y
Revised: Y

Original: [Figure]
Revised: [Figure]

Original: Y
Revised: Y

Original: [Figure]
Revised: [Figure]

Original: (cid:98) O ( x, y, j ) = (cid:88) s 1 ,st (cid:88)
Revised: (cid:98) O ( x, y, j ) = (cid:88) s 1 ,st (cid:88)

Original: z =
Revised: z =

Original: M/t (cid:88) k
Revised: M/t (cid:88) k

Original: =
Revised: =

Original: I ( x + s 1 , y + s 2 , z + ( k
Revised: I ( x + s 1 , y + s 2 , z + ( k

Original: [Equation]
Revised: [Equation]

Original: 1) t ) · (cid:98) Q ( k )
Revised: 1) t ) · (cid:98) Q ( k )

Original: j
Revised: j

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: , z ) (3) =
Revised: , z ) (3) =

Original: (cid:88) s 1 ,st (cid:88)
Revised: (cid:88) s 1 ,st (cid:88)

Original: z =
Revised: z =

Original: M/t (cid:88) k
Revised: M/t (cid:88) k

Original: =
Revised: =

Original: I ( x + s 1 , y + s 2 , z + ( k − 1) t ) · s (cid:88) i =
Revised: I ( x + s 1 , y + s 2 , z + ( k − 1) t ) · s (cid:88) i =

Original: P ( k ) j ( i ) · B i ( s 1 , s 2 , z ) (4) = s (cid:88) i = (cid:88) s 1 ,st (cid:88)
Revised: P ( k ) j ( i ) · B i ( s 1 , s 2 , z ) (4) = s (cid:88) i = (cid:88) s 1 ,st (cid:88)

Original: [Equation]
Revised: [Equation]

Original: J
Revised: J

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: x, y, i ) : a point-wise convolution of (cid:101)
Revised: x, y, i ) : a point-wise convolution of (cid:101)

Original: I z .
Revised: I z .

Original: Let (cid:101)
Revised: Let (cid:101)

Original: I z ( x, y, k ) be a tensor of size X × Y × M/t defined by (cid:101)
Revised: I z ( x, y, k ) be a tensor of size X × Y × M/t defined by (cid:101)

Original: I z ( x, y, k ) ≜
Revised: I z ( x, y, k ) ≜

Original: I ( x, y, z + ( k − 1) t ) .
Revised: I ( x, y, z + ( k − 1) t ) .

Original: Wedefine J ( z ) ( x, y, i ) ≜ (cid:80) M/tk =1 P ( k ) j
Revised: Wedefine J ( z ) ( x, y, i ) ≜ (cid:80) M/tk =1 P ( k ) j

Original: ( i ) · (cid:101)
Revised: ( i ) · (cid:101)

Original: I
Revised: I

Original: z ( x + s 1 , y + s 2 , k ) which is a point-wise convolution of (cid:101)
Revised: z ( x + s 1 , y + s 2 , k ) which is a point-wise convolution of (cid:101)

Original: I z .
Revised: I z .

Original: Next, we define J i ( x, y, z ) ≜
Revised: Next, we define J i ( x, y, z ) ≜

Original: J ( z ) ( x, y, i ) and let J be the reshaped tensor which is the concatenationof J 1 , . .
Revised: J ( z ) ( x, y, i ) and let J be the reshaped tensor which is the concatenationof J 1 , . .

Original: , J s , that is J (
Revised: , J s , that is J (

Original: x, y, z
Revised: x, y, z

Original: + ( i − 1) t ) =
Revised: + ( i − 1) t ) =

Original: J i ( x, y, z ) .
Revised: J i ( x, y, z ) .

Original: Now Equation 5 can be rewritten as
Revised: Now Equation 5 can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Finally, Equation 6 is just a group-wise convolution of the tensor J with s groups.
Revised: Finally, Equation 6 is just a group-wise convolution of the tensor J with s groups.

Original: Let us compute the computational cost (MAdds) of the implementation for BlkSConv.
Revised: Let us compute the computational cost (MAdds) of the implementation for BlkSConv.

Original: By Equation 5(Step a in Figure 2), the computational cost of s point-wise convolutions on the concatenation of(cid:101)
Revised: By Equation 5(Step a in Figure 2), the computational cost of s point-wise convolutions on the concatenation of(cid:101)

Original: I 1 , . .
Revised: I 1 , . .

Original: , (cid:101) I t is sXY M .
Revised: , (cid:101) I t is sXY M .

Original: In addition, by Equation 6 (Step b in Figure 2), the computational cost of thegroup-wise convolution on the tensor J is sXY tℓ 2 .
Revised: In addition, by Equation 6 (Step b in Figure 2), the computational cost of thegroup-wise convolution on the tensor J is sXY tℓ 2 .

Original: Finally, the computational cost of the point-wisesummation in the last step is sXY .
Revised: Finally, the computational cost of the point-wisesummation in the last step is sXY .

Original: The total MAdds of a BlkSConv operation is sXY ( M + tℓ 2 + 1)while the MAdds of a standard convolution is XY Mℓ 2 .
Revised: The total MAdds of a BlkSConv operation is sXY ( M + tℓ 2 + 1)while the MAdds of a standard convolution is XY Mℓ 2 .

Original: Again, let us take convolutions in the laststage of ResNet-34 as examples.
Revised: Again, let us take convolutions in the laststage of ResNet-34 as examples.

Original: We set s = 5 and t = 4 as the hyperparameters of the BlkSConvbased convolution.
Revised: We set s = 5 and t = 4 as the hyperparameters of the BlkSConvbased convolution.

Original: Now the ratio between the MAdds of a BlkSConv-based convolution and astandard convolution is approximately 0.595.
Revised: Now the ratio between the MAdds of a BlkSConv-based convolution and astandard convolution is approximately 0.595.

Original: Thus the proposed BlkSConv operation is much moreefficient than the standard convolution in practical cases.
Revised: Thus the proposed BlkSConv operation is much moreefficient than the standard convolution in practical cases.

Original: We remark that the proposed implementationrequires much GPU memory due to using many group-wise and pointwise convolutions.
Revised: We remark that the proposed implementationrequires much GPU memory due to using many group-wise and pointwise convolutions.

Original: 3.3 Hyperparameter search via principal component analysis
Revised: 3.3 Hyperparameter search via principal component analysis

Original: Designing a BlkSConv-based CNN involves hyperparameters including the block depth and the number of basis block vectors in each convolutional layer that affect the performance of the corresponding
Revised: Designing a BlkSConv-based CNN involves hyperparameters including the block depth and the number of basis block vectors in each convolutional layer that affect the performance of the corresponding

Original: CNN model.
Revised: CNN model.

Original: To realize an efficient BlkSConv-based CNN, we conduct a hyperparameter search algorithm based on principal component analysis of trained CNNs.
Revised: To realize an efficient BlkSConv-based CNN, we conduct a hyperparameter search algorithm based on principal component analysis of trained CNNs.

Original: Given a trained CNN, the algorithmgenerates the block depth and the number of basis block vectors for each standard convolutional layer of the trained CNN in the following way.
Revised: Given a trained CNN, the algorithmgenerates the block depth and the number of basis block vectors for each standard convolutional layer of the trained CNN in the following way.

Original: First, for each individual ℓ × ℓ × M kernel K of the trained
Revised: First, for each individual ℓ × ℓ × M kernel K of the trained

Original: CNN where we assume that M = 2 α for some α ∈ N , the kernel K is partitioned into M/t blockvectors B 1 , B 2 , . . .
Revised: CNN where we assume that M = 2 α for some α ∈ N , the kernel K is partitioned into M/t blockvectors B 1 , B 2 , . . .

Original: , B M/t each of size ℓ × ℓ × t with t ∈ { 1 , 2 , . . .
Revised: , B M/t each of size ℓ × ℓ × t with t ∈ { 1 , 2 , . . .

Original: , 2 β } for some integer β < α .
Revised: , 2 β } for some integer β < α .

Original: Next, we perform principal component analysis (PCA) on the set { B 1 , B 2 , . . .
Revised: Next, we perform principal component analysis (PCA) on the set { B 1 , B 2 , . . .

Original: , B M/t } .
Revised: , B M/t } .

Original: Then, fora fixed integer γ and, for each q ∈ { 1 , 2 , . . .
Revised: Then, fora fixed integer γ and, for each q ∈ { 1 , 2 , . . .

Original: , γ } , the algorithm computes the variance V t,q of thekernel K which is explained by the first q principal components PC1,PC2,..,PCq.
Revised: , γ } , the algorithm computes the variance V t,q of thekernel K which is explained by the first q principal components PC1,PC2,..,PCq.

Original: In addition, let
Revised: In addition, let

Original: CC t,q and MS t,q denote the MAdds and the model size of the BlkSConv under the setting that theblock depth is t and the number of basis block vectors is q , respectively.
Revised: CC t,q and MS t,q denote the MAdds and the model size of the BlkSConv under the setting that theblock depth is t and the number of basis block vectors is q , respectively.

Original: Note that the MAdds and themodel size of the standard convolution is exactly CC M, 1 and MS M, 1 , respectively.
Revised: Note that the MAdds and themodel size of the standard convolution is exactly CC M, 1 and MS M, 1 , respectively.

Original: After computingall V t,q , CC t,q , and MS t,q , the algorithm generates the feasible set
Revised: After computingall V t,q , CC t,q , and MS t,q , the algorithm generates the feasible set

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , and
Revised: , and

Original: [Equation]
Revised: [Equation]

Original: for fixed positive constants α v , α c , α s ∈ (0 , 1) .
Revised: for fixed positive constants α v , α c , α s ∈ (0 , 1) .

Original: Finally, the algorithm chooses the hyperparameter( t, q ) from H α v ,α c ,α s according to the computational cost or the model size.
Revised: Finally, the algorithm chooses the hyperparameter( t, q ) from H α v ,α c ,α s according to the computational cost or the model size.

Original: On one hand, note that the goal of BlkSConv is to maintain the prediction performance of the trainedstandard CNN.
Revised: On one hand, note that the goal of BlkSConv is to maintain the prediction performance of the trainedstandard CNN.

Original: In general, the prediction accuracy is proportional to the model size of the constructed
Revised: In general, the prediction accuracy is proportional to the model size of the constructed

Original: CNN.
Revised: CNN.

Original: Therefore, in this sense, we choose the hyperparameters (ˆ t, ˆ q ) from H α v ,α c ,α s such that theconstructed BlkSConv has the largest parameter size, that is
Revised: Therefore, in this sense, we choose the hyperparameters (ˆ t, ˆ q ) from H α v ,α c ,α s such that theconstructed BlkSConv has the largest parameter size, that is

Original: [Equation]
Revised: [Equation]

Original: One can expect that the generated BlkSConv-based CNN has nice prediction performance comparedto the original CNN with standard convolutions.
Revised: One can expect that the generated BlkSConv-based CNN has nice prediction performance comparedto the original CNN with standard convolutions.

Original: On the other hand, one of the advantage of BlkSConv operations is that BlkSConv can greatly reducethe model size of the original standard CNN.
Revised: On the other hand, one of the advantage of BlkSConv operations is that BlkSConv can greatly reducethe model size of the original standard CNN.

Original: Thus, in this sense, we can select the hyperparameters(˜ t, ˜ q ) from H α v ,α c ,α s such that the constructed BlkSConv has the smallest parameter size, that is
Revised: Thus, in this sense, we can select the hyperparameters(˜ t, ˜ q ) from H α v ,α c ,α s such that the constructed BlkSConv has the smallest parameter size, that is

Original: [Equation]
Revised: [Equation]

Original: However, the prediction performance may degrade when the parameter size of the BlkSConv-basedmodel decreases.
Revised: However, the prediction performance may degrade when the parameter size of the BlkSConv-basedmodel decreases.

Original: We will demonstrate in the experimental section that the BlkSConv-based CNNsgenerated according to Equation 8 also have acceptable prediction accuracy compared to the standard
Revised: We will demonstrate in the experimental section that the BlkSConv-based CNNsgenerated according to Equation 8 also have acceptable prediction accuracy compared to the standard

Original: CNNs.
Revised: CNNs.

Original: In summary, both Equation 8 and Equation 9 provide ways to determine hyperparameters from thefeasible set H α v ,α c ,α s such that corresponding BlkSConv-based CNNs have smaller model size and 214 fewer multiply-adds operations than the original CNN with standard convolutions.
Revised: In summary, both Equation 8 and Equation 9 provide ways to determine hyperparameters from thefeasible set H α v ,α c ,α s such that corresponding BlkSConv-based CNNs have smaller model size and 214 fewer multiply-adds operations than the original CNN with standard convolutions.

Original: Finally, let us consider the extreme case that two constants β and γ are set by β = 0 and γ = 1 .
Revised: Finally, let us consider the extreme case that two constants β and γ are set by β = 0 and γ = 1 .

Original: Letus further set the search parameter α v = 0 .
Revised: Letus further set the search parameter α v = 0 .

Original: Under this restricted search condition, the cardinality ofthe feasible set H 0 ,α c ,α s is always 1.
Revised: Under this restricted search condition, the cardinality ofthe feasible set H 0 ,α c ,α s is always 1.

Original: Thus the outputs of Equation 8 and Equation 9 are the same.
Revised: Thus the outputs of Equation 8 and Equation 9 are the same.

Original: In fact, the resulting BlkSConv-based CNN is exactly the same as the CNN where the standardconvolutions are replaced by the blueprint separable convolutions previously developed in (4).
Revised: In fact, the resulting BlkSConv-based CNN is exactly the same as the CNN where the standardconvolutions are replaced by the blueprint separable convolutions previously developed in (4).

Original:  Experiments
Revised:  Experiments

Original: We evaluate BlkSConv and the proposed hyperparameter architecture search algorithm combiningwith ResNet-10, ResNet-18, and ResNet-26 (5) on ImageNet (22), Stanford Dogs, (14), and Oxford102 Flowers (19).
Revised: We evaluate BlkSConv and the proposed hyperparameter architecture search algorithm combiningwith ResNet-10, ResNet-18, and ResNet-26 (5) on ImageNet (22), Stanford Dogs, (14), and Oxford102 Flowers (19).

Original: The proposed methods are also evaluated combining with ResNet-20 and ResNet-56on CIFAR 10/100 (15).
Revised: The proposed methods are also evaluated combining with ResNet-20 and ResNet-56on CIFAR 10/100 (15).

Original: 4.1 Hyperparameter Search Details
Revised: 4.1 Hyperparameter Search Details

Original: We apply the PCA-based hyperparameter search algorithm (HSA) developed in Section 3.3 on severalvariants of ResNet models.
Revised: We apply the PCA-based hyperparameter search algorithm (HSA) developed in Section 3.3 on severalvariants of ResNet models.

Original: In the first part, we consider the large-scale classification scenarios.
Revised: In the first part, we consider the large-scale classification scenarios.

Original: Several standard ResNets are trained on ImageNet first and their architectures are shown in Table 1.
Revised: Several standard ResNets are trained on ImageNet first and their architectures are shown in Table 1.

Original: The HSA for searching BlkSConv architectures is only applied to conv3_x, conv4_x, conv5_xlayers of these standard ResNets.
Revised: The HSA for searching BlkSConv architectures is only applied to conv3_x, conv4_x, conv5_xlayers of these standard ResNets.

Original: Next, the search hyperparameters α v , α c , α s are set as 0.5 or
Revised: Next, the search hyperparameters α v , α c , α s are set as 0.5 or

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 75 , max)
Revised: 75 , max)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: It is possible that the feasible set H α v ,α c ,α s is empty.
Revised: It is possible that the feasible set H α v ,α c ,α s is empty.

Original: In this case, the corresponding standardconvolutional layer is not replaced and is denoted by conv as shown in Table 1.
Revised: In this case, the corresponding standardconvolutional layer is not replaced and is denoted by conv as shown in Table 1.

Original: Moreover, theproposed HSA has two selection strategies: one is based on the largest parameter size, denoted by
Revised: Moreover, theproposed HSA has two selection strategies: one is based on the largest parameter size, denoted by

Original: SS = max , and the other is based on the smallest parameter size, denoted by SS = min as shown in
Revised: SS = max , and the other is based on the smallest parameter size, denoted by SS = min as shown in

Original: Table 2.
Revised: Table 2.

Original: The selected BlkSConv, s i t j , which means i basis block vectors and j depth of the blocks.
Revised: The selected BlkSConv, s i t j , which means i basis block vectors and j depth of the blocks.

Original: The feasible set H α v ,α c ,α s is likely to be empty when the parameter α v is large.
Revised: The feasible set H α v ,α c ,α s is likely to be empty when the parameter α v is large.

Original: In the case that α v islarge, it often requires many principal components to accumulate enough explained variance and thusthis causes large numbers of parameters or MAdds.
Revised: In the case that α v islarge, it often requires many principal components to accumulate enough explained variance and thusthis causes large numbers of parameters or MAdds.

Original: Therefore, the feasible set H α v ,α c ,α s is probablyempty when we further require small α c and α s .
Revised: Therefore, the feasible set H α v ,α c ,α s is probablyempty when we further require small α c and α s .

Original: On the other hand, the parameter α v cannot be toosmall because the prediction performance of the network is highly proportional to the amount of theaccumulated variance as discussed in Section 3.3 where we will demonstrate it in the ablation studyof this section.
Revised: On the other hand, the parameter α v cannot be toosmall because the prediction performance of the network is highly proportional to the amount of theaccumulated variance as discussed in Section 3.3 where we will demonstrate it in the ablation studyof this section.

Original: For the above reason, we only present the results for ResNet-18 and ResNet-26 on
Revised: For the above reason, we only present the results for ResNet-18 and ResNet-26 on

Original: ImageNet under the setting that α v = 0 .
Revised: ImageNet under the setting that α v = 0 .

Original: which are shown in Table 2.
Revised: which are shown in Table 2.

Original: In the second part, we consider the small-scale classification on CIFAR10/100.
Revised: In the second part, we consider the small-scale classification on CIFAR10/100.

Original: We use the standard
Revised: We use the standard

Original: ResNet-20 and ResNet-56 as the experimental models where the architectures are slightly modifiedto suit the small-scale images.
Revised: ResNet-20 and ResNet-56 as the experimental models where the architectures are slightly modifiedto suit the small-scale images.

Original: The proposed HSA is only applied to conv4_x layers of these twostandard ResNets.
Revised: The proposed HSA is only applied to conv4_x layers of these twostandard ResNets.

Original: More BlkSConv-based architecture search results can be found in the appendix.
Revised: More BlkSConv-based architecture search results can be found in the appendix.

Original: 4.2 Performance on large-scale classification: ImageNet
Revised: 4.2 Performance on large-scale classification: ImageNet

Original: To evaluate the performance of BlkSConv-based models in large-scale recognition, we conductexperiments on ImageNet(22).
Revised: To evaluate the performance of BlkSConv-based models in large-scale recognition, we conductexperiments on ImageNet(22).

Original: Each model takes 3 days to be trained on a single GPU (Nvidia
Revised: Each model takes 3 days to be trained on a single GPU (Nvidia

Original: Tesla V100).
Revised: Tesla V100).

Original: ImageNet contains nearly 1.3M training images and 50,000 testing images.
Revised: ImageNet contains nearly 1.3M training images and 50,000 testing images.

Original: For
Revised: For

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the experimental setup, ResNet-10, ResNet-18, and ResNet-26 are trained on ImageNet under thefollowing setting.
Revised: the experimental setup, ResNet-10, ResNet-18, and ResNet-26 are trained on ImageNet under thefollowing setting.

Original: The number of epochs is 100 and the batch size is 256.
Revised: The number of epochs is 100 and the batch size is 256.

Original: SGD is used as theoptimizer and the initial learning rate, the momentum, and the weight decay are set to 0.1, 0.9, and10 − 4 , respectively.
Revised: SGD is used as theoptimizer and the initial learning rate, the momentum, and the weight decay are set to 0.1, 0.9, and10 − 4 , respectively.

Original: The learning rate is scheduled to decay by a factor of 0 . 1 at epochs 30, 60, and 90.
Revised: The learning rate is scheduled to decay by a factor of 0 . 1 at epochs 30, 60, and 90.

Original: We augment the data via random resized crop to 224px and random horizontal flip.
Revised: We augment the data via random resized crop to 224px and random horizontal flip.

Original: The performanceresults are shown in Table 2, More experimental results can be found in the appendix.
Revised: The performanceresults are shown in Table 2, More experimental results can be found in the appendix.

Original: On one hand, let us focus the cases that α v = 0 .
Revised: On one hand, let us focus the cases that α v = 0 .

Original: 5 and SS = max in Table 2.
Revised: 5 and SS = max in Table 2.

Original: The predictionaccuracies of the selected BlkSConv-based models and the standard model are close within 1%.
Revised: The predictionaccuracies of the selected BlkSConv-based models and the standard model are close within 1%.

Original: It confirms our expectation that BlkSConv-based models have smaller parameter sizes and fewer
Revised: It confirms our expectation that BlkSConv-based models have smaller parameter sizes and fewer

Original: MAdds than standard models while preserving prediction performance if the proposed HSA adopts aselection strategy based on the maximum parameter size.
Revised: MAdds than standard models while preserving prediction performance if the proposed HSA adopts aselection strategy based on the maximum parameter size.

Original: On the other hand, let us consider the case that α v = 0 .
Revised: On the other hand, let us consider the case that α v = 0 .

Original: 5 and SS = min in Table 2.
Revised: 5 and SS = min in Table 2.

Original: The parametersand MAdds of the BlkSConv-based models are only 12.6% and 29.8% of the standard modelwhile the gap of their prediction accuracies is about 3%.
Revised: The parametersand MAdds of the BlkSConv-based models are only 12.6% and 29.8% of the standard modelwhile the gap of their prediction accuracies is about 3%.

Original: We adopt an interesting way based onrestricting the parameter size and MAdds to interpret the advantage of the generated BlkSConv-basedmodels where the selection strategy SS is set as min .
Revised: We adopt an interesting way based onrestricting the parameter size and MAdds to interpret the advantage of the generated BlkSConv-basedmodels where the selection strategy SS is set as min .

Original: We also compare the standard ResNet-10, the
Revised: We also compare the standard ResNet-10, the

Original: BlkSConv-based ResNet-18, and the BlkSConv-based ResNet-26 in Table 3 where the parametersizes or MAdds of three given models are similar.
Revised: BlkSConv-based ResNet-18, and the BlkSConv-based ResNet-26 in Table 3 where the parametersizes or MAdds of three given models are similar.

Original: The BlkSConv-based ResNet-26 with parameter(0 .
Revised: The BlkSConv-based ResNet-26 with parameter(0 .

Original: 5 , min) and the BlkSConv-based ResNet-18 with parameter (0 . 5 , 0 . 5 , 0 .
Revised: 5 , min) and the BlkSConv-based ResNet-18 with parameter (0 . 5 , 0 . 5 , 0 .

Original: 5 , max) greatlyoutperform the standard
Revised: 5 , max) greatlyoutperform the standard

Original: ResNet-10 where both the BlkSConv-based models lead to an accuracy gainof at least 6.5%.
Revised: ResNet-10 where both the BlkSConv-based models lead to an accuracy gainof at least 6.5%.

Original: In addition, the BlkSConv-based ResNet-26 with parameter (0 . 5 , 0 . 5 , 0 .
Revised: In addition, the BlkSConv-based ResNet-26 with parameter (0 . 5 , 0 . 5 , 0 .

Original: 5 , min) onlyhas half the parameter size of the BlkSConv-based ResNet-18 with parameter (0 . 5 , 0 . 5 , 0 .
Revised: 5 , min) onlyhas half the parameter size of the BlkSConv-based ResNet-18 with parameter (0 . 5 , 0 . 5 , 0 .

Original: 5 , max) .
Revised: 5 , max) .

Original: 4.3 Performance on small-scale classification: CIFAR 10/100
Revised: 4.3 Performance on small-scale classification: CIFAR 10/100

Original: The performance results are shown in Table 4.
Revised: The performance results are shown in Table 4.

Original: The BlkSConv-based ResNet-56 models have muchsmaller model sizes and fewer MAdds than the standard model while all BlkSConv-based variantsoutperform the standard model on CIFAR 10 and have comparable accuracies on CIFAR 100.
Revised: The BlkSConv-based ResNet-56 models have muchsmaller model sizes and fewer MAdds than the standard model while all BlkSConv-based variantsoutperform the standard model on CIFAR 10 and have comparable accuracies on CIFAR 100.

Original: In thebottom of Table 3, the BlkSConv-based ResNet-20 with (0 . 5 , 0 . 5 , 0 . 5 , max) and the standard ResNet20 both have a comparable accuracy while the BlkSConv-based ResNet-20 model is compressed64% of the parameter size and MAdds is decreased 61% compared to the standard ResNet-20 model.
Revised: In thebottom of Table 3, the BlkSConv-based ResNet-20 with (0 . 5 , 0 . 5 , 0 . 5 , max) and the standard ResNet20 both have a comparable accuracy while the BlkSConv-based ResNet-20 model is compressed64% of the parameter size and MAdds is decreased 61% compared to the standard ResNet-20 model.

Original: Furthermore, the BlkSConv-based ResNet-56 with (0 . 5 , 0 . 5 , 0 . 5 , min) and the standard ResNet-20model both have similar parameter sizes and MAdds while the BlkSConv-based ResNet-56 modelhas an accuracy gain of 2%.
Revised: Furthermore, the BlkSConv-based ResNet-56 with (0 . 5 , 0 . 5 , 0 . 5 , min) and the standard ResNet-20model both have similar parameter sizes and MAdds while the BlkSConv-based ResNet-56 modelhas an accuracy gain of 2%.

Original: More experimental results can be found in the appendix.
Revised: More experimental results can be found in the appendix.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Performance on fine-grained classification
Revised: Performance on fine-grained classification

Original: We conduct experiments for fine-grained recognition on two datasets Stanford Dogs and Oxford102 Flowers.
Revised: We conduct experiments for fine-grained recognition on two datasets Stanford Dogs and Oxford102 Flowers.

Original: For the experimental setup, the standard ResNet-18 and its BlkSConv-variants are alltrained from scratch by augmenting data through random crops, horizontal flips, and random gammatransform.
Revised: For the experimental setup, the standard ResNet-18 and its BlkSConv-variants are alltrained from scratch by augmenting data through random crops, horizontal flips, and random gammatransform.

Original: We use SGD as the optimizer and the initial learning rate, the moment, and the weightdecay are set to 0.1, 0.9, and 10 − 4 , respectively.
Revised: We use SGD as the optimizer and the initial learning rate, the moment, and the weightdecay are set to 0.1, 0.9, and 10 − 4 , respectively.

Original: The number of epochs is 200, and the learning rateis scheduled to decay at epochs 100, 150, and 200 by a factor of 0.1.
Revised: The number of epochs is 200, and the learning rateis scheduled to decay at epochs 100, 150, and 200 by a factor of 0.1.

Original: The proposed BlkSConv-based
Revised: The proposed BlkSConv-based

Original: ResNet-18 models significantly outperform the standard ResNet-18 model both on Stanford Dogsand Oxford 102 Flowers as shown in shown Table 5..
Revised: ResNet-18 models significantly outperform the standard ResNet-18 model both on Stanford Dogsand Oxford 102 Flowers as shown in shown Table 5..

Original: 4.5 Ablation Study: Necessity to have large explained variance
Revised: 4.5 Ablation Study: Necessity to have large explained variance

Original: Here, we demonstrate how the variance hyperparameter α v affects the prediction accuracy of
Revised: Here, we demonstrate how the variance hyperparameter α v affects the prediction accuracy of

Original: BlkSConv-based CNNs.
Revised: BlkSConv-based CNNs.

Original: We use ResNet-18 as the experimental model.
Revised: We use ResNet-18 as the experimental model.

Original: After training the standard ResNet-18 on Stanford Dogs, the next goal is to find several BlkSConv-variants of ResNet18 all of which have different explained variances such that their accuracies can be compared.
Revised: After training the standard ResNet-18 on Stanford Dogs, the next goal is to find several BlkSConv-variants of ResNet18 all of which have different explained variances such that their accuracies can be compared.

Original: Note that H a, 0 .
Revised: Note that H a, 0 .

Original: 75 ⊆ H b, 0 . 75 , 0 .
Revised: 75 ⊆ H b, 0 . 75 , 0 .

Original: 75 for any a, b with a ≥ b .
Revised: 75 for any a, b with a ≥ b .

Original: Based on this observation, the 299 model in H b, 0 . 75 , 0 .
Revised: Based on this observation, the 299 model in H b, 0 . 75 , 0 .

Original: 75 which has the smallest parameter size is likely to have a small explainedvariance as well.
Revised: 75 which has the smallest parameter size is likely to have a small explainedvariance as well.

Original: Therefore, the selection strategy of the proposed HSA is set by SS = minin order to select several BlkSConv-based ResNet-18 models with different explained variances.
Revised: Therefore, the selection strategy of the proposed HSA is set by SS = minin order to select several BlkSConv-based ResNet-18 models with different explained variances.

Original: Now we apply the proposed HSA to the trained ResNet-18 under six search hyperparameters{ ( α v , 0 . 75 , 0 .
Revised: Now we apply the proposed HSA to the trained ResNet-18 under six search hyperparameters{ ( α v , 0 . 75 , 0 .

Original: 75 , min) :
Revised: 75 , min) :

Original: α v = 0 . 0 , 0 . 1 , 0 . 2 , . . .
Revised: α v = 0 . 0 , 0 . 1 , 0 . 2 , . . .

Original: The comparison result is shown in Table 6.
Revised: The comparison result is shown in Table 6.

Original: Itcan be seen that the accuracy of the BlkSConv-based model is greater than that of the standard modelonly when the variance hyperparameter α v is large enough, that is α v ≥ 0 .
Revised: Itcan be seen that the accuracy of the BlkSConv-based model is greater than that of the standard modelonly when the variance hyperparameter α v is large enough, that is α v ≥ 0 .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce the block-wise separable convolutions (BlkSConv) to replace standardconvolutions.
Revised: In this paper, we introduce the block-wise separable convolutions (BlkSConv) to replace standardconvolutions.

Original: An efficient implementation of the BlkSConv operation via a combination of pointwiseand group-wise convolutions is also given.
Revised: An efficient implementation of the BlkSConv operation via a combination of pointwiseand group-wise convolutions is also given.

Original: Moreover, we also propose an efficient hyperparametersearch algorithm based on principal component analysis in order to select an optimal BlkSConv-basedconvolutional network under certain constraints on model size and model efficiency.
Revised: Moreover, we also propose an efficient hyperparametersearch algorithm based on principal component analysis in order to select an optimal BlkSConv-basedconvolutional network under certain constraints on model size and model efficiency.

Original: Finally, theexperimental results demonstrate the advantage of the BlkSConv-based CNN models selected by theproposed hyperparameter search algorithm.
Revised: Finally, theexperimental results demonstrate the advantage of the BlkSConv-based CNN models selected by theproposed hyperparameter search algorithm.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes] Section
Revised: [Yes] Section

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Section 3.
Revised: [Yes] Section 3.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: Supplemental
Revised: Supplemental

Original: material
Revised: material

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] Section 4.1, Section 4.2, Section 4.4, Appendix(c)
Revised: [Yes] Section 4.1, Section 4.2, Section 4.4, Appendix(c)

Original: Did you report error bars (e
Revised: Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [No](d) Did you include the total amount of compute and the type of resources used (e
Revised: [No](d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] Section 4.2, Appendix4.
Revised: [Yes] Section 4.2, Appendix4.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Section
Revised: [Yes] Section

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Appendix(c)
Revised: [Yes] Appendix(c)

Original: Did you include any new assets either in the supplemental material or as a URL?
Revised: Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] Appendix
Revised: [Yes] Appendix

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] Appendix
Revised: [Yes] Appendix

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]