Original: D Y N ET : D YNAMIC C ONVOLUTION FOR A CCELERATING C ONVOLUTION N EURAL N ETWORKS
Revised: D Y N ET : D YNAMIC C ONVOLUTION FOR A CCELERATING C ONVOLUTION N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Convolution operator is the core of convolutional neural networks (CNNs) and occupies the most computation cost.
Revised: Convolution operator is the core of convolutional neural networks (CNNs) and occupies the most computation cost.

Original: To make CNNs more efﬁcient, many methods have been proposed to either design lightweight networks or compress models.
Revised: To make CNNs more efﬁcient, many methods have been proposed to either design lightweight networks or compress models.

Original: Although some efﬁcient network structures have been proposed, such as MobileNet or ShufﬂeNet, we ﬁnd that there still exists redundant information between convolution kernels.
Revised: Although some efﬁcient network structures have been proposed, such as MobileNet or ShufﬂeNet, we ﬁnd that there still exists redundant information between convolution kernels.

Original: To address this issue, we propose a novel dynamic convolution method named DyNet in this paper, which can adaptively generate convolution kernels based on image contents.
Revised: To address this issue, we propose a novel dynamic convolution method named DyNet in this paper, which can adaptively generate convolution kernels based on image contents.

Original: To demonstrate the effectiveness, we apply DyNet on multiple state-of-the-art CNNs.
Revised: To demonstrate the effectiveness, we apply DyNet on multiple state-of-the-art CNNs.

Original: The experiment results show that DyNet can reduce the computation cost remarkably, while maintaining the performance nearly unchanged.
Revised: The experiment results show that DyNet can reduce the computation cost remarkably, while maintaining the performance nearly unchanged.

Original: Speciﬁcally, for ShufﬂeNetV2 (1.0), MobileNetV2 (1.0), ResNet and ResNet50, DyNet reduces 37 .
Revised: Speciﬁcally, for ShufﬂeNetV2 (1.0), MobileNetV2 (1.0), ResNet and ResNet50, DyNet reduces 37 .

Original: 7% , 67 . 2% and 71 .
Revised: 7% , 67 . 2% and 71 .

Original: 3% FLOPs respectively while the Top-1 accuracy on ImageNet only changes by +1 .
Revised: 3% FLOPs respectively while the Top-1 accuracy on ImageNet only changes by +1 .

Original: 0% , − 0 .
Revised: 0% , − 0 .

Original: 27% , − 0 .
Revised: 27% , − 0 .

Original: 6% and − 0 .
Revised: 6% and − 0 .

Original: Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87 × , 1.32 × and 1.48 × on
Revised: Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87 × , 1.32 × and 1.48 × on

Original: CPU platform respectively.
Revised: CPU platform respectively.

Original: To verify the scalability, we also apply DyNet on segmentation task, the results show that DyNet can reduces 69.3% FLOPs while maintaining the Mean IoU on segmentation task.
Revised: To verify the scalability, we also apply DyNet on segmentation task, the results show that DyNet can reduces 69.3% FLOPs while maintaining the Mean IoU on segmentation task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional neural networks (CNNs) have achieved state-of-the-art performance in many computer vision tasks (Krizhevsky et al., 2012; Szegedy et al., 2013), and the neural architectures of CNNs are evolving over the years (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015;
Revised: Convolutional neural networks (CNNs) have achieved state-of-the-art performance in many computer vision tasks (Krizhevsky et al., 2012; Szegedy et al., 2013), and the neural architectures of CNNs are evolving over the years (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015;

Original: He et al., 2016; Hu et al., 2018; Zhong et al., 2018a;b).
Revised: He et al., 2016; Hu et al., 2018; Zhong et al., 2018a;b).

Original: However, modern high-performance CNNs often require a lot of computation resources to execute large amount of convolution kernel operations.
Revised: However, modern high-performance CNNs often require a lot of computation resources to execute large amount of convolution kernel operations.

Original: Aside from the accuracy, to make CNNs applicable on mobile devices, building lightweight and efﬁcient deep models has attracting much more attention recently (Howard et al., 2017; Sandler et al., 2018;
Revised: Aside from the accuracy, to make CNNs applicable on mobile devices, building lightweight and efﬁcient deep models has attracting much more attention recently (Howard et al., 2017; Sandler et al., 2018;

Original: Zhang et al., 2018; Ma et al., 2018).
Revised: Zhang et al., 2018; Ma et al., 2018).

Original: These methods can be roughly categorized into two types: efﬁcient network design and model compression.
Revised: These methods can be roughly categorized into two types: efﬁcient network design and model compression.

Original: Representative methods for the former category are
Revised: Representative methods for the former category are

Original: MobileNet (Howard et al., 2017; Sandler et al., 2018) and ShufﬂeNet (Ma et al., 2018; Zhang et al., 2018), which use depth-wise separable convolution and channel-level shufﬂe techniques to reduce computation cost.
Revised: MobileNet (Howard et al., 2017; Sandler et al., 2018) and ShufﬂeNet (Ma et al., 2018; Zhang et al., 2018), which use depth-wise separable convolution and channel-level shufﬂe techniques to reduce computation cost.

Original: On the other hand, model compression based methods tend to obtain a smaller network by compressing a larger network via pruning, factorization or mimic (Chen et al., 2015; Han et al., 2015a; Jaderberg et al., 2014; Lebedev et al., 2014; Ba & Caruana, 2014).
Revised: On the other hand, model compression based methods tend to obtain a smaller network by compressing a larger network via pruning, factorization or mimic (Chen et al., 2015; Han et al., 2015a; Jaderberg et al., 2014; Lebedev et al., 2014; Ba & Caruana, 2014).

Original: Although some handcrafted efﬁcient network structures have been designed, we observe that the signiﬁcant correlations still exist among convolutional kernels, and introduce large amount of redundant calculations.
Revised: Although some handcrafted efﬁcient network structures have been designed, we observe that the signiﬁcant correlations still exist among convolutional kernels, and introduce large amount of redundant calculations.

Original: Moreover, these small networks are hard to compress.
Revised: Moreover, these small networks are hard to compress.

Original: For example, Liu et al.
Revised: For example, Liu et al.

Original: (2019) compress MobileNetV2 to 124M, but the accuracy drops by 5 .
Revised: (2019) compress MobileNetV2 to 124M, but the accuracy drops by 5 .

Original: 4% on ImageNet.
Revised: 4% on ImageNet.

Original: We theoretically analyze above observation, and ﬁnd that this phenomenon is caused by the nature of static convolution, where correlated kernels are cooperated to extract noise-irrelevant features.
Revised: We theoretically analyze above observation, and ﬁnd that this phenomenon is caused by the nature of static convolution, where correlated kernels are cooperated to extract noise-irrelevant features.

Original: Thus it is hard to compress the ﬁxed convolution kernels without information loss.
Revised: Thus it is hard to compress the ﬁxed convolution kernels without information loss.

Original: We also ﬁnd that if we linearly fuse several convolution kernels to generate one dynamic kernel based on the input, we can obtain the noise-irrelevant features without the cooperation of multiple kernels, and further reduce the computation cost of convolution layer remarkably.
Revised: We also ﬁnd that if we linearly fuse several convolution kernels to generate one dynamic kernel based on the input, we can obtain the noise-irrelevant features without the cooperation of multiple kernels, and further reduce the computation cost of convolution layer remarkably.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Fixed convolution kernels
Revised: Fixed convolution kernels

Original: Dynamic Generation
Revised: Dynamic Generation

Original: [Figure]
Revised: [Figure]

Original: Based on above observations and analysis, in this paper, we propose a novel dynamic convolution method named DyNet.
Revised: Based on above observations and analysis, in this paper, we propose a novel dynamic convolution method named DyNet.

Original: The overall framework of DyNet is shown in Figure 1, which consists of a coefﬁcient prediction module and a dynamic generation module .
Revised: The overall framework of DyNet is shown in Figure 1, which consists of a coefﬁcient prediction module and a dynamic generation module .

Original: The coefﬁcient prediction module is trainable and designed to predict the coefﬁcients of ﬁxed convolution kernels.
Revised: The coefﬁcient prediction module is trainable and designed to predict the coefﬁcients of ﬁxed convolution kernels.

Original: Then the dynamic generation module further generates a dynamic kernel based on the predicted coefﬁcients.
Revised: Then the dynamic generation module further generates a dynamic kernel based on the predicted coefﬁcients.

Original: Our proposed dynamic convolution method is simple to implement, and can be used as a drop-in plugin for any convolution layer to reduce computation cost.
Revised: Our proposed dynamic convolution method is simple to implement, and can be used as a drop-in plugin for any convolution layer to reduce computation cost.

Original: We evaluate the proposed DyNet on state-of-the-art networks such as MobileNetV2, ShufﬂeNetV2 and ResNets.
Revised: We evaluate the proposed DyNet on state-of-the-art networks such as MobileNetV2, ShufﬂeNetV2 and ResNets.

Original: Experiment results show that DyNet reduces 37 . 0% FLOPs of ShufﬂeNetV2 (1.0) while further improve the Top-1 accuracy on
Revised: Experiment results show that DyNet reduces 37 . 0% FLOPs of ShufﬂeNetV2 (1.0) while further improve the Top-1 accuracy on

Original: ImageNet by 1 . 0% .
Revised: ImageNet by 1 . 0% .

Original: For MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 54 .
Revised: For MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 54 .

Original: 7% , 67 . 2% and 71 .
Revised: 7% , 67 . 2% and 71 .

Original: 3% FLOPs respectively, the Top-1 accuracy on ImageNet changes by − 0 .
Revised: 3% FLOPs respectively, the Top-1 accuracy on ImageNet changes by − 0 .

Original: 27% , − 0 .
Revised: 27% , − 0 .

Original: 6% and − 0 .
Revised: 6% and − 0 .

Original: Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet and ResNet50 by 1.87 × ,1.32 × and 1.48 × on CPU platform respectively.
Revised: Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet and ResNet50 by 1.87 × ,1.32 × and 1.48 × on CPU platform respectively.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We review related works from three aspects: efﬁcient convolution neural network design, model compression and dynamic convolutional kernels.
Revised: We review related works from three aspects: efﬁcient convolution neural network design, model compression and dynamic convolutional kernels.

Original: 2. E FFICIENT CONVOLUTION NEURAL NETWORK DESIGN
Revised: 2. E FFICIENT CONVOLUTION NEURAL NETWORK DESIGN

Original: In many computer vision tasks (Krizhevsky et al., 2012; Szegedy et al., 2013), model design plays a key role.
Revised: In many computer vision tasks (Krizhevsky et al., 2012; Szegedy et al., 2013), model design plays a key role.

Original: The increasing demands of high quality networks on mobile/embedding devices have driven the study on efﬁcient network design (He & Sun, 2015).
Revised: The increasing demands of high quality networks on mobile/embedding devices have driven the study on efﬁcient network design (He & Sun, 2015).

Original: For example, GoogleNet (Szegedy et al., 2015) increases the depth of networks with lower complexity compared to simply stacking convolution layers; SqueezeNet (Iandola et al., 2016) deploys a bottleneck approach to design a very small network; Xception (Chollet, 2017), MobileNet (Howard et al., 2017) and MobileNetV2 (Sandler et al., 2018) use depth-wise separable convolution to reduce computation and model size.
Revised: For example, GoogleNet (Szegedy et al., 2015) increases the depth of networks with lower complexity compared to simply stacking convolution layers; SqueezeNet (Iandola et al., 2016) deploys a bottleneck approach to design a very small network; Xception (Chollet, 2017), MobileNet (Howard et al., 2017) and MobileNetV2 (Sandler et al., 2018) use depth-wise separable convolution to reduce computation and model size.

Original: ShufﬂeNet (Zhang et al., 2018) and ShufﬂeNetV2 (Ma et al., 2018) shufﬂe channels to reduce computation of 1 × 1 convolution kernel and improve accuracy.
Revised: ShufﬂeNet (Zhang et al., 2018) and ShufﬂeNetV2 (Ma et al., 2018) shufﬂe channels to reduce computation of 1 × 1 convolution kernel and improve accuracy.

Original: Despite the progress made by these efforts, we ﬁnd that there still exists redundancy between convolution kernels and cause redundant computation.
Revised: Despite the progress made by these efforts, we ﬁnd that there still exists redundancy between convolution kernels and cause redundant computation.

Original: 2. M ODEL COMPRESSION
Revised: 2. M ODEL COMPRESSION

Original: Another trend to obtaining small network is model compression.
Revised: Another trend to obtaining small network is model compression.

Original: Factorization based methods (Jaderberg et al., 2014; Lebedev et al., 2014) try to speed up convolution operation by using tensor decomposition to approximate original convolution operation.
Revised: Factorization based methods (Jaderberg et al., 2014; Lebedev et al., 2014) try to speed up convolution operation by using tensor decomposition to approximate original convolution operation.

Original: Knowledge distillation based methods (Ba & Caruana, 2014; Romero et al., 2014; Hinton et al., 2015) learn a small network to mimic a larger teacher network.
Revised: Knowledge distillation based methods (Ba & Caruana, 2014; Romero et al., 2014; Hinton et al., 2015) learn a small network to mimic a larger teacher network.

Original: Pruning based methods (Han et al., 2015a;b; Wen et al., 2016;
Revised: Pruning based methods (Han et al., 2015a;b; Wen et al., 2016;

Original: Liu et al., 2019) try to reduce computation by pruning the redundant connections or convolution channels.
Revised: Liu et al., 2019) try to reduce computation by pruning the redundant connections or convolution channels.

Original: Compared with those methods, DyNet is more effective especially when the target network is already efﬁcient enough.
Revised: Compared with those methods, DyNet is more effective especially when the target network is already efﬁcient enough.

Original: For example, in (Liu et al., 2019), they get a smaller model of 124M FLOPs by pruning the MobileNetV2, however it drops the accuracy by 5 . 4% on ImageNet compared with the model with 291M FLOPs.
Revised: For example, in (Liu et al., 2019), they get a smaller model of 124M FLOPs by pruning the MobileNetV2, however it drops the accuracy by 5 . 4% on ImageNet compared with the model with 291M FLOPs.

Original: While in DyNet, we can reduce the FLOPs of MobileNetV2 (1.0) from 298M to 129M with the accuracy drops only 0 .
Revised: While in DyNet, we can reduce the FLOPs of MobileNetV2 (1.0) from 298M to 129M with the accuracy drops only 0 .

Original: 2. D YNAMIC CONVOLUTION KERNEL
Revised: 2. D YNAMIC CONVOLUTION KERNEL

Original: Generating dynamic convolution kernels appears in both computer vision and natural language processing (NLP) tasks.
Revised: Generating dynamic convolution kernels appears in both computer vision and natural language processing (NLP) tasks.

Original: In computer vision domain, Klein et al.
Revised: In computer vision domain, Klein et al.

Original: (Klein et al., 2015) and Brabandere et al.
Revised: (Klein et al., 2015) and Brabandere et al.

Original: (Jia et al., 2016) directly generate convolution kernels via a linear layer based on the feature maps of previous layers.
Revised: (Jia et al., 2016) directly generate convolution kernels via a linear layer based on the feature maps of previous layers.

Original: Because convolution kernels has a large amount of parameters, the linear layer will be inefﬁcient on the hardware.
Revised: Because convolution kernels has a large amount of parameters, the linear layer will be inefﬁcient on the hardware.

Original: Our proposed method solves this problem via merely predicting the coefﬁcients for linearly combining static kernels and achieve real speed up for CNN on hardware.
Revised: Our proposed method solves this problem via merely predicting the coefﬁcients for linearly combining static kernels and achieve real speed up for CNN on hardware.

Original:  Yang et al.
Revised: The idea of linearly combining static kernels using predicted coefﬁcients has been proposed by Yang et al.

Original: (Yang et al., 2019) share similar idea with ours but they focus on using more parameters to make models more expressive, while we focus on reducing redundant calculations in convolution.
Revised: (Yang et al., 2019), but they focus on using more parameters to make models more expressive while we focus on reducing redundant calculations in convolution.

Original: We make theoretical analysis and conduct correlation experiment to prove that correlations among convolutional kernels can be reduced via dynamically fusing several kernels.
Revised: We make theoretical analysis and conduct correlation experiment to prove that correlations among convolutional kernels can be reduced via dynamically fusing several kernels.

Original: In NLP domain, some works (Shen et al., 2018; Wu et al., 2019; Gong et al., 2018) incorporate context information to generate input-aware convolution ﬁlters which can be changed according to input sentences with various lengths.
Revised: In NLP domain, some works (Shen et al., 2018; Wu et al., 2019; Gong et al., 2018) incorporate context information to generate input-aware convolution ﬁlters which can be changed according to input sentences with various lengths.

Revised: These methods also directly generate convolution kernels via a linear layer, etc.

Revised: Because the size of CNN in NLP is smaller and the dimension of convolution kernel is one, the inefﬁciency issue for the linear layer is alleviated.

Revised: Moreover, Wu et al.

Revised: (Wu et al., 2019) alleviate this issue utilizing the depthwise convolution and the strategy of sharing weight across layers.

Original: These methods are designed to improve the adaptivity and ﬂexibility of language modeling, while our method aims to cut down the redundant computation cost.
Revised: These methods are designed to improve the adaptivity and ﬂexibility of language modeling, while our method aims to cut down the redundant computation cost.

Original:  D Y N ET : D YNAMIC C ONVOLUTION IN CNN S
Revised:  D Y N ET : D YNAMIC C ONVOLUTION IN CNN S

Original: In this section, we ﬁrst describe the motivation of DyNet.
Revised: In this section, we ﬁrst describe the motivation of DyNet.

Original: Then we explain the proposed dynamic convolution in detail.
Revised: Then we explain the proposed dynamic convolution in detail.

Original: Finally, we illustrate the DyNet based architectures of our proposed Dy-mobile,
Revised: Finally, we illustrate the DyNet based architectures of our proposed Dy-mobile,

Original: Dy-shufﬂe, Dy-ResNet18, Dy-ResNet50.
Revised: Dy-shufﬂe, Dy-ResNet18, Dy-ResNet50.

Original: [Figure]
Revised: [Figure]

Original: 3. M OTIVATION
Revised: 3. M OTIVATION

Original: As illustrated in previous works (Han et al., 2015a;b; Wen et al., 2016;
Revised: As illustrated in previous works (Han et al., 2015a;b; Wen et al., 2016;

Original: Liu et al., 2019), convolutional kernels are naturally correlated in deep models.
Revised: Liu et al., 2019), convolutional kernels are naturally correlated in deep models.

Original: For some of the well known networks, we plot the distribution of Pearson product-moment correlation coefﬁcient between feature maps in Figure 2.
Revised: For some of the well known networks, we plot the distribution of Pearson product-moment correlation coefﬁcient between feature maps in Figure 2.

Original: Most existing works try to reduce correlations by compressing.
Revised: Most existing works try to reduce correlations by compressing.

Original: However, efﬁcient and small networks like MobileNets are harder to prune despite the correlation is still signiﬁcant.
Revised: However, efﬁcient and small networks like MobileNets are harder to prune despite the correlation is still signiﬁcant.

Original: We think these correlations are vital for maintaining the performance because they are cooperated to obtain noiseirrelevant features.
Revised: We think these correlations are vital for maintaining the performance because they are cooperated to obtain noiseirrelevant features.

Original: We take face recognition as an example, where the pose or the illumination is not supposed to change the classiﬁcation results.
Revised: We take face recognition as an example, where the pose or the illumination is not supposed to change the classiﬁcation results.

Original: Therefore, the feature maps will gradually become noise-irrelevant when they go deeper.
Revised: Therefore, the feature maps will gradually become noise-irrelevant when they go deeper.

Original: Based on the theoretical analysis in appendix A, we ﬁnd that if we dynamically fuse several kernels, we can get noise-irrelevant feature without the cooperation of redundant kernels.
Revised: Based on the theoretical analysis in appendix A, we ﬁnd that if we dynamically fuse several kernels, we can get noise-irrelevant feature without the cooperation of redundant kernels.

Original: In this paper, we propose dynamic convolution method, which learns the coefﬁcients to fuse multiple kernels into a dynamic one based on image contents.
Revised: In this paper, we propose dynamic convolution method, which learns the coefﬁcients to fuse multiple kernels into a dynamic one based on image contents.

Original: We give more in depth analysis about our motivation in appendix A.
Revised: We give more in depth analysis about our motivation in appendix A.

Original: 3. D YNAMIC CONVOLUTION
Revised: 3. D YNAMIC CONVOLUTION

Original: The goal of dynamic convolution is to learn a group of kernel coefﬁcients, which fuse multiple ﬁxed kernels to a dynamic one.
Revised: The goal of dynamic convolution is to learn a group of kernel coefﬁcients, which fuse multiple ﬁxed kernels to a dynamic one.

Original: We demonstrate the overall framework of dynamic convolution in Figure 1.
Revised: We demonstrate the overall framework of dynamic convolution in Figure 1.

Original: We ﬁrst utilize a trainable coefﬁcient prediction module to predict coefﬁcients.
Revised: We ﬁrst utilize a trainable coefﬁcient prediction module to predict coefﬁcients.

Original: Then we further propose a dynamic generation module to fuse ﬁxed kernels to a dynamic one.
Revised: Then we further propose a dynamic generation module to fuse ﬁxed kernels to a dynamic one.

Original: We will illustrate the coefﬁcient prediction module and dynamic generation module in detail in the following of this section.
Revised: We will illustrate the coefﬁcient prediction module and dynamic generation module in detail in the following of this section.

Original: [Figure]
Revised: [Figure]

Original: Coefﬁcient prediction module Coefﬁcient prediction module is proposed to predict coefﬁcients based on image contents.
Revised: Coefﬁcient prediction module Coefﬁcient prediction module is proposed to predict coefﬁcients based on image contents.

Original: As shown in Figure 3, the coefﬁcient prediction module can be composed by a global average pooling layer and a fully connected layer with Sigmoid as activation function.
Revised: As shown in Figure 3, the coefﬁcient prediction module can be composed by a global average pooling layer and a fully connected layer with Sigmoid as activation function.

Original: Global average pooling layer aggregates the input feature maps into a 1 × 1 × C in vector, which serves as a feature extraction layer.
Revised: Global average pooling layer aggregates the input feature maps into a 1 × 1 × C in vector, which serves as a feature extraction layer.

Original: Then the fully connected layer further maps the feature into a 1 × 1 × C vector, which are the coefﬁcients for ﬁxed convolution kernels of several dynamic convolution layers.
Revised: Then the fully connected layer further maps the feature into a 1 × 1 × C vector, which are the coefﬁcients for ﬁxed convolution kernels of several dynamic convolution layers.

Original: Dynamic generation module For a dynamic convolution layer with weight [ C out × g t , C in , k, k ] , it corresponds with C out × g t ﬁxed kernels and C out dynamic kernels, the shape of each kernel is [ C in , k, k ] .
Revised: Dynamic generation module For a dynamic convolution layer with weight [ C out × g t , C in , k, k ] , it corresponds with C out × g t ﬁxed kernels and C out dynamic kernels, the shape of each kernel is [ C in , k, k ] .

Original: g t denotes the group size, it is a hyperparameter.
Revised: g t denotes the group size, it is a hyperparameter.

Original: We denote the ﬁxed kernels as w it , the dynamic kernels as (cid:102) w t , the coefﬁcients as η it , where t = 0 , ..., C out , i = 0 , ..., g t .
Revised: We denote the ﬁxed kernels as w it , the dynamic kernels as (cid:102) w t , the coefﬁcients as η it , where t = 0 , ..., C out , i = 0 , ..., g t .

Original: After the coefﬁcients are obtained, we generate dynamic kernels as follows:
Revised: After the coefﬁcients are obtained, we generate dynamic kernels as follows:

Original: [Equation]
Revised: [Equation]

Original: Training algorithm For the training of the proposed dynamic convolution, it is not suitable to use batch based training scheme.
Revised: Training algorithm For the training of the proposed dynamic convolution, it is not suitable to use batch based training scheme.

Original: It is because the convolution kernel is different for different input images in the same mini-batch.
Revised: It is because the convolution kernel is different for different input images in the same mini-batch.

Original: Therefore, we fuse feature maps based on the coefﬁcients rather than kernels during training.
Revised: Therefore, we fuse feature maps based on the coefﬁcients rather than kernels during training.

Original: They are mathematically equivalent as shown in Eq. 2:
Revised: They are mathematically equivalent as shown in Eq. 2:

Original: [Equation]
Revised: [Equation]

Original: where x denotes the input, (cid:101)
Revised: where x denotes the input, (cid:101)

Original: O t denotes the output of dynamic kernel (cid:101) w t , O it denotes the output of ﬁxed kernel w it .
Revised: O t denotes the output of dynamic kernel (cid:101) w t , O it denotes the output of ﬁxed kernel w it .

Original: Input
Revised: [Figure]

Original: Dy
Revised: Dy

Original: [Figure]
Revised: [Figure]

Revised: Dy-Conv

Original: Dy
Revised: Dy

Original: [Figure]
Revised: [Figure]

Original: Conv
Revised: Conv

Original: Dy-Conv

Original: Golbal Pool
Revised: Golbal Pool

Original: FC +
Revised: FC +

Original: Input
Revised: Input

Original: Channel Split
Revised: Channel Split

Original: Dy-Conv Dy-Conv
Revised: Dy-Conv Dy-Conv

Original: Dy-Conv
Revised: Dy-Conv

Original: Channel Shuffle
Revised: Channel Shuffle

Original: Golbal Pool
Revised: Golbal Pool

Original: FC
Revised: FC

Original: Input
Revised: Input

Original: Dy-Conv Dy-Conv
Revised: Dy-Conv Dy-Conv

Original: Dy-Conv
Revised: Dy-Conv

Original: Golbal Pool
Revised: Golbal Pool

Original: FC
Revised: FC

Original: FC +
Revised: FC +

Original: Input
Revised: Input

Original: Dy-Conv Dy-Conv
Revised: Dy-Conv Dy-Conv

Original: Golbal Pool
Revised: Golbal Pool

Original: FC FC
Revised: FC FC

Original: [Figure]
Revised: [Figure]

Original: 3. D YNAMIC CONVOLUTION NEURAL NETWORKS
Revised: 3. D YNAMIC CONVOLUTION NEURAL NETWORKS

Original: We equip the MobileNetV2, ShufﬂeNetV2 and ResNets with our proposed dynamic convolution, and propose Dy-mobile, Dy-shufﬂe, Dy-ResNet18 and Dy-ResNet50 respectively.
Revised: We equip the MobileNetV2, ShufﬂeNetV2 and ResNets with our proposed dynamic convolution, and propose Dy-mobile, Dy-shufﬂe, Dy-ResNet18 and Dy-ResNet50 respectively.

Original: The building blocks of these 4 network are shown in Figure 4.
Revised: The building blocks of these 4 network are shown in Figure 4.

Original: Based on above dynamic convolution, each dynamic kernel can get noise-irrelevant feature without the cooperation of other kernels.
Revised: Based on above dynamic convolution, each dynamic kernel can get noise-irrelevant feature without the cooperation of other kernels.

Original: Therefore we can reduce the channels for each layer of those base models and remain the performance.
Revised: Therefore we can reduce the channels for each layer of those base models and remain the performance.

Original: We set the hyper-parameter g t as 6 for all of them, and we give details of these dynamic CNNs below.
Revised: We set the hyper-parameter g t as 6 for all of them, and we give details of these dynamic CNNs below.

Original: Dy-mobile
Revised: Dy-mobile

Original: In our proposed Dy-mobile, we replace the original MobileNetV2 block with our dymobile block, which is shown in Figure 4 (a).
Revised: In our proposed Dy-mobile, we replace the original MobileNetV2 block with our dymobile block, which is shown in Figure 4 (a).

Original: The input of coefﬁcient prediction module is the input of block, it produces the coefﬁcients for all three dynamic convolution layers.
Revised: The input of coefﬁcient prediction module is the input of block, it produces the coefﬁcients for all three dynamic convolution layers.

Original: Moreover, we further make two adjustments:
Revised: Moreover, we further make two adjustments:

Original: • We do not expand the channels in the middle layer like MobileNetV
Revised: • We do not expand the channels in the middle layer like MobileNetV2.

Original: If we denote the output channels of the block as C out , then the channels of all the three convolution layers will be C out .
Revised: If we denote the output channels of the block as C out , then the channels of all the three convolution layers will be C out .

Original:  Since the depth-wise convolution is efﬁcient, we set groups =
Revised: • Since the depth-wise convolution is efﬁcient, we set groups =

Original: C out 6 for the dynamic depthwise convolution.
Revised: C out 6 for the dynamic depthwise convolution.

Original: We will enlarge C out to make it becomes the multiple of 6 if needed.
Revised: We will enlarge C out to make it becomes the multiple of 6 if needed.

Original: After the aforementioned adjustments, the ﬁrst dynamic convolution layer reduce the FLOPs from 6 C 2 HW to C 2 HW .
Revised: After the aforementioned adjustments, the ﬁrst dynamic convolution layer reduce the FLOPs from 6 C 2 HW to C 2 HW .

Original: The second dynamic convolution layer keep the FLOPs as 6 CHW × 3unchanged because we reduce the output channels by 6x while setting the groups of convolution 6x smaller, too.
Revised: The second dynamic convolution layer keep the FLOPs as 6 CHW × 3unchanged because we reduce the output channels by 6x while setting the groups of convolution 6x smaller, too.

Original: For the third dynamic convolution layer, we reduce the FLOPs from 6 C 2 HW to
Revised: For the third dynamic convolution layer, we reduce the FLOPs from 6 C 2 HW to

Original: C 2 HW as well.
Revised: C 2 HW as well.

Original: The ratio of FLOPs for the original block and our dy-mobile block is:
Revised: The ratio of FLOPs for the original block and our dy-mobile block is:

Original: [Equation]
Revised: [Equation]

Original: Dy-shufﬂe
Revised: Dy-shufﬂe

Original: In the original ShufﬂeNet V2, channel split operation will split feature maps to rightbranch and left-branch, the right branch will go through one pointwise convolution, one depthwise convolution and one pointwise convolution sequentially.
Revised: In the original ShufﬂeNet V2, channel split operation will split feature maps to rightbranch and left-branch, the right branch will go through one pointwise convolution, one depthwise convolution and one pointwise convolution sequentially.

Original: We replace conventional convolution with dynamic convolution in the right branch as shown in Figure 4 (b).
Revised: We replace conventional convolution with dynamic convolution in the right branch as shown in Figure 4 (b).

Original: We feed the input of right branch into coefﬁcient prediction module to produce the coefﬁcients.
Revised: We feed the input of right branch into coefﬁcient prediction module to produce the coefﬁcients.

Original: In our dy-shufﬂe block, we split channels into left-branch and right-branch with ratio 3 : 1 , thus we reduce the 75% computation cost for two dynamic pointwise convolutuon.
Revised: In our dy-shufﬂe block, we split channels into left-branch and right-branch with ratio 3 : 1 , thus we reduce the 75% computation cost for two dynamic pointwise convolutuon.

Original: Similar with dy-mobile, we adjust the parameter ”groups” in dynamic depthwise convolution to keep the FLOPs unchanged.
Revised: Similar with dy-mobile, we adjust the parameter ”groups” in dynamic depthwise convolution to keep the FLOPs unchanged.

Original: Dy-ResNet18/

Revised: Dy-ResNet18/

Original: In Dy-ResNet18 and DyResNet50, we simple reduce half of the output channels for dynamic convolution layers of each residual block.
Revised: In Dy-ResNet18 and DyResNet50, we simple reduce half of the output channels for dynamic convolution layers of each residual block.

Original: Because the input channels of each block is large compared with dy-mobile and dy-shufﬂe, we use two linear layer as shown in Figure 4 (c) and
Revised: Because the input channels of each block is large compared with dy-mobile and dy-shufﬂe, we use two linear layer as shown in Figure 4 (c) and

Original: Figure 4 (d) to reduce the amount of parameters.
Revised: Figure 4 (d) to reduce the amount of parameters.

Original: If the input channel is C in , the output channels of the ﬁrst linear layer will be C in 4 for Dy-ResNet18/50.
Revised: If the input channel is C in , the output channels of the ﬁrst linear layer will be C in 4 for Dy-ResNet18/50.

Original:  E XPERIMENTS 4. I MPLEMENTATION DETAILS
Revised:  E XPERIMENTS 4. I MPLEMENTATION DETAILS

Original: For the training of the proposed dynamic neural networks.
Revised: For the training of the proposed dynamic neural networks.

Original: Each image has data augmentation of randomly cropping and ﬂipping, and is optimized with SGD strategy with cosine learning rate decay.
Revised: Each image has data augmentation of randomly cropping and ﬂipping, and is optimized with SGD strategy with cosine learning rate decay.

Original: We set batch size, initial learning rate, weight decay and momentum as 2048, 0.8, 5e-5 and 0.
Revised: We set batch size, initial learning rate, weight decay and momentum as 2048, 0.8, 5e-5 and 0.

Original: respectively.
Revised: respectively.

Original: We also use the label smoothing with rate 0.1.
Revised: We also use the label smoothing with rate 0.1.

Original: We evaluate the accuracy on the test images with center crop.
Revised: We evaluate the accuracy on the test images with center crop.

Original: 4. E XPERIMENT SETTINGS AND COMPARED METHODS
Revised: 4. E XPERIMENT SETTINGS AND COMPARED METHODS

Original: [Figure]
Revised: [Figure]

Original: We evaluate DyNet on ImageNet (Russakovsky et al., 2015), which contains 1.28 million training images and 50K validation images collected from 1000 different classes.
Revised: We evaluate DyNet on ImageNet (Russakovsky et al., 2015), which contains 1.28 million training images and 50K validation images collected from 1000 different classes.

Original: We train the proposed networks on the training set and report the top-1 error on the validation set.
Revised: We train the proposed networks on the training set and report the top-1 error on the validation set.

Original: To demonstrate the effectiveness, we compare the proposed dynamic convolution with state-of-the-art networks under mobile setting, including MobileNetV1 (Howard et al., 2017), MobileNetV2 (Sandler et al., 2018),
Revised: To demonstrate the effectiveness, we compare the proposed dynamic convolution with state-of-the-art networks under mobile setting, including MobileNetV1 (Howard et al., 2017), MobileNetV2 (Sandler et al., 2018),

Original: ShufﬂeNet (Zhang et al., 2018), ShufﬂeNet
Revised: ShufﬂeNet (Zhang et al., 2018), ShufﬂeNet

Original: V2 (Ma et al., 2018), Xception (Chollet, 2017),
Revised: V2 (Ma et al., 2018), Xception (Chollet, 2017)

Revised: [Table]

Original: DenseNet (Huang et al., 2017), IGCV2 (Xie et al., 2018) and IGCV3 (Sun et al., 2018).et al.,
Revised: DenseNet (Huang et al., 2017), IGCV2 (Xie et al., 2018) and IGCV3 (Sun et al., 2018).

Original: [Table]
Revised: [Table]

Original: 4. E XPERIMENT RESULTS AND ANALYSIS
Revised: 4. E XPERIMENT RESULTS AND ANALYSIS

Original: Analysis of accuracy and computation cost We demonstrate the results in Table 1, where the number in the brackets indicates the channel number controller (Sandler et al., 2018).
Revised: Analysis of accuracy and computation cost We demonstrate the results in Table 1, where the number in the brackets indicates the channel number controller (Sandler et al., 2018).

Original: We partitioned the result table into three parts: (1) The proposed dynamic networks; (2) Compared state-of-the-art networks under mobile settings; (3) The original networks corresponding to the implemented dynamic networks.
Revised: We partitioned the result table into three parts: (1) The proposed dynamic networks; (2) Compared state-of-the-art networks under mobile settings; (3) The original networks corresponding to the implemented dynamic networks.

Original: Table 1 provides several valuable observations: (1) Compared with these well known models under mobile setting, the proposed Dy-mobile and Dy-shufﬂe achieves the best classiﬁcation error with lowest computation cost.
Revised: Table 1 provides several valuable observations: (1) Compared with these well known models under mobile setting, the proposed Dy-mobile and Dy-shufﬂe achieves the best classiﬁcation error with lowest computation cost.

Original: This demonstrates that the proposed dynamic convolution is a simple yet effective way to reduce computation cost.
Revised: This demonstrates that the proposed dynamic convolution is a simple yet effective way to reduce computation cost.

Original: Compared with the corresponding basic neural structures, the proposed Dy-shufﬂe (1.0), Dy-mobile (1.0), Dy-ResNet18 and Dy-ResNet50 reduce 37 .
Revised: Compared with the corresponding basic neural structures, the proposed Dy-shufﬂe (1.0), Dy-mobile (1.0), Dy-ResNet18 and Dy-ResNet50 reduce 37 .

Original: 7% , 67 . 2% and 71 .
Revised: 7% , 67 . 2% and 71 .

Original: 3% computation cost respectively with little drop on Top-1 accuracy.
Revised: 3% computation cost respectively with little drop on Top-1 accuracy.

Original: This shows that even though the proposed network signiﬁcantly reduces the convolution computation cost, the generated dynamic kernel can still capture sufﬁcient information from image contents.
Revised: This shows that even though the proposed network signiﬁcantly reduces the convolution computation cost, the generated dynamic kernel can still capture sufﬁcient information from image contents.

Original: The results also indicate that the proposed dynamic convolution is a powerful plugin, which can be implemented on convolution layers to reduce computation cost while maintaining the accuracy.
Revised: The results also indicate that the proposed dynamic convolution is a powerful plugin, which can be implemented on convolution layers to reduce computation cost while maintaining the accuracy.

Original: Furthermore, we conduct detailed experiments on MobileNetV2.
Revised: Furthermore, we conduct detailed experiments on MobileNetV2.

Original: We replace the conventional convolution with the proposed dynamic one and get Dy-MobileNetV2.
Revised: We replace the conventional convolution with the proposed dynamic one and get Dy-MobileNetV2.

Original: The accuracy of classiﬁcation for models with different number of channels are shown in Figure 5.
Revised: The accuracy of classiﬁcation for models with different number of channels are shown in Figure 5.

Original: It is observed that DyMobileNetV2 consistently outperforms MobileNetV2 but the ascendancy is weaken with the increase of number of channels.
Revised: It is observed that DyMobileNetV2 consistently outperforms MobileNetV2 but the ascendancy is weaken with the increase of number of channels.

Original: [Figure]
Revised: [Table]

Original: have small correlation values

Original: [Figure]
Revised: [Figure]

Original:  Analysis of the dynamic kernel Aside from the quantitative analysis, we also demonstrate the redundancy of the generated dynamic kernels compared with conventional ﬁxed kernels in Figure 6.
Revised: W M Analysis of the dynamic kernel Aside from the quantitative analysis, we also demonstrate the redundancy of the generated dynamic kernels compared with conventional ﬁxed kernels in Figure 6.

Original: We calculate the correlation between each feature maps output by the second last stage for the original
Revised: We calculate the correlation between each feature maps output by the second last stage for the original

Original: MobileNetV2(1.0) and Dy-MobileNetV2 (1.0).
Revised: MobileNetV2(1.0) and Dy-MobileNetV2 (1.0).

Original: Note that Dy-MobileNetV2 (1.0) is different with
Revised: Note that Dy-MobileNetV2 (1.0) is different with

Original: Dy-mobile(1.0).
Revised: Dy-mobile(1.0).

Original: Dy-MobileNetV2(1.0) keeps the channels of each layer the same as the original one, while replace the conventional convolution with dynamic convolution.
Revised: Dy-MobileNetV2(1.0) keeps the channels of each layer the same as the original one, while replace the conventional convolution with dynamic convolution.

Original: As shown in Figure 6, we can observe that the correlation distribution of dynamic kernels have more values distributed between − 0 .
Revised: As shown in Figure 6, we can observe that the correlation distribution of dynamic kernels have more values distributed between − 0 .

Original: 1 and 0 .
Revised: 1 and 0 .

Original: 2 compared with ﬁxed convolution kernel, which indicates that the redundancy between dynamic convolution kernels are much smaller than the ﬁxed convolution kernels. 
Revised: 2 compared with ﬁxed convolution kernel, which indicates that the redundancy between dynamic convolution kernels are much smaller than the ﬁxed convolution kernels. [Figure] [Figure] L a t e n c y

Original:  Analysis of speed on the hardware We also analysis the inference speed of DyNet.
Revised: R e du ce d R a ti o [Figure] Analysis of speed on the hardware We also analysis the inference speed of DyNet.

Original: We carry out experiments on the CPU platform (Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz) with Caffe (Jia et al., 2014).
Revised: We carry out experiments on the CPU platform (Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz) with Caffe (Jia et al., 2014).

Original: We set the size of input as 224 and report the average inference time of 50 iterations.
Revised: We set the size of input as 224 and report the average inference time of 50 iterations.

Original: It is reasonable to set mini-batch size as 1, which is consistent with most inference scenarios.
Revised: It is reasonable to set mini-batch size as 1, which is consistent with most inference scenarios.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: Moreover, the latency of fusing ﬁxed kernels is independent with the input size, thus we expect to achieve bigger acceleration ratio when the input size of networks become larger.
Revised: Moreover, the latency of fusing ﬁxed kernels is independent with the input size, thus we expect to achieve bigger acceleration ratio when the input size of networks become larger.

Original: We conduct experiments to verify this assumption, the results are shown in Figure 7.
Revised: We conduct experiments to verify this assumption, the results are shown in Figure 7.

Original: We can observe that the ratio of reduced latency achieved by DyNet gets bigger as the input size becomes larger.
Revised: We can observe that the ratio of reduced latency achieved by DyNet gets bigger as the input size becomes larger.

Original: As shown in (Tan & Le, 2019), a larger input size can make networks perform signiﬁcantly better, thus DyNet is more effective in this scenario.
Revised: As shown in (Tan & Le, 2019), a larger input size can make networks perform signiﬁcantly better, thus DyNet is more effective in this scenario.

Original: We also analysis the training speed on the GPU platform.
Revised: We also analysis the training speed on the GPU platform.

Original: The model is trained with 32 NVIDIA
Revised: The model is trained with 32 NVIDIA

Original: Tesla V100 GPUs and the batch size is 2048.
Revised: Tesla V100 GPUs and the batch size is 2048.

Original: We report the average training time of one iteration in
Revised: We report the average training time of one iteration in

Original: Table 2.
Revised: Table 2.

Original: It is observed that the training speed of DyNet is slower, it is reasonable because we fuse feature maps rather than kernels according to Eq.
Revised: It is observed that the training speed of DyNet is slower, it is reasonable because we fuse feature maps rather than kernels according to Eq.

Original: 2 in the training stage.
Revised: 2 in the training stage.

Original: [Table]
Revised: [Table]

Original: 4. E XPERIMENTS ON SEGMENTATION
Revised: 4. E XPERIMENTS ON SEGMENTATION

Original: To verify the scalability of DyNet on other tasks
Revised: To verify the scalability of DyNet on other tasks

Original: [Table]
Revised: [Table]

Original: we conduct experiments on segmentation.
Revised: we conduct experiments on segmentation.

Original: Compared to the method Dilated FCN with ResNet50 as basenet (Fu et al., 2018)
Revised: Compared to the method Dilated FCN with ResNet50 as basenet (Fu et al., 2018)

Original: [Table]
Revised: [Table]

Original: Dilated FCN with DyResNet50 reduces 69.3% FLOPs while maintaining the MIoU on Cityscapes validation set.
Revised: Dilated FCN with DyResNet50 reduces 69.3% FLOPs while maintaining the MIoU on Cityscapes validation set.

Original: The result are shown in Table 3.
Revised: The result are shown in Table 3.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 4. A BLATION STUDY
Revised: A BLATION STUDY

Original: Comparison between dynamic convolution and static convolution We correspondingly design two networks without dynamic convolution.
Revised: Comparison between dynamic convolution and static convolution We correspondingly design two networks without dynamic convolution.

Original: Speciﬁcally, we remove the correlation prediction module and use ﬁxed convolution kernel for Dy-mobile (1.0) and Dy-shufﬂe (1.5) [Table] and we keep the channel number the same as the dynamic convolution neural networks.
Revised: Speciﬁcally, we remove the correlation prediction module and use ﬁxed convolution kernel for Dy-mobile (1.0) and Dy-shufﬂe (1.5), and we keep the channel number the same as the dynamic convolution neural networks.

Original: We denote the baseline networks as Fix-mobile(1.0) and Fix-shufﬂe (1.5) respectively.
Revised: We denote the baseline networks as Fix-mobile(1.0) and Fix-shufﬂe (1.5) respectively.

Original: The results are shown in Table 4,[Table] [Table] compare with baseline networks Fix-mobile (1.0) and Fix-shufﬂe (1.5), the proposed Dy-mobile (1.0) and Dy-shufﬂe (1.5) achieve absolute classiﬁcation improvements by 5 .
Revised: The results are shown in Table 4, compare with baseline networks Fix-mobile (1.0) and Fix-shufﬂe (1.5), the proposed Dy-mobile (1.0) and Dy-shufﬂe (1.5) achieve absolute classiﬁcation improvements by 5 .

Original: 19% and 2 .
Revised: 19% and 2 .

Original: 82% respectively.
Revised: 82% respectively.

Original: This shows that directly decreasing the channel number to reduce computation cost inﬂuences the classiﬁcation performance a lot.
Revised: This shows that directly decreasing the channel number to reduce computation cost inﬂuences the classiﬁcation performance a lot.

Original: While the proposed dynamic kernel can retain the representation ability as mush as possible. 
Revised: While the proposed dynamic kernel can retain the representation ability as mush as possible. [Table] [Table] [Table]

Revised: [Table]

Original:  Effectiveness of g t for dynamic kernel
Revised: [Table] [Table] [Table] Effectiveness of g t for dynamic kernel

Original: The group size g t in Eq.
Revised: The group size g t in Eq.

Original: 1 does not change the computation cost of dynamic convolution, but affects the performance of network.
Revised: 1 does not change the computation cost of dynamic convolution, but affects the performance of network.

Original: Thus we provide ablative study on g t .
Revised: Thus we provide ablative study on g t .

Original: We set g t as 2,4,6 for dy-mobile(1.0) respectively and the results are shown in Table 5.
Revised: We set g t as 2,4,6 for dy-mobile(1.0) respectively and the results are shown in Table 5.

Original: The performance of dy-mobile(1.0) becomes better when g t gets larger.
Revised: The performance of dy-mobile(1.0) becomes better when g t gets larger.

Original: It is reasonable because larger g t means the number of kernels cooperated for obtaining one noise-irrelevant feature becomes larger.
Revised: It is reasonable because larger g t means the number of kernels cooperated for obtaining one noise-irrelevant feature becomes larger.

Original: When g t = 1 , the coefﬁcient prediction module can be regarded as merely learning the attention for different channels, which can improve the performance of networks as well (Hu et al., 2018).
Revised: When g t = 1 , the coefﬁcient prediction module can be regarded as merely learning the attention for different channels, which can improve the performance of networks as well (Hu et al., 2018).

Original: Therefore we provide ablative study for comparing g t = 1 and g t = 6 on Dy-mobile(1.0) and
Revised: Therefore we provide ablative study for comparing g t = 1 and g t = 6 on Dy-mobile(1.0) and

Original: Dy-ResNet18.
Revised: Dy-ResNet18.

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: From the table we can see that, setting g t = 1 will reduce the Top-1 accuracy on ImageNet for Dy-mobile(1.0) and Dy-ResNet18 by 2.58% and 2.79% respectively.
Revised: From the table we can see that, setting g t = 1 will reduce the Top-1 accuracy on ImageNet for Dy-mobile(1.0) and Dy-ResNet18 by 2.58% and 2.79% respectively.

Original: It proves that the improvement of our proposed dynamic networks does not only come from the attention mechanism.
Revised: It proves that the improvement of our proposed dynamic networks does not only come from the attention mechanism.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a DyNet method to adaptively generate convolution kernels based on image content, which reduces the redundant computation cost existed in conventional ﬁxed convolution kernels.
Revised: In this paper, we propose a DyNet method to adaptively generate convolution kernels based on image content, which reduces the redundant computation cost existed in conventional ﬁxed convolution kernels.

Original: Based on the proposed DyNet, we design several dynamic convolution neural networks based on well known architectures, i.e., Dy-mobile, Dy-shufﬂe, Dy-ResNet18, Dy-ResNet50.
Revised: Based on the proposed DyNet, we design several dynamic convolution neural networks based on well known architectures, i.e., Dy-mobile, Dy-shufﬂe, Dy-ResNet18, Dy-ResNet50.

Original: The experiment results show that DyNet reduces 37 . 0% , 54 .
Revised: The experiment results show that DyNet reduces 37 . 0% , 54 .

Original: 7% , 67 . 2% and 71 .
Revised: 7% , 67 . 2% and 71 .

Original: 3% FLOPs respectively, while maintaining the performance unchanged.
Revised: 3% FLOPs respectively, while maintaining the performance unchanged.

Original: As future work, we want to further explore the redundancy phenomenon existed in convolution kernels, and ﬁnd other ways to reduce computation cost, such as dynamically aggregate different kernels for different images other than ﬁxed groups used in this paper.
Revised: As future work, we want to further explore the redundancy phenomenon existed in convolution kernels, and ﬁnd other ways to reduce computation cost, such as dynamically aggregate different kernels for different images other than ﬁxed groups used in this paper.

Original: A A PPENDIX A. D ETAILED ANALYSIS OF OUR MOTIVATION
Revised: A A PPENDIX A. D ETAILED ANALYSIS OF OUR MOTIVATION

Original: We illustrate our motivation from a convolution with output f ( x ) , i.e.,
Revised: We illustrate our motivation from a convolution with output f ( x ) , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where ⊗ denotes the convolutional operator, x ∈ R n is a vectorized input and w ∈ R n means the ﬁlter.
Revised: where ⊗ denotes the convolutional operator, x ∈ R n is a vectorized input and w ∈ R n means the ﬁlter.

Original: Speciﬁcally, the i th element of the convolution output f ( x ) is calculated as:
Revised: Speciﬁcally, the i th element of the convolution output f ( x ) is calculated as:

Original: [Equation]
Revised: [Equation]

Original: ) where (cid:104)· , ·(cid:105) provides an inner product and x ( i ) is the circular shift of x by i elements.
Revised: ) where (cid:104)· , ·(cid:105) provides an inner product and x ( i ) is the circular shift of x by i elements.

Original: We deﬁne the index i started from 0 .
Revised: We deﬁne the index i started from 0 .

Original: We denote the noises in x ( i ) as (cid:80) d − 1 j =

Revised: We denote the noises in x ( i ) as (cid:80) d − 1 j =

Original: α j y j , where α j ∈ R and { y 0 , y 1 , ..., y d − 1 } are the base vectors of noise space Ψ .
Revised: α j y j , where α j ∈ R and { y 0 , y 1 , ..., y d − 1 } are the base vectors of noise space Ψ .

Original: Then the kernels in one convolutional layer can be represented as { w 0 , w 1 , ..., w c } .
Revised: Then the kernels in one convolutional layer can be represented as { w 0 , w 1 , ..., w c } .

Original: The space expanded by { w 0 , w 1 , ..., w c } is Ω .
Revised: The space expanded by { w 0 , w 1 , ..., w c } is Ω .

Original: We can prove if the kernels are trained until Ψ ⊂ Ω , then for each w k / ∈ Ψ , we can get the noise-irrelevant f i ( x white )
Revised: We can prove if the kernels are trained until Ψ ⊂ Ω , then for each w k / ∈ Ψ , we can get the noise-irrelevant f i ( x white )

Original: =
Revised: =

Original: (cid:104) x white ( i ) , w k (cid:105) by the cooperation of other kernels w 0 , w 1 , ... .
Revised: (cid:104) x white ( i ) , w k (cid:105) by the cooperation of other kernels w 0 , w 1 , ... .

Original: Firstly x ( i ) can be decomposed as:
Revised: Firstly x ( i ) can be decomposed as:

Original: [Equation]
Revised: [Equation]

Original: 6) where β ∈ R and ¯ x ∈ R n is vertical to w k and y j .
Revised: 6) where β ∈ R and ¯ x ∈ R n is vertical to w k and y j .

Original: For concision we assume the norm of w k and y j is 1.
Revised: For concision we assume the norm of w k and y j is 1.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: When there is no noise, i.e. α j = 0 for j = 0 , 1 , ..., d − 1 , the white output f i ( x white ) becomes:
Revised: When there is no noise, i.e. α j = 0 for j = 0 , 1 , ..., d − 1 , the white output f i ( x white ) becomes:

Original: [Equation]
Revised: [Equation]

Original: It is proved in the Appendix A.2 that:
Revised: It is proved in the Appendix A.2 that:

Original: [Equation]
Revised: [Equation]

Original: where β 0 , ..., β c is determined by the input image.
Revised: where β 0 , ..., β c is determined by the input image.

Original: Eq. 9 is fulﬁlled by linearly combine convolution output (cid:104) w k , x ( i ) (cid:105) and (cid:104) w t , x ( i ) (cid:105) for those β t (cid:54) = 0 in the following layers.
Revised: Eq. 9 is fulﬁlled by linearly combine convolution output (cid:104) w k , x ( i ) (cid:105) and (cid:104) w t , x ( i ) (cid:105) for those β t (cid:54) = 0 in the following layers.

Original: Thus if there are N coefﬁcients in Eq. 9 that are not 0, then we need to carry out
Revised: Thus if there are N coefﬁcients in Eq. 9 that are not 0, then we need to carry out

Original: N times convolution operation to get the noise-irrelevant output of kernel w t , this causes redundant calculation.
Revised: N times convolution operation to get the noise-irrelevant output of kernel w t , this causes redundant calculation.

Original: InEq.9,we can observe that the computation cost can be reduced to one convolution operation by linearly fusing those kernels to a dynamic one:
Revised: InEq.9,we can observe that the computation cost can be reduced to one convolution operation by linearly fusing those kernels to a dynamic one:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In Eq.
Revised: In Eq.

Original: 10, the coefﬁcients β 0 , β 1 , ... is determined by α 0 , α 1 , ... , thus they should be generated based on the input of network.
Revised: 10, the coefﬁcients β 0 , β 1 , ... is determined by α 0 , α 1 , ... , thus they should be generated based on the input of network.

Original: This is the motivation of our proposed dynamic convolution.
Revised: This is the motivation of our proposed dynamic convolution.

Original: A. P ROVING OF E Q .
Revised: A. P ROVING OF E Q .

Original: We denote g ij ( x ) as (cid:104) x
Revised: We denote g ij ( x ) as (cid:104) x

Original: ( i ) , y j (cid:105) ,
Revised: ( i ) , y j (cid:105) ,

Original: j = 0 , 1 , ..., d − 1 .
Revised: j = 0 , 1 , ..., d − 1 .

Original: Then,
Revised: Then,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By summarize Eq. 7 and Eq. 11, we get the following equation:
Revised: By summarize Eq. 7 and Eq. 11, we get the following equation:

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:    βα
Revised:    βα

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original:   f i
Revised:   f i

Original: [Equation]
Revised: [Equation]

Original: x ) g i
Revised: x ) g i

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: g i
Revised: g i

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: We simplify this equation as:
Revised: We simplify this equation as:

Original: [Equation]
Revised: [Equation]

Original: Because w k / ∈ Ψ , we can denote w k as:
Revised: Because w k / ∈ Ψ , we can denote w k as:

Original: [Equation]
Revised: [Equation]

Original: 14) where w ⊥ is vertical to y 0 , ..., y d − 1 and γ ⊥ (cid:54) = 0 .
Revised: 14) where w ⊥ is vertical to y 0 , ..., y d − 1 and γ ⊥ (cid:54) = 0 .

Original: moreover because | w k | = 1 ,thus
Revised: moreover because | w k | = 1 ,thus

Original: [Equation]
Revised: [Equation]

Original: It can be easily proved that:
Revised: It can be easily proved that:

Original: [Equation]
Revised: [Equation]

Original: thus,
Revised: thus,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =

Original: (cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 − γ 20 − γ  ...
Revised: (cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 − γ 20 − γ  ...

Original: γ d − 1 γ   ...
Revised: γ d − 1 γ   ...

Original: γ  ... γ d −0 . . .
Revised: γ  ... γ d −0 . . .

Original: (cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
Revised: (cid:12) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 − γ 20 − γ 21 − ... − γ 2 d −  ...γ  ...γ  ...... γ d −0 . . .
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 − γ 20 − γ 21 − ... − γ 2 d −  ...γ  ...γ  ...... γ d −0 . . .

Original: ...(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
Revised: ...(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =

Original: (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) γ 2 ⊥  ...γ  ...γ  ......
Revised: (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) γ 2 ⊥  ...γ  ...γ  ......

Original: γ d −0 . . .
Revised: γ d −0 . . .

Original: ...(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
Revised: ...(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =

Original: γ 2 ⊥ (cid:54) = 0 .
Revised: γ 2 ⊥ (cid:54) = 0 .

Original: thus,
Revised: thus,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If we denote the elements of the ﬁrst row of A − 1 as a 00 , a 01 , ..., a 0 d , then
Revised: If we denote the elements of the ﬁrst row of A − 1 as a 00 , a 01 , ..., a 0 d , then

Original: [Equation]
Revised: [Equation]

Original: Because Ψ ⊂ Ω , there exists { β t ∈ R | t = 0 , 1 , ..., c } that
Revised: Because Ψ ⊂ Ω , there exists { β t ∈ R | t = 0 , 1 , ..., c } that

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]