A DAPTIVE C ONVOLUTIONAL N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The quest for increased visual recognition performance has led to the development of highly complex neural networks with very deep topologies.
To avoid high computing resource requirements of such complex networks and to enable operation on devices with limited resources, this paper introduces adaptive kernels for convolutional layers.
Motivated by the non-linear perception response in human visual cells, the input image is used to deﬁne the weights of a dynamic kernel called Adaptive kernel.
This new adaptive kernel is used to perform a second convolution of the input image generating the output pixel.
Adaptive kernels enable accurate recognition with lower memory requirements; This is accomplished through reducing the number of kernels and the number of layers needed in the typical CNN conﬁguration, in addition to reducing the memory used, increasing 2X the training speed and the number of activation function evaluations.
Our experiments show a reduction of 66X in the memory used for MNIST, maintaining 99% accuracy and 16X memory reduction for CIFAR10 with 92.5% accuracy.
I NTRODUCTION
Convolutional Neural Networks (CNN) have demonstrated their capacity to achieve state of the art accuracy in image classiﬁcation, semantic segmentation, and object detection.
In order to increase the accuracy of recognition, most of these works rely on deeper and deeper architectures with millions of parameters.
The reason behind this is that more complex features can be abstracted as we add more layers to a network.
Unfortunately such models can not be used in embedded devices, cellphones or drones due to the size of the models.
Recently there are many approaches like ShufﬂeNet Xiangyu Zhang (2017), MobileNet Andrew G. Howard (2017), HENet Qiuyu Zhu (2018) and
SqueezeNet Forrest N. Iandola (2017) trying to generate small models with an small drop in accuracy.
The proposed method generates the smallest model with comparable accuracy.
In addition this method can be combined with most of the existent techniques like ResNet generating an improved version of it.
Previous research has demonstrated that the response of visual cells is a non-linear function of their stimuli (Szulborski & Palmer, 1990).
Thus, ﬁnding non-linear models that best represent data is imperative.
Given that a typical convolutional layer is a linear system, its ability to express this response is limited by the layers and the number of neurons in the intermediate layers.
The use of a non-linear neuron, as it was ﬁrstly explored to solve the XOR problem by Minsky & Papert (1969), (which cannot be solved by a ﬁrst order neuron, but it can by a second order neuron), seems an appropriate way to tackle this kind of non linearity issue.
Ideally, such non-linear approaches should be able to provide similar accuracy as compared to traditional CNNs, albeit at a much lower computation and memory costs.
In other hand, in computer vision the ﬁlter used to extract borders is different to the ﬁlter used to extract corners, etc.
Our method uses the input image to deﬁne the ﬁlter that better ﬁts in on the speciﬁc location. Motivated by this, we developed a convolutional kernel, that includes non-linear transformations obtaining similar results as the state of the art algorithms, while yielding a reduction in required memory up to 14x in the CIFAR10 (Krizhevsky, 2009) classiﬁcation, and up to 66x for the MNIST classiﬁcation.
The main contributions of this paper are: (i) we present the non-linear convolutions designed for high visual classiﬁcation accuracy under memory constraints; (ii) using the proposed convolutions, we present a network design that is partially pre-deﬁned and is capable of completing self-deﬁnition during the pattern evaluation phase, including deﬁning the convolutional kernels on the ﬂy depending of the input pattern; (iii) we present a method to tackle problems associated with higher order neural networks like the saturation of the activation due to the N-order of multiplications used; (iv) a method to constrain every new dynamically generated weight to a pre-known range deﬁned by the activation function, for instance if hyperbolic tangent function was used, the dynamically generated weights would be all in ( − 1 , 1) range; and (v) a pytorch-based implementation located in https://github.com/adapconv/adaptive-cnn (2018).
This work is organized as follows: Section 2 describes previous approaches, Section 3 explains in detail our proposed method, Section 4 shows the results for MNIST, CIFAR10 and Navigation.
We end with our conclusions in Section 5.
R ELATED WORK
Different ways of increasing the accuracy of neural networks have been addressed in the existing scientiﬁc literature, most of these rely on the use CNNs, as they are generalized linear models, and their level of abstraction is low (Lin et al., 2013).
Some approaches have dealt with this low abstraction by having additional layers (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014), resulting in a considerable increase in the accuracy on different datasets, e.g. CIFAR10 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009).
However, although the network depth has shown a crucial importance in neural network performance, the difﬁculty to train the network also increases, and moreover, the accuracy of the network drops (He et al., 2015).
A proposal to tackle this problem is the use of ResNet blocks He et al.
(2015), which given a set of inputs X , with an associated label Y and a function H ( x ) that maps X to Y , the networks deﬁne a building block y
= F ( x ) + x , where F ( x ) represents the residual mapping to be learned.
This residual network allows to train a deeper network without being affected by the degradation problem, using a larger amount of layers and parameters.
In the work presented in Lin et al.
(2013) a nonlinear function approximator is proposed as a solution to increase the level of abstraction.
The typical convolutional kernel is replaced with a micro network, i.e. a non linear approximator.
A multilayer perceptron is used as the instantiation of this micro network, and by sliding the micro network over the input in a similar manner as convolutional neural networks the feature maps are obtained.
In the approach of Brabandere et al.
(2016), a dynamic ﬁlter module is introduced, where the ﬁlters used for the convolution are generated dynamically depending on the input.
This module consists of two parts: a ﬁlter generating network, which generates sample speciﬁc parameters given an input; and the dynamic ﬁltering layer, which applies the parameters to another input.
Although the mentioned work presents similar components to our proposal, it is important to notice the differences between these: in Brabandere et al.
a CNN is trained to get the convolutional kernels, and then another network is deﬁned to generate such kernels, where this second network is trained independently.
In our work, we deﬁne a second order convolutional kernel trained using a novel training rule, which is explained in detail in the next section.
M ETHOD
An Adaptive Kernel is a dynamic kernel that changes its weights by itself depending on the input image.
An Adaptive Kernel can be viewed as an array of traditional kernels.
For instance, each element in a 3x3 adaptive kernel is a 3x3 linear kernel, as shown in the Figure 1.
The convolution of each kernel element Q u,v with the window X of the input image generates a component of the new kernel K u,v after applying the activation function (Figure 1).
This new dynamically generated kernel K is convolved again with the same window X of the input image (Figure 2).
As result of this second order convolution and the activation function an output pixel value is obtained.
By sliding the window through all the input image, we generate the ﬁnal ﬁltered image.
3. F EED FORWARD
Convolving the kernel Q u,v in the Figure 1 and using tanh as activation function the weights K u,v are generated
[Equation]
where σ u,v is the convolution of each linear kernel Q u,v with the input image
[Equation]
This new kernel K u,v is now convolved with the input image to compute S in this location
[Equation]
and ﬁnally the output pixel is computed using hyperbolic tangent as activation function like f = tanh ( S ) .
Another activation functions can be used, like sigmoid or Relu.
We use hyperbolic tangent to generate weights in the range of ( − 1 , 1) .
3. T RAINING
For the Adaptive kernel a new training rule is obtained using gradient descent technique in order to adjust the weights of adaptive kernel Q .
The kernel can be seen as an array of NxN traditional kernels, using Q u,v to refer to each linear kernel and using ( i, j ) to refer the elements (weights) of each Q u,v kernel.
It means the element Q ( u,v ) ( i,j ) is a scalar value and represents a weight.
The kernel output f ( S ) k,l , where f is the activation function and S represents sum of input x u,v weighted by σ u,v as shown in Equation (3).
The error for output pixel ( k, l ) denoted as E k,l is given by:
[Equation]
where d k,l represents the expected output at coordinates ( k, l ) .
The training rule for the t − th iteration ( Q t ) is given by the error derivative per component ( u, v ) i,j :
[Equation]
The gradient descent updates of the weights Q t ( u,v ) i,j are given by:
[Equation]
where γ represents the learning rate.
Using the hyperbolic tangent as the activation function f
[Equation]
) k, [Equation] [Equation] where S k,l is the weighted sum computed as:
[Equation]
Here, N is the size of the kernel, and K k,lu,v ( σ ) =
tanh( σ k,lu,v ) , with:
[Equation]
Since the hyperbolic tangent was used as the activation function, the gradient is:
[Equation]
by replacing (10) in (6), the ﬁnal training rule is deﬁned by:
[Equation]
where γ is the learning rate, d k,l is the desired value at the window position ( k, l ) , f is the activation function and δ k,l is [Equation]
[Equation]
For this training rule tanh was used as activation function to generate the kernel weights, it ensures all weight are in the range of ( − 1 , 1) .
Additionally, this layer allows the reduction of ﬁlters in the subsequent layers without affecting the performance of the network, on the experimental results different experiments will be used to compared against state of the art for MNIST and CIFAR10 to show a signiﬁcant memory compression using the proposed models.
R ESULTS
Our implementation of the adaptive kernels was written as a layer of Caffe (Jia et al., 2014) with forward and backward propagation.
For CIFAR-10 (Krizhevsky, 2009) we use Nesterov (Nesterov, 1983) as training rule, with momentum set to 0.9, initializing the learning rate to 0.1, dropping it by a factor of 5 every 20 epochs, and a weight decay of 0.0001.
We use the same weight initialization as He et al.
A 32x32 crop is randomly sampled from a 40x40 image or its horizontal ﬂip, with the per-pixel mean subtracted and divided by the channel standard deviation.
For MNIST we use the same parameters, changing the learning rate to 0.01 and the weight decay to 0.0005.
With no data augmentation or pre-processing.
4. E XPERIMENT 1: MNIST
The MNIST is a public data set, it consists of 60,000 28x28 grey scale images in 10 classes (handwritten numbers), with 6000 images per class.
There are 50,000 training images and 10,000 test images in the ofﬁcial data.
Here our approach has three main advantages: memory reduction, increment of accuracy outperforming the traditional model of CNNs, and the learning speedup.
The results produced by the implementation of the MNIST digits recognition, show a big memory compression, using 66X less memory measured through parameter reduction, additionally high accuracy was achieved 2x faster.
In the table 1a the LeNet neural network architecture used as reference is detailed 1 and table 1b describes our topology.
[Table]
The (ﬁgure 3A) shows a few examples of how the Adaptive kernel is changing according to the input window.
The Figure 3B shows all the kernels generated for random sample of digit seven, For all the back ground pixels the kernels is simple neutral it does not extract any features there. There are many different NN Models of MNIST classiﬁcation, In the table 2 a subset is presented, selecting only the models having over 99% accuracy with small number of parameters [Table]
[Table] In order to perform a Hyper-parameter sensibility analysis, eight different models with three layers were trained using MNIST. By increasing the number of kernels in the ﬁrst layer, a saturation on the accuracy can be seen, but this saturation can be mitigated incrementing the number of kernels in the second layer (3). [Table]
In this scenario our technique is the smallest CNN model that reaches 99% without any preprocessing in only 5 epochs, in contrast, the LeNet reached 97% after 9 epochs.
In terms of the number of operations the LeNet as in the tutorial has 2.29M MAC operations, while our method has 1.23M MAC operations for MNIST.
4. E XPERIMENT 2:CIFAR
The CIFAR-10 is a public data set, it consists of 60,000 32x32 color images in 10 classes, withimages per class.
There are 50,000 training images and 10,000 test images in the ofﬁcial data.
In this scenario CIFAR10 was used for testing and we used horizontal ﬂipping, padding and 32x32 random cropping for data augmentation of the training dataset.
In the selected topology: only the ﬁrst layer uses adaptive kernels of (3 x 3) (3 x 3) in order to highlight the impact of a single layer in the full topology, being the ﬁrst layer where the main feature extraction takes place.
It has 8 convolutional layers and ends with ten outputs in the fully connected layer.
In order to compare against related work we include some of the latest topologies used for CIFAR pattern recognition problem.
Our intention is not to outperform the accuracy of these topologies.
Instead, the idea is to achieve similar results with a smaller model.
This approach can potentially enable us to target embedded systems.
In the Table 4 we provide a summary of the most popular methods recently used for CIFAR10.
As the Table 4 shows the focus of the prior work has been the improvement of recognition accuracy without regard to the amount of memory and time required for classiﬁcation.
The neural networks achieving the highest scores use over 20 million parameters, making it very hard to implement them in low capacity embedded devices.
Our solution reduces memory usage; it only uses 200K parameters.
While there is no simple way to determine the efﬁciency of a neural [Table] network, our target is the highest accuracy with the least amount of memory as shows (Table 4) [Table]
on accuracy the Volterra-based Wide ResNet Zoumpourlis et al.
(2017) provides the best score but uses 36M parameters.
Our solution, the one with less number of parameters represents a compression of 184X with 4% drop in accuracy or comparing with All-CNN Springenberg et al.
(2014) our solution represents a 6X memory reduction with 0 .
2% accuracy drop.
Additional experiments were performed taking as base the ResNet18 topology, in this experiment the number of parameters was not taken into account and only the ﬁrst layer was changed to useadaptive kernels instead of 64 convolutional kernels, as in the typical topology, in order to show the contribution of one adaptive layer.
Although the topology with an adaptive layer has less feature maps in the ﬁrst layer, it can achieve even better results than ResNet50 see table 5.
[Table]
4. E XPERIMENT 3:G ENERALIZATION
In this experiment an internal data set was used, in order to train a neural network model, that drives a robot to navigate inside of a known room.
Basically given an input image I from the robot camera, estimate a required direction α ∈ [0 , 360] and a distance d ∈ [0 , 100] , that drives the drone to reach the center c ( x c , y c ) of a known region.
For this purpose, a NN architecture was created inspired by
DroNet Loquercio et al.
(2018), instead of a standard convolutional layer, an adaptive convolutional layer was used (Table 6b).
The last two layers of the network are trained considering a classiﬁcation problem, in order to estimate a steering angle class and a distance class.
The compression is achieved by reducing the number of ﬁlters in the three ResNet blocks.
In the table we have k , 2 k , and 4 k kernels for each block respectively.
When the compression increases
[Table] the accuracy drops, but the Adaptive kernels helps to keep a better accuracy in comparison with the traditional convolutional layers (4). 4. C ONCLUSION In this paper we presented adaptive convolutional kernels, capable of redeﬁning the convolutional kernel during the inference time, depending on the input image.
Our technique not only reduces memory usage
[Table]
but also reduces the training time.
Additionally, our results show adaptive convolutional kernels generalize better than traditional CNNs.
Kernels adapt dynamically to extract better features depending of the input image [Table] and in terms of efﬁciency, our method has shown to be very compelling generating 10X lighter solutions with less than 1% accuracy drop.
Our solution should be able to provide direct impact to the computational cost of the inference on embedded systems, increasing the operational scope of these systems.
In terms of accuracy, our proposed Adaptive kernel can imitate any traditional constant kernel (by training the dynamic kernel to be constant independent of the input); this means that it can be trained to generate the weights in the best in class solution and start from there to outperform it.
As in traditional CNNs, the increment of the number of kernels in a layer produces some saturation, with a marginal increment of accuracy, thus the topology also plays an important role.
In our experiments it was observed that less adaptive kernels in a layer generates comparable or even better level of abstraction than a higher number of traditional convolutional kernels in a layer. For instance, in ResNet18, the 64 convolutional ﬁlters in the ﬁrst layer were changed to only 16 Adaptive ﬁlters producing better results than the topology with 50 layers. In addition fully connected layers can use this technique, one adaptive layer can replace two traditional layers.