Filter redistribution templates for iteration-less convolutional model reduction
Anonymous authors
Paper under double-blind review
A BSTRACT
Automatic neural network discovery methods face an enormous challenge caused by the size of the search space.
A common practice is to split this space at different levels and to explore only a fraction of it.
On one hand, neural architecture search methods look at how to combine a subset of layers to create an architecture while keeping a predeﬁned number of ﬁlters in each layer.
On the other hand, pruning techniques take a well known architecture and look for the appropriate number of ﬁlters per layer.
In both cases, the exploration is made iteratively, training models several times during the search.
Inspired by the constraints and advantages of these two approaches, we propose a straight-forward and fast option to ﬁnd models with improved characteristics.
We apply a small set of templates, that have been heuristically and experimentally evaluated, to make a one-shot redistribution of the number of ﬁlters in an already existing neural network.
When compared to the initial base models we found that the resulting architectures, when trained from scratch, surpass the original accuracy even after been reduced to ﬁt to the original amount of resources.
Speciﬁcally, we show accuracy improvement for some network-task pairs of up to 5.5%, a reduction of up to 45% in parameters and 60% reduction in memory footprint.
I NTRODUCTION
Convolutional neural networks are built by stacking layers of neurons following the principle explained by Fukushima’s Neocognitron model (Fukushima, 1980).
The Neocognitron design made neural networks invariant to shift in feature locations by arranging cells locally connected in a hierarchical architecture.
Instead of connecting every neuron of the previous layer to all the neurons in the next layer, convolutional networks connections are only made to a small region.
Given that different regions share the same weights, the layer can be implemented as a convolution operation using the set of shared weights known as kernel.
The complete input is processed by shifting the kernel at uniform steps, normally overlapping parts of the input.
To improve shift invariance, convolutional networks needs to rely less in the exact position of a feature and a simple solution is to have a lower resolution performed by averaging the values of neighbouring points in the image in a operation known as spatial subsampling.
An important consideration to create a convolutional network model is the number of ﬁlters, required at every layer.
The Neocognitron implementation for example, keeps a ﬁxed number of ﬁlters for each layer in the model.
A very common practice has been to use a by-pyramidal architecture.
The number of ﬁlters across the different layers is usually increased as the size of the feature maps decrease.
This pattern was ﬁrst proposed by LeCun et al.
(1998) with the introduction of LeNet and can be observed in a diverse set of models such as VGG, ResNet and MobileNet (See Figure 5 in Appendix).
Even models obtained from automatic model discovery, like NASNet, follow this principle inasmuch as neural network methods are mainly formulated to search for layers and connections.
It can be found in (LeCun et al., 1998) that the reason behind this progressive increase in the number of kernels is to compensated a possible loss of the representation caused by the spatial resolution reduction.
In recent models, what seems to be the real reason is a practical issue (Chu & Krzy˙zak, 2014), to improve performance by keeping a constant number of operations in each layer.
The Pyramidal distribution of ﬁlters has perpetuated over two areas of model discovery.
The methods in Automatic Neural Architecture Search (Liu et al., 2018a; Tan et al., 2019) explore the models built by combining a predeﬁned set of layers, commonly with a pyramidal distribution of ﬁlters.
On the other side, Network Pruning aims to reduce a model computational resources demands by selecting and removing weights that match some rule, commonly the closest to zero values, but starting from models that present this pyramidal distribution.
To the best of our knowledge, it remains unknown if this pyramidal distribution of ﬁlters is also beneﬁcial to different aspects of model performances other than the number of operations. How the distribution of resources in a deep network model affect accuracy, memory footprint, inference time and model compression level are also of high interest in a multi-performing space that networks have to operate in.
This paper explores the topic by comparing models against versions of themselves with the same general structures but with the distribution of ﬁlters across the layers changed.
We present a straightforward and fast to implement method for model discovery, that takes the goal of structured pruning methods, that is ﬁnding the lowest number of ﬁlters for a model while maintaining accuracy.
Our exploration technique only tests over a small subset of diverse ﬁlter distributions, that we call templates, and that reduce the model proportionally to match some resource budget.
Our experiments show that by using our proposed templates, the resulting models keep comparable accuracy as the original models in classiﬁcation tasks but present reductions in the number of parameters and/or memory footprint.
The contributions of this paper are the following: 1) it provides evidence that pyramidal distribution of ﬁlters in convolutional network models is usually optimised for a distributed GPU operation across layers, and simple changes to that distribution leads to improvements in metrics such as number of parameters or memory footprint; 2) it highlights that most recent models, which have had a more detailed tuning in the ﬁlter distribution, present resiliency in accuracy to changes in the ﬁlter distribution, a phenomena that requires further research and explanation; 3) it shows that redistributing ﬁlters in a model and then applying a width multiplier operation can be seen as a pruning technique which produces smaller models than just applying the width multiplier to the original models; 4) it gives classical models a repositioning of their merit when measuring other practical implementation resources including and beyond the number of operations. And, 5) A practical and fast alternative approach for model search that can work in addition of conventional architecture search and network pruning.
The rest of the paper is structured as follows: Chapter 2 explores the most recent methods to reduce the size of neural network architectures.
Chapter 3 describes the set of templates for ﬁlter distribution and how to implement this change in a convolutional network model.
Chapter 5. compares dissimilar allocations of ﬁlters and their effect in model performances.
Finally, chapterbrieﬂy explains the ﬁndings derived from experiments.
R ELATED W ORK
The process of designing a Neural Networks is a task that largely has been based on experience and experimentation that consumes a lot of time and computational resources.
Of note are reference models such as VGG, ResNet, Inception and similar that have been developed entirely or with signiﬁcant use of heuristics.
With the increase in the use of Neural Networks, and particularly Convolutional Networks for computer vision problems, a mechanism to automatically ﬁnd the best architecture has become a requirement in the ﬁeld of Deep Learning.
Although some works have been published several years ago (Pinto et al., 2009; Kuri-Morales, 2014) trying to address the topic of automatic architecture generation, they have not provided competitive results compared to many hand-crafted architectures.
However, current works start to lead the state of the art models (Elsken et al., 2018).
But even with automatic methods, one key feature that constantly has been adopted is the selection of the number of ﬁlters in each layer in the ﬁnal model.
The ﬁlters are set in such a way to have an increasing number as the layers go deeper.
Pruning methods have done some work in this ﬁeld {+Base   
 Layer N u m be r o f F il t e r s+}
{+Reverse−Base   
+}
Layer
N u m be r o f F il t e r s
Uniform   16 343.
Layer
N u m be r o f F il t e r s
Quadratic
Layer
N u m be r o f F il t e r s
{+Negative−Quadratic   
+}
Layer
N u m be r o f F il t e r s
Base distribution, which is the original distribution, shows the common design of growing the ﬁlters when resolution of feature maps decreases in deeper layers.
but with the belief that the weights obtained at the end of the training process are important to the pruning method.
One common characteristic of many model discovery methods is that the search process is very time consuming, normally in the order of thousands of GPU hours.
In this sense, a remarkable improvement is presented by Liu et al.
(2018b).
That method uses a relaxation condition to transform the selection of layers in the architecture to a continuous space using a softmax function.
The relaxation allows to perform simultaneous search of weights and architecture using gradient descent.
The method converges after one day of GPU time.
On the side of Pruning methods the search also involves training models for several iterations to select the correct weights to remove (Frankle & Carbin, 2018;
He et al., 2019), or at least increasing the computation during the training when doing jointly training and search (Leclerc et al., 2018).
Recently, Liu et al.
(2018c) suggest that accuracy obtained by pruning techniques can be reached by training from scratch.
Our work relates to (Gordon et al., 2018) in the sense that their method is not restricted to reducing ﬁlters but also to increase them to see if the increment is beneﬁcial.
Our approach differs however, because it does not require to train the model other than in the ﬁnal stage, after making some predeﬁned changes to the number of ﬁlters using the redistribution template.
F ILTER DISTRIBUTION TEMPLATES
Recent pruning methods such as Gordon et al. ; Leclerc et al. (2018), have shown different ﬁlter distribution patterns emerging when reducing models like VGG that defy the notion of pyramidal design as the best distribution for a model.
This is a motivational insight into what other distributions can and should be considered when designing models.
On one side the combinatorial space of distributions make this a challenging exploration, on the other however, it importantly highlights the need to pursue such exploration if gains in accuracy and overall performance can be made.
In this work, rather than attempting to ﬁnd the optimal ﬁlter distribution with expensive automatic pruning or growing techniques, we propose to ﬁrst adjust the ﬁlters of a convolutional network model via a small number of pre-deﬁned templates. These templates such as those depicted in ﬁgure 1, are inspired by existing models that have already been found to perform well and thus candidates that could be beneﬁcial for model performance improvement beyond the number of operations. Performance criteria such as accuracy, memory footprint and inference time and model compression level are arguably as important as the number of operations required.
In particular, we adopt as one template a distribution with a ﬁxed number of ﬁlters, as with the original Neocognitron design, but also other templates inspired by the patterns found in (Gordon et al., 2018) where at least three behaviours are present in different blocks from the resulting ResNet model: 1) ﬁlters increase in deeper layers, 2) ﬁlters agglomerate in the centre and 3) ﬁlters are reduced in the centre of the block. (Leclerc et al., 2018) shows also a ﬁlter pattern with more ﬁlters in the centre of a VGG model. Based on these observations we deﬁne the templates we use in this work.
We deﬁne a convolutional neural network base model as a set of layers L = 1 , ..., D + 1 each of them containing a number of ﬁlters f l and a total number of ﬁlters F = (cid:80) Dl
=1 f l .
We want to test if the common heuristic of distributing F having f l +1 = 2 f l each time the feature map is halved, is advantageous to the model over other distributions of f l when evaluating performance, memory footprint and inference time.
It should be noted that that ﬁnal layer D + 1 remains with the same number of ﬁlters according to the task under evaluation, therefore it is not taken into account in the equations.
Another important consideration is that, in architectures composed for modules or blocks (ResNet and Inception), it is easier to change the number of ﬁlters in the module as a whole than to change the ﬁlters in each particular layer inside the module and then to ensure that concatenations and additions from the previous layers match the correct number of ﬁlters.
We have adopted then, this assumption of taking blocks as single layers to set the values of f l .
As a result, a ﬁnal ResNet or Inception module marked with f l ﬁlters, is set to f l ﬁlters in each layer inside the module.
Uniform Template .
The most immediate distribution to evaluate is, as the original Neocognitron, an uniform distribution of ﬁlters.
Computing the number of ﬁlters in an uniform distribution is straightforward.
Adding up the ﬁlters in each layer from the base model and divide them by the number of layers give the number of ﬁlters to be set in each of them.
Formally, we compute the new number in each layer as f (cid:48)
l = F/D ∀ l ∈ { 1 , ..., D } .
In this way, changing the distribution for a VGG19 model built exclusively with sixteen convolutional layers, one ﬁnal unchangeable fully connected layer and a total number of ﬁlters of F =produces a model with f (cid:48)
l = 5504 / 16 = 344 ﬁlters in each layer.
Reverse Template .
Another straight-forward transformation for the ﬁlter distribution adopted in this paper is reversing the number of ﬁlters in every layer.
Our ﬁnal model with this template is deﬁned by the ﬁlters f (cid:48)
l = f D − l +1 .
[-Base   
-]
[-Reverse−Base   
-]
[-Negative−Quadratic   
-]
Quadratic Template .
The third distribution we evaluated is characterised by a quadratic equation f (cid:48)
l = al
+ bl + c
and consequently, has a parabolic shape with the vertex in the middle layer.
We set this layer to the minimal number of ﬁlters in the base model f min = min ( f l ) l ∈ { 1 , ..., D } so, the number of ﬁlters is described by f (cid:48)
D/ 2 = f min .
Also, we ﬁnd the maximum value in both the initial and ﬁnal convolutional layers , thus f (cid:48)
= f (cid:48) D .
To compute the new number of ﬁlters in each layer we solve the system of linear equations given by i) the restriction of the total number of ﬁlters in (cid:80) Dl =1 ( f (cid:48) l )
=
(cid:80) Dl =1 (cid:0)
al
+ bl + c (cid:1) =
F , that can be reduced to (cid:16) D 3 3 + D 2
+ D 6 (cid:17) a + (cid:16) D 2
+ D 2 (cid:17) b
+ Dc = F , ii) the equation produced by the value in the vertex f (cid:48)
D/ 2 = D 2 2 a + D 2 b
+ c = f min and iii) the equality from the maximum values which reduces to ( D 2 − 1)
a + ( D − 1) b = 0 .
Negative Quadratic Template .
The ﬁnal template is also a parabola but with the vertex in a maximum, that is, a negative quadratic curve.
The equation is the same quadratic equation that the previous template but the restrictions change.
Instead of deﬁning a value in the vertex, f (cid:48) l at the initial and ﬁnal convolutional layers are set to the minimal number of ﬁlters in the base model f (cid:48)
l = f min l ∈ { 1 , D } .
The number of ﬁlters in each layer is computed again with a system of equations speciﬁed by i) the restriction of the total number of ﬁlters as in the quadratic template, and the two points already known in the ﬁrst and last convolutional layers deﬁned by ii) a + b + c = f min and iii) D 2 a + Db + c = f min .
Once the model has been readjusted with the new number of ﬁlters per layer we use a width multiplier to test different levels of model compression to make comparable evaluations given that the change in the distributions of kernels modiﬁes the number of parameters, memory consumption and speed.
The width multiplier only reduces or increases the set of new ﬁlters f (cid:48)
l proportionally in every layer.
D ATASETS
We trained over two datasets traditionally used for convolutional network evaluation: CIFAR-10 and
CIFAR-100 (Krizhevsky et al., 2009).
Both datasets contain a train set of 50,000 images and a test set of 10,000 images with a resolution of 32x32 and three colour channels.
They were published for classiﬁcation tasks for ten and one hundred classes respectively.
Some models were tested also
C ONVOLUTIONAL N ETWORK M ODELS
The state-of-the-art networks evaluated represent some of the highest performing CNNs on the
ImageNet challenge in the previous years (Russakovsky et al., 2015).
They have been primarily tested on classiﬁcation tasks and also have demonstrated a strong ability to generalise to images outside the ImageNet dataset.
Therefore, it is expected they perform well in the CIFAR datasets.
The VGG network architecture (Simonyan & Zisserman, 2014) is recognised by its simplicity.
It is composed by sequential convolutional layers followed by max-pooling reduction layers.
The ﬁnal classiﬁcation is managed by fully-connected layer and a Softmax classiﬁer.
The main disadvantage of these networks is the size of their parameters.
In theses paper we use the version of the model with just one fully connected layer in the ﬁnal classiﬁcation section.
ResNet (He et al., 2016) succeeds on the problem of training very deep CNNs by reformulating the assumption that the network blocks are modelling a function closer to an identity mapping than to a zero mapping.
Therefore, it should be easier to ﬁnd differences with reference to an identity rather than to a zero mapping.
This assumption is carry out by adding additional references at the end of building blocks.
The Inception/GoogleNet architecture (Szegedy et al., 2015) make use of the Inception module conceived as a multi-level feature extractor allowing simultaneous extraction features of several sizes within the same module of the network.
The MobileNet network (Howard et al., 2017) is built on depthwise separable convolutions except for the ﬁrst layer which is a full convolution.
All layers are followed by a batch normalisation and
ReLU nonlinearity with the exception of the ﬁnal fully connected layer which consist of a softmax layer for classiﬁcation.
M ODELS C OMPARISON U NDER S IZE , M EMORY F OOTPRINT AND S PEED
In this section we ﬁrst investigate the effects of applying different templates to the global distribution of kernels in well known convolutional neural network models (VGG, ResNet, Inception and
MobileNet).
We compare models under the basis of size, memory and speed in two of the popular datasets for classiﬁcation tasks.
All experiments have models fed with images with the common augmentation techniques of padding, random cropping and horizontal ﬂipping.
Our experiments were run in a NVidia Titan
X Pascal 12GB GPU adjusting the batch size to 16 samples to ﬁt in GPU Memory at the maximum model scaling.
T EMPLATE EFFECT OVER THE BASELINE MODELS
We conducted a ﬁrst experiment to test our proposed templates on the selected architectures.
All convolutional models, with and without templates, were trained for 160 epochs using the same conditions: stochastic gradient descent (SGD) with a scheduled learning rate starting in 0.1 for the ﬁrst 80 epochs, 0.01 for the next 40 epochs and ﬁnally 0.001 for the remaining epochs.
The results are presented in Table 1.
It is shown that for VGG, ResNet and MobileNet, the model accuracy improves in CIFAR-10 when the templates are applied.
Inception architecture presents the highest accuracy of all base models in both datasets and templates are only able to change its accuracy in less than 1.5%.
This is surprising given the drastic modiﬁcations that the model is suffering after the change of ﬁlter distribution.
Models that share a sequential classical architecture such as VGG and MobileNet, show a better improvement when using templates.
We additionally performed an experiment with some of the models on the Tiny-Imagenet dataset (See Table 2).
While there is a very modest reduction in parameters with VGG, a very remarkable accuracy improvement is produced in MobileNet.
[Table]
[Table]
[Table]
Parameters
[Table] Millions) [Table] When evaluated under other metrics (Table 3), models are affected differently with each template and model.
The Reverse-Base, Uniform and Quadratic templates show some reductions in the number of parameters while Negative Quadratic template reduces the memory usage.
Inference Time is affected negatively for the templates.
This is an expected result as original models are designed to perform well in the GPU.
The Inception model shows an improvement in speed with a reduction of 14% over inference time respect to the base model.
It is important to notice that a reduced number of parameters does not correspond to a low consumption of memory, not even a small inference time.
Some of the causes are the difference in feature map resolution for ﬁlters in different layers, the need to keep early feature maps in memory for late layers and the restrictions for improving parallelisation in the computational graph of the model.
T EMPLATE EFFECT WITH SIMILAR RESOURCES
It can be argued that models obtained with templates make use of more resources such as memory or number of operations in the GPU (reﬂected in the low inference speed).
So, we formulated a second experiment that makes proportional changes in the models after applying the templates.
We not only apply reductions to the models but also increments in order to observe if the actual total number of ﬁlters is adequate for the task the model is performing or if the model accuracy could improve by adding more ﬁlters.
Thus, we create curves for each template applying an uniform reduction using a width multiplier with values of 1,6, 1,3, 1.0, 0.8, 0.5, 0.25, 0.1 and 0.05.
These curves of reduction allow comparison under the same amount of resources as well as compares the use of resources under the same accuracy.
The experiment also shows the level of reduction that our models can tolerate without a signiﬁcant loss in accuracy.
We add dashed lines to every plot to be used as a reference for the model with the original distribution and no reductions which is the point where both vertical and horizontal dashed lines cross.
In general, any arbitrary vertical line in the plot compares accuracy between models with amount of resources (parameters, memory or speed).
On the other side, any arbitrary horizontal line compares the resources taken for each model under each template to produce similar accuracy.
Evaluating a model performance using accuracy and parameters is by far the default approach.
We show models performances with these metrics in ﬁgure 2.
VGG and MobileNet models improve in accuracy almost with any template in CIFAR-10 and CIFAR-100.
Under reductions, their original accuracy can be reached with less than 25% of the original parameters in the two models.
ResNet shows less improvement when compared with networks with similar resources, yet templates can reduce the model further before accuracy drops.
Inception behavior considering the same resources remains similar no matter the template used.
In general, for this test, the uniform template seems to get the best parameter efﬁciency for all the models.
We are convinced that for practical implementations, comparing parameters is not a good option.
Table 3 has shown that models with a small number of parameters are not necessary related with a small memory footprint or bigger speed and more results are presented in Figure 3 in the Appendix.
We observe again that VGG and MobileNet accuracy is enhanced by templates.
More than 50% of memory consumption can be reduced in both models while producing the same accuracy.
With this metric, ResNet and Inception improve slightly in CIFAR-10 with the Negative-Quadratic template but they perform lower with the rest of templates.
We attribute the lower efﬁciency in memory to the fact that in all the templates but Negative-Quadratic, the number of ﬁlters is increased in the initial layers. At these layers the size of feature maps produced for each ﬁlter is bigger, and therefore, more memory costly.
One ﬁnal comparison also important for practical issues is inference time.
Our experiments show the patter of improvement for VGG and MobileNet and a degradation of inference time when adopting templates in ResNet and Inception (See Figure 4 in Appendix).
In particular Inception shows an improvement with the Negative-Quadratic template in CIFAR-10.
By looking results in inference time it can look unpromising to apply templates.
However we can take a different perspective, by sacriﬁcing inference speed it is possible to obtain models with a better accuracy.
This could be an unwanted decision but it is frequently taken.
It is clearly stated in the inference time between different original models.
For example by using ResNet the accuracy improves compared to the obtained by VGG in the two datasets tested, but at the cost of increasing the time for inference.
On the contrary, looking for speed enhancement MobileNet has sacriﬁced [Figure] accuracy.
In these sense, our templates are still competitive when compared to searching for a totally different model in order to improve accuracy.
C ONCLUSIONS
The most common design of convolutional neural networks when choosing the distribution of the number of ﬁlters is to start with a few and then to increase the number in deeper layers.
We challenged this design by evaluating some architectures with a varied set of distributions on the CIFAR and TinyImagenet datasets.
Our results suggest that this pyramidal distribution of ﬁlters is not necessarily the best option for obtaining the highest accuracy or even the highest parameter efﬁciency.
The method presented allows a model architect to apply a set of templates for redistributing the number of ﬁlters originally assigned to each layer in existent convolutional network models before training the models from scratch.
This redesign and the following proportional reduction can be achieved without any previous training process to select particular weights.
In essence, the application of ﬁlter redistribution templates offers an alternative and or additional approach to the iteration-intensive architecture search or model pruning.
Our experiments show that the models, with the same amount of ﬁlters but a different distribution produced by our templates, improves accuracy with up to 5.5% improvement for some model-task pairs.
But after being pruned uniformly, they can obtain the same accuracy than the original models using less resources such as number of parameters with up to 45% less parameters and a memory footprint up to 60% smaller.
Results also reveal an interesting behaviour in the evaluated models: a strong resilience to changes in ﬁlter distribution.
The variation in accuracy for all models after administering templates is less than 5% despite the modiﬁcations in the distributions, and therefore in the original design, are considerable.
This ﬁnding strengthen our belief that it is not worth exploring the whole space of ﬁlter distributions to ﬁnd the best solution at the cost of training models for a large number of iterations, as this solution possibly won’t be too distant of the baseline performance.
However, it is possible to explore just some distinct and easy to implement distributions such as those represented by our templates, that produces beneﬁts depending on the resource to be optimised.
Our work overall offers an additional tool to model designers, both automated and manual, and we hope it motivates further work for iteration-less methods and help gather data to build understanding of the design process for model-task pairs.
APPENDIX 5. E XPERIMENTS C OMPARING M ODELS M EMORY F OOTPRINT AND S PEED
[Figure]
[Figure]
5. F ILTER D ISTRIBUTION FOR E XISTING C ONVOLUTIONAL D EEP N ETWORK M ODELS
During the design of a convolutional deep network architecture the number of ﬁlters, required at every layer, is selected.
Neocognitron model keeps the same number of ﬁlters across all layers in the model.
A distribution of ﬁlter widely used forms a by-pyramidal architecture.
The number of ﬁlters across the different layers is usually increased as the size of the feature maps decrease.
We present in Figure 5 the distribution for the models VGG, ResNet, Inception and MobileNet which were tested in our experiments.
[Figure]