B INARY F LEX : O N - THE -F LY K ERNEL G ENERATION IN B INARY C ONVOLUTIONAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work we present BinaryFlex , a neural network architecture that learns weighting coefﬁcients of predeﬁned orthogonal binary basis instead of the conventional approach of learning directly the convolutional ﬁlters.
We have demonstrated the feasibility of our approach for complex computer vision datasets such as ImageNet.
Our architecture trained on ImageNet is able to achieve top-5 accuracy of 65.9% while being around 2x smaller than binary networks capable of achieving similar accuracy levels.
By using deterministic basis, that can be generated on-the-ﬂy very efﬁciently, our architecture offers a great deal of ﬂexibility in memory footprint when deploying in constrained microcontroller devices.
I NTRODUCTION
Since the success of AlexNet (Krizhevsky et al., 2012), convolutional neural networks (CNN) have become the preferred option for computer vision related tasks.
While traditionally the research community has been ﬁxated on goals such as model generalization and accuracy in detriment of model size, recently, several approaches (Iandola et al., 2016; Courbariaux & Bengio, 2016; Rastegari et al., 2016) aim to reduce the model’s on-device memory footprint while maintaining high levels of accuracy.
Binary networks and compression techniques have become a popular option to reduced model size.
In addition, binary operations offer ∼ 2x speedup (Rastegari et al., 2016) in convolutional layers since they can be replaced with bitwise operations, enabling this networks to run on CPUs.
It is also a common strategy to employ smaller convolutional ﬁlter sizes (e.g. 5x5, 3x3 or even 1x1) aiming to reduce the model’s memory footprint.
Regardless of the ﬁlter dimensions, they represent a sizeable percentage of the total model’s size.
We present a ﬂexible architecture that facilitates the deployment of these networks on resource-constrained embedded devices such as ARM Cortex-M processors.
Table 1 shows a range of Cortex-M based microcontrollers running at different frequencies and with different on-chip RAM-Flash conﬁgurations.
[Table] We offer the following contributions: (1) a new architecture that generates the convolution ﬁlters by using a weighted combination of orthogonal binary basis that can be generated on-the-ﬂy very efﬁciently and offers competitive results on ImageNet.
This approach translates into not having to store the ﬁlters and therefore reduce the model’s memory footprint.
The number of parameters needed to be updated during training is greatly reduced since we only need to update the weights and not the entire ﬁlter, leading to a faster training stage.
(3) We present an scenario where the on-the-ﬂy nature of BinaryFlex is beneﬁtial when the model does not ﬁt in RAM.
R ELATED W ORK
Our work is closely related to the following areas of research.
CNN for Computer Vision.
Deep CNNs have been adopted for various computer vision tasks and are widely used for commercial applications.
While CNNs have achieved state-of-the-art accuracies, a major drawback is their large memory footprint caused by their high number of parameters.
AlexNet (Krizhevsky et al., 2012) for instance, uses 60 million parameters.
Denil et al.
(2013) have shown signiﬁcant redundancy in the parameterization of CNNs, suggesting that training can be done by learning only a small number of weights without any drop in accuracy.
Such overparameterization presents issues in testing speed and model storage, therefore a lot of recent research efforts have been devoted to optimizing these architectures so that they are faster and smaller.
Network Optimization.
There have been many attempts in reducing model size through use of low-precision model design.
Linear quantization is one approach that has been rigorously studied:
One direction in quantization involves taking a pre-trained model and normalizing its weights to a certain range.
This is done in Vanhoucke et al.
(2011) which uses an 8 bits linear quantization to store activations and weights.
Another direction is to train the model with low-precision arithmetic as in Courbariaux et al.
(2014) where experiments found that very low precision could be sufﬁcient for training and running models.
In addition to compressing weights in neural networks, researchers have also been exploring more light-weight architectures.
SqueezeNet (Iandola et al., 2016) uses 1 × 1 ﬁlters instead of 3 × 3 , reducing the model to 50 × smaller than AlexNet while maintaining the same accuracy level.
Binary Networks.
In binary networks, parameters are represented with only one bit, reducing the model size by 32 × .
Expectation BackPropagation (EBP) (Soudry et al., 2014) proposes a variational Bayes method for training deterministic Multilayer Neural Networks, using binary weights and activations.
This and a similar approach (Esser et al., 2015) give great speed and energy efﬁciency improvements.
However, the improvement is still limited as binarised parameters were only used for inference.
Many other proposed binary networks suffer from the problem of not having enough representational power for complex computer vision tasks, e.g. BNN (Courbariaux & Bengio, 2016), DeepIoT (Yao et al., 2017), eBNN (McDanel et al., 2017) are unable to support the complex ImageNet dataset.
In this paper, we also considered the few binary networks that are able to handle the ImageNet dataset, i.e. BinaryConnect (Courbariaux et al., 2015) and Binary-WeightNetworks (BWN) (Rastegari et al., 2016), achieving the later AlexNet level accuracy on ImageNet.
BinaryConnect extends the probabilistic concept of EBP, it trains a DNN with binary weights during forward and backward propagation, done by putting a threshold for real-valued weights.
B INARY F LEX 3. C ONVOLUTION USING B INARY O RTHOGONAL B ASIS
Training CNNs for image classiﬁcation fundamentally consists in learning the set of ﬁlters that hierarchically transform an input image to an image label.
The ﬁlters are usually learn via backpropagation (LeCun et al., 1989). BinaryFlex differs from standard CNNs in the sense that ﬁlters are not directly learnt. Instead, they are generated from a weighted linear combination of deterministic orthogonal binary basis.
Our architecture learns the per-basis weighting coefﬁcients and uses them to generate the ﬁlters at inference time. We introduce in our architecture a basis generator that outputs orthogonal variable spreading factor (OVSF) codes that are later combined to generate the convolutional ﬁlters. This generator has been widely used in W-CDMA based 3G mobile cellular systems 1 to provide multi-user access. Its simplicity, makes it suitable for efﬁcient real-time implementations on power-constrained devices.
The ﬁlter generation process using the OVSF basis and the learned weights is illustrated in Figure 1. The OVFS generator outputs a set of binary sequences of length L , where L = 2 l , l ∈ N that are orthogonal to each other; they get reshaped to match the shape of the desired convolutional ﬁlter; and ﬁnally linearly combined using the weights obtained during training. The quality of our ﬁlter
[Table] [Table]
˜ b 1 i , ˜ b 2 i , ˜ b
3 i ,...
{+˜ b Ni ] w i = [ w 1 i , w 2 i , w 
 i ,... w Ni ] b ki = [1, − 1,2,1,... − 2] -0.4 -1.3 0.9 0.2 1.7 -1.3 -1.1 1. b i =+}
[˜ b 1 i , ˜ b
2 i , ˜ b
3 i ,...
{+˜ b Ni ] w i = [ w 1 i , w 2 i , w 
 i ,... w Ni ] b ki = [1, − 1,2,1,... − 2] f i [Figure]+}
approximation can be measured as:
[Equation]
Where ρ is the total number of basis to use in order to approximate ﬁlter f k ,
b ik is a base and w i j ∈ R its associated weight.
(cid:15) is the difference between the the approximated, f (cid:48)
k and the real ﬁlter, f k .
Intuitively, (cid:15) → 0 as we increase the number of binary basis, ρ .
In the following sections we will describe how these basis are generated and why it is beneﬁcial for our ﬂexible model with on-the-ﬂy ﬁlter generation.
3. G ENERATION OF B INARY O RTHOGONAL B ASIS
The design space to generate high-dimensional binary and orthogonal arrays is very broad.
Our target platforms constrains this space to those codes that can be generated very eﬁciently while been applicable to complex image classiﬁcation tasks. We ﬁnd OVSF codes to be an optimal solution given those constrains.
C 1,1 = (1) C 2,1 =
C 2,2 =
C 4,3 = C 4,4 =
C 4,1 = C 4,2 = (1,1,-1,-1)
C 8,1 =
C 8,2 =
(1,1,1,1,-1,-1,-1,-1) C 8,3 =
C 8,3 =
C 8,5 =
C 8,6 =
(1,-1,1,-1,-1,1,-1,1) C 8,7 = (1,-1,-1,1,1,-1,-1,1)
C 8,8 =
SF= SF=
SF= SF=
Figure 2 shows the procedure for generating OVSF codes of different lengths as a recursive process in a binary tree (Adachi et al., 1997).
Assuming code C with length N on a branch, the two codes on branches leading out of C are derived by arranging C in ( C, C ) and ( C, − C ) .
It is clear from this arrangement that C can only be deﬁned for N s that are powers of 2.
Efﬁcient hardware implementations of OVSF code generators have been crucial for designing power-efﬁcient cellular transceivers and given the maturity of W-CDMA standard is a well-studied problem in the wireless community (Andreev et al., 2003;
Kim et al., 2009; Rintakoski et al., 2004;
Purohit et al., 2013).
Availability of such designs makes OVSF codes a suitable choice for efﬁcient on-device ﬁlter generation where memory for storing the model comes at a premium.
3. T RADING C OMPUTE FOR M EMORY : O N - THE - FLY B ASIS G ENERATION
In scenarios with very memory-limited platforms, a network making use of on-the-ﬂy basis generation would only be required to store the basis coefﬁcients (the weights) and generate the ﬁlters on-the-ﬂy .
This ﬂexibility is possible since the generation of binary basis is deterministic, repeatable and can be achieved using simple hardware logic on silicon.
In addition, this process can be done with high granularity.
For example, the application can maintain just a subset of basis in memory or, given that OVSF codes are conventionally generated recursively from the code tree described in 3.2, the model can store only parts of the basis and generate the remaining parts when required.
When memory footprint is not a concern the application can cache kernel coefﬁcients and make
BinaryFlex behave exactly similar to conventional CNNs while a middle-ground approach would be to only store the basis in memory and combine them to recover the kernels.
The decision of which approach to take can either be made at design time based on the speciﬁcation and requirements of the target device but a smart implementation can mix-and-match these approaches at run-time depending on the amount of available resources at inference time.
The deterministic nature of basis generation also gives hardware implementations ﬂexibility in terms of the type of memory that can be used.
For on-device learning on very resource-stringent devices this approach allows the implementation to move a big portion of network parameters into ROM instead of RAM resulting in much smaller silicon area and power consumption.
3. B INARY F LEX A RCHITECTURE O VERVIEW
Our BinaryFlex architecture, Figure 3, is inspired by SqueezeNet(Iandola et al., 2016) and
ResNet(He et al., 2015).
Macroarchitecturally, it resembles SqueezeNet in the sense that after the initial convolution and max-pooling layers, the rest of the pipeline is comprised of 3 cascades of modules or blocks separated by max-pooling layers.
The ﬁnal elements are a convolutional and pooling layer followed by a softmax stage.
Microarchitecture wise, Binaryﬂex resembles ResNet in the sense the ﬂow of data in the building blocks follow the same pattern as in ResNet’s bottleneck architecture.
In BinaryFlex we name this building block FlexModule .
[Figure]
[Figure]
BinaryFlex is comprised of eight FlexModules and no fully connected layer aiming to reduce the number of parameters in our model.
We adopted SqueezeNet’s strategy of late down-sampling that in summary consists in using stride s = 1 in each of the convolution layers in the FlexModules and perform downsampling after each cascade of those modules (i.e. after blocks FlexModule3 , FlexModule6 and FlexModule8 ).
This microarchitecture enables the convolutional layers to have large activation maps.
Delaying the downsampling in CNNs leads to better accuracy in certain architectures (He & Sun, 2014).
In isolation, a FlexModule looks like Figure 4.
Each convolution layer uses a different set of ﬁlters from the ﬁlter bank that has been generated as illustrated in Figure 1.
In our BinaryFlex implementation, a per- FlexModule OVSF basis-generator is not needed since the basis are deterministic and we only have to decide how many basis to combine in order to create each ﬁlter.
Further explanation on how the basis are generated and how they are efﬁciently used during inference can be found in 3.2 and 3.3, respectively. [Table]
E VALUATION
In this section, we evaluate BinaryFlex’s performance when compared to binary networks on image classiﬁcation tasks when deploying in compute and memory constrained platforms.
We ﬁrst present results on different vision problems to demonstrate the generalizability of BinaryFlex.
Then we present a detailed study of BinaryFlex’s performance on ImageNet, followed by a case study of BinaryFlex’s ﬂexibility in memory footprint.
The main ﬁndings are:
• Binaryﬂex offers the best accuracy to size ratio (ASR) previously seen.
One conﬁguration is able to support ImageNet at a reasonable accuracy of 56.5% while being just 1.6 MB.
• BinaryFlex is comparable in accuracy under common image datasets.
In particular, under ImageNet BinaryFlex accuracies dominate the low end but only lags the high end by 17%.
• Our work demonstrates the feasibility of using a combination of deterministic binary basis as convolutional ﬁlters, achieving fairly high accuracy models in image classiﬁcation tasks.
• BinaryFlex can be easily adjusted to meet the computation and memory requirements very constrained platform (see Table 1).
The breadth of the inference operations vs memory size space spanned by our architecture is much wider than that offered by exiting models.
4. E XPERIMENTAL S ETUP
To train BinaryFlex on ImageNet, we used learning rate 0.1, decade factor 0.1 per 30 epochs and batch size 64. For the case of CIFAR-10 and MNIST we train each BinaryFlex conﬁguration for a total of 90 epochs with the same decay rate as before and batch size of 128.
No pre-processing or data augmentation is used.
Our architecture is implemented in TensorFlow. 4. R ESULTS On ImageNet, BinaryFlex is compared to BinaryConnect (Courbariaux et al., 2015) and
BNN(Courbariaux & Bengio, 2016); for MNIST and CIFAR-10, BinaryFlex is compared to BinaryConnect and BWN (Rastegari et al., 2016).
Several BinaryFlex conﬁgurations resulting in different model sizes are trained on ImageNet, CIFAR-10 and MNIST.
The fundamental difference between each conﬁguration is the ratios of binary basis used to generate the convolutional ﬁlters. Intuitively, combining more basis results in ﬁlters that can better capture complex image features. The results in Table 3 show the robustness of BinaryFlex even when using a very small portion of binary basis, resulting in smaller models.
When comparing the performance of several BinarFlex conﬁgurations against other models for the same dataset, we use BinaryFlex appended by the model size in MB, e.g. BinaryFlex-3.6, to refer to a particular BinaryFlex conﬁguration.
BinaryFlex-10. uses almost all the OVSF to generate the ﬁlters, where as BinaryFlex-3.4 uses around 30% and BinaryFlex-0.7 close to 5%. As shown in Table 2, ratios values differ between FlexModules. Our ﬁndings suggest that lower OVSF ratios in deeper layers provide higher model size reductions at the cost of little to none accuracy loss when shallower layer maintain high OVSF ratios.
ImageNet.
In Table 5, we see BinaryFlex’s ability to offer good ASR which are otherwise not possible.
We proof that, OVSF basis can provide comparable results to those achieved by BWN at a similar model size even when BinaryFlex has not been ﬁne tuned during training.
BinaryFlex can be made very small, as BinaryFlex-1.6 gives a 4 . 5 × reduction in model size but only gives up 4% in accuracy when comparing to BinaryConnect.
This small model size of 1.6MB is of critical importance for hardware implementations as many processors (e.g. ARM Cortex processor M7) only allows a maximum combined ROM/RAM space of 2MB.
By ﬁtting into ROM/RAM of embedded devices without paging to SD card, BinaryFlex offers great computational advantage, as the timescales of data movement from SD card to RAM can be 1250 × longer than that from ROM to RAM (Gregg, 2013).
Other datasets.
Table 4 shows that BinaryFlex gives competitive results in a general setting, despite not having its architectures optimized for each task.
On MNIST, BinaryFlex gives the best accuracy amongst baselines.
In addition, with the aims to further reduce memory footprint, we applied post-training 8-bit quantisation to the BinaryFlex models shown in Table 3 for MNIST and CIFAR-10 datasets resulting in models ∼ 75% smaller. Some results are shown in Table 4. [Table]
[Table] [Table] [Table] OF B INARY F LEX UNDER I MAGE N ET .
In this section we elaborate on the performance of BinaryFlex model on ImageNet, as our main goal is to improve the performance of binary architecture on large-scale dataset.
There two things we want to emphasize, the ﬂexibility of different model sizes and good memory trade-off.
First, by adjusting the number of basis used for each ﬁlter generation, we can easily ﬁnd a model that meets the memory constraint of any device which the model is going to be deployed on.
Second, it is very important to take into account both the accuracy and model size when training models, since we do not want to compromise the accuracy too much while reducing the model size.
And so, we introduce the concept of accuracy to size ratio (ASR) to evaluate the trade-off between accuracies and model sizes.
As shown in Figure 5, BinaryFlex not only has a wide range of model sizes, from 10MB to 1MB but also maintains high ASR, ranging from 10 to 35(%/MB), when the model size gets reduced.
4. C ASE S TUDY : F LEXIBILITY OF B INARY F LEX
In this case study, we further emphasize the beneﬁt attributed to other on-the-ﬂy generation.
The overhead of on-the-ﬂy generation is extremely low.
To generate basis with dimension n , it only requires 2 × (2 log 2 n − 1) operations.
And with dedicated hardware, the whole generation can be [Figure] done in just one operation cycle.
As a result [Figure] on-the-ﬂy generation gives rise to substantial memory footprint reduction with just little overhead.
In Figure [Figure] we analyzed the model size and number of Mult-Adds resulting from different number of basis used for ﬁlter generation and percentage of basis that are generated during runtime.
We examined three different generation conﬁgurations: 1) 100 % generation – All the binary basis are generated during runtime.
2) 50 % generation – Half of the binary basis are generated during during runtime and other half are stored in the memory and retrieved during ﬁlter generation.
No generation – All the basis are stored in the memory.
The on-the-ﬂy generation results in 5X to 10X memory with negligible compute overhead.
The gain is even more pronounced for models with large size.
This suggests that in scenarios where we have deploy large model on a resource-limited device, we can apply our method to reduce the model size without compromising the accuracy.
In other words, we can determine the fraction of basis to be generated during runtime based on the memory and compute resource on the device
[Figure]
D ISCUSSION
Here we further describe the main contributions of BinaryFlex and why we believe they are relevant.
Our network architecture is ﬂexible.
We could deﬁne this as a 2-dimensional property in the sense that it can be split into two distinctive components: (1) reduction of the model size by enabling on-the-ﬂy generation of basis and (2) reduction in the number of forward operations depending on the number of basis we use to generate each ﬁlter.
The ﬁrst component could be deﬁned as onthe-ﬂy ﬂexibility, F OTF ( T RAM , model MB ) , where subscript OTF stands for on-the-ﬂy , T RAM is the available RAM in the target platform and model MB is the size of the BinaryFlex model.
This property was ﬁrst described in 3.3.
Figure 7 provides an intuition on how BinaryFlex could be deployed into platforms with limited on-chip memory.
The model accuracy is not being affected by F OTF .
[Figure] [Figure] We deﬁne the second ﬂexibility component as operations ﬂexibility, F OP ( T COMP , A (cid:15) ) , where T COMP is the compute capability of the target platform T and A (cid:15) is the maximum allowed accuracy error allowed in application A (i.e. image classiﬁcation).
Intuitively, the F OP ﬂexibility component enables BinaryFlex to run on compute-constrained devices without having to modify the network architecture, at the cost of reducing the model accuracy.
We believe that, unlike other methods that limit their efforts to reduce the complexity of the operations (e.g. by using binary operations), this characteristic of BinaryFlex, F = { F OTF , F OP } , would lead to a new type of CNNs speciﬁcally designed to perform well in a broad range of constrained devices.
In this work we have focused on reducing already small CNNs, in the order of a few MBs, to sub-MB models by avoiding to store the ﬁlters of each convolutional layer.
This characteristic of
BinaryFlex is equally applicable to the other side of spectrum, big models like AlexNet(240MB) or
VGG(540MB).
This would enable training much larger models while keeping their memory footprint at an order of magnitude less.
Figure 6 visually represents this idea.
So far, we have discussed why BinaryFlex’s property of on-the-ﬂy basis generation leads to a reduction in the model size.
In addition to this, our approach dramatically reduces the number of parameters that need to be updated during each training iteration.
Figure 8 compares the number of parameters learned in each architecture for the ImageNet classiﬁcation task.
Because BinaryFlex only needs to store the weights for each basis and not the entire ﬁlters, the amount of parameters is reduced by an order of magnitude.
Implicity to this property is that, since the size of a BinaryFlex model depends on the number of basis used to generate the ﬁlters and not on the dimensions of this ﬁlters, FlexModules in our architecture enables the usage of arbitrarily big ﬁlters without translating this into an increase in model memory footprint.
We ﬁnd this novel approach to be the to-go option to enable training with more and bigger ﬁlters that better capture relevant image features.
C ONCLUSION
We have introduced BinaryFlex, a small-memory, ﬂexible and accurate binary network which learns through a predeﬁned set of orthogonal basis.
We have shown that BinaryFlex can be trained on
ImageNet, CIFAR-10 and MNIST with good balance of accuracy and model size.
On ImageNet, BinaryFlex models are able to reduce the size by 4.5 × without great compromise in accuracy.
Further, we have introduced techniques such as on-the-ﬂy basis generation, allowing powerful management of inference-time overhead on low-resource devices.