Paper: OneBNet: Binarized Neural Networks using Decomposed 1-D Binarized Convolutions on Edge Device (ICLR-2024)

Reviewer comment (Presentation):
The reference format in this paper can be improved, e.g.  
- "Reactnet: Towards precise binary neural network with generalized activation functions" is an ECCV paper,  
- "Recu: Reviving the dead weights in binary neural networks" is an ICCV paper.

Author response:
* **Question:**
The reference format in this paper can be improved.
* **Answer:** 
In agreement with your concerns, the references of officially published papers have been changed in **Reference** section of the revised version as:  
***"Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In European Conference on Computer Vision (ECCV), pp. 143-159, 2020."  
"Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5198-5208, 2021."***  
Besides, the references of several works have been revised if they were officially published.

Reviewer comment (Novelty):
It is not new to replaces NxN 2D binarized convolution by Nx1 and 1xN 1D binarized convolutions. For example, SqueezeNext (CVPR’18 workshop) decompose the KxK convolutions into two separable convolutions of size 1xK and Kx1. Although this paper focuses on binary neural network, the novelty of using such strategy for BNNs is still quite unclear.

Author response:
Thanks for your novel comments. In agreements with your concerns, 1-D convolutions were proposed in SqueeNext and Inceptionv3. I agree that the naive application of 1-D convolutions to BCNNs cannot be a main contribution, as shown in **Introduction** section of the previous version. 
However, we provide several unique things in this paper as: the downsampling layer using 1-D binarized convolutions was proposed, so that the proposed model can have significant internal differences based on backboned ResNet, compared with the model naively using 1-D binarized convolution. Besides, the effects of adjusting the activation distribution and non-linear activation function can be doubled by applying specific layers for BCNNs to the decomposed 1-D convolutions in the proposed OneBNet. In order to avoid the confusion of the main contribution, the explanation has been rewritten in **Abstract** as:  
***"It was known that the decomposed 1-D convolutions can replace the spatial 2-D convolutions in several convolutional neural networks (CNNs) for computer vision. However, the proper usage of 1-D convolutions was not shown in the field of binarized CNNs. This paper shows This paper proposes a new structure called OneBNet to maximize the effects of 1-D binarized convolutions, thus producing  excellent performance on CPU-based edge devices. To double the effects of adjusting the activation distribution and non-linear activation function, specific layers for BCNNs are doubled by applying them to  both $n \times 1$ row-wise and $1 \times n$ column-wise 1-D binarized convolutions. The proposed 1-D downsampling can perform information compression gradually through two 1-D convolutions, which can contribute tremendously to the performance improvement in binarized convolutional neural networks (BCNNs) in our analysis."***

Reviewer comment (Experiments):
Could the proposed method also suitable for object detection tasks using binary neural network?

Author response:
Thanks for your good recommendation for model applications. At this time, we did not verify the proposed OneBNet to object detection in the experiments. However, because similar works using BCNNs can produce acceptable performance in object detection, we expect that the proposed model could be applicable. For example, BiDet: An Efficient Binarized Object Detector (CVPR 2020) and Recurrent Bilinear Optimization for Binary Neural Networks (RBONN in ECCV 2022) applied BCNNs such as XNOR-Net, Bi-RealNet, and ReActNet in the object detection. Notably, when RBONN was applied to the image classification, its Top-1 performance (66.7\%) on ImageNet dataset cannot reach our results. 
Although the experimental results for the object dection were not listed,
the successful application with the counterparts and the outstanding image classification of the proposed model make sure that the proposed OneBNet could be useful in object detection.

Reviewer comment (Experiments):
Several recent methods reported in Table 2 are not included in Table 3.

Author response:
Table 3 listed the comparison with several BCNNs if a model has difference in layer structure and **the specific layer supported by Larq compute engine**.
When the specific layer was not supported by Larq, we cannot evaluate the latency using Larq. For example, XNOR-NET++ has row-wise and column-wise scaling parameters in binarized convolutions, which cannot be supported in Larq and Tensorflow Lite. In RB-Net, the reshaped output could not be supported by Larq. Therefore, we have listed models that can be developed by Larq in Table 3.

Reviewer comment (Novelty):
The contribution and innovation of this paper are insufficient. The decomposition of 2D convolutions into two 1D convolutions used in this article is not a new idea, but a widely studied method. Although its combination with Binarized neural networks may make it more effective, it is easy to consider or attempt.

Author response:
In agreements with your concerns, 1-D convolutions were proposed in previous works such as SqueeNext and Inceptionv3. I agreed that the naive application of 1-D convolutions to BCNNs cannot be a main contribution, as shown in Introduction section.  
On the other hand, our works proposed the downsampling layer using 1-D binarized convolutions, so that the proposed model can have significant internal changes. Besides, the effects of adjusting the activation distribution and non-linear activation function can be doubled by applying specific layers for BCNNs to the decomposed 1-D convolutions. To focus on the main contribution, the explanation has been rewritten in **Abstract** as:  
***"It was known that the decomposed 1-D convolutions can replace the spatial 2-D convolutions in several convolutional neural networks (CNNs) for computer vision. However, the proper usage of 1-D convolutions was not shown in the field of binarized CNNs. This paper proposes a new structure called *OneBNet* to maximize the effects of 1-D binarized convolutions, thus producing excellent performance on CPU-based edge devices. To double the effects of adjusting the activation distribution and non-linear activation function, specific layers for BCNNs are doubled by applying them to both $n \times 1$ row-wise and $1 \times n$ column-wise 1-D binarized convolutions. The proposed 1-D downsampling can perform information compression gradually through two 1-D convolutions, which can contribute tremendously to the performance improvement in binarized convolutional neural networks (BCNNs) in our analysis.''***

Reviewer comment (Experiments):
The generalizability of this method has not been verified. The author's experiment only trained and tested the smaller ResNet model, and the dataset only included CIFAR10 and ImageNet.

Author response:
Our evaluations were based on baselined ResNet18 and ResNet20. Target datasets included FashionMNIST, CIFAR10, and ImageNet. In agreement with your concerns, we adopted the multiplier $m$ for the number of channels in Appendix A.3 of the revised version, . Both the number of input and output channels can be scaled by multiplying with $m$. New Table 7 shows the summary of Top-1 accuracies by varying multiplier on CIFAR10 dataset.

Reviewer comment (Theory):
The basic theory of the method in this paper is not sound enough. Binary quantization and 2D convolutional decomposition are methods that sacrifice accuracy for less computational complexity. Why can a combination of the two achieve better accuracy? Substantive improvements may come from element wise calculations and learnable bias in more layers after decomposition, but this is not without cost, as FLOPs cannot accurately reflect the additional hardware overhead this brings. The explanation of Figure 3 also lacks quantitative data support.

Author response:
Because the newly added adjustments and functions are doubled, we concluded that the doubled effects can mitigate the degradation. I agree that good performance of the proposed method required additional computational costs. Therefore, we want to show the trade-off between the additional costs and enhanced performance to provide better model structure, in Figure 5 and Table 1. 
We agree with you that OPs in Figure 5 cannot be accurate to estimate the latencies. Additionally, Table 1 compares the latencies on Raspberry Pi when1-D binarized convolutions were adopted, so that the guideline for developing structures were concluded in terms of accuracies and latencies.

Reviewer comment (Experiments):
This method lacks a determination method for parameter selection. The paper mentions that not all convolutional layers of blocks are suitable for such transformations. In Table 1, several selection combinations are attempted, and the best ones are selected for subsequent comparison. This will bring difficulties to practical applications. If the model structure is different and there are more blocks with different channel numbers, it will not be suitable for such selection.

Author response:
Although it is hard to extract any quantitative metric, we think that the below guideline can introduce the priority to apply the proposed structure. In the second paragraph of **Experimental Results and Analysis** section, we concluded the meaning of data as:  
***"Table 1 shows that the effectiveness of 1-D binarized convolutions in downsampling, where deep convolutional layers with many channels and small activation map can have significant benefits for enhancing performance."***  
The above conclusion gave the guideline to determine which layer adopted 1-D binarized convolutions considering latencies and Top-1 accuracies.

Reviewer comment (Writing):
P3: section 3.1 The section is very difficult to follow, please rewrite this.

Author response:
According to your recommendation, **Motivations** section has been modified.  
Notably, the second paragraph was rewritten as:  
 **"*To double the effects of the adjustment of activation distribution, we apply the adjustment of activation distribution to the decomposed 1-D binarized convolutions. Whereas the conventional 2-D convolutions use 3 × 3 kernels, the decomposed convolutions use 3 × 1 and 1 × 3 kernels. The adjustment of activation distribution is applied to each 1-D convolution. Although the decomposition had the same receptive field with 2-D convolution (Szegedy et al., 2016), it requires additional element-wise operations for the adjustments and activation functions. This paper explains the structure of 1-D convolutional layer. Then, it analyzes the model structures in terms of latency and accuracy and shows the idea to deploy 1-D binarized convolutions on the above explained pros and cons*."**

Reviewer comment (Theory):
P4: "However, it shrinks the receptive fields", I am not sure whether this is true, please verify.

Author response:
According to Inceptionv3, when $1 \times n$ and $n \times 1$ convolutions are used, the receptive fields are the same. The expression has been modified as:  
***"Although the decomposition had the same receptive field with 2-D convolution (Szegedy et al., 2016), it shrinks the receptive fields and requires additional element-wise operations for the adjustment."***

Reviewer comment (Presentation):
Fig. 2. It is not clear how the results from 1D convolution are summed together with output of the 1x1 FP32 convolution, given they are of different shape.

Author response:
The numbers of channels and size of the feature map after 1x1 FP32 and binarized convolutions have been denoted in Fig.2.

Reviewer comment (Presentation):
Table 1 is very confusing, please clear it up.

Author response:
To avoid confusion in Table 1, the inference latencies of all structures have been listed in the revised version.

Reviewer comment (Experiments):
I suggest adding a model performance comparison with a similar OP level.

Author response:
To show the comparison with BCNN models having similar OPs, counterparts listed in Table 3 were compared in the previous manuscript because they have the same first FP32 convolutional and last fully connected layers on the same baseline model. 
Instead, I think that your recommendation means the comparison with mobile-friendly FP32 models such as MobileNet or ShuffleNet. Theoretically, TypeI had about **$170 \times 10^6$** OPs. On the other hand, MobileNetv1 and MobileNetv2 has about **500MFLOPs and 300MFLOPs** with 70.6 and 72.0 Top-1 accuracies. The inference latency on a single thread of MobileNetv2 was **117ms** using Larq. Considering the results, we can conclude that the proposed OneBNet can consume small computational resources, although accuracies were degraded. 
The performance comparison with other mobile-friendly models requires huge space because there have been too many researches. Besides, additional model optimization related to number of channels and feature size should be done. Thanks for recommendation.