Paper: Training Binary Neural Networks in a Binary Weight Space (ICLR-2024)

Reviewer comment (Experiments):
While I like the overall method of the submission, the experimental section leaves a lot of questions as it is not at the scale of most well-received computer vision or even BNN evaluations.

Author response:
We thank the reviewer for suggesting an experimental design to better evaluate our method. We have answered each of the reviewer's questions below.

> the experimental section leaves a lot of questions as it is not at the scale of most well-received computer vision or even BNN evaluations.

> Particularly, the error rates on CIFAR10 and Tiny-ImageNet are very high, regardless of whether STE training or the proposed EMP mask training is used. Comparing such underfitted (in my opinion) models does not paint a clear picture.

As the reviewer mentioned, the results for CIFAR-10 and Tiny-ImageNet in Table 3 show underfitting and this is mainly due to the limited expressive power of the 4-layer MLP. To adequately evaluate the training performance of our method (and STE) on the middle-sized datasets, we conducted additional experiments using AdamBNN (MobileNetV1 architecture) and ReActNet (ResNet-18 architecture), and have reported the detail in the global response. In summary of the global response, our method achieved a (top-1) accuracy of 91.94% on CIFAR-10 and 74.64% on CIFAR-100 (we utilized instead of Tiny-ImageNet), which were comparable to STE, demonstrating that our binary-space approach performs sufficiently on the middle-sized datasets.

Reviewer comment (Experiments):
Particularly, the error rates on CIFAR10 and Tiny-ImageNet are very high, regardless of whether STE training or the proposed EMP mask training is used. Comparing such underfitted (in my opinion) models does not paint a clear picture.

Author response:
We thank the reviewer for suggesting an experimental design to better evaluate our method. We have answered each of the reviewer's questions below.

> the experimental section leaves a lot of questions as it is not at the scale of most well-received computer vision or even BNN evaluations.

> Particularly, the error rates on CIFAR10 and Tiny-ImageNet are very high, regardless of whether STE training or the proposed EMP mask training is used. Comparing such underfitted (in my opinion) models does not paint a clear picture.

As the reviewer mentioned, the results for CIFAR-10 and Tiny-ImageNet in Table 3 show underfitting and this is mainly due to the limited expressive power of the 4-layer MLP. To adequately evaluate the training performance of our method (and STE) on the middle-sized datasets, we conducted additional experiments using AdamBNN (MobileNetV1 architecture) and ReActNet (ResNet-18 architecture), and have reported the detail in the global response. In summary of the global response, our method achieved a (top-1) accuracy of 91.94% on CIFAR-10 and 74.64% on CIFAR-100 (we utilized instead of Tiny-ImageNet), which were comparable to STE, demonstrating that our binary-space approach performs sufficiently on the middle-sized datasets.

Reviewer comment (Experiments):
The experimental setup is limited to fully-connected layers. While these layers are certainly well utilized, most modern DNNs and BNNs employ either convolutional or transformer layers, which questions the practical applicability of the method.

Author response:
> The experimental setup is limited to fully-connected layers. While these layers are certainly well utilized, most modern DNNs and BNNs employ either convolutional or transformer layers, which questions the practical applicability of the method.

> What obstacles do the authors forsee in applying the proposed method to BNNs using convolution layers?

We agree with the reviewer that many modern BNNs are CNN-based, and the training performance on these architectures is important for the realistic evaluation of training algorithms. Therefore, as described above, we evaluated the binary-space training using the popular ResNet-18 and memory-efficient MobileNetV1, and have reported the results in the global response.
In the original manuscript, the evaluations of the proposed method were limited to MLP, not because our method is unable to train CNN-based BNNs, but to provide a pure comparison of real- and binary-space training in the most basic structure. In fact, as the additional experimental results (Ex.2 in global response) show, our method has been experimentally demonstrated to achieve comparable performance to STE in training convolutional layers.

Reviewer comment (Theory):
The analogy from Figure 1(a) to 1(b) is more like binarizing the gradients in a real-valued space. It is not obvious to the reader that $w_t^{*}$ is the ideal target weights in the binary space since the loss landscape may have been changed a lot after binarization. It needs more discussion or justification.

Author response:
Let us confirm your question. Do you mean, "In Figure 1 (b), it appears that w is being updated in the real-valued space"? This confusion might have stemmed from the placement of omega in Figure 1(a) and w in Figure 1(b) at identical positions; therefore, we have updated the figure in the revised paper to alter the positioning of the weights.

> It is not obvious to the reader that wt\* is the ideal target weights in the binary space since the loss landscape may have been changed a lot after binarization. It needs more discussion or justification.

The target weight merely represents the direction, toward which a point that reduces the loss is found. Here, we use the word 'merely', since such a point may not be necessarily located in the binary coordinates and w\* might increase the loss. We added a footnote that describes the word choice of 'target weight' so that readers won't experience the same confusion you had.

In addition, previous studies like XNOR-Net and AdamBNN have shown that reflecting changes in the loss landscape of BNNs in the optimizer is crucial for enhancing BNN training performance. Our study mainly focuses on emulating the real-space SGD for BNNs, and the loss landscape problem is not directly addressed; however, the combining these methods with our binary-space training is not difficult. For instance, by using the gradient scaling of the XNOR-Net in our binary-space trainining, we can partially alleviate the change in the loss landscape, and in fact, we used this combination in Ex. 1 in the global response.

Reviewer comment (Theory):
It seems that the proposed update rule cannot be applied to propagating the gradients to 1-bit activations, which means STE is still needed when training a BNN.

Author response:
It seems that the proposed update rule cannot be applied to propagating the gradients to 1-bit activations, which means STE is still needed when training a BNN.

We do use STE in our method, but the real-valued gradients are computed strictly at the points in the binary coordinates. It means we do not hold real weights at all during training. 

- For your interest, our proposed random mask relies solely on the direction of the gradient, not its magnitude. As demonstrated in Table 4 of our original paper, this hypermask performs nearly as well as the EMP mask in larger networks. This indicates that our method is effective even when the gradient is binary. Therefore, in future work, if gradients can be computed entirely within the binary space, our random mask could enable complete binary-space training of BNNs, both in terms of weights and gradients. This is a particularly intriguing possibility for the field.

Reviewer comment (Experiments):
The experiments need to be done using at least some popular models, such as ResNet.

Author response:
The experiments need to be done using at least some popular models, such as ResNet.

Acknowledging the reviewer's feedback, we conducted further experiments with ReActNet, which is based on the well-known ResNet-18 structure, and AdamBNN, which is based on the memory-efficient MobileNetV1 structure. The results of these additional evaluations have been detailed in the global response. Notably, in the fine-tuning experiments (Ex. 2), our method attained a top-1 accuracy of 91.94% on CIFAR-10 and 74.64% on CIFAR-100.

Reviewer comment (Experiments):
The paper does not take BOP [1] as a baseline. BOP should be a closely related work that aims at not using latent weights for training BNNs. *[1] K. Helwegen, Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization, NeurIPS’19.*

Author response:
The paper does not take BOP [1] as a baseline. BOP should be a closely related work that aims at not using latent weights for training BNNs.

We thank the reviewer for suggesting a proper baseline. We have added BOP to the baseline methods in the additional experiment (Ex. 1) shown in the global response and the revised paper.

- Just for the record, BOP by Helwegen et al. is a study that proposed to reinterpret the latent real-valued weight in the BNN as a real-valued accumulator of the gradient. In contrast, our binary-space training eliminates the real-valued accumulator because the l-th gradient can be deleted after computing the (l-1)-th gradient and there is no need to maintain gradients for all layers at the same time. Thus, our binary-space training and BOP are essentially different optimizations.