From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper1510/Area_Chair_T4Uu
title: Effectiveness of Selected bases
comment: In the outset, I continue the discussion because I would like to achieve better clarity and exclude misunderstanding.

1) weight scaling factor.

I understand your argument. However you would have to convince me that having learnable real valued weights with the scaling factor transfer through quantization (and hence BN) has any advantage over having a learnable affine scale parameter in the end of BN. I think maky works found the idea of the weight approximation of XNOR-Net, inferior or at least not bringing an improvement over the latter alternative, which is much simpler. In my view, we do not need to "approximate" any real-valued weights, we just need to learn discrete weights. Sorry I can't recall exact references.

2) I somehow understand your argument why APoT requires 3 bits per activation in that example. It renders this method impractical in my view because 2 bits would suffice to represent 2-bit numbers. Since I still deem that the global scaling factor should be a learnable parameter, the simple uniform quantization would be more efficient in this case.

> For the RealToBinary network, the learnt bases are  $\{2^-1, 2^{-2}\}$, leading to the same effect as uniform quantization.

If the learned bases are equivalent to the uniform quantization, from the modelling point of view we do not need the learned bases. The paper does not demonstrate why learning the bases would have any advantage.

> The experimental results in Table 1 has demonstrated the effectiveness of our selected bases as compared to the uniform quantization method DoReFa-Net with different bases.

First, the difference between DoReFa-Net and the proposed method in accuracy in Table 1 is not shown to be statistically significant. 
Second, even if there was such a difference, it would be due to differences in the learning approaches  and not due to the model. If for the model the uniform quantization suffices, other training methods can be applied. In particular the plain straight-through quantization approach, e.g. Gupta (2015) Deep Learning with Limited Numerical Precision,
(they quantize gradients as well -- this part is not relevant for our discussion).

From: ICLR.cc/2023/Conference/Paper1510/Area_Chair_T4Uu
title: Still not clear
comment: > 1. The scaling factor $\alpha$ does not have gradients
This point is minor and is about formal correctness. If you don't want to differentiate the $||W||_1$ then you cannot write $\frac{d W_t}{d \W_r}  = \dots $ after (6).

> 2. However, the number of operations of our solutions is smaller.

First, convolution of $X$ with the kernel of ones is the same number for all output channels. This convolution is also separable over spatial and channel dimensions. With an efficient implementation its cost would be negligible compared to the convolution with $V$ (and I have mentioned that in the very first comment).

Second, I do not see how it is possible that both solutions are valid since they are to compute the same result $X\star W$. You step "we replace the value -1 with 0", written mathematically is exactly $W = 2V -1$. I do not see how your solution follows from this replacement.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: More explanations
comment: Thank you again for the reviewer's helpful comments! We will revise the paper accordingly. Answers to your questions as below:  
**1. I think this is misleading, the scale will be "eaten" by batch normalization after the convolution and replaced with a new scale in its affine transform. The view that the quantization should approximate the real-valued activations well in MSE is your design choice, but it does not follow from anywhere. If our goal is to train a quantized network, it does not necessarily have to approximate real-valued activations. It may not be relevant for other methods. Therefore I deem the MSE metric generally irrelevant. Perhaps if you observe experimentally that improving it leads to improvements in training in your setup, it would empirically justify it for your model, but not for the comparison in Fig. 4.**  
For the scaling factor, at the training stage, we think that the scaling factor cannot be eaten by the batch normalization as it is specified as $\frac{|| \mathbf{W}r^t||_{1}}{ | \mathbf{W}r^t |}$.  While at the inference stage, the scaling factor, as a constant number, can be merged with the batch normalisation layer. 

For the MSE, our idea originates from the XNOR-Net (**XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks**), which obtains the binary weights by minimising the error of $|| \mathbf{W}_r - \alpha \mathbf{W_b}||^2$.  The difference between our methods and XNOR-Net is that we want to minimize the expectation of $\frac{|| \mathbf{W}_r - \alpha \mathbf{W_b}||^2}{n}$, where $n$ is the number of elements in the weights. Intrinsically, when the real value is quantized, the quantization error between the quantized value and the real value will lead to the information loss. Therefore, to reduce the information loss, it is natural to design a loss function to minimize the distance between the quantized output and the real-valued output.  
**2. Our method has learnt the base (2−1,2−2). Let me clarify why this is a red light for me. The bases (2−1,2−2) are consecutive. An encoding in these bases takes the form (b2+2b1)/4. The expression z=(b2+2b1) is a normal binary encoding of a 2-bit integer z∈0,1,2,3. The scaling factor 1/4 can be taken outside of the binarization and is irrelevant. If b1,b2 is your learned adaptive binary representation of x, then z is simply the quantization of x∗4 to the grid {0,1,2,3}. The quantization to a uniform grid has a natural straight-through gradient estimator: ∇zL:=∇xL. This is what I meant by "the experiments do not clearly demonstrate that learned bases by the proposed method have some advantage". If the method learns consecutive powers of two, it has no advantage over a plain quantization into 2 bits, just a more complex detour with less accurate gradients and hence slower learning.**  
Our method is an on-the-fly quantization, possibly leading to both uniform and non-uniform quantization. We sets a pool of bases $\\{2^{-1}, 2^{-2}, 2^{-3}, …\\}$ and then selects the best bases based on the importance sampling approach during the training. For the RealToBinary network, the learnt bases are $\\{2^{-1}, 2^{-2}\\}$ , leading to the same effect as uniform quantization. The experimental results in Table 1 has demonstrated the effectiveness of our selected bases as compared to the uniform quantization method DoReFa-Net with different bases. To illustrate the gradient computation, we want to give an example here. Given a fixed point real value $0.1011_2$ in binary format, we select the first two significant bits as $0.\underline{\textbf{10}}11_2$ and thus the gradients of the first two significant bits are 1 and others are 0, which is like a clipping operation.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Explanation 
comment: Thanks very much for your helpful comments! Answers to your questions as below:  
**1. Derivative of (5) When you differentiate (5) formally by a chain rule there should be a derivative of ||Wr||1 as well, you see?**  
The scaling factor $\alpha$ does not have gradients. Using the binary $\mathbf{W}_B$ to approximate $\mathbf{W}$ by minimizing the $||\mathbf{W} - \alpha \mathbf{W}_B||$,  we could compute out the optimal value for the scaling factor as $\frac{|| \mathbf{W}r^t||_1}{|\mathbf{W}r^t|}$.  
**2. Convolution of X and W. I still have the following confusion. If X∈0,1 and W∈−1,1 and we want to express the convolution X⋆W in binary operations, we need to encode W=2V−1, where V∈0,1. Then X⋆W=X⋆(2V−1)=X⋆V−X⋆1=2popcount(XANDV)−popcount(XAND mask of ones the shape of W Where am I wrong?**  
We think that the reviewer’s solution is correct to compute the convolution and the number of popcount-and operations is 2*size($\mathbf{W}_b$), where size($\mathbf{W}_b$) denotes the number of elements. However, the number of operations of our solutions is smaller. We replace the value -1 with 0 in $\mathbf{W}_b$ and the other value remains the same. Then the convolution operation of $\mathbf{X}_b$ and $\mathbf{W}_b$ is represented as: $\text{popcount(AND}(\mathbf{X}_b, \mathbf{W}_b))−\text{size}(\mathbf{W}_b)$. Then the total number of popcount-and operations is size($\mathbf{W}_b$), which is smaller.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: More explanations on idea
comment: Thank you for the reviewer's helpful comments! We will revise the paper accordingly. Answers to your questions as below:  
**1. Overall the approach is more similar to quantization rather than binarization. Hence, the terminology is confusing, since even if individual bases are binarized, the overall effect is quantization.**  
We want to elaborate on the difference between quantization and binarization for the neural network model. To our best knowledge, quantization is to constrain the real-valued weights or activations to a discrete set of predefined quantized values and binarization is to binarize the weights or activations into binary values. Therefore,  if we want to use the quantization approaches to address the binarization problem, we need to convert the quantized values into binary values.  
However, not all quantization approaches can work well in the binarization domain. For example, although APoT (ICLR 2021) only uses 2-bit representation for the quantization, it indeed requires 3 bits when expressed in binary representation (The detailed induction is shown in Appendix 4.2). This is due to the intrinsic nature of APoT quantization which is not tailored for binary quantization.  
To demonstrate the effectiveness of our methods over other benchmark quantization methods, we have conducted a thorough comparison between our methods and other benchmark quantization approaches.  
**2. The overall idea of approximating values with fixed point powers of 2 has long been a common optimization approach, especially in the embedded systems/hardware community, and the contributions of this paper are limited in this regard.**  
Different from the previous work using power-of-two values, our method represents the real value into power-of-two bases with binary representations using importance sampling. It is to improve the performance of the binarized network. We have conducted a thorough literature review in the last paragraph of Section 2.1.  
We want to briefly elaborate more to let the reviewer have a better understanding of our idea. Firstly, we observe that the distribution of weights and activations are imbalanced and thus we attempt to improve the performance by reducing the number of weights and increasing the number of activations. Secondly, based on the importance sampling, we select the power-of-two values to approximate the real values. Thirdly, to avoid the redundancy of the activations, we use computational-based pruning to reduce the number of activations.  
**3. In terms of the learnable aspects they have novelty but some design choices are made ad-hoc and are unjustifiable.**  
To the best of our knowledge, the main concern of the design choices is about the induction of Eq. (8) to Eq. (9). And In Section 4.2 of the revised version, we have provided a thorough explanation of how we induce Eq. (9) from Eq. (8).

From: ICLR.cc/2023/Conference/Paper1510/Area_Chair_T4Uu
title: Major concerns
comment: > We assume the scaling factor is a part of the quantization

I think this is misleading, the scale will be "eaten" by batch normalization after the convolution and replaced with a new scale in its affine transform. The view that the quantization should approximate the real-valued activatinos well in MSE is your design choice, but it does not follow from anywhere. If our goal is to train a quantized network, it does not necessarily have to approximate real-valued activations. It  may not be relevant for other methods. Therefore I deem the MSE metric generally irrelevant. Perhaps if you observe experimentally that improving it leads to improvements in training in your setup, it would empirically justify it for your model, but not for the comparison in Fig. 4.

> Our method has learnt the base $(2^{-1}, 2^{-2})$

Let me clarify why this is a red light for me. The bases $(2^{-1}, 2^{-2})$ are consecutive. An encoding in these bases takes the form 
$(b_2 + 2 b_1)/4$. The expression $z = (b_2 + 2 b_1)$ is a normal binary encoding of a 2-bit integer $z \in {0,1,2,3}$. The scaling factor $1/4$ can be taken outside of the binarization and is irrelevant. If $b_1,b_2$ is your learned adaptive binary representation of $x$, then $z$ is simply the quantization of $x*4$ to the grid {0,1,2,3}. The quantization to a uniform grid has a natural straight-through gradient estimator: $\nabla_z L := \nabla_x L$. This is what I meant by "the experiments do not clearly demonstrate that learned bases by the proposed method have some advantage". If the method learns consecutive powers of two, it has no advantage over a plain quantization into 2 bits, just a more complex detour with less accurate gradients and hence slower learning.

From: ICLR.cc/2023/Conference/Paper1510/Area_Chair_T4Uu
title: Response
comment: I thank the authors for the detailed response, which I have red. I cannot check everything again, just a few comments on the technical side.

> Derivative of (5)

When you differentiate (5) formally by a chain rule there should be a derivative of $||W_r||_1$ as well, you see?

> Convolution of  X and W

I still have the following confusion. If $X \in \{0,1\}$ and $W \in \{-1, 1\}$ and we want to express the convolution $X \star W$ in binary operations, we need to encode $W = 2 V -1$, where $V \in \{0,1\}$. Then

$ X\star W = X \star (2 V -1) = X \star V - X \star 1 = 2 \text{popcount}(X \text{AND} V) - \text{popcount}(X \text{AND mask of ones the shape of W}$
Where am I wrong?

From: ICLR.cc/2023/Conference/Paper1510/Reviewer_h8Kp
title: Improved Revision - Limited Novelty
comment: First, I would like to thank the reviewers that took the time to revise the paper and address all the reviewer comments. 

Taking into account the revised manuscript and after considering the comments from the other reviewers as well, I still am of the opinion that the manuscript fails short of being publishable.  

The overall idea of approximating values with fixed point powers of 2 has long being a common optimization approach especially in the embedded systems/hardware community and contributions of this paper are limited in this regard.

In terms of the learnable aspects they have novelty but some design choices are made ad-hoc and are unjustifiable.

Overall the approach is more similar to quantization rather than binarization. Hence, the terminology is confusing, since even if individual bases are binarized, the overall effect is quantization. 

I believe that the authors need to revise the paper in such a way as to make it more clear and position it accordingly against the related work, and target a future submission.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Reviewer comments incorporated in newest revision
comment: Thanks again to all the reviewers for the helpful suggestions! We’ve uploaded a new version of the paper that incorporates some of the reviewer comments.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Explanations on methods and experiments
comment: Thank you for the reviewer's helpful comments! We will revise the paper accordingly. Answers to your questions as below

1. **Questions: It increases the activation count compared to other methods. This might be restrictive for some devices.**  
Even though the number of activation is slightly increased, the total run-time and memory cost of weights plus activations is reduced, as shown in Table 2. We have revised the paper accordingly.

2. **Questions: The actual runtime is not considered.  The latency is mentioned as an important consideration, however, no results are given to support the suitability of the approach for low latency applications. The BFLOPS and activations might not be a good proxy for the latency. real benefits might not materialize on actual runtime for standard devices.**  
To the best of our knowledge, three major elements affect the run-time: a) the hardware design (for example, a GPU is tailored for matrix multiplication while a CPU is tailored for sequential processing, and thus the GPU is faster to process matrix operations.), b) the computational cost, and c) the memory cost. Commodity hardware like GPUs or CPUs only support floating-point arithmetic and the lowest precision is 8-bit (e.g., TensorRT from NVIDIA or TensorLite from Google). Therefore, to report the actual runtime, one needs to design custom hardware, which is highly expensive. Instead, following the practice of most previous methods, we provide a theoretical computation cost based on BOPS and FLOPS, where BOPS is the number of binary operations and FLOPS is the number of floating-point operations. 

3. **Questions: The proposed pruning strategy removes spatial features which is unstructured pruning. The proposed pruning strategy is not compared with other pruning methods.**  
Our approach is to prune the whole channel of the activations, rather than element-level unstructured pruning, belongs to structured pruning. We have compared our proposed pruning approach with the traditional l1-norm sparsity regularisation approach to prune the activations in the paper (Liu, Zhuang, et al. "Learning efficient convolutional networks through network slimming.”) on EBN network. With the BOPS of $1.7 \times 10^9$, our pruning approach achieves an accuracy of 67.4% while the baseline pruning approach achieves 61.9%. And we will show more comparison results in Appendix later.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Explanations on methods and experiments
comment: Thank you for the reviewer's helpful comments! We will revise the paper accordingly. Answers to your questions as below

1. **Questions: The representation of real-values in computer is not simply a sum of power of two. a floating point number consists of a sign bit (s), mantissa (m), radix (r) and exponent (e) parts. And the number is calculated byn=(−1)s×m×2e.Therefore, based on this, I wonder if the proposed method is still valid.**  
The IEEE-754 standard mentioned by the reviewer is a technical standard for floating-point arithmetic on many types of hardware. Our approach is motivated by the fixed-point representation, which has a fixed number of bits for the integral and fractional parts.  For example, $0.7865 = 2^{-1} + 2^{-3} + 2^{-4}$, is $0.1011_2$ in binary format. Then the real-valued $\mathbf{X}_r$ can be represented as the matrix multiplication between a binary matrix and a power-of-two base vector. We have revised the paper accordingly. 
2. **Questions: The transition from Eq. (8) to Eq. (9) is also confusing. The authors only explain that removing S and pit for the sake of hardware-friendly. However, I wonder if the approximation is still true. And more details should be given in Appendix about the derivation of Eq. (8) to Eq. (10).**  
We want to simplify Eq. (8) to not only construct a more succinct expression, but also make the approximation more hardware-friendly, by removing unnecessary floating point operations. Firstly, we can remove $\frac{1}{S}$ from the above expression because it is a constant and can be infused into the weights. Secondly, we ask the question if we can remove $1/p_{i_t}$, which is a floating point number, from the expression. To answer this question, we resolve to empirical evidence; we conducted experiments that are with $1/p_{i_t}$ and without $1/p_{i_t}$. Our experiment results show that $1/p_{i_t}$ does not affect model's performance when S is kept small, which is the case in our implementation with $S=2$ or $3$. We have revised the paper accordingly.

3. **Questions: In Sec.1, the authors declare that the number of activations in a convolution layer in ResNet-like architecture is 1/6 to 1/7 of the weights, which causes the imbalance of activation and weight size and further cause the information bottleneck issue after binarization. However, in their method, the spatial masking is designed for sparse the binarized activation, which makes the activation size even smaller. I do not know how can the masks balance the size of activations and weights.**  
The increase in bit-width for binarized activations increases the computational cost. To reduce the computational cost, we propose the computational cost based pruning approach in terms of BOPS (number of binary operations). Even though our pruning method reduces the activation size, its primary aim is to reduce the computational cost but not the activation size.

4. **Questions: Authors provide comprehensive experiments for analysis and demonstration. But the training strategy and the performance of MSE seems to have little relationship with the proposed method. It seems to have no special designs of training strategy for the MABA-Net, neither does the authors provide any analysis and explanations for the possible reason of why the selected training strategy could be better than others.**  
In Section 5.1.1, we have compared different training strategy and shown that binarising the activation first in a two-stage scheme gets best accuracy. It is an empirical observation. It might be due to that activations are more sensitive to binarization as compared to the weights, and searching the optimal weights from quantized activations is easier.  
The MSE study showcases that our activation binarization method produces the closest estimation to the original full-precision activation in terms of MSE or l2 norm. This result serves as empirical evidence to support that our method is superior than other methods.

5. **Questions: according to the experimental results, the improvement is somewhat incremental compared with previous binarization methods.**  
The advantages of our method are both improved accuracy and reduced memory cost. In Table 1, it shows that for ResNet18, BiRealNet and RealToBinary network, our methods increases the performance more than 1%. For the compact network like EBN and MobileV1-based ReActNet, our methods increases the accuracy by 0.2 or 0.3, but the memory cost is largely reduced.

6. **Questions: Tables and Figures have invalid hyperlink in the text.**  
We have checked all the tables and figures and are unable to find the invalid hyperlink. Finally, we  add subtitle for Fig. 4 to avoid any confusion.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Experiments explanation
comment: Thank you for the reviewer's helpful comments! We will revise the paper accordingly. Answers to your questions as below

1. **Questions: However, the above analysis seems to be only applicable to ResNet structures, but not the compact architectures like MobileNet.**  
Among the various baselines, ReActNet is a MobileNetV1-based architecture. Based on the experimental results on ReActNet from Table 2, it can be seen that our proposed method still outperforms the baseline with much less memory cost.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Experiments explaination
comment: Thank you again for the reviewer's helpful comments! We will revise the paper accordingly.

1. **Questions: In figure 3 I do not observe a significant difference in the end results. In 5.1.3 it is unclear which MSE is measured and why it is relevant or what insight it has for the performance of the training or at the inference stage.**  
We have reported the final performance in the legend of Fig. 3.  The MSE is measured between original unquantized activation distribution and the corresponding quantized values. The smaller the MSE is, the better. 
2. **"ABCNet learns the same bases for different branches" This sounds interesting. Perhaps detailing this phenomenon and explaining the shortcomings of ABCNet could improve the paper.**  
Given a real-valued input $x$, ABCNet compute the binarized value $x_b^i$ of the $i$ branch as $x_b^i = sign (h_{v_i}(x) - (0.5 - v_i))$
where $v_i$ is the shifting scalar for the $i^{th}$ branch and $h_\{v_i\}(x) = \text{clip} (x+v_i, (0, 1))$. Suppose the scalar parameters $v_i$ is sorted, namely, $v_i < v_{i+1}$. Given the same input $x$, $x_b^i$ and $x_b^{i+1}$ are different only when $ 0.5 - v_{i+1} \leq x < 0.5-v_i$. If $v_i$ and $v_{i+1}$ are very close, branches $i$ and $i+1$ will have a lot of same values and thus are highly correlated.  
ABCNet does not specify the gradient for the shifting parameter $v_i$; however, it can be deduced as follow: $\frac{\partial{\mathcal{L}}}{\partial{v_i}} = \frac{\partial{\mathcal{L}}}{\partial{\mathbf{X}_b^i}} \times 1\_{0 \leq { \mathbf{X}_r - v_i \leq 1}}$ where $\mathcal{L}$ denotes the loss function, $\mathbf{X}_b^i$ denotes the binarized activations of the $i^{th}$ branch and $\mathbf{X}_r$ denotes the real-valued activations. For simplification, we assume branch $i$ and branch $j$ has the same scaling factor and then $\frac{\partial{\mathcal{L}}}{\partial{\mathbf{X}_b^i}}$ = $\frac{\partial{\mathcal{L}}}{\partial{\mathbf{X}_b^j}}$. Therefore, if $v_i$ and $v_j$ are initialised close to each other, their gradients,$\frac{\partial{\mathcal{L}}}{\partial{v_i}}$ and $\frac{\partial{\mathcal{L}}}{\partial{v_j}}$, will be close. Thus if $v_i$ and $v_j$ are kept close, based on Eq. (15), $\mathbf{X}_b^i$ and $\mathbf{X}_b^j$ are also close. This effect extends to the learning of the scaling factor $\beta_i$, too. The gradient of $\beta_i$ is calculated as follows: $\frac{\partial{\mathcal{L}}}{\partial{\beta_i}} = \frac{\partial{\mathcal{L}}}{\partial{\overline{A}}} \times \frac{\partial{\overline{A}}}{\partial{\beta_i}} = \frac{\partial{\mathcal{L}}}{\partial{\overline{A}}} \times \mathbf{X}_b^i$, where $\overline{A} = \sum_i \beta_i*\mathbf{X}_b^i$. If $\mathbf{X}_b^i$ and $\mathbf{X}_b^j$ are close to each other, then the gradients of $\beta_i$ and $\beta_j$ are also close. So the learnt $\beta_i$ and $\beta_j$ would be close. In the end, we can see that the performance of ABCNet is really affected by the initialization of the shifting parameter and the scaling factor. More details are shown in Appendix A.3.

3. **This leaves me with the question: was the whole introduction of the multi-branch network necessary? With uniform quantization, the plain straight-through quantization can be applied. I think also a global scaling factor should be discounted from the different bases learned in LQNets visualized in Fig .4.**  
The multi-branch network is tailored for binary encodings but not general quantization. For example, as mentioned in Appendix A.2, although APoT only uses 2-bit representation in the case, it indeed requires 3 bits when it is expressed in terms of binary representation.  
We assume the scaling factor is a part of the quantization and thus we take the scaling factor into consideration for all the methods in Fig.4.

4. **I find that the experiments do not clearly demonstrate that learned bases by the proposed method have some advantage.**  
The learnt bases are power-of-two values that allow the custom hardware to conduct shifting rather than real-valued multiplication.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Additional method explanation
comment: 8. **Questions: Eq. (9) the approximation is not justified. In the end why to do incorrect derivations and unjustified approximations? Just simply call the unnormalized probabilities scores and output C and Q as subsets of X and D for the top S scores.**  
$\mathbf{X}_r$ is real number represented in fixed point representation. Firstly, we remove $\frac{1}{S}$ from Eq. (8) because it is a constant and can be infused into the weights. Secondly, we ask the question if we can remove $1/p_\{i_t\}$, which is a floating point number, from the expression. To answer this question, we resolve to empirical evidence; we conducted experiments that are with $1/p_\{i_t\}$ and without $1/p_\{i_t\}$. Our experiment results show that $1/p_\{i_t\}$ does not affect the model's performance when $S$ is kept small, which is the case in our implementation with $S=2$ or $3$. We have revised the paper accordingly.

9. **Questions: Eq. (10) X_b is already defined. The shift of a fractional N-component vector Xr is unclear.**  
$\mathbf{X}r$ is a fixed point representation. Through left shifting $\mathbf{X}_r$ by $i_t$ bits and a bit-wise AND operation with 1, the ${i_t}^{th}$ most significant bit is obtained. For example, 0.7865 represented in binary format is $0.1011_2$.  If we left shifting $0.10\underline{\textbf{1}}1_2$ by 3 bits, and bit-wise AND with 1, we will get the $3^{rd}$ most significant bit. We have revised the paper accordingly. 

10. **Questions: Eq. (13) Should it be AND instead of XNOR? I think it can be made simpler: 2 * popcount(AND(X_b,W_b)) - popcount(X_b). Note the second popcount is the same for all input-output channel combinations and thus costs little.**. 
In our paper, $\mathbf{X}_b \in \{0,1\}$ and $\mathbf{W}_b \in \{-1, 1\}$. In actual implementation at the hardware, we represent $\mathbf{W}_b \in \{0, 1\}$ by replacing -1 with 0. Then the convolution operation of $\mathbf{X}_b$ and $\mathbf{W}_b$ is represented as: $\text{popcount(AND(}\textbf{X}_b, \textbf{W}_b)) - \text{size}(\mathbf{W}_b)$, where $\text{size}(\mathbf{W}_b)$ denotes the number of elements in $\mathbf{W}_b$ and is a constant number.

11. **Questions: The multi-bit representation seems just a variant of quantization. It is a non-uniform quantization combining powers of two and it is adaptive in the sense that the powers are chosen based on a particular input by the scoring and sorting algorithm. However, this selection algorithm is not differentiable, and seems to be not handled in training with any method. It is not discussed how the gradient propagation works in the multi-bit representation. The independent STE heuristic per bit seems inferior.**  
We represent the real value into the fixed-point representation and the output binary matrix $\mathbf{X}_b$ is a subset of $\mathbf{X}_B$. Therefore, the selected bits have the gradients of 1 while the unselected have the gradient of 0. The approximation is to minimise the expectation of MSE as $E(|| \mathbf{X_B}D - \mathbf{X_b} Q ||_F^2)$ as mentioned in Sec. 4.2. 

12. **Questions: Learnable power-of-two quantization methods have been proposed before. In particular Additive Power-of-Two quantization by Lin et al. (2020) also selects suitable bases adaptively to the weight distribution. The empirical approach in the submission does not stand out as a more justified or a more flexible, etc.**  
APoT defines a quantized value set $Q$ and then selects the quantized value that is closest to the real value $x$. When only 2 bits are used, APoT has quantization levels: $Q = \\{0, 2^{-1}, 2^{-2}, 2^{-3} \\}$. APoT then selects the base that results in smallest quantization error.  APoT requires 3 binary activations bases, which correspond to the bases $\\{2^{-1}, 2^{-2}, 2^{-3}\\}$. Although APoT only uses 2-bit representation in the case, it indeed requires 3 bits when it is expressed in terms of binary representation. This is due to the intrinsic nature of APoT quantization which is not tailored for binary quantization. In contrast, with tailored binarization technique for activation, our method achieves comparable performance to APoT with only 2 binary branches, resulting in 1 bit reduction, as shown in Table 1. The detailed mathematical analysis is shown in Appendix A.2. 

13. **Questions: The masking approach appears interesting. However it is not clear how the theoretical saving is going to be achieved in practice for convolutional networks.**  
As shown in Eq. (10), the BOPS after pruning is: $\sum_l \frac{||\mathbf{M}^l||_1}{{W^l \times H^l}} \times \text{BOPS}^{l}$. The theoretical memory saving is: $\sum_l \frac{||\mathbf{M}^l||_1}{{W^l \times H^l}} \times \text{size}(\text{X}^{l})$.  Our BOPS-based pruning approach belongs to structured pruning by pruning the whole channel of the activation. We have shown our pruning results on EBN in Appendix A.4.

From: ICLR.cc/2023/Conference/Paper1510/Authors
title: Methods explanation
comment: Thanks very much for your helpful suggestions! Answers to your questions as below
1. **Questions: It would be more appropriate to call it therefore the empirical ST method, which somehow defines the gradient but has nothing to do with estimation.**  
Our proposed methods share the idea of straight-through back-propagating the gradients with STE as the sign function has a derivative of zero almost everywhere. However, as mentioned by AC, the difference is that we re-define but not estimate the gradients.

2. **Questions: Eq. (3): the sybmols are not clear.**  
In our paper, we try to use Eq. (3) to explain how the convolution operation between the matrices $\mathbf{X}_b \ \ \text{and} \ \ \mathbf{W}_b$ is converted to XNOR and popcount.  We have revised the Eq. (3) as: $\mathbf{W}_b \circledast \mathbf{X}_b = w_b \cdot x_b = \text{popcount}(\text{xnor}(w_b, x_b))$ where $w_b$ and $x_b$ are the flattened representation of $\mathbf{W}_b$ and $\mathbf{X}_b$, and $\cdot$ denotes the dot product.  In the actual implementation, the inputs of $\\{-1, +1\\}$ will be replaced by $\{0,1\}$ to conduct the XNOR and popcount operation. 

3. **Questions: Eq. (5): It is not clarified what is understood by binary. There is no justification why to introduce this scaling factor.**  
A binary value only takes 2 states/values. Binarization is a function that maps real-value number to one of the 2 values. We use $sign(\cdot)$ function as illustration. $sign(\cdot)$ takes a real-value number $x$, and splits out $x_b ∈ \\{-1,+1\\}$ depending on the sign of $x$. This naive way of binarizing values can be improved by replacing {-1,+1} to {-α, +α}, where α is the scaling factor. The improvement can be attributed to the fact that the scaled binary tensor is closer to the span of the original full-precision tensor. In fact, this idea was proposed in XNOR-Net (Rastegari, Mohammad, et al. "XNOR-Net: Imagenet classification using binary convolutional neural networks." ECCV, 2016.), and it has become a commonly used binarization method to enhance the top-1 accuracy. Due to its popularity among other works, we omitted the justification of using it. 

4. **Questions: In expanding the partial derivative after (6) the chain rule is applied incorrectly and the clipping indicator occurs out of nowhere.**  
Because the derivative of $sign(\cdot)$ function is almost zero everywhere and thus preventing the flow of gradients, conventional methods approximate $\frac{\partial sign(\mathbf{W}_r^t)}{\partial \mathbf{W}_r^t}$ with the identity mapping, namely $\frac{\partial sign(\mathbf{W}_r^t)}{\partial \mathbf{W}_r^t}=1$. As empirically not cancelling the gradients when weights is too large significantly worsen the performance, most methods include a clipping to cancel the gradients when weights is too large.  Therefore, we define an indicator function to set the gradients of the weights that are outside of the range [-1,1] to 0.

5. **Questions: Eq. (7)  can approximate an integer but not a fractional number early in the example.**  
Our idea is to approximate a fixed-point representation number in the range of [0, 1]. In practice, $D = [2^{-31}, 2^{-30}, 2^{-29}, ..., 2^{-1}]^T$, and thus using Eq. (7) we can approximate the fractional numbers. We have revised the paper accordingly.

6. **Questions: Algorithm 1 confusing of symbols. The outputs C and Q do not appear to be binary , resp. power of two as declared. Finally, suppose we use this algorithm for approximation. What does it do? Does it minimize (approximately, in the expectation, etc.) the L_2 error?**  
We have revised the notations of Algorithm 1 to make it clear. In particular, $\mathbf{X}_B^{(i_t)}$ refers to the $i_t$ row of $\mathbf{X}_B$.  We set the $t^{th}$ column of $\mathbf{X}_b$ as $\mathbf{X}_b^{(t)} = \mathbf{X}_B^{(i_t)}$ and the $t^{th}$ row of $Q$ as $Q{(t)} = \frac{D{(i_t)}}{Sp{i_t}}$, and thus the output matrix $\mathbf{X}_b$ is binary. This algorithm is to minimize the the expectation of MSE as $E(|| \mathbf{X_B}D - \mathbf{X_b} Q ||_F^2)$, where $||\cdot||_F$ denotes the Frobenius norm. The reference paper “Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix Multiplication” provide detailed proofs.

7. **Questions: Eq. (8) If p_{i_t} is substituted, then the expression appears to contain powers 2^{-i_t}, which seems incorrect. It recovers an incorrect order for e.g. x=25 (N=1).**  
Our idea is to approximate a fixed-point representation number in the range of [0, 1]. For example, $0.7865 = 2^{-1} + 2^{-3} + 2^{-4}$.  We have revised the paper accordingly.

From: ICLR.cc/2023/Conference/Paper1510/Area_Chair_T4Uu
title: Experiments
comment: In figure 3 I do not observe a significant difference in the end results.
In 5.1.3 it is unclear which MSE is measured and why it is relevant or what insight it has for the performance of the training or at the inference stage.

"ABCNet learns the same bases for different branches"
This sounds interesting. Perhaps detailing this phenomenon and explaining the shortcomings of ABCNet could improve the paper.

"Our method has learnt the base as $(2^{-1}, 2^{-2})$, identified as uniform quantization"

This leaves me with the question: was the whole introduction of the multi-branch network necessary? With uniform quantization, the plain straight-through quantization can be applied.
I think also a global scaling factor should be discounted from the different bases learned in LQNets visualized in Fig .4.

I find that the experiments do not clearly demonstrate that learned bases by the proposed method have some advantage.

From: ICLR.cc/2023/Conference/Paper1510/Area_Chair_T4Uu
title: Additional Feedback
comment: ### Language Errors

"the partial gradient of the binarized value $z_b$ to the real value $z_r$"
 -> Derivative of $z_b$ with respect to $z_r$

... many more

###  Formal Errors

The STE in Bengio et al. applies to networks with stochastic binary neurons, where the gradient of the expected loss exists and needs to be estimated.
In the context of the submission (deterministic network) this does not apply. It does not estimate anything because there is no reference gradient to estimate.
It would be more appropriate to call it therefore the empirical ST method, which somehow defines the gradient but has nothing to do with estimation.

* Equation (3) is rather schematic at best:
What are $X_b$ and $W_b$ -- matrices or vectors?
What it the dot in $W_b\cdot X_b$? Later on the same dot is used for multiplication by a scalar.
The sum has no variable, the small $x_b$ and $w_b$ occur for the first time: are they scalars or bit strings or what? Where is the index to sum over? What is $*$?
How does XOR and popcount apply to {-1,1}-valued inputs? Aa reduction to {0,1} is needed or at least some clarification.
What are $x_b$ and $w_b$ on the right hand side?

* Equation (5):
It is not clarified what is understood by binary. There is no justification why to introduce this scaling factor.

* In expanding the partial derivative after (6) the chain rule is applied incorrectly and the clipping indicator occurs out of nowhere. It would be better to postulate the update or the surrogate gradient, without pretending that it is derived from somewhere. 

* Equation (7):
"We can approximate $X$ by... ": $X$" -> $X_r$. We can't approximate because D contains positive powers and so (7) can approximate an integer but not a fractional number early in the example.


* Algorithm 1: 
Small $x_b$ is not defined and the indexing $x_b^{i_t}$ is unclear. 
D_{i_t} is just 2^{i_t} and so the norm is unnecessary and confusing.
The outputs C and Q do not appear to be binary , resp. power of two as declared.
Finally, suppose we use this algorithm for approximation. What does it do? Does it minimize (approximately, in the expectation, etc.) the L_2 error?

* Equation (8):
If p_{i_t} is substituted, then the expression appears to contain powers 2^{-i_t}, which seems incorrect. It recovers an incorrect order for e.g. $x = 2^5$ (N=1).

* Equation (9): the approximation is not justified. In the end why to do incorrect derivations and unjustified approximations?
Just simply call the unnormalized probabilities scores and output $C$ and $Q$ as subsets of $X$ and $D$ for the top S scores. A clear and transparent heuristic.
But still, (9) has negative powers of two, and so does fit the task. 
Please clearly state that $X_r$ is an integer or a fixed point number represented as (7).

* Equation (10): The value $X_b$ was already defined in the beginning of the section to represent $X_r$. The equation is presented as a definition, but it is a (unclear) identity.
The shift of a fractional N-component vector $X_r$ is unclear.

* The jump to a multi-branch network in the end of Section 4.2 is unclear.

* Equation (13): Should it be AND instead of XNOR? I think it can be made simpler:
2 * popcount(AND(X_b,W_b)) - popcount(X_b). Note the second popcount is the same for all input-output channel combinations and thus costs little.


### Method

The multi-bit representation seems just a variant of quantization. It is a non-uniform quantization combining powers of two and it is adaptive in the sense that the powers are chosen based on a particular input by the scoring and sorting algorithm. However, this selection algorithm is not differentiable, and seems to be not handled in training with any method. See 
James Diffenderfer, Bhavya Kailkhura (2021, ICLR): Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network.
for an empirical approach for differentiating through ranking.

Learnable power-of-two quantization methods have been proposed before. In particular Additive Power-of-Two quantization by Lin et al. (2020) also selects suitable bases adaptively to the weight distribution. The empirical approach in the submission does not stand out as a more justified or a more flexible, etc.
                                                                                                                                       
Finally, it is not discussed how the gradient propagation works in the multi-bit representation. The independent STE heuristic per bit seems inferior. In particular, if bases use consecutive powers of 2 (as is observed in Table 1), one would expect that a straight-through quantization heuristic (as commonly applied in quantization) to be more reasonable.

The masking approach appears interesting. However it is not clear how the theoretical saving is going to be achieved in practice for convolutional networks.

From: ICLR.cc/2023/Conference/Paper1510/Reviewer_X9gr
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposed multiplying binary weights with binary activations that are weighted by power-of-2 factors in order to approximate real-valued activations. Further, the authors proposed spatial masking to reduce the information redundancy introduced by increase in bit-width for binarized activations.
strength_and_weaknesses: (+) The idea of using power-of-two factors to multiply with the activations and approximate the real-valued activations is interesting and reasonable

(+) The analysis that the activation size at most of the layers is an order of magnitude smaller than that of the weight size is reasonable and gives a good background for the proposed idea.

(-) However, the above analysis seems to be only applicable to ResNet structures, but not the compact architectures like MobileNet.

(+) The results are promising. The authors demonstrated higher results than the previous state-of-the-art binary neural networks.

Can authors discuss more about the potential of extending this work to MobileNet structures?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The quality is good. For the novelty, although there are several similar works in the literature, for example, the power-of-two quantization and the ABC-Net that use multiple groups of binarized paths, this paper still has its uniqueness by multiplying the multiple branches in the binary neural networks by the power-of-two factors. 
summary_of_the_review: Overall, this paper proposes a reasonable solution to bridge the performance gap between the binary and real-valued neural networks and shows good experimental results.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1510/Reviewer_D6ip
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This manuscript propose an insteresting point of view that balancing the size of activation and weight can help maintain the accuracy while binarization. And the authors propose the Masked Additive Binary Activation Network to reduce the quantization error and improve the accuracy. 
strength_and_weaknesses: Strength:
(1) The authors claim that balancing the size of activation and weight can help maintain the accuracy while binarization, which is interesting.

Weakness:
(1) The representation of real-values in computer is not simply a sum of power of two. a floating point number consists of a sign bit ($s$), mantissa ($m$), radix ($r$) and exponent ($e$) parts. And the number is calculated by
$$
n = (-1)^s \times m \times 2^e.
$$
Therefore, based on this, I wonder if the proposed method is still valid.

(2) The transition from Eq. (8) to Eq. (9) is also confusing. The authors only explain that removing $S$ and $p_{it}$ for the sake of hardware-friendly. However, I wonder if the approximation is still true. And more details should be given in Appendix about the derivation of Eq. (8) to Eq. (10).

(3) In Sec.1, the authors declare that the number of activations in a convolution layer in ResNet-like architecture is 1/6 to 1/7 of the weights, which causes the imbalance of activation and weight size and further cause the information bottleneck issue after binarization. However, in their method, the spatial masking is designed for sparse the binarized activation, which makes the activation size even smaller. I do not know how can the masks balance the size of activations and weights. 

(4) Authors provide comprehensive experiments for analysis and demonstration. But the training strategy and the performance of MSE seems to have little relationship with the proposed method. It seems to have no special designs of training strategy for the MABA-Net, neither does the authors provide any analysis and explanations for the possible reason of why the selected training strategy could be better than others. 

(5) Meanwhile, according to the experimental results, the improvement is somewhat incremental compared with previous binarization methods. 

(6) Tables and Figures have invalid hyperlink in the text. 

clarity,_quality,_novelty_and_reproducibility: Overall, the paper proposes some interesting ideas. However, the clarity of the work needs to be improved.
summary_of_the_review: Overall, the paper propose an interesting view of binarization. However, the motivation of the proposed method need to be strengthen. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper1510/Reviewer_h8Kp
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes MABA-Net to approximate real-valued activations by accumulating a limited set of binary activations weighted by power-of-2 factors. In addition, it proposes a trainable binary mask as an indicator to keep or drop activation features at the spatial position. Overall, it combines binarization and pruning. Results on ImageNet demonstrate that the method provides improved accuracy with reduced memory for weight values. 
strength_and_weaknesses: Strengths:
- The proposed method provides higher accuracy on ImageNet compared to other methods.
- The memory for weight values is reduced

Weaknesses:
- It increases the activation count compared to other methods. This might be restrictive for some devices. 
- The actual runtime is not considered.
- The proposed pruning strategy removes spatial features which is unstructured pruning. Hence, real benefits might not materialize on actual runtime for standard devices. 
- The latency is mentioned as an important consideration, however, no results are given to support the suitability of the approach for low latency applications. The BFLOPS and activations might not be a good proxy for the latency.
-The proposed pruning strategy is not compared with other pruning methods.

clarity,_quality,_novelty_and_reproducibility: The quality of this paper is good, and it reads well. The background and related work aspects are also covered in adequate manner. Overall, the approach does not show major novelty with regards to approaches for weight binarization. The novelty is rather on the network structural side that handles the binarized weights. The reduction of parameters can also be partially attributed to the use of group convolution. Ablation studies with and without group convolutions should be performed. 
summary_of_the_review: While this paper proposes a method that improves accuracy on ImageNet the results are not conclusive to assess how good the approach is compared to the state of the art. The approach reduces weight memory but the activation memory which is important for on device storage is increased. Furthermore, it is not clear how effective the pruning strategy is, compared to other pruning approaches. Overall, while the approach is partially novel there are many aspects that this paper needs to improve on.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold