Paper: SPFQ: A Stochastic Algorithm and Its Error Analysis for Neural Network Quantization (ICLR-2024)

Reviewer comment (Evaluation):
the error analysis was not validated by the simulations. Simulations only reflect the accuracy of the quantizer compare to unquantized method but it does not say anything about how good the error analysis is.

Author response:
Thank you for this comment, with which we strongly disagree. We believe that theoretical work is of value in machine learning. As such, we focused our attention in this paper on proving that a variant of GPFQ, namely SPFQ, can provide full-network error bounds. We provided a short numerical experiments section to validate that the method is viable in practice, and performs comparably to GPFQ which was extensively compared to other methods in the original GPFQ paper (J. Zhang, Y. Zhou, R. Saab, "Post-training quantization for neural networks with provable guarantees", SIAM Journal on Mathematics of Data Science, 2023). Indeed, the numerics suggest that SPFQ can outperform GPFQ in some cases. Due to its stochasticity, SPFQ if run multiple times, can potentially even provide significantly improved performance at the cost of multiple runs. While we appreciate the practical importance of such explorations, we note that in order to explore these issues one would need to run extensive numerical experiments (as done in the GPFQ paper). Given the purpose of this paper, and the computational resources needed for running such experiments, we choose to keep the focus on theory.

Reviewer comment (Experiments):
experiments does not use any baseline method for comparison with the proposed method. Only unquantized model accuracy was compared. This is unacceptable.

Author response:
Thank you for this comment, with which we strongly disagree. We believe that theoretical work is of value in machine learning. As such, we focused our attention in this paper on proving that a variant of GPFQ, namely SPFQ, can provide full-network error bounds. We provided a short numerical experiments section to validate that the method is viable in practice, and performs comparably to GPFQ which was extensively compared to other methods in the original GPFQ paper (J. Zhang, Y. Zhou, R. Saab, "Post-training quantization for neural networks with provable guarantees", SIAM Journal on Mathematics of Data Science, 2023). Indeed, the numerics suggest that SPFQ can outperform GPFQ in some cases. Due to its stochasticity, SPFQ if run multiple times, can potentially even provide significantly improved performance at the cost of multiple runs. While we appreciate the practical importance of such explorations, we note that in order to explore these issues one would need to run extensive numerical experiments (as done in the GPFQ paper). Given the purpose of this paper, and the computational resources needed for running such experiments, we choose to keep the focus on theory.

Reviewer comment (Theory):
The original version of SPFQ is motivated by minimizing the 2-norm of the pre-activation's quantization error, from which the authors deviate in their two proposed variants (though the second variant contains the original as a special case). Since the variants no longer strictly control the 2-norm of the quantization error and the authors do not discuss the benefits of this deviation, it appears that their only purpose is to enable derivation of the error bounds in sections 3 and 4.

Author response:
Thank you for the opportunity to clarify. The data alignment is indeed for technical reasons related to the proof. That said, the reviewer is not correct that the variants are no longer controlling the 2-norm of the quantization error. They, in fact, are. This is clearly highlighted in Theorems 3.1, and 4.1 which control precisely the 2-norm of the quantization error.

Reviewer comment (Experiments):
The authors do not numerically evaluate any of their bounds. Due in part to their complexity, it is impossible to gauge the bounds' tightness in any scenario.

Author response:
We derive full-neural network error bounds. To validate their tightness, we would need to run experiments with multiple neural networks of varying width, depth, and data-sizes. In other words, we would need to vary $m, N_i, i=1,...,L$ which would be very computationally expensive for real-world networks.

Reviewer comment (Theory):
The usefulness of the relative error bound in Corollary 3.2 and related results seems very limited. I am not convinced they deserve to be highlighted as one of the premier contributions in Eq (2), because the authors need to assume that network weights follow a standard Gaussian distribution for the results to hold, an assumption that usually holds before training but certainly not after. This means that the results regarding the number of bits required to quantize a network seem to only hold for untrained networks in practice.

Author response:
The purpose of Corollary 3.2 is just to illustrate the use of Theorems 3.1 and 4.1 (which provide absolute error bounds) to obtain relative error bounds when extra assumptions on the weights are available. In the corollary, we made the Gaussian weight assumption to illustrate this use. That said, there are regimes, e.g. lazy training (see "On lazy training in differentiable programming" by L. Chizat et al. and "Disentangling feature and lazy training in deep neural networks" by M. Geiger et al.), where the final values of the trained weights are very close to their initialization, which is itself Gaussian. This is very close to the setting of our corollary.

Reviewer comment (Experiments):
The authors do not state what SPFQ’s intended purpose is—whether it is to compress a network or to speed up inference. This is important because it determines the relevant benchmarks the method should be compared with.

Author response:
The purpose is to quantize the network so it can be implemented in hardware, more cheaply, and with faster inference times. Compression is an added benefit.

Reviewer comment (Experiments):
The experiments are relegated to Appendix H and are limited to small architectures and comparison with only one other related work. There are no comparisons with state-of-the-art post-training quantization methods.

Author response:
We again highlight the importance of theoretical work in machine learning. We also point out that we compared SPFQ to GPFQ and found them quite similar for reasonable choices of $m$. In turn, the GPFQ paper (J. Zhang, Y. Zhou, R. Saab, "Post-training quantization for neural networks with provable guarantees", SIAM Journal on Mathematics of Data Science, 2023). has extensive comparisons to state of the art methods.

Reviewer comment (Novelty):
The authors should mention the related work of [2], which also explores the idea of sequentially correcting for poor stochastic quantization in a greedy/myopic way, though in the context of variational autoencoders.

Author response:
We thank the reviewer for the reference, but unfortunately could not see the connection.

Reviewer comment (Novelty):
It is not clear what the novelty of this work is. It seemed to me that the core framework regarding the greedy quantization algorithm was already proposed in the prior work Lybrand & Saab (2021). In terms of proposed algorithm, is the only contribution of this work the replacement of the deterministic quantizer in Lybrand & Saab (2021) by a stochastic quantizer in order to make the analysis feasible for deeper neural networks? If so, the authors should clarify the fact that stochastic quantizer is NOT actually proposed first in this paper, since I could not find any relevant references to the same where stochastic quantization is introduced. Eq. (6) in this work is a well-known quantizer design. See for example:  
(a) Non-subtractive dither (https://ieeexplore.ieee.org/document/823976) -- This is exactly the same, or for instance, see  
(b) Eq. (4) of (https://ieeexplore.ieee.org/abstract/document/4542554)  
The idea may date even earlier, but stochastic quantization is quite widely used for quantization in distributed optimization, etc. Keeping this in mind, it seems like the idea of replacing the deterministic quantizer of GPFQ with a stochastic quantization is straightforward. Although, the analysis of this and the conclusions is not trivial.

Author response:
**Response to weakness 1** We thank the reviewer for this comment. To be clear, nowhere do we claim that we are the first people in the history of quantization to propose stochastic quantization, or randomized rounding. As the reviewer noted, this has a long history in electrical engineering. That said, our contribution is to employ the stochastic quantizer in order to prove the first full-network theoretical error bounds, as previously noted. We have added a sentence (highlighted in blue) to this effect, see bottom of page 3.

Reviewer comment (Experiments):
It is very important to keep in mind that stochastic quantization cannot actually be implemented in hardware as it is not easy to quantize to values that are not allowed by current hardware primitives provided by the GPUs (e.g., 4-bit Floating point). This is important to note because although the theory is pretty comprehensive with stochastic quantization, modifications must be made to the algorithm to account for hardware limitations. For instance, the stochastic quantizer ensures that the quantizer output is unbiased, and this gives a nice upper bound on the expected quantization error. However, since this problem is highly inspired from practical purposes, it is important to show some numerical simulation results for the non-deterministic version, especially since it might be likely that the bias in quantization errors from deterministic quantizers may propagate and accumulate in the greedy algorithm. For example, parallel research directions such as this: https://github.com/TimDettmers/bitsandbytes also work on neural network quantization, and assume that the weights are Gaussian, but translate to actual hardware benefits when deployed.

Author response:
**Response to weakness 2:** We again thank the reviewer for the opportunity to clarify a misconception. First, we note that the mapping of weights to  alphabet elements  (i.e., weight quantization) is not intended to be done in hardware but is in fact expected to be done on a computer after the neural network is trained. The quantized network can then be built into hardware. This differs from signal processing applications where measurements are taken and quantized in real-time. That said, in the context of neural networks, the quantities that need to be "quantized" in hardware in real-time are the activations, but this is done deterministically, via simple bit-shift operations and is not the subject of our paper. Nevertheless, we stress that our analysis has no trouble with such quantization of activation functions. Activation quantization (in contrast with weight quantization) mathematically amounts to rounding the activation to its nearest element in the alphabet, and this rounding is a Lipschitz function hence all our theory applies.  

Stochastic quantization is a process that is not directly executed by hardware. Rather, it's a method used to derive quantized weights, which are values of limited precision. It is these quantized weights that are actually used in hardware.

Separately, the procedure that converts high-precision weights into these quantized weights is carried out on a computer. This involves a stochastic quantizer. After this process is completed and the (finite precision) quantized weights are determined, they can be applied in hardware devices. To summarize, an algorithm is run on a computer to produce these limited-precision weights, and then these weights are utilized in hardware systems.

Reviewer comment (Experiments):
The work only considers fully connected neural networks and subsequently, uses results from vector quantization. In other neural network models like attention, parameters can be matrices instead, where low rank compression becomes important in addition to quantization. It is important that the authors highlight this limitation.

Author response:
**Response to weakness 3:** This is not a limitation of the algorithm, though we did not theoretically analyze self-attention. To see that the algorithm can be readily applied to attention layers, note that these are of the form $\rho(XW_Q W_K^T X^T)X W_V$. As such, one can simply use SPFQ to quantize each column $w$, of $W_{K,Q,V}$, such that $XW \approx XQ$. If instead, a low-rank approximation is employed, so that $W=LR$, one can first quantize $L$ using SPFQ, then quantize $R$ as if it were the weights of the next layer.

Reviewer comment (Experiments):
Please move the numerical simulations to the main paper. A problem which is inspired by a practical issue such as this should be justified with numerical simulations. Also, most of the time, GPFQ performs better than SPFQ. Why should SPFQ be preferred in that case? Except for the theoretical guarantees?

Author response:
**Response to weakness 4:** Thank you for this comment. We believe that theoretical work is of value in machine learning. As such, we focused our attention in this paper on proving that a variant of GPFQ, namely SPFQ, can provide full-network error bounds. We provided a short numerical experiments section to validate that the method is viable in practice,
and performs comparably to GPFQ (which was extensively compared to other methods in the original GPFQ paper: J. Zhang, Y. Zhou, R. Saab, "Post-training quantization for neural networks with provable guarantees", SIMODS, 2023). Indeed, the numerics suggest that SPFQ can outperform GPFQ in some cases. Due to its stochasticity, SPFQ if run multiple times, can potentially even provide significantly improved performance at the cost of multiple runs. While we appreciate the practical importance of such explorations, we note that in order to explore these issues one would need to run extensive numerical experiments (as done in the GPFQ paper). Given the purpose of this paper, and the computational resources needed for running such experiments, we choose to keep the focus on theory.

Reviewer comment (Theory):
Comparing the assumptions in Thm 3.1 (infinite quantization points) and Thm. 4.1 (finite quantization points), it looks like the Gaussian weight assumption is only used in the latter. If I understand correctly (please correct me if I'm mistaken), then this assumption allows for a finite bit requirement because of strong concentration properties of the Gaussian distribution. Analysis of the expected quantization error for the Gaussian weights assumption yields a bit-requirement of log log N per parameter.  
A similar result can be obtained without the Gaussian weight assumption in the worst-case analysis. This is done by side-stepping the Gaussian assumption by utilizing a Randomized Hadamard transform (which has similar concentration properties that can be obtained using Chernoff bounds). This was done in this recent work (https://ieeexplore.ieee.org/abstract/document/10095529) for linear regression and classification models. See Table 1. Since the error bounds essentially depend on the vector quantization error for Lipschitz activations, can similar results be derived utilizing the Hadamard transform-based randomized quantizer as proposed in this work to sidestep the Gaussian weights assumption, and without changing the final conclusion of the results?

Author response:
**Response to question 1:** Thank you for giving us the opportunity to clarify this misconception. The Gaussian assumption is NOT used in theorem 4.1, which holds deterministically. Gaussianity is only assumed in Theorem 4.2, simply to illustrate how Theorem 4.1 can also be used to obtain relative (vs absolute) error bounds. Here, the reviewer's intuition is correct. The randomness of the weights allows for the denominator to concentrate in norm. 

**Response to question 2 ("Since the error bounds essentially...conclusion of the results?"):**  This is a good question. The idea of using a randomized Hadamard transform (which is standard in compressed sensing literature, and the literature on fast Johnson-Lindenstrauss transforms, and in randomized numerical linear algebra) is often a reasonable one. However, in this context, it is not clear how it applies since it would require transforming the weights which would in turn require transforming the inputs to each layer in order to preserve the output.