Paper: Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators (ICLR-2024)

Reviewer comment (Theory):
The proposed solution described in Section 3 is too qualitative. For instance, the underflow region is described as a "hard" region to "escape" from. Can the "escape" and "hardness" be presented in a mathematical manner? Similar for later description of parameters being "stuck" and so on.

Author response:
We agree this was not sufficiently clear. Consider a neural network, where a specific scalar weight $w$ is connected to its scalar activation $x$ as input, and their product is $z=xw$. Suppose $w$ is small enough so that $z$ consistently results in product underflow, i.e. $Q_{\mathrm{prod}}(z) = 0$. In this case,  during forward propagation, the value of $x$ has little to no direct effect on the output neuron to which $w$ is connected. Therefore, it is reasonable to assume that the computed neural gradient $g \equiv \frac{d\mathcal{L}}{dz} $ (where $\mathcal{L}$ is the loss) will be uncorrelated with $x$. Consequently, the gradient update of the weight $w$ will be $\Delta_{w} \propto gx$, with the expected value $\mathbb{E}[\Delta_{w}] \propto \mathbb{E}[g x] = \mathbb{E}[g] \mathbb{E}[x]$. Based on previous quantization literature [1], we have approximately $E[g]=0$, and so $\mathbb{E}[\Delta_{w}]= 0 $. Therefore, any sufficiently small weight $w$ will become ``stuck", so that its $z$ cannot escape underflow for a long time. 

We added a new section in Appendix (Section F), for the theoretical of this issue.

[1] Banner, Ron, et al. "Scalable methods for 8-bit training of neural networks." Advances in neural information processing systems 31 (2018).

Reviewer comment (Experiments):
Benchmarks employed are diverse, but relatively simple (ResNets and Berts). Can the empirical results be augmented with Transformer models, such as GPTs?

Author response:
Please note Table 5: the results there are on the Lamma2-7B model (i.e., which has 7 billion parameters). This is as close to GPT as we can get given our experiments rely on simulated hardware for the LBA (which is much less efficient than standard training). 
    
 Also note, that this model introduces a new difficulty since the range of values in the network varies greatly from layer to layer (which we attribute to splitting the layers to LoRA a and LoRA b). Therefore, a single exponent bias is not sufficient for using $M7E4$ accumulator. This issue can be solved using a dynamic exponent-bias (changes from layer to layer), and using this method, we were able to achieve $45.1$% validation MMLU score when fine-tuning Llama2 (baseline is $45.3$%), or $35.2$% when fine-tuning LLamaV1 (baseline $35.4$%). The results were added to the paper (Table 5), as well as a larger version of Bert in table (Table 5).

Reviewer comment (Evaluation):
The method proposed in this paper is difficult to be applied to accelerate real-world low bit width neural networks. Although this paper claims that an accumulator with a low bit width of 12 bits FP can be implemented with cheaper hardware, standard hardware typically only provides floating-point bit widths of 8 bits, 16 bits, and 32 bits. Implementing such acceleration requires specialized hardware design, which requires more collaborative design and additional costs. The optimization of low bit floating-point accumulators under the same cost, as well as the accuracy that can be achieved by fixed-point quantization models with the same hardware cost, remains to be discussed.

Author response:
We respectfully disagree with how the reviewer classified numerical formats as "standard" vs "non-standard", in the context of deep learning accelerators. After all, the FP16 format (half precision, IEEE 2008), which is the default setup for deep learning applications, only had a bare-bone implementation in hardware before being heavily utilized in the Volta microarchitecture (2017). The FP8 formats (M2E5 /M3E4/ M4E3) do not have an IEEE definition and are not yet supported in standard hardware (Specifications were only published last year[1]). Our work is focused on a primary component of cutting-edge hardware AI accelerators. We believe the fact that FP12 is not standardized yet is a rather small obstacle in this context.

Furthermore, if any hardware vendor does consider FP12 to be problematic, it can still make use of applying our method for half-precision accumulators. Half-precision accumulators do exist in the market, but they are not commonly used. We do not know the hardware design of existing products, but, as we discussed in the paper and the hardware analysis, our implementation does have clear potential to be more efficient than the FP16 LBA accelerators previously discussed in the literature, while providing more accurate models.

[1] https://developer.nvidia.com/blog/nvidia-arm-and-intel-publish-fp8-specification-for-standardization-as-an-interchange-format-for-ai/

Reviewer comment (Evaluation):
This paper lacks accurate evaluation and theoretical analysis of the error and accuracy requirements of floating-point models. Although Table 1 in the paper categorizes the errors in several cases of floating-point quantization, and Figure 2 shows several cases of errors, it is still not possible to quantitatively evaluate the impact of low bit width floating-point accumulation on model inference errors. It is difficult to make people believe with certainty the scalability and reliability of this scheme.

Author response:
We agree with the reviewer's main point, that the effect of floating-point accumulation on models is not sufficiently clear from the paper, and we thank the reviewer for pointing this out. We added the following table (see: comment) to the appendix, describing the zero-shot accuracies of pre-trained models (ResNet, in this case) with different types of accumulator quantization. While zero-shot accuracies are not fully indicative of the quality of the fine-tuned LBA network, the table does showcase the challenge of reducing the bit-width, and the motivation behind the settings we had selected (M=7, E=4, $b_{\text{acc}}=10$). The new section, with this table, was added as Appendix B.

| Mantissa Effect             |          |       |                |        |        |                                        |
|-----------------------------|----------|-------|----------------|--------|--------|----------------------------------------|
| Model                       | Baseline | M10E5 | M9E5           | M8E5   | M7E5   | M6E5                                   |
| ResNet18                    | 69.75    | 69.50 | 68.95          | 66.70  | 57.09  | 20.49                                  |
| ResNet34                    | 73.31    | 73.17 | 72.68          | 70.46  | 60.07  | 17.19                                  |
| ResNet50                    | 76.12    | 75.95 | 75.57          | 73.70  | 64.94  | 19.48                                  |
| **Exponent Bias Effect (M7E4)** |          |       |                |        |        |                                        |
| Model                       | $b=8$    | $b=9$ | $b=10$         | $b=11$ | $b=12$ | $b_{\text{acc}},b_{\text{prod}}=10,12$ |
| ResNet18                    | 55.68    | 60.64 | 60.00          | 58.84  | 56.96  | **60.14**                         |
| ResNet34                    | 50.80    | 63.30 | **63.88** | 62.46  | 59.90  | 63.65                                  |
| ResNet50                    | 26.41    | 64.25 | **68.69** | 67.57  | 66.12  | 68.49                                  |

Reviewer comment (Theory):
In the last paragraph before section 3.1, the author chooses b_acc = b_prod - 1/2 log_2(chunk size) as an offset, which does not seem to guarantee that overflow will not occur under any condition. Therefore, more discussion and theoretical analysis are needed regarding parameter selection and error evaluation.

Author response:
To be clear, the equation $b_{\text{acc}}=b_{\text{prod}}-\frac12 \log_2(\texttt{chunk size})$ only concerns the relative difference between the exponent bias terms. This is a relatively minor change, which is not intended or expected to prevent overflows on its own. To prevent overflows, the absolute value of $b_\text{acc}$ should be chosen using either layer-output statistics or zero-shot evaluation of pre-trained networks.

Reviewer comment (Evaluation):
The content of this paper only includes an evaluation of the model accuracy and does not discuss or analyze the actual cost. For example, regarding the estimation of multiplier and accumulators’ power and silicon area, how much performance improvement or resource savings can be achieved through the optimization proposed in this paper. It would be interesting to discuss the benefits of these optimizations in the context of the additional accuracy loss and design complexity required by the methods presented.

Author response:
See the general comment for our hardware analysis. The quantization scheme in this paper was made with chip-design limitations in mind, and should not result in any additional design complexity. For example, this is the reason we insisted on using bit-masking (no additional logic), and avoided more accurate yet expensive methods, like nearest-neighbour-rounding.

Reviewer comment (Theory):
In section 4, one method is using STE as a recursive way on FMA. This seems to require all FMA operations to be unfolded in sequence. As far as I know, the FMA gradient of deep learning training seems to be treated equally, otherwise it will seriously affect efficiency and be unrealistic. This is because GEMM is a highly optimized parallel operation, and additional branches are not suitable for use here.

Author response:
The reviewer is correct about GEMM being highly parallelized, and our experiments were designed with this in mind. However, it is not true that the recursive STE requires the operation to be unfolded. All simulated GEMM operations and gradient estimation in this paper used a chunk size of $16$, which means that an operation with an accumulation width of $N$ is initiated with $\frac{N}{16}$ parallel operations (i.e., the first hierarchy), before aggregating the results (i.e., the second hierarchy). In the case of recursive STE, every detection of OF/DIFF during re-computation will result in a `$0$' in all preceding operations, just as we saw for sequential accumulation. The only difference for parallel accumulation is that the hierarchy tree can expand in two directions (Like Figure 1 (right) in the paper, with all the arrows reversed). We added this explanation to Appendix D (Old Appendix C, where we discussed fine grained methods in details)

Reviewer comment (Novelty):
The paper needs to provide more context on related works. For example, what is the key difference between the proposed method and the prior work Wrapnet? Is it training vs. non-training?

Author response:
WrapNet, like our work, is also directed toward inference. The main difference in the setting between our work and WrapNet is that WrapNet uses Integer quantization, while our work uses floating point quantization and arithmetic. The integer quantization of the accumulator results in different challenges, as there, swamping is not an issue, and overflow becomes the main problem. (Presumably, this is why using very low bit weights seems to be mandatory to reduce overflow in WrapNet; but this also directly harms its final test accuracy). We expanded the related work discussion in section 2.2, and made sure this is better clarified in the last paragraph of 3.1, where we discuss the differences.

Reviewer comment (Evaluation):
There is a lack of evaluation or estimation on the hardware benefits of the proposed method. What will be the gate count/computational energy/latency improvement if using 12-bit accumulators compared to FP16/BF16 accumulators?

Author response:
See the general comment for our hardware analysis. As we explained there, we could not reliably evaluate computation energy or area, but we did include a gate count estimation.

Reviewer comment (Experiments):
Low-bitwidth accumulator under the context of integer quantization is not explored in the main experiments. Integer quantization is mentioned in Section 2.2 when introducing fixed-point quantization, but it seems to be disconnected from the rest of the evaluation.

Author response:
Correct. Our work is strictly focused on floating point quantization (as we explained in the introduction, we were motivated to do this by the recent popular trend of floating point quantization in transformers [1]). Integer accumulators were already explored in prior work (WrapNet). Fixed-point quantization was mentioned in the preliminary section to give full context and as a warm-up for the less trivial, floating-point quantization

[1] https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html

Reviewer comment (Evaluation):
It is recommended that the paper include a comparison of the computational complexity of MAC operations between the proposed method and the previous works [1,2,3]. Additionally, the author suggested discussing the quantization overhead associated with the new approach in comparison to [1,2,3].

Author response:
> It is recommended that the paper include a comparison of the computational complexity of MAC operations between the proposed method and the previous works [1,2,3]. 
  
See the general comment for our hardware analysis. As for [2,3] (4-bit quantization), we did not include a direct comparison with 4-bit neural networks, since the changes are orthogonal, and we fully expect 4-bit neural networks to work with LBA once their implementation is standardized. From our experience, 4-bit can be adapted to residual neural networks with some effort, but is still too experimental for modern transformers (especially for activations).
 
> Additionally, the author suggested discussing the quantization overhead associated with the new approach in comparison to [1,2,3].

There is no explicit quantization overhead in our method. Since we rely on bit-masks (with no rounding), our method is similar to just removing logic connections within the FMA design, which should always translate to cheaper hardware post-synthesise. We added a small clarification for this in section 3, where we present the FMAq component.

Reviewer comment (Experiments):
The impact of chunk size requires further exploration to determine if reducing the chunk size also diminishes the bit size.

Author response:
> The impact of chunk size requires further exploration to determine if reducing the chunk size also diminishes the bit size. 

Typically, higher chunk-size reduces the error in floating point accumulation [4] and was shown to do so in neural networks as well [1], up until the point where chunk-size $=\sqrt{N}$. In almost all experiments we have chunk-size $<\sqrt{N}$. The only exception is the case where LoRA is applied (like in our Table 5 experiment). There, some layers have a very low accumulation size, and the chunk-size can easily exceed $\sqrt{N}$. 
Generally, we aimed to avoid treating the chunk-size as a configurable hyperparameter, since in practice, the chunk-size used during GEMM operations is a result of the accelerator architecture. This is highlighted in section 2.4, where we discuss the previous works, including [1].

Reviewer comment (Experiments):
Can the distribution of the accumulations (which might follow a normal or other distribution) affect the performance of the quantization approach?

Author response:
> Can the distribution of the accumulations (which might follow a normal or other distribution) affect the performance of the quantization approach?

It is always true that the ideal quantization approach (e.g., in terms of MSE), will change depending on the statistics of the quantized values. Due to the large range of represented values and the low relative error, floating-point quantization is relatively robust, but it is expected to be sub-optimal in cases where the distribution of values requires a smaller range.