Paper: Hardware-Friendly Post-Training Quantization: Input- and Output-Channelwise Scale and Offset (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the paper raises concerns, as the input-channel scale and shift resemble group-wise quantization that has been studied in prior works like Q-BERT. Although the authors emphasize advantages in hardware implementation, the improvement over group-wise quantization appears minor. Can you provide the comparison between group-wise quantization (with or without the power of two scales) in computation complexity and performance?

Author response:
**Q1. Compare with other input channel-wise grouping**

**A1-1. Differences from Existing Channel Grouping Methods.**

> Previous works in group-wise quantization, like Q-BERT[12], require each group to have different scale values. As highlighted in QLoRA[9], this requirement creates significant memory overhead during low-bit quantization. For example, if weights for 64 groups are quantized with a single scale per group, it demands 32-bit scales for each of those 64 weights. This scenario is similar to each weight incurring an additional memory overhead of 0.5 bits. While this impact is less pronounced in higher-bit quantizations, in the case of 2-bit quantization, it equates to an approximate 25% increase in memory requirements.

> Our method, instead of creating groups based on unrelated weight positions, undergoes a calibration stage to find the optimal group for predetermined scale values. Since shuffled channels are independent, they can be reorganized through permutation, eliminating the need for additional scale information memory. Furthermore, we opted for shift operations to stay within the integer domain to avoid requiring floating point multipliers and adders if the scale multiplication remains in the floating domain. Table 4 in our ablation study demonstrates that significant performance improvements can be achieved through shift operations alone.

**A1-2. ASIC/FPGA Area/Latency comparison**

> We conducted experiments using the structure of a batch-norm fused systolic array from an NPU developed for edge devices to compare overhead and latency in ASICs. The results below show the area of the batch-norm fused systolic array when each method is implemented in ASIC. The synthesis was done using the TSMC 12nm process. The MAC has a 16x16 structure, and IOSO, batch norm, and activation can process 8 data in 1 cycle. The sequence is MAC(->IOSO)-> BN -> ACT. Please refer to the "Common Response from Authors." Due to time and resource constraints, the actual performance time, or latency, was tested only on the resnet 50 layer with 28x28x128 feature 3x3x128x128 kernel layer.

|  | **integer Quant.** | **+IOSO** | **FP16 quant** |
|---|---:|---:|---:|
| **Total Area(um^2)** | 429,342 | 439,762 | 882,233 |
| **Latency(us)** | 1129.2775 | 1129.2825 | 1129.3150 |

> As seen in the table above, IOSO can be implemented with an integer shifter adder without modifying the MAC structure, resulting in only a 2.37% increase in area. In addition, the implementation is very simple as it does not alter the existing data path. In contrast, methods partially implemented with FP16 points (like GPTQ[10], QLoRA[9], LLM.int8()[11]) for increased accuracy and reduced memory are useful in GPUs that already have FP16 point kernels. However, from an ASIC perspective, these are challenging to use due to significant increases in MAC's area and power. The latency results are similar across all three methods because all data operates in a pipeline. The speed is comparable if the number of MACs is the same. The additional logic slightly alters the pipeline length, causing a minor increase in latency, but it's negligible compared to the overall data.

Reviewer comment (Experiments):
Similar to Weakness.1, shifting and scaling the output per channel is similar to finetune/update the BatchNorm statistics after the quantized convolutions. Does this method still work if we finetune BN after quantization on Conv-BN networks? (This method is used widely to recover accuracy after quantization.)

Author response:
**Q2. The authors only explained the motivation of channel-wise scaling/offset. But the difference between channel-wise scaling/offset and finetuning BatchNormalization is not discussed. It seems to me that these two methods are the same.**

Methodologically, I agree with the reviewer's concern that output channel-wise scaling/offset can be achieved through batch-norm fine-tuning. Our *output-channel-wise scaling/offset* essentially adopts the parameters used in batch normalization and applies input scaling and rounding error-aware fine-tuning. And from the perspective of applying the method, we have presented the following perspectives:

**1. [With] When used in conjunction with Input Channel-wise scaling and rounding-based quantization, better results can be obtained (Chapters 3.2 and 4.2).**

> Input Channel scaling alone was insufficient to prevent bias from the input channel. 
From Equation (6), it becomes  
$z_k= \gamma^z_k \alpha_k\sum^g_p\gamma^G_p\sum_{i \in G_p}w_{ki}x_{i}+\alpha_k\sum_{i }^{c}\varphi^y_i +\gamma_k^z\beta_k+\varphi^z_k$.
The presence of the $\alpha_k\sum_{i }^{c}\varphi^y_i$(*input channel bias*) necessitates additional floating addition operations. 
When only Input Channel scaling is applied, it cannot prevent bias arising from the input channel/rounding. Therefore, to reduce this, we incorporate the role in output-wise offset and scaling to mitigate these effects.

**2. [How] Applied to reduce *Block-Based* reconstruction error (Chapter 3.4).**
>While focusing on batch-norm finetuning could be effective layer-wisely and applicable in reducing the final loss, we found the scale and offset that further reduce loss at the block level. For instance, when applying rounding/input channel-wise scaling at the block level and output channel scaling/offset at the layer level to reduce reconstruction error, there was a performance decrease of 66.62 (-0.60%) in resnet18 at 2/32 bit.

**3. [Where] The placement of output scaling/offset (before/after activation) - (Appendix D.7).**
> It's possible to apply scaling/offset after activation, not necessarily at the current batch-norm position, but we found that applying it at the conventional batch-norm position generally yields slightly better performance.

**4. [Extension] Additionally, in models like transformers that do not use batch normalization, the performance can be enhanced with just 1 add/mult floating operation per activation (Chapter 3.4).**

Reviewer comment (Evaluation):
Is the shift operation expected to introduce more latency than a single integer operation due to non-local memory access at inference?

Author response:
**Q3. Shift operation overhead.**
> We can consider the scenarios for GPUs and ASICs separately. In the case of GPUs, shift operations need to be performed on accumulated values to be processed similarly to adders(accumulators). For ASICs, there are two possibilities. When using a fixed $\gamma_y$, the operation can be implemented through wire connections, eliminating timing issues. If the number of shifts in gamma_y is variable, it can be implemented with a multiplexer (mux) for each required shift. In this context, as demonstrated in Experiment Figure 6, values of N up to 6 are meaningful, which means only a small number of multiplexers are needed.

Reviewer comment (Experiments):
Lack of real HW results. It's clear that the author thoroughly considered the potential limitations if the proposed method was to be implemented. However, there are still some potential concerns, such as channel permutation's impact on computation efficiency and grouping fragments effect. A few examples on a representative HW would make the paper much stronger and convincing.

Author response:
**A1-1. Our Objective**

> For hardware, especially ASICs/FPGAs, to operate with low power and low area, the most computation-intensive Multiply-Accumulate (MAC) operations must be performed using integer calculations. To achieve this, weights and activations must be quantized into the smallest possible bit integers. We believe our method, compared to previously studied round-only based Post-Training Quantization (PTQ), offers an option for better accuracy with no additional cost for Output channel-wise (utilizing batch-norm) quantization and low hardware cost for Input Channel-wise operations, enhancing expressiveness.

**A1-2. ASIC/FPGA Area/Latency comparison**

> We conducted experiments using the structure of a batch-norm fused systolic array from an NPU developed for edge devices to compare overhead and latency in ASICs. The results below show the area of the batch-norm fused systolic array when each method is implemented in ASIC. The synthesis was done using the TSMC 12nm process. The MAC has a 16x16 structure, and IOSO, batch norm, and activation can process 8 data in 1 cycle. The sequence is MAC(->IOSO)-> BN -> ACT. For more detailed information, please refer to the "Common Response from Authors." Due to time and resource constraints, the actual performance time, or latency, was tested only on the resnet 50 layer with 28x28x128 feature 3x3x128x128 kernel layer.

|  | **integer Quant.** | **+IOSO** | **FP16 quant** |
|---|---:|---:|---:|
| **Total Area(um^2)** | 429,342 | 439,762 | 882,233 |
| **Latency(us)** | 1129.2775 | 1129.2825 | 1129.3150 |

> As seen in the table above, IOSO can be implemented with an integer shifter adder without modifying the MAC structure, resulting in only a 2.37% increase in area. In addition, the implementation is very simple as it does not alter the existing data path. In contrast, methods partially implemented with FP16 points (like GPTQ[10], QLoRA[9], LLM.int8()[11]) for increased accuracy and reduced memory are useful in GPUs that already have FP16 point kernels. However, from an ASIC perspective, these are challenging to use due to significant increases in MAC's area and power. The latency results are similar across all three methods because all data operates in a pipeline. The speed is comparable if the number of MACs is the same. The additional logic slightly alters the pipeline length, causing a minor increase in latency, but it's negligible compared to the overall data.

**A1-3. IOSO overhead on GPU**

|  | **Baseline(ms)** | **IOSO(ms)** | **Increased latency** |
|:---|---:|---:|---:|
| **resnet18** | 15.37±1.37 | 17.13±1.66 | 111.5% |
| **resnet50** | 38.67±0.83 | 42.26±1.58 | 108.1% |
| **mobilenet_v2** | 29.94±2.74 | 34.53±2.87 | 117.3% |
| **regnetx_600m** | 36.29±3.65 | 40.11±1.77 | 110.8% |

> We indirectly tested the latency using a non-optimized (naive) GPU. After adding the add and shift operations to the output feature in the baseline and running the ImageNet validation set with a batch size of 64, we observed the latency results. We used the RTX3080 GPU. On the non-optimized GPU, there was about a max of 17% increase in latency. We expect this value to represent the upper bound of our method's latency.  The latency could further decrease if there is support for a CUDA kernel for IOSO or operations after the accumulator, such as shift-add.

Reviewer comment (Theory):
Eq.9 and the explanations in the following paragraph shows that the group scaling factor, gamma_y, is a linear combination of preset gamm_G based on probability. However, that will likely make gamma_y not compatible with bit shifter. In AdaRound, the handling of h(V) is different during calibration/PTQ stage and inference stage. Author might want to clarify/comment on this part or it might cause confusion to the readers.

Author response:
**Q3.  Clarify group scaling factor.**
> The value of $\gamma_y$, as depicted in Figure 3, is predetermined as a hyper-parameter (we used values $1.0$, $1.0-2^{−4}$, and $1.0+2^{−4}$) and is determined by the calibration/PTQ stage ratio. However, these values tend to converge towards one side during calibration, allowing them to function as shift operations at inference. The variable corresponding to $h(V)$ is $s(R_{ij})$. I will offer a more comprehensive explanation of the concept of $s(R_{ij})$ to enhance clarity in the rebuttal.

Reviewer comment (Evaluation):
Based on the example in Fig. 6, the 3 scaling factors used are 1 and 1+-2^-N. One may interpret this scheme as some input channels will be up-weighted, some will be down-weighted, and the remaining channels will be unchanged. However, the optimized factor here seems to be very small, implying that simply not applying the scaling factor might work as well? But comparing to Table 4, it seems like the case without input scaling will be close to the case of N=1? Please comment.

Author response:
**Q4. Scaling factor applying**
> As commented by the reviewer, some channels increase, others decrease, and the rest remain unchanged. However, when $N=\infty$, channel scaling has no effect, similar to not applying any scaling at all, whereas at $N=1$, it has the most significant impact (50%, 100%, 150% scaling). As observed in Fig 6., excessive scaling ($N=1$) degrades performance, while moderate Group Gamma $N$ ($N=4$ for Weight Only, $N=6$ for Weight+Feature) yields the best results, outperforming very small scaling ($N=7$). Increasing $N$ gradually results in outcomes similar to those in Table 4's output scaling/offset case.

Reviewer comment (Evaluation):
Instead of using unstructured input channel-wise grouping, another frequently used quantization scheme is structured grouping, such as used by GPTQ and other LLM works. Maybe the author could consider including a few comments on the pros and cons with the proposed method?

Author response:
**Q5. Compare with other input channel-wise grouping**
> Previous works on group-wise quantization require each group to maintain different scale values, which, as indicated in QLoRA[9], become a significant memory overhead during low-bit quantization. For instance, quantizing weights for 64 groups with a single scale per group necessitates 32-bit scales for each of the 64 weights. This is akin to each weight carrying an overhead of 0.5 bits of memory. While the impact is less significant in larger bit quantizations, it becomes comparable to a 25% memory increase for 2-bit quantization.

> Instead of creating groups based on unrelated weight positions, our method undergoes a calibration stage to find the optimal group for predetermined scale values. Since shuffled channels are independent, they can be reorganized through permutation, eliminating the need for additional scale information memory. Furthermore, we opted for shift operations to stay within the integer domain to avoid requiring floating point multipliers and adders if the scale multiplication remains in the floating domain. Table 4 in our ablation study demonstrates that significant performance improvements can be achieved through shift operations alone.

> Similar to what was mentioned in A1-2, methods like GPTQ and LLM.int8() keep part or all of the activations in a floating point to reduce memory usage. While this is an effective approach for GPUs, where power and area are not major constraints, it poses a problem for edge devices requiring low power and small-area implementation. This distinction highlights the importance of considering the specific hardware environment when choosing a quantization strategy, as techniques suitable for high-resource settings may not be feasible or efficient in more constrained environments like edge devices.

Reviewer comment (Evaluation):
There is no discussion on why the proposed method is considered hardware-friendly. Which hardware is this work referring to? How its efficiency was measured?

Author response:
Thank you for taking the time to review our paper thoroughly and for providing valuable comments and feedback. In response to your questions, below are detailed answers to each one. We hope you find them helpful.

**Q1. Why is it  Hardware-friendly?**

**A1-1. Our Objective**

> For hardware, especially ASICs/FPGAs, to operate with low power and low area, the most computation-intensive Multiply-Accumulate (MAC) operations must be performed using integer calculations. To achieve this, weights and activations must be quantized into the smallest possible bit integers. We believe our method, compared to previously studied round-only based Post-Training Quantization (PTQ), offers an option for better accuracy with no additional cost for Output channel-wise (utilizing batch-norm) quantization and low hardware cost for Input Channel-wise operations, enhancing expressiveness.

**A1-2. ASIC/FPGA Area/Latency comparison**

> We conducted experiments using the structure of a batch-norm fused systolic array from an NPU developed for edge devices to compare overhead and latency in ASICs. The results below show the area of the batch-norm fused systolic array when each method is implemented in ASIC. The synthesis was done using the TSMC 12nm process. The MAC has a 16x16 structure, and IOSO, batch norm, and activation can process 8 data in 1 cycle. The sequence is MAC(->IOSO)-> BN -> ACT. For more detailed information, please refer to the "Common Response from Authors." Due to time and resource constraints, the actual performance time, or latency, was tested only on the resnet 50 layer with 28x28x128 feature 3x3x128x128 kernel layer.

|  | **integer Quant.** | **+IOSO** | **FP16 quant** |
|---|---:|---:|---:|
| **Total Area(um^2)** | 429,342 | 439,762 | 882,233 |
| **Latency(us)** | 1129.2775 | 1129.2825 | 1129.3150 |

> As seen in the table above, IOSO can be implemented with an integer shifter adder without modifying the MAC structure, resulting in only a **2.37%** increase in area. In addition, the implementation is very simple as it does not alter the existing data path. In contrast, methods partially implemented with FP16 points (like QLoRA[9], GPTQ[10], and LLM.int8()[11]) for increased accuracy and reduced memory are useful in GPUs that already have FP16 point kernels. However, from an ASIC perspective, these are challenging to use due to significant increases in MAC's area and power. The latency results are similar across all three methods because all data operates in a pipeline. The speed is comparable if the number of MACs is the same. The additional logic slightly alters the pipeline length, causing a minor increase in latency, but it's negligible compared to the overall data.

**A1-3. IOSO overhead on GPU**
|  | **Baseline(ms)** | **IOSO(ms)** | **Increased latency** |
|:---|---:|---:|---:|
| **resnet18** | 15.37±1.37 | 17.13±1.66 | 111.5% |
| **resnet50** | 38.67±0.83 | 42.26±1.58 | 108.1% |
| **mobilenet_v2** | 29.94±2.74 | 34.53±2.87 | 117.3% |
| **regnetx_600m** | 36.29±3.65 | 40.11±1.77 | 110.8% |

> We indirectly tested the latency using a non-optimized (naive) GPU. After adding the add and shift operations to the output feature in the baseline and running the ImageNet validation set with a batch size of 64, we observed the latency results. We used the RTX3080 GPU. There was about a max 17% increase in latency on the non-optimized GPU. We expect this value to represent the upper bound of our method's latency. The latency could further decrease if there is support for a CUDA kernel for IOSO or operations after the accumulator, such as shift-add.

Reviewer comment (Experiments):
Have you considered fine-tuning your results by doing some limited amount of training, say one epoch? It will be interesting to see if the accuracy improves significantly.

Author response:
Our method involves finding appropriate rounding values while simultaneously performing grouping and scaling/offset. We couldn't achieve favorable results with just one epoch (1024 samples). Also, fixing the rounding and performing grouping later did not yield satisfactory outcomes.
Additionally, rounding and grouping use a temperature hyperparameter, and the importance of each determines the specific group/rounding. Therefore, a certain number of epochs are necessary. In the presented results, we used 35,000 cycles (about 2187 epochs) and found that at least 15,000 cycles (about 937 epochs) are required to achieve satisfactory results.

Reviewer comment (Experiments):
Can your method be extended to cover training?

Author response:
Based on my understanding of the reviewer's question, it seems to refer not to post-training quantization but to the use of our method in normal training. We assumed training minimizes changes to the given pre-trained weights within a limited dataset (small calibration set). Therefore, we calibrated the network using rounding and scaling/offset.
Applying our method to normal training without quantization seems meaningless. If the optimal point requires specific channel scaling/biasing, training the weights in that direction would be more appropriate.
However, the method might be meaningful in quantization-aware training (QAT) in low-bit quantization. It can enhance the representational power of quantized weights/activation through scaling factors and offsets.