Paper: Fisher-aware Quantization for DETR Detectors with Critical-category Objectives (ICLR-2024)

Reviewer comment (Novelty):
My major concern w.r.t. this paper is that, while I agree on the importance of the proposed problem that we should particularly focus on certain classes instead of all classes, I feel that this can be achieved in many ways. For example, during training, one can particularly control the loss weight for certain particular classes. Moreover, during quantization, one can perform input-aware bit assignment [1] to set higher bit for images in which critical classes are hardly distinguishable and lower bit for other images. Thus, while I admit that the Fisher information can be a feasiable perspective, I hope that the authors can discuss more on its advantages over other seems feasible ways.

Author response:
A1: Thank you for bringing up these possible methods. As we do not claim Fisher-aware quantization to be the only way of improving the fine-grained performance of the quantized model, here we discuss the advantage of our method over other alternatives. 
 
For adjusting the weight of fine-grained losses, we have compared our proposed scheme with performing regular QAT with the summation of overall and critical objectives in Tab. 12 of Appendix D. For the Fisher-trace regularization, the motivation comes from our Insight 2 in Sec. 4.1, where the QAT gap is caused by the sharp loss landscape leading to a poor generalization. This cannot be simply resolved with the addition of critical objectives in the training loss, as illustrated by the results in Tab. 12, where the summed loss does not consistently lead to critical performance improvements.

As for the Instance-aware dynamic quantization method, it may not bring real efficiency when deployed on a hardware system. All the weights need to be stored at the highest possible precision, and the switching of precision at inference time will bring additional overhead. Furthermore, multiple objects with different classes are likely to exist in the same scene for object detection tasks, which brings further challenges for instance-aware methods on deciding the quantization scheme to use. Our proposed method works with a static quantization scheme for all the inputs, which does not suffer from the aforementioned drawbacks.

Reviewer comment (Evaluation):
The last concern I have is w.r.t. the experiment. Specifially, from my perspective, there can exist a mismatch between the experiment and the claim of this paper. In other words, the improvement of Fisher-Critical over Fisher-Overall or even Uniform is often limited. This made me have the feeling that, this paper fails to convince me that it proposes a good method for solving the "the fine-grained critical-category performance" problem.

Author response:
A3: We would like to note that both the Fisher-aware quantization scheme (both overall and critical) and the Fisher-trace regularization on the DETR model are our contributions derived from our theoretical analysis in Sec. 3. When combined, they lead to an improved critical mAP performance over uniform quantization baselines by up to 1.15% on 4-bit DETR-R50 on the COCO dataset, and up to **10.4%** on COCO-Panoptic where the critical performance is hindered more due to the large number of classes (Sec. 5.3).

Also, we would like to argue that the significance of the empirical improvements should not be solely determined by the value itself. Our results of the Fisher-critical quantization scheme are significant in the sense that 1) the improvements are mostly 3x larger than the standard error induced by the training randomness, and 2) the improvement is consistent across different precisions, architectures, and datasets. This verifies the improvements brought by our proposed method are significant and not random. Furthermore, we find the improvement of our method is higher when applied to more complicated architectures. Critical mAP improves by up to 0.5% on DETR-R101, 0.8% on DAB DETR-R50, and 0.4% on Deformable DETR-R50 with Fisher-critical over Fisher-overall, compared to the 0.2% improvement on DETR-R50. This further shows the potential of the proposed method as larger models may require more compression. 

Besides the observable improvements brought by our method, we would like to emphasize that this paper is the first to investigate the impact of quantization on the fine-grained critical-category performance of DETR-based object detectors, which is a practical setting yet largely overlooked by the community. As we take an initial step to tackle this challenge in this work, it will motivate future explorations on the fine-grained impacts of other compression methods in the computer vision area and general machine learning research.

Reviewer comment (Novelty):
Are you the first to conduct this research? As far as I know Q-DETR （CVPR2023) is the first paper about QUANTIZATION FOR DETR. However, your research contents are different. Q-DETR focuses on how to enhance the performance of quantized DETRs. You focus on CRITICAL-CATEGORY OBJECTIVES. I think you should cite Q-DETR.

Author response:
We thank reviewer ka68 for your thorough review and valuable comments. Hopefully, the following responses can address your concerns.

Thank you for bringing up the Q-DETR paper to our attention. We acknowledge the contribution of Q-DETR as a pioneer work in designing better QAT objectives and algorithms for DETR. We are happy to cite the paper in our related work section. 

As you have clearly identified, Q-DETR and our paper work on different aspects of DETR quantization. Q-DETR focuses on better distillation objectives to improve the overall DETR performance in QAT. In contrast, our work proposes a Fisher-aware mixed-precision quantization scheme design and a Fisher trace regularization, with a special focus on improving critical-category objectives. In this sense, the contributions of both papers are orthogonal to each other. Following your suggestion, we have included an experiment of training DETR models quantized with different mixed-precision quantization schemes using the Q-DETR distillation objective. Due to the limitation in time, we ran all experiments with the DETR-R50 model on the COCO dataset, with **a total of 50 epochs using a 1e-5 learning rate**. All the other settings are the same as described in Sec. 5.1. Though the reported performance here is lower than that of the Q-DETR paper due to the 10x less training epochs, we believe the trend in relative performance remains if longer training is conducted.

|   Model  |   Quant scheme  | Overall 4-bit mAP | Person 4-bit mAP | Animal 4-bit mAP |
|:--------:|:---------------:|-------------------|:----------------:|:----------------:|
| DETR-R50 |     Uniform     |  34.85 $\pm$ 0.04 | 33.00 $\pm$ 0.04 | 35.53 $\pm$ 0.03 |
|          |  Fisher-Overall |  **35.10** $\pm$ 0.12 | 33.47 $\pm$ 0.09 | 36.08 $\pm$ 0.06 |
|          | Fisher-Critical |  35.04 $\pm$ 0.11 | **33.72** $\pm$ 0.07 | **36.43** $\pm$ 0.04 |

The results are consistent with our findings in Tab. 3 that the proposed Fisher-overall quantization scheme improves over uniform quantization on the overall model performance, whereas the Fisher-critical quantization scheme further improves the corresponding critical performance. This result verifies that the disparate impact on critical performance exists for different QAT objectives, while our method is generalizable to other QAT objectives.

Reviewer comment (Experiments):
I think you should compare it to Q-DETR. Q-DETR's model has been released on Git Hub. Although you are different, I think you are comparable.

Author response:
We thank reviewer ka68 for your thorough review and valuable comments. Hopefully, the following responses can address your concerns.

Thank you for bringing up the Q-DETR paper to our attention. We acknowledge the contribution of Q-DETR as a pioneer work in designing better QAT objectives and algorithms for DETR. We are happy to cite the paper in our related work section. 

As you have clearly identified, Q-DETR and our paper work on different aspects of DETR quantization. Q-DETR focuses on better distillation objectives to improve the overall DETR performance in QAT. In contrast, our work proposes a Fisher-aware mixed-precision quantization scheme design and a Fisher trace regularization, with a special focus on improving critical-category objectives. In this sense, the contributions of both papers are orthogonal to each other. Following your suggestion, we have included an experiment of training DETR models quantized with different mixed-precision quantization schemes using the Q-DETR distillation objective. Due to the limitation in time, we ran all experiments with the DETR-R50 model on the COCO dataset, with **a total of 50 epochs using a 1e-5 learning rate**. All the other settings are the same as described in Sec. 5.1. Though the reported performance here is lower than that of the Q-DETR paper due to the 10x less training epochs, we believe the trend in relative performance remains if longer training is conducted.

|   Model  |   Quant scheme  | Overall 4-bit mAP | Person 4-bit mAP | Animal 4-bit mAP |
|:--------:|:---------------:|-------------------|:----------------:|:----------------:|
| DETR-R50 |     Uniform     |  34.85 $\pm$ 0.04 | 33.00 $\pm$ 0.04 | 35.53 $\pm$ 0.03 |
|          |  Fisher-Overall |  **35.10** $\pm$ 0.12 | 33.47 $\pm$ 0.09 | 36.08 $\pm$ 0.06 |
|          | Fisher-Critical |  35.04 $\pm$ 0.11 | **33.72** $\pm$ 0.07 | **36.43** $\pm$ 0.04 |

The results are consistent with our findings in Tab. 3 that the proposed Fisher-overall quantization scheme improves over uniform quantization on the overall model performance, whereas the Fisher-critical quantization scheme further improves the corresponding critical performance. This result verifies that the disparate impact on critical performance exists for different QAT objectives, while our method is generalizable to other QAT objectives.

Reviewer comment (Evaluation):
The performance improvements are marginal, as shown in Tab. 3 and Tab. 4.

Author response:
We thank reviewer WkhP for your thorough review and valuable comments. Hopefully, the following responses can address your concerns.

We would like to note that both the Fisher-aware quantization scheme (both overall and critical) and the Fisher-trace regularization on the DETR model are our contributions derived from our theoretical analysis in Sec. 3. When combined, they lead to an improved critical mAP performance over uniform quantization baselines by up to 1.15% on 4-bit DETR-R50 on the COCO dataset, and up to **10.4%** on COCO-Panoptic where the critical performance is hindered more due to the large number of classes (Sec. 5.3).

Also, we would like to argue that the significance of the empirical improvements should not be solely determined by the value itself. Our results of the Fisher-critical quantization scheme are significant in the sense that 1) the improvements are mostly 3x larger than the standard error induced by the training randomness, and 2) the improvement is consistent across different precisions, architectures, and datasets. This verifies the improvements brought by our proposed method are significant and not random. Furthermore, we find the improvement of our method is higher when applied to more complicated architectures. Critical mAP improves by up to 0.5% on DETR-R101, 0.8% on DAB DETR-R50, and 0.4% on Deformable DETR-R50 with Fisher-critical over Fisher-overall, compared to the 0.2% improvement on DETR-R50. This further shows the potential of the proposed method as larger models may require more compression. 

Besides the observable improvements brought by our method, we would like to emphasize that this paper is the first to investigate the impact of quantization on the fine-grained critical-category performance of DETR-based object detectors, which is a practical setting yet largely overlooked by the community. As we take an initial step to tackle this challenge in this work, it will motivate future explorations on the fine-grained impacts of other compression methods in the computer vision area and general machine learning research.