Paper: Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks (ICLR-2024)

Reviewer comment (Writing):
It would be better if the authors can provide more details about Gradient conflict ratio.

Author response:
Thanks for your helpful suggestion. Upon your advice, we updated the manuscript to add more details about the gradient conflict ratio.
In the manuscript, we detailed how the gradient conflict ratio is measured: the ratio of parameters that the sign of gradient from reconstruction loss and that of regularization is different.
Also, we additionally emphasized that, according to Figure 2 of the main paper, the conflict ratio decreases minimally during training, which indicates that variance regularization can hinder the reconstruction loss throughout training.
Moreover, instead of disregarding the gradient of the regularization when the two gradient conflicts, we added analysis for weighting the regularization loss by the degree of conflict between two losses (measured by the cosine similarity).
According to the results, it was slightly better to disregard the variance regularization loss when it is not cooperative.

Table R3: **Analysis on the degree of gradient conflict** on EDSR x4 (2-bit). cos() measures cosine similarity.
| Method    | Set5  | Set14 | B100  |Urban100|
|:-------------|:-----:|:-----:|:-----:|:------:|
| VR disregarded when $\nabla_\theta L_R \cdot \nabla_\theta L_V <0$ | 31.49 | 28.12 | 27.26 | 25.15  |
| VR weighted with cos($\nabla_\theta L_R, \nabla_\theta L_V$)    | 31.45 | 28.12 | 27.25 | 25.14  |

Reviewer comment (Theory):
In addition to the variance regularization, the selective distribution offsets also employ a learnable parameter about the standard deviation of the features. It would be better if the author could provide more discussions about the relation of these two terms.

Author response:
Thank you for your advice. As the reviewer mentioned, variance regularization and selective distribution offsets both function to reduce the distribution mismatch (feature-wise std).
Each component largely reduces the mismatch and results in higher accuracy over the baseline.
When the two components are jointly used, the accuracy increases compared to using each component separately, although the increasing amount is relatively minor.
This indicates that there is, to some extent, an overlapping effect of selective offsets and variance regularization. Nevertheless, both components contribute to reducing the mismatch, resulting in a further accurate quantized SR network.
We added the discussion to the manuscript.

Reviewer comment (Reproducibility):
Does the selective distribution offsets are learned on the features not processed in cooperative variance regularization?

Author response:
We would like to clarify that selective distribution offsets are also processed through cooperative variance regularization. The parameters of the quantized network (including the selective distribution offsets) are end-to-end trained. We updated the manuscript followingly.

Reviewer comment (Experiments):
The main motivation is that feature distributions of SR networks are significantly divergent for each channel or input image. In Figure 1, does the distribution mismatch only occur in the SR network? Does such a distribution mismatch occur in other networks? In addition, could you show the distribution after quantization?

Author response:
**Q1-1: Does the distribution mismatch occur only in SR or other networks?** \
**A1-1:** 
The distribution mismatch problem is especially severe for SR networks. For example, a classification network (ResNet20) shows a much more minor image-wise and channel-wise distribution mismatch compared to SR networks (EDSR, RDN). We measure the average variance of features on DIV2K validation set for SR networks and ImageNet validation set for the classification network. Thanks for your comments, we added such analyses to the supplementary material.

Table R2: **Average feature mismatch**.
| Model | Image-wise Variance  | Chanel-wise Variance |
| ---------|:-----:|:-----:|
| EDSR (x4) | 15.08 | 40.29 |
| RDN (x4) | 6.40 | 58.14  |
| ResNet-20 | 0.04 | 0.09  |

\
**Q1-2: The distribution after quantization** \
**A1-2:** 
Thanks for your suggestion. Based on your comments, we added new figures to the supplementary material of distribution after quantization and the distribution after using our framework, ODM. The distribution of the ODM-applied network shows that ODM is effective in reducing the distribution mismatch.

Reviewer comment (Experiments):
In the experiments, the authors mainly use EDSR, EDN and SRResNet. However, these methods are very old. Could you compare the new SOTA SR networks, e.g., SwinIR?

Author response:
**Q2: Comparison with more recent models (e.g., SwinIR)** \
**A2:** 
Please kindly refer to our supplementary material (Table S1), in which we reported results on more recent models SwinIR [A] and CARN [B]. According to the results, our method achieves consistent gain over existing methods also on recent models.

\
[A] SwinIR: Image Restoration Using Swin Transformer, CVPR2021. \
[B] Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network, ECCV2018.

Reviewer comment (Evaluation):
In Table 1, for Bit=2, the results of EDSR-DAQ do not correspond to the original results of the DAQ paper. Could you discuss these results?

Author response:
**Q3: EDSR-DAQ does not correspond to the original results of the DAQ paper** \
**A3:** 
We note that the result reported in the DAQ paper uses the EDSR backbone of 32 residual blocks (of 256 channel dimensions). In comparison, we use the EDSR-baseline backbone that consists of 16 residual blocks (of 64 channel dimensions), following PAMS and DDTB. We have reproduced EDSR-DAQ directly by the official codebase, which also matches the accuracy reported for EDSR-baseline in the official GitHub repository.

Reviewer comment (Experiments):
The experiments only address the scale of 4. It would be better to conduct more experiments on other scales and put the results in supplementary.

Author response:
**Q4: Experiments only on a scale of 4** \
**A4:** 
Also, kindly refer to our supplementary material (Table S2) for the results on scale 2 SR models. The results show that our method is also beneficial for scale 2 SR models.

Reviewer comment (Experiments):
In the ablation study, could you conduct an experiment with only the cooperative variance regularization? In addition, the network with only the selective distribution offsets is comparable with Coop.+Var. Reg.+Sel. Off. This result demonstrates that Coop.+Var. Reg. are not important.

Author response:
**Q5-1: Ablation only using the cooperative variance regularization** \
**A5-1:** 
If we understood correctly, Table 5 (b) presents the result of only using cooperative variance regularization.




**Q5-2: Ablation indicates that the coop. var. reg is not important** \
**A5-2:** 
We would like to emphasize that, compared to the baseline, using cooperative variance regularization (Coop. Var. Reg.) brings 0.8 dB gain in PSNR, which demonstrates that Coop. Var. Reg is important.
The two main components we propose (Coop. Var. Reg. and Sel. Off.) both serve to reduce the distribution mismatch in SR; using two components together gives a relatively minor gain compared to the gain from each component (0.09 dB / 0.18 dB).
However, this does not mean that each component is useless; each brings 0.8 dB / 0.9 dB gain over the baseline. It rather hints that the two attributes can have overlapping effects for reducing distribution mismatch.

Reviewer comment (Evaluation):
The comparisons seem not fair in terms of training epochs, and the proposed method does not reduce BitOPs compared to the previous method, i.e., DDTB, which contradicts the motivation of the method.

Author response:
**Q1-1: Fair comparison regarding training epochs** \
**A1-1:** 
We follow DDTB [A] to use 60 epochs, and for a fair comparison, we reproduce PAMS [B] and DAQ [C] for 60 epochs using the official codebase. 

**Q5: ODM does not reduce BitOPs compared to DDTB** \
**A5:** 
We would like to note that, although the bitOPs of ODM is similar to that of DDTB, ODM occupies a smaller storage size, which is also an important computational cost benefit.

Reviewer comment (Experiments):
The authors reproduce the results of other methods using the same training epochs. Does the number of epochs influent the performance of other methods? Why not use their optimal training epochs for comparisons?

Author response:
**Q1-2: Influence on the number of epochs** \
**A1-2:** 
If we set the training epochs to 300 (the epochs reported in DAQ), the accuracies of the overall methods increase. However, we note that the order is preserved, and ours outperforms other methods. 

Table R1: **Different training epochs** on 2-bit EDSR for Urban100 PSNR.
| Epochs | EDSR-PAMS  | EDSR-DAQ | EDSR-DDTB | EDSR-ODM (Ours) |
| -------------------------------|:-----:|:-----:|:-----:|:------:|
| 60   | 23.72 | 24.88 | 24.82 | 25.15  |
| 300 | 24.09 | 24.90 | 25.01 | 25.51  |

Reviewer comment (Evaluation):
Compared to competitors, the authors use seemingly complicated methods to address the quantizing problem beforehand. Will the proposed method increase the training time?

Author response:
**Q2: Comparison of training time** \
**A2:** 
Using a single RTX 2080Ti GPU, the calibration for the selective offsets takes $\sim$30 seconds. The overall training time for ODM is $\sim$4.0 hours for quantizing EDSR, which is no less than that of DDTB ($\sim$4.0 hours).

Reviewer comment (Evaluation):
We can observe from Tab. 4 that the proposed method achieves a better tradeoff between the storage size and BitOPs, to be precise. What makes the ODM need higher storage space than DAQ.

Author response:
**Q3: What makes ODM need higher storage space than DAQ?** \
**A3:** 
The additional storage space of ODM originates from the selective offset parameters. However, the storage overhead is relatively minimal compared to the significant bitOPs overhead of DAQ: as DAQ adaptively adjusts the quantization range parameters by calculating mean and variance at test-time, the bitOPs overhead is substantial.

Reviewer comment (Writing):
The verb is missing in the sentence after Eq.4.

Author response:
**Q4: Typo after Eq. (4)** \
**A4:** 
Thank you for pointing out. We have revised our manuscript.