Paper: Long-Tailed Recognition on Binary Networks by Calibrating A Pre-trained Model (ICLR-2024)

Reviewer comment (Writing):
There are some instances where the writing or logical flow could be improved.

Author response:
Thank you for the valuable suggestions to improve 2.1 2.2, 2.3 and 2.4. We will change them accordingly in the revision. 

For your question about Sec. 2.5: Multi-resolution training can be understood as data augmentation where we explicitly obtain more training data by using differently resized images. This is particularly beneficial for our problem of binary LT as binary LT is more prone to the data-scarcity in long-tailed classes (please refer to our answer regarding the question on special difficulties of binary LT ).

Reviewer comment (Presentation):
Regarding section 3.3, it appears that multi-resolution learning is primarily for efficient training. Would it be more fitting to discuss this within the implementation of experiments? Alternatively, is it integral to the overall framework's effectiveness?

Author response:
We agree that the implementation of the multi-resolution could be discussed within the experiment section. Rather, we should provide the motivation for the multi-scale training in the approach instead, as described in our second answer to the question regarding flow improvement in Sec. 2.5. While it is not integral to our approach, it certainly does help when we can utilize it (please see Tab 3 for ablation results).

Reviewer comment (Experiments):
The scalability of this approach raises concerns, given its utilization of a teacher model of substantial size. Notably, in Table 2, only single-resolution is utilized, primarily due to VRAM limits.

Author response:
$\to$ We cordially argue that the scalability concern is not critical for multiple reasons. First, we can simply calculate the teacher model’s output on the dataset beforehand which will substantially reduce VRAM overhead. Second, the multi-scale training is not integral to our calibrate and distill framework. If we did not have enough VRAM to utilize it, we could simply forgo it and still obtain good performance. Last, at the rates in which VRAM size is advancing with today’s hardware improvement, we believe that the size of our teacher models are not that limiting with regards to VRAM.

Reviewer comment (Experiments):
The incorporation of a non-LT pretrained network in advance does not appear to be a practical approach. One of the underlying motivations for addressing the long-tailed recognition problem stems from the inherent imbalances frequently encountered in natural datasets.

Author response:
$\to$ As great efforts have been devoted into curating publicly available large-scale datasets that are balanced such as ImageNet-1k and ImageNet-21K, we argue that obtaining non-LT pretrained models in advance is indeed not difficult, $i.e.*, quite practical. We also note that we do **NOT** assume that a pretrained model is trained on either LT or non-LT distributions. Instead, we utilize a cheap calibration process (training only a linear classifier) for the pretrained model, which makes it possible to utilize the pretrained model as a distillation teacher no matter what the pretrained data distribution was. Empirically, the calibration noticeably improves the performance by more than 10% compared to using a non-calibrated teacher (please see Tab 3 for the ablation studies).

Reviewer comment (Experiments):
It is advisable to update the set of compared methods in the experimental evaluation, as the current selection appears somewhat outdated. A more comprehensive comparison with contemporary methodologies is suggested.

Author response:
$\to$ We will definitely update them for the final version. But we believe that the strong performance of CANDLE,  as noted by other reviewers (udmW, Hnun, HnuF) as well,  is enough to provide empirical validations..

Reviewer comment (Evaluation):
The mechanism by which the parameter $\lambda_\phi$ facilitates generalization across multiple data distributions remains unclear. In Figure 6, it is evident that, across all datasets, $\lambda_\phi$ consistently converges to a value of 1, indicating that towards the conclusion of the training process, improvements are predominantly focused on the classifier rather than the encoder. If the variation in the scheduling of $\lambda_\phi$ is considered essential for enhancing generalization, a more thorough explanation is needed.

Author response:
$\to$ As $\lambda_\phi$ approaches the value of 1, the binary encoder is primarily trained with the supervision from the pretrained model’s feature extractor (please see Eq. 4), which is inline with the reviewer’s comment and also mentioned in Sec. 4.3 of the paper.  However, as pointed out, the scheduling of the $\lambda_\phi$ is important for generalization as it varies substantially from dataset to dataset. Our formulation of $\lambda_\phi$ allows a schedule to be learned solely by the target LT data and not by humans. This makes it applicable to multiple target LT datasets without the need for human intervention.

Reviewer comment (Experiments):
How the large teacher model can effectively contribute to the performance of the binary network, particularly when a substantial domain gap exists between the dataset used for pretraining, such as ImageNet, and the actual target dataset, which could be, for instance, an MRI dataset in practical applications.

Author response:
$\to$ Thank you for the interesting question. Currently, the teacher models with the size as used in our work perform well for addressing domain gaps such as from ImageNet to Places dataset. While we are also curious about the effect of teacher model size on the performance of addressing domain gaps, the experimental undertaking is not trivial for two reasons. First, it is difficult to quantify the domain gap, which makes plotting the teacher size vs its effect on addressing the domain gap hard. Second, isolating the effect of teacher size scaling is hard as the architectural differences of differently sized models also have an effect. Thus, we leave this as an interesting future research avenue.

Meanwhile, to better address larger domain gaps, we believe that improving the calibration process to better handle the large domain gaps given the same model size budget also could be another interesting future work (*i.e.*, gaining hints from the transfer learning and domain adaptation literature).