Paper: Hybrid Classification-Regression Adaptive Loss for Dense Object Detection (ICLR-2024)

Reviewer comment (Experiments):
There are many parameters involved in the loss, such as theta, mu, alpha, ep, gamma, and l in EATSS, which makes the design complicated and has low generalization ability.

Author response:
While some parameters in our formula require extensive experimentation to obtain optimal parameters, the parameters of HCRAL exhibit strong generalization abilities, consistently delivering favorable results across multiple one-stage models. As demonstrated in Tables 1 to 5, we conducted separate searches for the optimal parameters of HCRAC and HCRAR on the Retinanet model, and then applied the same parameters to the ATSS and FCOS+ATSS single-stage classical models. Tables 1 to 4 show that the inclusion of HCRAC and HCRAR in the Retinanet model outperforms the baseline model (36.5 AP vs 37.6 AP, 36.5 AP vs 37.4 AP). In the ATSS model, the addition of HCRAC and HCRAR similarly enhances model performance (40.0 AP vs 40.4 AP, 40.0 AP vs 40.2 AP). Table 5 further highlights the generalization effectiveness of HCRAL in FCOS+ATSS (41.3 AP vs 42.1 AP).

Reviewer comment (Evaluation):
The EATSS seems to have a small influence on the results (41.3 vs. 41.2 AP).

Author response:
In Table 5, it can be observed that replacing EATSS with ATSS results in only a 0.1 AP increase for FL+GIoU. However, to verify the effectiveness of individual methods, we added them one by one and summarized the results in Table 6. When the additional samples for EATSS increased from 2 to 3, there was a 0.4 AP improvement. This indicates that EATSS is sensitive to HCRAL, confirming that our loss function has a significant optimization effect on samples with high scores and low IoU or low scores and high IoU. Moreover, the strategy of EATSS aligns well with HCRAL.

Reviewer comment (Experiments):
Can the method be applied to more advanced detectors, like the two-stage Faster RCNN, and transformer-based DETR? Since the detector architecture FCOS, ATSS, and RetinaNet is classical but old, the proposed method will be more solid if the loss is valid on the recent stronger detector.

Author response:
Two-stage models demonstrate superior performance due to their ability to filter out a large portion of the background in the first stage. However, this advantage comes at the cost of slower inference speeds compared to single-stage models. In recent years, Transformer-based models, such as DETR with a backbone-50, have shown promising performance. Still, they often require an extensive number of training epochs and face challenges in practical deployment.In domains like autonomous driving and pedestrian recognition, single-stage models continue to be widely used due to their fast inference speed and ease of deployment. However, their performance is hindered by issues such as class imbalance and smaller model sizes. In response to the limitations of single-stage models, we propose the RCI module to enable the model to adaptively optimize the classification and regression performance of samples and the CF module to focus on high-quality challenging samples.
| Model               | #epochs | AP   | AP50 | AP75 | APS  | APM  | APL  | 
|---------------------|---------|------|------|------|------|------|------|
| DETR-R50         | 500     | 42.0 | 62.4 | 44.2 | 20.5 | 45.8 | 61.1 |
| Anchor DETR-R50| 50      | 42.1 | 63.1 | 44.9 | 22.3 | 46.2 | 60.0 |
| Conditional DETR-R50| 50 | 40.9 | 61.8 | 43.3 | 20.8 | 44.6 | 59.2 |
| DAB-DETR-R50   | 50      | 42.2 | 63.1 | 44.7 | 21.5 | 45.7 | 60.3 |
| DN-DETR-R50    | 50      | 44.1 | 64.4 | 46.7 | 22.9 | 48.0 | 63.4 |
| HCRAL(ours)    | 24  | 44.4 | 62.2 | 48.6 | 26.9 | 47.6 | 55.5 |
| HCRAL(ours)  (with auxiliary modules)    | 24 | **45.3** | **62.8** | **49.4** | **27.7** | **48.5** | **56.1** |
Furthermore, we conducted a comparative analysis with the Transformer-based DETR, particularly when the backbone is ResNet-50. It can be observed that when the backbone is ResNet-50, HCRAL outperforms DETR with only 24 epochs, establishing itself as the current state-of-the-art single-stage model we are aware of.

Reviewer comment (Experiments):
The proposed methods introduce many hyper-parameters, which must be tuned carefully. This undermines the generality of the methods. The method designs are technically complicated, involving both loss functions and label assignment. I suggest the author simply the method.

Author response:
Our motivation in designing the loss function was to maximize the performance of the model by simultaneously considering the consistency between classification and regression tasks and focusing on challenging samples. Existing loss functions do not address this aspect. Based on the rules established in Sections 3.1 and 3.2, we introduced the RCI and CF modules. While these modules come with a certain parameter overhead, each serves a specific purpose. Extensive experimentation was conducted to determine the optimal parameters, yet the parameters of HCRAL demonstrate strong generalization.
As shown in Tables 1~5, we conducted separate searches for the best parameters of HCRAC and HCRAR on the Retinanet model, applying the same parameters to the ATSS and FCOS+ATSS one-stage classical models. The inclusion of HCRAC and HCRAR in the Retinanet outperformed the baseline model (36.5 AP vs 37.6 AP, 36.5 AP vs 37.4 AP). In the ATSS, the addition of HCRAC and HCRAR similarly improved the model's performance (40.0 AP vs 40.4 AP, 40.0 AP vs 40.2 AP). Table 5 further demonstrates the generalization effectiveness of HCRAL in FCOS+ATSS (41.3 AP vs 42.1 AP).
Our method, serving as a loss function, differs from label assignment. Label assignment is designed based on individual model frameworks and lacks universality.  Our loss function has undergone experimentation across multiple models and can be flexibly combined with other loss functions, outperforming all other existing loss functions.

Reviewer comment (Presentation):
The presentation of the experiment section is poor. What is the baseline in Tables 1, 2, 3 and 4? Also, Table 2 is weird since you report the results for various $\gamma$ in HCRAC, however, it is $\mu$. Where do you mention Table 5 in the context? It loses the connection. What is the baseline in Table 5? It is hard to believe that FCOS+ATSS got a 41.2 AP score.

Author response:
The parameters in Table 2 should be denoted as μ. As mentioned in Section 4.2,  the results in Table 5 were discussed. The baseline model, FCOS+ATSS, achieves an AP of 41.2 because we introduced auxiliary learning modules, as explained at the end of the introduction. This clarification has also been added in Section 4.2.

Reviewer comment (Theory):
The motivation is somewhat unclear. Why did you choose to design the three rules in Sec. 3.1.1 and Sec. 3.1.2?

Author response:
For classification
(a) If divide the classification gradients of samples into several sub-intervals, where difficult-to-train samples are distributed in intervals with fewer samples, resulting in larger penalties. However, within the same sub-interval, each sample has a different IoU. If samples with high IoU can receive higher penalties, the performance of the model is improved. 
(b) When the classification score approaches a higher IoU, the model should reduce the penalty for that particular sample.
(c) In negative samples, high IoU samples that are predicted incorrectly and fall into intervals with fewer samples can disrupt the model's original learning. For low IoU samples, despite prediction errors, substantial penalties are necessary to facilitate model learning.

For  regression, 
The first two points are easily understandable.
(c) For regression tasks, it is crucial to combine the sample's classification status to correctly impose penalties. In high-score, low-IoU samples, there are usually certain texture details, indicating greater regression potential. For low-score, high-IoU samples, even if regression accuracy continues to improve, the classification score remains challenging to elevate.

Reviewer comment (Evaluation):
The improvement is almost non-existent. More importantly, the reported SOTA results of previous methods are lower than the reference. As shown in Table 7, under the backbone R50, HCRAL only achieves 44.4 vs VFNet 44.8 [4] and GFLv2 44.3 [5]. Under the backbone R101, HCRAL 46.1 vs VFNet 46.7 and GFLv2 46.2. Under R101-DCN, HCRAL 49.3 vs VFNet 49.2.

Author response:
Comparing in this way is unfair; the results I mentioned include the effects of other loss functions with the addition of auxiliary learning modules. To make the comparison more equitable, we have supplemented the necessary experiments

Reviewer comment (Theory):
There is a mistake in the proposed method. In the last paragraph of page 5, "the coordinates are partitioned into two regions based on $y(Score) = x(IoU) + \alpha$. In region 1, where samples exhibit higher scores compared to IoU values..." Clearly, $\alpha<0$. The samples just between the red line and the yellow line do not exhibit higher scores compared to IoU values.

Author response:
In the latest manuscript, we have updated Eqn.10.

Reviewer comment (Writing):
The quality of the paper writing is low. It is difficult to understand the proposed method. Some variables are described before their formal appearance, e.g., "c is the diagonal length of the smallest enclosing box..." in Sec. 3.1.2.

Author response:
I sincerely apologize for any confusion caused by the writing errors.

Reviewer comment (Evaluation):
Why would ATSS 'omit certain samples characterized by high scores and high IoU that hold promise'? This isn't clear to me. Moreover, is this even an issue? I can see that the improvement is marginal.

Author response:
1. ATSS selects positive samples by calculating the mean and standard deviation of k candidate boxes around each layer of the target. On one hand, for larger-sized targets, effective anchor points are often located farther away from the target center. ATSS only considers k points from a specific layer to calculate the mean and standard deviation, potentially neglecting anchor points with good regression potential. On the other hand, for anchor points closer to the target center, the potential for classification is usually higher than regression potential. However, due to their IoU being lower than the calculated mean and standard deviation, these points may also be overlooked. We discussed this aspect in Section 3.2. To address this, we propose EATSS, which includes additional high-score low-IoU samples and low-score high-IoU samples. Moreover, existing industry-standard loss functions do not optimize for this aspect. From Table 5, it is observed that combining EATSS with Focus + GIoU increases performance by 0.1 AP. Furthermore, in Table 6, when the number of additional samples for EATSS increases from 2 to 3, there is a notable improvement of 0.4 AP. This sensitivity of EATSS to HCLAL parameters confirms that EATSS's strategy aligns well with HCRAL.

Reviewer comment (Experiments):
Why is Res2Net not applied to VFL? This isn't a fair comparison.

Author response:
3. In Table 7, we conducted experiments using ResNet-101 as the backbone for VFL, GFL, and HCRAL. To ensure fairness, we did not use their respective auxiliary modules (as seen in the third column). As shown in Table 7, the map50 of HCRAL reaches 46.1, whereas GFL and VFL achieve 45 AP and 44.9 AP, respectively, resulting in improvements of 1.1 AP and 1.2 AP

Reviewer comment (Evaluation):
Where do the quoted improvements of 1.1 and 1.2 mAP over VFL and GFL come from?

Author response:
3. In Table 7, we conducted experiments using ResNet-101 as the backbone for VFL, GFL, and HCRAL. To ensure fairness, we did not use their respective auxiliary modules (as seen in the third column). As shown in Table 7, the map50 of HCRAL reaches 46.1, whereas GFL and VFL achieve 45 AP and 44.9 AP, respectively, resulting in improvements of 1.1 AP and 1.2 AP

Reviewer comment (Novelty):
The innovation of this paper is limited, and the two parts proposed, HCRAL and EATSS, are relatively independent.

Author response:
1. The primary motivation behind the design of this loss function is to address situations where single-stage models encounter a higher number of negative samples and fewer positive samples, focusing on difficult-to-train samples and fostering interaction between classification and regression tasks. To validate the effectiveness of our proposed loss function in single-stage models, we consider three classic models: Retinanet, ATSS, and FCOS+ATSS. As shown in Tables 8 and 9, our designed HCRAC and HCRAR achieve optimal performance in Retinanet experiments (36.5 AP vs 37.6 AP, 36.5 AP vs 37.4 AP), and we apply the corresponding optimal parameters to ATSS, confirming the generalization of both classification and regression tasks (40.0 AP vs 40.4 AP, 40.0 AP vs 40.2 AP). To further validate the effectiveness of the loss function across multiple one-stage models, experiments are conducted on the FCOS+ATSS model (41.2 AP vs 42.1 AP).

Reviewer comment (Theory):
The motivation of equation 8 is very clear, but the article does not seem to detail the reasons for this design. It can be explained in more detail whether this RCI_reg is optimal when the conditions are met, and whether there is a clearer theoretical proof.

Author response:
3. Equation 8 is designed based on the analysis results from Figure 3(a). Samples in Region 1 satisfy RCI >= 0, while in Region 2, RCI < 0. When samples satisfied the condition that s-a=iou, the model does not need to emphasize optimizing both classification and regression, i.e., RCI_reg=1. In both Region 1 and Region 2, RCI_reg needs to adjust the regression penalty based on the difference between score and IoU. For samples in Region 1, where the score-IoU difference is larger, the model should focus more on optimization. This is achieved by $Reg_{reg}(RCI>=0)= \frac{(s+a)^2 +IoU^2}{2*(s+a) * IoU}$. Conversely, in Region 2, where the score-IoU difference is larger, the degree to which samples need to be correctly regressed is much smaller than classification. Therefore, the regression penalty should be weakened, as given by $Reg_{reg}(RCI<0)= \frac{2*(s+a) * IoU}{(s+a)^2 +IoU^2}$