Paper: PBADet: A One-Stage Anchor-Free Approach for Part-Body Association (ICLR-2024)

Reviewer comment (Novelty):
The main problem of this paper is that part-to-body offsets have been proposed in previous works [1]. Furthermore, in [1], they also try local center offsets to improve the performance. Hence, considering the not new idea and no other designs, the technical learning of this paper is limitted.  
[1] Jin L, Wang XJ. Grouping by Center: Predicting Centripetal Offsets for the Bottom-up Human Pose Estimation. TMM, 2022.

Author response:
A:  Thank you for highlighting the paper [1] that discusses a concept similar to our part-to-body center offset. However, it is crucial to note the distinct focus of our work on the part-body association task, as opposed to the multi-person pose estimation task emphasized in [1]. This distinction is elaborated in Section 2 "Related Work" of our paper.

The part-body association task, exemplified by models like BodyHands, Hier R-CNN, and BPJDet, involves detecting the bounding boxes of both parts and bodies and establishing their association relationships. In contrast, multi-person pose estimation primarily focuses on detecting individual keypoints of each person.

To illustrate the practical application of our approach, consider hand gesture recognition in a scan room setting. Here, it is essential for the system to respond exclusively to the technician's gestures while ignoring the patient’s hands to prevent unintended interactions. This scenario requires not just the detection of hands (parts) but also the identification of their associated body, necessitating bounding boxes for both hand and body for effective gesture recognition and person classification.

As discussed in [4], while multi-person pose estimation can be adapted for the part-body association task, it requires additional models for detecting part and body bounding boxes. Using pose estimation for association, i.e., determining whether a keypoint lies within a part's bounding box, introduces a two-stage heuristic process that can add complexity and potentially reduce efficiency. Our method addresses this gap by providing a more streamlined, single-stage solution for the part-body association, enhancing both the efficiency and applicability of the approach in practical scenarios.

[4] Who are raising their hands? Hand-raiser seeking based on object detection and pose estimation

Reviewer comment (Experiments):
To verify the association efficiency, author should compare your association method with other grouping methods, such as associative embeddings and methods in [1], under the same part detection accuracy.  
[3] Associative Embedding: End-to-End Learning for Joint Detection and Grouping.

Author response:
A: We thank you for suggesting relevant works for comparison. In our revised paper, we will include discussions on these works, particularly [1] and [3], in the Related Work section. Considering that [1] is not open-sourced and [3] was proposed in 2017 with performance no longer considered state-of-the-art, we have chosen to compare our method with ED-Pose [6], a current leading method in multi-person pose estimation. We have presented a qualitative comparison with ED-Pose on images from COCO in Figure 5. 

For a fair and meaningful quantitative comparison, we conducted evaluations on the COCO val-set, as ED-Pose was also trained on the COCO dataset. This validation set contains 2,693 images, each featuring at least one person. To ensure consistency, we utilized the ground truth body-part bounding boxes from COCO-HumanParts, which are derived from the original COCO annotations. We excluded facial and head parts due to their inconsistent number of keypoints and focused on four challenging and smaller parts: left hand, right hand, left foot, and right foot, which can be determined by one keypoint. For evaluation, we used the ED-Pose model (Swin-L) trained on the COCO train-set and our PBADet model (YOLOv5l6) trained on the COCO-HumanParts train-set.

In assessing performance, we considered a predicted keypoint by ED-Pose as a true positive if it fell within a ground truth part box. In favor of ED-Pose, we do not use a part detection model but directly use the ground truth part boxes. For PBADet, a predicted part box with successful part-body association was deemed a true positive when it achieved an IoU greater than 0.5 with the ground truth box. The results, as shown in the table below, indicate that PBADet demonstrates superior robustness in part-body association compared to ED-Pose, especially in the precision, recall, and F1 score metrics for the selected body parts. This comparative analysis underscores the efficacy of PBADet in handling part-body associations, affirming its robustness and precision over existing methods in this domain.

| Methods                  | Parts     | Precision | Recall | F1   |
|--------------------------|-----------|-----------|--------|------|
|                          | lefthand  | 18.36     | 30.34  | 22.87|
|                          | righthand | 19.57     | 31.59  | 24.17|
| ED-Pose                  | leftfoot  | 29.32     | 65.28  | 40.46|
|                          | rightfoot | 29.13     | 65.58  | 40.34|
|                          | all parts | 24.09     | 45.51  | 31.51|
|--------------------------|-----------|-----------|--------|------|
|                          | lefthand  | 37.63     | 60.82  | 46.50|
|                          | righthand | 38.58     | 61.26  | 47.34|
| Ours                     | leftfoot  | 41.56     | 66.24  | 51.08|
|                          | rightfoot | 40.86     | 65.55  | 50.34|
|                          | all parts | 39.41     | 63.09  | 48.52|


[6] Explicit box detection unifies end-to-end multi-person pose estimation

Reviewer comment (Evaluation):
The focus of this paper is association, but the evaluation metric is AP. More metrics on association quality should be discussed.

Author response:
A: In Section 4.1 "EVALUATION PROTOCOLS", we outlined various metrics for assessing association accuracy. Alongside AP, we utilize Conditional Accuracy (defined as the percentage of correctly associated bodies among the correctly detected hand instances) and Joint AP, as defined by BodyHands, and log-average miss matching rate (mMR−2) on FPPI of part-body pairs, as defined by BPJDet (please refer to the original papers for detailed definitions). These metrics, presented in Tables 1, 4, 3, 5, and 6, offer a comprehensive assessment of our method's association accuracy.

We hope these revisions address your concerns and provide a more comprehensive understanding of our method and its unique contributions. We are dedicated to making the necessary updates to enhance the clarity and impact of our paper.

Reviewer comment (Presentation):
In Figure 2, please highlight the parts where the proposed approach makes design changes. In Fig. 1, the direction of arrows are not very obvious. Besides, differences in qualitative results are difficult to distinguish. Please consider emphasizing it further.

Author response:
A: We appreciate your suggestions regarding Figure 2 and Figure 1. In the revised version of the paper, we will highlight the parts in Figure 2 where our approach introduces design changes. We will also address the clarity of the arrows in Figure 1 and enhance the differentiation in qualitative results to ensure that the unique aspects of our approach are more evident.

Reviewer comment (Reproducibility):
Will the proposed method be open-sourced? It seems hard to re-implement the results with limited details.

Author response:
A: We have outlined the primary details under Section 4.1 "EVALUATION PROTOCOLS" in the main paper. Regarding the availability of our code, we are in the process of finalizing its release, pending approval. We are actively working towards this goal and plan to update the revised paper accordingly. Furthermore, we are committed to providing even more detailed implementation to ensure that other researchers and practitioners can reliably reproduce our results.

Reviewer comment (Experiments):
The paper doesn't explicitly address how the model deals with occluded parts.

Author response:
A: Our method does not explicitly address occluded parts. For instance, in hand-body associations, if a hand is not visible (occluded), it is not associated with a body. This approach demonstrates an advantage over methods like BPJDet, which outputs a constant number of part center offsets regardless of visibility, making the association decoding more complex. Our part-to-body center offset definition ensures that no offset is attached to undetected (invisible) parts, enhancing the method's relevance and efficiency.

Reviewer comment (Evaluation):
The dense, per-pixel prediction approach might lead to increased computational demands.

Author response:
A: The one-stage anchor-free object detection approach that we employ, with its dense per-pixel prediction, is established for real-time performance on many platforms. It is more efficient than many two-stage proposal-based object detection methods and traditional anchor-based models, which require multiple predefined anchor boxes for each prediction. Please refer to our response to Reviewer AkLB: "3. Model Capacity and Computational Complexity".

Reviewer comment (Experiments):
The performance might be highly sensitive to the tuning of loss function hyper-parameters.

Author response:
A: We have calibrated the association loss weight (λ_assoc=0.2) to ensure that all losses are of the same magnitude. Other hyper-parameters are set as defaults in established object detection models. Note that, we maintain these parameters consistently across different datasets to demonstrate the generalizability of our method. For NMS thresholds, we select optimal values for each dataset considering their specific levels of crowdedness, which is in line with protocols followed in previous methods, such as BPJDet, ensuring a fair and effective comparison.

Reviewer comment (Experiments):
There's a lack of in-depth comparative analysis with traditional anchor-based models.

Author response:
A: We acknowledge the importance of an in-depth comparative analysis with traditional anchor-based models. BPJDet, a state-of-the-art anchor-based method, has already been compared in our paper. Additionally, we have introduced a new ablation study in our revised paper to compare our model with the traditional anchor-based YOLOv7 model, using the same association definition of part-to-body center offset as in our approach. This comparison has been instrumental in highlighting the advantages of the anchor-free design, especially in the context of part-body association tasks. Our findings show that the anchor-free version of YOLOv7 outperforms its anchor-based counterpart, validating the effectiveness of our method. Furthermore, we compared the traditional anchor-based YOLOv7 model with BPJDet, which also adopts an anchor-based approach but utilizes a body-to-part association definition. This comparison demonstrates that our part-to-body association definition significantly enhances performance, offering clear evidence of the superiority of our approach over previous methods.

| Methods                  | Param (M) | Size | Hand AP↑ | Cond. Accuracy↑ | Joint AP↑ |
|--------------------------|-----------|------|----------|-----------------|-----------|
| BPJDet (anchor-based body-to-part)    | 41.2      | 1536 | 85.3     | 86.80           | 78.13     |
| Ours (anchor-based part-to-body)     | 36.9      | 1024 | 88.4     | 92.31           | 85.28     |
| Ours (anchor-free part-to-body)      | 36.9      | 1024 | 89.1     | 92.62           | 85.98     |

Reviewer comment (Experiments):
The model's efficiency in handling images with multiple overlapping objects is unclear.

Author response:
A: We have conducted experiments on the COCOHumanParts dataset, which includes scenarios where people overlap and are closely situated, as illustrated in Figure 4. Additionally, our Appendix includes experiments on the CrowdHuman dataset, featuring more densely overlapping situations.

Reviewer comment (Writing):
How does PBADet address the challenge of occluded parts in the detection process?

Author response:
A: Our method does not explicitly address occluded parts. For instance, in hand-body associations, if a hand is not visible (occluded), it is not associated with a body. This approach demonstrates an advantage over methods like BPJDet, which outputs a constant number of part center offsets regardless of visibility, making the association decoding more complex. Our part-to-body center offset definition ensures that no offset is attached to undetected (invisible) parts, enhancing the method's relevance and efficiency.

Reviewer comment (Experiments):
How does the model perform in scenarios with multiple overlapping or closely situated objects?

Author response:
A: We have conducted experiments on the COCOHumanParts dataset, which includes scenarios where people overlap and are closely situated, as illustrated in Figure 4. Additionally, our Appendix includes experiments on the CrowdHuman dataset, featuring more densely overlapping situations.

Reviewer comment (Experiments):
Have you tested the model’s scalability and performance across various domains other than human part-to-body associations?

Author response:
A: We have expanded our research to include experiments on the AnimalPosedataset [1] [2], demonstrating our model's scalability and performance in domains beyond human part-to-body associations. The qualitative results of these experiments are included in the revised paper.

[1] Cross-Domain Adaptation for Animal Pose Estimation

[2] AP-10K: A Benchmark for Animal Pose Estimation in the Wild

Reviewer comment (Experiments):
Could you provide more comparative performance of PBADet against traditional anchor-based detection models?

Author response:
A: We acknowledge the importance of an in-depth comparative analysis with traditional anchor-based models. BPJDet, a state-of-the-art anchor-based method, has already been compared in our paper. Additionally, we have introduced a new ablation study in our revised paper to compare our model with the traditional anchor-based YOLOv7 model, using the same association definition of part-to-body center offset as in our approach. This comparison has been instrumental in highlighting the advantages of the anchor-free design, especially in the context of part-body association tasks. Our findings show that the anchor-free version of YOLOv7 outperforms its anchor-based counterpart, validating the effectiveness of our method. Furthermore, we compared the traditional anchor-based YOLOv7 model with BPJDet, which also adopts an anchor-based approach but utilizes a body-to-part association definition. This comparison demonstrates that our part-to-body association definition significantly enhances performance, offering clear evidence of the superiority of our approach over previous methods.

| Methods                  | Param (M) | Size | Hand AP↑ | Cond. Accuracy↑ | Joint AP↑ |
|--------------------------|-----------|------|----------|-----------------|-----------|
| BPJDet (anchor-based body-to-part)    | 41.2      | 1536 | 85.3     | 86.80           | 78.13     |
| Ours (anchor-based part-to-body)     | 36.9      | 1024 | 88.4     | 92.31           | 85.28     |
| Ours (anchor-free part-to-body)      | 36.9      | 1024 | 89.1     | 92.62           | 85.98     |

Reviewer comment (Evaluation):
Table 2 doesn't mention AP for small objects. The proposed method performs better than previous methods on medium and large objects. How does this compare to small objects?

Author response:
A: We acknowledge your observation about the absence of AP for small objects in Table 2. It is important to note that in the original Hier R-CNN paper, small objects are further categorized into small and tiny objects, and AP for small objects is not explicitly provided. Similarly, another comparison method, BPJDet, also does not offer AP details for small objects. We will include the APs for small objects in our revised paper. We emphasize that Table 2 demonstrates our method's competitive object detection accuracy relative to state-of-the-art approaches. However, the critical metric for our task is association accuracy, where, as shown in Table 3, our method achieves superior performance. This distinction underscores our method's effectiveness in part-body association, which is central to our research objective.

Reviewer comment (Experiments):
How do the ablation studies change on the COCOHumanParts dataset? I am curious about the performance of different components of the proposed method on different object sizes: small, medium, and large.

Author response:
A: We have not conducted ablation studies on the COCOHumanParts dataset in our original submission. We acknowledge the importance of such analysis for a comprehensive understanding of our method's performance across different object sizes (small, medium, and large). We would like to highlight that our ablation studies were conducted on the BodyHands dataset, a large-scale dataset with unconstrained images and annotations for hand and body locations and correspondences. This dataset comprises 18,861 training images and 1,629 test images. The results from the BodyHands dataset should provide convincing evidence of our method's efficacy. Nevertheless, we understand the value of additional ablation studies on the COCOHumanParts dataset and, time permitting, we aim to conduct this analysis and share the findings.

Reviewer comment (Reproducibility):
The anchor-free object representation uses multi-scale feature maps. How many scales have been used? Is this the same as the backbone (e.g., YOLOv7) feature representation?

Author response:
A: In our experiments, the multi-scale feature maps align with the backbone feature representations. Specifically, the YOLOv7 model generates P3-P5 outputs (i.e., three scales), and the YOLOv5l6 model provides P3-P6 outputs (i.e., four scales). This consistency with the backbone ensures optimal utilization of the multi-scale feature capabilities.

Reviewer comment (Experiments):
The approach uses a singular part-to-body center offset. The body center is the bounding box center or something else. Also, how important is accuracy in detecting the body center influencing the overall performance?

Author response:
A: In our model, the body center is determined as the center of the bounding box, following standard practices in bounding box detection. Specifically, the center is calculated using the formula ((r+l)/2, (b+t)/2) for a bounding box defined by coordinates (r, l, t, b). Our ablation study (Table 4) demonstrates that incorporating association loss (for detecting part-body center offset) does not adversely affect hand detection accuracy, showcasing the robustness of our approach.

Reviewer comment (Evaluation):
It would be nice to have a section on model capacity and computational complexity (e.g. Params, GFLOPS, per-image inference time, etc) to further improve the article. This should be compared to the other SOTA models.

Author response:
A: We thank you for highlighting the importance of discussing model capacity and computational complexity. Our paper provides detailed information on model parameter sizes and input image sizes in Tables 1 and 3. To further emphasize the efficiency of our model, we have included a comprehensive analysis of inference speed.

Specifically, we have assessed the frames per second (FPS) of PBADet using YOLOv7 and YOLOv5l6 detectors on a GPU V100. With an input size of 1280x1280, PBADet achieves approximately 84 FPS with YOLOv7 and 63 FPS with YOLOv5l6. These numbers indicate that PBADet is well-suited for real-time applications, demonstrating significant efficiency in processing.

For context, we compared this with the ResNet-101-FPN-based detector Mask R-CNN, which is utilized in methods such as BodyHands [1] and Hier R-CNN [2]. The official FPS for Mask R-CNN is about 10 to 20 per image on an NVIDIA V100 GPU & NVLink [3] [4]. When considering the additional computational load due to the association network in BodyHands [1] and Hier R-CNN [2], it is evident that these methods lag significantly behind PBADet in terms of speed. Note that our method has an input size of 1024x1024 while BodyHands [1] and Hier R-CNN [2] use an input size of 1536x1536 (note that the original Mask-CNN uses 800x800), which further enlarges the FPS gap. This disparity in processing efficiency highlights the advanced capability of PBADet, particularly when it comes to real-time processing demands.

[1] Whose Hands Are These? Hand Detection and Hand-Body Association in the Wild, CVPR 2022

[2] Hier R-CNN: Instance-Level Human Parts Detection and A New Benchmark, TIP 2020

[3] Mask R-CNN

[4] https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md

Reviewer comment (Evaluation):
The multi-scale module has a significant impact on the performance improvement w.r.t. the baseline. What could be the reason?

Author response:
A: Our ablation study in Table 4 indicates that interpreting the center offset as a normalized absolute distance without multi-scale considerations leads to a reduction in hand prediction accuracy and joint AP. This is because each point in the multi-scale dense feature predicts a center offset, and it is challenging to achieve convergence for different scales to output the same absolute distance. The multi-scale module thus plays a critical role in enhancing performance.

Reviewer comment (Experiments):
The loss weights ($\lambda$ and NMS thresholds $\tau$ values) are optimal for a given dataset?

Author response:
A: The loss weights during training remain the same across different datasets. However, the NMS thresholds are not loss weights and are post-processing parameters in object detection. NMS thresholds are optimally selected for each dataset since different datasets present varying levels of crowdedness. This approach is in line with protocols followed in previous methods, such as BPJDet, ensuring a fair and effective comparison.

Reviewer comment (Theory):
The author states that the proposed part-to-body center offset guarantees a one-to-one correspondence between parts and bodies. However, the paper does not elaborate on how the situation is handled when the predicted center offsets of multiple parts with the same category point to the same body. How to define the order for associating multiple parts with the same category? These details are not described in Section 3.4.

Author response:
A: We acknowledge your concern regarding how our method handles situations where multiple parts of the same category point to the same body. As detailed in Section 3.4 "DECODING PART-TO-BODY ASSOCIATIONS", our network outputs the center offset of a part, pointing to the body to which the part belongs. We use Euclidean distance between the estimated body center (part center plus the center offset) and the centers of unassigned bodies that enclose the part. The body with the smallest distance (i.e., the distance of the estimated body center and body center) is then assigned as the corresponding body for the given part. This approach ensures accurate and efficient part-to-body association.

Reviewer comment (Presentation):
In Figure 2, it seems that 'P3' and 'P5' in the 'Multi-scale features' are labeled incorrectly.

Author response:
A: We are grateful for your observation regarding the labeling error in Figure 2, where 'P3' and 'P5' in the 'Multi-scale features' were incorrectly labeled. We have corrected this typo in the revised version of the paper to ensure clarity and accuracy in our visual representations.

Reviewer comment (Reproducibility):
The reviewer wonders whether the models are trained from scratch or the pretrained YOLO weights are used. It is not mentioned in the paper.

Author response:
A: We appreciate your inquiry about whether our model was trained from scratch or using pretrained YOLO weights. To clarify, we followed the same protocol established in the BPJDet paper [1] that uses pretrained YOLO weights. This implementation detail will be explicitly stated in the revised manuscript to avoid any ambiguity.