Paper: P2P: Transforming from Point Supervision to Explicit Visual Prompt for Object Detection and Segmentation (ICLR-2024)

Reviewer comment (Evaluation):
The performance of the model largely depends on the capability of SAM itself. From the quantitative results, the proposed approach does not have a significant improvement compared with directly using SAM.

Author response:
Thanks for your comment. The emergence of large models has brought new insights and performance improvements to many downstream tasks. We think that the trend of applying foundation models to solve problems is irreversible. 

For point-supervised detection and segmentation, the absence of size information makes it challenging. SAM, due to its lack of semantic understanding, also struggles to fully address this issue. In our method, P2P is the first attempt to iteratively enhance the quality of prompts, guiding SAM to generate accurate masks and achieving further improvement upon SAM.

we conducted a more detailed comparison with SAM. We found that, for mIoU and CorLoc, our approach improves mIoU and CL@0.5 by 11.4 and 18.37, respectively. Additionally, our approach also demonstrates a notable improvement in $AP_{50}$, with a performance increase of 8.5 on COCO dataset. 



| Method | mIoU  | CL@0.5 | CL@0.7 | CL@0.9 |  AP  | AP50 | AP75 |
| :----: | :---: | :----: | :----: | :----: | :--: | :--: | :--: |
| P2BNet | 57.5  |   -    |   -    |   -    | 22.1 | 47.3 |  -   |
|  SAM   | 58.25 | 60.32  | 47.32  | 21.27  | 27.3 | 45.3 | 28.5 |
|  Ours  | 69.70 | 79.05  | 62.40  | 24.07  | 31.6 | 53.8 | 32.7 |

Reviewer comment (Reproducibility):
The method seems to be computationally demanding since the training process is conducted with 4*RTX 4090 GPUs.

Author response:
As suggested, we have supplemented detailed comparisons of memory, training time, and inference time in the appendix of revised version. This is also explained in the *unified response* part. 

P2P consumes a bit higher time and memory during the training phase than P2BNet, but it greatly enhances the quality of pseudo-labels, which is crucial for subsequent detection and segmentation tasks. With high-quality pseudo-labels, one can freely choose any task-specific subnetwork for optimization, achieving commendable performance. Therefore, compared to the substantial improvement in performance, the time consumption during training is not an issue.

Reviewer comment (Experiments):
Could the author provide a comparison of training time and memory cost compared with other methods?

Author response:
As suggested, we have supplemented detailed comparisons of memory, training time, and inference time in the appendix of revised version. This is also explained in the *unified response* part. 

P2P consumes a bit higher time and memory during the training phase than P2BNet, but it greatly enhances the quality of pseudo-labels, which is crucial for subsequent detection and segmentation tasks. With high-quality pseudo-labels, one can freely choose any task-specific subnetwork for optimization, achieving commendable performance. Therefore, compared to the substantial improvement in performance, the time consumption during training is not an issue.

Reviewer comment (Evaluation):
The authors claim that the existing methods aim to release the annotation burden while still achieving decent performance. However, the proposed method seems still far behind fully-supervised method.

Author response:
1. **point-supervised tasks is challenging:** Due to the absence of target size information, accurate detection and segmentation are challenging under point supervision alone. In theory, the performance of fully supervised methods serves as an upper bound for the performance of all weakly supervised methods. As shown in the following tables, under equivalent conditions (i.e., point supervision using the same fully supervised method fine-tuned), previous state-of-the-art point-supervised methods achieved only around **60% (for detection) or 50% (for segmentation)** of the fully supervised performance. 
2. **We further narrows the performance gap:** With only point annotations, our method can achieve performance levels of around **80%** of a fully supervised detector, significantly reducing the required annotation effort.



|    Method    | Sup. |  AP  | AP50 | AP75 |
| :----------: | :--: | :--: | :--: | :--: |
| Faster  RCNN |  F   | 37.4 | 58.1 | 40.4 |
|     DINO [1]     |  F   | 57.2 | 75.7 | 62.7 |
|  P2BNet-FR   |  P   | 22.1 | 47.3 |  -   |
|   Ours-FR    |  P   | 31.6 | 53.8 | 32.7 |
|  Ours-DINO   |  P   | 45.1 | 66.1 | 48.9 |

|    Method     | Sup. |  AP  | AP50 | AP75 |
| :-----------: | :--: | :--: | :--: | :--: |
|   Mask RCNN   |  F   | 35.4 | 56.4 | 37.9 |
|  Mask2Former [2]  |  F   | 46.1 | 69.4 | 49.8 |
| AttenShift-MR |  P   | 21.2 | 42.0 | 19.4 |
|    Ours-MR    |  P   | 26.4 | 48.6 | 26.2 |
|    Ours-MF    |  P   | 34.9 | 58.9 | 36.1 |

*Note: FR indicates Faster RCNN, MR indicates Mask RCNN, and MF indicates Mask2Former*

------

*[1] Zhang, Hao, et al. "DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection." The Eleventh International Conference on Learning Representations. 2022.*

*[2] Cheng, Bowen, et al. "Masked-attention mask transformer for universal image segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.*

Reviewer comment (Evaluation):
It is better to give an inference time comparison, which can help readers better understand the propose method.

Author response:
As suggested, we have supplemented detailed comparisons of memory, training time, and inference time in the appendix of revised version. This is also explained in the *unified response* part. 

For P2P, the inference speed is a bit lower than P2BNet. However, we think that the time consumption of point-supervised methods itself may not be a primary limitation in practical applications. As illustrated  in the section 3.1, the point supervision task involves training a regressor to predict pseudo-labels. One can choose an appropriate subnetwork (*e.g.*, a detector ) for specific tasks or applications and opimized it in a fully supervised manner with the predicted pseudo labels, (*e.g.*, we can use more powerful detectors, such as DINO, to ensure performance, or choose faster detectors like YOLO to guarantee real-time capabilities.).

Reviewer comment (Novelty):
SAM has a good ability to generate the mask of objects based on point. Thus, it may give a good performance using SAM and Cascade structure, which is similar to PGSR.

Author response:
Thank you for your comment.  

**Results**: Following your advice, we devised the cascaded-SAM. However, experimental results (in the following table) indicate that it **did not yield satisfactory performance**. 

**Analysis**: We attribute SAM's performance to the quality of prompts. A critical issue with point prompts is semantic ambiguity. SAM cannot ascertain whether the semantic of a point is local or global (e.g., a car or a car window). Under point prompts, numerous local masks are generated (e.g., clothing on a person, the glass of a car), which becomes more pronounced in complex scenes, such as the COCO dataset. Even when using the bounding box of local masks as prompts and reapplying SAM, the resulting masks remain local and fail to enhance performance effectively. In contrast, **PGSR progressively regresses proposal seeds to the correct semantic regions based on given semantic labels**, (e.g., transitioning from clothing to the entire person), which used as a prompt can better guide SAM in generating accurate masks.

|   Method    | mIoU  | CL@0.5 | CL@0.7 | CL@0.9 |  AP  | AP50 | AP75 |
| :---------: | :---: | :----: | :----: | :----: | :--: | :--: | :--: |
|     SAM     | 58.25 | 60.32  | 47.32  | 21.27  | 24.3 | 43.8 | 24.3 |
| Cascade SAM | 58.56 | 60.64  | 47.76  | 22.05  | 24.7 | 44.3 | 24.8 |
|    Ours     | 69.70 | 79.05  | 62.40  | 24.07  | 26.4 | 48.6 | 26.2 |

Reviewer comment (Novelty):
Novelty: Some parts of the proposed method lack novelty, particularly in the proposed Prompt Refiner II, phrases like "similar to previous works" and "follow the common practice" resemble the refinement stage of P2BNet [1].

Author response:
Dear Reviewer iqSb,

We would like to express our sincere gratitude for your time and efforts in improving our paper.

Considering your concerns about our paper, we have revised some of the content again and summarized it below:

- We've reorganized the methods section, merging 3.4 into 3.1 to give the reader a better understanding of our framework.

- Regarding *Prompt refiner II*, we placed the homogenization with the previous method in the Appendix to highlight the innovations. 

We hope the above improvement could clarify your concerns. Looking forward to your feedback and discussion. Thanks again!

Best,

Paper 1568 Authors.

Reviewer comment (Writing):
Writing: Certain parts of the language are not as concise and straightforward as desired. Additionally, there is a slight delay in providing specific explanations for some concepts, such as "proposal seeds" and "selected high-quality embedding features,". Furthermore, Section 3.4 should be repositioned closer to the beginning of the paper to provide readers with a clearer understanding of the overall method's workflow.

Author response:
Q3: *The concept ‘proposal seeds’ is proposed in Section 3.1, but its specific generation way (i.e., outer rectangle of the mask generated by SAM) is given until Section 3.4, it is suggested to explain its specific concept early on to avoid confusion. Furthermore, Section 3.4 should be repositioned closer to the beginning of the paper to provide readers with a clearer understanding of the overall method's workflow.*

A3: Thanks for your suggestion. We have reorganized Sections 3.1 and 3.4 in the revised version.


Dear Reviewer iqSb,

We would like to express our sincere gratitude for your time and efforts in improving our paper.

Considering your concerns about our paper, we have revised some of the content again and summarized it below:

- We've reorganized the methods section, merging 3.4 into 3.1 to give the reader a better understanding of our framework.

- Regarding *Prompt refiner II*, we placed the homogenization with the previous method in the Appendix to highlight the innovations. 

We hope the above improvement could clarify your concerns. Looking forward to your feedback and discussion. Thanks again!

Best,

Paper 1568 Authors.

Reviewer comment (Writing):
In Section 3.1, the paper mentions that “the core of this task lies in designing an accurate point-to-box regressor”, and introduces the P2P framework (point-to-visual prompt), but it is inconsistent between “point-to-box” and “point-to-visual prompt”; it is suggested to add the description of the process of visual prompt-to-box/mask after.

Author response:
Q1: *In Section 3.1, the paper mentions that “the core of this task lies in designing an accurate point-to-box regressor”, and introduce the P2P framework (point-to-visual prompt), but it is inconsistent between “point-to-box” and “point-to-visual prompt”, it is suggested to add the description of process of visual prompt-to-box/mask after*.

A1: Thanks for your detailed comment. In P2P, the two stages, SEPG and PGSR, respectively serve the roles of ‘point to prompt’ and ‘prompt to pseudo-mask’. We have added this description in Section 3.1 to make the logic more complete.

Reviewer comment (Presentation):
What’s the meaning of the orange arrow in the lower part of the Figure 2? (i.e., the arrow on the right of ‘Refine box b*’).

Author response:
Q2: *What’s the meaning of the orange arrow in the lower part of the Figure 2? (i.e., the arrow on the right of ‘Refine box b\*’)*.

A2: Thanks for your detailed comment. This refined box b* is generated by SEPG stage and serves as one part of the prompt, subsequently inputted into PGSR.

Reviewer comment (Writing):
The concept ‘proposal seeds’ is proposed in Section 3.1, but its specific generation way (i.e., outer rectangle of the mask generated by SAM) is given until Section 3.4; it is suggested to explain its specific concept early on to avoid confusion.

Author response:
Q3: *The concept ‘proposal seeds’ is proposed in Section 3.1, but its specific generation way (i.e., outer rectangle of the mask generated by SAM) is given until Section 3.4, it is suggested to explain its specific concept early on to avoid confusion. Furthermore, Section 3.4 should be repositioned closer to the beginning of the paper to provide readers with a clearer understanding of the overall method's workflow.*

A3: Thanks for your suggestion. We have reorganized Sections 3.1 and 3.4 in the revised version.

Reviewer comment (Reproducibility):
What’s the concrete content of ‘sharing some of the weights’ about the two refiners mentioned in Section 3.2?

Author response:
Q4: *What’s the concrete content of ‘sharing some of the weights’ about the two refiners mentioned in Section 3.2?*

A4: The two refiners involve a shared backbone and, additionally, in the head, there are two shared fully connected (FC) layers. In the revised version, we have added additional details in the Appendix.

Reviewer comment (Novelty):
What are the key differences in the workflow of the proposed Refiner2 compared to the refining stage in previous works [1]?

Author response:
Q5: *What are the key differences in the workflow of the proposed Refiner2 compared to the refining stage in previous works [1]?*

A5: The main differences lies in:

1) **Structural differences**: The refiner in P2P comprises a classification head and an embedding head, which calculate classification scores and feature embeddings, respectively. The classification scores are utilized for optimizing proposals, while the feature embeddings are employed for updating semantic prototypes. On the other hand, P2BNet utilizes a multi-instance learning structure with two classification heads for computing classification scores and instance scores. 
2) **Optimization differences**: We use the proposal scores from *Refiner II* as indicators to update the semantic prototypes with high-quality embeddings. The high-quality prototypes are then used in *Refiner I* to compute instance probabilities, obtaining high-quality proposals. When these proposals are input into Refiner 2, it further enhances the quality of semantic prototypes. We believe there is a mutually reinforcing relationship between *Refiner I* and *Refiner II*, which is absent in [1].

Reviewer comment (Evaluation):
How about the computational complexity of the proposed method compared to other methods?

Author response:
Q6: *How about the computational complexity of the proposed method compared to other methods?*

A6: As suggested, we added a comparison of memory and time consumption in the appendix, and the explanation for this is provided in the *unified response* part.

Reviewer comment (Experiments):
In Section 3.4, the actual input of the subsequent P2P refiner is the initial proposal seed box (i.e., outer rectangle of the mask generated by SAM). If possible, it is recommended to use the box (e.g., the fully supervised bounding box label) as the prompt of SAM to conduct a more comprehensive experimental comparison in segmentation, rather than just using the point as the prompt of SAM for comparison.

Author response:
Q7: *In Section 3.4, the actual input of the subsequent P2P refiner is the initial proposal seed box (i.e., outer rectangle of the mask generated by SAM). If possible, it is recommended to use the box (e.g., the fully supervised bounding box label) as the prompt of SAM to conduct a more comprehensive experimental comparison in segmentation, rather than just using the point as the prompt of SAM for comparison.*

A7: In the revised version, we supplemented the results using GT bounding box as a prompt in Tab. 3, which can be considered an upper bound for point supervision methods. Using only point as supervision, we reached performance levels of **85% (26.4 vs 31.1)**, **91% (48.6 vs 53.1)**, and **80% (26.2 vs 32.6)** on three respective metrics when using bounding box prompts. This indicates that P2P approaches the performance of bounding box supervision, reaffirming the effectiveness of our method.  

| Method | Sup. |AP | AP50 | AP75 |
| :-------------: | :-----------: | :---------: | :--------------: | :--------------: |
|     SAM-MR      |     Point     |    24.3     |       43.8       |       24.3       |
|     SAM-MR      |      box      |    31.1     |       53.1       |       32.6       |
|     Ours-MR     |     Point     |    26.4     |       48.6       |       26.2       |

Reviewer comment (Experiments):
The author said ‘we observe that only 40% of the masks covered more than 70% foreground pixels’. However, as I know, because the semantic-free of SAM, the highest score mask generated by SAM may not what we want, but the top-3 masks may contain what we want in most situations. What we should do is to select the best one (or with extra refinement) from the top-3 masks. But the paper only chooses the highest score one and claimed ‘only 40% of the masks covered more than 70% foreground pixels’. I think this is a handmade problem. In other words, the author does not make full use of the foundation model and makes the problem more difficult.

Author response:
Thank you for your insightful comment. We perceive this as two distinct approaches. To obtain semantically accurate masks, as you mentioned, one could design a selector to choose the most accurate mask from the three. This might yield better results, but it does not guarantee success in complex scenarios.

We analysis that the significant semantic differences in masks generated by SAM can be attributed fundamentally to the semantic uncertainty in the provided prompts. Thus, we can address this by improving the quality of the prompts themselves. By progressively enhancing the semantic confidence of the prompts, we aim to ensure the foundation model outputs accurate masks.

Reviewer comment (Novelty):
I think the framework is lack of novelty and a little engineering. If I understand you correctly, the structure is (SAM+P2BNet(CBP stage + PBR stage))+(SAM+ P2BNet)+ (SAM+ P2BNet) ....... The whole paragraph of ‘SEMANTIC-EXPLICIT PROMPT GENERATION’ is P2BNet (the network, the loss, the sampling is similar) and the ‘PROMPT GUIDED SPATIAL REFINEMENT’ is the application of SAM. I think the combination of P2BNet and SAM is OK, but the author did not propose some insights or other challenges in combination. And the ITERATIVE LEARNING is a little engineering, I am interested in the time cost in practical application.

Author response:
1. Firstly, the *main distinctions between SEPG and P2BNet* are as follows: 

   - **differences in Structure**: Diverging from classical MIL frameworks, we employ a more stable feature prototype for computing instance-level probabilities. The refiner in P2P comprises a classification head and an embedding head, which calculate classification scores and feature embeddings, respectively. P2BNet utilizes the MIL structure with two classification heads for computing classification scores and instance scores. 

   - **Differences in Sampling**: In order to mitigate the randomness associated with sampling and optimization, we employ a **seed-based group sampling** strategy, leveraging prior knowledge provided by the foundation model. In contrast, P2BNet employs point-centered neighbor sampling. 

   - **Differences in optimization**: P2P follows a **group-then-individual** fashion to reduce the solution space. In Prompt refiner I, semantic and instance probabilities for each group are computed, and through optimizing the joint probability, the group with the highest score is selected. And then, the proposals in the selected group are augmented as the input of the Prompt refiner II. P2BNet employs a multi-level individual optimization approach.
2. Furthermore, we would like to emphasize that our novelty extends beyond individual components. Our work contributes novel insights to the community in addressing weakly supervised or point-supervised tasks, specifically focusing on how to better leverage foundation models in the era of the foundation model to improve solutions for weakly supervised problems.
3. Regarding time consumption, while P2P consumes a bit higher time and memory during the training phase, it yields pseudo-labels with higher quality. We think that the time consumption of point-supervised methods itself may not be a primary limitation in practical applications. Leveraging these high-quality pseudo-labels, it is free for users to employ any detectors to conduct specific tasks or applications. Therefore, compared to the substantial improvement in performance, the time consumption during training is not an issue.

Reviewer comment (Presentation):
The visualization is unclear if I do not enlarge the image or I print it. The line is too thin.

Author response:
Thank you for your reminder. In the revised version, we have redrawn the visualization to address the concerns.

Reviewer comment (Novelty):
I think the title is not suitable because another paper is named as P2P: Rethinking Counting and Localization in Crowds:A Purely Point-Based Framework, ICCV2021. The tasks are different but relevant, and are all point-based.

Author response:
Thanks for the comment. The paper you referenced presents a point-to-point network named **P2PNet** designed for crowd counting tasks. While both employ the term *P2P*, the tasks and expressions hold distinct meanings. Our reference to "point to prompt" reflects a different context. Hence, we consider this to be coincidental. We appreciate your feedback and are open to making adjustments if necessary.

Reviewer comment (Experiments):
I think the experiments on better detectors or segmentation network (or better backbone?) are needed. The faster RCNN is classic but too old. The fully-supervised method is far beyond 30+ or 40+ on COCO.

Author response:
Thank you for your insightful advice. To ensure a fair comparison with previous methods, we refrained from employing more advanced detectors or segmentation networks. In the revised version, as suggested, we have supplemented the experiments in the appendix on advanced detectors and segmentation networks.

1) When **DINO (both R-50 and Swin-L are used as backbone)** [2] is adopted as the detector, our method achieves a 38.2 AP (R-50) and 45.1 AP (Swin-l) on COCO dataset, approximately 80% of the fully supervised performance, and a 57.2 AP50 (R-50) and 66.1 AP50 (Swin-l), about 87% of the fully supervised performance. 

2) When the **Mask2Former** [3] is employed as the segmentation network, achieving a 34.9 AP and 58.9 AP50. 

Due to the limited time, we do not adjust the parameters carefully, but it is sufficient to verify the effectiveness of the proposed method. We believe that the performance will be further improved. In summary, when utilizing more advanced detector or segmentation network, our method still closely approaches the performance of fully supervised methods.

|   Method    | backbone |  AP  | AP50 | AP75 |
| :---------: | :------: | :--: | :--: | :--: |
|    DINO     |   R-50   | 49.0 | 66.4 | 55.3 |
|    DINO     |  Swin-l  | 57.2 | 75.7 | 62.7 |
|  Ours-DINO  |   R-50   | 38.2 | 57.2 | 40.9 |
|  Ours-DINO  |  Swin-l  | 45.1 | 66.1 | 48.9 |
| Mask2former |  Swin-s  | 46.1 | 69.4 | 49.8 |
|   Ours-MF   |  Swin-s  | 34.9 | 58.9 | 37.6 |