Paper: Discriminatively Matched Part Tokens for Pointly Supervised Instance Segmentation (ICLR-2024)

Reviewer comment (Novelty):
The section of related work is inadequate in weakly supervised instance segmentation. Some weakly image-level supervised methods [1] and box-supervised methods [2][3] are not listed.

Author response:
As suggested, we have integrated the previous researches, including IRNet, BoxInst and Box-LevelSet, into the section of related work for a comprehensive survey.

Reviewer comment (Evaluation):
The inference speed of the whole method could be reported and it can better demonstrate the superiority of the proposed method.

Author response:
In the supplementary, we measured and reported the FPS metric of our DMPT model and others, which displays DMPT's superiority of inference efficiency.  As suggested, we will move this section in the main paper to improve the presentation of our strengths.

Reviewer comment (Writing):
Some typos exist, like "Combed with ..." in section 3.5 should be "Combined with ...".

Author response:
Thinks for your advice, the typos have been revised and we think the fluency of our paper has been enhanced.

Reviewer comment (Experiments):
If some parts of an object are directly filtered as background due to low activation values in the self-attention map, how could this be further optimized in subsequent steps?

Author response:
A1: While some object parts are occluded in some images, in statistical sense, all object parts of the same category will appear in the dataset. Specially, our part classifiers statistically learn the fine-grained semantics from the whole dataset. As long as most of the parts visible in the images, part classifiers can be optimized to represent and recognize the stable fine-grained semantics by using visible parts. In addition, our token-classifier matching mechanism allows the part classifier not matching any parts if its corresponding part is occluded. This characteristic ensures that the classifiers not only are able to represent the fine-grained pattern from visible parts, but also can keep steady optimized when object part absents in some images.

Reviewer comment (Experiments):
It would be beneficial to include comparisons with more weakly supervised methods as the performance reference for the readers.

Author response:
A2: Thanks for your constructive and insightful comments. We have added more comparisons with weakly supervised methods in the revised version.

Reviewer comment (Evaluation):
The authors present the result for an image of a person. It would be advantageous to include more image samples in the main paper. I am particularly interested in the extent to which the part-classifiers effectively learn semantically meaningful parts and consistently activate similar parts in diverse images. Interestingly, the person sample in Figure 2 in the supplementary material does not seem to achieve this. Could the authors explain this?

Author response:
Due to challenge of less supervision information, the location precision of object remains imperfect in pointly supervised instance segmentation, which limits accuracy of part tokens generation. Meanwhile, part deformation and occlusion result in difficulty for clustering methods to produce part tokens to represent anthropogenic defined object parts, like "head" or "leg". This does limit the ability of the part classifier to actually activate the object part uniformly. 

Surprisingly, DMPT is still able to learn relative stable and consistent fine-grained semantics than the baseline method. 
We conduct statistical analysis of feature similarity on different aspects in Figure 1, updated in "rebuttal_visualization.pdf" of the new supplementary materials. 
In Figure 1(a), the similarity between part features and background of DMPT is lower than that of AttentionShift, which showcases that our part-based modeling mechanism can effectively suppress background noise, obtaining stable semantics. As we utilize part classifiers to uniformly represent fine-grained semantics, DMPT exhibits stronger discrimination among objects of different categories, In Figure 1(b). Particularly, we calculate the feature similarity of different parts of the same object, Figure 1(c). It can be seen that the part features of DMPT are more discriminative than AttentionShift, implying that we do learn consistent fine-grained semantics.

Reviewer comment (Experiments):
I have reservations about the validity of the token-classifier matching when some parts are missing in the input image due to occlusion or other factors. In such situations, do the part-classifiers corresponding to the missing parts get correctly excluded in the matching matrix? In the rebuttal, visual results for these cases would be appreciated.

Author response:
R2.1: In Equ.8 of main paper, the token-classifier matching allows the case when part tokens' number is less than that of part classifiers ($K \textless N$), due to part occlusion or other factors. We visualize the matching matrix of some cases in Figure 2 of the "rebuttal_visualization.pdf".

Reviewer comment (Evaluation):
I have reservations about the validity of the token-classifier matching in cases of over-segmentation. It seems possible that sometimes K can significantly exceed N, especially as there is no constraint on K. In such cases, a single part-classifier should ideally be matched with multiple tokens. In the rebuttal, visual results for these cases would be appreciated.

Author response:
R2.2: No, the matching mechanism do not handle cases of over-segmentation, because it has been handled before token-classifier matching. First, we pre-define that each object has $N$ parts, based on which we initialize $N$ part classifiers and partition the object into $N$ cluster centers. Then, similar cluster centers are merged according to their cosine distances, avoiding over-segmentation on consecutive part semantics. After this procedure, we produce $K$ part clusters for each object.  Note that $K$ might be different values for different objects. In order to uniformly represent the number of part clusters across all objects, we use $K$ to control the upper limit of the number of object parts after merging similar clusters. In addition, $K\leq N$ is an implicit setting in our paper. 

Thanks for this constructive advice. For more clear presentation, we will add the details about how to handle the over-segmentation problem and relation of $K$ and $N$ in the paper.

Reviewer comment (Experiments):
It would be valuable for the authors to explain their criteria for determining N, the number of part-classifiers. The optimal number of parts may vary across different classes and datasets. Complex classes like bicycles might require more parts, while classes with simple shapes (e.g., a ball) may need fewer. Can the authors elaborate on their approach to determining the number of part classifiers in various scenarios?

Author response:
$N$ is a hyper-parameter to pre-define the number of classifiers optimized. $N$ is experimentally determined for achieving the balanced performance of all the classes within a dataset. 

For special usage for other scenarios, $N$ is empirically set to a relative high value to ensure classifiers can sufficiently cover various fine-grained semantics. As our DMPT owns two characteristics, (i) token-classifier matching allows some classifiers not matching any part tokens during model training (R2.1), and (ii) the number ($K$) of part tokens of different objects dynamically adapts to object semantics (R2.2), part classifiers have the potential to be dynamically optimized with objects with various part semantics. However, mismatched number of tokens ($K=3$) and classifiers ($N=7$) could result in insufficient optimization on stable fine-grained semantics, Figure 6(c) of main paper.

Reviewer comment (Experiments):
I think SAM itself can handle this task to some extent. Using the given point annotations serve as prompts for SAM, we can obtain pseudo-labels simply. Can you check this setting and compare it with your DMPT-SAM?

Author response:
The setting you mentioned is right. However, getting one accurate pseudo-mask needs a selection upon three pseudo-masks when using one single point to prompt SAM. Note that producing three masks from one single point prompt is a design of SAM to address ambiguity of the prompt. In the setting of pointly supervised methods, there is no ground-truth mask as guidance to select the most accurate one. We only can select pseudo-masks leveraging location scores (IoU predictions) of these masks predicted by SAM. However, these scores are usually unreliable for obtaining the most accurate mask.

In order to address this issue, DMPT acts a prompter before prompting SAM by generating part points with the the single point, which we termed as DMPT-SAM. The generated part points contains more spatial information ($e.g.$, the deformation, scale and extent of object), which is less ambiguous than one single point. With these points as prompt, the location scores are more reliable to select the accurate pseudo-masks. 

Quantitative results in Table 1 and qualitative visualizations in Figure 4(b) of main paper demonstrate that DMPT-SAM facilitates generating more precise pseudo-labels for pointly supervised instance segmentation.

Reviewer comment (Writing):
The notations are somewhat distracting, but honestly, I haven't come up with a better alternative. The core concept of this paper appears to be pretty intuitive to me. However, its mathematical formulation makes the understanding rather complex. It would greatly enhance the paper's clarity if the authors could improve its presentation.

Author response:
Thanks for your constructive advice and we will further simplify and refine our notations for higher presentation in the camera-ready version.

Reviewer comment (Experiments):
The limitation of the part deformation constraint is not considered. For example, will it work properly if the target is a snake with a long and thin shape?

Author response:
The proposed part deformation constraint has the potential to handle these extreme cases. Specially, thanks to the ability of global attention mechanism to effectively capture the overall deformation of the object, DMPT can dynamically decompose the object into various parts, acting as reference points for local spatial deformation, which makes the model focus attention on object parts. Based upon these reference points, the proposed part deformation constraint further enhances local features by coupling the deformation representation to counter part deformation. Therefore, incorporating our part deformation constraint with the global attention mechanism is crucial for effectively resolving substantial object deformations.

Undoubtedly, there are some limitations of the spatial deformation. On the one head, initializing deformation constrain $d(x_i,y_i)$, defined in Equ.3, as a square distance feature might be less rational, as the part of some objects like snake sometimes represents a slender shape, whose spatial feature should have different initialized weights upon horizontally and vertically. On the other head, the value of $d(x_i,y_i)$ should be dynamically reduced before coupling with part classification scores when meeting extreme cases, as spatial constrain might be less unreliable in this situation.  

Thanks for your advice, which inspires us to further improve our method.