From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The paper proposes to utilize prior knowledge from CLIP, e.g. by pruning human-object associations from pretrained models. Reviewers appreciate the idea (especially in the context of weakly-supervised learning), the results and ablations, and the writing. They raise concerns about parts of the method, aspects of evaluation (e.g. data leakage), and novelty. For most reviewers, the concerns are well addressed by the rebuttal. 
justification_for_why_not_higher_score: The reviewers recommending acceptance comment that they prefer to rate the paper a 7 rather than 8, so the paper is clearly above the bar, but enthusiasm for the ideas presented is not sufficient for oral presentation recommendation.
justification_for_why_not_lower_score: No reviewer recommends rejection.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper933/Reviewer_S5GP
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper targets on HOI detection in a weakly supervised manner (i.e., only image description is provided, no instance-level annotation). To improve the generalization towards fine-grained human-object interactions and mitigate the biased learning caused by imbalanced data distribution, this paper proposes a training strategy to distill the knowledge from CLIP model. Specifically, they incorporate CLIP in HOI at both the image and the instance levels. At the image level, they generate a global HOI knowledge bank and predict image-level HOI descriptions. At the instance/object level, the attention mechanism is employed to enrich the representation of HOI features and then improve the performance. A comprehensive evaluation is conducted and a detailed analysis is provided. 
strength_and_weaknesses: Strength:
1.	The motivation is clear. Applying CLIP in weakly supervised learning and facilitating the learning of an HOI detector is essential. 
2.	The approach is interesting. Though combining the detector with CLIP has been studied, but combining them for weakly-supervised learning has been less explored.
3.	The experimental study is promising and comprehensive. The visualizations are also provided.
4.	The ablation study is in detail.

Weakness:
1.	Regarding the whole framework, which part is vital for using CLIP to guide weakly supervised learning? I think the discussion is necessary (but I didn’t find clear answer in the discussion) and help this paper to be distinguished from the other related work.
2.	The knowledge bank is based on classes appearing in the full dataset and is defined by the text. Can you explain how the size of the knowledge bank affects performance? After all, when there exists a brunch of interaction classes, I am not sure about the training efficiency and workload.
3.	SRC only does not help much with the detection, according to Table 2. It is not very effective and is kind of counterintuitive. I would recommend providing a more detailed explanation or removing the SRC part.
4. When a CLIP model is used, it is always necessary to explain the potential issue of fair comparison. After all, CLIP has seen quite a lot of training data during pretraining, and there is a risk of potential data leakage. As such, explanation is necessary.

clarity,_quality,_novelty_and_reproducibility: The approach is new and interesting. It could be useful for weakly-supervised learning.
The implementation details are useful for reproduction.
summary_of_the_review: The overall presentation of this paper is good. However, a few more discussions should be provided. Please see details in weakness.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper933/Reviewer_qCHU
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper presents a new weakly supervised method in the human-object interaction task. The method exploits the prior knowledge from CLIP.
strength_and_weaknesses: [Strength] 
The idea of incorporating external knowledge from CLIP is interesting. 

[Weakness]
1. The authors promise to release source codes. This would eliminate the issue of reproducibility. However, the paper presentation is bad, which makes readers hard to fully understand and hence, potentially hard to reproduce the model based on the main text.

2. The paper presentation is bad (see more points below).

3. There exists an unfair comparison with baselines in the experiments (see elaborations below).

I elaborated on the specific points below:
[clarity] 
The paper presentation is very bad. I have trouble understanding some parts of the model designs and the problem setting. The authors are strongly encouraged to explicitly list out what are the supervised signals in the problem setting in both fully supervised and weakly supervised cases. Please clearly define every math notation before using them. I list out several key confusions:

1.1. What is the detection score s_h defined in sec 3.1? Ground truth bounding box coordinates? How many scores are there?

1.2. Define relatedness and interaction scores in the introduction. 

1.3. Add vg defined in sec3.2.1 to fig1. Also including several other key variables in Fig1 would help readers understand the designs better.

1.4. I have trouble understanding this relatedness score. Up to 3.2.5, there is no definition of sb. What is the relatedness score exactly? Is this binary score given as ground truth? If so, the authors should spell it out at the beginning of the paper.

1.5. In sec3.3, it talks about self-taught relatedness classification loss. What is exactly given in this case? If binary cross-entropy loss is used and bm is given, this relatedness acts as the ground truth and it should be explicitly introduced in the problem setting in Sec 3.1.

1.6. what is given in the weakly supervised setting? What are the image-level annotations only? Go back to the tuples O defined in Sec3.1, what is exactly missing or not given in a weakly supervised setting?


1.7. In table1, what are APS1 and S2 in the rightmost column?

[other questions]
2. I am familiar with scene graph generation. HOI seems to be a subset of that problem. Would this weakly supervised method generalize to scene graph generation?

3. I found the comparisons of methods are unfair and the results are puzzling. In table 1, baselines are using different backbones and object detectors. If so, it is very hard to say whether the performance improvement is simply because of the change of backbones and detectors, or because the proposed weakly supervised methods are efficient.

4. Utilizing the external knowledge from CLIP is an interesting idea! However, would this put other methods at disadvantage as CLIP has been presented with more data than existing methods?
clarity,_quality,_novelty_and_reproducibility: I have problems with the paper presentation (points above in weakness).
The paper is hard to follow. Many notations are used without clear definitions beforehand.
summary_of_the_review: The paper presentation is bad which creates troubles for me to fully understand the method section.
This subsequently creates problems for me to appreciate the novelty of the method.
I am also having issues with the baseline comparisons. I do not think it is fair to some extent (see elaborations above).

I would vote for REJECT. However, it is possible that I might revise my ratings based on the authors' feedback after the rebuttal period.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: none as far as i can see
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper933/Reviewer_RZBb
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: For weakly-supervised HOI identification, this paper provides a bi-level knowledge integration technique that integrates the prior information from CLIP. To augment the HOI representation using an image-wise HOI recognition network and a pairwise knowledge transfer network,  the authors specifically use CLIP textual embeddings of HOI labels as a relational knowledge bank. To get over unclear human-object connection, the authors also suggest including a self-taught binary pairwise relatedness classification loss. Finally, in a weakly supervised context, the proposed method surpasses the previous state of the art on the HICO-DET and V-COCO benchmarks.
strength_and_weaknesses: Strength
- The method achieves the sota performance among weak-supervised hoi detection benchmark.
- Utilizing the prior knowledge from encoded prompt feature bank to enhance the human-object feature is interesting

Weaknesses
- Literature review should be enhanced, e.g., [1].
[1] Weakly Supervised Learning of Interactions between Humans and Objects
- From my humble view,  the score fusion is much better since union region brings lots of noise. Therefore, using the union region feature as  query to get attention score still be noisy.
- It seems that over 50 percent images of HICO—DET only contains one human-object pair (pls correct me if wrong). Therefore, HICO-DET is probably not the best choice to validate idea, especially for ablation study. 
- From my prospective, the main contribution of the paper is exploring how to introduce CLIP text prior but it is not strong related to weakly supervised learning. How about the result if taking into account fully supervision? 

clarity,_quality,_novelty_and_reproducibility: Quality, clarity and originality are all good to me.
summary_of_the_review: This paper looks good to me. The main concern is that CLIP text prior seems not to be strongly related to weakly supervised learning considering pseudo labels contribute only a little improvement.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper933/Reviewer_q7LZ
confidence: 1: You are unable to assess this paper and have alerted the ACs to seek an opinion from different reviewers.
summary_of_the_paper: This paper proposes a bi-level knowledge integration strategy that incorporates the prior knowledge from CLIP for weakly-supervised HOI detection.

This paper also exploits CLIP textual embeddings of HOI labels as a relational knowledge bank, which is adopted to enhance the HOI representation with an image-wise HOI recognition network and a pairwise knowledge transfer network.

This paper further proposes the addition of a self-taught binary pairwise relatedness classification loss to overcome ambiguous human-object association.


strength_and_weaknesses: The proposed approach achieves the new state of the art on both HICO-DET and V-COCO benchmarks under the weakly supervised setting.
clarity,_quality,_novelty_and_reproducibility: There is a lack of contribution and novelty.
summary_of_the_review:  It lacks any innovative contribution.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper933/Reviewer_frLC
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This work addressed the problem of HOI detection tasks under a weakly-supervised learning paradigm. The key inspiration is to leverage the rich vision-language model (i.e., CLIP) to improve the model learning on highly noisy and long-tailed HOI datasets. The key novelties are the novel local relation-specific HOI representation and the relatedness classification terms. The experimental results show that the proposed approach improves the corresponding baseline. 
strength_and_weaknesses: Strength:
+ This work acknowledges the rich information encoded in the pre-trained CLIP model and proposes several modules to better leverage such information for the HOI detection task. This include (1) HOI knowledge bank, and (2) knowledge transfer network. 
+ Design a local relation-specific HOI representation that transfer relation-level semantic knowledge from CLIP to pair-wise representations. 
+ Design a model-guided relatedness classification loss term (i.e., some form of trustworthiness score) to address the problem in weakly supervised learning. 
+ The paper conducted several ablation studies to empirically show the efficacy of the proposed modules. Improvement on rare classes is more significant, demonstrate the benefit of having strong pre-trained model.

Weakness:
- The technical novelty of this work is relatively weak. The relatedness classification loss term is very straightforward and used in earlier work (e.g., using a threshold to remove low-quality prediction). In addition, this work largely benefits from the strong CLIP model (e.g., HOI knowledge bank, visual encoder, and text encoder). Effectively, the long-tailed issue is partially addressed as the encoder is trained with external dataset. The strength is that this work shows how to better leverage the CLIP model in the context of HOI detection. But I would argue that the technical novelty is welcome but limited.  

Other concerns:
- Introduction highlights that JOI annotations are prone to image-level labelling errors. Can the author elaborate on how can the model deal with labelling errors if it is presented in training data? 
- In Fig 1, please clearly annotate which are the HOI recognition network (is it both local and global branch?)
- In Section 3.2, "The global branch includes a backbone network (Sec.3.2.1) that generates human-
object proposals and extracts image features". The human-object proposals seems to be generated with a pre-trained object detector and provided as an input to global brunch, please clarify this. 
- Please comment about the baseline model in ablation study. It seems that the performance is actually higher than the PPR-FCN (which use the same backbone) in Table 1. Is there some clear factors that give the baseline a higher performance?
- Please change "strongly supervised learning" to "supervised learning". The term is well established in the community in the past decades. 
clarity,_quality,_novelty_and_reproducibility: The proposed approach is detailed in the manuscript and should be able to reproduce without concerns. The author promise to release all codes and trained model for reproducibility purposes. 
summary_of_the_review: Overall, this paper is well-written and easy to follow. Generally, the work is highly beneficial from the strong visual-language model, where the object semantics are better learned in the pre-trained model. The key contribution of this work is to better exploit CLIP embedding for learning relational information. The designed model is intuitive and provides insights on how to leverage such information for a specific task. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A.
recommendation: 6: marginally above the acceptance threshold