Paper: Zero-shot Human-Object Interaction Detection via Conditional Multi-Modal Prompts (ICLR-2024)

Reviewer comment (Novelty):
Overall, this work is very similar with the following ICCV2023 paper, including the overall framework, the conditional vision prompts and the learnable modules. What's the difference between the proposed method and the ICCV2023 paper. A1: Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory, ICCV2023.

Author response:
**1. The differences between the proposed method and the ICCV2023 paper.**

Thank you once again for your careful consideration of our work.
We propose an approach for zero-shot HOI detection that addresses the issue of **performance degradation between seen and unseen classes.** While the concurrent ICCV2023 paper focuses on improving efficiency when adapting CLIP to HOI detection, our method prioritizes generalizability. Our approach differs from it in several key aspects:

+ **Conditional Vision Prompts for Region-Level Interactiveness Knowledge:**
While both the ICCV paper and our approach utilize prior knowledge from DETR, our method specifically focuses on leveraging instance-level visual priors to distinguish between non-interactive human-object pairs and interactive ones. This differentiation is crucial, especially in zero-shot HOID settings where unseen HOI concepts are encountered.

+ **Learnable Embeddings as Context Words:**
To augment the CLIP text encoder's representation capabilities for actions, we introduce learnable embeddings as context words, i.e., an additional input known as language prompts.

+ **Regularization Loss for Generalizability:**
Since the hand-engineered prompts outperform the learnable soft prompts for the zero-shot setting, then, in order to avoid overfitting to the seen classes and strengthen generalizability, we further propose that the learnable ones should be trained so that they can be correctly classified in language space where the class weights are given by the hand-engineered prompts. This is achieved through the **constrastive loss** in Equation (6) of the main text. 

+ **Exclusive Reliance on Language for Interaction Classification:** For interaction recognition, the ICCV paper proposes a key-value mechanism storing both visual and linguistic knowledge. In contrast, we argue that visual knowledge is challenging to generalize to unseen classes. In other words, the visual features of unseen classes are hard to acquire. As a result, we exclusively employ language for interaction classification.

Reviewer comment (Experiments):
This work only presents the HOI results using zero-shot settings. What's the result using the typical experimental settings?

Author response:
**3. Experimental results using the typical settings.**

We are truly grateful for this insightful consideration. To further validate the effectiveness of our method, we conduct experiments on the typical experimental settings on both the HICO-DET and V-COCO datasets. Empirically, we find that our method achieves competitive results on the default setting on both datasets. Notably, our method achieves sota on rare classes on HICO-DET, aligning with our methodology's design since rare classes are typically challenging to detect, similar to the scenario of detecting unseen interactions.


|  Method  | Full  | Rare  | Non-Rare  |
|   ----   | ----  | ----  |  ----   |
| GEN-VLKT | 33.75 | 29.25 |  35.10  |
| HOICLIP  | 34.54 | 30.71 |  35.70  |
| Ours     | 34.26 | 32.22 |  34.86  |


|  Method  | AP(Scenario 1) | AP(Scenario 2) |
|   ----   | ----  | ----  | 
| GEN-VLKT | 62.4 | 64.4 |
| HOICLIP  | 63.5 | 64.8 |
| Ours     | 57.8 | 63.2 |

Reviewer comment (Novelty):
Some important works from CVPR2023 are missing. Besides, the formats of some references are not consistent.

Author response:
**4. Missing related works.**

Thank you for bringing this to our attention. We will incorporate discussions about the literature on HOI in CVPR2023 [1-4] into the related work section, and revise the formats of references in the final version.

[1] Open-Category Human-Object Interaction Pre-Training via Language Modeling Framework

[2] Relational Context Learning for Human-Object Interaction Detection

[3] Category Query Learning for Human-Object Interaction Classification

[4] ViPLO: Vision Transformer Based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection

Reviewer comment (Novelty):
Overall, the paper mainly borrows the popular adapt large models and prompt strategy for down-stream tasks. Considering that there are massive similar approaches in other fields, the novelty is limited. However, the reviewer still thinks it is beneficial for the development of zero-shot HOI detection.

Author response:
**1. The novelty of our method.**

Thank you once again for your careful consideration of our work.
CoOp[1] first proposes to use context tokens as language prompts in the image classification task. Co-CoOp [2] proposes to explicitly condition language prompts on image instances. Recently, other approaches for adapting V-L models through prompting have been proposed. For example, ProGrad [3] utilizes gradient matching, while TTTuning [4] employs test time tuning.

Different from them, our innovations mainly lie in the following two aspects:

+ **Introducing Multi-Modal Prompts:** Previous methods have mainly focused on unimodal solutions, which means adding prompts either in the vision or language branch, particularly in the language branch when adapting Vision-Language pretrained models. We have discovered that unimodal prompts are not effective for the HOID task. This is because the HOID task requires not only the generalizability of the text branch through language prompts but also the inclusion of vision prompts to enhance the vision encoder's understanding of region-level spatial relationships. In light of this, we first propose multi-modal prompts in zero-shot human-object interaction detection to improve visual-language feature alignment and zero-shot knowledge transfer.

+ **Leveraging Prior Knowledge for Vision and Language Prompts:** Previous methods involving conditional prompts, like CoCoOp, generate an input-conditional token for each image and only apply it in the text branch. Our method differs from it in the following two aspects: (1) First, we propose a vision condition that incorporates spatial information for each image, making it more fine-grained and suitable for the HOID task in the vision branch. (2) Second, we introduce a language condition as a regularization term to prevent the model from deviating excessively from the CLIP text feature space. 


[1] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models."

[2] Zhou, Kaiyang, et al. "Conditional prompt learning for vision-language models."

[3] Zhu, Beier, et al. "Prompt-aligned gradient for prompt tuning."

[4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models."

Reviewer comment (Novelty):
To some extent, the core idea is similar to CoOp and the following work Co-CoOp, though this paper also incorporates the visual prompts and has made some HOI-specific designs.

Author response:
**1. The novelty of our method.**

Thank you once again for your careful consideration of our work.
CoOp[1] first proposes to use context tokens as language prompts in the image classification task. Co-CoOp [2] proposes to explicitly condition language prompts on image instances. Recently, other approaches for adapting V-L models through prompting have been proposed. For example, ProGrad [3] utilizes gradient matching, while TTTuning [4] employs test time tuning.

Different from them, our innovations mainly lie in the following two aspects:

+ **Introducing Multi-Modal Prompts:** Previous methods have mainly focused on unimodal solutions, which means adding prompts either in the vision or language branch, particularly in the language branch when adapting Vision-Language pretrained models. We have discovered that unimodal prompts are not effective for the HOID task. This is because the HOID task requires not only the generalizability of the text branch through language prompts but also the inclusion of vision prompts to enhance the vision encoder's understanding of region-level spatial relationships. In light of this, we first propose multi-modal prompts in zero-shot human-object interaction detection to improve visual-language feature alignment and zero-shot knowledge transfer.

+ **Leveraging Prior Knowledge for Vision and Language Prompts:** Previous methods involving conditional prompts, like CoCoOp, generate an input-conditional token for each image and only apply it in the text branch. Our method differs from it in the following two aspects: (1) First, we propose a vision condition that incorporates spatial information for each image, making it more fine-grained and suitable for the HOID task in the vision branch. (2) Second, we introduce a language condition as a regularization term to prevent the model from deviating excessively from the CLIP text feature space. 


[1] Zhou, Kaiyang, et al. "Learning to prompt for vision-language models."

[2] Zhou, Kaiyang, et al. "Conditional prompt learning for vision-language models."

[3] Zhu, Beier, et al. "Prompt-aligned gradient for prompt tuning."

[4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models."

Reviewer comment (Experiments):
The proposed method achieves smaller gap between seen and unseen category. According to Tab.1, PD is larger in RF-UC setting. Could you explain it? Moreover, do you have any ablation studies to check which module is more important for reducing the PD.

Author response:
**2. Discussion on the PD.**

+ **PD is larger in RF-UC setting.**

The PD could be attributed to the semantic gap between the seen and unseen classes. In RF-UC setting, the rare classes are filtered out during the training process. Given that rare classes in real-life scenarios are inherently infrequent and semantically challenging, they are not well-understood by the original CLIP text embedding space, leading to a larger PD.


+ **Ablation studies on which module is more important for reducing the PD.**

We propose a constraint for each modality to alleviate the performance degradation. Following the reviewer's suggestion, we further conduct ablation studies to check which condition is more important for reducing the PD in RF-UC setting. As shown in the table below, we find that the language condition holds greater significance for reducing PD, showing the efficacy of utilizing hand-crafted prompts for regularization.


|         Method         | Unseen | Seen | Full | PD |
|          ----          | ----  | ----  |  ---- |  ---|
|  w/o Vision Condition  | 27.97 | 32.95 | 31.95 | 4.98 |
| w/o Language Condition | 26.27 | 32.60 | 31.71 | 6.33 |
|      PID (Ours)        | 28.82 | 33.35 | 32.45| 4.53 |

Reviewer comment (Theory):
The paper aims to achieve verb-agnostic prior knowledge. Could you explain why the verb-agnostic feature is helpful for interactiveness-aware features? By the way, the local spatial structure is actually verb-dependent, e.g., different action pattern demonstrates different relative human-object positions. Thus, capturing local spatial structure seems to contradict to verb-agnostic representations.

Author response:
**3. Verb-agnostic knowledge for interactiveness-aware feature extraction.**


+ Verb-agnostic knowledge is independent of the specific actions or verbs involved in a given situation. Similarly, interactiveness judgment is inherently binary, devoid of the necessity for specific verb-related information. As a result, we propose to utilize verb-agnostic knowledge to facilitate interactiveness-aware feature extraction.


+ We would like to kindly remind you that the local spatial structure we utilize only involves the spatial information of the objects instead of the spatial arrangement of human-object pairs. Since an interaction may manifest at virtually any position within an image (taking into account the different shooting angles of the images), we argue that the local spatial structure is inherently verb-agnostic.

Reviewer comment (Novelty):
The whole framework seems like a combination of multiple existing modules, e.g., DETR, CoOp/CoCoOp style prompts. The novelty and the motivation behind each of the design are unclear.

Author response:
**1. The novelty and the motivation behind each module.**

We concur with your perspective that leveraging vision foundation models for the HOI detection task is a recent trend. In this work, we aim to apply CLIP for the HOI detection task through decoupled multi-model prompts:

+ **Conditional Vision Prompts for Region-Level Interactiveness Knowledge:** Since CLIP is originally designed for image-level recognition, we design vision prompts for extracting feature maps with region-level pair-wise interactiveness knowledge. Leveraging the instance-level prior knowledge from DETR, our encoder is guided to allocate heightened attention to regions with potential interactions.

+ **Learnable Embeddings as Context Words:** To augment the CLIP text encoder's representation capabilities for actions, we introduce learnable embeddings as context words, i.e., an additional input known as language prompts.

+ **Regularization Loss for Generalizability:** In order to utilize the original structure of CLIP's text embedding space and avoid overfitting to the seen classes, we further introduce a regularization loss. This loss prevents the model from becoming too specialized on the seen classes and strengthens its generalizability when dealing with unseen classes.

Reviewer comment (Experiments):
In 4.2, the authors mentioned that the DETR used for detecting all the humans and objects in the first stage is fine-tuned on the whole HICO-DET dataset. Does the 'whole' here mean both training and validation sets? If so, this is a weird setting as previous works (including Bansal et al. 2020 and Hou et al. 2020 that the authors claimed) never fine-tuned their detectors on the validation set, which would lead to extremely unfair comparison since the detector can significantly affect the overall performance.

Author response:
**2. The data split used for fine-tuning DETR.**

To clarify, in our approach, we fine-tune DETR **only on the training set**. The 'whole' in 4.2 means all the instance-level annotations, without distinction for instances engaged in unseen interactions, which is the same as previous works (including Bansal et al. 2020 and Hou et al. 2020). We apologize for the confusion caused by the earlier statement and will revise it in the final version of our work.

Reviewer comment (Experiments):
The experiments are only conducted on a single dataset. Why the method is not tested on V-COCO?

Author response:
**3. Experimental dataset.**

We do not conduct experiments on v-coco dataset since the official evaluation utilities do not support a zero-shot setting.

To further validate the effectiveness of our method, we conduct experiments on the typical experimental settings on both the HICO-DET and V-COCO datasets. Empirically, we find that our method achieves competitive results on the default setting on both datasets. Notably, our method achieves sota on rare classes on HICO-DET, aligning with our methodology's design since rare classes are typically challenging to detect, similar to the scenario of detecting unseen interactions.


| Method   | Full  | Rare  | Non-Rare  |
|   ----   | ----  | ----  |  ----   |
| GEN-VLKT | 33.75 | 29.25 |  35.10  |
| HOICLIP  | 34.54 | 30.71 |  35.70  |
| Ours     | 34.26 | 32.22 |  34.86  |


| Method | AP(Scenario 1) | AP(Scenario 2) |
|   ----   | ----  | ----  | 
| GEN-VLKT | 62.4 | 64.4 |
| HOICLIP  | 63.5 | 64.8 |
| Ours     | 57.8 | 63.2 |

Reviewer comment (Writing):
The conclusion part lacks objective reflections on the deficiencies of this study and future prospects for improvements.

Author response:
**4. Reflections on the Study's Deficiencies and Future Directions.**

Thank you for your constructive feedback. In light of your feedback, we will expand our conclusion section to incorporate the following points:

+ **Deficiencies:** In this paper, we employ CLIP for HOI detection via conditional prompt learning. However, a limitation of this approach is its dependency on the input resolution of CLIP, specifically constrained to 224 $\times$ 224 pixels, which may impede the accurate detection of interactions involving small objects.

+ **Future Prospects:** Considering the quick-evolving trends in Vision Language Models (VLM), and Large Language Models (LLM), in future, there are several promising directions: (1) We aim to employ VLMs for **flexible input resolutions** to deal with HOIs with different scales or distances. (2) We will explore to **integrate the semantic spaces of VLMs and LLMs** to improve the model's comprehension of interactions, especially for unseen interactions, aiming for a more generic HOI detection system in the real-world scenario. (3) We will explore efficient **knowledge transfer** methods in the future to facilitate the integration and collaboration of specialized HOI detectors powered by VLMs or LLMs, thereby reducing the inference costs of large models.

Reviewer comment (Experiments):
Does $\mathbb{A}$ contain unseen verbs? If so, can this model recognize HOIs that are not present in HICO-DET? In other words, if I want to detect a HOI using this model, does the corresponding interaction verb have to be included in $\mathbb{A}$?

Author response:
**b) The capability of our method:** Following the previous experimental protocol, $\mathbb{A}$ contains both seen and unseen verbs. However, it is worth noting that since our method treats verb labels in a semantic manner, embedding them into a unified visual-and-text space as opposed to employing traditional one-hot labels, our method can detect any HOI in the wild given its name.

Reviewer comment (Evaluation):
It is not clear why the vision prompts are useful for zero-shot HOI detection. Concretely, the visual features extracted in this model do not seem to be very special compared to those in most two-stage based HOI detectors.

Author response:
In contrast to most previous two-stage HOI detectors, our approach integrates prior knowledge from the pretrained detector directly into the image encoder's architecture, in addition to applying it to the output of the image encoder through ROI-Align. This integration **empowers the vision encoder with a heightened awareness of regions with greater potential for interactions**, thereby guiding the image encoder to produce feature maps that are more sensitive to interactiveness, which is crucial for zero-shot HOI detection.