Paper: Interaction-centric Hypersphere Reasoning for Multi-person Video HOI Recognition (ICLR-2024)

Reviewer comment (Writing):
Some claims are ambiguous, please give more explanations: usually lack of ability to capture global context information: which works and why?

Author response:
* ***Capturing global context information.*** All of the CNN-based works listed  Section 2 "Related Works-HOI detection" lacks the ability to capture global context information due to the inherent local nature of the CNN structure.

Reviewer comment (Experiments):
How to handle the multi-label classifications given hyperspheres, and the long-tailed bias?

Author response:
* ***Multi-label and long-tail.*** 

     (1) At each time step, there is only one sub-activity for each person. Therefore there is no multi-label problem in HOI recognition task. 

     (2) We deal with long-tail problem with focal loss $\mathcal{L}_{cls}$, which is illustrated in Section 4.3.

Reviewer comment (Theory):
Why is it called a hypersphere? What is the meaning of hypersphere and does it have any theoretical implications? Hypergraphs and graphs are different theories. In this paper, what is the difference between the concept of a hypersphere and a sphere?

Author response:
* ***Hypersphere.*** 

    **(1)** Hypersphere indicates a high-dimensional sphere, indicating the boundary of a high-dimensional ball. We call "hypersphere" in this work because the feature dimensions in Equation (6) are high-dimensional vectors (larger than 3). 

    **(2)** The interaction-centric hypersphere is crafted to introduce a **structured bias** of HOI into our model. Specifically, this bias asserts that **human-object pairs adhere to their respective interaction classes**. Consequently, the model is compelled to make predictions within the confines of this structured bias. It is essential to clarify that the hypersphere serves merely as a means of visualizing this bias. Alternately, various other techniques may be employed for visualizing the structure bias. Hence, our paper's fundamental proposition is to instill the desired structure bias, emphasizing that hypersphere is just one illustrative manifestation of this broader concept.

Reviewer comment (Novelty):
The hypersphere appears to be used to predict interaction probabilities. Please explain how it differs from traditional classifiers. This explanation is necessary since this hypersphere is the key idea.

Author response:
* ***Differnece of hypersphere from traditional classifier.*** As we explained in the above section, hypersphere is a way to visualize the introduced structured bias underlying HOI. This structure bias forces the model to make predictions within the confines of this structured bias. Compared to tranditional classifiers which usually employ multi-layer perceptron (MLP) to generate prediction logits, hypersphere introduce a strong prior into model prediction, which is abscent in traditional classifiers. Moreover, hypersphere do not introduce any training parameters like traditional classifiers, featuring higher computational efficiency.

Reviewer comment (Writing):
What is the HOI manifold structure mentioned several times in this paper? It is hard to understand. Is it related to Riemannian geometry? Please elaborate with a clear explanation.

Author response:
* ***HOI manifold structure.*** In mathematics and machine learning, a "manifold structure" refers to a topological space that locally resembles Euclidean space. In other words, in the vicinity of any point on the manifold, the space behaves like a flat, n-dimensional Euclidean space. Manifolds can have various dimensions and shapes, and they are often used to represent complex data structures or patterns in high-dimensional spaces. In this work, *HOI manifold structure* specifically describes the topological relationship between **human-object paris** and **interaction class**, which is a structure bias that we introduce into our model.

Reviewer comment (Writing):
Is there any connection between complex HOI and multi-person HOI, as mentioned in “To enhance the awareness of complex HOI structures in our representations, we introduce the Context Fuser (CF)…”?

Author response:
* ***Complex HOI.*** Complex HOI incorporates multi-person HOI, meaning there would be interdependencies between human and object in single-person cases, or interdependencies between human and human under multi-person scenarios.

Reviewer comment (Writing):
In “To facilitate interaction reasoning, we place the ISR module on top of the context fuser module, yielding entity representations capable of capturing interaction transition dynamics,” does entity representation represent the characteristics of human and object entities, or the transition characteristics of the same person or object between different states? This sentence is confusing about what exactly it represents.

Author response:
* ***Entities representations.*** Human and object are called "entity", so entity representations denote representation of human-object pair.

Reviewer comment (Presentation):
In Figures 2 and 3, it is better to replace the letters with specific features. Using a large number of letters is too unintuitive and makes it difficult for readers to understand.

Author response:
* ***Figure 2 and 3.*** We have added annotations  for the letters in the figures.

Reviewer comment (Writing):
In “We follow 2G-GCN to extract feature of humans and objects from backbone network,” you used 2G-GCN to capture features, but the input {vt}t=1T seems to be a clip. Is the input of 2G-GCN a video? Is the output the characteristics of people and objects in each frame of the video? Do ZH and ZO represent the characteristics of people and objects in each frame, or the characteristics of the entire video? The reviewer is totally confused.

Author response:
* ***Follow 2G-GCN to extract feature.*** {${v_t}$}$_{t=1}^T$ denote video frames, whose features are extracted from 2G-GCN model. We use CLIP model to extract text features, not video frame features. The input of 2G-GCN is the same with our model, which is a video. The output is the human sub-activity class at each time step. $z_H$ and $z_O$ are extracted features of human and object from 2G-GCN, respectively.

Reviewer comment (Experiments):
Although three datasets are compared, the algorithm is not fully verified. Why is the VidHOI dataset not used for comparison, since it is a well-known video-based human-object interaction dataset?

Author response:
* ***Experiment on VidHOI dataset.*** VidHOI is used for video HOI detection task, while our paper works on video HOI recognition, which is a different task. Consequently, the VidHOI dataset is not suitable for comparison in our study.

Reviewer comment (Experiments):
The main focus of the paper is HOI recognition for multi-person videos. In the experiment, there is only 1 multi-person dataset used for evaluation but 2 single-person datasets. Showing model performance on different multi-person datasets will help strength the claims in the paper.

Author response:
* ***Experiment datasets.*** In our multi-person video HOI recognition experiment, we exclusively leverage the MPHOI dataset, as it stands as the sole dataset tailored for multi-person video HOI recognition at present. Aligning with the SOTA methodology, specifically 2G-GCN, which also centers on multi-person video HOI recognition, our experimental design mirrors theirs by utilizing the MPHOI dataset. Furthermore, 2G-GCN augments its investigation with two single-person datasets. Hence, our experimental settings maintain congruence with 2G-GCN to facilitate a meaningful comparative analysis.

Reviewer comment (Theory):
After reading the paper, there is still a lack of proof or explanation about why an interaction-centric hypersphere will help in the task theoretically. The ablation study does not show an ablation study on it.

Author response:
* ***Why interaction-centric hypersphere helps?*** The interaction-centric hypersphere is crafted to introduce a **structure bias** of HOI into our model. Specifically, this bias asserts that **human-object pairs adhere to their respective interaction classes**. Consequently, the model is compelled to make predictions within the confines of this structured bias. It is essential to clarify that the hypersphere serves merely as a means of visualizing this bias. Alternately, various other techniques may be employed for visualizing the structure bias. Hence, our paper's fundamental proposition is to instill the desired structure bias, emphasizing that hypersphere is just one illustrative manifestation of this broader concept.

Reviewer comment (Experiments):
In the method, the context fuser and interaction state reasoner extract CLIP features for the representation. Ablation on the features is necessary to test if the feature is important or over-complex where a simple binary feature is enough. For example, in the interaction state reasoner, the two possible states “continue” and “stop” can be represented by binary labels or simpler features of lower dimensions compared with CLIP.

Author response:
* ***Ablating CLIP features.*** We conduct ablation studies in the revised peper by utilizing random initialization for interaction features and context features, instead of using CLIP representations. Ablation study results in Table 1 (highlighted in red in the revised paper) show that random initialization results in around 10% in the $F_1$ scores in MPHOI dataset, while still higher than SOTA method 2G-GCN around 15%. Furthermore, Tables 2 and 3 showcase ablation study outcomes on the CAD-120 and Bimanual Actions datasets, indicating a drop of less than 3% in $F_1$ scores with random initialization. Notably, these scores remain competitive, if not superior, to the 2G-GCN model. Thus, our findings suggest that extracting CLIP features is not imperative for achieving strong performance. We also show the results below for better understanding:

  |***MPHOI*** | $F_1 @10$  |  $F_1 @25$  |  $F_1 @50$  |  
  | ------------- | ------------- | --------------  | -------------- |
  | 2G-GCN     |  68.6$\pm$10.4 |  60.8$\pm$10.3 |  45.2$\pm$6.5 | 
  |**Ours**       | **91.6**$\pm$**0.9** |  **84.5**$\pm$**2.6** |  **67.7**$\pm$**2.2** | 
  |Ours ($\Delta$ CLIP+BLIP)  |  80.0$\pm$6.7  |   73.0$\pm$9.9  |   55.6$\pm$7.4  | 

  |***CAD-120*** | $F_1 @10$               |  $F_1 @25$  |  $F_1 @50$  |  
  | -------------    | -------------              | --------------  | -------------- |
  | 2G-GCN     |  89.5$\pm$1.6 |  87.1$\pm$1.8 |  76.2$\pm$2.8 | 
  |**Ours**          | **90.7**$\pm$**2.9** |  **88.1**$\pm$**2.8** |  **77.6**$\pm$**4.7** | 
  |Ours ($\Delta$ CLIP+BLIP)  |  89.4$\pm$2.3  |   85.5$\pm$3.9  |   74.9$\pm$5.7  | 

  |***Bimanual Actions*** | $F_1 @10$  |  $F_1 @25$  |  $F_1 @50$  |  
  | ------------- | ------------- | --------------  | -------------- |
  | 2G-GCN     |  **85.0**$\pm$**2.2** |  82.0$\pm$2.6 |  69.2$\pm$3.1 | 
  |**Ours**       | 85.0$\pm$2.5 |  **82.9**$\pm$**2.9** |  **74.2**$\pm$**4.3** | 
  |Ours ($\Delta$ CLIP+BLIP)  |  84.3$\pm$1.4  |   81.8$\pm$1.8  |   73.2$\pm$2.7  |

Reviewer comment (Experiments):
In 4.3.1 Model inference, during model inference, the interaction probability is predicted on each frame. It is not clear who is in interaction if there are multiple people in the video. Interaction prediction for each person is more detailed and straightforward.

Author response:
* ***Interaction prediction for each person.***  In the case of multi-person video HOI recognition, our model is able to predict interaction class for each person, as qualitative examples shown in Figure 5 and Figure 7 in the paper.

Reviewer comment (Experiments):
Based on the question above, ablation studies with methods of HOI detection on images are necessary. HOI detection can detect interaction for each person. If there are multiple people, the results for comparison from the HOI detection is whether there is any interaction from all people or not.

Author response:
* ***Ablation studies with methods of HOI detecion.*** Our emphasis lies specifically on the **HOI recognition** task rather than HOI detection. These two tasks differs in both problem formulation and technical challenges. As articulated in Section 4.1 "Problem Formulation," our focus is on predicting the HOI class present in a given video frame. It is essential to clarify that our paper does not address HOI detection, which involves generating bounding box predictions for each individual and object in the scene, a distinct task from the scope of our work.

Reviewer comment (Evaluation):
In the Conclusion Sec, it mentions that the method outperforms SOTA on the multiple-people dataset but is on par with the single-person dataset. Is it because in the single-person video, there is only one person so the interaction prediction is determined by that single person? But for multi-person videos, the model does not need to predict correctly for each person to get the correct answer for the image.

Author response:
* ***Performacne on single-person videos.*** The relatively modest improvement of our model's performance on single-person videos compared to the SOTA method in contrast to its notable gains in multi-person videos can likely be attributed to the absence of a dedicated context fuser module for single-person scenarios, as illustrated in Figure 2. As evident from the outcomes presented in Table 1, the contextual fusion module plays a pivotal role in enhancing performance in multi-person settings. Consequently, we hypothesize that the less remarkable results observed in single-person instances are attributable to the lack of a context fuser module.

Reviewer comment (Experiments):
The primary weakness of the proposed method is the unfair comparisons with other video HOI methods and the lack of thorough ablation studies. The Context Fuser employs large Vision-Language Models (VLMs), CLIP and BLIP, pre-trained on big data, while ASSIGAN and 2G-GCN do not. In Table 1, removing Context Fuser results in a notable decline in $F_1@10$. This raises suspicions that the significant performance enhancement could be largely attributed to the text-image alignment capabilities inherent in large VLMs rather than the proposed Context Fuser. The performance is below the benchmark set by Qiao et al., 2020, in the absence of the Context Fuser. The paper lacks critical ablation studies to disentangle the contributions of the VLMs and the proposed method.

Author response:
* ***Comparison with other methods and ablation studies.***: In our revised paper, we employ ablation studies to elucidate the impact of excluding prominent Vision-Language Models (VLMs) CLIP and BLIP (highlighted in red in the revised paper). In the ablation studies, we adopt random initialization for the interaction feature $z_I$ and context feature $z_C$. The results, detailed in Table 1, indicate approximately 10% in $F_1$ scores within the MPHOI dataset upon removing CLIP and BLIP. Despite this reduction, our scores remain superior to the SOTA method 2G-GCN by approximately 15% $F_1$ score. Moreover, Tables 2 and 3 reveal that in the CAD-120 and Bimanual Actions datasets, the omission of CLIP and BLIP yields a drop of less than 3% in the $F_1$ scores, still comparable to or surpassing the performance of the 2G-GCN model. These findings underscore that the comparison between our model and SOTA methods is equitable, as the reliance on large VLMs proves non-essential for achieving commendable performance. Rather, it is the intricately designed structure of our model that substantiates the substantial enhancement in performance. We also show the results below for better understanding:

  |***MPHOI*** | $F_1 @10$  |  $F_1 @25$  |  $F_1 @50$  |  
  | ------------- | ------------- | --------------  | -------------- |
  | 2G-GCN     |  68.6$\pm$10.4 |  60.8$\pm$10.3 |  45.2$\pm$6.5 | 
  |**Ours**       | **91.6**$\pm$**0.9** |  **84.5**$\pm$**2.6** |  **67.7**$\pm$**2.2** | 
  |Ours ($\Delta$ CLIP+BLIP)  |  80.0$\pm$6.7  |   73.0$\pm$9.9  |   55.6$\pm$7.4  | 

  |***CAD-120*** | $F_1 @10$               |  $F_1 @25$  |  $F_1 @50$  |  
  | -------------    | -------------              | --------------  | -------------- |
  | 2G-GCN     |  89.5$\pm$1.6 |  87.1$\pm$1.8 |  76.2$\pm$2.8 | 
  |**Ours**          | **90.7**$\pm$**2.9** |  **88.1**$\pm$**2.8** |  **77.6**$\pm$**4.7** | 
  |Ours ($\Delta$ CLIP+BLIP)  |  89.4$\pm$2.3  |   85.5$\pm$3.9  |   74.9$\pm$5.7  | 

  |***Bimanual Actions*** | $F_1 @10$  |  $F_1 @25$  |  $F_1 @50$  |  
  | ------------- | ------------- | --------------  | -------------- |
  | 2G-GCN     |  **85.0**$\pm$**2.2** |  82.0$\pm$2.6 |  69.2$\pm$3.1 | 
  |**Ours**       | 85.0$\pm$2.5 |  **82.9**$\pm$**2.9** |  **74.2**$\pm$**4.3** | 
  |Ours ($\Delta$ CLIP+BLIP)  |  84.3$\pm$1.4  |   81.8$\pm$1.8  |   73.2$\pm$2.7  |

Reviewer comment (Novelty):
There is a relevant ICLR’23 paper you should refer to, Gao et al., ICLR’23, which also uses large VLMs for the HOI problem. Unlike the fixed prompt used in Context Fuser, Gao et al. delve into learnable prompts.  
>Gao, K., Chen, L., Zhang, H., Xiao, J. & Sun, Q. Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection. _ICLR_ (2023).

Author response:
* ***Reference.*** Thanks for the suggestion, we have added the reference in the revised version.