W EAKLY - SUPERVISED HOI D ETECTION VIA P RIORGUIDED B I - LEVEL R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Human object interaction (HOI) detection plays a crucial role in human-centricscene understanding and serves as a fundamental building-block for many vision tasks.
One generalizable and scalable strategy for HOI detection is to useweak supervision, learning from image-level annotations only.
This is inherentlychallenging due to ambiguous human-object associations, large search space ofdetecting HOIs and highly noisy training signal.
A promising strategy to addressthose challenges is to exploit knowledge from large-scale pretrained models (e g.,
CLIP) but a direct knowledge distillation strategy (Liao et al., 2022) does not perform well on the weakly-supervised setting.
In contrast, we develop a CLIP-guided
HOI representation capable of incorporating the prior knowledge at both imagelevel and HOI instance level, and adopt a self-taught mechanism to prune incorrecthuman-object associations.
Experimental results on HICO-DET and V-COCOshow that our method outperforms the previous works by a sizable margin, showingthe efficacy of our HOI representation.
I NTRODUCTION
Human object interaction detection aims to simultaneously localize the human-object regions in animage and to classify their interactions, which serves as a fundamental building-block in a wide rangeof tasks in human-centric artificial intelligence, such as human activity recognition (Heilbron et al.
,2015; Tina et al., 2021), human motion tracking (Wafae et al., 2019;
Nishimura et al., 2021) andanomalous behavior detection (Liu et al., 2018; Pang et al., 2020).
Usually, HOI detection adopts a supervised learning paradigm (Gupta & Malik, 2015; Chao et al.
,2018; Wan et al., 2019; Gao et al., 2020; Zhang et al., 2021c).
This requires detailed annotations(i.e. human and object bounding boxes and their interaction types) in the training stage.
However,such HOI annotations are expensive to collect and prone to labeling errors.
In contrast, it is mucheasier to acquire image-level descriptions of target scenes.
Consequently, a more scalable strategyfor HOI detection is to learn from weak annotations at the image level, known as weakly-supervised
HOI detection (Zhang et al., 2017).
Learning under such weak supervision is particularly challengingmainly due to the lack of accurate visual-semantic associations, large search space of detecting HOIsand highly noisy training signal from only image level supervision.
Most existing works (Zhang et al., 2017; Baldassarre et al., 2020; Kumaraswamy et al., 2021) attemptto tackle the weakly-supervised HOI detection in a Multiple Instance Learning (MIL) framework (Ilseet al., 2018).
They first utilize an object detector to generate human-object proposals and then train aninteraction classifier with image-level labels as supervision.
Despite promising results, these methodssuffer from several weaknesses when coping with diverse and fine-grained HOIs.
Firstly, they usuallyrely on visual representations derived from the external object detector, which mainly focus on thesemantic concepts of the objects in the scene and hence are insufficient for capturing the concept offine-grained interactions.
Secondly, as the image-level supervision tends to ignore the imbalancein HOI classes, their representation learning is more susceptible to the dataset bias and dominatedby frequent interaction classes.
Finally, these methods learn the HOI concepts from a candidate setgenerated by pairing up all the human and object proposals, which is highly noisy and often leads toerroneous human-object associations for many interaction classes.
To address the aforementioned limitations, we introduce a new weakly-supervised HOI detectionstrategy.
It aims to incorporate the prior knowledge from pretrained foundation models to facilitatethe HOI learning.
In particular, we propose to integrate CLIP (Radford et al., 2021b), a large-scalevision-language pretrained model.
This allows us to exploit the strong generalization capability of the CLIP representation for learning a better HOI representation under weak supervision.
Comparedto the representations learned by the object detector, the CLIP representations are inherently lessobject-centric, hence more likely to incorporate also aspects about the human-object interaction, asevidenced by Appendix A.
Although a few works have successfully exploited CLIP for supervised HOI detection in the past, experimentally we find they do not perform well in the more challengingweakly-supervised setting (c.f. Appendix.B).
We hypothesize this is because they only transferknowledge at image level, and fail without supervision at the level of human-object pairs.
To this end, we develop a CLIP-guided HOI representation capable of incorporating the priorknowledge of HOIs at two different levels.
First, at the image level, we utilize the visual and linguisticembeddings of the CLIP model to build a global HOI knowledge bank and generate image-level HOIpredictions.
In addition, for each human-object pair, we enrich the region-based HOI features bythe HOI representations in the knowledge bank via a novel attention mechanism.
Such a bi-levelframework enables us to exploit the image-level supervision more effectively through the shared HOIknowledge bank, and to enhance the interaction feature learning by introducing the visual and textrepresentations of the CLIP model.
We instantiate our bi-level knowledge integration strategy as a modular deep neural network witha global and local branch.
Given the human-object proposals generated by an off-the-shelf objectdetector, the global branch starts with a backbone network to compute image feature maps, whichare used by a subsequent HOI recognition network to predict the image-wise HOI scores.
The localbranch builds a knowledge transfer network to extract the human-object features and augment themwith the CLIP-guided knowledge bank, followed by a pairwise classification network to compute theirrelatedness and interaction scores 1 .
The relatedness scores are used to prune incorrect human-objectassociations, which mitigates the issue of noisy proposals.
Finally, the outputs of the two branchesare fused to generate the final HOI scores.
To train our HOI detection network with image-level annotations, we first initialize the backbonenetwork and the HOI knowledge bank from the CLIP encoders, and then train the entire model inan end-to-end manner.
In particular, we devise a novel multi-task weak supervision loss consistingof three terms: 1) an image-level HOI classification loss for the global branch; 2) an MIL-like lossfor the interaction scores predicted by the local branch, which is defined on the aggregate of allthe human-object pair predictions; 3) a self-taught classification loss for the relatedness of eachhuman-object pair, which uses the interaction scores from the model itself as supervision.
We validate our methods on two public benchmarks: HICO-DET (Chao et al., 2018) and VCOCO (Gupta & Malik, 2015).
The empirical results and ablative studies show our method consistently achieves state-of-the-art performance on all benchmarks.
In summary, our contributionsare three-fold: (i) We exploit the CLIP knowledge to build a prior-enriched HOI representation,which is more robust for detecting fine-grained interaction types and under imbalanced data distributions.
(ii) We develop a self-taught relatedness classification loss to alleviate the problem ofmis-association between human-object pairs.
(iii)
Our approach achieves state-of-the-art performanceon the weakly-supervised HOI detection task on both benchmarks.
R ELATED WORKS
HOI detection: Most works on supervised HOI detection can be categorized in two groups:two-stage and one-stage HOI detection.
Two-stage methods first generate a set of human-objectproposals with an external object detector, then classify their interactions.
They mainly focus onexploring additional human pose information (Wan et al., 2019;
Li et al., 2020a; Gupta et al., 2019),pairwise relatedness (Li et al., 2019a;
Zhou et al., 2020) or modeling relations between object andhuman (Gao et al., 2020; Zhang et al., 2021c; Ulutan et al., 2020; Zhou & Chi, 2019), to enhancethe HOI representations.
One-stage methods predict human & object locations and their interactiontypes simultaneously in an end-to-end manner.
They are currently dominated by transformerbased architectures (Carion et al., 2020), which promote HOI representation with multi-scale visualfeatures (Chen & Yanai, 2021;
Kim et al., 2022), leverage external knowledge for representationlearning (Liao et al., 2022; Dong et al., 2022), or decouple instance and interaction detection in acascade manner (Zhang et al., 2021a;b).
Supervised methods show superior performance, but require labor-intensive HOI annotations that are infeasible to obtain in many scenarios.
Thus, in this workwe focus on HOI detection under weak supervision.
Weakly-supervised HOI detection: Weakly-supervised HOI detection aims to learn instancelevel HOIs with only image-level annotations .
(Prest et al., 2011) learns a set of binary actionclassifiers based on detected human-object pairs, where human proposal is obtained from a partbased human detector and object is derived from the relative position with respect to the human.
PPR-FCN (Zhang et al., 2017) employs a parallel FCN to perform pair selection and classification.
Explainable-HOI (Baldassarre et al., 2020) adopts graph nets to capture relations for better imagelevel HOI recognition, and uses backward explanation for instance-level HOI detection.
MXHOI (Kumaraswamy et al., 2021) proposes a momentum-independent learning strategy to utilizestrong & weak labels simultaneously.
AlignFormer (Kilickaya & Smeulders, 2021) proposes analign layer in transformer framework, which utilizes geometric & visual priors to generate pseudoalignments for training.
Those methods focus on learning HOIs with advanced network structuresor better pseudo alignments.
However, they still suffer from noisy human-object associations andambiguous interaction types.
To address those challenges, we exploit prior knowledge from CLIP tobuild a discriminative HOI representations.
Knowledge exploitation of pretrained V&L models: Recently, CLIP (Radford et al., 2021a)model has demonstrated strong generalization to various downstream tasks (Ghiasi et al., 2021; Duet al., 2022; Gu et al., 2021).
Some works also explore CLIP knowledge in supervised HOI detection,e.g., CATN (Dong et al., 2022) initializes the object query with category-aware semantic informationfrom CLIP text encoder and GEN-VLTK (Liao et al., 2022) employs image feature distillation andclassifier initialization with HOI prompts.
However, they only exploit CLIP knowledge at a coarselevel and require detailed annotations of human-object pairs.
It is non-trivial to extend such strategiesto the weak supervision paradigm due to highly noisy training signals.
In our work, we build a deepconnection between CLIP and HOI representation by incorporating the prior knowledge of HOIs atboth image and HOI instance levels.
M ETHOD
3. P ROBLEM SETUP AND METHOD OVERVIEW
Problem setup Given an input image I , the task of weakly-supervised HOI detection aims tolocalize and recognize the human-object interactions, while only the corresponding image-level HOIcategories are available for training.
Formally, we aim to learn a HOI detector M , which takesan image I as input and generates a set of tuples
O = { ( x h , x o , c
o , a h,o , R ah,o ) } , i.e., O =
M ( I ) .
Here each tuple indicates a HOI instance, in which x h , x o ∈ R 4 represent human and objectbounding boxes, c o ∈ { 1 , ..., C } is the object category, a h,o ∈ { 1 , ..., A } denotes the interactionclass associated with x h and x o , and R ah,o ∈ R is the HOI score.
For the weakly-supervised setting,each training image is annotated with a set of HOI categories R = { r ∗ } at the image level only, where r ∗ ∈ { 1 , ..., N } is an index to a combination of ground-truth object category c ∗ and interactioncategory a ∗ , and N denotes the number of all possible HOI combinations defined on the dataset.
Method Overview As we lack supervision for the HOI locations, we adopt a typical hypothesizeand-recognize strategy (Zhang et al., 2017; Baldassarre et al., 2020; Kumaraswamy et al., 2021) for
HOI detection: first we generate a set of human and object proposals with an off-the-shelf objectdetector (Ren et al., 2015) and then predict the interaction class for all human-object combinations.
Unlike other methods, we do not re-use the feature maps of the object or human detector - we onlykeep the bounding boxes.
Instead, we learn a new representation optimized for the HOI task.
This ischallenging under the weak setting as the model learning is noisy, but feasible by leveraging the richsemantic knowledge from a pretrained large-scale multimodal model, like CLIP.
However, the naiveknowledge integration strategies for supervised setting fail when directly applied in the weak setting,as evidenced by our experiments in Appendix.
B
Our framework adopts two philosophies to address the challenges in the weakly-supervised HOI task:the first is to integrate the prior knowledge into discriminative representation learning, and the secondis to suppress noise in learning.
For the first philosophy, we utilize the prior knowledge from CLIPto guide the representation learning in both global image-level and fine-grained human-object pairs,which is instantiated by a bi-level knowledge integration strategy.
For the second philosophy, weadopt an effective self-taught learning mechanism to suppress the irrelevant pairs.
We instantiate the bi-level knowledge integration strategy with a two-branch deep network.
Ourdetection pipeline starts with a set of human proposals with detection scores { ( x h , s h ) } , and objectproposals with their categories and detection scores { ( x o , c o , s o ) } .
Then, the global branch performsimage-level HOI recognition by utilizing a CLIP-initialized HOI knowledge bank as a classifier.
Thisallows us to exploit both visual and text encoders from CLIP to generate better HOI representations.
In parallel, for each human-object pair ( x h , x o ) , the local branch explicitly augments the pairwise
HOI features with the HOI knowledge bank to then identify their relatedness and interaction classes.
To train our model, we use a multi-task loss, which incorporates a HOI recognition loss defined onimage-wise HOIs for the visual encoder and knowledge bank finetuning, and a self-taught relatednessclassification for suppressing the background human-object associations, on top of the standard
MIL-based loss.
We first present model details in Sec.3.2, followed by the training strategy in Sec.3.3.
3. M ODEL D ESIGN
Now we introduce our bi-level knowledge integration strategy, where the aim is to exploit CLIPtextual embeddings of HOI labels as a HOI knowledge bank for the HOI representation learning,and to transfer such knowledge both at image level as well as at the level of human-object pairs forinteraction predictions.
Specifically, as shown in Fig.
1, our network consists of a global branchand a local branch.
The global branch includes a backbone network (Sec.3.2.1) that extracts imagefeatures, and a HOI recognition network (Sec.3.2.2) that uses a HOI knowledge bank based on CLIPto predict image-level HOI scores.
For each human-object proposal generated by an off-the-shelfobject detector, the local branch employs a knowledge transfer network (Sec.3.2.3) to compute itsfeature representation with enhancement from the HOI knowledge bank, and a pairwise classificationnetwork (Sec.3.2.4) to compute their relatedness and interaction scores.
Finally, we generate the final HOI detection scores by combining global HOI scores with local predictions (Sec. 3.2.5).
HOI Knowledge Bank Generation CLIP builds a powerful vision-language model by pretrainingon large-scale image-text pairs.
It consists of a visual encoder F V and textual encoder F T , mappingboth visual and textual inputs to a shared latent space.
Here, we exploit CLIP to generate a HOIknowledge bank.
We take a similar prompt strategy as in CLIP, adopting a common template ‘aperson { verb } a/an { object }’ to convert HOI labels into text prompts (e.g., converting ‘drive car’ to‘a person driving a car’).
Then we input the sentences into the CLIP textual encoder F T to initializethe HOI knowledge bank W T ∈ R N · D , with D denoting the feature dimension.
One can think of W T as a set of ‘prototypes’ in feature space, one for each HOI in the dataset.
3.2. G LOBAL BRANCH : B ACKBONE N ETWORK
To incorporate CLIP for feature extraction, we initialize the backbone network (e.g., a ResNet-101 (Heet al., 2016)) with CLIP’s visual encoder F V to generate a feature map Γ for the input image I .
Wefurther compute a global feature vector v g ∈ R D with self-attention operation (Radford et al., 2021b).
3.2. G LOBAL BRANCH : HOI R ECOGNITION N ETWORK
We perform an image-wise HOI recognition task with the HOI knowledge bank W T .
We obtainglobal HOI scores s g ∈ R N by computing the inner product between the image feature v g and theknowledge bank W T : s g = W T × v g , where × is matrix multiplication.
This has the effect ofadapting the visual encoder and knowledge bank parameters to the HOI recognition task, fully takingadvantage of the knowledge from CLIP.
3.2. L OCAL BRANCH : K NOWLEDGE T RANSFER N ETWORK
Given the CLIP-initialized visual encoder, a standard HOI representation can be formed by concatenating the human and object appearance features along with their spatial encoding.
However, evenafter the finetuning as described above, such a representation still mainly focuses on object-levelsemantic cues rather than relation-level concepts.
In this module, we explicitly exploit the HOIknowledge bank W T to learn a local relation-specific HOI representation.
To achieve this, we proposean attention-based architecture as shown in Fig.2(a).
Specifically, for each human proposal x h and object proposal x o , we use RoI-Align (He et al. ,2017) to crop the feature maps from Γ followed by a self-attention operation to compute theirappearance features v h , v o ∈ R D .
Then we compute a spatial feature v sp by encoding the relativepositions of their bounding boxes ( x h , x o )
(Zhang et al., 2021c) 2 .
The holistic HOI representationv p ∈
R D is an embedding of the human and object appearance features and their spatial feature, i.e.,v p = F E ([ v h ; v o ; v sp ]) , where [; ] is the concatenation operation and F E is a multi-layer perceptron(MLP).
To enhance relation-level concepts, we further compute its union region x u ∈ R 4 (see Fig. 2a) andextract the corresponding appearance feature v u ∈ R D via RoI-align over the feature map Γ .
Theunion region is important as it encodes relational context cues, but it potentially also contains a largeamount of background that is noisy for model learning.
We thus devise an attention module that issimilar in design to the HOI recognition network, but uses the union feature v u as query to extracta meta-embedding v meta ∈ R D from the HOI knowledge bank W T .
The final HOI representationˆ v p ∈ R D is built by fusing the holistic representation v p and v meta with a MLP F K .
[Equation]
Here α ∈ R N is the normalized attention weight and ⊺ is the transpose operation.
v meta encodes adiscriminative representation from CLIP and facilitates feature sharing between HOI classes.
3.2. L OCAL BRANCH : P AIRWISE C LASSIFICATION N ETWORK
Given the relation-aware HOI representation ˆ v p , our final module performs a coarse-level classification on human-object association and a fine-level classification for interaction recognition.
Specifically, we use two MLPs F P and F B to predict the interaction scores s p ∈ R A and therelatedness score s b ∈ R for each human-object pair:
[Equation]
To train the model under weak supervision (see Sec. 3.3), we further aggregate the pairwise interaction 221 scores into image-level interaction scores .
Assume we have M pairs of human-object proposals fora given image, and denote the interaction scores for the m -th pair as s mp .
We first concatenate all theinteraction scores to compose a bag S =
[ s 1 p ; ... ;
s Mp ] ∈ R M · A , then we maximize over all pairs toobtain the image-wise interaction scores: ˜ s p = max m S .
3.2. M ODEL I NFERENCE
During model inference, we do not use the local interaction scores s p directly.
Instead, we normalize Swith a Softmax operation defined on all pairs: ¯ S =
Softmax m ( S ) , and then compute the normalizedpairwise interaction scores e p
= σ (˜ s p ) · ¯ s p , where ¯ s p is a row from ¯ S and σ is Sigmoid function.
This has the effect of measuring the contribution of a given pair, in case of multiple pairs in an imageshare the same interaction.
The final interaction score s ah,o for human-object pair ( x h , x o ) combines multiple scores, includingthe global HOI scores s g , the normalized pairwise interaction scores e p , and the relatedness score s b .
The overall HOI score R ah,o is a combination of the interaction score and the object detection scores.
[Equation]
where s a,c o g is the HOI score corresponding to a -th interaction and c o -th object category in s g , e apis the score of a -th interaction in e p , and γ is a hyper-parameter to balance the scores (Zhang et al. ,2021c; Li et al., 2019b).
3. L EARNING WITH W EAK S UPERVISION
To train our deep network in a weakly supervised setting, we use a multi-task loss defined on threedifferent levels.
Specifically, our overall loss function L consists of three terms: i) an image-wise
HOI recognition loss L g to adapt CLIP features to the task of human-object interaction detection;ii) a pairwise interaction classification loss L p to guide the knowledge transfer towards fine-grainedrelation-aware representations; and iii) a self-taught relatedness classification loss L b to prunenon-interacting human-object combinations.
Formally, the overall loss is written as:
[Equation]
Image-wise HOI recognition loss L g :
Given the HOI scores s g and ground-truth HOI categories
R , L g is a standard binary cross-entropy loss for multi-label classification: L g =
L BCE ( s g , R ) .
Pairwise interaction classification loss L p : We adopt a MIL strategy that first aggregates thepairwise interaction scores and supervises this with image-level interaction labels as A = { a ∗ } .
Given the image-wise interaction scores ˜ s p , L p is a standard binary cross-entropy loss for multi-labelclassification as: L p =
L BCE (˜ s p , A ) .
Self-taught relatedness classification loss L b : As human-object associations are not annotated,we devise a novel pseudo relatedness label generation mechanism for training a self-taught binaryclassifier to identify valid human-object associations.
Specifically, we observe that the human-objectpairs with confident interaction scores are often associated after a short period of initial trainingwithout self-taught classification loss.
Motivated by this, we use the interaction scores s p from themodel under training to supervise the relatedness classification.
Concretely, we generate pseudo labels B = { b 1 , ..., b M } for all human-object pairs in an image,where b m ∈ { 0 , 1 } indicates the relatedness for the m -th combination.
To this end, as illustratedin Fig.2(b), we first propose a binary mask Z ∈ { 0 , 1 } M · A for all interaction scores S with respectto the ground-truth object categories
C
= { c ∗ } .
For each human-object pair where the objectlabel c o is included in C , we consider it as a potential interactive combination and thus assign thecorresponding row in Z as 1, and other rows as 0.
For the latter, we also immediately set b m = 0 .
Then we generate pairwise scores t a ∈ R M for each ground-truth interaction a ∗ by selecting thecorresponding row from S ⊙ Z .
The pseudo label for the pair with the highest score is assigned as1, i.e., m a = arg max m t a and b m a = 1 .
We only select one positive pair 3 for each a ∗ .
Finally, L b is3 We also explore top-K selection in Appendix F [Table]
[Table]
defined as a binary cross-entropy loss: L b = (cid:80) m L BCE ( s mb , b m ) , where s mb is the relatedness scorefor the m -th pair.
E XPERIMENTS
4. E XPERIMENTAL SETUP
Datasets: We benchmark our model on two public datasets: HICO-DET and V-COCO.
HICO-DETconsists of 47776 images (38118 for training and 9658 for test).
It has N = 600 HOI categories,which are composed of C = 80 common objects (the same as MSCOCO (Lin et al., 2014)) and
A = 117 unique interaction categories.
V-COCO is a subset of MSCOCO, consisting of 2533 imagesfor training, 2867 for validation and 4946 for test.
It has 16199 human instances, each annotated withbinary labels for A = 26 interaction categories.
Evaluation Metric:
Following (Chao et al., 2015), we use mean average precision (mAP) to evaluate
HOI detection performance.
A human-object pair is considered as positive when both predictedhuman and object boxes have at least 0.5 IoU with their ground-truth boxes, and the HOI class isclassified correctly.
4. I MPLEMENTATION D ETAILS
We use an off-the-shelf Faster R-CNN (Ren et al., 2015) pretrained on MSCOCO to generate atmost 100 object candidates for each image.
For V-COCO, it is worth noting that we train the objectdetector by removing the images in MSCOCO that overlap with V-COCO to prevent informationleakage.
The backbone network is initialized with the visual encoder from CLIP-RN101 model andthe feature dimension D = 1024 .
For model learning, we set the detection score weight γ = 2 . 8 as default by following previousworks (Zhang et al., 2021c; Li et al., 2019b), then optimize the entire network with AdamW and aninitial learning rate of 1e-5 for backbone parameters and 1e-4 for others.
We detach the parameters ofthe knowledge bank on the local branch for better model learning.
We train up to 60K iterations withbatch-size 24 in each on 4 NVIDIA 2080TI GPUs, and decay the learning rate by 10 times in 12Kand 24K iteration.
4. Q UANTITATIVE R ESULTS
For HICO-DET (Tab.1), our approach outperforms the previous state of the arts on the weaklysupervised setting by a clear margin, achieving 22.89 mAP with ResNet-50 and 25.70 mAP with
ResNet-101 as backbone.
For a fair comparison, we also re-implement PPR-FCN with CLIP visualencoder.
The results show that we still outperform PPR-FCN by a sizeable margin, which validatesthe superiority of our framework.
Besides, we even perform comparably with HOTR and IDN underan inferior experimental setting where HOTR adopts a more advanced transformer encoder-decoderarchitecture, and both methods are trained with strong supervision.
Furthermore, the mAP gapbetween Rare (training annotations < 10) and Non-rare HOI classes in our results is much smaller
[Table]
than other methods, demonstrating the superior generalization capability of our HOI representationfor solving the long-tailed distribution issue.
In detail, we achieve a 0.62 mAP gap with ResNet-50and 1.53 with ResNet-101 backbone, which is much smaller than AlignFormer (3.14) and PPR-FCN(2.64), and supervised methods SSRT (9.01) and GEN-VLKT (4.9).
For V-COCO dataset, we report the performance of AP role in both scenario1 and scenario2 for acomplete comparison, which are 42.97 / 48.06 AP role with ResNet-50 and 44.74 / 49.97 AP role with
ResNet-101 as backbone.
As shown in Tab.1, our model achieves significant improvement comparedwith AlignFormer, and even is comparable with supervised methods TIN and iCAN.
4. A BLATION S TUDY
In this section, we mainly validate the effectiveness of each component with detailed ablation studieson HICO-DET dataset.
We use ResNet-50 as the backbone network to reduce experimental costs.
Baseline: The baseline adopts the visual encoder from CLIP-RN50 to generate the vanilla HOIrepresentation v p , which is directly used to predict the interaction scores s p .
Only pairwise interactionclassification loss L p is used for model learning.
HOI recognition: We augment the baseline with a HOI recognition network and observe the fullmAP improves from 19.52 to 20.41, as reported in Exp 1 of Tab. 2.
It suggests that the learnableknowledge bank W T serves as a powerful classifier to perform image-level HOI recognition andupdate the visual encoder for better HOI representation.
We visualize the learned parameters ofknowledge bank in Appendix D to demonstrate its effectiveness.
Furthermore, as in Exp 2 , theperformance slightly decreases from 20.31 to 20.09 when we freeze the training of the knowledgebank, indicating that joint learning of visual features and the knowledge bank is more appropriate for
HOI detection.
Knowledge Transfer Network (KTN): KTN explicitly transfers the CLIP meta-knowledge topairwise HOI features.
As a result, it contributes 0.55 Full mAP improvement ( Exp 3 v.s. Exp 1 ) andmost of the performance gains come from Non-rare classes.
This result shows KTN is capable ofextracting discriminative features from the relational knowledge bank to our HOI representation.
Wealso study the effectiveness of the attention mechanism of KTN in Appendix E.
Score fusion: In Tab. 2, we largely improve the Full mAP from 20.86 ( Exp 3 ) to 22.40 ( Exp 4 ) byfusing the global HOI scores s g to pairwise interaction score s p .
As the HOI recognition networkseamlessly inherits the visual-linguistic features from CLIP and directly adopts image labels assupervision, the global interaction scores are pretty accurate and largely enhance the pairwise scores,demonstrating its strong capabilities to cope with long-tailed and fine-grained HOI recognition.
Self-taught Relatedness Classification (SRC): Self-taught classification aims to identify therelatedness between human and objects.
The improvements from Exp 4 to ours show the effectivenessof our self-taught strategy, which is capable of figuring out the irrelevant human-object pairs andsuppressing their interaction scores during inference.
Combining KTN & SRC:
The ablation results of Exp 5-7 in Tab.
2 show the KTN and SRC areable to facilitate each other.
In detail, the SRC obtains 0.49 Full mAP improvement when the KTN is introduced ( ours v.s. Exp 4 ), which is only 0.36 without KTN ( Exp 5 v.s. baseline ).
Similarly,the KTN contributes 0.78 Full mAP improvement with SRC ( Exp 7 v.s. Exp 6 ), which is only 0.55without SRC ( Exp 3 v.s. Exp 1 ).
Parameter initialization: Our visual encoder and knowledge bank are both initialized from CLIP.
We also explore different parameter initialization strategy in Exp 8-10 .
Specifically, we initialize thevisual encoder with a ResNet50-FPN pretrained on COCO detection task for the baseline ( Exp 8 ), andthe knowledge bank with random parameters ( Exp 9 ) or embeddings of HOI labels from RoBERTamodel ( Exp 10 ) for the final model.
We observe severe drops with all these initialization methodscompared with ours, demonstrating the effectiveness and generalization ability of CLIP model.
It isworth noting that the mAP of Rare classes decreases from 16.20 in Exp 8 to 15.57 in Exp 9 , whichsuggests the randomly initialized knowledge bank even aggravates the imbalance issue in final model.
4. Q UALITATIVE R ESULTS
We show some qualitative results of our method in Fig.3.
For each HOI prediction, we report (i)normalized pairwise interaction score, (ii) global HOI score and (iii) relatedness score for ours , andonly pairwise interaction score for baseline .
In Fig.3(a), ours interaction scores are more confidentthan baseline in Rare HOI classes, demonstrating the generalization ability of our CLIP-guided
HOI representation.
Besides, when incorporating relational knowledge bank into pairwise HOIrepresentation, our method is capable of distinguishing the subtle differences among similar HOIs in
Fig.3(b)
(e.g., repair_truck:0.23 v.s. inspect_truck:0.48 in the bottom figure).
Moreover, in Fig.3(c),the global branch suppresses background HOIs by predicting low global scores for them (e.g., theglobal HOI score is 0.033 for sit_on_motorcycle while the ground-truth is sit_on_bicycle ).
Finally, in
Fig.3(d), our self-taught relatedness classification strategy shows strong capability at recognizing theambiguous human-object associations (e.g., 0.079 v.s. 0.994 in the upper figure).
C ONCLUSION
In this paper, we propose a bi-level knowledge integration strategy that incorporates the priorknowledge from CLIP for weakly-supervised HOI detection.
Specifically, we exploit CLIP textualembeddings of HOI labels as a relational knowledge bank, which is adopted to enhance the HOIrepresentation with an image-wise HOI recognition network and a pairwise knowledge transfernetwork.
We further propose the addition of a self-taught binary pairwise relatedness classificationloss to overcome ambiguous human-object association.
Finally, our approach achieves the new stateof the art on both HICO-DET and V-COCO benchmarks under the weakly supervised setting.
E THICS S TATEMENT
Hereby, we consciously assure that our study is original work which has not been previously publishedelsewhere, and is not currently being considered for publication elsewhere.
We do not have ethicsrisks as mentioned in the author guidelines.
R EPRODUCIBILITY S TATEMENT
We use publicly available benchmarks, HICO-DET and V-COCO, to validate our method.
We willrelease all the codes and the trained model after this paper is accepted.
A PPENDIX
In this appendix, we first describe the spatial feature generation, and then supplement more experimental results of different CLIP knowledge integration strategies for weakly-supervised HOIdetection.
For Explanation-HOI (Baldassarre et al., 2020), we further clarify the difference betweentheir mAP evaluation protocol and the standard one.
Finally, we demonstrate the limitations, potentialnegative societal impacts as well as the result error bars of our method.
A T HE A DVANTAGE OF O UR HOI R EPRESENTATION
To verify the improvement obtained with our CLIP-based HOI representation, we visualize the HOIrepresentation ˆ v p in feature space with t-SNE(van der Maaten & Hinton, 2008).
For clarity, werandomly sample 80 HOI categories, and collect 50 samples for each category.
For comparison, wealso demonstrate the object-based HOI representation derived from ‘Exp 9’ in Tab.2 (i.e., the modelwithout CLIP knowledge and using a random knowledge bank).
As shown in Fig.4, we observethat CLIP-based HOI representations for different HOI categories are diverse and well separated infeature space, which is better for HOI detection.
In contrast, the object-based representations are notwell separated in feature space (see the red box region in Fig.4b).
Besides, the experimental results inthe ablation study (ours v.s. ‘Exp 9’) also validate the advantage of CLIP-based HOI representation,improving full mAP from 19.61 to 22.89.
B A BLATION ON CLIP K NOWLEDGE I NTEGRATION
To further demonstrate the superiority of our CLIP knowledge integration strategy, we study severalproven techniques for CLIP knowledge transfer in Tab. 3.
In Abl 1 , for each human-object pair, wedirectly infer the HOI scores with CLIP by computing the cross-modal similarities between theirvisual union region and the HOI prompts.
Without introducing any HOI priors, the promising resultsindicate the powerful generalization ability of CLIP and motivate the design of incorporating CLIPknowledge for weakly-supervised HOI detection.
In Abl 2 , we duplicate the experiment setting andresults from Exp 8 in Tab.
2 of the main paper.
It is a simplified baseline model but initializes thevisual encoder with a ResNet50-FPN pretrained on COCO detection task.
Then we introduce threedifferent CLIP knowledge transfer strategies ( Abl 3-4 and ours ) based on Abl 2 .
In Abl 3 , we directly enhance baseline scores in Abl 2 with the CLIP similarity scores in Abl 1 on theinference stage.
Without bells and whistles, we obtain 1.12 gain in Full mAP.
Furthermore, in Abl 4 , we adopt a similar knowledge transfer strategy as GEN-VLKT (Liao et al.,2022), where we initialize the HOI classifier F P with HOI prompt and regularize the global HOIrepresentation with CLIP image feature v g .
In detail, we first compute the global HOI representationv mean with mean pooling on all pairwise HOI representations, i.e., v mean =
MeanPool m ( { v mp } Mm =1 ) .
Here v mp is the holistic HOI representation (c.f.
Sec.
3.2.3 in the main paper) for m -th human-objectpair.
Then we develop an additional L 2 loss L reg to transfer the knowledge from CLIP to HOIrepresentations: L reg =
L 2( v mean , v g ) .
The performance even decreases slightly from 19.44 to19.39, which might be caused by the incompatibility of parameters between backbone network(ResNet50-FPN pretrained on COCO) and F P (HOI prompt embeddings from CLIP).
When directlyapplying the knowledge transfer strategy of GEN-VLKT to a weakly-supervised setting, it is difficultto map the unmatched HOI representation and classification weights to a joint space as the supervisorysignals are noisy.
Finally, our approach achieves the best performance compared with other strategies, demonstratingthe effectiveness of our bi-level knowledge integration strategy.
[Equation] G ENERATION
Following (Zhang et al., 2021c), we generate the spatial feature v sp ∈ R D for each pair of humanobject proposals ( x h , x o ) .
Specifically, we first compute the bounding boxes information for x hand x o separately, including their center coordinates, widths, heights, aspect ratios and areas, all
[Table]
normalized by the corresponding dimension of the image.
We also encode their relative spatialrelations by estimating the intersection over union (IoU), a ratio of the area of x h and x o , a directionalencoding and the distance between center coordinates of x h and x o .
We concatenate all the abovementioned preliminary spatial cues and obtain a spatial encoding p ∈ R 18+ .
To encode the second andhigher order combinations of different terms, the spatial encoding is concatenated with its logarithmand then embedded to v sp : v sp = F sp ([ p ; log ( p + ϵ )]) .
Where ϵ > 0 is a small constant to guaranteethe numerical stability, and F sp is a multi-layer fully connected network.
D V ISUALIZATION OF HOI K NOWLEDGE B ANK W T
To further understand W T , we visualize the knowledge bank features initialized by CLIP (Fig.5(a))and learned from scratch (Fig.5(b)) in feature space by t-SNE.
It is worth noting that the knowledgebank learned from scratch is derived from ‘Exp 9’ in Tab.2.
As shown in Fig.5, we observe thatthe knowledge features of HOI classes initialized with CLIP are more discriminative than randominitialized, and show a better clustering result (e.g. the HOI classes in red box regions).
E D IFFERENT DESIGNS OF KTN
To further validate the effectiveness of our attention mechanism in KTN, we compare our designwith some variants in Tab. 4.
First of all, we directly encode the relation-level features within theunion region to enhance the pairwise representation rather than the external knowledge bank.
As aresult, the mAP even decreases a little bit from 20.75 ( Exp 6 ) to 20.69 ( Exp 11 ).
The potential reasonis that the union region contains more ambiguous visual relations and background clutters, whichare difficult to learn in a weak setting.
Besides, we also explore different normalization strategies in
KTN.
The results in Tab.
4 demonstrate that Softmax operation ( ours ) performs better than uniformattention ( Exp 12 ) or Sigmoid operation ( Exp 13 ), indicating our attention mechanism is non-trivialand more effective on aggregating the relational cues from HOI knowledge bank.
[Table]
F T OP -K POSITIVE PAIR SELECTION FOR SRC
In this section we show the results of selecting top-2 and top-5 pairs as positive in Tab. 5.
We noticethat there is a small performance drop, which is likely to be caused by mislabeling more negativepairs as positive, resulting in model learning with more noise.
G T HE P ROMPT G ENERATION FOR V-COCO
For the V-COCO dataset, each action has two different semantic roles (’instrument’ and ’object’)for different objects, like ‘cut cake’ and ‘cut with knife’.
We use two different prompt templates toconvert a HOI label to a language sentence.
For the former one, we take template “a person verb a/anobject”, and use “a person verb with object” for the latter.
H E VALUATION M ETRIC FOR V-COCO
V-COCO dataset has two scenarios for role AP evaluation.
In Tab. 1, APS1&2 refer to ‘Average Precision in scenario 1&2’. V-COCO dataset has two different annotations for HOIs: the first is a [Table]
full label of (human location, interaction type, object location, object type), and the second missestarget object (also denoted as ‘role’ in the original paper (Gupta & Malik, 2015)) annotations, and thelabel only includes (human location, interaction type). For the second case, there are two differentevaluation protocols (scenarios) when taking a prediction as correct 4 :
In scenario 1, it requires theinteraction is correct & the overlap between the human boxes is > 0 . 5 & the corresponding role isempty, which is more restricted; in scenario 2, it only requires the interaction is correct & the overlapbetween the person boxes is > 0 . 5 . I E VALUATION OF E XPLANATION -HOI The Explanation-HOI (Baldassarre et al., 2020) has a misunderstanding of mAP evaluation protocol.
As shown in Fig.6(a) L200-L205, the Explanation-HOI only takes some specific predicted HOIs intothe evaluation process, which has the same HOI labels as groundtruth HOIs.
Thus, they ignore lotsof false-positive HOI predictions when calculating mAP, leading to an untrustable high mAP score(reported in their original paper).
In Fig.6(b)
L204-L208, we evaluate all predicted HOIs, which isthe same as the standard evaluation protocol proposed in HICO-DET (Chao et al., 2015).
The correctresults have already been reported in Tab.1 in the main paper.
J L IMITATIONS
As described in Sec. 3.1, we adopt an external object detector to generate human-object proposalsand then recognize their interactions.
Consequently, our method is faced with two limitations broughtby erroneous object detection results.
Firstly, the positive human-object pairs are not recalled if thehuman or object proposals are not detected.
Secondly, the proposals are kept fixed during learning,which leads to the problem of inaccurate localization and object types.
K L ICENSE
The licenses of the assets used in our work are listed below, including open-sourced CLIP model,
HICO-DET dataset, and V-COCO dataset.
As for HICO-DET, we cannot find its license in the paperand the official project page.
Thus we provide the official project page instead here for clarity.
CLIP: https://github.com/openai/CLIP MIT License
VCOCO: https://github.com/s-gupta/v-coco/ MIT License
HICO-DET: http://www-personal.umich.edu/ ywchao/hico/
No license