Relational Language-Image Pre-training for Human-Object Interaction Detection
Abstract
The task of Human-Object Interaction (HOI) detection targets fine-grained visualparsing of humans interacting with their environment, enabling a broad range of applications.
Prior work has demonstrated the benefits of effective architecture designand integration of relevant cues for more accurate HOI detection.
However, thedesign of an appropriate pre-training strategy for this task remains underexploredby existing approaches.
To address this gap, we propose Relational LanguageImage Pre-training (RLIP), a strategy for contrastive pre-training that leveragesboth entity and relation descriptions.
To make effective use of such pre-training,we make three technical contributions: (1) a new Par allel entity detection and
Se quential relation inference (ParSe) architecture that enables the use of both entityand relation descriptions during pre-training; (2) a synthetic data generation framework, Label Sequence Extension, that expands the scale of language data availablewithin each minibatch; (3) mechanisms to account for ambiguity, Relation Quality
Labels and Relation Pseudo-Labels, to mitigate the influence of ambiguous/noisysamples in the pre-training data.
Through extensive experiments, we demonstratethe benefits of these contributions, collectively termed RLIP-ParSe, for improvedzero-shot, few-shot and fine-tuning HOI detection performance as well as increasedrobustness to learning from noisy annotations.
Introduction
Driven by improvements in storage, sensors and networking technology, humanity is amassing vastarchives of image and video data.
A significant fraction of this media is human-centric —it is contentfocused on humans and their actions.
The task of human-object interaction (HOI) detection [4] aimsto provide a step towards fine-grained parsing of such content by detecting all possible triplets of theform <
human, relation, object > present in visual data.
Robust HOI detection has myriad uses forimage/video data analysis and represents essential functionality for visual and language applicationssuch as image/video captioning [60, 46], image retrieval [26], image synthesis [25] and video actionunderstanding [23].
Given that sustained progress in object detection has yielded increasingly robust systems for detectingpeople and objects [51, 15, 9], a key remaining challenge for HOI detection is developing methodscapable of generalising to the many possible pairs of interactions between these entities whenprovided with non-exhaustive training data.
To tackle this challenge, we draw inspiration from recentdevelopments demonstrating that contrastive language-image pre-training can induce remarkablegeneralisation for zero-shot classification tasks [50, 24].
These methods perform classification by casting it a retrieval problem , ensuring that the downstream task aligns closely with the pre-training objective.
Recent work by Alayrac et al.
[2] hypothesises that it is this close alignment betweenthe downstream and pre-training objectives that explains why contrastive methods have proven soeffective for zero-shot classification.
In light of this hypothesis, in this work, we explore whether it ispossible to achieve a similarly close alignment between the HOI detection task and its pre-trainingstrategy.
While HOI detection has been widely studied [14, 47, 59, 39, 28, 30, 65, 67, 29, 55, 62], the topic ofdesigning pre-training to reflect the final task objective remains under-explored.
Indeed, a widelyadopted strategy [64, 55, 7, 67] has been to employ object detection pre-training to initialise theparameters of the model responsible for both entity detection and relation inference.
However, whilesuitable for entity detection, such pre-training may be suboptimal for the detection of relationsbetween entities which often requires the model to take account of groups of entities with greaterspatial context, rather than individual entities in isolation.
To address this shortcoming of HOI detection, we propose Relational Language-Image Pre-training(RLIP) which tasks the model with establishing correspondences from both entities and relationsto free-form text descriptions.
By doing so, RLIP endows the model with the ability to performzero-shot HOI detection 1 .
Moreover, in contrast to previous pre-training schemes that are limited topredefined finite category lists, RLIP benefits from the rich descriptive nature of natural languagesupervision.
We encountered three barriers to a naive implementation of RLIP for existing methods: (1) Recent
HOI detection architectures [55, 64, 67] typically employ joint representations of (some subset of)subject, object and relation triplets.
As a consequence, it is difficult to leverage text descriptions forseparate humans, objects and relations provided by existing datasets such as VG [31].
(2) Contrastivepre-training requires negative samples to train effectively, but it is unclear a priori how such negativesshould be constructed.
(3) Free-form text descriptions exhibit label noise and semantic ambiguity(since there can be many ways to describe the same concept in the absence of a canonical list ofcategories), rendering optimisation challenging.
To overcome these barriers, we make several technical contributions in addition to the RLIP framework.
First, to allow contrastive pre-training with distinct descriptions of subsets, objects and relations,we propose the Par allel entity detection and Se quential relation inference (ParSe) architecture.
ParSeemploys a DETR [3]-like design that allocates separate learnable query groups for subject and objectrepresentations, together with an additional set of conditional queries that encode relations.
While
ParSe enables (and works best with) RLIP, we also find that it yields gains for traditional objectdetection pre-training schemes.
To address the second barrier, we synthesise label sequences byextending in-batch labels with out-of-batch sampling to ensure a plentiful supply of negatives—weterm this Label Sequence Extension (LSE).
For the third barrier, we exploit cross-modal cues toresolve label noise and relation ambiguity.
In particular, to mitigate label noise we use the qualityof the visual entity detection phase [36] to assign quality scores to relation-text correspondences,an approach we term Relational Quality Labels (RQL).
To mitigate relation ambiguity, we leveragesimilarities between labels to propagate relations via a pseudo-labeling scheme, which we term
Relational Pseudo-Labels (RPL).
We demonstrate through experiments that relational pre-training outperforms traditional objectdetection pre-training schemes on comparable data.
We further find that the a zero-shot applicationof our combined approach, RLIP-ParSe, surpasses several existing fine-tuned methods.
Related Work
Human-object interaction detection.
There is a rich body of work on HOI detection.
One themehas focused on the development of effective architectures for this task [39, 28, 55].
A secondtheme has sought to leverage informative cues ranging from interaction points [39, 65], interaction boxes [28], contextualized embeddings [55, 29, 67, 29], poses [14, 38] and statistical priors [30, 62]to external knowledge in the form of language embeddings [59, 47].
However, relational pre-trainingand open-vocabulary recognition remains underexplored.
The inter-pair transformations of IDN [37]and affordance transfer learning of ATL [19] can be interpreted as entity augmentations to train astronger verb classifier, but these methods do not directly optimise for all the components of HOIdetection.
Similarly to ParSe, CDN [64] explores disentangled embeddings.
However, it still couplesthe embedding of subjects and objects, rendering it suboptimal for RLIP.
The concurrent work
GEN-VLKT [40] aims to mimic the knowledge from image-level language-image pre-training [50],but we aim to achieve aligned entity- and relation-level pre-training for HOI detection.
Leveraging free-form text for visual pre-training.
A series of recent papers have illustratedthe significant value of employing free-form language to provide supervision for vision systems.
CLIP [50] and ALIGN [24] demonstrated striking improvements in zero-shot image classificationability through contrastive training of image-level representations.
Further work has sought toadditionally leverage correspondences between objects/regions and text to learn flexible groundingmodels [27, 35, 63].
We similarly seek to benefit from natural language supervision.
However, to thebest of our knowledge, we are the first to leverage correspondences between descriptions of relationsand explicit pairings of subjects and objects (rather than descriptions of images, objects or regions)as a pre-training signal.
Methodology
[Figure]
In this section, we first present our triplet detection architecture, ParSe.
Second, we describehow ParSe is used to perform relational language-image pre-training (RLIP).
Finally, we introducetechniques to synthesise contrastive negatives and mitigate noise and ambiguities among labels.
Theoverall RLIP-ParSe framework is illustrated in Fig.
3.1 ParSe for Triplet Detection
Structure overview.
The core idea underpinning the ParSe architecture is to allocate distinct representations of subjects, objects and relations (rather than representing their combination, as commonlypursued in prior work [55]).
The motivation for doing so is two-fold: (i) distinct representationsenable the direct use of contrastive RLIP, since these representations can be put in correspondencewith separate entity and relation annotations; (ii) the separation of responsibilities allows for a morefine-grained control over the context available for each decision (a theme that has proven importantfor detection tasks [54]).
In particular, note that when detecting subjects and objects, local contextis typically most useful.
However, when it comes to relations, detection will benefit not only frominformative local cues, but also neighbouring context (for instance, it is useful to be aware of waterand hoses when inferring the relation in the triplet <human, wash, car> ).
To instantiate this ideawe follow [64] and implement triplet detection in a two-stage end-to-end manner.
Our probabilisticmodel factorises as follows:
[Equation]
where Q s , Q o ∈ R N Q × D define two sets of independent queries for N Q subjects and N Q objects;
C denotes features from the detection encoder; B s , B o , R denote sets of detected subject boxes,object boxes and relations, respectively (these collectively comprise the detection results G );
θ Parand θ
Se represent learnable parameters from the entity detection decoder and the relation inferringdecoder, respectively.
To construct ParSe, we design two components, Parallel Entity Detection and
Sequential Relation Inference , to implement the second and third terms in Eq.
(1), respectively.
Theseare described next.
Parallel Entity Detection.
Following the DETR family of architectures [3, 66, 42], we first extractvisual features using an image encoder, add positional encodings and then pass the result througha customized Transformer encoder according to the detector we adopt (we explore both DETR [3]and DDETR [66] variants) to obtain detection features C .
Then, two sets of queries Q s and Q o arefed into the entity decoder to perform self-attention [56], cross-attention and feed-forward network(FFN) inference, obtaining ˜ Q s , ˜ Q o ∈ R N Q × D which are used to predict box locations and classes.
Sequential Relation Inference.
To encode relations, we perform Sequential Relation Inference asa sequential step after entity detection (similarly to [64]).
In the first stage, subjects and objectsare detected via Parallel Entity Detection .
In the second stage, we adopt a simple parameter-freematching scheme between subjects and objects to generate relation queries: matching by their indices.
Using this pairing scheme, we obtain relation queries via a conditional query generation function:
[Equation]
where for simplicity, we adopt addition as the query generation function.
Since we match by indices,
Q r ∈ R N Q × D contains N Q relation queries.
Q r is then fed into the second decoder to perform
Sequential Relation Inference via self-attention, cross-attention and FFN inference to obtain thecorresponding relation features ˜ Q r ∈ R N Q × D which are then used for relation classification.
3.2 RLIP-ParSe for Relational Language-Image Pre-training
For each iteration of pre-training, we construct a minibatch of images and their annotated relationtriplets comprising all entities’ locations, N E unique entity text labels as well as N R unique relationtext labels.
We describe how these are used for contrastive pre-training next.
Formation of target label sequences.
We construct targets from in-batch labels (free-form textdescriptions forming subject, object, relation triplets).
In more detail, we first aggregate all entitylabels within the batch and append to this sequence a no objects label.
Next, we similarly aggregateall in-batch relation labels.
Then, all entity and relation labels are respectively fed into a text encoder(RoBERTa [43] in our implementation) to extract label features denoted as L E and L R , respectively.
Note that a free-form text label can have multiple tokens after tokenization—we use only the featurederived from the [CLS] token to represent the label.
We concatenate the label feature sequencewith features from the image encoder as shown in Fig.
To fuse the concatenated features, weadopt a simple approach: applying a Transformer encoder [33, 27, 1] to obtain fused label features˜ L E ∈ R N E × D and ˜ L R ∈ R N R × D .
Cross-modal alignment through classification.
To implement RLIP, we task the model with establishing correspondences between entities/relations and their text descriptions using a classificationobjective, following [35].
In particular, we align the i th relation ˜ Q
r ( i ) ∈ R D with relation its text viaa Focal loss [41]:
[Equation]
where ˜ Q r ( i ) W Tb + W c is the learnable bias term introduced in [41]; W b ∈ R N R
× D is a learnablelinear projection and W c ∈ R N R is a constant vector filled with − log((1 − π ) /π ) with π = .
The
Focal loss is defined via Focal( p ) =
− (1 − p ) γ log( p ) where γ is set as a hyperparameter.
In theargument to this loss in Eq.
(3), j indexes along P r ( i ) ∈
R N R .
To encourage matching of subjectsand objects with their corresponding entity descriptions, an analogous objective to Eq.
(3) is usedexcept that a softmax and a CE loss are applied and W c is omitted (note that entities are uni-label andrelations multi-label, as defined by the downstream task).
The central benefit of the RLIP objectivesdefined above is that they bring the pre-training and downstream HOI detection losses into close alignment since the task of classifying entities and relations in the downstream task reflects thesame matching task used in pre-training.
As a result, RLIP produces models that can perform HOIdetection under zero-shot with no fine-tuning (NF) evaluation protocols.
Label Sequence Extension (LSE).
Within a given batch, the number of negative samples availablefor matching is limited.
However, provision of plentiful negatives has been widely shown to improvecontrastive learning [8, 10, 22, 57].
To this end, we propose Label Sequence Extension as a mechanismto leverage out-of-batch text descriptions.
Concretely, we sample additional text descriptions with aratio of two thirds entity labels and one third relation labels.
To ensure computational tractability inthe presence of the quadratic complexity of Transformer, we limit the label sequence to a predefinedlength N L .
We experiment with two sampling strategies: (i) Uniform sampling that draws amongcandidate labels with equal probability; (ii) Frequency-based sampling that samples according to thelabel frequency in the training set.
3.3 Addressing Relational Semantic Ambiguity
Datasets with crowd-sourced language annotations [31, 32] exhibit significant label noise and ambiguity.
First, the descriptions themselves may be noisy (inaccurate), particularly when the underlyingimage is challenging to interpret.
A second challenge for traditional training schemes is that similarrelations can be described differently, thanks to synonyms.
For example, the stand near relationmay be annotated “stand near”, “stand next to”, “stand by”, etc. .
These forms of semantic ambiguitymake supervised cross-modal pre-training (which relies on access to consistent labels) challenging.
To mitigate this issue, we focus on two aspects of the pre-training input data: (i) the quality of therelation text labels; (ii) the presence of semantically-similar labels in sampled label sequences.
Relational Quality Labels (RQL).
To tackle the first challenge, we propose a label smoothing [44]approach.
The key idea is that we expect the difficulty of subject and object detection for a particularinstance to correlate with the confidence of the annotated relation.
We therefore propose to estimateannotation quality from the quality of the entity detection stage.
Drawing inspiration from thegeneralised focal loss [36], we instantiate this idea by assessing the quality of the i th subject andobject detection after bipartite matching [55] as
[Equation]
where GIoU 0 – 1 denotes generalized IoU from [53] together with a linear scaling function to scale the
GIoU value to the range of 0 to 1 and ˆ denotes ground-truth annotation.
The resulting value e ( i ) isthen employed to calibrate the relation label confidence via multiplication: ˜ R ( i ) =
e ( i )
ˆ R ( i ) .
Relational Pseudo-Labels (RPL).
To address the second issue, we propose a pseudo-labellingstrategy [61] to account for synonyms in the extended sequence.
We exploit the fact that textembeddings with high semantic similarity will lie close together, as measured by an appropriatedistance function M ( · , · ) .
We define the distance between the i th annotated relation label ˆ R ( i )
={ 0 , 1 } N R and the j th relation text feature
˜ L R ( j ) ∈ R D from the extended sequence as
[Equation]
(5) where m ( · , · ) denotes Euclidean distance.
Given the i th relation label, we apply a scaling function to
M ( i, j ) via ¯ M ( i, j ) = max k ( M ( i,k ))
− M ( i,j ) max k ( M ( i,k )) where we have abbreviated M ( ˆ R ( i ) , ˜ L R ( j )) as M ( i, j )for clarity.
Next, we use a global threshold η to select label texts with high similarities: we set thej th label in the i th relation labels as ¯ M ( i, j ) if ¯ M ( i, j ) > η .
Note that when applying either RQL or
RPL, the ground truth labels are continuous (rather than discrete).
We therefore employ the Quality
Focal Loss [36] (rather than the standard Focal Loss [41]) as our objective function.
3.4 Pre-training, Fine-tuning and Inference
By design, our pre-training (RLIP) and fine-tuning phases follow a similar process.
For a given batchof images with corresponding annotations, we aggregate the results from Parallel Entity Detectionand Sequential Relation Inference to form N Q triplets per image.
During pre-training and fine-tuning,we employ bipartite matching similarly to prior work [55, 67, 6, 64], following in particular the matching cost proposed in [55].
The overall loss is then constructed as follows:
[Equation]
where L l 1 , L GIoU , L s , L o , L r denote the ℓ 1 loss for box regression, GIoU loss [53], CE loss forsubject and object classes, and Focal loss for relations (or Quality Focal loss [36] when applying
RQL or RPL), respectively.
The λ terms are fixed weights to balance multi-task training following[55], with λ 1 = 2 . 5 , λ 2 = 1 , λ 3 = 1 , λ 4 = 1 .
During pre-training, the label sequence is constructedfrom both in-batch and out-of-batch labels.
During fine-tuning, we use all text labels contained inthe dataset to form the label sequence (unlike pre-training, these labels fall within a pre-definedcategory list of limited size).
Note that we follow [55, 64] and exclude L s during fine-tuning since HOI detection detects only humans as subjects.
During inference, the confidence score for an objectis simply the top-1 score from the softmax distribution over objects, and the relation score is obtainedby multiplying the original score from the sigmoid function and the object score.
We rank relationscores and filter out the top- K within those correctly localised triplets (IoU > 0.5) for evaluation.
Kis set to 100 by default following [55, 64, 39].
Experiments
Datasets.
We use the Visual Genome (VG) [31] dataset for RLIP.
This dataset contains 108,077images annotated with free-form text for a wide array of objects and relations (100,298 objectannotations and 36,515 relation annotations).
The dataset is pre-processed prior to use (see supplementary material for details).
For downstream tasks, we conduct experiments on HICO-DET [5]and V-COCO [13].
HICO-DET contains 37,536 training images and 9,515 testing images, annotatedwith 600 HOI triplets derived from combinations of 117 verbs and 80 objects.
We evaluate under the Default setting.
V-COCO comprises 2,533 training images, 2,876 validation images and 4,946 testingimages annotated with 24 interactions and 80 objects.
Results are assessed under two scenariosdenoted as AP #1 role and AP #2 role as defined by the official evaluation code [13].
Note that all objectclasses in HICO-DET and V-COCO are identical to COCO.
Implementation details.
The basic architecture of the encoder and decoder are based on DETR [3]for ParSe and DDETR [66] for an additional variant named ParSeD .
A detailed architecture description of RLIP-ParSeD (which uses an additional transformer for cross-modal fusion) is providedin the supplementary.
For Parallel Entity Detection and Sequential Relation Inference , 3 decodinglayers are used.
The number of queries N Q is set to 100 during pre-training and 64 during fine-tuning(following [64]).
γ in the Focal loss is set to 2 following [55, 64].
N L in LSE is set to 500 to ensurecomputational tractability.
η in RPL is set to 0.3.
For pre-training and fine-tuning, the initial learningrate (LR) of the image and text encoders is set to 1e-5, while all other modules are set to 1e-4.
For
RLIP-ParSeD and ParSeD (object detection and relation detection pre-training), we pre-train modelon VG for 50 epochs and drop LR by a factor of 10 at epoch 40.
For ParSeD and RLIP-ParSeD, Wefine-tune for 60 epochs and drop LR at epoch 40 by a factor of 10.
For ParSe and RLIP-ParSe, Wefine-tune for 90 epochs and drop LR at epoch 60 by a factor of 10. The pre-training and fine-tuningstrategy follow above descriptions unless stated otherwise. Experiments are conducted on 8 Tesla V100 GPU cards with a minibatch size of 32.
Experimental protocols.
(i) To assess performance under fine-tuning and zero-shot with no finetuning (NF) scenarios, we evaluate on HICO-DET across three HOI sets under the mAP metric: Rare(HOIs with training samples less than 10, of which there are 138), Non-Rare (HOIs with samplesequal to or more than 10, of which there are 462) and Full (all HOIs, of which there are 600).
(ii) Toevaluate performance under the zero-shot formulation considered by [19, 20, 18], we report resultson unseen combinations (UC).
In particular, we report results under two settings: UC with rare-first(UC-RF) selection and UC with non-rare first selection (UC-NF), both of which are assessed acrossthree subsets: Unseen (120 HOIs), Seen (480 HOIs) and Full (600 HOIs).
(iii) To evaluate few-shottransfer performance, we follow [27] and sample subsets of training annotations from HICO-DET.
In detail, we sample 1% and 10% of the total annotations available among the HICO-DET trainingdata, ensuring that all objects and verbs (but not all combinations) exist in the selected annotations.
Similarly to the fine-tuning protocol, we evaluate on the Rare , Non-Rare and Full sets.
(iv) To assess [Table] [Table] the robustness of RLIP and its sensitivity to noise in the relation labels, we follow [34, 58] andartificially inject noise into the relation labels by randomly flipping a fixed ratio of verbs in HOItriplets across the training set.
4.1 Results and Analysis
Comparing object detection and relation detection pre-training with RLIP.
An assessment ofthe benefits of object detection pre-training using the COCO dataset may offer a somewhat optimisticevaluation of this approach, since COCO shares identical object classes with the downstream HOIdetection evaluation datasets HICO-DET and V-COCO (and in the latter case shares training images).
If we control for this effect by performing object detection pre-training on VG rather than COCO, weobserve a significant drop in performance for our ParSeD baseline (from 29.12 to 23.78 across the Full set on HICO-DET and from 61.8 to 41.4 on V-COCO for AP #1 role , as shown in Tab. 1).
However,since COCO lacks relation annotations, we investigate the benefits of RLIP on VG.
We observe that RLIP outperforms vanilla object detection pre-training by a wide margin (boosting performance from23.78 to 29.21 across the Full set on HICO-DET and from 41.4 to 53.1 on V-COCO for AP #1 role ),demonstrating the value of incorporating relations as a pre-training cue.
Another way to pre-train withrelations is to perform relation detection which is still inferior to RLIP (27.45 < 29.21), demonstratingthe importance of addressing relational semantic ambiguity in RLIP.
Leveraging off-the-shelf object detection data without relations.
As shown in the previousexperiment, while VG provides relation annotations, it provides a much weaker basis for objectdetection pre-training than COCO.
More broadly, we may expect that object annotations are likely tobe more readily available (and greater in scale) than relation annotations.
To mitigate this, a simplesolution is to simply load object detection parameters pre-trained from an object-annotated dataset(like COCO), to complement the abilities of RLIP.
Tab. 1 indicates that RLIP-ParSeD indeed benefitsfrom this approach, surpassing both object detection pre-training and RLIP (29.12, 29.21 → 30.70on the Full set).
We pre-trained on DETR for 150 epochs, outperforming an expert object detectionpre-training (31.79 → 32.84).
On V-COCO, there is a degradation of performance (62.5 → 61.9)which we believe may be caused by the reduced domain alignment (i.e. common training images)relative to COCO object-detection pre-training [55].
Comparing cross-modal regional alignment pre-training with RLIP.
We next compare to theuse of language-region alignment pre-training introduced by MDETR [27], which employed the
GoldG+ dataset (this comprises VG, COCO and Flickr30k [48] together with the correspondingannotations for referring expressions, VG regions, Flickr entities, and GQA [21]).
For comparison,we initialise RLIP-ParSe with MDETR’s parameters and then fine-tune on HICO-DET.
The resultsare reported in Tab. 1 as MDETR-ParSe.
Although MDETR makes use of a heavier backbone(ResNet-101) and additional pre-training data, RLIP-ParSe nevertheless surpasses this baseline witha lighter ResNet-50 backbone, demonstrating the effectiveness of RLIP for this task
[Table] [Table]
[Table]
Zero-shot HOI detection.
To assess performance under the zero-shot NF protocol, we comparewith other methods using RLIP-ParSe (initialised with COCO parameters followed by RLIP on VG)and ParSe (initialised with COCO parameters).
Note that during RLIP, we match subjects againsta diverse collection of categories.
However HOI detection only needs to detect person as subjects.
Consequently, for zero-shot NF inference, we filter out subjects that are not classified as person .
Wereport results in Tab. 2, where we observe that RLIP outperforms several fully fine-tuned methods.
We find, as expected, that regional language-image pre-training methods like MDETR fail under an NF evaluation, since its pre-training lacks the notion of explicit relations.
Under UC-RF and UC-NFprotocols (fine-tuning for 40 epochs under UC-NF to avoid over-fitting), RLIP-ParSe outperformsprevious methods and ParSe by performing RLIP on VG.
Few-shot transfer on HICO-DET.
To evaluate few-show transfer, we fine-tune ParSeD for 60epochs as above, while RLIP-ParSeD is fine-tuned for 10 epochs to avoid over-fitting.
The results areshown in Tab. 3.
We observe that RLIP significantly benefits few-shot fine-tuning relative to objectdetection pre-training, especially when data is scarce.
[Figure]
different sampling strategies for label sequence extension
DET.
[Table]
The influence of relation label noise.
To assess sensitivity to noise, we report fine-tuning resultson HICO-DET with increasing ratios of relation label noise in Fig.
We observe that as label noiseincreases, the COCO object detection pre-training adopted by prior work exhibits a greater degradationin performance (29.12 → 24.52, -4.60) than RLIP (29.21 → 25.68, -3.53) We also observe that wheninitialising RLIP-ParSeD with COCO pre-trained parameters, RLIP again helps to ameliorate noise,
[Table]
with a more limited loss of performance (30.70 → 26.87, -3.83) than COCO pre-training and a similar
[Table]
degradation to RLIP-ParSeD with random initialization.
Consequently, we deduce that RLIP offers aroute to mitigating label corruption and improving model robustness [17].
4.2 Ablation studies and analysis
human blow cake human blow cake human wash bus human fly kite human hold backpack
Ablation study: sampling strategy.
In Tab. 4, we present anablation study to assess the efficacy of out-of-batch samplingstrategies.
Intuitively, uniform sampling will up-weight descriptions from the tail of the distribution while frequency-basedsampling will preserve the distribution.
Both bring improvements to fine-tuning by providing additional negative samples.
However, by over-sampling descriptions from the tail, uniformsampling performs less well with common texts in the downstream task, and thus fares less well overall.
Visualisation of ParSe attention weights.
We visualise attention weights from ParSe for several example images in Fig.
We observe that ParSe attends distinct regions for subjects, objects and relations.
This aligns with our motivation that entitydetection is best supported by local context, while relation inference draws on additional spatial context, conditioning onsubjects and objects like hands and string, as well as the wet ground and sky where appropriate.
Ablation study of RLIP techniques.
In Tab. 5, we present an ablation study of the three proposedtechnical contributions.
We observe that each benefits both fine-tuning and zero-shot NF under allmetrics.
LSE attains a greater boost for the Non-Rare set (by sampling according to the training setdistribution).
On the other hand, RPL enhances results for the Rare set, likely due to the propensityof RPL to label rare descriptions as positive (due to the long-tailed distribution of text labels).
Understanding LSE, RQL and RPL with Uniformity and Alignment.
To gain insight intorepresentation quality, Wang et al.
[57] proposed two metrics, uniformity and alignment , whichwe employ here to better understand the influence of our contributions.
To this end, we performa zero-shot (NF) evaluation on HICO-DET, using bipartite matching to assign predicted tripletsto ground-truth labels.
We then calculate uniformity and alignment metrics for relation featuresvia
L u ( f ; t ) = log( E ( x,y ) i .
i .
d ∼ P data [ e − t ∥
f
( x ) − f ( y ) ∥ 22 ]) and L a ( f ; α ) =
E ( x,y ) ∼P pos [ ∥ f ( x ) − f ( y ) ∥
α 2 ] ,where α, t = 2 .
We present the results in Tab. 5, where we observe that LSE and RQL both reduceuniformity, the former through additional negative pairs and the latter by reducing the loss assignedto over-confident text labels.
We also observe that RPL yields better alignment through discoveringadditional positive labels.
The results indicate, however, a trade-off between the metrics—a usefuldirection for future work would be to determine how to find an appropriate balance between them.
Conclusion
In this paper, we propose RLIP as a pre-training strategy for HOI detection.
We show that RLIP,together with our additional technical contributions, boosts HOI detection performance under finetuning, zero-shot and few-shot evaluations, and improves robustness against noisy annotations.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] It’s mentioned in paragraph " Leveraging off-the-shelf object detection data without relations ", and discussed in
more detail in the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] It’s included in the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Data is publicly
available and instructions are included in the experiment section.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] They are included in Methodology and Experiments section.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] It’s included in the supplementary
material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]