C OMPOSITIONAL P ROMPT T UNING WITH M OTION C UES FOR O PEN - VOCABULARY V IDEO R ELATION D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Prompt tuning with large-scale pretrained vision-language models empowersopen-vocabulary predictions trained on limited base categories, e g. , object classification and detection.
In this paper, we propose compositional prompt tuningwith motion cues : an extended prompt tuning paradigm for compositional predictions of video data.
In particular, we present Relation Prompt (RePro) for
Open-vocabulary Video Visual Relation Detection (Open-VidVRD), where conventional prompt tuning is easily biased to certain subject-object combinationsand motion patterns.
To this end, RePro addresses the two technical challengesof Open-VidVRD: 1) the prompt tokens should respect the two different semantic roles of subject and object, and 2) the tuning should account for the diverse spatio-temporal motion patterns of the subject-object compositions.
Without bells and whistles, our RePro achieves a new state-of-the-art performance on two VidVRD benchmarks of not only the base training object and predicate categories,but also the unseen ones.
Extensive ablations also demonstrate the effectivenessof the proposed compositional and multi-mode design of prompts
I NTRODUCTION
[Figure]
Video visual relation detection (VidVRD)aims to detect the visual relationships betweenobject tracklets in videos as 〈 subject, predicate, object 〉 triplets (Shang et al., 2017; Chenet al., 2021;
Gao et al., 2022), e.g. , dogtowards - child shown in Figure 1.
Compared to its counterpart in still images (Chenet al., 2019), due to the extra temporal axis,there are usually multiple relationships withdifferent temporal scales, and a subject-objectpair can have several predicates with ambiguous boundaries.
For example, as shown in Figure 1, the action feed of child to dogco-occurs with several other predicates ( e.g. ,away , towards ).
This characteristic makes
VidVRD have more plentiful and diverse relations between objects than its image counterpart.
As aresult, it is impractical to collect sufficient annotations for all categories for VidVRD.
Therefore, tomake VidVRD practical, we should know how to generalize the model, trained on limited annotations, to new object and predicate classes unseen in training data.
To this end, we propose a new task: Open-vocabulary VidVRD (Open-VidVRD).
In particular,“open” does not only mean unseen relationship combinations, e.g. , dog-sit on-floor , but alsounseen objects and predicates, e.g. , bread and feed , as shown in Figure 1.
Recent works on suchgeneralization only focus on the unseen combinations (Chen et al., 2021; Shang et al., 2021) in VidVRD, or zero-shot transfer among semantically related objects in zero-shot object detection (Huanget al., 2022), e.g. , the seen dog class can help to recognize the unseen wolf .
However, they fail
[Figure] [Figure]
Groups
MotionCues [
[Figure]
[Figure]
][of][dog] [away] [
[Figure]
[ 𝒘𝒘] [ 𝒘𝒘] ... [ 𝒘𝒘 𝑳𝑳 ] [away] [ 𝒔𝒔] [ 𝒔𝒔] ...
[ 𝒔𝒔 𝑳𝑳 ] [towards] [away]
[Figure]
[Figure]
[Figure]
... [ 𝒘𝒘] [ 𝒘𝒘] ... [ 𝒘𝒘 𝑳𝑳 ] [ [Figure] [Figure] [Figure]
Selector [ 𝒐𝒐] [ 𝒐𝒐] ...
[ 𝒐𝒐 𝑳𝑳 ] ...
# #K [ 𝒔𝒔] [ 𝒔𝒔] ...
[ 𝒔𝒔 𝑳𝑳 ] [ 𝒐𝒐] [ 𝒐𝒐] ...
[ 𝒐𝒐 𝑳𝑳 ] ...
[Figure]
to generalize to the categories totally unrelated to the limited seen ones, where the transfer gap isunbridgeable, e.g. , bread in testing has no visual similarity with dog and child in training.
Thanks to the encyclopedic knowledge acquired by large vision-language models (VLMs) pretrained on big data (Radford et al., 2021; Li et al., 2022), we can achieve open-vocabulary relationdetection with only training data of limited base categories.
To bridge the gap between the pretrained and downstream tasks without extra fine-tuning the whole VLM model, a trending techniquenamed prompt tuning is widely adopted (Liu et al., 2021; Jin et al., 2022;
Zhou et al., 2022b).
For example, we can achieve zero-shot relation classification for the tracklets pair in Figure 2.
Wefirst crop the object tracklet regions in the video, and feed them into the visual encoder of VLM toobtain corresponding visual embeddings.
Then we use a simple prompt like “a video of [CLASS]”,feed it to the VLM’s text encoder to obtain the text embedding, and classify the object based on thesimilarities between visual and text embeddings.
Based on the tracklet classification results, for theexample of the pair dog and child , we can craft a prompt like “a video of dog [CLASS] child”, asshown in Figure 2(a), and similarly classify their predicates based on the predicate text embeddings.
Furthermore, we can replace the fixed prompt tokens with learnable continuous tokens, as shown in Figure 2(b), known as prompt representation learning, which has been widely applied to many vision tasks, e.g. , image classification (Zhou et al., 2022b;a) and open-vocabulary object detection (Guet al., 2021; Du et al., 2022; Ma et al., 2022).
Learning the prompt representation is actually introducing some priors for describing the context oftarget classes, and it excludes some impossible classes with the constraint of the context.
However,the prompt representations (either handcrafted or learned) in above approaches are monotonous andstatic, and learning the prompt sometimes might break the “open” knowledge due to overfitting tothe base category training data.
Modeling the prompt representation for video visual relations hassome specific characteristics that need to be considered:
• Compositional : The prompt context for predicates is highly related to the semantic roles ofsubject and object.
A holistic prompt representation might be sub-optimal for predicates.
Forexample, as shown in Figure 1, even the same predicate ( sit on ) in different relation triplets( dog-sit on-floor and child-sit on-stool ) have totally different visual context.• Motion-related : Predicates with different motion patterns naturally should be prompted withdifferent context tokens.
The naive prompt representation fails to consider the spatio-temporal
motion cues of tracklet pairs.
For example, the predicate towards shown in Figure 1 can beprompted as “a relation of [CLASS], moving closer”.
In contrast, eat and sit on can beprompted as “a relation of [CLASS], relative static”.
In this paper, we propose a compositional and motion-based Re lation Pro mpt learning framework:
RePro, as shown in Figure 2(c).
To deal with the compositional characteristic of visual relations,we set compositional prompt representations specified with subject and object respectively.
Withthis design, we can model the prompt context w.r.t.
semantic roles ( i.e. , subject or object ).
Forexample, a possible prompt can be “sth.
doing [CLASS]” for the subject and “sth.
being [CLASS]”for the object.
To consider the motion-related characteristic of predicate contexts, we design multimode prompt groups, where each group ( i.e. , each mode) is assigned with a certain motion pattern,and has its own compositional prompts for the subject and object. During the implementation, weselect a proper group according to the motion cues (patterns) in the subject-object tracklet pairs (cf.
Sec. 3.3).
Compared to some prompt tuning works which focus on category-based context (Zhouet al., 2022b) or instance-conditioned context (Zhou et al., 2022a; Ni et al., 2022), our motion-cuebased grouping has better cross-category generalization ability, and can avoid the over-fitting to basecategories.
We evaluate our RePro on the VidVRD (Shang et al., 2017) and VidOR (Shang et al., 2019) benchmarks.
Our experiment results show that RePro trained with only the samples of baserelation categories has a good generalizability to detect novel relations, and achieves the new stateof-the-art.
For example, it outperforms the top-performing method, i.e. , VidVRD-II (Shang et al.,2021), by 2 . 54% and 3 .
91% absolute mAP for SGDet and SGCls settings, respectively.
Our contributions in this paper are thus three-fold.
A new open-vocabulary setting for video visual relation detection task, i.e. , Open-VidVRD.
A compositional prompt representation learningmethod that models the prompt contexts for the subject and object separately.
A motion-cue-basedmulti-mode prompt groups that achieve a strong generalization ability.
R ELATED W ORK
Video Visual Relation Detection (VidVRD) was defined in Shang et al.
(2017; 2019) together withthe proposals of the VidVRD and VidOR benchmarks.
The task aims to spatio-temporally localizevisual relations between object tracklets.
Existing methods mainly focus on modeling better visualor spatio-temporal contexts (Qian et al., 2019; Shang et al., 2021), and detecting visual relationswith more granularity either by sliding windows (Liu et al., 2020) or temporal grounding (Gao et al.,2022). They mainly worked on the pre-defined (closed) sets of object and predicate categories.
Incontrast, our work is the first one to study the open-vocabulary VidVRD setting, where some objectand predicate categories are unseen in the training set. Zero-Shot Setting in Image and Video VRD . Existing VRD works, either in image domain (Tanget al., 2020; Kan et al., 2021) or video domain (Shang et al., 2021), only achieve zero-shot transferon the unseen triplet combinations, where the objects and predicates are seen in the training set. They ignore the model’s generalization ability to unseen object/predicate categories.
There is oneconcurrent work (He et al., 2022) proposes the open-vocabulary setting in image VRD.
However,they put the main emphasis on unseen object categories. Different from them, RePro generalizes themodel to recognize both object and predicate categories totally unrelated to the seen training ones.
Prompt Tuning for Open-vocabulary Visual Recognition .
Prompt tuning (Liu et al., 2021) hasbeen widely adopted in both image ( e.g. , open-vocabulary object detection (OV-Det) (Gu et al. ,2021; Du et al., 2022; Ma et al., 2022)) and video ( e.g. , zero-shot video action recognition (Linet al., 2022;
Ni et al., 2022; Ju et al., 2022;
Nag et al., 2022)) domains. For OV-Det, recent worksmainly focus on knowledge distillation from VLM and simply using handcrafted prompt (Gu et al. ,2021; Ma et al., 2022), or focus on prompt representation learning for object regions (Du et al., 2022;
Feng et al., 2022). For video action recognition, existing works mainly use fixed prompt (Nag et al.,2022), conventional learnable prompt (Ju et al., 2022), or prompt conditioned on the input videocontexts (Ni et al., 2022), and they all focus on the cross-frame attention or feature interaction.
Incontrast, our RePro learns the compositional prompt by leveraging the motion cues of subject-objectpairs, and has better cross-category generalization ability to detect novel visual relations in videos.
M ETHOD
To build the Open-VidVRD setting, we first divide the categories of a dataset into base and novelsplits.
Specifically, we denote C Ob and C On as the sets of base and novel object categories, respectively.
We use C Pb and C Pn to denote the sets of base and novel predicate categories, respectively.
Inthe training stage, we use all visual relation triplet samples from C Ob ×C Pb ×C Ob .
In the testing stage,we evaluate the model with the triplets sampled from all categories, i.e. , C Ob ∪ C
On and C Pb ∪ C Pn .
Based on this new setting, we first briefly introduce the preliminaries for open-vocabulary classification with pre-trained VLMs (Sec. 3.1).
Then, we introduce the proposed Open-VidVRD method
RePro, as illustrated in Figure 3, in which we first extend open-vocabulary object detection methods (Gu et al., 2021; Du et al., 2022) to open-vocabulary tracklet detection (Sec. 3.2), and thenperform open-vocabulary relation classification for each tracklet pair (Sec. 3.3).
3. P RELIMINARIES : O PEN -V OCABULARY C LASSIFICATION WITH P ROMPT
Fixed Prompt .
Pre-trained VLMs have a strong open-vocabulary classification ability (Li et al.,2022).
They first extract text embeddings for all categories by feeding handcrafted prompt ( e.g. , “a
[Figure]
[Figure] ,j . video of [CLASS]”) into the text encoder of VLM, where [CLASS] can be replaced with the classname of an arbitrary object or predicate.
Their output text embedding t c ∈ R d for each class c ist c = VLM txt ( W c ) , W c = [ w 1 , . .
. , w L , ˜ w c ] , ∀ c ∈ C Ob ∪ C
On or C Pb ∪ C Pn , (1) where W c is the prompt representation with L context token vectors and the class token vector ˜ w c .
Then, for each object tracklet with cropped video region, the corresponding visual embedding canbe extracted by the visual encoder of VLM, denoted as v i ∈ R d .
Similarly, the visual embedding oftracklet pair ( i, j ) can also be extracted ( e.g. , based on the union region), denoted as v i,j .
Therefore,the i -th region (generally denoted as v i ) can be classified by the cosine similarities w.r.t { t c } :ˆ c i = arg max c cos( v i , t c ) ,
∀ c ∈ C Ob ∪ C
On or C Pb ∪ C Pn , where cos( x , y )
=
x T y / ( ∥ x ∥∥ y ∥ ) .
Learnable Prompt .
Manually tuning the words in the prompt requires domain expertise and istime-consuming or not robust (Radford et al., 2021).
The substitute method is to learn the promptrepresentations from the training data (Zhou et al., 2022b;a).
Specifically, the context vector w i in
W c can be set as a learnable vector while ˜ w c is kept as fixed.
In the training stage, samples and{ ˜ w c } are from base categories.
In the testing stage, the L learned vectors in each W c are fixed andthen the model performs classification in the same way as in Eq.
3. O PEN -V OCABULARY O BJECT T RACKLET D ETECTION
Tracklet Proposal Generation .
Given a video, we first detect all the class-agnostic object trackletsusing a pre-trained tracklet detector, denote as T =
{ T i } Ni =1 , as shown in Figure 3(a).
Specifically, each tracklet T i is characterized with a bounding box sequence and the corresponding RoI
Aligned (He et al., 2017) visual feature, To reduce the computational overhead, we average the RoIfeatures of all bounding boxes ( i.e. , along the temporal axis of the tracklet) following (Shang et al.,2021), and denote it as f i ∈ R 2048 .
Tracklet Classification .
Instead of directly classifying object tracklets using VLM as Eq.
(2), wetrain a visual-to-language (V2L) projection module ϕ
o ( · ) to further utilize the annotations of baseclasses.
In particular, ϕ o ( · ) maps the RoI
Aligned feature
f i of each tracklet to the same semanticspace R d , i.e. , v ′ i =
ϕ
o ( f i ) .
Let t oc be the text embedding of object class c ∈ C Ob .
The probability oftracklet T i being classified as class c can be calculated as
[Equation]
where τ is a temperature parameter for softmax.
Training Objectives .
To train the object tracklet classification module, we assign base categorylabels to detected tracklets according to the IoU w.r.t ground-truth tracklets.
We call those trackletswith assigned labels as positive tracklets, otherwise negative tracklets.
Note that two cases can be recognized as negative tracklets: 1) the content is background; and 2) the content contains a novelobject category.
For these negative tracklets, we follow the loss used by Du et al.
(2022) that forcesthe prediction (from any negative tracklet) on each base class to be 1 / |C Ob | , i.e. , unlike any basecategory.
Therefore, the classification loss for positive and negative tracklets can be calculated as:
[Equation]
where T p and T n are the sets of positive and negative tracklets, respectively ( i.e. , T p ∪ T n = T ),and c ∗ i is the ground-truth label for the i -th positive tracklet.
We empirically found (in Sec. 3.2) thatusing the above negative tracklet loss works better than using the loss with a unique “background”class (Zareian et al., 2021; Gu et al., 2021).
Besides, following (Gu et al., 2021), we distill theknowledge from a pre-trained visual encoder to ϕ o ( · ) by aligning v ′ i to v i using l 1 loss, i.e. ,
[Equation]
Therefore, the overall loss for object tracklet classification is L cls =
L cls
[Equation]
pos +
[Equation]
neg + λ L distill ,where λ is a hyper-parameter to weight the classification and distillation.
3. O PEN -V OCABULARY V ISUAL R ELATION C LASSIFICATION
Based on the classified object tracklets, we perform open-vocabulary relation classification for eachtracklet pair, as shown in Figure 3(b).
First, we learn the prompt representations based on the preextracted visual embeddings of tracklet pairs, for which we introduce the compositional prompt representations and the motion-based prompt groups .
Second, we utilize the pre-extracted RoI Alignedfeatures to train a visual-to-language (V2L) projection module based on the learned prompt representations.
Finally, for testing, we extract all predicate text embeddings and classify the predicatesof each tracklet pair by using the RoI Aligned features and the trained V2L projection module.
Compositional Prompt Representations .
The compositional prompt consists learnable promptrepresentations S c and O c (of predicate class c ) for subject and object, respectively:
[Equation]
where s i and o i are the learnable context vectors and ˜ w c is the fixed class token for predicate c (forc ∈ C Pb in training phase and for all c in testing phase).
Then, the predicate text embedding t pc isgenerated by concatenating the two outputs of VLM given two prompts (respectively) as inputs, i.e. ,
[Equation]
[Equation]
Motion-based Prompt Groups .
We vary the prompt contexts based on the motion cues, i.e. , therelative spatio-temporal motion patterns, between each pair of subject and object.
In specific, wetake the generalized IoU Rezatofighi et al.
(2019) ( i.e. , GIoU) as the metric to calculate the motionpatterns.
For each tracklet pair 〈 T i , T j 〉 , we use a vector to represent a motion pattern:
m i,j = sign([ G si,j − γ, G ei,j − γ, G ei,j − G si,j ]) , and m i,j ∈ { + , −} 3 ,
[Equation]
8) where G si,j , G ei,j are the GIoU between subject-object for the start and end bounding boxes of theirtemporal intersection, respectively, and γ is a threshold for GIoU.
This definition considers twoperspectives: 1) whether the two tracklets are near or far ( i.e. , the first two terms of Eq.
(8)), and 2)whether they move toward or away to each other ( i.e. , the third term of Eq.
Overall, we have6 motion patterns (cf.
Sec.
A.2 for more details) and build 6 prompt groups correspondingly.
Eachgroup consists its own compositional prompt representations S c and O c as defined in Eq.
It’s worth noting that we aim to build a framework for learning motion-based multi-mode prompts.
The used GIoU-based approach (in our framework) is a simple and intuitive way to calculate motioncues.
This approach is not perfect, e.g. , it is poor to capture the motion pattern of tracklets movingback and forth.
We leave other fancier (motion capturing) approaches as future work.
Training Objectives .
Based on the above definition, we train the prompt representations with visual embeddings and relative position features.
For simplicity, we show the training process of asingle group (and in the end, we derive the final loss by averaging the losses across all groups).
For each tracklet pair 〈 T i , T j 〉 , we first calculate its motion cue, select the corresponding promptgroup, and extract the class text embeddings t pc for each predicate class c ∈ C Pb .
Then, we take thepre-extracted visual embeddings v i and v j , and concatenate them as the pair’s visual embeddingv i,j = [ v i , v j ] ∈ R 2 d .
Following Shang et al.
(2021), we additionally compute the relative positionfeature between bounding boxes of T i and T j , denoted as f si,j ∈ R 12 (cf.
Sec.
A.2 in the Appendixfor more details).
The predicted probability of predicate class c in this tracklet pair is thus
[Equation]
where ϕ pos projects f si,j to the same dimension as v i,j .
Going through all base classes (in C Pb ),the probability vector of tracklet pair 〈 T i , T j 〉 is generated and can be denoted as p i,j , i.e. , eachdimension is calculated by Eq.
In the training time, we assign the predicate labels according tothe IoU for each tracklet pair w.r.t the ground-truth tracklet pair by following Shang et al.
We denote the sets of positive and negative tracklet pairs as P p and P n , respectively.
Due to themulti-label setting of VidVRD, we use binary cross-entropy loss for relation classification.
Theground truth for positive tracklet pair in P p is a binary vector of dimension
|C Pb | , denoted as p ∗ i,j .
For those negative tracklet pairs in P n , we optimize the probability of each base class to zero, i.e. ,the ground truth is an all-zero vector.
The classification loss is thus calculated as
L pred-cls =
|P p | (cid:80)
[Equation]
Training V2L Projection Module .
Once the prompt representations are learned, we train a visualto-language (V2L) projection module to use RoI Aligned features { f i } as training data, and get ridof VLM’s visual encoder at inference time.
Given the learned prompt representations, we pre-extractthe predicate class text embeddings (denoted as { ˜ t pc } ) for each prompt group and fix them.
Formally,for each tracklet pair 〈 T i , T j 〉 , we concatenate their RoI features as f i,j = [ f i , f j ] ∈ R 4096 .
Then,we use a V2L projection module ϕ p to project it to the same dimension as text embeddings.
Similarto Eq.
(9), the probability of predicate class c is predicted as
[Equation]
where ϕ pos is the learned spatio-temporal projection layer and is fixed.
Then, we apply the same lossas defined in Eq.
(10), and compute the final total loss by averaging across all groups.
Discussions .
Intuitively, we can train the prompt representations together with the V2L projectionmodule ϕ p , and use the l 1 loss to align ϕ p ( f i,j ) to v i,j , i.e. , distill the knowledge from the pre-trainedvisual encoder to the V2L module.
We name this variant as RePro † .
We justify that our RePro worksbetter than RePro † due to: 1) directly using the teacher ( i.e. , v i,j ) to train the prompt is intuitivelybetter than using student ( i.e. , projected visual embedding), and 2) the distillation makes the V2Lmodule focus too much on the static visual alignment, rather than the dynamic relation informationlearned in the prompt.
In experiments, we empirically show the superiority of RePro over RePro † .
E XPERIMENTS
4. D ATASETS AND E VALUATION M ETRICS
Datasets .
We evaluated our method on the VidVRD (Shang et al., 2017) and VidOR (Shang et al.
,2019) benchmarks: 1) VidVRD consists of 1,000 videos, and covers 35 object categories and 132predicate categories.
We used official splits: 800 videos for training and 200 videos for testing. 2)
VidOR consists of 10,000 videos, which covers 80 object categories and 50 predicate categories.
Weused official splits: 7,000 videos for training, 835 videos for validation, and 2,165 videos for testing.
Since the annotations of VidOR-test are not released, we only evaluated models on validation set.
Evaluation Settings .
To build the open-VidVRD setting, we manually split base and novel categories by selecting the common object and predicate categories as the base split, and selecting therare ones as the novel split.
The detailed splits are given in Sec.
A.8 of the Appendix.
We trained themodel on the triplet samples of both base object and predicate categories in the training set.
Duringtesting, we evaluated the model on two settings: 1) Novel-split : triplet samples with all object categories and novel predicate categories, and 2) All-splits : triplet samples with all object and predicatecategories, in the testing set of VidVRD (or the validation set of VidOR). [Table]
Metrics .
We follow three standard evaluation tasks in scene graph generation (Zellers et al., 2018):scene graph detection (SGDet), scene graph classification (SGCls), and predicate classification(PredCls).
We apply these metrics to VidVRD: a detected triplet is considered to be correct if thereis the same triplet tagged in the ground truth, and both subject and object tracklets have a sufficientvolume IoU ( e.g. , 0.5) with the ground truth.
Following the standard setting (Shang et al., 2017), weuse mAP and Recall@K (R@K, K=50,100) as evaluation metrics.
4. I MPLEMENTATION D ETAILS
Tracklet Detector & Pre-trained VLM .
We used the Faster-RCNN (Ren et al.,
2015)-based VinVLmodel (Zhang et al., 2021) to detect frame-level object bounding boxes and extracted corresponding
RoI Alinged features, and then adopted Seq-NMS (Han et al., 2016) to generate class-agnostic objecttracklets.
The VinVL model was trained on out-of-domain image data without seeing any VidVRDdata.
For pre-trained VLM, we used ALPro (Li et al., 2022), which was pre-trained on a wide rangeof video-language data, and learned the fine-grained visual region to text entity alignment.
Relation Detection Details .
Following the popular segment-based methods (Qian et al., 2019;
Shang et al., 2017; 2021), we first detected visual relations in short video segments, and then adoptedgreedy relation association algorithm (Shang et al., 2017) to merge the same relation triplets.
Hyperparameters .
The detailed hyperparameter settings are left in Sec.
A.3 of the Appendix.
4. E VALUATE O PEN - VOCABULARY O BJECT T RACKLET D ETECTION
We evaluated the tracklet detection part of RePro on novel object categories, as shown in Table 1.
Comparison to ALPro . A straightforward baseline to achieve open-vocabulary tracklet detection isdirectly applying the pre-trained VLM (ALPro in our case) by inputting the tracklet regions into itsvisual encoder to perform classification, as in Eq. However, this has a significant computationaloverhead due to the heavy pipeline of ALPro’s visual encoder. In contrast, our RePro requiresmuch less computational cost, since we only use one projection layer ( i.e. , ϕ o ).
We thus compareour RePro with the above ALPro baseline. The results in row #4 show that RePro can achievecomparable performances on both datasets, with the projection layer ϕ o .
Negative Tracklet Classification .
How to model the negative sample is a key challenge as widelydiscussed in many open-vocabulary object detection works.
There are usually two approaches: 1)using a unique background embedding (BG-Embd) in addition to the class text embeddings (Zareianet al., 2021; Gu et al., 2021), and 2) only using the class text embeddings, and computing theloss of negative sample as L cls-neg in Eq.
(4) (Gu et al., 2021).
By comparing rows #3 and #4 of Table 1, we find that without using background embedding ( i.e. , as L cls-neg ) achieves better recall, andoutperforms the other by a large margin, especially on the more challenging VidOR benchmark.
Thisis because the tracklets recognized as negative may be due to the fact that they contain novel objects(rather than backgrounds), and aligning their embeddings ( i.e. , different novel class embeddings) toa unique background embedding hurts the model’s recognition ability on novel objects.
Distillation .
We verified the effectiveness of visual distillation ( i.e. , Eq.
(5)) by comparing rows #2and #3 of Table 1.
Obviously, the distillation helps RePro improve the detection recall by a largemargin, especially for the more challenging VidOR benchmark.
For row #1 , we can observe thatcomputing the negative tracklet classification loss as L cls-neg without distillation has extremely lowperformance. This is because forcing the classification probability of negative tracklet to be 1 / |C b |( i.e. , by L cls-neg ) and without the guidance from the teacher ( i.e. , without distillation) make the modelhas poor generalize ability to novel categories.
(RelTag) only considers the precision of relation triplets and ignores the localization of tracklets
[Table]
[Table] [Table]
The relation classification part of our RePro was trained separately by keeping the results of trackletdetection fixed. All of our experiments for relation classification used the same tracklet detectionresults (which is row #4 in Table 1).
Comparison to Conventional VidVRD SOTA Methods .
We compared our RePro with several
SOTA methods in the conventional VidVRD setting, and showed the results in Table 2.
The objecttracklets and features used in SOTA methods are not uniform since VidVRD is a very challengingtask (see Sec.
A.6 for details).
We can observe that even when our RePro is trained with only basecategory samples (while others are with both base and novel category samples), our performance on SGDet tasks is comparable to others’. When trained with both base and novel category samples, our
RePro outperforms all other SOTA methods.
Comparisons in the Setting of Open-VidVRD .
We compared the model performances in the setting of Open-VidVRD and showed results in Table 3.
Since our RePro is the first Open-VidVRDmethod, we compared it to ALPro (implemented as Eq. We also re-implemented the SOTAmethod VidVRD-II (Shang et al., 2021) and trained it on base category samples.
We replaced itsclassifier with text embeddings extracted by ALPro’s text encoder.
For both ALPro and VidVRD-II,we used a fixed (handcrafted) prompt “a video of relation [CLASS]”.
In addition, we reported theresults of RePro’s intuitive variant RePro † as mentioned in the “ Discussion ” of Sec. 3.3.
From the results in Table 3, we can observe that our RePro outperforms ALPro, VidVRD-II and
RePro † by a large margin on both Novel-split and All-splits.
By comparing RePro to ALPro, weshow that, unlike that in tracklet classification, directly applying pre-trained VLM to relation classification is sub-optimal and achieves poor performance.
By comparing RePro to VidVRD-II, wedemonstrate the superiority of our prompt tuning framework over the fixed prompt design.
By comparing RePro to RePro † , we validate the effectiveness of our training scheme for RePro.
4. A BLATION S TUDIES
We conducted careful ablation studies as shown in Table 4. Since the compositional and motionbased prompt design is one of our main contributions, we conducted ablations w/o either of them(rows #1, #2 and #5 ). To further show the effectiveness of our motion pattern design, we designedtwo variants, i.e. , rows #3 (Ens) and #4 (Rand). Their detailed settings are enumerated as follows:
#1 : training with neither compositional prompt nor motion
cues Specifically, we learned a singleprompt representation W c as in Eq.
The obtained predicate text embedding has the half dimensions of the t pc in Eq.
Therefore, we calculated the visual embeddings of a tracklet pair [Table] as v i,j = v i − v j (different from the concatenated vector v i,j in Eq.
#2 : training with compositional prompt but without motion cues.
#3 (Ens) : training with compositional prompt, but theprompt is randomly selected from the 6 groups without considering motion cues.
For testing, theprompts are ensembled by averaging. #4 (Rand) : training as in #3 , but the prompt is randomlyselected at testing. #5 : our RePro, where the prompt is chosen according to the motion cues.
[Table]
Compositional Prompt .
By comparing theresults in #1 and #2 , we can observe that thecompositional prompt can effectively improvethe performance on Novel-split.
Meanwhile,the improvement on All-splits is not significant.
We conjecture that the base relations,the majority of All-splits, require less compositional semantic contexts for prompt learning.
Motion-based Prompt Groups .
By comparing our RePro ( #5 ) vs. #2 , we can observethat with the help of motion cues, our ReProachieves significant improvements of recall onall tasks, and also achieves considerable improvements in mAP on most tasks.
We can seethat the improvement on Novel-split is moresignificant than that on All-splits, e.g. , 13.55% → 15.20% vs. 15.09% → 15.84% on R@100 of SGDet,showing that the motion-based prompt has a better generalizability for detecting novel relations.
Besides, if comparing RePro ( #5 ) to Ens ( #3 ), we can see that RePro outperforms Ens in Novel-spliton most tasks, and achieves considerable improvements for All-splits on all tasks.
Compared to Rand ( #4 ), RePro achieves clear improvements on most metrics for both Novel-split and All-splits.
Ablations on VidOR .
We conducted the same ablation studies on VidOR-val, as shown in Table 5.
Firstly, we can observe that the compositional prompt representation shows its efficiency on both
Novel-split and All-splits, e.g. , 0.86% → 1.72% and 9.49% → 10.06% on R@50 of SGCls.
For themotion-based prompt groups, the improvement of RePro is small due to the biased data distribution (Li et al., 2021), i.e. , the predicate categories strongly depend on the visual cues of subject andobject tracklets, making the model predict relations simply based on object appearances withoutconsidering motion cues.
More results on VidOR are left in the Appendix (Sec. A.7).
C ONCLUSIONS
In this paper, we introduced a challenging video visual relation detection task on open-vocabularydata — Open-VidVRD.
We analyzed two key characteristics, i.e. , compositional and motion-related,when applying the method of prompt tuning in this new setting. We proposed a novel method called
RePro that learns compositional prompt representations while considering motion-based contextsin the video.
Our evaluations on both conventional and open-vocabulary datasets show a clearsuperiority of RePro for tackling video visual relation detection tasks.
E THICS AND R EPRODUCIBILITY STATEMENTS
Ethics statement .
The open-vocabulary video visual relation detection (Open-VidVRD) that weintroduced in this paper is a general extension of conventional VidVRD, and there are no knownextra ethical issues in terms of the Open-VidVRD task and the proposed RePro model.
As forthe pre-trained visual-language model (VLM) used for Open-VidVRD, the large-scale pre-trainingdata might contain some videos and captions involved with discrimination/bias issues.
When applying pre-trained VLM to Open-VidVRD, the model tends to predict relations based more on thepre-trained knowledge, and focus less on the visual cues.
For example, when the pre-trained datainvolved with unethical videos and captions, a model might predict “person-punch-dog” given avideo of person caressing dog, which implies a person is abusing animals.
To avoid the potentialethical issues, we can design algorithms to filter out those unethical training data for VLMs.
For
Open-VidVRD models
, we can also introduce some common sense knowledge and design somerule-based methods to filter out those unreasonable relation triplets that involve ethical issues.
Reproducibility Statement .
Our RePro is mainly implemented based on the realsed code of ALPro (Li et al., 2022), VinVL (Zhang et al., 2021), and VidVRD-II (Shang et al., 2021).
We firstmodified the code of VinVL to fit the video data and to extract object tracklets in each video.
Then
We modified the code of VidVRD-II to be compatible with the visual and text encoder of ALPro, andto fit the Open-VidVRD setting.
We provide the the detailed base/novel split information of objectand predicate categories in the Appendix (Sec. A.8) to ensure all experiments can be reproduced.
When training the RePro model and its variants, we manually set the random seed and fixed the seedfor all experiments to ensure they can be reproduced.
We also provide the code of our RePro modeland the training/evaluate scripts in the supplementary materials.
A A PPENDIX
This Appendix has the following contents:
• More details about the relative position feature are in Sec.
A.1.
• More details about the motion pattern are in Sec.
A.2.• The detailed hyperparameters are given in Sec. A.3.• Analysis of the performance improvement in different predicate groups are in Sec. A.4.• Potential improvements of the motion pattern design are introduced in Sec. A.5.•
The detailed experiment settings of the compared SOTA methods are introduced in Sec.
A.6.• More experiment results on VidOR are provided at Sec.
A.7.• The detailed base/novel split information of object and predicate categories are in Sec.
A.8.
A. R ELATIVE P OSITION F EATURE FOR T RACKLET P AIRS
We compute the relative position between bounding boxes of subject-object tracklet pair 〈 T i , T j 〉 by following Shang et al.
Specifically, we compute the relative position feature between subject-object for the beginning and ending bounding boxes of their temporal intersection.
For the beginning bounding boxes of the subject and object, the position feature is calculated as:
[Equation]
(12) where ( x i , y i ) is the central coordinates of T
i ’s beginning bounding box, ( w i , h i ) is its width andheight, and t i is the frame ID of this beginning bounding box.
( x j , y j , w j , h j , t j ) is defined similarlyfor T j .
L seg is the number of frames in each video segment, and following Shang et al.
(2021), we set
L seg = 30 .
The relative position feature between the ending bounding boxes of 〈 T i , T j 〉 is definedsimilarly as f Bi,j , and is denoted as f Ei,j .
The final relative position of 〈 T i , T j 〉 is concatenated asf s i,j = [ f B [Figure]
A.
D ETAILS A BOUT THE M OTION P ATTERNS
[Figure] We provide a schematic of the motion patterns defined in Eq.
(8), as shown in Figure 4.
According tothe definition in Eq.
(8), m i,j has only 6 possible values, i.e. , [+ , − , +] and [ − , + , − ] are impossible. A. H YPERPARAMETERS
We set ϕ o , ϕ p and ϕ pos
all as two-layer MLPs with hidden dimension 768.
The λ for weightingthe distillation loss ( i.e. , Eq.
(5)) was set as 5.0.
The prompt length L was set as 10. The softmaxtemperature τ was set as learnable. The GIoU threshold γ was chosen based on the statistics of thetraining set, by making the tracklet pairs evenly distributed w.r.t different motion patterns. In ourimplementation, γ was set as -0.3 for VidVRD and -0.25 for VidOR.
We trained our RePro using
Adam (Kingma & Ba, 2014) with a learning rate 1e-4, and stopped the training when SGDet mAPdrops.
A.
A NALYSIS OF THE PERFORMANCE IMPROVEMENT IN DIFFERENT PREDICATE GROUPS
We evaluated the Recall@100 in the PredCls setting of some predicate groups (grouped by theprefix of predicate words) at the novel-split of the VidVRD dataset.
We compared our RePro withthe mean ensemble (Ens) and random select (Rand) variants of RePro (refer to Sec. 4.5).
The resultsshow that the improvements of motion-related predicates are much larger than other context-relatedpredicates.
For example, we have 9.68% absolute improvements on “run” (e.g., “run past”, “run nextto”).
While for those predicates that can be roughly inferred by the context (e.g., “fly”, “swim”), ourapproach has limited contributions.
This indicates that the performance improvements of our ReProare largely attributed to motion cues.
[Table]
A. P OTENTIAL I MPROVEMENTS OF THE M OTION P ATTERN D ESIGN
The current proposed GIoU-based motion pattern design can be further improved.
Based on our proposed motion-pattern based prompt group learning framework, we can design other fancier motioncapturing approaches, e.g. , automatically learning the motion primitives from the training set.
Thenfor each test sample, the motion pattern can be decomposed as the weighted combination of motionprimitives.
Consequently we can use the weighted combination of the prompt representations as thedesired prompt representation.
We leave this as future work.
A. D ETAILED E XPERIMENTAL S ETTINGS OF T HE C OMPARED SOTA M ETHODS
Since VidVRD is a very challenging task, The object tracklets and features used in SOTA methodsin Table 2 are not uniform.
The object tracking algorithms for tracklet generation include SeqNMS (Han et al., 2016) and deepSORT (Wojke et al., 2017).
The features include RoI Aligned features, I3D features (Carreira & Zisserman, 2017), and improved dense trajectory (iDT) features (Shang et al., 2017)
Here we enumerate their details as follows:
• Su et al.
(2020) uses Seq-NMS for tracklets generation, and improved dense trajectory (iDT)feature and relative motion feature of tracklet pairs for relation classification.
• Liu et al.
(2020) uses deepSORT for tracklets generation, and uses RoI feature, I3D feature, and
relative motion feature of tracklet pairs for relation classification.
• Li et al.
(2021) uses Seq-NMS for tracklets generation, and uses RoI feature and relative motion
feature of tracklet pairs for relation classification.
• Gao et al.
(2022) uses deepSORT for tracklets generation, and uses RoI feature, I3D feature for
relation classification.
• Our RePro uses Seq-NMS for tracklets generation, and uses RoI feature and relative motionfeature of tracklet pairs for relation classification.
A. M ORE E XPERIMENT R ESULTS ON V ID OR
[Table]
We provide more experiment results for our RePro on the validation set of VidOR, as shown in
Table 7.
We first compare our RePro with using ALPro directly perform relation classification as
Eq.
We find that ALPro performs slightly better than RePro on novel-split, because ALProdoesn’t have the trend of fitting base categories.
However, ALPro performs much worse than RePro in All-splits due to not trained on base categories.
Furthermore, we also compare RePro withthe VidVRD-II (Shang et al., 2021) baseline and the variant RePro † .
We can observe that ReProoutperfoms both VidVRD-II and RePro † by a large margin on both novel-split and all-split.
A. D ETAILED BASE / NOVEL CATEGORIES OF OBJECT AND PREDICATE FOR V ID VRD ANDV ID OR
We list the base/novel categories of object and predicate for training and evaluating our RePro andother baselines in all experiments.
We also provide more statistics information in the supplementarymaterials
VidVRD Object 25 base object categories:“airplane”, “bicycle”, “bird”, “bus”, “car”, “dog”, “domestic cat”, “elephant”, “hamster”,“lion”, “monkey”, “rabbit”, “sheep”, “snake”, “squirrel”, “tiger”, “train”, “turtle”, “whale”,“zebra”, “ball”, “frisbee”, “sofa”, “skateboard”, “person”10 novel object categories“horse”, “watercraft”, “giant panda”, “fox”, “red panda”, “cattle”, “motorcycle”, “bear”,“antelope”, “lizard”
VidVRD Predicate71 base predicate categories:
“jump past”, “jump with”, “kick”, “lie above”, “lie beneath”, “lie inside”, “lie with”,“move above”, “move away”, “move next to”, “move past”, “move toward”, “past”,“pull”, “run above”, “run away”, “run beneath”, “run next to”, “run toward”,“sit behind”, “sit beneath”, “sit inside”, “sit next to”, “stand above”, “stand beneath”,“stand inside”, “stand with”, “stop above”, “stop beneath”, “stop next to”, “stop with”,“swim behind”, “swim beneath”, “swim next to”, “toward”, “walk above”, “walk away”,“walk beneath”, “walk past”, “walk toward”
VidOR Object50 base object categories:“adult”, “child”, “toy”, “dog”, “baby”, “car”, “chair”, “table”, “sofa”, “ball/sports ball”,“screen/monitor”, “cup”, “bicycle”, “guitar”, “bottle”, “backpack”, “handbag”,“baby seat”, “camera”, “cat”, “cellphone”, “bird”, “sheep/goat”, “laptop”, “ski”, “stool”,“watercraft”, “duck”, “bus/truck”, “bench”, “fruits”, “baby walker”, “horse”, “bat”, “dish”,“electric fan”, “kangaroo”, “motorcycle”, “lion”, “hamster/rat”, “refrigerator”, “elephant”,“faucet”, “cake”, “penguin”, “sink”, “piano”, “microwave”, “cattle/cow”, “aircraft”30 novel object categories:“antelope”, “vegetables”, “panda”, “rabbit”, “fish”, “train”, “snowboard”, “suitcase”,“squirrel”, “leopard”, “chicken”, “skateboard”, “traffic light”, “surfboard”, “camel”,“racket”, “bread”, “bear”, “oven”, “scooter”, “frisbee”, “stop sign”, “turtle”, “stingray”,“pig”, “crab”, “crocodile”, “toilet”, “tiger”, “snake”
VidOR Predicate30 base predicate categories:“next to”, “in front of”, “watch”, “behind”, “away”, “towards”, “beneath”, “above”,“hold”, “lean on”, “speak to”, “ride”, “hug”, “touch”, “carry”, “hold hand of”, “bite”,“push”, “pull”, “play(instrument)”, “grab”, “release”, “pat”, “inside”, “lift”, “caress”,“point to”, “press”, “hit”, “use”20 novel predicate categories:“kick”, “chase”, “wave”, “smell”, “throw”, “feed”, “kiss”, “wave hand to”, “shout at”,“drive”, “clean”, “lick”, “squeeze”, “shake hand with”, “get off”, “knock”, “cut”, “open”,“get on”, “close”