Text-Adaptive Multiple Visual Prototype Matching for Video-Text Retrieval
Abstract
Cross-modal retrieval between videos and texts has gained increasing interestbecause of the rapid emergence of videos on the web.
Generally, a video containsrich instance and event information and the query text only describes a part of theinformation.
Thus, a video can have multiple different text descriptions and queries.
We call it the Video-Text Correspondence Ambiguity problem.
Current techniquesmostly concentrate on mining local or multi-level alignment between contents ofvideo and text (e.g., object to entity and action to verb).
It is difficult for thesemethods to alleviate video-text correspondence ambiguity by describing a videousing only one feature, which is required to be matched with multiple different textfeatures at the same time.
To address this problem, we propose a Text-Adaptive
Multiple Visual Prototype Matching Model.
It automatically captures multipleprototypes to describe a video by adaptive aggregation on video token features.
Given a query text, the similarity is determined by the most similar prototype tofind correspondence in the video, which is called text-adaptive matching.
To learndiverse prototypes for representing the rich information in videos
, we propose avariance loss to encourage different prototypes to attend to different contents ofthe video.
Our method outperforms state-of-the-art methods on four public videoretrieval datasets.
Introduction
Cross-modal retrieval has gained popularity as a means of matching semantically similar samplesacross multiple modalities (14; 13; 2).
With the rapid increase in the number of videos, Text-Video
Retrieval (TVR) has garnered a lot of interest.
Given a text description/video, we aim to retrieve thecorresponding video/text description in the gallery.
TVR remains a challenging task since the videoand text have different semantical descriptions which results in modality gap.
To overcome the modality gap issue, most existing methods generally follow a dual encoding networkparadigm which encodes video and text through a visual encoder and a language encoder into thesame embedding space, respectively (14; 13; 2; 10; 15).
And then they focus on learning a similarityfunction between the representations of text and video, which provides the basis for ranking thegallery video/text.
However, the modality gap issue is still not well solved due to the misalignment.
Some methods aim to extract additional information from the video to get more fine-grained videofeatures to be aligned with corresponding text features (10; 29; 16).
They focus on properly combiningdifferent video modalities, such as appearance, audio, speech, etc.
Some methods pay attention tomulti-level alignment, including text entity to video object, text verb to video action, and text phraseto video event(15; 31).
However, most of them ignore the fact that a video contains rich instanceand event information, while the query text only describes a part of the information, which leads tovideo-text correspondence ambiguity problem.
Concretely, as shown in Figure 1, due to the multiplecharacters in the video, the video has different text descriptions: “a woman does something”, “‘a man
Motivation
[Figure]
does something” or “a group does something”.
The text descriptions are very diverse and scattered inthe feature space.
With diverse video contents , it is infeasible to align a single video feature with thediverse text features.
Neither additional information-based methods nor multi-level alignment-basedmethods consider such correspondence ambiguity since both of them match a single video featurewith multiple text features.
In this work, we focus on video-text retrieval problem and aim to mine fine-grained and diverseevent descriptions in a video and adaptively match them with multiple text descriptions to mitigatecorrespondence ambiguity issue for better alignment.
Specifically, we automatically capture multipleprototypes for each video to account for its rich instance and event information by aggregatingon token features.
Based on multiple prototypes of a video, we aim at finding many-to-manycorrespondence between prototypes and text representations to reduce ambiguity.
To this end, wepropose a Text-Adaptive Multiple Visual Prototype Matching Model.
We follow the dual encodingnetwork paradigm as previous methods(2).
First, we input video into visual encoder to get the videotoken features.
Then we employ a token-adaptive aggregation method to generate several visualprototypes from the final video token features.
As shown in Figure 3, one prototype is the class tokenand the others are the linear combinations of all the token features.
The text feature encoded by thelanguage encoder is matched with the most similar video prototype.
This text-adaptive matching canhelp find text-to-prototype correspondence in video and alleviate the correspondence ambiguity.
Tofurther force the prototypes to represent rich information in the video, we propose a variance loss toenhance the diversity of prototypes to attend to different contents of the video.
Extensive experiments were conducted on the MSR-VTT (30), DIDEMO(11), MSVD(5), and
LSMDC (24) to evaluate video-text bi-directional retrieval performance.
Experimental results showthat our proposed model can achieve state-of-the-art results on all the above datasets.
Ablation studieswere carried out to evaluate the effectiveness of each single part of our model.
The contributions of our work are three folds.
We propose to automatically generate multipleprototypes for video to account for its rich information and introduce a text-adaptive matching strategyto adaptively find correspondence between texts and prototypes.
We further design a varianceloss to effectively encode diverse information into prototypes to attend to different contents in thevideo and boost alignment.
Our proposed model can achieve new state-of-the-art results on fourbenchmark datasets.
Extensive experiments demonstrate its effectiveness and generalization.
Related Work
Text-Video Retrieval.
The methods for text-video retrieval can be classified into two categories:pre-extracted feature based (16; 29; 15; 19) and end-to-end training (2; 13).
Given the small size ofmost video-text retrieval datasets, the dominant paradigm for video retrieval has been to combinepre-extracted features from “expert” models, such as models trained for a variety of tasks and onmultiple modalities such as face, scene, and object recognition, action classification, and soundclassification.
CE (16) and MMT (10) compute the overall similarity for a video-text pair obtainedas a weighted sum of each expert’s similarity with the text.
DMM (29) uses hierarchical video-textalignment to explore the diverse properties in multi-modal cues for fine-grained alignment.
Another
[Figure]
Inner Production ⨂ ⨂ ⨂ ⨂ 0.
MAX 0.
TextProj
VideoProj
Similarity !
[Figure] [Figure]
Then we employ a token-adaptive aggregation method to generate several visual prototypes from the final video token features.
The text feature from the language model is matched with the video prototype which is most similar to text after the projection layer.
important paradigm is to first train on a large-scale vision-and-language dataset and then finetuneon downstream dataset.
ClipBERT (13) proposes an efficient end-to-end technique based on sparsesampling and demonstrates that the pretrained model by image-language dataset enables superiorvideo-text retrieval initialization.
Frozen (2) proposes an end-to-end trainable model that can benefitfrom large-scale image and video captioning datasets.
Most of them, however, ignore the fact that avideo contains rich instance and event information, whereas the query text only describes a portionof the information, resulting in video-text correspondence ambiguity.
We propose a Text-Adaptive
Multiple Visual Prototype Matching Model to help mitigate this issue.
Vision Transformer.
Now vision transformer becomes the visual basic backbone in most of thestate-of-the-are methods for vision-language tasks (22; 13; 2; 14).
BLIP(14) makes effective use ofnoisy web data by bootstrapping captions, in which a captioner generates synthetic captions and afilter removes the noisy ones.
Frozen(14) makes a minor modification to the Divided Space-Timeattention introduced by (4).
Our model is based on the dual framework of Frozen.
Method
A video, in general, contains a lot of rich instance and event information, and the text description onlydescribes a portion of it.
A video can have many characters/events/actions, whereas a text typicallyonly describes one character, action, or event.
So mining representation of multiple essential eventsfacilitates the text-video matching by reducing the modality gap properly.
As a result, a model thatgenerates multiple visual prototypes is better than that generates one feature.
With this rationalein mind , we propose our Text-Adaptive Multiple Visual Prototype Matching Model.
To generateseveral visual prototypes from video, we use a token-adaptive aggregation method on the final videotoken features.
Given a query text, the most similar prototype is used to find correspondence in thevideo, a process called as text-adaptive matching.
We also propose a variance loss to encouragedifferent prototypes to attend to different contents of the video in order to learn diverse prototypesfor representing the rich information in videos.
We define the problem in Section 3.1, present
Text-Adaptive Multiple Visual Prototype Matching Model in Section 3.2, describe our variance lossin Section 3.3, and explain our dual encoder in Section 3.4.
3.1 Problem Definition
For the video-text retrieval task, we are given M videos V =
{ V i } M − 1 i =0 and N text descriptions T =
{ T i } N − 1 i =0 .
This function should maximize positive cross-modal sample similarity while minimizingnegative cross-modal sample similarity.
The goal of this task is to learn a similarity function
S : ( T, V ) → R .
It should maximize positive cross-modal sample similarity while minimizing negative cross-modal sample similarity.
In general, we fed video and text to into an vision encoderand language encoder to get normalized feature f v , f v .
Then the similarity function S is defined by
S t,v = ⟨ f t , f v ⟩ .
The similarity of text-video is calculated as the inner production of f v , f t .
3.2 Text-Adaptive Multiple Visual Prototype Matching Model
The similarity function of Equation 1 ignores the Video-Text Correspondence Ambiguity problem:some videos have multiple different text descriptions and queries.
To furthermore illustrate thisproblem, we show the text descriptions diversity of videos by showing the inter-text similarity andthe minimum of intra-similarity distribution in 3.3.
We define inter-text similarity as the similarityof different text descriptions for different videos, intra-text similarity as the similarity of differenttext descriptions for same video.
The less the minimum of intra-text similarity, the diversity of thetext descriptions for a video.
As shown in Figure 3.3, at least one-third of the minimum of intra-textsimilarity is less than the mean of inter-text similarity on features from both models.
It demonstratesthat many videos have diverse captions and contents.
So a single feature in some videos cannot bealigned with multiple text features at the same time.
As a result, using multiple visual prototypes toalign with different text is critical.
An intuitive idea for producing multiple visual prototypes is to generate a feature for each frame/clipof video as a prototype.
However, one frame/clip can contain multiple characters and events atthe same time.
So a better way is to generate many potential prototypes for each frame, then weaggregate them together into multiple visual prototypes.
Vision Transformer (2; 13) have shownit’s effectiveness on learning multi-modality shared representation.
So we take the final layer tokenfeatures z ∈ R B × D of vision transformer as the potential prototypes.
B, D represents the number oftokens, the dimension of token feature.
There are many methods to aggregate the token features, such as transformer decoder, graph neuralnetwork, LSTM etc.
However, we find that the mask-based production of k th is enough.
The k thprototype
[Equation]
, where z j , f mask represents the jth token feature of z , mask-generating function.
f mask is implemented by a linear layer, followed by relu function.
In general, the class token feature is regardedas the final output of vision transformer (2; 13).
So we expand p
k
+
=
z cls , which means we have
K + 1 visual prototypes.
For comparing visual prototypes and text feature on same embeddingspace, we input p to a projection linear layer, do normalization and get multiple vision featuresf
ve ∈ R ( K +1)
× D .
Matching text to video becomes a matter of matching one text feature to multiplevisual prototypes.
The text is aligned with the video if one of the prototypes is very close to it.
Basedon this text-adaptive matching, we have new similarity function as
[Equation]
3.3 Variance Loss
Define mask m ji,k = f mask ( z j ) i,k , which represents the mask weight on j th token from k th prototypeof i th sample .
It’s an intuitive idea to different prototypes of same video to attend to different contentsand the masks have different attention on different tokens.
However, as shown in Figure 3.3,in somevideos, the text descriptions are very similar.
The videos may be too short and only have a character,event or action.
The prototypes and masks are similar .
Instead, we’d like all masks of all videos topay attention to different token positions.
Inspired by the variance loss on feature of (3), we designthe variance loss on mask as
[Equation]
,where D is the regularized standard deviation defined by
[Equation]
[Figure]
As a result, the diversity of video contents can be represented by the minimum of intra-text similarity.
It can be seen that at least one-third of the minimum of intra-text similarity is less than the mean of inter-text similarity on both graphs.
It demonstrates that many videos have diverse captions and contents and it is necessary to use multiple visual prototypes to align with different text descriptions.
, γ is a constant target value for standard deviation, ϵ is a small scalar to prevent numerical instabilities.
The variance loss forces the variance of all prototype mask values for each token to be large.
It alsoencourages diverse prototypes for representing the rich information in videos.
3.4 Dual-Encoder Framework
We use a end-to-end training dual-encoder framework.
Frozen (2), a basic and successful dual-encoderarchitecture (separate visual encoder and text encoder), serves as our foundation in this study.
It’sdesigned for video-language pretraining to match raw-pixel video and raw text with a contrastive lossin dual-encoder framework.
The language and vision encoder is multi-layer bidirectional transformerand Timesformer (4) with some changes.
To train this dual-encoder framework, the matched text-video pairings in the batch are regarded aspositive, while all other pairwise combinations in the batch are deemed negative.
Each video appearsat most once in each batch to prevent duplicates.
With L text-video pairs in a batch, the symmetricalcontrastive loss is calculated as follows
[Equation]
where τ is the temperature and s ii , s ij is the similarity of i th paired text-video, i th text to j th video.
Then the ranking of videos/texts is determined by this similarity calculation for single text/videoquery.
So overall loss is defined by
[Equation]
where α controls the weight of L var .
Experiment
We evaluate our model’s performance on several public datasets.
In the sections that follow, we firstintroduce the datasets and metrics for comparing results, as well as implementation details.
Then wedemonstrating the overall performance of our model on the datasets.
Finally, we present ablationstudies to evaluate the effectiveness of each component of our model.
[Table] [Table]
Frozen
[Table] [Table]
[Table]
Ours 17.
[Table] [Table]
[Table]
4.1 Datasets and Metrics
MSR-VTT
MSR-VTT(30) contains ten thousand YouTube videos, each of which is accompanied by 20 naturalsentences that describe it.
Following (2), we train on 9K train+val videos and report results on the1K-A test set.
MSVD
MSVD (5) consists of 80K English descriptions for 1,970 videos from YouTube, with each videocontaining 40 sentences each.
Following (2), we use the standard split of 1200, 100, and 670 videosfor training, validation, and testing.
DIDEMO
DIDEMO (11) contains 10K Flickr videos annotated with 40K sentences.
Following (2), we evaluateparagraph- to-video retrieval, where all sentence descriptions for a video are concatenated into asingle query.
LSMDC
LSMDC (24) consists of 118,081 video clips sourced from 202 movies.
The validation set contains7,408 clips and evaluation is done on a test set of 1,000 videos from movies disjoint from the trainand val sets.
This follows the protocol outlined in (25).
[Table] [Table]
Metrics
We evaluate retrieval performance using common information retrieval metrics such as Recall at K(R@K and K=1, 5, 10) and Median Rank (MedR).
R@K is the proportion of test queries for which atleast one relevant item is found among the top-K retrieved results.
The MedR calculates the medianrank of correct items in the returned ranking list, with a lower score indicating a better model.
Toreflect overall retrieval performance, we also take the sum of all R@K as SumR.
4.2 Implementation Details
We fintune the pretrained model from Frozen (2) on text-video retrieval dataset.
The visual encoderhave 12 attention blocks, patch size P = 16, sequence dimension D = 768, and 12 heads.
The textencoder takes the architecture of DistilBERT base-uncased (27).
The dimensionality of the commontext-video space is set to 256.
We take some training settings same with Frozen (2): we use 8frames per video when training model;
The temperature hyperparameter σ for the loss defined in
Equation 6 is set to 0.05; We randomly crop and horizontally flip during training and center crop themaximal square crop at test time for visual augmentation; All videos are resized to 224 × 224 as input;
We use text augmentation during training for paragraph-retrieval settings, randomly sampling andconcatenating a variable number of corresponding captions per video;The optimizer is Adam.
We usea different learning rate scheduler: warmup for 5 epochs with a linearly growing learning rate from0.0 to 0.00003, then use cosine decay scheduler to guide the learning rate from 0.00003 to 0.0 forother 45 epochs.
The batch size is set as 64.
The number of extra prototypes besides class token K isset as 3 .
The variance loss parameter γ, ϵ are set as 0.75, 1e-4.
Variance loss weight α is set as 5.0.
4.3 Comparison to the State of the Art
The Table 1, 2, 3, 4 present the retrieval results of HiT on MSR-VTT, LSMDC, DIDEMO and MSVD.
We also compare TMVM with other state-of-the-art methods: improving the robustness of pretrainedmodel (34; 19; 1; 17; 26); designing better architecture or loss (6; 7; 8; 31; 21; 15; 28; 33; 32) onpre-extracted features; focusing on mining information of different visual modalities(10; 29; 18);endto-end trainable model (13; 2).
As demonstrated by the results, TMVM outperforms all comparison methods clearly.
We presentvideo-to-text and text-to-video retrieval results for MSR-VTT and LSMDC.
For MSR-VTT, ourretrieval performance at SumR, in particular, is 348.4, outperforming recent state-of-the-art methods(29) by a margin of 10.2.
For LSMDC, our retrieval performance at SumR also outperforms othermethods by a clear margin.
It accurately reflects TMVM’s overall retrieval quality.
We reportretrieval performance in terms of text-to-video retrieval for DIDEMO and MSVD.
TVMM continuesto outperform comparison methods.
Our method doesn’t outperform R@10 of some other methodsin Table 1, 2 .
It is possibly because the methods use multi modality information of video, such asspeech, audio and appearance.
Our method is based on Frozen and only use the frames.
4.4 Ablation Study
We discover that the proposed components, such as Multiple Visual Prototypes and Variance Loss,help to improve retrieval performance.
We exhaustively and comprehensively ablate two componentsto demonstrate their effectiveness and robustness.
[Table] [Table]
Production of Multiple Visual Prototypes.
As mentioned above, we use mask-based method toaggregate token features from the final layer.
In this section, we design two other variants to studywhether this production is good enough.
• TVMM-Part.
We divide the token features into several parts according to the frame order.
The part’s feature centers are then treated as prototypes.
• TVMM-Decoder.
We use several embeddings as queries, and the token features as memory,
then we throw query and memory into two-layer decoder to process the queries as visual
prototypes.
TVMM-Mask is our method which is introduced in Section 2.
As shown in Table 5 ,TVMM-Maskoutperforms TVMM-Part and TVMM-Decoder with a clear margin.
TVMM-Part improved littlecompared with the baseline, only 1.3 on SumR. It shows that TVMM-Part is very limited in its abilityto model multiple prototypes.
The Number of Visual Prototypes.
As shown in Table 5, multiple prototypes is better than singleprototype.
With K = 3 and its variance loss, TVMM-Mask outperforms the baseline with 5.3 on
RSum.
This reflects a video can contain rich instance and event information and the positive textonly describes a part of it.
TVMM find correspondence in the video with the help of multiple visualprototypes matching.
The choice of number of prototypes is critical.
K=3 is the best among 1,2,3,4and 1569.
When K is greater than or equal to 3, as K gets larger, the performance continues to degrade.
A appropriate number of prototypes can assist the model in learning representative prototypes.
Whenthere are too many prototypes, the learned prototypes become noise rather than representative.
The Number of Input Visual Frames.
In end-to-end training framework, the number of videoframes is a parameter that has a big impact on performance.
As shown in Table 6, as the numberof video frames increases, so does the performance.
It’s worth noting that even on single-frameinput, our method also improves the baseline with 4.0 on SumR. It proves the importance of multipleprototype a car driving down a road goes from and to video man talking about old car
[Figure]
Some Retrieval Examples.
Each test video in the original MSR-VTT split has only one textdescription.
In this experiment, we take some samples from the training data set and add them to thetest set to demonstrate the superiority of the method.
When compared to the baseline, our methodprovides correct retrieval results for the two test text queries, as shown in Figure 4.4.
It’s because
our
TVMM model can alleviate the ambiguity problem in video-text correspondence.
Visualization of Multi Prototypes.
We choose a video and its three captions from MSR-VTT.
Thecaptions are matched with the most similar prototype.
The heatmaps based on the mask values on thefour sampled frames are then plotted to visualize the prototypes.
The four frames show a serious man,a sad woman, a serious woman, and a pair conversing.
As you see, the three captions correspond wellwith the prototypes, which attend to different contents of the video.
Conclusion
In this paper, we propose a Text-Adaptive Multiple Visual Prototype Matching Model.
It generatesmultiple prototypes for video automatically to account for its rich information, and we presenta text-adaptive matching strategy to find correspondence between texts and videos.
Then, wepropose a variance loss to encourage different prototypes to pay attention to different video contents.
Experimental results on four public video retrieval datasets demonstrate the advantages of our model.
Ablation studies are also carried out to verify the effectiveness of each part of our model.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[No](b) Did you include complete g xproofs of all theoretical results?
[No]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No](d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] The requirement method is public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The data is publicly available.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[No] We didn’t use crowdsourcing and conducted research with human
[Table]
subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[No] We didn’t use crowdsourcing and conducted
[Table]
research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[No] We didn’t use crowdsourcing and conducted
research with human subjects.