Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations
Anonymous Author(s)
AffiliationAddress email
Abstract
Most video-and-language representation learning approaches employ contrastivelearning, e.g., CLIP [42], to project the video and text features into a commonlatent space according to the semantic similarities of text-video pairs.
However,such learned shared latent spaces are not often optimal, and the modality gapbetween visual and textual representation can not be fully eliminated.
In this paper,we propose Expectation-Maximization Contrastive Learning (EMCL) to learncompact video-and-language representations.
Specifically, we use the ExpectationMaximization algorithm to find a compact set of bases for the latent space, wherethe features could be concisely represented as the linear combinations of these bases.
Such feature decomposition of video-and-language representations reduces therank of the latent space, resulting in increased representing power for the semantics.
Extensive experiments on three benchmark text-video retrieval datasets prove thatour EMCL can learn more discriminative video-and-language representations thanprevious methods, and significantly outperform previous state-of-the-art methodsacross all metrics.
More encouragingly, the proposed method can be applied toboost the performance of existing approaches either as a jointly training layer oran out-of-the-box inference module with no extra training, making it easy to beincorporated into any existing methods.
Introduction
Text-video retrieval [60], which aims to fetch relevant videos using textual queries or vice versa,is an important yet challenging cross-modal task.
The dominating paradigm of text-video retrievalis contrastive learning [56, 25, 6, 24, 8, 7], which is a commonly adopted framework for videoand-language representation learning.
The core idea of contrastive learning is to pull the textualand visual representations of matched text-video pairs together and push the representations ofunmatched text-video pairs apart.
In this manner, contrastive learning enables neural networks tolearn discriminative video-and-language representations.
However, standard contrastive learning has intrinsic limitations for text-video retrieval tasks, sincethe success of contrastive learning largely depends on the volume and variety of negative samples[6].
Without adequate negative samples, it would be hard to guide the direction of sample learning,and the features could not be contrasted well.
As shown in Figure 1a, we randomly select threevideos (each video has 8 video frames and 20 text captions) from the benchmark dataset [57] andanalyze the entire feature space of a recent state-of-the-art standard contrastive learning method
[Figure]
Text
Different Classes
Hyperplane between videos and texts
[Figure]
surfboard.
[Figure]
: A
[Figure]
a
[Figure]
Text: A horse jumps inside his stable.
Text:
A guy driving a car .
Text: A horse jumps inside his stable.
Text: A man is riding a surfboard.
) shows that although the representations belonging to the same class are well-clustered, there is still a dividing line (
Hyperplane) representations language representations
Besides, the representations with different semantics (i.e., from different semantic classes) overlap with each other.
On the contrary, (b) shows that our method can learn more discriminative video-andlanguage representations.
In detail, the video-and-language representations with the same semantics (i.e., belonging to the same semantic class) are well-grouped; and there are clear dividing spaces between different semantic classes.
Here we visualize the feature space with t-SNE [33].
In an ideal feature space, the instancesof the same semantic classes should be close to each other.
However, as shown in Figure 1a, wenotice that the feature space learned by standard contrastive learning fails to preserve inter-modalsemantic relatedness, where videos and texts with the same class semantics are still far away.
In otherwords, the semantic-relevant but modal-different instances can not be grouped together in this featurespace.
To improve the performance of contrastive learning, recent image-text representation learningmethods either increase batch sizes or maintain large data memory banks [6, 8, 35, 56, 24].
Theseworks target to collect sufficient negative samples for contrastive learning.
However, such text-videoretrieval approaches are relatively expensive due to the incurred extra computational costs, especiallyon large-scale datasets [46].
So an urgent challenge here is to find an efficient method to learn asemantically relevant feature space.
To efficiently bridge the modality gap and group visual and textual representation according tosemantics, we propose to learn a low-rank and compact latent feature space.
For this purpose, wepropose a novel method named Expectation-Maximization Contrastive Learning (EMCL), whichuses the same parametric model to abstract and reconstruct both textual and visual representations.
Indetail, we find a set of reconstruction bases for subspace representation by estimating the parametersin Expectation-Maximization (EM) algorithm [13].
In the learned subspace, both video and textfeatures are represented with the distributions over the same sets of hidden variables, which canpreserve strong semantic relations across modalities.
As shown in Figure 1b, our EMCL effectivelylearns a semantically related subspace which has smaller intra-class variance and larger inter-classvariance compared to the previous contrastive learning method (Figure 1a).
We further propose
EMCL-Net based on the EMCL and apply EMCL-Net to the task of text-video retrieval.
In particular,for better adapting the proposed EMCL into the downstream task, our EMCL-Net introduces aparameter initialization strategy of the EM algorithm.
Experimental results on three text-videoretrieval benchmark datasets, i.e., MSR-VTT [57], ActivityNet [27], and LSMDC [44], show theadvantages of the proposed EMCL.
The main contributions are as follows:
• We identify the intrinsic limitation of contrastive learning for text-video retrieval, i
e.,the cross-modal representation bias could not be fully eliminated via standard contrastive
learning approaches.
• To alleviate such limitation, we reformulate the contrastive learning for video-and-languagerepresentations into an expectation-maximization iteration manner and propose a plug-andplay feature projection module named Expectation-Maximization Contrastive Learning(EMCL), which learns the subspace that aims to become semantic-relevant representation.
• Based on our EMCL, we further propose the EMCL-Net, which introduces a parameterinitialization strategy for the EM algorithm.
Experiments show that our approach achievesstate-of-the-art results on three text-video retrieval datasets.
More encouragingly, our methodcan be easily applied to boost the performances of existing approaches either as a jointlytraining layer or an out-of-the-box inference module with no extra training.
Related Work
Cross-modal learning is widely studied in many areas, including cross-modal retrieval [60, 56, 25],transfer learning [40, 37], and domain adaptation [45, 29], in which different modalities/domainscome from different distributions.
The main challenge of cross-modal learning is to use givenvision-and-language pairs to learn common representations shared between modalities.
Most existingworks of text-video retrieval [6, 24, 8, 7, 21] map text and video to the same latent space, where thesimilarity between them can be directly calculated [15, 20, 18, 4, 51, 14, 53, 54, 5, 12, 58, 55, 41, 23].
Contrastive learning methods [22, 17, 56, 2, 24, 35, 7] try to learn data representations from positiveand negative pairs, making the representations of positive pairs (usually data augmentations thatretain semantic information) have high similarity, and negative pairs (different semantic examples)have low similarity.
Inspired by the great success of contrastive learning, we employ CLIP [42] forlearning video-and-language representations.
In this paper, a positive pair is a matching text-videopair, and a negative pair contains non-matching text and video.
However, due to the multi-modalnature and spatial-temporal evolution of video [9, 30, 36], it is essential to capture the most importantsemantic concept for the video.
Meanwhile, the visualization in Figure 1 also shows that the featurespace of video-and-language representations has semantically irrelevant redundancy which leads tonon-ideal contrast in the feature space.
Therefore, we propose to conduct contrastive learning inan expectation-maximization iteration manner to eliminate the non-optimal semantically redundantdimensions, acquiring compact representations.
Approach
In this section, we first introduce how to reformulate the Contrastive Learning into an expectationmaximization iteration manner, i.e., Expectation-Maximization Contrastive Learning (EMCL).
Then,we introduce how to incorporate the proposed EMCL into the neural network for video-and-languagerepresentations, which are used to perform the text-video retrieval task.
3.1 PreliminariesExpectation-Maximization Algorithm
The expectation-maximization (EM) algorithm [13] isan iterative optimization strategy, which was originally designed to solve the problem whendata are missing in the process of parameter estimation.
Briefly, given unobserved hidden variables Z = { z 1 , z 2 , ..., z N } and observed data sets X = { x 1 , x 2 , ..., x N } with N samples,the goal of EM algorithm is to estimate the maximum likelihood solution of model parametersˆ θ = arg max (cid:80) Ni =1 log (cid:80)
z
i p ( x i , z i ; θ ) .
In step E, the EM algorithm calculates the conditionalprobability expectation Q i ( z i )
=
p ( z i | x i , θ ) .
In the M step, the likelihood function is maximized toget the revised parameter ˆ θ = arg max (cid:80) Ni
=1 (cid:80) z
i Q i ( z i ) log p ( x i ,z
i ; θ ) Q i ( z i ) .
Gaussian Mixture Model
The Gaussian Mixture Model (GMM) [43] combines multiple single Gaussian models.
Assuming that GMM consists of K Gaussians, given the input X
={ x 1 , x 2 , ..., x N } with hidden variables Y , the probability of GMM is as follows:
[Equation]
where N ( ∗| µ k , Σ k ) is the probability function of the k th Gaussian and π k is the prior probability.
Through the E step of the EM algorithm, the estimated value of y n,k can be obtained by:
[Equation]
[Figure]
M Initialize
Output we show the case when K = 3 .
The input features come from 4 samples, which are colored differently
EMCL projects
In the M step of the EM algorithm, the parameters of the Gaussian mixture model are updated as:
[Equation]
3.2 Expectation-Maximization Contrastive Learning (EMCL)
In this section, we introduce our EMCL in detail.
Representing the features in low-dimensional spaceis a fundamental method to eliminate the inherent redundancies in features, which is also the goal ofour method.
We leverage these compact features for contrastive learning to efficiently bridge the gapand improve their performance.
The overview of our EMCL is shown in Figure 2 and Algorithm 1.
Mathematically, the subspace can be defined as follows: let
X ∈ R B
× D be the features, where B isthe sample size, D is the dimension of the feature.
Suppose that these features are distributed in manyunknown linear subspaces, where the intrinsic dimensions of the linear subspaces are less than D .
Wedefine the union of K semantically related low-dimensional linear subspaces as the semantic subspace.
To this end, our method includes two parts: Firstly, we use the EM algorithm [13] to find the bases ofthe K optimal subspaces, which is called “ Maximum Probability Projection ”; Secondly, we needto re-represent the features in these subspaces, which is called “ Feature Reconstruction ”; Finally, inthe “ Contrastive Learning ”, the compact features we re-represent are used in contrastive learning tobridge the gap and improve their performance.
Maximum Probability Projection
Given original video features C v ∈ R B × D and originaltext features C t ∈ R B × D , we combine both video features and text features to be the input data
X ∈ R 2 B × D , where x i,j is the j th coding bit of the i th sample.
In our method, each coding bit x i,jis assigned to a specific subspace, and meanwhile we introduce the hidden variable Y , where y j,krepresents whether the k th subspace is selected by x i,j .
We take the base λ of the subspace as theparameter of EM algorithm [13], where λ ∈ R 2 B × K represents the distribution of samples in Ksemantically related low-dimensional linear subspaces.
Given input data X ∈ R 2 B × D and hiddenvariable Y ∈ R D
×
K , the complete likelihood of the input feature X can be expressed as:
[Equation]
where P ( ∗ ) is the probability function and p ( ∗ ) is a kernel function.
Here we assume that x i,j can berepresented by K shared Gaussian distributions like GMM [43].
For simplicity, we freeze Σ and π ,and only consider the mean λ = µ of the Gaussian model.
There are many choices for the kernel function p (ˆ x, ˆ y ) , such as linear kernel (ˆ x T
ˆ y + c ) , polynomialkernel ( a ˆ x T
ˆ y + c ) d , Gaussian kernel exp ( − ˆ γ ∥
ˆ x − ˆ y ∥ 2 ) and so on.
We find that different kernelfunctions have slightly different impacts on the final results.
For easy implementation, we use
Gaussian kernel and rewrite it in a form similar to the attention model [3, 49].
The formula is:
[Equation]
where σ is a hyper-parameter to adjust the distribution, similar to the mean and covariance in the
Gaussian distribution.
In this paper, we define x : ,j to be the j th column of X , and λ : ,k to be the k thcolumn of λ .
Reviewing the EM algorithm [13], it estimates the parameters of the model by continuously executing
E steps and M steps.
In step E, it calculates the conditional probability expectation with currentparameters.
In step M, it maximizes the likelihood function to update the parameters.
In step E, referring to the Eq.
(2), the estimated value of y j,k can be obtained by:
[Equation]
In the M step, we update base λ i,k
according to Eq.
(3), which is formulated as:
[Equation]
By repeatedly iterating step E and step M, our algorithm forces this set of optimal subspaces torepresent the original video features and original text features at the same time.
After several iterations,we could keep the information that appears in both the video and the text to remove the redundancy.
Feature Reconstruction
When reconstructing features, we use Y and λ to linearly reconstruct theoriginal features.
Learning from the meaning of Y and λ mentioned above, y j,k indicates whetherthe k th subspace is selected by x : ,j and λ : ,k represents the base of the k th subspace.
The formula forfeature reconstruction is:
[Equation]
As shown in Eq.
(8), we only use the corresponding base
λ i,k (for all k ∈ [1 , K ] ) to reconstruct thei th sample x i,j .
Therefore, we estimate different x i,j from K subspaces.
Contrastive Learning
The above feature reconstruction method generates the low-rank andcompact features stripped of redundancy.
Therefore, we can leverage these compact features forcontrastive learning to efficiently bridge the modality gap and improve task performance.
3.3 EMCL for Cross-Modal Learning
In this section, we discuss in detail how to incorporate the EMCL into the neural network for crossmodal downstream tasks.
Targeting on the cross-modal tasks, we propose the ECML-Net, whichintroduces a parameter initialization strategy for the EM algorithm that can establish the connectionbetween batches.
At last, we take an important yet challenging cross-modal task, i.e., the text-videoretrieval, as an example to illustrate how to employ our EMCL-Net to perform downstream tasks.
Initial Value Maintenance
Considering that a corpus of video-based cross-modal tasks usuallycontains thousands of text-video pairs, it is often not enough to only establish connections betweensamples within single batches.
Therefore, our module builds an inter-batch information transfermechanism by maintaining a separate initial value M ∈ R K .
In implementations, each time a newbatch of features is fed into the model, the EM algorithm needs to initialize the λ .
Rather thanrandomly initializing λ for each batch, we use M as the initial value of the λ .
For all i, k , we initialize λ with λ
i,k
= m k .
We update M using an average moving method similar to the Batch
Normalization (BN) layer [26].
The moving method is formulated as:
[Equation]
where α ∈ [0 , 1] is the momentum.
Similar to the average moving method in the BN layer, wedon’t update M in the inference stage.
Because the initial value is crucial to the stability of the EMalgorithm [1, 28], we need to limit the value range of the initial value.
In each iteration, we use the
L2 normalization operation to limit the value range of λ .
EMCL-Net
For fair comparisons, we follow common practice [32, 10, 50] to extract the videorepresentations of input videos and the language representations of input texts.
In detail, for videorepresentations, we first extract the frames from the video clip as the input sequence of video; Thenwe use ViT [16] to encode the frame sequence, by exploiting the transformer architecture to modelthe interactions between image patches; Followed by the CLIP [42], the output from the [class] tokenis used as the frame embedding; Finally, we aggregate the embedding of all frames and obtain thevideo representation C v ∈ R B × D .
For text representation, we directly use the text encoder of CLIPto acquire the text representation C t ∈ R B × D .
As a plug-and-play module, our proposed EMCL is inserted at the end of the video-text encoders.
We concatenate video features C v and text features C t , generating the input data of our method
X =
[ C v ; C t ] ∈ R 2 B × D .
Following [11], we add reconstructed features f EMCL ( X ) by the EMCLand original features X to obtain final text-video representations
ˆ X =
[ ˆ C v ; ˆ C t ]
= βf EMCL ( X ) +
X ∈ R 2 B × D where β is the scale factor.
Here, β is the scale factor which is used to balance theoriginal and new embeddings.
By adjusting β , we can add some flexibility in the reconstructedvideo-and-language representations for the robustness of the model.
Training Objective
During the training stage, the EMCL module is trained with the neural network.
We use M to initialize the λ .
Then the components Y and λ are updated in unsupervised wayby iteration, as shown in Algorithm 1.
Moreover, we update the initial value M using an averagemoving method.
The goal of the EMCL-Net is to map text and video into a joint representationspace to measure the text-video similarity.
Following common practice, we use cosine similaritys ( t, v )
=
ˆ c
Tt ˆ c v ∥
ˆ c t
∥∥ ˆ c v ∥ as the similarity measure between text t and video v , where ˆ c t represents thereconstructed text representation of text t ; and ˆ c v represents the reconstructed video representation ofvideo v .
We train our model with InfoNCE loss [48]:
[Equation]
where B is the batch size and τ is the temperature hyper-parameter.
This loss function maximizes thesimilarity of positive pairs s ( t i , v i ) and minimizes the similarity of negative pairs.
During the inference stage, given a set of queries (text/video) and a set of candidates (videos/texts),we use the trained M to initialize the λ .
Then Y and λ can be updated in an unsupervised wayby iteration.
The goal of cross-modal retrieval task is to map the query and candidate into a jointvideo-and-language representation space to measure the query-candidate similarity s ( t, v ) .
In thisway, we can output the similarity scores for all the input candidates, and take the candidates with the
Top-1/5/10/50 similarity scores with input query as the final prediction.
Experiments
Datasets, Metrics and Implementation Details 2 Datasets.
We conduct the experiments on threepopular text-video retrieval datasets, i.e., MSR-VTT [57], ActivityNet Captions [27], LSMDC [44],and follow common practice [32, 10, 50] to pre-process the datasets for fair comparison.
[Table] [Table]
(+Number) denotes the absolute improvements.
[Table]
Metrics.
We choose standard retrieval metrics: Recall at K (R@K) and Median Rank (MdR) toevaluate the text-to-video and video-to-text retrieval performance.
Implementation Details.
We utilize the CLIP (ViT-B/32) [42] equipped with Temporal Transformer[32] as pre-trained Bi-Encoder (Base Model).
We set the temperature τ
= 100 , σ = 1 , β = 3 , themomentum α = 0 .
9 , the number of iterations is set to 9 and the parameter K is set to 32.
Comparisons to State-of-the-art
Table 1 shows the results of our method on three text-videoretrieval datasets.
As we can see, our ECML-Net method consistently outperforms the recentlyproposed state-of-the-art methods on both text-to-video retrieval and video-to-text retrieval tasksacross all datasets and metrics.
Generalization Analysis
Since the input and output dimensions of the EMCL module are thesame, it is model-agnostic and can be applied to features extracted from any language and videoencoders.
Therefore, we further equip our EMCL with three strong baseline models, i.e., MMT[19], CLIP4Clip [32], DCR [50], and evaluate the performance of these models on the MSR-VTTdatasets.
Specifically, we insert the EMCL module at the end of the video-text encoders.
Table 5shows that our EMCL can be applied to successfully boost all baselines either as a jointly traininglayer (Table 5(a)) or an out-of-the-box inference module with no extra training (Table 5(b)).
Thesignificant improvements demonstrate the generalization ability of EMCL.
We perform the analysis on the Text->Video task.
B is the sample size.
D is the dimension of the original feature.
K is the number of subspaces.
[Table]
[Table] [Figure]
[Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Comparisons to Other Dimensionalty Reduction Methods
We further compare our method withother representative dimensionality reduction methods in Table 3.
PCA [47] is a popular methodfor finding salient features shared by the two modalities so that the modality gap can be reducedto a certain extent.
“Fully Connected Layers” represents using two shared fully connected layersbetween two modalities, which can explicitly project the data from two modalities into a shared space.
“Sparse Autoencoders” is a common sparse autoencoder [38], which reduces the average responseof the encoding layer and learns the compact representation.
Different from these representativedimensionality reduction methods, the motivation of our method is to use contrastive learning topreserve video-text semantic relatedness.
By maximizing the joint posterior probability of video andtext, we find a semantically related subspace for compact video-text representation.
In contrast, PCAand its variants are feature dimension reduction methods, which aim to maximize the variance ofprojected data and cannot guarantee semantic relationships.
Sparse autoencoder reduces the averageresponse of the encoding layer for sparsity, which may potentially discard semantic information.
Ablative Analysis
Effect of the parameter initialization strategy.
As shown in Table 4, with orwithout parameter initialization strategy, our method can successfully promote the base model.
Itis worth noting that the EM algorithm [13] is sensitive to initial values [1, 28].
In other words, theconvergence of EM algorithm depends mainly on initial parameters.
Random initialization leadsto large fluctuations in convergence results.
This fatal flaw limits the performance of our module.
Fortunately, with the proposed parameter initialization strategy, our EMCL method surpasses the basemodel by a large margin with 9.2% R@1 and 8.6% R@1 on the text-to-video and video-to-text tasks,respectively, proving the effectiveness of our parameter initialization strategy used for EMCL-Net.
Effect of the number of subspaces.
In Figure 3a, we show the effect of the number of subspaces
K .
On the one hand, we find that fewer subspaces mean fewer semantic centers, which limits ourmodule’s ability to reconstruct the features.
On the other hand, a larger number of subspaces requiresmore training data, which increases the cost of our model learning.
We set the center size K = 32 toachieve the best performance for all datasets in practice.
Effect of the iteration number.
In Figure 3b, we show the influence of EM iteration number T .
Overall performance improves slightly before leveling off.
We find that the algorithm has convergedwhen the number of iterations is 9, so we set the number of iterations to 9 as default for all datasets.
Hyper-parameter selection.
The parameter σ are hyper-parameters to adjust the distribution (Eq.
5),similar to the mean and covariance in the Gaussian distribution.
We evaluate the scale range setting
Classes
[Figure]
i n t r a - c l a ss v a r i a n ce
The iteration number
[Figure]
The iteration number T i n t e r-
c l a ss v a r i a n ce
K= K=
K=
K=128 K=
X
[Figure]
view
X-Y-Z view
X-Y-Z view X-Y-Z view
X-Y-Z view X-Y-Z view
X-Y view X-Y view
X-Y view X-Y view
X-Y view X-Y view
Y-Z view Y-Z view
Y-Z view Y-Z view
Y-Z view
Y-Z view σ ∈ [0 . 3 , 2 . 0] as shown in Figure 3c.
We find that R@1 is improved from 43.8% to 51.6% whenσ = 0 .
7 and saturated with σ = 1 .
As a result, we adopt σ = 1 to achieve the best performance.
Qualitative Analysis Analysis of EMCL iterative process.
As shown in Figure 4a, reducingthe intra-class variance makes videos and texts belonging to the same semantic class gather, andincreasing the inter-class variance makes those belonging to different semantic classes separate fromeach other.
The two conclusions we get from Figure 4b and Figure 4c are as follows.
(1) EMCLmodule reduces the intra-class variance of the same semantic classes and increases the inter-classvariance of different semantic classes.
The number K of subspaces has a great influence on the
EMCL module.
The effect of the EMCL module is limited if K is too small, while the intra-classvariance increases if K is too large.
Visualization.
To better understand EMCL, we provide the visualization form of both the originalrepresentations and the filtered representations.
As shown in Figure 5, even though features haveintense noise in the redundancy dimension, the EMCL module still learns semantic information.
The
EMCL module forces the differences between classes in subspace to be more obvious than original,which is helpful for the contrastive learning to learn the semantic centers shared by videos and texts.
Conclusion
In this paper, we studied the intrinsic limitation of classic contrastive learning for text-video retrieval.
We found that the contrastive method in the entire representation space fails to preserve inter-modalsemantic relatedness, which makes the features gather or separate in the subspace which is irrelevantto semantics.
To mitigate this effect, we propose to directly learn the subspace that is related to sharedsemantics and do contrastive learning in it.
By learning the subspaces related to semantics, we areable to learn the common semantic center of video and text in the semantic subspace.
Further, it isworth noting that our method could be applied for other contrastive learning tasks, which includesimilar samples containing redundant dimensions or with a limited number of negative samples.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 4 and
Appendix A.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix A.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.
and Section 3.2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3.1 andSection 3.2.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section B and Appendix B.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Table 5.
We conducted 5 runs with different seedsfor all experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Additional Discussions
Potential negative societal impacts
Although our work improves the performance of text-videoretrieval, but may reduce the difficulty of cross-modal retrieval of sensitive information on the network.
It may raise challenges to protecting information security.
Limitations of our work
Iterative approaches are sensitive to initialization and parameters such asthe dimensions and the number of subspaces.
In our work, although we use the L2 normalization operation to limit the value range of the parameters, the EM algorithm [13] may still converge to badresults.
At the same time, the selection of the number of subspaces also has a relatively significantimpact on the model effect.
In the future, it is still a promising research direction to explore morestable subspace search algorithms.
B Datasets and Implementation DetailsDatasets
MSR-VTT [57] contains 10,000 YouTube videos, each with 20 text descriptions.
Wefollow the training protocol in [31, 19, 34] and evaluate on text-to-video and video-to-text searchtasks on the 1k-A testing split with 1,000 video or text candidates defined by [59].
ActivityNet Captions [27] dataset consists densely annotated temporal segments of 20K YouTubevideos.
Following [19, 39, 52], we concatenate descriptions of segments in a video to construct“video-paragraph” for retrieval.
We use the 10K training split to train or finetune the model and reportthe performance on the 5K “val1” split.
LSMDC [44] contains 118081 video clips from 202 movies.
We follow the split of [19] with 1000videos for testing.
Implementation Details
The EMCL module is trained with the neural network.
During the trainingstage, the EMCL module is trained with the neural network.
Each time a new batch of features is fedinto the model, we use M to initialize the λ .
Then the components Y and λ are updated by iteration.
Moreover, we update the initial value M using an average moving method.
During the inference stage, given a set of queries (text/video) and a set of candidates (videos/texts),we use the trained M to initialize the λ .
Then the components Y and λ can be updated by iteration.
When the EMCL module is incorporated into trained baselines as an out-of-the-box inference modulewith no extra training, λ is randomly initialized.
Then the components Y and λ can be updated in anunsupervised way by iteration.
We use 2 V100 GPUs for MSR-VTT and LSMDC.
For ActivityNet, a long video retrieval dataset, weuse 8 V100 GPUs.
We utilize the CLIP (ViT-B/32)
[42] as pre-trained Bi-Encoder.
The temporaltransformer is composed of 4-layer blocks, each including 8 heads and 512 hidden channels, and isinitialized from the CLIP’s text encoder.
Following CLIP4Clip [32], the frame length and captionlength are 12 and 32 for MSR-VTT and LSMDC.
For ActivityNet, a long video retrieval dataset, weset the frame length and caption length to 64 and 64.
We follow training schedules from CLIP4Clip[32].
Concretely, we use the Adam optimizer with a linear warmup.
The initial learning rate is 1e-7for text encoder and video encoder and 1e-4 for other modules.
We set the temperature τ
= 100 ,σ = 1 , β = 3 , the momentum α = 0 .
9 , the number of iterations is set to 9 and the parameter K is setto 32.
The network is optimized with the batch size of 128 in 5 epochs.
The experiment setup in “Comparisons to other Dimensionality Reduction Methods”
We chosethree baselines, e.g., PCA, “Fully Connected Layers” and “Sparse Autoencoders”.
For all methods,we concatenate video features C v and text features C t , generating the input data X =
[ C v ; C t ] ∈ 540 R 2 B × D .
Finally, we add reconstructed features f ( X ) and original features X to obtain final textvideo representations
ˆ X =
[ ˆ C v ; ˆ C t ] =
f ( X ) + X ∈ R 2 B × D .
All networks are optimized withthe batch size of 128 in 5 epochs.
In “PCA”, we adopt PCA at the end of the video-text encoders.
We use PCA to reduce the dimensions of the original features from 512 to 32, then restore themto 512 dimensions.
In “Fully Connected Layers”, we pass the original features through a commonfeed-forward network where the inner-layer has a dimensionality of 256, and the activation function is
Relu.
In “Sparse Autoencoders”, we adopt Sparse Autoencoders at the end of the video-text encoders.
The inner-layer has a dimensionality of 256, and the activation function is Sigmoid.
We reduce theaverage response of the encoding layer to ρ = 0 . 05 .
[Table] [Table]
[Table] [Table]
C Additional ExperimentsAdditional Generalization Analysis
To further verify the generalization of our method, we testit on other datasets such as ActivityNet [27] and LSMDC [44].
As a plug-and-play module, ourapproach EMCL can be easily integrated into existing contrastive learning methods.
Therefore, wefurther equip our EMCL with two strong baseline models, i.e., MMT [19] and CLIP4Clip [32], andevaluate the performance of these models on the ActivityNet and LSMDC datasets.
Table 5 showsthat our EMCL can be applied to successfully boost all baselines either as a jointly training layer oran out-of-the-box inference module with no extra training.
For ActivityNet, our approach can boostthe baselines with the improvement up to 3.0% and 5.2% for text-to-video task and video-to-texttask in terms of R@1 metric, respectively.
For LSMDC, our approach can boost the baselines withthe improvement up to 4.1% and 3.4% for text-to-video task and video-to-text task in terms of R@1metric, respectively.
The improvements demonstrate the generalization ability of EMCL.
Effect of the scale factor β
The parameter β is the scale factor.
By adjusting β , we can add someflexibility in the reconstructed video-and-language representations.
We evaluate the scale rangesetting β ∈ [0 . 0 , 4 . 0] as shown in Table 6.
We find that R@1 is improved from 45.1% to 51.6% whenβ = 2 . 0 and saturated with β = 3 . 0 .
As a result, we adopt β = 3 . 0 to achieve the best performance.
Generalize to other contrastive learning tasks
To further verify the generalization of our method,we adopt the EMCL to other contrastive learning tasks, such as Self-supervised Visual representationand Few-shot Classification.
Table 7 shows that our EMCL can be applied to boost other contrastive
[Table]
[Figure]
[Table]
[Figure]
[Table]
learning tasks successfully.
For the Self-supervised Visual representation, our approach can boostthe baseline with an improvement up to 0.16%.
For the few-shot classification, our approach with
EMCL outperforms the baseline by 0.2%.
It shows that EMCL can further improve the performanceof contrastive learning, especially for the tasks with similar samples containing redundant dimensionsand unable to maintain a large number of negative samples.
D Additional Qualitative Analysis
Visualization of semantic similarity in semantic subspace
In EMCL, we project video and textfeatures into semantic subspaces.
We hope that videos and texts with similar semantics can share acommon semantic center.
In Figure 6, we associate all video frames with words in the text.
As shownin Figure 6, all video frames assigned to “skateboard" contain content related to “skateboard".
Aninteresting observation is that in addition to the semantic information, quantitative information suchas “person" and “others" are also distinguished in semantic subspace.
For example, all video framesassigned to “person" contain only one person, while video frames assigned to “others" contain morethan one person.
All the semantic similarities in Figure 6 are low because limited training data arenot sufficient to understand low-frequency words.
This experiment shows that the EMCL can helpthe model learn adequate semantic information in video-text retrieval.
Visualization of the text-to-video retrieval
We show two examples of the videos retrieved by the basemodel and our EMCL-Net.
As shown in Figure 7, our EMCL-Net successfully retrieves the groundtruth video while the model without the EMCL module returns several videos that are unrelated tothe query sentence.
Our EMCL-Net retrieves the correct videos with higher confidence than themethod without the EMCL module.
These results demonstrate that our EMCL module can effectivelyimprove the retrieval performance of the model.
Query 7060 ： a guy extinguishes something then talks into the camera with another guy.
[Figure]
Query 7894 ： a woman getting interviewed by a man.
[Figure]
report