S UPPORT - SET BOTTLENECKS FOR VIDEO - TEXT REPRESENTATION LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The dominant paradigm for learning video-text representations – noise contrastive learning – increases the similarity of the representations of pairs of samples that are known to be related, such as text and video from the same sample, and pushes away the representations of all other pairs.
We posit that this last behaviour is too strict, enforcing dissimilar representations even for samples that are semanticallyrelated – for example, visually similar videos or ones that share the same depicted action.
In this paper, we propose a novel method that alleviates this by leveraging a generative model to naturally push these related samples together: each sample’s caption must be reconstructed as a weighted combination of other support samples’ visual representations.
This simple idea ensures that representations are not overly-specialized to individual samples, are reusable across the dataset, and results in representations that explicitly encode semantics shared between samples, unlike noise contrastive learning.
Our proposed method outperforms others by a large margin on MSR-VTT, VATEX, ActivityNet, and MSVD for video-to-text and text-to-video retrieval.
I NTRODUCTION
Noise contrastive learning (Gutmann & Hyvärinen, 2010) is emerging as one of the best approaches to learn data representations both for supervised (Khosla et al., 2020) and unsupervised regimes (Chen et al., 2020c).
The idea is to learn a representation that discriminates any two data samples while being invariant to certain data transformations.
For example, one might learn a representation that identiﬁes a speciﬁc image up to arbitrary rotations (Misra & van der Maaten, 2020).
In a multi-modal setting, the transformations can separate different modalities, for example, by extracting the audio and visual signals from a video.
The resulting noise contrastive representation associates audio and visual signals that come from the same source video, differentiating others (Patrick et al., 2020).
The noise contrastive approach is motivated by the fact that the transformations that are applied to the data samples leave their ‘meaning’ unchanged.
For example, rotating an image does not change the fact that it contains a cat or not (Gidaris et al., 2018).
However, in most cases, we expect to ﬁnd many data samples that share the same content without being necessarily related by simple transformations (e.g. think of any two images of cats).
Existing noise contrastive formulations are unaware of these relationships and still try to assign different representations to these samples (Wu et al., 2018), despite the fact that they are semantically equivalent.
If the representation is learned for a downstream task such as semantic video retrieval, this might degrade performance.
This suggest that there might be other learning signals that could complement and improve pure contrastive formulations.
In this paper, we explore this idea in the case of learning from two modalities: videos and text, in the form of video transcripts or captions.
Given a state-of-the-art contrastive formulation that learns from these two modalities, we investigate complementary pretext objectives to improve it.
First, we consider the (instance) captioning task, namely mapping a video to the corresponding text, casting this as a conditional stochastic text generation problem.
We show that this brings only a modest beneﬁt.
We observe that the captioning task is highly sample-speciﬁc, as the goal is to produce a caption which describes a speciﬁc video and not any other video, and thus it suffers from the same disadvantages (discouraging concept sharing among samples) as contrastive learning.
Thus, we propose
[Figure]
to address this issue by switching to a different text generation task.
The idea is to modify the text generator to take as input a learnable mixture of a support-set of videos, which we call cross-instance captioning .
The mixture weights are generated by comparing the learned video representations to captions’ representations in an online way over the batch.
The limited set of support samples acts as a bottleneck that encourages extraction of shared semantics.
In this manner, the embeddings can associate videos that share similar captions even if the contrastive loss tries to push them apart.
We show that, when the captioning task is added in this manner, it brings a sensible improvement to already very strong video representation learning results, further improving our own state-of-the-art baseline by a signiﬁcant margin.
R ELATED W ORKS
Learning data representations from unlabelled data has been a long standing goal of machine learning.
These approaches are called “self-supervised learning” because the learning signals, termed pretext tasks, are obtained from the data itself.
In the image and video domain, pretext tasks include colorization (Zhang et al., 2016), rotation (Gidaris et al., 2018), or clustering (Asano et al., 2020a;b;
Caron et al., 2018; Ji et al., 2018), while in the natural language domain, masked language modeling (Devlin et al., 2019), and next word prediction (Mikolov et al., 2013; Pennington et al., 2014) are extremely popular.
These pretext tasks can be broadly classiﬁed into two classes: generative and discriminative.
Discriminative approaches learns representations by differentiating input samples, using objectives such as the contrastive loss (Gutmann & Hyvärinen, 2010; Hadsell et al., 2006).
Discriminative approaches have proven to be particularly successful for image (Chen et al., 2020c;
He et al., 2020;
Misra & van der Maaten, 2020; Wu et al., 2018) and video (Han et al., 2019; Morgado et al., 2020;
Patrick et al., 2020) representation learning.
Generative approaches, on the other hand, try to reconstruct its input.
GANs (Donahue & Simonyan, 2019; Goodfellow et al., 2014; Radford et al., 2015), autoencoders (Hinton & Salakhutdinov, 2006) and sequence-to-sequence models (Sutskever et al., 2014) are popular generative models.
In this work, we show the importance of combining both discriminative and generative objectives to learn effective video-text representations.
The success of representation learning has also been due to advances in model architectures, such as the Transformer (Vaswani et al., 2017).
BERT (Devlin et al., 2019) demonstrated that a transformer architecture pretrained on large-scale textual data can learn transferable text representations that can be ﬁne-tuned on a variety of downstream tasks.
Subsequent works (Clark et al., 2020; Lewis et al., 2020a;b; Radford et al., 2019; Raffel et al., 2019), have improved upon the transformer architecture or training objective to learn even better representations.
Inspired by the success of transformers in the NLP domain, several works have leveraged transformers to learn transferable image (Chen et al., 2020a; Desai & Johnson, 2020;
Sariyildiz et al., 2020) or multi-modal image-text representations (Chen et al., 2019; Li et al., 2020a; 2019;
Lu et al., 2019; Su et al., 2019; Tan & Bansal, 2019).
In this work, we leverage the transformer architecture to better encode and represent text and video.
[Figure]
Large-scale training data has enabled the more effective pretraining of image (Sun et al., 2017;
Yalniz et al., 2019), video (Ghadiyaram et al., 2019; Thomee et al., 2016) and textual representations (Raffel et al., 2019).
The release of the HowTo100M dataset (Miech et al., 2019), a large-scale instructional video dataset, has spurred signiﬁcant interest in leveraging large-scale pretraining to improve video-text representations for tasks such as video question-answering (Lei et al., 2018), text-video retrieval (Liu et al., 2019) and video captioning (Zhou et al., 2018b) on smaller datasets such as YouCookII (Zhou et al., 2018a), MSVD (Venugopalan et al., 2015a), MSR-VTT (Xu et al., 2016), LSMDC (Rohrbach et al., 2017), DiDeMo (Hendricks et al., 2018) and ActivityNet (Krishna et al., 2017).
Although semantically rich and diverse, instructional videos from the web are super noisy and therefore a few approaches have been proposed to combat this.
A few works (Luo et al., 2020; Sun et al., 2019a;b; Zhu & Yang, 2020) extend the BERT model to accept both visual and textual tokens to learn high-level semantic video-text representations.
Other works have leveraged the contrastive loss (Miech et al., 2020) and show that using the raw audio (Alayrac et al., 2020;
Rouditchenko et al., 2020) and other modalities (Gabeur et al., 2020) can be used to better align and improve video-text representations.
While all these approaches rely on a contrastive objective, VidTranslate (Korbar et al., 2020) shows that a generative objective can also be used to learn joint video-text representations.
In contrast to Korbar et al.
, we show that combining contrastive and generative objectives to pre-train video-text representations on large-scale data such as
HowTo100M is very effective.
The generative objective serves as regularizer to mitigate the strictness of the instance discrimination task of the constrastive objective, showing beneﬁts similar to approaches such as clustering (Caron et al., 2020;
Li et al., 2020b) and feature mixing (Kalantidis et al., 2020) which have been applied in the image domain.
M ETHOD
We consider the problem of learning multimodal representations from a corpus C of video-text pairs ( v, t ) , where v is a video and t is its corresponding text (caption or transcription).
Our goal is to learn a pair of representation maps c v = Ψ( v ) and c t = Φ( t ) , with outputs in a d -dimensional embedding space c v , c t ∈ R d , where semantically similar instances are close to each other.
3. O BJECTIVE FOR L EARNING M ULTIMODAL R EPRESENTATIONS
We consider two learning objectives, also illustrated in ing them apart otherwise.
This assumes that every sample is its own class and does not beneﬁt from modelling similiarities across instances.
The second objective is generative captioning.
In its most basic variant, it maximizes the probability of generating the text t given the corresponding video v .
However, we suggest that variants that explicitly promote concept sharing between instances will result in better downstream performance, in tasks such as video retrieval.
These variants, illustrated in Figure 2, have in common that the caption t is reconstructed from a learned weighted combination over other videos ˆ v .
This is a form of attention (Bahdanau et al., 2014) which encourages the network to learn about which videos share similar semantics, compensating for the contrastive loss and grouping them implicitly.
In the following, we denote with B ⊂ C a batch of multi-modal samples, i.e. a ﬁnite collection of video-text pairs ( t, v ) ∈ C .
For simplicity, we denote the batch as B = { ( t i , v i ) }
Bi =1 } .
3.1. C ONTRASTIVE OBJECTIVE
To deﬁne the contrastive objective, let s ( a, b ) =
a
[Equation]
b
[Equation]
be the similarity measure between vectors a and b .
Following Faghri et al.
(2018), we adopt the hinge-based triplet ranking loss with hard negative mining:
[Equation]
where α is the correlation margin between positive and negative pairs and [ · ] + = max { 0 , ·} is the hinge function.
In our experiments, we set α = 0 .
3.1. C ROSS - CAPTIONING OBJECTIVES
In the conventional captioning, the decoder seeks to optimize the negative log-likelihood of a text sequence t given its corresponding video v :
[Equation]
Here, the log-likelihood is obtained via auto-regressive decoding (Vaswani et al., 2017) from an intermediate video embedding e iv = Φ (cid:48) ( v i ) .
For the cross-captioning objective, we modify this loss to condition the generation process on a weighted average of the embeddings of the other videos in the batch, which we call the support-set .
The weights themselves, which can be interpreted as a batch-wise attention, are obtained as a softmax distribution with temperature T over batch indices based on the video embeddings, as follows:
[Equation]
By default, the summation in the softmax is conducted over a support set S i containing all indices except i .
In the experiments, we consider the following attention types for reconstruction.
Identity captioning ( S i =
{ i } ) generates the caption from the corresponding video and reduces to the standard captioning objective, eq.
Full support ( S i = { 1 , . . .
, B } ) considers all videos as possible candidates for captioning.
Hybrid captioning sets the weights in eq.
(3) as the average of the weights for identity captioning and full support.
Cross-captioning ( S i = { j (cid:54) = i } ) considers all but the video that one wishes to caption.
This variant forces the network to extract all information required for captioning from other videos in the batch.
Figure 2 compares graphically these attention mechanisms.
Considering both discriminative and generative objectives for learning multimodal representations, our full objective is L =
L contrast + λ L cross-captioning , where λ balances two objectives.
We set λ
=to ensure similar magnitudes for both losses in our experiments.
In the training phase, we use
Adam (Kingma & Ba, 2015) to minimize our loss.
At inference time, we directly use Φ( t ) and Ψ( v ) to encode video and text representations for retrieval.
3. M ODEL A RCHITECTURE
We now discuss the details of the encoders and decoder components in our architecture, illustrated in ﬁg. 2.
For the text decoder p ( t | e v ) in eq.
(2) and (3), we use a pre-trained T-5 decoder (Raffel et al., 2019).
For the video representation c v = Ψ( v ) =
Ψ (cid:48)(cid:48)
(Ψ (cid:48) ( v )) , we use a video encoder e v = Ψ (cid:48) ( v ) followed by a multi-layer transformer pooling head c v = Ψ
(cid:48)(cid:48)
( e v ) .
The encoder Ψ (cid:48) ( v ) concatenates the output of pretrained ResNet-152 (He et al., 2016) and R(2+1)D-34 (Tran et al., 2018) networks applied to individual video frames, resulting in a code e v =
[ e v 1 · · · e vM ] where M is the maximum duration of a video clip.
For the pooling head c v = Ψ (cid:48)(cid:48) ( e v ) , we consider a transformer architecture to attend to important context and summarize it into a ﬁxed-length representation c v .
For this, we follow MMT (Gabeur et al., 2020), but with two important differences.
First, while MMT uses 7 expert features that results in 7 × the sequence length, we only use a transformer to attend to early-fused motion and appearance features as the video representation, thus signiﬁcantly reducing the sequence length and computational cost.
Second, instead of stacking 6 transformer layers to encode the visual stream as in MMT, we only use a shallow two-layer transformer architecture with additional pre-encoders, further increasing model efﬁciency.
As temporal 1D-convolutional neural networks (CNNs) (LeCun et al., 1998) were shown to effectively capture temporal dependencies in videos (Dong et al., 2019), we integrate CNNs into our transformer pooling heads to better capture video temporal signals.
In more detail, we compute c v = Ψ
(cid:48)(cid:48)
( e v ) by chaining two transformer layers, each of the type:
[Equation]
Here f is a pre-encoder that reﬁnes the video representation; we found empirically that a 1D CNN works well for this purpose.
Then, we apply multi-head self-attention (MHA) (Vaswani et al., 2017) followed by a feed-forward network (FNN) with batch normalization (BN) (Ioffe & Szegedy, 2015).
The architecture maps the input sequence e v to a new ‘contextualized’ sequence of representation vectors; we take the ﬁrst one as c v .
The text representation decomposes in the same way as c t =
Φ( t ) =
Φ (cid:48)(cid:48) (Φ (cid:48) ( t )) .
The text encoder e t =
Φ (cid:48) ( t ) uses a pretrained T-5 network resulting in a code e t =
[ e t 1 · · · e tN ], where N is the maximum length of a sentence.
The pooling head c t = Φ (cid:48)(cid:48) ( e t ) follows the same design as the video case, but f is set to a recurrent neural network (RNN) instead of a CNN.
Please refer to the appendix for details.
In practice, for computational reasons, we use eq.
(3) to ﬁnetune the parameters of all networks except the video encoder Ψ (cid:48) ( v ) , which is ﬁxed.
E XPERIMENTS
We validate empirically the ability of our method to learn better representations for the downstream tasks of text-to-video and video-to-text retrieval.
First, in sec.
4.2 we ablate various model components on the MSR-VTT dataset.
Then, in sec. 4.3 we show that our best model signiﬁcantly outperforms state-of-the-art retrieval systems on three datasets, MSR-VTT, ActivtyNet and VATEX.
Finally, in sec.
4.4 we analyse qualitatively the effect of the attention mechanism used during training.
4. E XPERIMENTAL S ETUP Datasets.
HowTo100M (Miech et al., 2019) is a large-scale instructional video collection of 1. million YouTube videos, along with automatic speech recognition transcripts.
We use this dataset for our pre-training experiments.
MSR-VTT (Xu et al., 2016) contains 10000 videos, where each video is annotated with 20 descriptions.
We report results on the 1k-A split (9000 training,testing) as in Liu et al.
VATEX (Wang et al., 2019) is a multilingual (Chinese and English) video-text dataset with 34,911 videos.
We use the ofﬁcial training split with 25991 videos and report on the validation split as in HGR (Chen et al., 2020b).
The ActivityNet Caption (Krishna et al., 2017) dataset consists of densely annotated temporal segments of 20K YouTube videos.
We use the 10K training split to train from scratch/ ﬁnetune the model and report the performance on the 5K ‘val1’ split.
The MSVD (Chen & Dolan, 2011) dataset consists of 80 K English descriptions for
[Table] [Table]
[Table] [Table]
1970 videos from YouTube, with each video associated with around 40 sentences each.
We use the standard split of 1200, 100, and 670 videos for training, validation, and testing (Liu et al., 2019;
Venugopalan et al., 2015b; Xu et al., 2015).
Evaluation Metrics.
To measure the text-to-video and video-to-text retrieval performance, we choose Recall at K (R@K) and Median Rank (MedR), which are common metrics in information retrieval.
4. A BLATIONS
[Table]
In Tab. 1, we ﬁrst only ablate the cross-modal retrieval part of our network architecture, while the generative objectives are analysed in Tab. 2.
Video Encoder.
In Tab.
1a, we show the effect of the choice of visual input features.
We ﬁnd that for text-to-video retrieval at Recall at 1 and 5 ( R @1 , R @5 ), features obtained from a video R(
+1)D
ResNet achieve 2 .
9% and 7 . 0% higher performance compared to only image-frame based features from a ResNet-152.
A further 3 .
5% and 2 . 0% can be gained by concatenating both features, yielding the strongest MdR of 3 . 0% .
Feature Aggregation.
While the features from both video and image-based visual encoders have reduced spatial extent after a fully-connected layer, the temporal dimension can be reduced in various ways.
In Tab.
1b, we ﬁnd that our multi-head, parameterized attention reduction yields strong gains over the mean- or max-pooling baselines of over 4% for
R @1 .
This shows that learning attention over the temporal dimension of ﬁxed feature sets can give strong gains even without ﬁne-tuning the encoder.
Text Encoder.
In Tab. 1c, we ﬁnd decent gains of 2 .
7% and 0 .
4% for R@1,5 for using T5-base, instead of T5-small.
We do not use the T-5-Large model, as in Korbar et al.
(2020), due to the prohibitively large relative model size increase of +220%.
Text Decoder.
In Tab.
1d, we ﬁnd that using a larger text decoder gives a 1% increase in performance when using the cross-captioning objective.
Contrastive Loss.
To validate the choice of a triplet loss in eq.
(1), in Tab.
1e, we compare the results of the InfoNCE contrastive loss (Oord et al., 2018) with a triplet loss, with both the intra and inter-intra modality variants.
We ﬁnd that InfoNCE (Oord et al., 2018) loss does not work well in our case, likely due to the difﬁculty in tuning this loss to have the right combination of temperature and batch-size.
Support-Set Size.
Lastly, in Tab. 1f, we show the effect of the size of the support set used for cross-instance captioning.
We ﬁnd that our reconstruction loss indeed acts as a bottleneck, with both smaller and very large sizes degrading the performance.
Captioning Objective.
In Tab. 2, we show the effect of the different variants of our learning objective eq.
First, we ﬁnd that the naive addition of a reconstruction objective (“Identity”) does not improve the contrastive-only baseline (“None”) much.
Considering reconstruction from other videos improves the performance more.
In particular, the “Hybrid” variant, which combines “Identity” and “Full” (sec. 3.1.2) improves Recall at 1 and 5 from 25 .
9% and 53 .
0% to 26 .
0% and 54 .
8% , respectively.
However, the best result by far ( 27 . 2 / 55 .
2% ) is obtained forcing captions to be reconstructed only from other videos, via our cross-instance attention mechanism (“Cross”).
This variant cannot use information contained in a video to generate the corresponding caption and thus entirely relies on the model to discover meaningful relationship between different videos.
This newly-proposed scheme seems to have the most beneﬁcial effect for semantic retrieval.
[Table]
4. C OMPARISON TO S TATE - OF - THE -A RT
In this section, we compare the results of our method to other recent text-to-video and video-to-text retrieval approaches on various datasets.
In Tab. 3 to 5, we show the results of our model applied to text-to-video and video-to-text retrieval on MSR-VTT, VATEX, ActivityNet and MSVD with and without pre-trainig on HowTo100M. Without pre-training, our method outperforms all others in all metrics and datasets.
In particular, for the VATEX dataset, our retrieval performance at recall at 1 and 5 is 45 .
9% and 82 .
4% , exceeding recent state-of-the-art methods (Chen et al., 2020b) by a margin of 9% .
For ActivityNet, our model outperforms MMT by a margin of 4% at recall at 1.
With
[Table] [Table]
[Table]
-training on HowTo100M, our performance further increases across the board.
Notably, unlike
MMT which uses 7 features, our model uses only 2 features and achieves state-of-the-art in most metrics
[Table]
4. A NALYSIS
In order to better understand the effect of our learning objective, we visualize the soft attention of our best-performing cross-instance reconstruction model in ﬁg. 3.
As we can see in the top-left square, which shows the pairwise attention between all pairs of videos in the batch, it is highly focused, with the model mostly attending one or two other instances in the batch.
For the ﬁrst video’s caption reconstruction (second row), we ﬁnd that the model solely attends to another musical performance video that is in the batch, ignoring the others.
For the second video (third row), the model focuses on another sample that shows the sea but differs in most other aspects since there are no semantically-equivalent clips in the batch.
The third video shares a similar scenario.
These examples show that the bottleneck is effective at forcing the model to avoid memorising the videocaption association of each clip in isolation, and attempt to match other clips more broadly, since an exact (or very close) match is not guaranteed.
C ONCLUSION
In this work, we studied classic contrastive learning methods such as the triplet loss to learn videotext representations for cross-model retrieval.
We suggested that the contrastive approach might pull apart videos and captions even when they are semantically equivalent, which can hinder downstream retrieval performance.
To mitigate this effect, we propose to consider a captioning pretext task as an additional learning objective.
In particular, we show that cross-instance captioning can encourage the representation to pull together videos that share a similar caption, and are thus likely to be equivalent for retrieval.
Leveraging these ideas, our model achieves state-of-the-art performance on the text-to-video and video-to-text retrieval tasks, on three datasets.
While we demonstrated these ideas in the speciﬁc case of text-to-video retrieval, they can in principle generalize to any setting that utilizes a contrastive loss, including self-supervised learning, provided that it is possible to learn reasonable conditional generators of a modality or data stream given another.
A PPENDIX
The appendix is organized as follows: First, we provide more details about our model.
Then we introduce the datasets and the experimental setup.
Finally, we provide additional qualitative and quantitative experimental results for video-text retrieval and captioning.
6. M ODEL D ETAILS
Implementation details and hyper parameters.
For our text encoder, we use the T5-base model pre-trained on the “Colossal Clean Crawled Corpus” (C4) (Raffel et al., 2019).
We use its corresponding text tokenizer and encode a sentence into a sequence of 1024 dimensional vectors.
For our visual encoder, our model utilizes only the motion and the appearance features.
For the motion feature, we use a 34-layer, R(
+1)-D (Tran et al., 2018) model pre-trained on IG65M (Ghadiyaram et al., 2019) and apply a spatial-temporal average pooling over the last convolutonal layer, resulting in a 512-dimensional vector.
For the appearance feature, we use the 2048-dimension ﬂattened pool-5 layer of the standard ResNet152 (He et al., 2016) pre-trained on Imagenet (Deng et al., 2009).
We extract features at a rate of 1 feature per second and simply concatenate the two features, resulting in a 2560-dimension visual input stream.
Noteworthily, instead of using 9 and 7 different types of visual features as in CE (Liu et al., 2019) and MMT (Gabeur et al., 2020), we use only the above 2 features and achieve on par or superior performance.
Also, with early fusion, our model does not suffer from additional computation required for the extended sequence length in MMT.
For the text decoder, we use the T5-base model decoder, also pre-trained on C4.
[Figure]
As illustrated in Fig.
4, our transformer pooling head is composed of a preencoder, a multi-head self-attention, and a FFN layer.
For pre-encoders, we use a one-layer MLP with a d -dimensional output for mapping video features into the common embedding space.
We use 1024-dimension bidirectional GRU as the text pre-encoder.
For the 1D-CNN prior, we use kernels with size [2 , 3 , 4 , 6] as the visual and text pre-encoders.
We set the embedding dimension to 1024 and use 4 attention heads in the transformer pooling layers.
The hidden dimension of FFN is 2048.
Training and Inference time.
Pre-training on 1.2 million HowTo100M videos takes around 160 GPU hours (NVIDA V100) for 20 epochs.
We speed up the pre-training process by distributing the workload over 8 GPUs.
We use 1 GPU for the ﬁne-tuning or training from scratch experiments.
For the MSR-VTT 1k-A split, it takes 12 GPU hours to train our full model on 180K video-text pairs for 20 epochs.
For Vatex, it takes 32 GPU hours to train on 260K video-text pairs for 30 epochs.
For ActivityNet, it takes 2.
GPU hours to train on 10K video-text paris for 28 epochs.
For inference, the encoding speed is around 250-300 video/sec and 200text query/sec.
The overall text-to-video search speed on 5,000 video-text pairs (5,000 text queries over 5,000 videos) is 30-34 seconds including encoding.
The speed of text-to-video retrieval is similar to video-to-text retrieval.
6. E XPERIMENT D ETAILS
The margin α of the max-margin loss is 0.2, and the temperature T is set to 0.1 as used in SimCLR Chen et al.
(2020c).
We use the Adam (Kingma & Ba, 2015) optimizer with a initial learning rate 5 · 10 − 5 and clip gradients greater than 0.2 during the training phase.
Dropout rate is 0.3 for all datasets besides ActivityNet (0.0).
As the average video/text lengths and videos available are quite different across datasets, we adjust our training scheme accordingly.
When training on MSR-VTT, ActivtyNet and Vatex, batch-size is set to 64.
For MSR-VTT training, we sample and truncate videos to 32 seconds, text to 100 tokens and train for 20 epochs.
For Vatex, videos are at most 64 seconds
and we train for 30 epochs.
For
ActivtityNet training, videos are at most 512 seconds and 256 tokens for the text part.
We train for 28 epochs on ActivityNet.
For ﬁne-tuning HowTo100M pre-trained model, we reduce training epochs into quarters.
6. D ATASET D ETAILS
HowTo100M (Miech et al., 2019) is a large-scale instructional video collection of 1.2 million
Youtube videos, along with automatic speech recognition transcripts.
There are more thanmillion clips (ASR segments) deﬁned in HowTo100M.
We use this dataset for pretraining.
MSR-VTT (Xu et al., 2016) contains 10,000 videos, where each video is annotated with 20 descriptions.
For retrieval experiments and ablation studies, we follow the training protocol and deﬁned in Gabeur et al.
; Liu et al.
; Miech et al.
and evaluate on text-to-video and video-to-text search tasks on the 1k-A testing split with 1,000 video or text candidates deﬁned by Yu et al.
For captioning task, we evaluate on the standard testing split with 2,990 videos.
VATEX (Wang et al., 2019) is a multilingual (Chinese and English) video-text dataset with 34, videos.
We use the ofﬁcial split with 25,991 videos for training.
As the testing annotations are private in VATEX, we follow the protocol in Chen et al.
(2020b) to split the validation set equally (1,500 validation and 1,500 testing videos) for model selection and testing.
For each video,
English and 10 Chinese descriptions are available, and we only use the English annotations.
ActivityNet Dense Caption dataset consists densely annotated temporal segments of 20K YouTube videos.
Following Gabeur et al.
; Zhang et al.
(2018), we concatenate descriptions of segments in a video to construct “video-paragraph” for retrieval and captioning.
We use the 10K training split to train from scratch/ ﬁnetune the model and report the performance on the 5K ’val1’ split.
MSVD dataset consists of 80 K English descriptions for 1,970 videos from YouTube, with each video associated with around 40 sentences each.
We use the standard split of 1200, 100, andvideos for training, validation, and testing (Liu et al., 2019; Venugopalan et al., 2015b; Xu et al., 2015).
6. V IDEO C APTIONING E XPERIMENTS
To measure captioning/text generation performance
[Table]
we report BLEU4 (Papineni et al., 2002), METEOR (Denkowski & Lavie, 2014), Rogue-L (Lin, 2004) and CIDEr (Vedantam et al., 2015) metrics.
We report results on the MSR-VTT, VATEX and ActivityNet datasets
[Table] [Table]
[Table] [Table]
Z ERO -S HOT R ETRIEVAL E XPERIMENTS
We also evaluate our model in the zero-shot setting on MSR-VTT, Vatex, ActivityNet and MSVD, after pre-training on HT100M. While we are able to get reasonable results on MSR-VTT and MSVD, our results are not great on Vatex and Activity-Net due to signiﬁcant domain gap.
[Table] [Table]
A CTION R ECOGNITION
Lastly
[Table]
we evaluate our model on the video action recognition task on HMDB-51 (Kuehne et al., 2011) and UCF-101 (Soomro et al., 2012).
For this, we use the R(2+1)D-34 (pretrained on IG65M) model as well as a ResNet-152 model (pretrained on Imagenet), as in our method.
We extract a feature per second per video by concatenating the features from each model (2560-D), and obtain an average representation per video using either average pooling (2560-D) or our proposed transformer pooling head (1024-D) pre-trained on HT100M using cross-captioning objective.
We then train a linear classiﬁer for 1500 epochs for HMDB-51 ( 500 for UCF-101) on these features using Adam (Kingma & Ba, 2015) optimizer with learning rate of 1 e − 4 and weight decay 1 e − 4 with early stopping.
We also drop the learning rate by 10 at epochs 200 , 400 for UCF-101 and 1000 ,for HMDB-51. In Table 11, we show the results of training only a linear-layer on features extracted from our ﬁxed backbone with or without a learned transformer-pooling head.
We ﬁnd that our transformer temporal pooling head provides signiﬁcant beneﬁts over the baseline of simply average pooling the features, demonstrating the effectiveness of building contextualized representations using our proposed transformer.
In particular, we see improvements of over 7% on HMDB-51 and 34% on UCF-101 by replacing average pooling with our transformer pooling head to aggregate features. We observe that naive average pooling performs signiﬁcantly worse than our transformer pooling under evaluation protocol.
This is likely because 1) the average pooling collapses temporal information, making the linear layer based classiﬁcation difﬁcult 2) compared to the transformer pooling, it does not beneﬁt from large-scale pretraining on a wide variety of action videos of HT100M. We further compare very favorably to the current state-of-the-art approaches. In particular, we outperform all other approaches, both supervised and self-supervised, except the recently introduced Omni (Duan et al., 2020) which was ﬁnetuned on both UCF-101 and HMDB-51, while we only trained a linear classiﬁer on extracted features. However, it should be noted that it is very difﬁcult to fairly compare all these different approaches because they may use different modalities (images, RGB video, optical ﬂow, audio, ASR outputs), pretraining datasets (Kinetics-400, HT100M, IG65M, Imagenet), architectures (S3D, I3D, R(2+1)D, R3D), pre-training (supervised, self-supervised) and downstream training (frozen, ﬁnetuned) strategies.
6. S TATISTICAL SIGNIFICANCE
In Table 12, we show the results of ﬁnetuning our pretrained model for 3 times on the VATEX dataset.
We ﬁnd that the variance is quite low and our model consistently beats the state of the art. [Table] [Table] [Table]
[Table]
6. A DDITIONAL Q UALITATIVE R ESULTS
We provide addition qualitative text-to-video retrieval results on MSR-VTT, VATEX, ActivityNet in
Fig.
Given a text query, in most cases, our model successfully retrieves the correct videos marked in green.
a person is swimming in some white water rapids (0.72) (0.62)  (0.61) a man is showing the interior of a car (0.77) (0.71)   (0.69) (a) MSR-VTT
a kid riding on a horse while a woman is talking   a man is snow skiing down the mountain slope smoothly   (0.83) (0.74) (0.64) (0.80) (0.73) (0.68) (b) VATEX A woman is seen speaking to the camera while holding an accordion and moving her hands around .
She demonstrates how to play the instrument while still speaking to the camera and moving all around
[Figure]
A close up of nails are seen followed by a shot of brushes and nail polish .
A person is then seen wiping polish onto a pad and rubbing the object all over her nails .
She then puts a coating over the nail and shows it off again   (0.74) (0.73) (0.68) (0.68) (0.67) (0.64) (
[Figure]
) ActivityNet