A CTIVE C ONTRASTIVE L EARNING OF A UDIO -V ISUAL V IDEO R EPRESENTATIONS
A BSTRACT
Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound on the mutual information (MI) between different views of an instance.
However, obtaining a tight lower bound requires a sample size exponential in MI and thus a large set of negative samples.
We can incorporate more samples by building a large queue-based dictionary, but there are theoretical limits to performance improvements even with a large number of negative samples.
We hypothesize that random negative sampling leads to a highly redundant dictionary that results in suboptimal representations for downstream tasks.
In this paper, we propose an active contrastive learning approach that builds an actively sampled dictionary with diverse and informative items which improves the quality of negative samples and improves performances on tasks where there is high mutual information in the data, e g., video classiﬁcation.
Our model achieves state-of-the-art performance on challenging audio and visual downstream benchmarks including UCF101, HMDB51 and ESC
I NTRODUCTION
Contrastive learning of audio and visual representations has delivered impressive results on various downstream scenarios (Oord et al., 2018; H ´ enaff et al., 2019; Schneider et al., 2019;
Chen et al., 2020).
This self-supervised training process can be understood as building a dynamic dictionary per mini-batch, where “keys” are typically randomly sampled from the data.
The encoders are trained to perform dictionary look-up: an encoded “query” should be similar to the value of its matching key and dissimilar to others.
This training objective maximizes a lower bound of mutual information (MI) between representations and the data (Hjelm et al., 2018; Arora et al., 2019).
However, such lower bounds are tight only for sample sizes exponential in the MI (McAllester & Stratos, 2020), suggesting the importance of building a large and consistent dictionary across mini-batches.
Recently, He et al.
(2020) designed Momentum Contrast (MoCo) that builds a queue-based dictionary with momentum updates.
It achieves a large and consistent dictionary by decoupling the dictionary size from the GPU/TPU memory capacity.
However, Arora et al.
(2019) showed that simply increasing the dictionary size beyond a threshold does not improve (and sometimes can even harm) the performance on downstream tasks.
Furthermore, we ﬁnd that MoCo can suffer when there is high redundancy in the data, because only relevant – and thus limited – parts of the dictionary are updated in each iteration, ultimately leading to a dictionary of redundant items (we show this empirically in Fig. 3).
We argue that random negative sampling is much responsible for this: a randomly constructed dictionary will contain more “biased keys” (similar keys that belong to the same class) and “ineffective keys” (keys that can be easily discriminated by the current model) than a carefully constructed one.
Furthermore, this issue can get aggravated when the dictionary size is large.
In this paper, we focus on learning audio-visual representations of video data by leveraging the natural correspondence between the two modalities, which serves as a useful self-supervisory signal (Owens & Efros, 2018; Owens et al., 2016; Alwassel et al., 2019).
Our starting point is contrastive learning (Gutmann & Hyv¨arinen, 2010; Oord et al., 2018) with momentum updates (He et al., 2020).
However, as we discussed above, there are both practical challenges and theoretical limits to the dictionary size.
This issue is common to all natural data but is especially severe in video; successive frames contain highly redundant information, and from the information-theoretic perspective, audiovisual channels of video data contain higher MI than images because the higher dimensionality – i.e., temporal and multimodal – reduces the uncertainty between successive video clips.
Therefore, a dictionary of randomly sampled video clips would contain highly redundant information, causing the contrastive learning to be ineffective.
Therefore, we propose an actively sampled dictionary to sample informative and diverse set of negative instances.
Our approach is inspired by active learning (Settles, 2009) that aims to identify and label only the maximally informative samples, so that one can train a high-performing classiﬁer with minimal labeling effort.
We adapt this idea to construct a non-redundant dictionary with informative negative samples.
Our approach, Cross-Modal Active Contrastive Coding ( CM-ACC ), learns discriminative audiovisual representations and achieves substantially better results on video data with a high amount of redundancy (and thus high MI).
We show that our actively sampled dictionary contains negative samples from a wider variety of semantic categories than a randomly sampled dictionary.
As a result, our approach can beneﬁt from large dictionaries even when randomly sampled dictionaries of the same size start to have a deleterious effect on model performance.
When pretrained on AudioSet (Gemmeke et al., 2017), our approach achieves new state-of-the-art classiﬁcation performance on
UCF101 (Soomro et al., 2012), HMDB51 (Kuehne et al., 2011), and ESC50 (Piczak, 2015b).
B ACKGROUND
Contrastive learning optimizes an objective that encourages similar samples to have similar representations than with dissimilar ones (called negative samples)
(Oord et al., 2018):
[Equation]
The samples x + and x − are drawn from the same distribution as x ∈ X , and are assumed to be similar and dissimilar to x , respectively.
The objective encourages f ( · ) and h ( · ) to learn representations of x such that ( x, x + ) have a higher similarity than all the other pairs of ( x, x − ) .
We can interpret it as a dynamic dictionary look-up process:
Given a “query” x , it ﬁnds the correct “key” x
+ among the other irrelevant keys x − in a dictionary.
Denoting the query by q = f ( x ) , the correct key by k + = h
( x + ) , and the dictionary of K negative samples by { k i
= h ( x i ) } , i ∈ [1 , K ] , we can express equation 1 in a softmax form, min θ q ,θ k E x
[Equation]
p X (cid:104) − log
[Equation]
Ki =
[Equation]
e q
[Equation]
ki
[Equation]
(cid:105) , where θ q and θ k are parameters of the query and key encoders, respectively, and τ is a temperature term that controls the shape of the probability distribution computed by the softmax function.
Momentum Contrast (MoCo) decouples the dictionary size from the mini-batch size by implementing a queue-based dictionary, i.e., current mini-batch samples are enqueued while the oldest are dequeued (He et al., 2020).
It then applies momentum updates to parameters of a key encoder θ k with respect to parameters of a query encoder, θ k ← mθ k + (1 − m ) θ q , where m ∈ [0 , 1) is a momentum coefﬁcient.
Only the parameters θ q are updated by back-propagation, while the parameters θ k are deﬁned as a moving average of θ q with exponential smoothing.
These two modiﬁcations allow MoCo to build a large and slowly-changing (and thus consistent) dictionary.
Theoretical Limitations of Contrastive Learning.
Recent work provides theoretical analysis of the shortcomings of contrastive learning.
McAllester & Stratos (2020) show that lower bounds to the
MI are only tight for sample size exponential in the MI, suggesting that a large amount of data are required to achieve a tighter lower bound on MI.
He et al.
(2020) empirically showed that increasing negative samples has shown to improve the learned presentations.
However, Arora et al.
(2019) showed that such a phenomenon does not always hold: Excessive negative samples can sometimes hurt performance.
Also, when the number of negative samples is large, the chance of sampling redundant instances increases, limiting the effectiveness of contrastive learning.
One of our main contributions is to address this issue with active sampling of negative instances, which reduces redundancy and improves diversity, leading to improved performance on various downstream tasks.
𝑒𝑛𝑐𝑜𝑑𝑒𝑑 𝑘𝑒𝑦 𝑝𝑜𝑜𝑙𝑠 𝑞 !
𝑘 "! ,𝑘 #! …
𝑒𝑛𝑞𝑢𝑒𝑢𝑒 𝐾𝑚𝑒𝑎𝑛𝑠 $$ 𝐷 ! 𝑑𝑒𝑞𝑢𝑒𝑢𝑒 𝑔𝑟𝑎𝑑𝑖𝑒𝑛𝑡 𝑠𝑝𝑎𝑐𝑒 𝑞 % 𝑒𝑛𝑞𝑢𝑒𝑢𝑒 𝐾𝑚𝑒𝑎𝑛𝑠 $$ 𝑑𝑒𝑞𝑢𝑒𝑢𝑒 𝑔𝑟𝑎𝑑𝑖𝑒𝑛𝑡 𝑠𝑝𝑎𝑐𝑒 𝑘 "% ,𝑘 #% … 𝐷 " 𝑓 " ℎ " ℎ !
𝑓 !
𝑞 !
𝑘 "! ,𝑘 #! …
𝑘 "% ,𝑘 #% … 𝑞 % 𝑣 &’()* 𝑣 "+(* ,𝑣 #
+(* … 𝑎 "+(* ,𝑎 #+(* … 𝑎 &’()* 𝑎𝑐𝑡𝑖𝑣𝑒 𝑠𝑎𝑚𝑝𝑙𝑖𝑛𝑔 𝑚𝑜𝑚𝑒𝑛𝑡𝑢𝑚𝑢𝑝𝑑𝑎𝑡𝑒 𝑚𝑜𝑚𝑒𝑛𝑡𝑢𝑚𝑢𝑝𝑑𝑎𝑡𝑒 𝑏𝑎𝑐𝑘𝑝𝑟𝑜𝑝 𝑏𝑎𝑐𝑘𝑝𝑟𝑜𝑝 𝑐𝑜𝑛𝑡𝑟𝑎𝑠𝑡𝑖𝑣𝑒 𝑙𝑜𝑠𝑠 𝑐𝑜𝑛𝑡𝑟𝑎𝑠𝑡𝑖𝑣𝑒 𝑙𝑜𝑠𝑠 𝐷 !
𝐷 "
[Figure]
A PPROACH 3. C ROSS -M ODAL C ONTRASTIVE R EPRESENTATION L EARNING
Our learning objective encourages the representations of audio and visual clips to be similar if they come from the same temporal block of a video.
Let A =
{ a 0 , · · · , a N − 1 } and V = { v 0 , · · · , v N − 1 } be collections of audio and visual clips, where each pair ( a i , v i ) is from the same block of a video.
We deﬁne query encoders f a and f v and key encoders
h a and h v for audio and visual clips, respectively, with learnable parameters { θ aq , θ vq } for the query encoders and { θ ak , θ vk } for the key encoders.
These encoders compute representations of audio and visual clips as queries and keys,
[Equation]
We train our encoders to perform cross-modal dictionary look-up, e.g., given a query video clip v query , we ﬁnd the corresponding audio clip a key from a dictionary D a .
Adapting MoCo (He et al., 2020) to our cross-modal setup, we implement a queue-based dictionary D a that stores keys of audio clips { k ai } Ki =1 , where K is the dictionary size.
We compute the contrastive loss and backpropagate the gradients only to the visual query encoder f v and update the parameters θ vq .
For the audio encoder h a , we apply the momentum update (He et al., 2020),
[Equation]
The parameter θ aq is not updated in this contrastive coding step; we update it during the audio-tovisual step (similar as above with the opposite modalities).
Here we explain the visual-to-audio step only; we perform bi-directional contrastive coding and train the whole model end-to-end.
3. A CTIVE S AMPLING OF N EGATIVE I NSTANCES : U NCERTAINTY AND D IVERSITY
The quality of negative samples is crucial in contrastive learning.
Existing work typically adopts random negative sampling.
However, we want a diverse set of negative samples so that comparisons between positive and negative pairs are the most informative they can be.
Motivated by active learning (Settles, 2009), we propose a gradient-based active sampling approach to improve the quality of negative samples.
In active learning, the learner chooses samples that seem maximally informative and queries an oracle for labels to obtain an optimal solution with a minimal labeling budget.
Adapting this to our setting, we can empower the learner to choose the maximally informative negative samples to construct a dictionary; the main question is how to measure the informativeness of samples without labels.
One way to measure informativeness is through the lens of uncertainty :
If a model is highly uncertain about its prediction of a sample, we can ensure the maximum update to the model by including the sample in a mini-batch (conversely, if the uncertainly is low for all samples in a mini-batch, the model update will be small).
Ash et al.
(2020) showed that gradients of a loss function with respect to the model’s most conﬁdent predictions can approximate the uncertainty of samples, demonstrating its effectiveness in active learning.
They provide a theoretical justiﬁcation by showing that gradient norms of the last layer of a neural network with respect to pseudo-labels provides a lower bound on gradient norms induced by any other labels.
In this work, we use gradients of the last layer to measure the uncertainty and encourage our model to include samples that have the highest gradient magnitudes to constitute a dictionary.
While the uncertainty of each individual samples is important, the diversity of samples is also a critical measure of informativeness.
Intuitively, it is possible that a model is highly uncertain about samples from particular semantic categories, but constructing a mini-batch of samples from just those categories can severely bias gradients and ultimately lead to a bad local minima.
There are several principled approaches to ensure diversity, e.g., submodular optimization (Fujishige, 2005) and Determinantal Point Processes (DPP) (Macchi, 1975; Kulesza & Taskar, 2011).
Unfortunately, those methods are typically inefﬁcient because of the combinatorial search space (Nemhauser et al., 1978;
Gilks et al., 1995).
In this work, instead of using the expensive solutions, we opt to the fast solution of Ash et al.
and use the initialization scheme of the k -MEANS ++ seeding algorithm (Arthur & Vassilvitskii, 2007) to sample a diverse set of negative samples.
3. C ROSS -M ODAL A CTIVE C ONTRASTIVE C ODING
Algorithm 1 describes our proposed cross-modal active contrastive coding (we provide a simpliﬁed version here; we include a more detailed version and another version without active sampling in
Appendix).
At a high-level, we initialize the dictionaries D v
and D a with K randomly drawn samples from V and A , respectively ( lines 3-4 ).
For each epoch, we construct “negative candidate pools” U v
and U a with N random samples from V and A , respectively ( lines 6-7 ).
For each iteration within an epoch, we actively select the most informative negative samples S v and S a from the pools U v
and U a , respectively, and enqueue them into the dictionaries D v and D a , respectively ( lines 9-21 ).
We then perform cross-modal contrastive coding, update the parameters of query encoders θ vq and θ aq via backpropagation, and apply momentum updates to the parameters of key encoders θ vk
and θ ak ( lines 22-27 ).
[Table]
Obtain S v ← k - MEANS ++ INIT ( { g v n : v n ∈ U v \ D v } , # seeds = M ) 15:
Update D v ← ENQUEUE ( DEQUEUE ( D v ) , S v ) 16: (cid:46)
Active sampling of negative audio keys for D a 17:
Encode mini-batch samples, q vi ← f v ( v i ) , ∀ v i ∈ B v 18:
Compute pseudo-label, ˜ y an ← arg max p (ˆ y an | a n , B v ) , ∀ a n ∈ U a \ D a 19:
Compute gradients g a n using the pseudo-labels ˜
y
an , ∀ n ∈ [1 , N ] 20:
Obtain S a ← k - MEANS ++ INIT ( { g a n : a n ∈ U a \ D a } , # seeds = M ) 21:
Update D a ← ENQUEUE ( DEQUEUE ( D a ) , S a ) 22: (cid:46) Cross-modal contrastive predictive coding 23:
Encode mini-batch samples, k vi ← h v ( v i ) , ∀ v i ∈ B v ,
k ai ← h a ( a i ) , ∀ a i ∈ B a 24:
Compute p ( y vi | v i , a i , D a ) and p ( y ai | a i , v i , D v ) , ∀ i ∈ [1 , M ] 25: (cid:46) Update model parameters 26:
Update parameters of query encoders θ vq and θ aq with backpropagation 27:
Momentum update parameters of key encoders θ vk
and θ ak 28: end for 29: end for 30: return Optimal solution θ vq , θ vk , θ aq , θ ak
Active sampling.
To measure uncertainty , we deﬁne a pseudo-label space induced by the queries from the other modality, and take the gradient of the last layer of a query encoder with respect to the most conﬁdent prediction, which we call the pseudo-label ˜ y .
For instance, in the case of sampling negative video keys from the pool U v ( lines 10-15 ), we compute the pseudo-posterior of a video key v n ∈ U v \ D a ,
[Equation]
where B a is the current mini-batch of audio queries and deﬁnes the pseudo-label space.
Note that we consider only the samples in U v \ D v to rule out samples already in D v .
Intuitively, this computes the posterior by the dot-product similarity between v n and all q ai ∈ B a , producing an M -dimensional probability distribution.
We then take the most conﬁdent class category as the pseudo-label ˜
y vn ( line 12 ) and compute the gradient according to the cross-entropy loss
[Equation]
where θ last is the parameters of the last layer of θ (in this case, θ aq of the audio query encoder h a ).
Intuitively, the gradient g v n measures the amount of change – and thus, the uncertainty – v n will bring to the audio query encoder h a .
One can interpret this as a form of online hard negative mining : The gradient is measured with respect to the most probable pseudo-label ˜
y vn induced by the corresponding audio query q aj .
When we compute the contrastive loss, the same audio query will be maximally confused by v n with its positive key v + per dot-product similarity, and v n in this case can serve as a hard negative sample.
Next, we obtain the most diverse and highly uncertain subset S v ⊆
U v \ D v using the initialization scheme of k - MEANS ++ (Arthur & Vassilvitskii, 2007) over the gradient embeddings g v ( line 14 ).
The k - MEANS ++ initialization scheme ﬁnds the seed cluster centroids by iteratively sampling points with a probability in proportion to their squared distances from the nearest centroid that has already been chosen (we provide the exact algorithm in the Appendix).
Intuitively, this returns a diverse set of instances sampled in a greedy manner, each of which has a high degree of uncertainty measured as its squared distances from other instances that have already been chosen.
Finally, we enqueue
S v into D v and dequeue the oldest batch from D v ( line 15 ).
We repeat this process to sample negative audio keys ( lines 16-21 ); this concludes the active sampling process for D v and D a .
Cross-modal contrastive coding.
Given the updated D v and D a , we perform cross-modal contrastive coding.
For visual-to-audio coding, we compute the posteriors of all video samples v i ∈ B v with respect to the negative samples in the audio dictionary D a ,
[Equation]
where the posterior is deﬁned over a cross-modal space with one positive and K negative pairs ( line 24 ).
Next, we backpropagate gradients only to the query encoders f v and f a ( line 26 ),
[Equation]
while applying momentum update to the parameters of the key encoders h v and h a ( line 27 ),
[Equation]
The momentum update allows the dictionaries to change their states slowly, thus making them consistent across iterations.
However, our cross-modal formulation can cause inconsistency in dictionary states because the gradient used to update query encoders are not directly used to update the corresponding key encoders.
To improve stability, we let the gradients ﬂow in a cross-modal fashion, updating part of f v and h
a using the same gradient signal from the contrastive loss.
We do this by adding one FC layer on top of all encoders and applying momentum update to their parameters.
For example, we apply momentum update to the parameters of the FC layer on top of h a using the parameters of the FC layer from f v .
We omit this in Alg. 1 for clarity but show its importance in our ablation experiments ( XMoCo (w/o fcl) in Table 1).
[Table]
R ELATED W ORK
Self-supervised learning has been studied in vision, language, and audio domains.
In the image domain, one popular idea is learning representations by maximizing the MI between different views of the same image (Belghazi et al., 2018;
Hjelm et al., 2018; Tian et al., 2019;
He et al., 2020).
In the video domain, several approaches have exploited the spatio-temporal structure of video data to design efﬁcient pretext tasks, e
[Table]
g. by adopting ordering (Sermanet et al., 2017; Wang et al., 2019b), temporal consistency (Dwibedi et al., 2019), and spatio-temporal statistics (Xu et al., 2019; Wang et al., 2019a; Han et al., 2019).
In the language domain, the transformer-based approaches trained with the masked language model (MLM) objective has been the most successful (Devlin et al., 2019;
Liu et al., 2019; Yang et al., 2019).
Riding on the success of BERT (Devlin et al., 2019), several concurrent approaches generalize it to learn visual-linguistic representations (Lu et al., 2019;
Li et al., 2020; Su et al., 2019; Tan & Bansal, 2019; Li et al., 2019).
CBT (Sun et al., 2019a) and
VideoBERT (Sun et al., 2019b) made efforts on adapting BERT-style pretraining for video.
Besides vision and language signals, several approaches learn audio-visual representations in a selfsupervised manner (Owens et al., 2016; Arandjelovic & Zisserman, 2017; Owens & Efros, 2018;
Owens et al., 2016).
Recently, audio-visual learning has been applied to enable interesting applications beyond recognition tasks, such as sound source localization/separation (Zhao et al., 2018;
Arandjelovic & Zisserman, 2018; Gao et al., 2018; Gao & Grauman, 2019a;b; Ephrat et al., 2018;
Gan et al., 2020; Zhao et al., 2019;
Yang et al., 2020) and visual-to-sound generation (Hao et al., 2018;
Zhou et al., 2018).
The work of Owens & Efros (2018), Korbar et al.
(2018), and Alwassel et al.
(2019) are similar in spirit to our own, but our technical approach differs substantially in the use of active sampling and contrastive learning.
Hard negative mining is used in a variety of tasks, such as detection (Li et al., 2020), tracking (Nam & Han, 2016), and retrieval (Faghri et al., 2017; Pang et al., 2019), to improve the quality of prediction models by incorporating negative examples that are more difﬁcult than randomly chosen ones.
Several recent work have focused on ﬁnding informative negative samples for contrastive learning.
Wu et al.
(2020) show that the choice of negative samples is critical in contrastive learning and propose variational extension to InfoNCE with modiﬁed strategies for negative sampling.
Iscen et al.
(2018) propose hard examples mining for effective ﬁnetuning of pretrained networks.
Cao et al.
(2020) utilize negative sampling to reduce the computational cost.
In the context of audio-visual selfsupervised learning, Korbar et al.
(2018) sample negatives under the assumption that the smaller the time gap is between audio and visual clips of the same video, the harder it is to differentiate them (and thus they are considered hard negatives).
Our proposed approach does not make such an assumption and estimates the hardness of negatives by directly analyzing the magnitude of the gradients with respect to the contrastive learning objective.
E XPERIMENTS
Experimental Setting.
We use 3D-ResNet18 (Hara et al., 2018)
as our visual encoders ( f v and h v ) in most of the experiments.
We also use R(2+1)D-18 (Tran et al., 2018) to enable a fair comparison with previous work (see Table 4).
For audio encoders ( f a and h a ), we adapt ResNet-18 (He et al., 2016) to audio signals by replacing 2D convolution kernels with 1D kernels.
We employ Batch
Normalization (BN) (Ioffe & Szegedy, 2015) with the shufﬂing BN (He et al., 2020) in all our encoders.
All models are trained end-to-end with the ADAM optimizer (Kingma & Ba, 2014) with an initial learning rate γ = 10 − 3 after a warm-up period of 500 iterations.
We use the mini-batch
[Figure]
a t
[Figure]
i
[Figure]
size M = 128 , dictionary size K = 30 × 128 , pool size N = 300 × 128 , momentum m = 0 .
999 , and temperature τ = 0 .
We used 40 NVIDIA Tesla P100 GPUs for our experiments.
We pretrain our model on Kinetics-700 (Carreira et al., 2019) and AudioSet (Gemmeke et al., 2017) when comparing with state-of-the-art approaches.
For Kinetics-700, we use 240K randomly selected videos that contain the audio channel.
On AudioSet, we use both a subset of 240K randomly selected videos and the 1.8M full set.
For our ablation study, we use Kinetics-Sound (Arandjelovic & Zisserman, 2017) that contains 22K videos from 34 classes that are potentially manifested both visually and audibly, and thus provides a relatively clean testbed for ablation purposes.
As for downstream tasks, we evaluate our models on action recognition using UCF101 (Soomro et al., 2012) and
HMDB51 (Kuehne et al., 2011), and on sound classiﬁcation using ESC50 (Piczak, 2015b).
Unimodal vs. cross-modal pretraining.
To validate the beneﬁts of cross-modal pretraining, we compare it to its unimodal counterparts.
We pretrain our model on Kinetics-Sound with a randomly sampled dictionary (similar to MoCo (He et al., 2020)); we call this XMoCo .
For the unimodal case, we pretrain two models on visual clips and audio clips, respectively; we call these SMoCo .
We also compare ours with a model trained from scratch ( Scratch ), and a model pretrained on KineticsSound in a fully-supervised manner ( Supervised ).
Lastly, we include XMoCo (w/o fcl) that is identical to XMoCo except that we do not include the additional FC layers on top of the encoders.
All these models are ﬁnetuned end-to-end on each downstream task using the same protocol.
Table 1 shows the top-1 accuracy of each downstream task.
We observe that all the self-supervised models outperform Scratch on all downstream tasks, suggesting the effectiveness of pretraining with contrastive learning.
We also see that our cross-modal objective outperforms the unimodal objective ( ∆ ( 4 (cid:13) - 3 (cid:13) )).
The comparisons between XMoCo vs. XMoCo (w/o fcl) and CM-ACC vs. CM-ACC (w/o fcl) show the effectiveness of the additional FC layer on top of the encoders ( ∆ ( 5 (cid:13) - 4 (cid:13) ), ∆ ( 6 (cid:13) - 7 (cid:13) )).
When adding the FC layer, the performance further improves on all three benchmarks.
This shows the importance of letting the gradients ﬂow in a cross-modal fashion.
Finally, the performance gap with the full-supervised case shows there is still room for improvement in the self-supervised approaches.
Next, we compare the number of unique categories the sampled instances originally belong to, using the ground-truth labels provided in the dataset.
Our logic is that the more categories the samples come from, the more diverse and less redundant the samples are.
We train these on UCF-101 over 300 iterations with different mini-batch sizes, M ∈ { 32 , 64 , 128 } .
Fig.
2 shows that active sampling selects more categories than random sampling across all three mini-batch sizes.
At M = 128 , active sampling (with gradient embedding) covers 60-70 % of categories on UCF101, which is substantially more diverse than random sampling (30-40 % ).
(A plot showing the probability of sampling unique negatives (instances from different categories) is shown in Appendix Figure 4.)
While both sampling schemes perform similarly in early iterations, active sampling starts choosing more diverse instances as the training progresses; this is because the gradient embedding becomes more discriminative with respect to the uncertainty.
Random vs. active sampling.
To validate the beneﬁt of active sampling over random sampling, we compare models pretrained with different sampling approaches on downstream tasks.
As shown in
Table 1, our CM-ACC outperforms the XMoCo, which uses random sampling, by large margins, i.e.
3.1 % , 1.9 % , and 6.2 % on UCF101, HMDB51, and ESC50, respectively ( ∆ ( 7 (cid:13) - 5 (cid:13) )).
[Table] [Table]
centroids with k -
Fig.
2 shows that gradient embeddings produce a more diverse set of negative samples than feature embeddings; this is consistent across all three batch sizes
Table 2 shows that this diversity helps achieve better downstream performances across all three benchmarks.
Fig.
5 (in Appendix) provides further insights, showing that the samples with high gradient magnitudes tend to be more informative negative samples
[Table]
From a theoretical aspect
[Table]
the gradient norm induced by each candidate with computed pseudo labels estimates the candidates inﬂuence on the current model.
The gradient embeddings convey information both about the model’s uncertainty and potential update direction upon receiving a candidate.
However, such messages are missing from the feature embeddings.
This shows the importance of considering both uncertainty and diversity when selecting random samples: the k - MEANS ++ ensures the diversity in the sample set, but without the uncertainty measure we lose important discriminative information from the candidates.
Online hard example mining vs. active sampling.
We compare our approach to online hard example mining (OHEM) (Shrivastava et al., 2016), which constructs negative samples by explicitly choosing the ones that incur high loss values.
Speciﬁcally, we compute the pseudo-labels for all keys (negative sample candidates) with a given mini-batch of queries.
We then compute the classiﬁcation loss based on these pseudo labels and select the top M keys with the highest loss values.
We pretrain the models on Kinetics-700 (Kay et al., 2017) and report the top-1 accuracy on UCF (Soomro et al., 2012) and HMDB51 (Kuehne et al., 2011).
We use the same architecture and hyperparameters; the only difference is the sampling approach.
Table 3 shows OHEM is generally less effective than both random sampling and our active sampling.
Intuitively, OHEM promotes the dictionary to contain the most challenging keys for a given minibatch of queries.
Unfortunately, this causes OHEM to produce a redundant and biased dictionary, e.g., negative samples coming from a particular semantic category.
Our results show that, when M (mini-batch size) is small, the performance of OHEM is even worse than random sampling, although the gap between OHEM and random sampling decreases as M increases.
We believe this is because
OHEM has a higher chance of selecting similar negative instances.
When M is large, this issue can be mitigated to some extent, but the performance still falls behind ours by a large margin.
This suggests the importance of having a diverse set of negative samples, which is unique in our approach.
Comparisons with SOTA.
Table 4 shows our approach outperforms various self-supervised approaches on action recognition.
For fair comparisons, we group the SOTA approaches by different pretraining dataset sizes, i.e. small-scale (UCF/HMDB), medium-scale (Kinetics), and large-scale (AudioSet).
Our gains are calculated according to this grouping.
As we can see, our approach outperforms SOTA approaches across all groups.
Compared with GDT (Patrick et al., 2020), the current top performing model on cross-modal self-supervised learning, our model outperforms it by 1.6 % on UCF101 and 1.1 % on HMDB51.
Table 5 shows audio classiﬁcation transfer results.
For
Kinetics and AudioSet (240K), our model outperforms the current state-of-the-art, AVID (79.1 % ) by 0.1 % and 1.8 % on Kinetics and AudioSet 240K, respectively.
Our approach also outperforms
AVID (89.2 % ) pretrained on AudioSet (1.8M) by 1.6 % .
[Table] [Table]
C ONCLUSION
We have shown that random sampling could be detrimental to contrastive learning due to the redundancy in negative samples, especially when the sample size is large
[Table]
and have proposed an active sampling approach that yields diverse and informative negative samples.
We demonstrated this on learning audio-visual representations from unlabeled videos.
When pretrained on AudioSet, our approach outperforms previous state-of-the-art self-supervised approaches on various audio and visual downstream benchmarks.
We also show that our active sampling approach signiﬁcantly improves the performance of contrastive learning over random and online hard negative sampling approaches.
A D ETAILS ON D ATA P ROCESSING
We preprocess video frames by sampling at 10 FPS and applying random cropping, horizontal ﬂipping, gray-scaling, and temporal jittering.
We resize video frames to 3-channel images of 224 × 224 ; we set the clip length to 16 frames during pretraining, and 32 frames during ﬁnetuning on downstream tasks.
For audio channel, we extract mel-spectrograms from the raw waveform using the
LibROSA library and get a 80 × T matrix with 80 frequency bands; T is proportionate to the length of an audio clip.
We then segment the mel-spectrogram according to the corresponding video clips to ensure temporal synchrony.
We treat the mel-spectrograms as an 80-channel 1D signal.
As for downstream tasks, we evaluate our models on action recognition using UCF101 (Soomro et al., 2012) and HMDB51 (Kuehne et al., 2011), and on sound classiﬁcation using ESC50 (Piczak, 2015b).
UCF101 contains 13K video clips from 101 action categories, HMDB51 contains 7K video clips from 51 categories, and ESC50 has 2K audio clips from 50 categories.
UCF101 and HMDB have 3 ofﬁcial train/test splits, while ESC50 has 5 splits.
We conduct our ablation study using split-1 of each dataset.
We report our average performance over all splits when we compare with prior work.
B A DDITIONAL E XPERIMENTS
Effect of mutual information.
We investigate the impact of the amount of MI on contrastive learning using the Spatial-MultiOmniglot dataset (Ozair et al., 2019).
It contains paired images ( x, y ) of Omniglot characters (Lake et al., 2015) with each image arranged in an m × n grid (each grid cell is 32 × 32 pixels).
Let l i be the alphabet size for the i th character in each image, then the MI
I ( x, y ) =
(cid:80) mni =1 logl i .
This way, we can easily control the MI by adding or removing characters.
We follow the experimental protocol of Ozair et al.
(Ozair et al., 2019), keeping the training dataset size ﬁxed at 50K and using the same alphabet sets: Tiﬁnagh (55 characters), Hiragana (52), Gujarati (48), Katakana (47), Bengali (46), Grantha (43), Sanskrit (42), Armenian (41), and Mkhedruli (41).
[Figure]
Fig. 3(a) shows the results as the number of characters (and thus the MI) increases.
We see that all approaches achieve nearly 99 % accuracy with less than 3 characters; this is the case when the exponent of the MI is smaller than the dataset size (50K), i.e., e I ( x,y )
=
55 with one character, e I ( x,y )
= 2 , 860 with 2 characters.
However, starting from 3 characters, the performance of the regular MoCo (SMoCo) drops signiﬁcantly; this is because the exponent of the MI (=137, (55 × 52 × 48)) is much larger than the dataset size.
Although our model also drops performance when the MI is increased, it outperforms the other approaches by a large margin.
We also observe that XMoCo outperforms SMoCo in mild conditions (1-5 characters) but performs nearly the same as SMoCo with severe conditions (6-9 characters).
This suggests that, while cross-modal prediction helps to learn good representations, it also suffers with the same issue when the MI is large, thus adopting active sampling is beneﬁcial.
Effect
of dictionary size.
Fig.
3 (b) shows how the dictionary size affects downstream task performance.
Here we pretrain our model on Kinetics-700 and ﬁnetune it on UCF-101.
Overall, all three approaches beneﬁt from large dictionaries up to a threshold (at about 10 3 ), which is consistent with previous empirical ﬁndings (He et al., 2020).
However, both XMoCo and SMoCo starts deteriorating performance after about 10 4 (which is consistent with previous theoretical claims of Arora et.
al (Arora et al., 2019)), whereas ours do not suffer even after 10 4 .
This suggests that there are performance limits by simply increasing the size of a randomly-sampled dictionary, and also shows the beneﬁt of our active sampling approach.
Effect of pretraining dataset sizes.
We investigate the effects of the size of pretraining datasets, using Kinetics-Sound (22k), Kinetics (240K), and AudioSet (1.8M).
We vary pretraining conditions while using the same protocol to ﬁnetune the models end-to-end on downstream tasks.
Table 6 shows that our model beneﬁts from pretraining on video data, and that the performance improves as we use a large pretraining video dataset (Kinetics and AudioSet) than the relatively smaller dataset (Kinetics-Sound).
Notably, our approach even outperforms the fully-supervised pretraining approaches by pretraining on a larger video dataset (1.0 % , 3.6 % , and 8.5 % improvement on
UCF101, HMDB51, and ESC50, respectively.)
[Table]
Diversity of random vs. active sampling.
To compare the diversity of the chosen negatives by random vs. active sampling, we plot the probability of them on sampling of unique negatives (instances from different categories).
The more categories the samples come from, we get more diverse and less redundant samples.
We train these on UCF-101 over 300 iterations with different mini-batch sizes, M ∈ { 32 , 64 , 128 } .
As shown in Figure 4, the active sampling selects more categories than random sampling across all three mini-batch sizes.
At M = 128 , active sampling (with gradient embedding) covers 60-70% of categories on UCF101, which is substantially more diverse than random sampling (30-40%).
[Figure] [Figure]
C V ISUALIZATION OF NEGATIVE INSTANCES
Figure 5 shows negative instances selected by active sampling and random sampling when we use audio clips as the query.
We visualize the center frames of the selected video clips.
We can see that our approach selects more challenging examples than the random sampling approach.
For instance, given a query opening bottle , our approach selected video clips from the same or similar semantic categories, e.g. drinking shot and opening bottle .
Given snowboarding , our approach selected more video clips related to categories containing the snow scene, e.g. ice fishing , snow kiting , and tobogganing .
Audio Query: Cheerleading
Singing
Sit-up
ThrowingAxe ThrowingAxe Tobogganing Tobogganing
[Figure]
m
Shooting Goal Shooting Goal
Skydiving
[Figure]
Salsa Dancing Jumpstyle Dancing
[Figure]
EatingChips OpeningBottle
OpeningBottle
[Figure]
Audio Query: Opening
[Figure]
ndo m
[Figure]
Blowing Leaves Blowing Leaves
Cheer- leading
[Figure]
Riding [Figure] Bike
Riding
[Figure]
Bike
Tapping Guitar
Unboxing Playing Violin Playing Recorder
A
[Figure]
i ve
Audio Query: Playing Guitar
R a ndo m
Reading a Book
Riding a Camel
Riding a Camel Getting a Tattoo
Reading a Book Reading a Book
[Figure]
American Football Shooting Basketball
Stretching Leg Zumba Hurling High Fiving
A c t i ve
Audio Query: Playing Tennis
R a ndo m
Picking
[Figure]
Taking Photo
[Figure]
Photo Helmet Diving
Picking Apples Picking Apples
[Figure]
Furthermore, we also ﬁnd that our approach selects more diverse negative samples.
For example, given a query snowboarding , active sampling selected video clips from 4 different categories related to the snow scene: ( ice fishing , playing ice hockey , snow kiting , and tobogganing ).
In comparison, the random sampling approach yields fewer semantic cate- gories in general.
This suggests that our active sampling approach produces more ‘challenging’ and ‘diverse’ negative instances than the random sampling approach.
To clearly investigate the relationship between negative samples and their gradient magnitudes, we show the gradient norm of each visualized sample in Figure 5.
We can see that hard negatives tend to have larger gradient norms than easy negatives.
Given a query Playing guitar , video clips containing the concept of “playing instruments” yield the higher gradient norms, i.e. playing violin (333.87) and tapping guitar (301.35), while concepts that are easy to discriminate, e.g., riding a camel yield a signiﬁcantly smaller gradient norm (5.92).
This provides evidence showing the gradient magnitude is effective in measuring the uncertainty of the current model, i.e., highly-uncertain samples (hard negatives) tend to yield gradients with larger magnitudes, while highly-conﬁdent samples (easy negatives) tend to have smaller gradient magnitudes.
D W HEN WOULD CROSS - MODAL CONTRASTIVE LEARNING FAIL ?
In general, cross-modal video representation learning is based on an assumption that the natural correspondence between audio and visual channels could serve as a useful source of supervision.
While intuitive, this assumption may not hold for certain videos in-the-wild, which may cause the model to learn suboptimal representations.
To investigate when our approach succeeds and fails, we conduct a post-hoc analysis by using thehttps://www.overleaf.com/project/5ded2abe1c17bc00011e5da ground-truth semantic category labels provided in Kinetics-700 (Carreira et al., 2019)
(which is not used during pretraining).
Speciﬁcally, we use our pretrained model to solve the audio-visual contrastive pretext task (Eqn.(7) in the main paper) and keep track of the prediction results (correct/incorrect).
We then average the pretext task accuracy over 100 randomly chosen samples for each action category.
[Figure]
Figure 6 shows the top-10 and bottom-5 classes by using both audio (left) and video (right) as the query.
We observe that, the top ranked classes for both audio and video are the activities that have highly correlated visual-audio signals.
For instance, playing bass guitar , play piano , and play violin are all activities related to music.
The correlation of audio-visual signals for these activities are obvious; such highly correlated signals are easier to be learned in a cross-modal manner.
On the contrary, the bottom ranked classes are those that have subtle audio-visual correlation, e.g. tossing coin , shaking hand , looking at phone , and hugging .
We also investigate the distribution of hard-easy classes with that reported in Kinetics-700 (Carreira et al., 2019) learned by the I3D-RGB model (Carreira & Zisserman, 2017).
Interestingly, we ﬁnd that some hard classes (e.g. karaoke and recording music ) are listed in our top ranked classes.
We suspect that, when only learned within visual modality, some classes with cluttered or complected spatial information will bring difﬁculties for classiﬁcation.
While, as our cross-modal approach can leverage information from both auditory and visual information, so our model does not limited by such problems.
[Table]
Compute pseudo-label, ˜ y vn ← arg max p (ˆ y vn | · ) 15: end for 16:
Compute gradient, g v n ← ∂ ∂θ last L CE ( p (ˆ y vn | · ) , ˜ y vn ) | θ = θ aq , ∀ n ∈ [1 , N ] 17:
Obtain S v ← k - MEANS ++ INIT ( { g v n : v n ∈ U v \ D v } , # seeds = M ) 18:
Update D v ← ENQUEUE ( DEQUEUE ( D v ) , S v ) 19: (cid:46)
Active sampling of negative audio keys for D a 20:
Encode mini-batch samples, q vi ← f v ( v i ) , ∀ v i ∈ B v 21: for ∀ a n ∈ U a \ D a : do 22:
Compute pseudo-posterior, p (ˆ y an | a n , B v ) ← exp( k an · q vj ) (cid:80) Mi
=1 exp( k an · q vi ) , ∀ j ∈ [1 , M ] 23:
Compute pseudo-label, ˜ y an ← arg max p (ˆ y an | · ) 24: end for 25:
Compute gradient, g a n ← ∂ ∂θ last L CE ( p (ˆ y an | · ) , ˜ y an ) | θ = θ vq , ∀ n ∈ [1 , N ] 26:
Obtain S a ← k - MEANS ++ INIT ( { g a n : a n ∈ U a \ D a } , # seeds = M ) 27:
Update D a ← ENQUEUE ( DEQUEUE ( D a ) , S a ) 28: (cid:46) Cross-modal contrastive predictive coding 29:
Encode mini-batch samples, k vi ← h v ( v i ) , ∀ v i ∈ B v ,
k ai ← h a ( a i ) , ∀ a i ∈ B a 30:
Compute p ( y vi |· ) = exp( q vi · k ai /τ ) (cid:80) Kj =
exp( q vi · k aj /τ ) , p ( y ai |· ) =
exp( q ai · k vi /τ ) (cid:80) Kj =
exp
( q ai · k vj /τ ) , ∀ i ∈ [1 , M ] 31: (cid:46) Update model parameters 32:
Update θ vq ← θ vq − γ ∇ θ L CE ( p ( y v | · ) , y vgt ) | θ = θ vq , θ aq ← θ aq − γ ∇ θ L CE ( p ( y a | · ) , y agt )
| θ = θ aq 33:
Momentum update θ vk ← mθ vk + (1 − m ) θ vq , θ ak ← mθ ak + (1 − m ) θ aq 34: end for 35: end for 36: return Optimal solution θ vq , θ vk , θ aq , θ ak end
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Compute the posterior, p ( y
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
20: (cid:46) Update model parameters 21:
[Equation] [Equation]
Momentum
[Table]
[Equation]
Momentum
[Table]
[Equation]
25: end for 26: end for 27: return Optimal solution θ
[Table]
, θ
[Equation]
[Table]
[Equation]
ak