D ISCOVERING I NFORMATIVE AND R OBUST P OSITIVES FOR V IDEO D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised domain adaptation for video recognition is challenging where the domain shift includes both spatial variations and temporal dynamics.
Previous works have focused on exploring contrastive learning for cross-domain alignment.
However, limited variations in intra-domain positives, false cross-domain positives, and false negatives hinder contrastive learning from fulfilling intra-domain discrimination and cross-domain closeness.
This paper presents a non-contrastive learning framework without relying on negative samples for unsupervised video domain adaptation.
To address the limited variations in intra-domain positives, we set unlabeled target videos as anchors and explored to mine "informative intra-domain positives" in the form of spatial/temporal augmentations and target nearest neighbors (NNs).
To tackle the false cross-domain positives led by noisy pseudo-labels, we reversely set source videos as anchors and sample the synthesized target videos as "robust cross-domain positives" from an estimated target distribution, which are naturally more robust to the pseudo-label noise.
Extensive experiments on several cross-domain action recognition benchmarks demonstrate the superiority of our approach over state-of-the-art methods.
I NTRODUCTION
Recent advances in deep neural networks have revolutionized many computer vision tasks, such as image and video recognition
He et al.
(2016), semantic segmentation Chen et al.
(2017), etc.
However, these impressive results heavily rely on human annotations, which are expensive and time-consuming to obtain.
To address this issue, semi-supervised learning Miyato et al.
and self-supervised learning (SSL)
He et al.
(2020) have been studied to utilize the knowledge from a label-rich dataset to a label-scarce dataset.
However, the domain shift problem between the source and target datasets usually exists in real-world scenarios, leading to performance degradation.
Unsupervised domain adaption (UDA) has been exploited to transfer knowledge across datasets with domain discrepancies to mitigate this problem.
While most of the methods have been developed only for images, the problem of UDA in videos remains largely underexplored.
Recently, a few works have attempted UDA for video action recognition by directly matching frame/clip-level features Chen et al. (2019a); Pan et al.
(2020a)
; Choi et al.
However, these methods usually extend the image-based UDA methods without considering long-term temporal information or action semantics.
Song and Kim et al.
Song et al.
; Kim et al.
(2021b) alleviate these issues with contrastive learning to learn such long-term spatial-temporal representations by instance discrimination.
To further understand how contrastive learning helps UDA, we firstly recall that domain-wise discrimination and class-wise closeness are the two main criteria to solve UDA problems Shi & Sha (2012);
Tang et al.
Considering an unlabeled video from the target domain as an anchor, we explain that Song et al.
(2021) introduces intra-domain positives (in the form of cross-modal correspondence, e.g, optical flow) to help UDA by learning discriminative representation in the target domain.
On the other hand, Kim et al.
(2021b) utilizes cross(source)- domain positives with the help of pseudo-labels and benefits the class-wise closeness by pushing the samples of the same class but different domains closer.
By thoroughly studying the effect of the positives and negatives selection from the existing contrastivebased methods in Figure 1 (a), we empirically find that limited variations in intra-domain positives, pseudo-label noise in cross-domain positives and false negatives are the three issues that largely hinder the performance.
Explicitly, when selecting the negatives based on the ground-truth, intra-
[Figure]
i
[Figure]
i c
[Figure]
i [Figure] domain positives from self-correspondence Song et al. (2021) is 3.5% below the ones from ground truth.
Further, selecting the cross-domain positives based on pseudo-labels leads to 2% drop compared to the ground-truth.
Importantly, when selecting the negatives from either different instance Song et al.
(2021) or pseudo-labels Kim et al.
(2021b), the performance of the four baselines in Figure 1 (a) drops by 2 ∼ 4 %.
Besides, the purity analysis in Figure 1 (b) also indicates that the pseudo-labels of target video are noisy and thus not reliable for selecting cross-domain positives and negatives.
Based on the observations, several straightforward questions might be raised:
Are there any unexplored intra-domain positives to enrich intra-domain variations?
How to alleviate the pseudo-label noise issue when selecting cross-domain positives?
How to address the adverse effect from false negatives?
To answer the first question, we propose to leverage the spatial and temporal augmentations of the unlabeled target video as intra-domain positives.
The rationale is that partially modifying the spatial/temporal information of videos could alter the samples without changing the whole action semantics.
Incorporating those samples as positives could help the model be invariant to spatial/temporal shift.
Take a step further, we also explore the anchor’s nearest neighbors (NNs) in target feature space, which capture rich target-domain variations.
By analyzing the purity of the NNs in both source and target domain in Figure 1 (b), we observe that target-domain NNs are clean and thus fit as intra-domain positives .
To address the second question, we are motivated from Xie et al.
(2018) that though the pseudo-labels of target samples are noisy, the class-conditional centroids µ tc of target features would weaken this noise by averaging all the target features of same pseudo class .
To incorporate it into our contrastive learning framework, we re-formulate the optimization reversely by setting source videos as anchors and then mining target videos as cross-domain positives.
We then estimate the gaussian target distribution N ( µ tc , Σ tc ) of target features based on pseudo-labels.
Given a source video as anchor, synthesized target features can be drawn from this distribution that shares the same class as the source anchor.
Consequently, we could leverage those synthesized features as cross-domain positives as the estimated target distribution is robust to pseudo-label noise.
To tackle the last question, we present a simple yet effective non-contrastive learning framework without relying on negative samples for video domain adaptation.
Specifically, we are motivated by the recent advances in SSL Grill et al.
; Chen & He (2021) to discard the negatives and maximize the similarity between the anchor and positives in feature space with the help of an extra
MLP head and a stop-gradient operation.
Beyond adopting this non-contrastive optimization into our setting, we highlight our compelling finding that the MLP head is crucial for intra-domain positives
[Figure]
optimization but largely hampers the convergence for cross-domain positives.
Consequently, we remove the MLP head for the latter case.
To summarize our main contributions:
• By studying the effects of the positives and the negatives from previous contrastive-based video DA methods, we make the first attempt to emphasize that limited variations in intradomain positives, pseudo-label noise in inter-domain positives and false negatives are the three under-explored bottlenecks •
We contribute a unified solution to address the above issues by introducing more informative and robust intra-domain and cross-domain positives without relying on negative samples for video DA problem.
• We conduct extensive experiments and ablation studies on challenging cross-domain action recognition benchmarks (UCF-HMDB and Epic-Kitchens) and thoroughly verify our effectiveness over state-of-the-art methods.
R ELATED W ORK 2. A CTION R ECOGNITION
Action recognition is a challenging problem with great application potential.
Many previous approaches are deployed in both 2D Karpathy et al.
(2014a), and 3D CNN-based framework (such as I3D Carreira & Zisserman (2017a), ResNet3D Hara et al.
(2017)), which achieves significant progress on RGB modality.
To incorporate multiple modalities or temporal dynamics, two-stream architecture Simonyan & Zisserman (2014) is widely used to process RGB and optical-flow in two separate CNNs with a late fusion in the upper layers Karpathy et al.
(2014b).
SlowFast network Feichtenhofer et al.
(2019) employs two pathways for recognizing actions by processing a video at different frame rates.
Despite promising results on supervised benchmarks, these models are critically dependent on large datasets that require careful human annotation effort.
In contrast, our focus is on unsupervised domain adaptation settings for action recognition where target domain data is unlabeled.
2. S ELF - SUPERVISED L EARNING
Self-supervised representation has witnessed tremendous progress on images and video by using different pretext tasks (i.e., rotations Gidaris et al.
(2018), jigsaw puzzle Noroozi & Favaro (2016))
and contrastive learning Chen et al.
; He et al.
; Han et al.
Recently, noncontrastive learning methods (i.e., BYOL Grill et al.
and SimSiam Chen & He (2021)) dominate the SSL performance by learning meaningful representation by maximizing the similarity between two augmented positive samples without considering negative pairs.
While our method is motivated by the non-contrastive learning above, we highlight our difference in mining "informative" and "robust" intra-domain and cross-domain positives beyond instance-level variations for better target-domain discrimination and class-wise closeness.
Importantly, we observe that the predictor in SimSiam Chen & He (2021), which is designed for preventing model collapse, would lead to convergence issue for cross-domain positives.
In contrast, we remove the predictor for cross-domain positives in optimization while maintain it for intra-domain positives.
We will explain the details in the experiment section.
2. U NSUPERVISED D OMAIN A DAPTATION
There are several directions to make UDA effective.
Specifically, mainstream methods seek for learning domain-invariant feature representations by minimizing some measurement of distribution discrepancy Tzeng et al.
(2014); Chung et al.
; Qiu et al.
(2021), utilizing adversarial learning Ganin et al.
; Long et al.
Saito et al.
(2018); Pan et al.
(2020b), cross-domain augmentation Li et al.
(2021) and contrastive alignment Saito et al.
Kim et al.
(2021a)
; Harary et al.
Self-training Xie et al.
; Zou et al.
(2018; 2019) also thrives on UDA by training the model with labeled source data and pseudo-labeled target data.
However, less attention has been paid to video domain adaptation until recently.
Existing video UDA methods can be summarized into two folds: temporal attention-based adversarial learning Chen et al.
(2019b);
Luo et al.
(2020); Pan et al.
(2020a)
and SSL-based cross-domain alignmentMunro & Damen (2020); Choi et al.
Song et al.
; Kim et al.
(2021b).
Explicitly, self-supervisions such as predicting video clips order Choi et al.
(2020), multi-modality correspondence Munro & Damen (2020) and contrastive learning Song et al.
; Kim et al.
(2021b)
; Sahoo et al.
(2021), have been exploited.
Compared to the image-based DA methods, our method provides a new perspective for domain discrimination and class-wise closeness via exploring positive samples in contrastive learning.
Compared to the contrastive-based video DA methods, (1) we provide empirical insights on the limitations of existing contrastive methods by analyzing the effect of positives ans negatives; (2) we address the issues by proposing more informative and robust intra-domain positives and cross-domain positives without replaying on negative samples.
P ROPOSED M ETHOD
In unsupervised video domain adaptation problem, we are given a source domain D s = { ( x si , y si ) | N s i =1 } of N s labeled source videos and a target domain D t = { ( x ti ) | N t
i =1 } of N t unlabeled target videos with a shared label space L .
The joint distributions of source and target domain are not identically and independently distributed, specifically defined as P ( x s , y s )
̸ = Q ( x t , y t ) .
Given these datasets, the goal is to train a deep neural network for action recognition on the labeled source videos D s and the unlabeled target videos D t such that the model can generalize well on unseen target domain videos.
To specify, the model is comprised of a feature extractor F followed by a classifier G .
3. I NTRA - DOMAIN P OSITIVES
The goal of intra-domain positives is to ensure domain-wise discrimination, which means that data in either source or target domains are discriminatively clustered.
Intra-source Discrimination.
As source domain videos have labels, we simply minimize the standard supervised cross-entropy loss ( L ce ) to achieve source-domain discrimination:
[Equation]
where p i,y si = G ( F ( x si ))
y si is the y si -th element of a K-dimensional prediction and K indicates the number of class.
Self-supervision as Intra-domain positives.
Without proper regularization on unlabeled target data, target-domain discrimination cannot be realized for free.
Motivated by the self-supervised learning methods Chen & He (2021); Bai et al.
(2021), we firstly propose to learn discriminative representation in target domain by maximizing the similarity between different augmented views of unlabeled target data.
Specifically, we consider 1) spatial augmentation, including standard image-based augmentation such as random flip, crop and color distortion.
2) temporal augmentation, including video play speed
Benaim et al.
(2020), and shuffled clips Fernando et al.
Precisely, we randomly double the sampling rate ( × 2 ) for video play speed or randomly shuffle a clip ( 1 / 3 consecutive frames) of a video.
The rationale behind is that the effectiveness of the intra-domain positives are highly coupled with how "informative" and "clean" the positives are given an anchor video.
"Informative" positives usually capture rich spatial variations (e.g., background/viewpoint changes) or temporal dynamics (e.g., video speed/action length).
Our choices of spatial and temporal augmentations above could partially change the anchor video along these variations without changing the whole action semantics.
Consequently, incorporating those samples as intra-domain positives could help the model be invariant to the variations in spatial and temporal axis.
Explicitly, our model takes two randomly augmented views x t 1 and x t 2 from a target video x t as input: one view is strongly augmented by one of the augmentations above while the other view is weakly augmented.
A feature extractor F processes the two views into feature space and a two-layers prediction MLP head, denoted as H , further transforms the feature of one view and matches it to the other view.
As shown in Figure 2 (a), our intra-target domain alignment loss can be formulated as maximizing the cosine similarity between the two views:
[Equation]
where h t 1 = H ( F ( x t 1 )) is the predictor output, z t 2 = F ( x t 2 ) is the spatial-temporal feature, ∥·∥ is l 2 -norm and D ( · , · ) is a cosine distance function.
Specifically, a stop-gradient operation is applied on the z t 2 and we implement it by modifying Eqn. 3 as D ( h t 1 , stopgrad ( z t 2 )) .
Explanation on optimization without negatives.
Unlike traditional contrastive learning objective Song et al.
; Kim et al.
(2021b) , we only involve positive pairs (e.g, ( x t 1 , x t 2 ) ) into the optimization without negative samples.
The insight Chen & He (2021) behind is that the negative samples in contrastive-learning play a role in preventing models from collapsing solutions (e.g, constant outputs).
Alternatively, it is found that applying the predictor H ( · ) on one view x t 1 and the stop-gradient operation stopgrad ( · ) on the other view x t 2 could also avoid model collapse.
As the existing contrastive video DA methods are largely constrained by the false negatives issue (Figure 1), we leverage this optimization to cut off the reliance on the negatives and thus avoid the error accumulation led by false negatives.
Target nearest neighbors as Intra-domain positives.
The empirical finding in Figure 1(b) shows that intra-domain NNs (target NNs) have higher purity than cross-domain NNs (source NNs) and pseudo-labels.
It motivates us to further explore the intra-domain NNs as intra-domain positives which capture rich spatial and temporal variations in target domain.
Following the notation of Eqn. 3, we add the detached feature z t 2 of the weakly-augmented view and its class pseudo label ˆ y t = arg max G ( z t 2 ) to a memory bank, denoted as M t = { ( z ti , ˆ y it ) | N t i =1 } in an online fashion where the size equals to the number of target dataset N t .
M t is updated with the newest features and pseudo labels given a current batch.
Then we find the k nearest neighbors of z t 2 in the memory bank to obtain k target features { ˆ z jt | kj =1 } .
Finally, we extend the Eqn.
3 to incorporate target NNs as intra-domain positives:
[Equation]
The final loss is the summation of the above loss for all target videos D t .
By pulling unlabeled target videos close to their NNs, the model could achieve the better target-domain discrimination.
Importantly, the Eqn. 4 is equivalent to the Eqn.
when k = 1 .
As k increases, the intra-domain positives could cover richer target-domain variations while under the risk of bringing false positives.
The trade-off between the "informativeness" and "robustness" of intra-domain positives is discussed in experiment section in Figure 4.
3. R OBUST C ROSS - DOMAIN P OSITIVES
While the intra-domain positive focuses on domain-wise discrimination, it ignores the cross-domain alignment in the shared action semantic space.
A natural solution is to extend Eqn. 4 by incorporating source domain positives which cover cross-domain spatial-temporal variations.
As target videos do not have labels, we resort to use the pseudo-labels for choosing source samples or directly using source NNs as positives.
However, as Figure 1 (b) suggests, both source NNs and pseudo labels have low purity.
Incorporating cross-domain positives based on such noisy supervisions would result in false domain alignment.
To tackle this issue, we formulate it reversely to utilize source videos as anchors and mine for more informative target domain positives.
Previous work Xie et al.
(2018) motivates us that, though the pseudo labels of target samples are noisy, the class-conditional centroids µ tc of target features would weaken this noise by averaging all the target features of same pseudo class .
Consequently, the target centroid of each class µ tc is more robust to pseudo-label noise.
To further explore target domain variations, we also estimate the corresponding covariance matrix Σ tc from the target features of the same pseudo-class.
Instead of directly mining real target videos as positives based on noisy NNs or PL, we randomly sample a synthesized target feature from a normal distribution N ( µ tc , Σ tc ) of the same action class c to the source anchor video in the form of
[Equation]
where α is a positive coefficient to control the strength of target data variations.
In the implementation, the target class centroids and the covariance matrix are computed based on the target features of the same pseudo class from the memory bank M t in Eqn. 4.
As the estimation in the first few epochs is not so accurate, we set α = ( t/T ) ×
α 0 where t and T are the current epoch and maximum epochs.
As a result, the negative impact from the estimated covariance in the early training stage will be reduced.
Similarly to Eqn. 4, our model takes a source video x s as input (anchor) and processes it with the feature extractor F .
Then we match it to the synthesized positives sampled from the class-conditional target distribution of class y s from Eqn.
5 in feature space.
As shown in Figure 2 (c), when the number of sampled positives is M , our cross-domain alignment loss can be formulated as:
[Equation]
where z s = F ( x s ) is the feature of the source anchor and
˜ z it is the target positive sampling from
N ( µ ty s , α Σ ty s ) .
We further derive a close-form upper-bound for the expectation of Eqn.
6 under all possible target features as L ∞ s − cross in appendix B.1.
Explanation on cross-domain optimization.
Unlike Eqn. 4, we do not transform the z s by the predictor H which is originally designed for avoiding model collapse.
Interestingly, we find that discarding the predictor H helps better convergence for cross-domain positives in optimization.
Our insight is that, as source feature z s and the synthesized target feature ˜
z t are from different distribution, pulling them close requires non-trivial optimization efforts and thus models are less prone to collapse.
In contrast, matching different views of a target video z t from the same distribution is relatively "easy".
Without regularization from the predictor H , models will easily match all the views (e.g, augmentations or intra-domain NNs) of the target video into a constant output.
Discussion on Xie et al.
It is worth noting some fundamental differences between our algorithm and Xie et al.
From the optimization view , they propose centroids alignment across domains where target centroids are updated with the current target batch based on pseudolabels and thus will receive gradients.
If the pseudo labels are noisy, wrong gradients will hurt the training.
In comparison, we align the cross-domain samples within constrastive framework where our synthesized target features are detached from gradients; thus, only the source batch will receive the robust gradients.
Algorithmically , besides the estimation of target centroids, we further integrate target covariance to estimate the whole gaussian distribution of target domain.
Table 2 shows that we have a significant improvement over Xie et al.
3. O VERALL O PTIMIZATION
In summary, the overall optimization includes the intra-domain positives in section 3.1 and the cross-domain positives in section 3.2 as follows:
[Table]
52.3 Supervised-target
I3D 62.
[Equation]
) where λ is a hyperparameter to control the strength of intra-domain alignment.
We empirically set the λ to 15 for the convergence of L t − intra due to the predictor.
Our framework can be easily extended to two-stream network by adding Eqn. 8 for optical flow modality as L flowall .
Also, the optical flow correspondence of target videos can be integrated as intra-domain positives in L t − intra (Eqn.15).
Our algorithm with pseudo code will be found in appendix Alg.1.
E XPERIMENT 4. D ATASETS
We evaluate the performance of our approach on two publicly benchmark datasets for video domain adaptation, namely UCF ↔ HMDB Chen et al.
(2019a)
and Epic-Kitchens Munro & Damen (2020).
We use the standard training and testing splits provided by the authors in Chen et al. (2019a); Munro & Damen (2020) to conduct our experiments on each dataset.
UCF ↔ HMDB is first assembled by Chen et al.
Chen et al. (2019a))
for studying video domain adaptation problem.
It is a subset of the original UCF Soomro et al.
and HMDB datasets Kuehne et al.
(2011), containing 3 , 209 videos across 12 overlapping classes for human activity recognition.
We show the performance of our method on two setting: UCF → HMDB and UCF ← HMDB.
EpicKitchens is a challenging egocentric fined-grained action recognition dataset that consists of videos across 8 largest action classes from three domains, namely D 1 , D 2 and D 3 , corresponding to P 08 , Pand P 22 kitchens on the full Epic-Kitchens dataset Damen et al.
It is released by the authors in Munro & Damen (2020) for exploring video domain adaptation on the fined-grained setting.
4. C OMPARISON WITH S TATE - OF - THE - ART M ETHODS
Baselines.
We compare our approach with the following baselines.
(1) source only and supervised target only baselines that trains the network using labeled source data and labeled target data respectively, (2) existing UDA methods based on adversarial learning, including DANN Ganin et al.
(2016)and ADDA Tzeng et al.
(2017), (3) existing UDA based on pseudo-labeling, including
CRST Zou et al. and MSTN Xie et al. (4) existing video domain adaptation methods, including TA 3 N Chen et al. (2019a), ABG Luo et al.
(2020), TCoN Pan et al.
(2020a)
, MM-SADA
Munro & Damen (2020), SAVA Choi et al.
STCDA Song et al.
(2021) and LCMC Kim et al.
(2021b).
Results on UCF-HMDB.
Table 2 shows results of our method with other competing approaches on
UCF-HMDB dataset.
Our framework achieves the best performance than previous state-of-the-art on different backbones.
Compared to the recent methods incorporating self-supervision such as clip-order prediction (SAVA Choi et al.
appearance-motion correspondence (MM-SADA
Munro & Damen (2020)) and contrastive learning (STCDA Song et al.
(2021) and LCMC Kim et al. (2021b))
, our method makes a significant improvement using I3D backbone with 1 . 8 % on
UCF → HMDB and 3 . 5 % on HMDB → UCF task respectively.
Though not specifically designed for motion space, our method can extend to two-stream backbone with additional gains to 86 .
1 % on
UCF → HMDB and 95 .
4 % on HMDB → UCF, achieving the state-of-the-art performance.
[Figure]
[Table]
Results on Epic-Kitchens.
Tableshows the results on Epic-Kitchens, which is another challenging dataset consisting of total 6 transfer tasks with a data imbalance across different action classes.
Our method achieves the best performance on all 6 tasks with the average performance of 52 .
Compared to adversarial alignment (AdaBN Li et al.
and MCD Saito et al.
(2018) ), self-correspondence learning (MM-SADA Munro & Damen (2020)) and contrastive learning methods (STCDA Song et al.
(2021) and
LCMC Kim et al.
(2021b)) , similar observations can be found that our method outperforms all the competing methods in the challenging EpicKitchens dataset by 1 . 1 % .
4. A BLATION S TUDY AND A NALYSIS Effectiveness of Proposed Positives.
As presented from Table 3, intradomain positives bring a significant improvement on UCF → HMDB by 2 . 6 % and HMDB → UCF by 5 . 0 % .
Cross-domain positives also improve the performance by a large margin.
Surprisingly, each proposed component individually outperforms the previous state-of-the-art methods (e.g.,
SAVA Choi et al.
(2020) and STCDA Song et al.
on the I3D backbone.
Incorporating these two components into the same framework brings further gains on both UCF → HMDB by 3 .
and
HMDB → UCF by 5 . 9 % .
These results suggest that intra/cross-domain positives could benefit each other.
Precisely, the better target domain discrimination brought by intra-domain positives helps the better estimation of target distribution in Eqn.5.
In return, cross-domain positives help cross-domain alignment and result in more accurate target NNs for intra-domain positives in Eqn. 4.
Last, the performance could be further boosted with our optimization without negatives.
Effect of Optimization w/o Negatives.
As shown in Table 3, optimization without negatives boosts the performance of both intra-domain positive, cross-domain positives and jointly training by a significant margin.
Explicitly, when jointly training, it improves 1 .
5 % on UCF → HMDB and 1 .
6 % on HMDB → UCF compared to the baseline of conventional contrastive optimization with negatives.
Effect of Augmentation Strategies.
When k = 1 , our Intra-Pos could generalize to self-supervised learning where the target anchor videos are pushed to be close to their augmentations.
Explicitly, we consider spatial augmentations (e.g., random cropping, color distortion, and temporal augmentations including video playrate change and clips shuffling.
Table 4 shows that all types of augmentations benefit the performance, and integrating all of them gives the best performance.
Interestingly,
UCF → HMDB favors the temporal augmentations more than the spatial augmentations while an opposite observation is found in HMDB → UCF.
[Table] [Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Effect of Class-conditional Covariance.
Our cross-domain alignment pulls a source video to the synthesized target video representations from the estimated target distribution.
We claim that the estimated class mean feature vector provides the shared action semantics from the target domain.
Its corresponding covariance brings target spatialtemporal variations to the source samples.
Table 4 shows a performance drop on Cross-domain without the covariance Σ tc on both tasks of UCF-HMDB, demonstrating the importance of integrating target variations.
Effect of Predictor in Cross-domain postives.
Unlike previous non-contrastive learning method
Chen & He (2021), we find that applying the predictor results in poor performance in Tab. 4.
We further tried increasing the optimization strength with larger loss ratio but still failed.
We hypothesize that, due to the domain shift, pulling the source features closer to the target requires non-trivial optimization effort and thus the model is less prone to the "collapsing solutions".
Therefore, adding a predictor would make the optimization even harder.
Hyperparameters Sensitivity.
Figure 4 presents the effect of choosing different neighbors k with λ = 15 .
We observe that adding neighbors in Eqn.4 generally boosts the performance and k =gives the best performance.
For λ , we set it λ from { 5 , 10 , 15 , 20 } with k = 5 .
Empirically, we observe that relatively large λ helps the convergence on the predictor in Figure 4, and λ = 15 achieves the best performance.
For α 0 , it can be implicitly considered as the radius of target class distribution.
It is shown that Σ tc becomes most effective when α 0 = 0 .
25 in Figure 4.
In this case, the sampled target variations from Σ tc are more likely to be clean.
In summary, our method is robust to different hyperparameters.
Feature Visualizations.
We visualize the feature embedding by t-SNE Maaten & Hinton (2008) among each component of proposed methods, including the source only model, intra-domain, crossdomain and ours.
As seen from Figure 3, intra-domain shows better class discrimination while cross-domain aligns two domains more tightly.
Ours incorporates the merits of both two methods.
C ONCLUSION AND F UTURE WORK
In this work, we introduce the bottlenecks of in existing contrastive-based video DA methods and propose a unified solution to address them without relying on negatives by mining informative and robust intra-domain positives and cross-domain positives.
Extensive experiments on several benchmark datasets demonstrate the superiority of our approach over state-of-the-art methods.
Regarding the future work, multimodal learning would be an interesting direction to be integrated with our algorithm by mining cross-modal nearest neighbors and estimating target distribution from other modalities.
E THICS S TATEMENT
In our paper we strictly follow the ICLR ethical research standards and laws.
All datasets we employed are publicly available and all related publications and source codes are cited appropriately.
R EPRODUCIBILITY S TATEMENT
We highlight several components within this submission that effectively support the reproducibility of this work.
1) Implementation details are provided in Appendix C, where the training dataset, backbone architecture, training schemes and hyperparameters are carefully stated.
A more detailed pseudo code for our algorithm can be found in Alg.
A A PPENDIX
In this material, we show the additional optimization detail, comparison to Imagge-based Domain adaptation methods and algorithm pseudo code to our main submission.
B O PTIMIZATION D ETAIL
In summary, our framework can be simply implemented and optimized in an end-to-end deep learning framework through the overall objective function:
[Equation]
B. C LOSE - FORM U PPER - BOUND FOR C ROSS -P OSITIVES O PTIMIZATION
The naive implementation is not computational efficient when M is large.
To tackle this issue, we consider the case that the number of M goes to infinity, and then derive a close-form upper-bound for the expectation of Eqn.
6 under all possible target features as:
[Equation]
For simplicity, all the z s , ˜
z it , µ ty s and Σ ty s above are l 2 -normalized if not coupled with D ( · , · ) .
The
Eqn.10 emerges from the Jensen inequality E log( X ) ≤ log E
[ X ] on the concave function log( · ) .
Based on the moment generation function E [ e aX ] =
e aµ + 12 a T Σ a where X ∼ N ( µ, Σ) , Eqn.11 can be obtained from Eqn.10 under Gaussian assumption.
The overall cross-domain alignment loss above is averaged for all source videos N s .
Notably, the computational cost of Eqn.
12 is different from backpropagating losses over M × N s source-target pairs.
As the N ( µ ty s , α Σ ty s ) is estimated offline based on the memory bank, Eqn.
12 can be computed efficiently without explicitly sampling target features.
B. E XTENSION TO TWO - STREAM NETWORK
For fair comparison to the methods that utilizing optical flow such as MM-SADA Munro & Damen (2020), STCDA Song et al.
(2021) and LCMC Kim et al.
(2021b), our method could be extended to two-stream network using both RGB and optical flow as inputs as well.
Following the setting above, we have:
[Equation]
Besides L rgball and L flowall , we also incorporate the cross-modality self-supervision in target domain as:
[Equation]
where z rgbt =
F r ( x rgbt ) , z flowt =
F f ( x flowt ) are the features of RGB and flow images.
Similar to the cross-domain positives case, we do not add predictor after features in optimization as cross-modality alignment in features space does not lead to the collapsed outputs.
To sum up, the overall objective function for two-stream network is shown as follows:
[Equation]
do
3: end do 5:
Generate target pseudo labels based p
Update memory bank M
= { features and pseudo labels in current
Randomly apply strong spatial or temporal augmentation on the target samples as the second view { ( ˆ i
B
Obtain their deep features and transform them as { h tj } Bj =1 by the predictor Θ H .
Given the current target features { z tj } Bj =1 , we mine the top-K nearest neighbors from M t as { ˆ z jt | kj =1 } and compute the intra-domain positives loss L t − intra .
13: for each class c do 14:
Estimate target features means µ ct according to memory module M t .
Estimate the target intra-class covariance matrix Σ ct according to memory module M t .
16: end for 17:
Based on N ( µ tc , α Σ tc ) , compute the cross-domain alignment loss L ∞ s − cross .
Update Θ F , Θ H by minimizing the loss L rgball in Eqn. 8 with stochastic gradient descent (SGD).
19: end for
C I MPLEMENTATION D ETAILS
Following Choi et al.
Song et al.
; Kim et al.
(2021b), we use I 3 D Carreira & Zisserman (2017b) as the backbone feature encoder network, initialized with Kinetics pre-trained weights.
The dimension of the features extracted from the I 3 D encoder is 1024 .
We randomly sampleconsecutive frames out of a video clip with a size of 224 × 224.
For inference, we use 16 uniformly sampled frames per video to recognize the action.
We follow the standard ‘pre-train then adapt’ procedure used in prior works Tzeng et al.
; Choi et al.
(2020) to train the model with only source data for 3 epochs as a warm start before the proposed approach is employed.
Then we train models in Eqn.
8 for 40 epochs in total.
For the hyperparameters, we select λ from { 5 , 10 , 15 , 20 } , α 0 from { 0 . 1 , 0 . 25 , 0 . 5 , 0 . 75 } and the number of neighbors k from { 1 , 2 , 5 , 10 , 20 } .
We conduct hyperparameter sensitivity experiments in sec.
We set k = 5 , λ = 15 and α 0 = 0 .
25 for all datasets.
We train all the models end-to-end using SGD with a momentum of 0 .
9 and a weight decay of 1 e- 7 .
We use an initial learning rate of 0 .
01 for the I 3 D with a cosine learning rate scheduler for our experiments.
We use a batch size of 32 equally split over the two domains.
We report the average action recognition accuracy over 3 random trials.
We use four 12G NVIDIA GPUs for training.
D C OMPARISON TO I MAGE - BASED D OMAIN ADAPTATION B ASELINE
We compare our approach with the following Image-based DA baselines.
(1) existing UDA based on advanced pseudo-labeling and entropy minimization methods
[Table]
We shows results of our method with other competing approaches on UCF-HMDB dataset in Table 5.
Our framework outperforms both inter-domain and intra-domain methods by a significant margin.