Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency
Abstract
Visual domain adaptation (DA) seeks to transfer trained models to unseen, unlabeled domains across distribution shift, but approaches typically focus on adapting convolutional neural network architectures initialized with supervised ImageNet representations.
In this work, we shift focus to adapting modern architectures for object recognition – the increasingly popular Vision Transformer (ViT) – initialized with modern pretraining based on self-supervised learning (SSL).
Inspired by the design of recent SSL approaches based on learning from partial image inputs generated via masking or cropping – either by learning to predict the missing pixels, or learning representational invariances to such augmentations – we propose PACMAC , a two-stage adaptation algorithm for self-supervised ViTs.
PACMAC ﬁrst performs in-domain SSL on pooled source and target data to learn task-discriminative features, and then probes the model’s predictive consistency across a set of partial target inputs generated via a novel attention-conditioned masking strategy, to identify reliable candidates for self-training.
Our simple approach leads to consistent performance gains over competing methods that use ViTs and self-supervised initializations on standard object recognition benchmarks.
Our code is available at https://github.com/virajprabhu/PACMAC .
Introduction
Deep models struggle to generalize to visual domains that deviate from the ones on which they were trained [1].
Unsupervised domain adaptation [2, 3, 4, 5, 6, 7, 8, 9] seeks to adapt models trained on labeled source domains to unseen and unlabeled target domains, but most existing approaches focus on adapting convolutional neural network (CNN) architectures initialized with supervised representations, typically from ImageNet [10].
In this work, we seek to “modernize” domain adaptation by focusing on adapting state-of-the-art architectures for object recognition – Vision Transformers
(ViTs) [11] – initialized with modern pretraining strategies based on self-supervised learning (SSL).
Vision Transformer (ViT)
[11] architectures have recently gained traction as an effective alternative to CNNs for computer vision tasks [12], achieving impressive performance despite fewer inductive biases.
With their in-built self-attention mechanisms and improved calibration under distribution shift over their CNN counterparts [13], ViTs may be particularly well-suited to domain adaptation [14, 15].
Similarly, self-supervised representation learning (SSL) is rapidly replacing supervised learning as the de-facto pretraining strategy for deep networks, due to improved scalability (unlabeled data is easier to collect) and generality (domain-speciﬁc SSL is often preferable to one-ﬁts-all
ImageNet pretraining [16, 17]).
Many successful methods that optimize proxy objectives based on pretext tasks [18, 19, 20], instance discrimination [21, 22, 23, 24], self-distillation [25], and image reconstruction [26], have been proposed.
Despite the increasing popularity of ViTs and SSL, to our knowledge no prior work has focused on adapting self-supervised ViTs.
Recent work has proposed additional self-supervised contrastive learning on the pooled source and target domain as a strong initialization for DA methods [27, 28], but focuses on adapting CNNs.
We follow these works to perform additional in-domain pretraining, ﬁnding that it leads to learning task-discriminative features with ViTs as well.
Interestingly, Shen et al.
[28] ﬁnd that such contrastive pretraining learns features that disentangle domain and class information and can generalize to the target without being domain invariant, subverting traditional wisdom in UDA.
We experimentally ﬁnd that this causes several off-the-shelf DA methods ( e.g. based on domain adversarial learning) to sometimes fail at adapting ViTs initialized with SSL representations, suggesting the need for specialized solutions.
Our work attempts to ﬁll this gap.
Concurrent work has also studied the problem of adapting ViTs [14, 15], but focuses on supervised rather than self-supervised initializations.
Crucially, none of these prior works propose an adaptation strategy speciﬁcally designed for adapting self-supervised initializations, whereas we explicitly incorporate components of the SSL pretraining in our adaptation algorithm for better transfer.
Concretely, we observe that several recent state-of-the-art SSL methods for ViTs focus on learning from partial inputs [22, 23, 26, 24, 25] generated via masking or cropping strategies.
In the context of vision transformers, such methods learn to either reconstruct the missing information [26, 29] or to be invariant to such cropping [25].
We ﬁnd that the predictive consistency of a model initialized with such representations across partial inputs generated via masking acts as an effective self-supervised reliability measure on the unlabeled target domain.
Further, rather than masking inputs randomly, we leverage the ViT attention mechanism to perform attention-conditioned masking : we generate a set of disjoint masks that only keep highly attended patches from the input image.
We then probe the reliability of a given target instance by measuring the model’s predictive consistency across such masked images and the original image, and mark target instances with high consistency as reliable.
We call our selection strategy Probing Attention-Conditioned Masking Consistency ( PACMAC ).
We further mark instances with high conﬁdence as reliable, and then selectively self-train the model against the current prediction on target instances identiﬁed as reliable via either scheme.
This leads to an easy to implement selective self-training adaptation algorithm that outperforms competing methods on standard benchmarks.
We make the following contributions:
• We propose an algorithm to adapt self-supervised ViTs to unseen domains that i) performs in-domain SSL on the pooled source and target data, and ii) self-trains the model on target instances identiﬁed to be reliable by probing the model’s predictive consistency across a set of attention-conditioned masks applied to the target image, in combination with conﬁdence.
• Our approach outperforms more complex DA methods from prior work at adapting selfsupervised ViTs on the OfﬁceHome [30], DomainNet [31], and VisDA [32] adaptation benchmarks for object recognition.
Related work
Unsupervised domain adaptation (UDA).
UDA seeks to transfer a model trained on a labeled source domain to an unlabeled target domain across distribution shift.
A variety of successful approaches based on aligning feature spaces via optimizing domain divergence measures [4, 3] and distribution matching via domain adversarial learning [6, 5, 7] (often incorporating additional pixel-level matching constraints), have been proposed.
Recently, selective self-training [33, 9] on the model’s predictions on target data has emerged as a simple and effective UDA technique.
With the upsurge in adoption of Vision Transformer (ViT [11]) architectures for object recognition, some recent works speciﬁcally focus on adapting ViTs [14, 15].
These methods leverage the ViT attention mechanism for incorporating patch-level transferability [14] and category-level feature alignment [15].
However, these works focus on adapting models initialized with supervised ImageNet weights.
In this work, we propose an adaptation algorithm designed for self-supervised ViTs.
Self-supervised learning.
Self-supervised learning (SSL) leverages unlabeled data to learn representations that can be transferred efﬁciently to downstream tasks.
Several approaches based on pretext tasks [18, 19, 20], masked-image modeling [26, 34], context prediction [35] and instance discrimination [22, 23], have been proposed to learn strong semantic representations in the absence of labeled data.
Some of the state-of-the-art SSL methods are designed for ViTs: DINO [25] performs self-distillation by training a student network that is given a strongly augmented image as input to match the output of a teacher network that sees a global view of the image.
MAE [26] is a masked image modeling (MIM) approach that learns to predict a complete image given only a fraction of the image patches as input.
Recently, ADIOS [36] improves MIM methods by learning an adversarial masking function to obscure salient regions of the image.
A concurrent work, MSN [29] improves upon DINO by passing in a masked version of an augmented image to the student network.
We derive inspiration from the local-to-global self-supervisory signal used by these methods in designing our adaptation algorithm, but instead of learning reconstruction or invariance to missing information, measure the model’s predictive conﬁdence under targeted missing information as a self-supervised probe to determine reliability.
Self-supervised learning for domain adaptation.
While most UDA methods focus on adapting models initialized with supervised initializations, typically from ImageNet, a few recent works have studied self-supervised domain adaptation [29, 27].
Kim et al.
[27] propose CDS, a pretraining strategy for domain adaptation that makes use of in-domain contrastive learning in conjunction with cross-domain matching as a superior initialization alternative to ImageNet pretraining.
Shen et al.
[28] study the transferability of representations learned via additional in-domain contrastive learning on the source and target domain, ﬁnding that such pretrained features can be transferred effectively to the target by simply ﬁnetuning on the source, despite not being domain invariant.
However, these methods restrict their experiments to ResNet [37] architectures while we work focus on adapting
ViTs.
Crucially, neither of these works propose a DA algorithm catered to the model’s self-supervised initialization, whereas we do so explicitly.
Approach 3.1 Notation
Let X and Y denote input and output spaces.
In unsupervised domain adaptation (UDA) we are given access to labeled source instances ( x S , y S ) ∼ P S ( X , Y ) , and unlabeled target instances x T ∼ P T ( X ) , where S and T correspond to source and target domains.
The goal is to learn a model f = h ( φ ( . ))
( φ denotes the encoder and h denotes classiﬁer), parameterized by Θ with minimum error on the target dataset.
In our experiments, Θ is parameterized as a Vision Transformer or
ViT [11]), which takes as input a linear embedding of of N image patches in addition to a class token embedding.
These N+1 embeddings are then appended with positional encodings and passed through several transformer layers, each of which comprises of a sequence of multi-headed self-attention (with M attention heads), multilayer perceptron, and layernorm modules.
Features from the ﬁnal encoder layer are typically fed to a classiﬁer layer and used to predict the output.
We consider adapting models trained for C -way image classiﬁcation: the inputs x are images, and labels y are categorical variables y ∈ { 1 , 2 , .., C } .
Let p Θ
( y | x ) denote the ﬁnal probabilistic output from the model.
For a target instance x T ∼ P T ( X ) , we compute a “pseudolabel” as ˆ y = argmax p Θ ( y | x T ) .
3.2 Preliminaries
Self-supervised learning (SSL) for ViTs.
We consider SSL methods for ViTs that learn from partial inputs, under two popular formulations: i) Masked Image Modeling, wherein the model is given a partial image and trained to predict the missing content (at a pixel or token level) via learning a masked or denoising autoencoder [26, 38, 39, 34].
ii) Joint-embedding networks, which learn a model that produces similar features for different views of a given image, across strong cropping and additional augmentation [25, 29].
Let m ( x T ) denote the partial image generated from target image x T under transformation m ( . ) which could correspond to either a masking or augmentation strategy.
Broadly speaking, SSL corresponds to minimizing an objective of the form L SSL ( φ ( m ( x T )) , φ ( x T )) , which could correspond to a reconstruction or invariance based objective operating on encoded features.
We now instantiate L SSL in the context of the the two SSL strategies we use in this paper: MAE [37] and DINO [25].
MAE [37].
MAE learns a visual transformer autoencoder to reconstruct images x T given only a random subset of patches from the original image m ( x T ) .
Let d denote a decoder learned for image reconstruction.
The MAE objective minimizes the mean square error between the original and reconstructed image in pixel space:
[Equation]
DINO [25].
For a target image x T , DINO passes two transformed versions of each target image m 1 ( x T ) and m 2 ( x T ) to a student network φ s and teacher network φ t respectively, and trains the student network to predict the output of the teacher by minimizing the cross-entropy H between the two output distributions:
[Equation]
Self-training for unsupervised DA.
Self-training for UDA typically involves training the model on its predictions on unlabeled target data (“pseudolabels”), optimizing either a conditional entropy minimization [40] or cross-entropy objective.
However, under a domain shift many of the model’s predictions may initially be incorrect [41, 42, 9] (especially under severe distribution shift), and unconstrained self-training may lead to amplifying model errors.
To combat this, a few recent works have proposed selective self-training on predictions that have a high likelihood of being correct.
Recent works propose to identify such reliable predictions either based on model conﬁdence [33, 43], predictive consistency under data augmentation [9, 44], or combinations of the two [45].
Let r ( x T ) denote a binary reliability value for target instance x T .
We optimize the following selective self-training objective on target data L SST :
[Equation] [Equation]
In this work, we propose an alternative selection criterion designed for self-supervised representations based on predictive consistency under attention-conditioned masking, which we now introduce.
3.3 PACMAC : Probing Attention-conditioned
Masking Consistency for UDA
Following prior work [28, 27], we ﬁrst learn task-discriminative features by optimizing the following in-domain self-supervised pretraining objective L IDP on pooled source and target data:
[Equation]
We then ﬁnetune the learned representations end-to-end on labeled source data:
[Equation]
Finally, we perform selective self-training.
For a target instance x T ∼ P T , we generate a committee of k masked versions.
However, applying random masks may lead to capturing irrelevant background features; instead, we propose attention-conditioned masking (see Algo. 1).
We ﬁrst obtain the model’s
1: Input: x T , per-patch attention ˆ a T , masking ratio mr , committee size k 2: L ← (1 − mr ) × len( ˆ a T ) 3: ˆ S T ← argsort( a T ) (cid:46)
4: for i ← k do M
(cid:46) 5: i ←6: while i < L do 7: i ← 8: for j do 9: p ← ˆ S T . _last
) (cid:46) next most attended index 10:
M pj ←(cid:46)
Greedy round-robin assignment 11: for j do m
= M
12: Return Consistent or Conﬁdent 10:
[Equation]
←
[Equation]
Minimize L CE ( x S , y S )
+ α L SST per-patch attention score ˆ a T over the original image (in practice, we obtain the patchwise selfattention from the last encoder layer of the class token with respect to all image patches for each attention head, and average across the M heads).
We then initialize all k masks with zeros (hide all patches).
Next, we sort patches in descending order of attention scores and perform a greedy round-robin assignment to the k masks to only unhide highly attended patches, until the desired masking ratio is satisﬁed.
We then apply these masks to the original image to generate masked images { m 1 ( x T ) , m 2 ( x T ) , ..., m k ( x T ) } .
We make predictions for each of these k masked versions (following prior work [37] we only feed in unmasked patches), and measure consistency between the model’s prediction for the original image and for each of the k masked versions.
If the model’s prediction for all masked versions matches its prediction on the original image, we consider the instance as “reliable”, and as “unreliable” if not.
Further, we also consider instances as reliable if the model’s predictive conﬁdence on the original image is higher than a threshold T .
Let 1 [ . ] denote an indicator function.
Formally, we assign target instance reliability r ( x T ) as:
[Equation]
[Equation]
Algorithm 2 lists the steps involved in computing the PACMAC objective.
Without loss of generality, we minimize the cross-entropy between the model’s prediction on the last consistent masked image m k
( x T ) and its consistent pseudolabel.
We empirically ﬁnd this to act as an effective data augmentation strategy and provide a stronger learning signal than when using the model’s prediction on the original unmasked image (see Sec. 4.5).
We additionally optimize a cross-entropy loss over labeled source examples.
For loss weight α , the full L PACMAC objective is:
[Equation]
[Table]
Experiments
We ﬁrst describe our datasets and metrics (Sec. 4.1), implementation details (Sec. 4.2)
[Table]
and baselines (Sec. 4.3).
Next, we present results (Sec. 4.4), method ablations (Sec. 4.5) [Table] and analysis (Sec. 4.6).
4.1 Datasets and metrics
We evaluate PACMAC on three classiﬁcation benchmarks for domain adaptation: i) OfﬁceHome [30] is a classiﬁcation-based benchmark comprising of 12 shifts spanning 65 categories of objects found in home and ofﬁce environments.
It consists of 4 domains: Real-world ( Rw ), Clipart ( Cl ), Product ( Pr ) [Table] and Art ( Ar ).
ii) DomainNet [31] is a large benchmark for adapting object recognition models.
Matching prior work [28, 9], we use the subset of DomainNet proposed in Tan et al .
[33] for our experiments, which reports performance over 12 shifts comprising 40 common classes fromdomains: Real ( R ), Clipart ( C ), Painting ( P ), and Sketch ( S ).
iii)
VisDA2017 [32] is a large image classiﬁcation benchmark for synthetic → real adaptation with >200k images from 12 classes.
Metric.
Matching prior work
we report standard accuracy on the target test set as our metric.
4.2 Implementation details
We use a ViT-base [11] architecture with 16x16 image patches.
We use ofﬁcial codebases for
MAE [26] and DINO [25] and initialize with checkpoints pretrained on ImageNet1K [10].
We pretrain on the combined source and target domain for 800 epochs (MAE) and 200 epochs (DINO).
For pretraining, we linearly scale the learning rate to 4 × 10 − 4 (MAE) and 5 × 10 − 5 (DINO) during aepoch warmup phase followed by a cosine decay.
We use the AdamW [46] optimizer.
For PACMAC , we use k = 2 , mr = 0 .
75 , T = 50% , and α = 0 .
We use RandAugment [47] with N = 3 and M = 4 .during pretraining and N = 1 and M = 2 . 0 during DA.
On OfﬁceHome and DomainNet, we ﬁnetune on the source and adapt for 100 epochs each, and perform 10 epochs of each phase on VisDA.
We use a learning rate of 2 × 10 − 4 and weight decay of 0 . 05 .
All experiments use PyTorch [48].
4.3 Baselines and SSL strategies
DA baselines: Lacking baselines designed for adapting self-supervised ViTs, we compare against diverse DA methods proposed for CNNs with supervised initializations: based on domain adversarial learning (CDAN [7]), minimizing classiﬁer confusion (MCC [49]), self-training (SENTRY [45]), and contrastive learning (Shen et al.
We also benchmark a concurrent method for adapting ViTs (TVT [14]) with supervised initializations.
1) CDAN [7]: CDAN improves upon domain adversarial learning with multilinear conditioning, by capturing cross-covariance between feature representations and classiﬁer predictions to improve discriminability.
Recent work [50] ﬁnds CDAN to outperform more recent DA methods when combined with new architectures like ViTs. 2) MCC [49]:
Minimum
Classiﬁer Confusion is a non-adversarial DA method that aligns domains by minimizing pairwise class confusion on the target domain estimated from model predictions.
3) SENTRY [9] : SENTRY measures model predictive consistency across randomly augmented versions of each target image and selectively minimizes entropy to increase predictive conﬁdence on highly consistent instances,
[Table]
Shen et al
[Table]
[Table]
[Table] [Table] [Table] SENTRY [9]
[Table]
PACMAC
[Table]
while maximizing it to decrease conﬁdence on highly inconsistent ones.
Shen et al.
[28] : Proposes contrastive learning on the pooled source and target domain followed by ﬁnetuning on source labels.
We adapt their approach by using SSL strategies designed for ViTs (described below), and by starting from ImageNet SSL weights rather than from scratch.
5) TVT [14] : Transferable Vision Transformer (TVT) injects a learned transferability measure into the transformer attention blocks, in addition to performing global domain-adversarial alignment and discriminative clustering.
SSL Strategies: As discussed in Sec. 3, PACMAC is designed as an adaptation strategy for selfsupervised representations.
We report benchmark performance on top of two popular SSL strategies for ViTs (see supp.
for more details): 1) MAE [26] : Masked Autoencoding (MAE) is a selfsupervised learning strategy that learns a visual transformer autoencoder to reconstruct images given only a random subset of patches from the original image.
By masking out large portions of images ( ∼ 75%), the MAE encoder is shown to learn strong representations that can be effectively ﬁnetuned to downstream tasks.
2) DINO [25] : Self-Distillation with No Labels is a visual selfsupervised learning strategy that passes two transformed versions of each image to a student and teacher network respectively, and trains the student network to predict the output of the teacher.
In supp.
for completeness we also benchmark PACMAC on top of a supervised ImageNet initialization.
4.4 Results
[Table]
Tables1, 2, and 3, present results.
We observe:
(cid:46)
Several existing DA methods underperform on top of SSL representations.
On both OfﬁceHome and DomainNet, we observe existing DA methods (CDAN [7], MCC [49], TVT [14]) to frequently underperform even the source model on average, despite hyperparameter tuning.
The fact that many existing DA methods struggle with such initializations suggests the need for specialized solutions, particularly as SSL pretraining becomes more common.
SENTRY [45], Shen et al.
[28], and PACMAC however offer consistent improvements.
(cid:46) PACMAC outperforms prior work across benchmarks and initializations.
On OfﬁceHome ( shift average), PACMAC improves over the next-best method by 1.3% (MAE init.) and 0.2% (DINO).
On DomainNet, we observe gains of 1.1% (MAE) and 0.4% (DINO), and 1.7% (MAE) on VisDA.
We note here that PACMAC and SENTRY [9] both make use of selective self-training on reliable instances identiﬁed via predictive consistency, but differ in important ways: i) PACMAC roughly matches the design of its SSL pretraining and measures consistency across partial masked inputs rather than random augmented images, ii) PACMAC incorporates model knowledge in its selection strategy by using attention-conditioning to focus on salient image regions, rather than random augmentations sampled from a manually pre-deﬁned set.
These differences lead to its improved performance, despite being considerably simpler (2 losses, Eq. 7, and no diversity regularizers or entropy maximization).
4.5 Ablating PACMAC
In Tables 4- 5 we ablate PACMAC with a DINO initialization on OfﬁceHome Cl
→ Pr.
We observe:
[Table]
(cid:46)
In-domain pretraining helps.(4a)
We report two metrics: cross-domain k-Nearest Neighbor (k=7) accuracy on the target domain using source domain embeddings from the trained encoder (kNN column), and target accuracy after ﬁnetuning on the source (FT column).
Across MAE and DINO initializations, we observe additional in-domain pretraining on the pooled source and target domain to improve ﬁnetuning performance (Row 2 v/s 1, +6.2% FT acc.
with MAE, +2.7% with DINO).
With
MAE, we additionally try pretraining only on source or target domains and ﬁnd them to underperform
S+T pretraining (not in table).
We also try S+T pretraining with MAE by learning separate decoders to reconstruct source and target images, and ﬁnd that to underperform no pretraining (Row 3 v/s 1, -2.3% ).
Interestingly , we ﬁnd pretraining to improve kNN accuracy in all cases except Row 3 with MAE, indicating better domain alignment in encoder feature space.
With DINO, we observe higher kNN accuracies than MAE, but see a slight drop after in-domain pretraining.
(cid:46) PACMAC beneﬁts from S+T pretraining (4b)
We apply PACMAC to ImageNet SSL features without S+T pretraining and observe worse accuracy (Row 3 v/s 2, -2.4% with MAE, -2.7% with DINO). (cid:46)
Combining masking consistency and conﬁdence is an effective selection measure.
In Table 4c, we ﬁrst self-train on all (Row 1) or only conﬁdent (model conﬁdence > 50%, Row 2) target examples.
We ﬁnd both to underperform selection based on our proposed attention conditioned masking
con- sistency scheme (Row 3, +11.5% and +
We further combine our consistency measure with conﬁdence and ﬁnd selecting instances marked as reliable by atleast one measure to perform best (Row 5). In Row 6, as
we report upper-bound performance of an oracle method which only self-trains on correct pseudolabels,
and
ﬁnd it to achieve a high accuracy of 94%.
We also try self-training on the original rather than masked image (
not in
table) and ﬁnd it to underperform by 3.2%.
However, self training
on the
masked image without using it for selection does signiﬁcantly worse ( 7% ): clearly, the primarly beneﬁt of masking is due to better selection rather than regularization
(cid:46)
Matching selection strategy to DINO [25].
So far, for simplicity we match MAE’s
[26] design for PACMAC ’s selection and measure consistency across masked images.
To demonstrate generality, we now match DINO’s design and instead measure predictive consistency across local-and-global image crops (both random and with attention-seeding).
We ﬁnd that for a DINO initialization this further improves accuracy ( 74.3 % on Cl → Pr), suggesting that closely matching the design of the selection criterion to the SSL pretraining is beneﬁcial (details and visualization in supp.). (cid:46)
Comparing selection strategy to SENTRY [9] .
We swap out PACMAC ’s attention-conditioned masking consistency scheme with the committee consistency across augmentation scheme used by the next best performing method, SENTRY [9].
We match original hyperparameters and use a committee size of 3, RandAugment [47] parameters of N=3 and M=2.0, and majority voting.
Averaged overOfﬁceHome shifts, we ﬁnd this to obtain a transfer accuracy of 66.1% (MAE) and 67.4% (DINO), while PACMAC obtains 66.8% (MAE) and 69.6% (DINO).
This clearly establishes that PACMAC ’s gains over SENTRY are due to improved selection rather than a stronger initialization.
(cid:46) Attention-conditioning helps, as does committee consistency.
In Table 5a, we vary the committee size k and masking strategy (random v/s attention-conditioned masking).
As seen, attention conditioning consistently improves upon random masking across k ( +0.5 to 1.8% ). (cid:46)
Ablating masking ratio and conﬁdence threshold.
In Tables 5b and 5c, we vary the masking ratio mr and conﬁdence threshold T , and ﬁnd mr =75% and T =50% to work best.
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
Classiﬁer
[Figure]
[Table]
[Figure]
PT
[Figure]
[Table]
[Figure]
[Table]
DINO 10.
Sup.
[Table]
4.6 Analyzing PACMAC
Evaluating reliability estimation.
In Fig.
2, we evaluate our proposed reliability estimation scheme across MAE and DINO initializations.
We observe a high precision (70-80% across both) across epochs, indicating that our method correctly identiﬁes reliable instances with a low false-positive rate (per-class analysis in supp.).
We also ﬁnd that the percentage of target instances selected for self-training increases over time, and is particularly high with DINO pretraining.
Variance across runs.
To measure the performance variance of our method PACMAC , we run it across 5 random seeds on the OfﬁceHome Clipart → Product shift and observe a target accuracy mean of 73.1% with a standard deviation of 0.59%.
Understanding SSL initialization In Tab. 6, we contrast self-supervised (MAE and DINO) and supervised ImageNet initialization, by reporting the error of linear classiﬁers [28] (averaged over all
OfﬁceHome shifts) trained to distinguish different sets of class token features, such as: C2.
Different domains.
We observe higher error for the supervised init, indicating better domain alignment.
C3.
Same class but different domains.
Low error with DINO indicates less per-class domain alignment.
C4.
Different classes but same domain.
High error with MAE indicates more inter-class confusion.
C5.
Different domains and classes.
C6.
We deﬁne a domain alignment (DA) score = C3-max(C4, C5) – higher means that cross-domain examples from a different domain but same class are on average closer than examples from different classes and either the same or different domains.
Intuitively, a high DA score will translate to better performance for CDAN, which aligns logits-conditioned source and target features 2 .
We ﬁnd that SSL inits indeed have lower DA scores, possibly explaining why methods like CDAN struggle.
Visualizing attention-conditioned masking consistency.
In Fig.
3, we visualize our proposed attention-conditioned masking consistency scheme for random target images from OfﬁceHome.
As seen, attention-conditioning ensures that the masked images focus on disjoint, highly attended regions of the target image (Row 2).
Columns 1-5 illustrate examples for which our selection strategy is correctly able to identify reliable (Cols 1-2) and unreliable instances (Cols 3-5).
Cols 6-7 denote
O r i g i n a l
M a s k e d M a s k e d
A tt e n t i o n c a nd l e c a nd l e c a nd l e b a tt e r i e s b a tt e r i e s b a tt e r
i e s p
e n c il s p oo n s p oo n n o t e b oo k n o t e b oo k p o s t i t
s p e n c il c a l e nd a r r u l e r f il e c a b i n e t
o v e
n c o m pu t
e r
GT candle paper clip eraser fridge post-its pen f l o w e r s f l o w e r s f l o w e r s flowers
[Figure]
failure cases, with the ﬁrst showing a false negative (correctly classiﬁed instance being misidentiﬁed as unreliable) and the second denoting a false positive.
Effect of in-domain pretraining on per-class kNN accuracy.
In supplementary, we report per-class cross-domain kNN accuracy on the OfﬁceHome Cl
→
Pr shift before and after in-domain pretraining across MAE and DINO initializations.
We ﬁnd that accuracy improves on several classes in both cases, particularly with MAE.
While prior work has observed beneﬁts from additional in-domain pretraining for CNNs [28, 27], we corroborate this ﬁnding with self-supervised ViTs.
In the appendix, we analyze the effect of in-domain pretraining on out-of-distribution conﬁdence calibration (trends vary across shifts) , and include t-SNE [51] visualizations of the feature space learned by encoders before and after in-domain pretraining.
Limitations
PACMAC requires additional i) in-domain pretraining and ii) forward passes over masked target images (but no additional backpropagation), which makes it slower.
Further, PACMAC ’s effectiveness at identifying reliable instances varies across categories (see supp.).
In our paper we only experiment with ViTs trained for image classiﬁcation, and the generality of our approach across architectures and tasks is not established.
Finally, the performance of DA methods with SSL initializations still lags behind supervised initializations on standard benchmarks like OfﬁceHome and DomainNet (comparison in supp.).
We hypothesize that this may be due to strong category overlap between
ImageNet and these benchmarks and may not translate to other datasets, but this requires further study.
Conclusion
We focus on adapting modern vision architectures (ViTs) initialized with state-of-the-art pretraining (self-supervised learning or SSL).
Our method PACMAC ﬁrst performs SSL on source and target data, and then uses predictive consistency across partial versions of target images generated via an attentionconditioned masking strategy to judge reliability for selective self-training.
Despite its simplicity,
PACMAC outperforms prior work at adapting self-supervised ViTs trained for object recognition.
Acknowledgements.
This work was supported in part by funding from the DARPA LwLL project and ARL.
Checklist
For all authors.. (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[N/A] See Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec.
4.2 and supplementary.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec 4.1.
(b) Did you mention the license of the assets?
[Yes] See supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] See supplementary.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] See supplementary.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]