Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency
Abstract
Visual domain adaptation (DA) seeks to transfer trained models to unseen, unlabeled domains across distribution shift, but approaches typically focus on adapting convolutional neural network architectures initialized with supervised ImageNet representations.
In this work, we shift focus to adapting modern architectures for object recognition – the increasingly popular Vision Transformer (ViT) – initialized with modern pretraining based on self-supervised learning (SSL).
Inspired by the design of recent SSL approaches based on learning from partial image inputs generated via masking or cropping – either by learning to predict the missing pixels, or learning representational invariances to such augmentations – we propose PACMAC , a two-stage adaptation algorithm for self-supervised ViTs.
PACMAC first performs in-domain SSL on pooled source and target data to learn task-discriminative features, and then probes the model’s predictive consistency across a set of partial target inputs generated via a novel attention-conditioned masking strategy, to identify reliable candidates for self-training.
Our simple approach leads to consistent performance gains over competing methods that use ViTs and self-supervised initializations on standard object recognition benchmarks. Our code is available at https://github.com/virajprabhu/PACMAC .
Introduction
Deep models struggle to generalize to visual domains that deviate from the ones on which they were trained [1].
Unsupervised domain adaptation [2, 3, 4, 5, 6, 7, 8, 9] seeks to adapt models trained on labeled source domains to unseen and unlabeled target domains, but most existing approaches focus on adapting convolutional neural network (CNN) architectures initialized with supervised representations, typically from ImageNet [10].
In this work, we seek to “modernize” domain adaptation by focusing on adapting state-of-the-art architectures for object recognition – Vision Transformers ( ViTs) [11] – initialized with modern pretraining strategies based on self-supervised learning (SSL).
Vision Transformer (ViT)
[11] architectures have recently gained traction as an effective alternative to CNNs for computer vision tasks [12], achieving impressive performance despite fewer inductive biases.
With their in-built self-attention mechanisms and improved calibration under distribution shift over their CNN counterparts [13], ViTs may be particularly well-suited to domain adaptation [14, 15].
Similarly, self-supervised representation learning (SSL) is rapidly replacing supervised learning as the de-facto pretraining strategy for deep networks, due to improved scalability (unlabeled data is easier to collect) and generality (domain-specific SSL is often preferable to one-fits-all
ImageNet pretraining [16, 17]).
Many successful methods that optimize proxy objectives based on pretext tasks [18, 19, 20], instance discrimination [21, 22, 23, 24], self-distillation [25], and image reconstruction [26], have been proposed.
Despite the increasing popularity of ViTs and SSL, to our knowledge no prior work has focused on adapting self-supervised ViTs.
Recent work has proposed additional self-supervised contrastive learning on the pooled source and target domain as a strong initialization for DA methods [27, 28], but focuses on adapting CNNs.
We follow these works to perform additional in-domain pretraining, finding that it leads to learning task-discriminative features with ViTs as well.
Interestingly, Shen et al.
[28] find that such contrastive pretraining learns features that disentangle domain and class information and can generalize to the target without being domain invariant, subverting traditional wisdom in UDA.
We experimentally find that this causes several off-the-shelf DA methods ( e.g. based on domain adversarial learning) to sometimes fail at adapting ViTs initialized with SSL representations, suggesting the need for specialized solutions.
Our work attempts to fill this gap.
Concurrent work has also studied the problem of adapting ViTs [14, 15], but focuses on supervised rather than self-supervised initializations.
Crucially, none of these prior works propose an adaptation strategy specifically designed for adapting self-supervised initializations, whereas we explicitly incorporate components of the SSL pretraining in our adaptation algorithm for better transfer.
Concretely, we observe that several recent state-of-the-art SSL methods for ViTs focus on learning from partial inputs [22, 23, 26, 24, 25] generated via masking or cropping strategies.
In the context of vision transformers, such methods learn to either reconstruct the missing information [26, 29] or to be invariant to such cropping [25].
We find that the predictive consistency of a model initialized with such representations across partial inputs generated via masking acts as an effective self-supervised reliability measure on the unlabeled target domain.
Further, rather than masking inputs randomly, we leverage the ViT attention mechanism to perform attention-conditioned masking : we generate a set of disjoint masks that only keep highly attended patches from the input image.
We then probe the reliability of a given target instance by measuring the model’s predictive consistency across such masked images and the original image, and mark target instances with high consistency as reliable.
We call our selection strategy Probing Attention-Conditioned Masking Consistency ( PACMAC ).
We further mark instances with high confidence as reliable, and then selectively self-train the model against the current prediction on target instances identified as reliable via either scheme.
This leads to an easy to implement selective self-training adaptation algorithm that outperforms competing methods on standard benchmarks.
We make the following contributions:
• We propose an algorithm to adapt self-supervised ViTs to unseen domains that i) performs in-domain SSL on the pooled source and target data, and ii) self-trains the model on target instances identified to be reliable by probing the model’s predictive consistency across a set of attention-conditioned masks applied to the target image, in combination with confidence. • Our approach outperforms more complex DA methods from prior work at adapting selfsupervised ViTs on the OfficeHome [30], DomainNet [31], and VisDA [32] adaptation benchmarks for object recognition.
Related work
Unsupervised domain adaptation (UDA).
UDA seeks to transfer a model trained on a labeled source domain to an unlabeled target domain across distribution shift.
A variety of successful approaches based on aligning feature spaces via optimizing domain divergence measures [4, 3] and distribution matching via domain adversarial learning [6, 5, 7] (often incorporating additional pixel-level matching constraints), have been proposed.
Recently, selective self-training [33, 9] on the model’s predictions on target data has emerged as a simple and effective UDA technique.
With the upsurge in adoption of Vision Transformer (ViT [11]) architectures for object recognition, some recent works specifically focus on adapting ViTs [14, 15].
These methods leverage the ViT attention mechanism for incorporating patch-level transferability [14] and category-level feature alignment [15].
However, these works focus on adapting models initialized with supervised ImageNet weights.
In this work, we propose an adaptation algorithm designed for self-supervised ViTs.
Self-supervised learning.
Self-supervised learning (SSL) leverages unlabeled data to learn representations that can be transferred efficiently to downstream tasks.
Several approaches based on pretext tasks [18, 19, 20], masked-image modeling [26, 34], context prediction [35] and instance discrimination [22, 23], have been proposed to learn strong semantic representations in the absence of labeled data.
Some of the state-of-the-art SSL methods are designed for ViTs: DINO [25] performs self-distillation by training a student network that is given a strongly augmented image as input to match the output of a teacher network that sees a global view of the image.
MAE [26] is a masked image modeling (MIM) approach that learns to predict a complete image given only a fraction of the image patches as input.
Recently, ADIOS [36] improves MIM methods by learning an adversarial masking function to obscure salient regions of the image.
A concurrent work, MSN [29] improves upon DINO by passing in a masked version of an augmented image to the student network.
We derive inspiration from the local-to-global self-supervisory signal used by these methods in designing our adaptation algorithm, but instead of learning reconstruction or invariance to missing information, measure the model’s predictive confidence under targeted missing information as a self-supervised probe to determine reliability.
Self-supervised learning for domain adaptation.
While most UDA methods focus on adapting models initialized with supervised initializations, typically from ImageNet, a few recent works have studied self-supervised domain adaptation [29, 27].
Kim et al.
[27] propose CDS, a pretraining strategy for domain adaptation that makes use of in-domain contrastive learning in conjunction with cross-domain matching as a superior initialization alternative to ImageNet pretraining.
Shen et al.
[28] study the transferability of representations learned via additional in-domain contrastive learning on the source and target domain, finding that such pretrained features can be transferred effectively to the target by simply finetuning on the source, despite not being domain invariant.
However, these methods restrict their experiments to ResNet [37] architectures while we work focus on adapting
ViTs.
Crucially, neither of these works propose a DA algorithm catered to the model’s self-supervised initialization, whereas we do so explicitly.
Approach 3.1 Notation
Let X and Y denote input and output spaces.
In unsupervised domain adaptation (UDA) we are given access to labeled source instances ( x S , y S ) ∼ P S ( X , Y ) , and unlabeled target instances x T ∼ P T ( X ) , where S and T correspond to source and target domains.
The goal is to learn a model f = h ( ϕ ( . ))
( ϕ denotes the encoder and h denotes classifier), parameterized by Θ with minimum error on the target dataset.
In our experiments, Θ is parameterized as a Vision Transformer or
ViT [11]), which takes as input a linear embedding of of N image patches in addition to a class token embedding.
These N+1 embeddings are then appended with positional encodings and passed through several transformer layers, each of which comprises of a sequence of multi-headed self-attention (with M attention heads), multilayer perceptron, and layernorm modules.
Features from the final encoder layer are typically fed to a classifier layer and used to predict the output.
We consider adapting models trained for C -way image classification: the inputs x are images, and labels y are categorical variables y ∈ { 1 , 2 , .., C } .
Let p Θ
( y | x ) denote the final probabilistic output from the model.
For a target instance x T ∼ P T ( X ) , we compute a “pseudolabel” as ˆ y = argmax p Θ ( y | x T ) .
3.2 Preliminaries
Self-supervised learning (SSL) for ViTs.
We consider SSL methods for ViTs that learn from partial inputs, under two popular formulations: i) Masked Image Modeling, wherein the model is given a partial image and trained to predict the missing content (at a pixel or token level) via learning a masked or denoising autoencoder [26, 38, 39, 34].
ii) Joint-embedding networks, which learn a model that produces similar features for different views of a given image, across strong cropping and additional augmentation [25, 29].
Let m ( x T ) denote the partial image generated from target image x T under transformation m ( . ) which could correspond to either a masking or augmentation strategy.
Broadly speaking, SSL corresponds to minimizing an objective of the form L SSL ( ϕ ( m ( x T )) , ϕ ( x T )) , which could correspond to a reconstruction or invariance based objective operating on encoded features. We now instantiate L SSL in the context of the the two SSL strategies we use in this paper: MAE [37] and DINO [25]. MAE [37].
MAE learns a visual transformer autoencoder to reconstruct images x T given only a random subset of patches from the original image m ( x T ) .
Let d denote a decoder learned for image reconstruction.
The MAE objective minimizes the mean square error between the original and reconstructed image in pixel space: [Equation] DINO [25]. For a target image x T , DINO passes two transformed versions of each target image m 1 ( x T ) and m 2 ( x T ) to a student network ϕ s and teacher network ϕ t respectively, and trains the student network to predict the output of the teacher by minimizing the cross-entropy H between the two output distributions:
[Equation] Self-training for unsupervised DA. Self-training for UDA typically involves training the model on its predictions on unlabeled target data (“pseudolabels”), optimizing either a conditional entropy minimization [40] or cross-entropy objective.
However, under a domain shift many of the model’s predictions may initially be incorrect [41, 42, 9] (especially under severe distribution shift), and unconstrained self-training may lead to amplifying model errors.
To combat this, a few recent works have proposed selective self-training on predictions that have a high likelihood of being correct.
Recent works propose to identify such reliable predictions either based on model confidence [33, 43], predictive consistency under data augmentation [9, 44], or combinations of the two [45].
Let r ( x T ) denote a binary reliability value for target instance x T .
We optimize the following selective self-training objective on target data L SST :
[Equation] [Equation]
In this work, we propose an alternative selection criterion designed for self-supervised representations based on predictive consistency under attention-conditioned masking, which we now introduce.
3.3 PACMAC : Probing Attention-conditioned Masking Consistency
for UDA
Following prior work [28, 27], we first learn task-discriminative features by optimizing the following in-domain self-supervised pretraining objective L IDP on pooled source and target data:
[Equation]
We then finetune the learned representations end-to-end on labeled source data:
[Equation]
Finally, we perform selective self-training.
For a target instance x T ∼ P T , we generate a committee of k masked versions.
However, applying random masks may lead to capturing irrelevant background features; instead, we propose attention-conditioned masking (see Algo. 1).
We first obtain the model’s
1: Input: x T , per-patch attention ˆ a T , masking ratio mr , committee size k 2: L ← (1 − mr ) × len( ˆ a T ) 3: ˆ S T ← argsort( ˆ a T )
▷ Sort patch-wise attention in ascending order 4: for i ← 1 to k
do M
i ← zeros_like
(ˆ a ) ▷ Initialize masks, 0 is hide, 1 is keep 5: i ←6: while i < L do 7: i ← i +8: for j ← 1 to k do 9: p ← ˆ S T .pop_last
() ▷ Pop next-most attended patch index 10:
M pj ←▷
Greedy round-robin assignment 11: for j ← 1 to k do m j ( x T ) =
M j ⊙ x
T 12: Return { m 1 ( x T ) , . . .
, m k ( x T ) }
1: Input: SrcLoader, TgtLoader, model parameters Θ , masking ratio mr , committee size k , confidence threshold T , loss weight α 2: for epoch ← 1 to MAX_EPOCH do 3: for x S , y S in SrcLoader and x T in TgtLoader do 4: ˆ p T , ˆ y T ← max p Θ ( y | x T )
▷ Get top-1 confidence and prediction on original image 5: ˆ a T ← att ( p Θ ( y | x T ))
▷ Get per-patch attention 6: { m 1 ( x T ) , . . .
, m k ( x T ) } ← AttentionConditionedMasking
( x T , ˆ a T , mr , k ) 7:
C ← { m i ( x T ) | ˆ y T = argmax p Θ ( y | m i ( x T )) } ki = 8:
L SST
=9: if len (C) =
k OR ˆ p T > T then ▷ Consistent or Confident 10:
L SST ← L CE ( p Θ ( y | m k ( x T )) , ˆ y T ) 11:
Minimize L CE ( x S , y S ) + α L SST per-patch attention score ˆ a T over the original image (in practice, we obtain the patchwise selfattention from the last encoder layer of the class token with respect to all image patches for each attention head, and average across the M heads). We then initialize all k masks with zeros (hide all patches). Next, we sort patches in descending order of attention scores and perform a greedy round-robin assignment to the k masks to only unhide highly attended patches, until the desired masking ratio is satisfied.
We then apply these masks to the original image to generate masked images { m 1 ( x T ) , m 2 ( x T ) , ..., m k ( x T ) } . We make predictions for each of these k masked versions (following prior work [37] we only feed in unmasked patches), and measure consistency between the model’s prediction for the original image and for each of the k masked versions.
If the model’s prediction for all masked versions matches its prediction on the original image, we consider the instance as “reliable”, and as “unreliable” if not. Further, we also consider instances as reliable if the model’s predictive confidence on the original image is higher than a threshold T . Let 1 [ . ] denote an indicator function. Formally, we assign target instance reliability r ( x T ) as:
[Equation]
Algorithm 2 lists the steps involved in computing the PACMAC objective.
Without loss of generality, we minimize the cross-entropy between the model’s prediction on the last consistent masked image m k
( x T ) and its consistent pseudolabel.
We empirically find this to act as an effective data augmentation strategy and provide a stronger learning signal than when using the model’s prediction on the original unmasked image (see Sec. 4.5).
We additionally optimize a cross-entropy loss over labeled source examples.
For loss weight α , the full L PACMAC objective is:
[Equation]
[Table]
Experiments
We first describe our datasets and metrics (Sec. 4.1), implementation details (Sec. 4.2) [Table] and baselines (Sec. 4.3).
Next, we present results (Sec. 4.4), method ablations (Sec. 4.5), and analysis (Sec. 4.6).
4.1 Datasets and metrics
We evaluate PACMAC on three classification benchmarks for domain adaptation: i) OfficeHome [30] is a classification-based benchmark comprising of 12 shifts spanning 65 categories of objects found in home and office environments.
It consists of 4 domains: Real-world ( Rw ), Clipart ( Cl ), Product ( Pr ), and Art ( Ar ).
ii) DomainNet [31] is a large benchmark for adapting object recognition models.
Matching prior work [28, 9], we use the subset of DomainNet proposed in Tan et al .
[33] for our experiments, which reports performance over 12 shifts comprising 40 common classes fromdomains: Real ( R ), Clipart ( C ), Painting ( P ), and Sketch ( S ).
iii)
VisDA2017 [32] is a large image classification benchmark for synthetic → real adaptation with >200k images from 12 classes.
Metric.
Matching prior work
we report standard accuracy on the target test set as our metric.
4.2 Implementation details
We use a ViT-base [11] architecture with 16x16 image patches.
We use official codebases for
MAE [26] and DINO [25] and initialize with checkpoints pretrained on ImageNet1K [10].
We pretrain on the combined source and target domain for 800 epochs (MAE) and 200 epochs (DINO).
For pretraining, we linearly scale the learning rate to 4 × 10 − 4 (MAE) and 5 × 10 − 5 (DINO) during aepoch warmup phase followed by a cosine decay.
We use the AdamW [46] optimizer.
For PACMAC , we use k = 2 , mr = 0 .
75 , T = 50% , and α = 0 .
We use RandAugment [47] with N = 3 and M = 4 .during pretraining and N = 1 and M = 2 . 0 during DA. On OfficeHome and DomainNet, we finetune on the source and adapt for 100 epochs each, and perform 10 epochs of each phase on VisDA. We use a learning rate of 2 × 10 − 4 and weight decay of 0 . 05 . All experiments use PyTorch [48].
4.3 Baselines and SSL strategies
DA baselines: Lacking baselines designed for adapting self-supervised ViTs, we compare against diverse DA methods proposed for CNNs with supervised initializations: based on domain adversarial learning (CDAN [7]), minimizing classifier confusion (MCC [49]), self-training (SENTRY [45]), and contrastive learning (Shen et al.
We also benchmark a concurrent method for adapting ViTs (TVT [14]) with supervised initializations.
1) CDAN [7]: CDAN improves upon domain adversarial learning with multilinear conditioning, by capturing cross-covariance between feature representations and classifier predictions to improve discriminability.
Recent work [50] finds CDAN to outperform more recent DA methods when combined with new architectures like ViTs.
MCC [49]: Minimum
Classifier Confusion is a non-adversarial DA method that aligns domains by minimizing pairwise class confusion on the target domain estimated from model predictions.
3) SENTRY [9] : SENTRY measures model predictive consistency across randomly augmented versions of each target image and selectively minimizes entropy to increase predictive confidence on highly consistent instances, [Table] SENTRY [Table]
[Table]
while maximizing it to decrease confidence on highly inconsistent ones.
Shen et al.
[28] : Proposes contrastive learning on the pooled source and target domain followed by finetuning on source labels.
We adapt their approach by using SSL strategies designed for ViTs (described below), and by starting from ImageNet SSL weights rather than from scratch.
5) TVT [14] : Transferable Vision Transformer (TVT) injects a learned transferability measure into the transformer attention blocks, in addition to performing global domain-adversarial alignment and discriminative clustering.
SSL Strategies: As discussed in Sec. 3, PACMAC is designed as an adaptation strategy for selfsupervised representations.
We report benchmark performance on top of two popular SSL strategies for ViTs (see supp.
for more details): 1) MAE [26] : Masked Autoencoding (MAE) is a selfsupervised learning strategy that learns a visual transformer autoencoder to reconstruct images given only a random subset of patches from the original image.
By masking out large portions of images ( ∼ 75%), the MAE encoder is shown to learn strong representations that can be effectively finetuned to downstream tasks.
2) DINO [25] : Self-Distillation with No Labels is a visual selfsupervised learning strategy that passes two transformed versions of each image to a student and teacher network respectively, and trains the student network to predict the output of the teacher.
In supp.
for completeness we also benchmark PACMAC on top of a supervised ImageNet initialization.
4.4 Results
[Table]
Tables1, 2, and 3, present results.
We observe:
▷ Several existing DA methods underperform on top of SSL representations.
On both OfficeHome and DomainNet, we observe existing DA methods (CDAN [7], MCC [49], TVT [14]) to frequently underperform even the source model on average, despite hyperparameter tuning.
The fact that many existing DA methods struggle with such initializations suggests the need for specialized solutions, particularly as SSL pretraining becomes more common.
SENTRY [45], Shen et al.
[28], and PACMAC however offer consistent improvements.
▷ PACMAC outperforms prior work across benchmarks and initializations.
On OfficeHome ( shift average), PACMAC improves over the next-best method by 1.3% (MAE init.) and 0.2% (DINO). On DomainNet, we observe gains of 1.1% (MAE) and 0.4% (DINO), and 1.7% (MAE) on VisDA.
We note here that PACMAC and SENTRY [9] both make use of selective self-training on reliable instances identified via predictive consistency, but differ in important ways: i) PACMAC roughly matches the design of its SSL pretraining and measures consistency across partial masked inputs rather than random augmented images, ii) PACMAC incorporates model knowledge in its selection strategy by using attention-conditioning to focus on salient image regions, rather than random augmentations sampled from a manually pre-defined set.
These differences lead to its improved performance, despite being considerably simpler (2 losses, Eq. 7, and no diversity regularizers or entropy maximization).
4.5 Ablating PACMAC
In Tables 4- 5 we ablate PACMAC with a DINO initialization on OfficeHome Cl
→ Pr.
We observe: [Table] ▷ In-domain pretraining helps.(4a)
We report two metrics: cross-domain k-Nearest Neighbor (k=7) accuracy on the target domain using source domain embeddings from the trained encoder (kNN column), and target accuracy after finetuning on the source (FT column).
Across MAE and DINO initializations, we observe additional in-domain pretraining on the pooled source and target domain to improve finetuning performance (Row 2 v/s 1, +6.2% FT acc.
with MAE, +2.7% with DINO).
With
MAE, we additionally try pretraining only on source or target domains and find them to underperform
S+T pretraining (not in table).
We also try S+T pretraining with MAE by learning separate decoders to reconstruct source and target images, and find that to underperform no pretraining (Row 3 v/s 1, -2.3% ).
Interestingly, we find pretraining to improve kNN accuracy in all cases except Row
with MAE, indicating better domain alignment in encoder feature space.
With DINO, we observe higher kNN accuracies than MAE, but see a slight drop after in-domain pretraining.
▷ PACMAC benefits from S+T pretraining (4b)
We apply PACMAC to ImageNet SSL features without S+T pretraining and observe worse accuracy (Row 3 v/s 2, -2.4% with MAE, -2.7% with DINO).
▷ Combining masking consistency and confidence is an effective selection measure.
In Table 4c, we first self-train on all (Row 1) or only confident (model confidence > 50%, Row 2) target examples.
We find both to underperform selection based on our proposed attention-conditioned masking consistency scheme (Row 3, +11.5%
and +13.5% ).
We further combine our consistency measure with confidence and find selecting instances marked as reliable by atleast one measure to perform best (Row 5).
In Row 6, as we report upper-bound performance of an oracle method which only self-trains on correct pseudolabels, and find it to achieve a high accuracy of 94%.
We also try self-training on the original rather than masked image (not in table) and find it to underperform by 3.2%.
However, self-training on the masked image without using it for selection does significantly worse ( -9.7% ): clearly, the primarly benefit of masking is due to better selection rather than regularization. ▷ Matching selection strategy to DINO [25].
So far, for simplicity we match MAE’s
[26] design for PACMAC ’s selection and measure consistency across masked images.
To demonstrate generality, we now match DINO’s design and instead measure predictive consistency across local-and-global image crops (both random and with attention-seeding).
We find that for a DINO initialization this further improves accuracy ( 74.3 % on Cl → Pr), suggesting that closely matching the design of the selection criterion to the SSL pretraining is beneficial (details and visualization in supp.). ▷ Comparing selection strategy to SENTRY [9] .
We swap out PACMAC ’s attention-conditioned masking consistency scheme with the committee consistency across augmentation scheme used by the next best performing method, SENTRY [9].
We match original hyperparameters and use a committee size of 3, RandAugment [47] parameters of N=3 and M=2.0, and majority voting.[-Averaged over 
-]
Averaged overOfficeHome shifts, we find this to obtain a transfer accuracy of 66.1% (MAE) and 67.4% (DINO), while PACMAC obtains 66.8% (MAE) and 69.6% (DINO).
This clearly establishes that PACMAC ’s gains over SENTRY are due to improved selection rather than a stronger initialization. ▷ Attention-conditioning helps, as does committee consistency.
In Table 5a, we vary the committee size k and masking strategy (random v/s attention-conditioned masking).
As seen, attention conditioning consistently improves upon random masking across k ( +0.5 to 1.8% ).
▷ Ablating masking ratio and confidence threshold.
In Tables 5b and 5c, we vary the masking ratio mr and confidence threshold T , and find mr =75% and T =50% to work best. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] 4.6 Analyzing PACMAC
Evaluating reliability estimation.
In Fig.
2, we evaluate our proposed reliability estimation scheme across MAE and DINO initializations.
We observe a high precision (70-80% across both) across epochs, indicating that our method correctly identifies reliable instances with a low false-positive rate (per-class analysis in supp.).
We also find that the percentage of target instances selected for self-training increases over time, and is particularly high with DINO pretraining. Variance across runs.
To measure the performance variance of our method PACMAC , we run it across 5 random seeds on the OfficeHome Clipart → Product shift and observe a target accuracy mean of 73.1% with a standard deviation of 0.59%.
Understanding SSL initialization In Tab. 6, we contrast self-supervised (MAE and DINO) and supervised ImageNet initialization, by reporting the error of linear classifiers [28] (averaged over all
OfficeHome shifts) trained to distinguish different sets of class token features, such as: C2.
Different domains.
We observe higher error for the supervised init, indicating better domain alignment.
C3.
Same class but different domains.
Low error with DINO indicates less per-class domain alignment.
C4.
Different classes but same domain.
High error with MAE indicates more inter-class confusion. C5. Different domains and classes.
C6. We define a domain alignment (DA) score = C3-max(C4, C5) – higher means that cross-domain examples from a different domain but same class are on average closer than examples from different classes and either the same or different domains. Intuitively, a high DA score will translate to better performance for CDAN, which aligns logits-conditioned source and target features 2 . We find that SSL inits indeed have lower DA scores, possibly explaining why methods like CDAN struggle.
Visualizing attention-conditioned masking consistency.
In Fig. 3, we visualize our proposed attention-conditioned masking consistency scheme for random target images from OfficeHome. As seen, attention-conditioning ensures that the masked images focus on disjoint, highly attended regions of the target image (Row 2). Columns 1-5 illustrate examples for which our selection strategy is correctly able to identify reliable (Cols 1-2) and unreliable instances (Cols 3-5).
Cols 6-7 denote O r i g i n a l M a s k e d M a s k e d
A tt e n t i o n c a nd l e c a nd l e c a nd l e b a tt e r i e s b a tt e r i e s b a tt e r
i e s p
e n c il s p oo n s p oo n n o t e b oo k n o t e b oo k p o s t i t
s p e n c il c a l e nd a r r u l e r f il e c a b i n e t
o v e
n c o m pu t
e r
GT candle paper clip eraser fridge post-its pen f l o w e r s f l o w e r s f l o w e r s flowers
[Figure]
failure cases, with the first showing a false negative (correctly classified instance being misidentified as unreliable) and the second denoting a false positive.
Effect of in-domain pretraining on per-class kNN accuracy. In supplementary, we report per-class cross-domain kNN accuracy on the OfficeHome Cl
→
Pr shift before and after in-domain pretraining across MAE and DINO initializations.
We find that accuracy improves on several classes in both cases, particularly with MAE.
While prior work has observed benefits from additional in-domain pretraining for CNNs [28, 27], we corroborate this finding with self-supervised ViTs. In the appendix, we analyze the effect of in-domain pretraining on out-of-distribution confidence calibration (trends vary across shifts) , and include t-SNE [51] visualizations of the feature space learned by encoders before and after in-domain pretraining.
Limitations
PACMAC requires additional i) in-domain pretraining and ii) forward passes over masked target images (but no additional backpropagation), which makes it slower.
Further, PACMAC ’s effectiveness at identifying reliable instances varies across categories (see supp.).
In our paper we only experiment with ViTs trained for image classification, and the generality of our approach across architectures and tasks is not established.
Finally, the performance of DA methods with SSL initializations still lags behind supervised initializations on standard benchmarks like OfficeHome and DomainNet (comparison in supp.).
We hypothesize that this may be due to strong category overlap between
ImageNet and these benchmarks and may not translate to other datasets, but this requires further study.
Conclusion
We focus on adapting modern vision architectures (ViTs) initialized with state-of-the-art pretraining (self-supervised learning or SSL).
Our method PACMAC first performs SSL on source and target data, and then uses predictive consistency across partial versions of target images generated via an attentionconditioned masking strategy to judge reliability for selective self-training.
Despite its simplicity,
PACMAC outperforms prior work at adapting self-supervised ViTs trained for object recognition. Acknowledgements.
This work was supported in part by funding from the DARPA LwLL project and ARL.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec.
4.2 and supplementary.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec 4.1.
(b) Did you mention the license of the assets?
[Yes] See supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See supplementary.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See supplementary.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[Table]  Improving PACMAC performance Recall that in Section 4.4 we pointed out that our method PACMAC outperfoms SENTRY [9] without additional diversity regularizers or entropy maximization losses. We now attempt to add these pieces to PACMAC : specifically, we replace the target cross-entropy objective on reliable instances with an entropy minimization loss L entmin ( x T ) =
(cid:80) Cc =1 − p Θ ( y = c | x T ) log p Θ ( y = c | x T ) , optimize an additional information entropy loss to encourage diverse predictions across all target instances L div =
(cid:80) C c =1 p Θ ( y
= c | x T ) log q (ˆ y = c ) ( q (ˆ y ) denotes a running average of model predictions, loss weight = 5 × 10 − 4 ), and perform additional entropy maximization to reduce model confidence on unreliable target instances L entmax ( x T )
=
(cid:80) Cc =1 + p Θ
( y = c | x T ) log p Θ ( y = c
| x T ) (loss weight = 1 . 0 ).
We denote this method as PACMAC *.
As shown in Table 7 below, across both MAE [37] and DINO [24] initializations this further improves performance by 0.5% (MAE) and 0.9% (DINO) on average.
In addition, we compare PACMAC and PACMAC * to a combination of SENTRY [9] and Shen et al.
[28] on OfficeHome shifts starting with DINO initialization.
This combination performs initial pre-training on pooled source and target domains followed by the full SENTRY method.
On average, we find
PACMAC * clearly outperforms this combination (+1.0%).
PACMAC : Additional analysis 8.1 Per-class accuracy change
In Fig.
4 we present per-class accuracy changes after applying PACMAC to the source model across
MAE and DINO initializations on the OfficeHome Clipart → Product shift.
As seen, across both plots
PACMAC maintains or improves accuracy across most categories.
However, performance for a few categories falls, which we analyze in the next experiment.
8.2 Reliability checker: Per-class analysis
In Fig.
5 we evaluate the performance of our consistency or confidence based reliability determination scheme on a per-class level.
We use a model pretrained on the OfficeHome Clipart → Product shift with DINO, and finetuned on the source domain.
We then compute per-class F1 score of the estimated reliability on the target domain so as to capture both precision (how often is a reliable instance actually correct?) and recall (what fraction of correct instances are identified by our method?).
As seen, F1 scores are high for a majority of classes.
However, performance is noticeably worse on some classes (such as TV, bottle, and alarm clock).
Unsurprisingly, we find that model accuracy on these categories also drops after applying PACMAC (Fig. 4b).
8.3 In-domain Pretraining: Per-class accuracy
In Fig.
6 we report per-category
cross-domain kNN accuracy on the OfficeHome Cl
→ Pr shift before and after in-domain pretraining across MAE and DINO initializations.
We find that accuracy improves on several classes in both cases, particularly so for MAE.
[Figure]
8.4 Reliability checker: Comparison to SENTRY [9]’s selection
Recall that in section 4.5 we compare PACMAC ’s selection criterion to SENTRY [9]’s selection criterion by swapping out the reliability checker while keeping all other components the same.
To compare the quality of target samples being selected for training, we measure reliability precision (how many of the selected target samples were actually predicted correctly?)
and reliability recall (how many of the correctly predicted samples are selected by the selection criterion?)
as training progresses and compute the F1-score.
From Fig.
7, we observe that PACMAC ’s selection criterion achieves higher F1-score across epochs while selecting more target samples for training across epochs.
[Figure]
8.5 In-domain Pretraining: OOD calibration
In Fig.
8 we analyze the effect of in-domain pretraining on out-of-distribution confidence calibration on the target test set after S+T pretraining with the MAE and DINO SSL strategies.
We report expected calibration error (ECE [ ? ]), lower is better.
We observe inconsistent trends across shifts, with additional MAE pretraining improving out-of-distribution confidence calibration on 5/12 shifts on the OfficeHome benchmark, while DINO improving it only on 4/12.
8.6 Encoder distance plots
In Fig.
9 we visualize histograms of the distance between class token embeddings extracted from the last transformer encoder layer, for target instances without and with random masks.
We visualize these histograms for models finetuned on the source domains but with different initializations –
SSL initializations with additional S+T pretraining with MAE (Fig. 9a) and DINO (Fig. 9b), and a supervised ImageNet initialization.
Instances that are classifier correctly and incorrectly are shown separately.
As seen, with SSL initializations correct instances tend to on average have more similar embeddings across masking than supervised initializations (as a result of being trained to learn from such missing inputs during SSL pretraining), but this is not the case for the supervised initialization.
This explains the efficacy of our masking consistency-based reliability scheme for SSL initializations.
8.7 In-domain pretraining: t-SNE [51] visualization
In Figures 10- 11, we present t-SNE visualizations of class token activations from the encoder, for the Clipart and Product OfficeHome domains.
We separately visualize features before and after in-domain pretraining with MAE 10 and DINO 11.
We note that these features are completely self-supervised as the model has not seen task labels yet.
Regardless, we observe a small degree of task discriminativeness (examples of the same class are clustered together) and domain invariance (examples of the same class but different domains are close) before additional pretraining.
After pretraining, we observe it to increase, particularly after DINO pretraining.
8.8 Comparison of total training time
We compare the total time taken to train different methods including all stages: PACMAC , SENTRY [9] and Shen et al.
We include results on the OfficeHome Product → Real shift that in general results in slower training due to large number of high resolution images in both domains.
We benchmark all methods on a single NVIDIA A40 GPU.
On the OfficeHome Product → Real shift, PACMAC takes 20h 23m to train, SENTRY [9] takes 28h 15m to train while Shen et al.
[28] takes 18h 39m to train.
Supervised ImageNet initializations 9.1 PACMAC performance
For completeness, we benchmark PACMAC on top of a supervised ImageNet initialization, and find it to improve average accuracy across 12 OfficeHome shifts from 75.1% to 76.8%.
However, we find a competing method designed for supervised initializations, TVT [14], to obtain an average accuracy of 80.1%, clearly outperforming our method, despite strongly underperforming PACMAC with MAE and DINO initializations (Tables 1-3 in the main paper).
This illustrates the importance of learning representations from missing information during pretraining, in the absence of which predictive consistency across masking proves to be an ineffective reliability measure.
In Fig. 9(c) we further highlight this behavior.
9.2 Most existing DA methods do not truly evaluate domain adaptation
Most existing DA methods are initialized with supervised ImageNet initializations (with ViTs, mostly on ImageNet-22K), and adapted to standard benchmarks like OfficeHome, DomainNet,
[Figure]
[Figure]
and VisDA.
We now measure the degree of label overlap between ImageNet-22K and thesebenchmarks.
Astonishingly, the overlap is near 100%: 61/65 (OfficeHome), 40/40 (DomainNet), and 12/12 (VisDA) categories from these benchmarks directly correspond to an ImageNet-22k category.
This is particularly problematic when evaluating adaptation to real domains as a target; by definition DA assumes that the model has never seen labeled images from the target domain, but we argue that methods initialization with supervised ImageNet pretraining and adapted to real domains have seen plenty!
However, in this paper, all methods are initialized with self-supervised
ImageNet initializations and we thus present a more realistic measure of adaptation performance, even when the target domain contains real images.
Going forward, we urge the community to rethink DA benchmarking when starting with supervised initializations, and consider self-supervised initializations as a potentially fairer alternative starting point.
Self-supervised initializations may also be a superior initialization choice when adapting to domains very different from ImageNet.
For e.g., Kim et al.
[50] find that SSL initializations outperform supervised ones when generalizing to a benchmark like WILDS [ ? ], which contains very distinct images from ImageNet.
Similarly, Azizi et al.
[17] find that self-supervised pretraining on ImageNet followed by domain-specific pretraining strongly outperforms supervised ImageNet pretraining for medical image classification tasks.
Selection criteria matching DINO’s augmentations
DINO [25] performs self-supervised learning by trying to match representations of a local crop (of scale 0.1-0.4) with the representations of a global crop (of scale 0.4-1.0).
Our method uses guidance from attention and generates disjoint masks that select disconnected portions of images as augmented versions of the original image (see visualisations in Figure 3).
While we don’t explicitly match the DINO’s augmentations exactly, our method is inspired by the common theme of recent SSL methods [26, 25, 29] that try to pull closer representations of different portions of the image.
In this section, we try to exactly match the augmentations DINO uses.
We use the predictions from a committee consisting of crops of original image to determine the reliability of target samples.
Before MAE pretraining
After MAE pretraining
ClassesRulerCouchFlowersComputerKettleShelfToysTrash
[Figure]
BottleScissorsRadioChair
[Figure]
CalendarWebcamSpeakerTableScrewdriverBedMugMouseBatteriesForkSodaPanKeyboardExit
[Figure]
FanOvenLamp
[Figure]
BucketSneakersFile Cabinet
TVHammerMopCandlesMonitorSpoonToothBrushPaper
Clip
DrillFolderPrinterFlipflopsDesk
Lamp
HelmetPencilMarkerPostit Notes
TelephoneEraserBackpackCalculatorPush
Pin GlassesAlarm Clock
NotebookRefrigeratorLaptopPenCurtainsBikeClipboardsSinkDomainsClipartProduct
Before DINO pretraining
After DINO pretraining
ClassesRulerCouchFlowersComputerKettleShelfToysTrash
[Figure]
BottleScissorsRadioChair
[Figure]
CalendarWebcamSpeakerTableScrewdriverBedMugMouseBatteriesForkSodaPanKeyboardExit
[Figure]
FanOvenLamp Shade
BucketSneakersFile Cabinet
TVHammerMopCandlesMonitorSpoonToothBrushPaper
Clip
DrillFolderPrinterFlipflopsDesk
Lamp
HelmetPencilMarkerPostit Notes
TelephoneEraserBackpackCalculatorPush
Pin GlassesAlarm Clock
NotebookRefrigeratorLaptopPenCurtainsBikeClipboardsSinkDomainsClipartProduct
[Table]
Specifically, we use a committee consisting of two crops of sizes: 112x112 (local view) and 196x (global view).
We start out by selecting the crops randomly and also experiment with a scheme that uses model’s last layer attention weights to guide the crop selection.
For the attention-guided local-global cropping scheme, we first center the global crop on the most highly attended image patch, and then select the local crop over the most highly attended image patch that is at least D = 48 pixels away from the centre of the global crop.
We visualize the crops obtained using this attention-guided strategy in Figure
We report results for these attention-seeded local-global cropping augmentations on the OfficeHome’s
Clipart → Product shift and compare them to other augmentations in Table 8.
We find that both random local-global cropping and attention-seeded local-global cropping outperform their masking counterparts as well as the next best baseline (SENTRY [9]) indicating that both matching DINO’s augmentations and seeding with attention are beneficial.
We also experiment with other choices for selecting crops by leveraging attention: selecting the crop with maximum sum of attention, selecting the local crop before global crop, using different crop sizes and different values for D .
We find these to underperform in comparison to the attention-seeded local-global cropping scheme we described earlier on the OfficeHome Cl
→ Pr shift.
[Table]
[Table]
Code
We include code to reproduce our results with MAE initializations with the supplemental material.
We will publicly release all our code and pretrained models.
Datasets and implementation details
Data Licenses: Images from the OfficeHome, DomainNet, and VisDA datasets are freely available for non-commercial and academic use.
The creators note that while the datasets contain some copyrighted material, scientific research is considered a fair use of such material.
To the best of our knowledge, none of the above datasets contain personally identifiable information or offensive content.
Hyperparameters.
Tables
9- 10 include a detailed list of hyperparameters for the in-domain pretraining and adaptation phases.
Compute details.
For most experiments, we use NVIDIA A40 GPUs (4 for pretraining and 1 for finetuning/adaptation) on an internal compute cluster.