Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency
Anonymous Author(s)
AffiliationAddress email
Abstract
Visual domain adaptation (DA) seeks to transfer trained models to unseen, unlabeled domains across distribution shift, but approaches typically focus on adaptingconvolutional neural network architectures initialized with supervised ImageNetrepresentations.
In this work, we shift focus to adapting modern architectures forobject recognition – the increasingly popular Vision Transformer (ViT) – initializedwith modern pretraining based on self-supervised learning (SSL).
Inspired by thedesign of recent SSL approaches based on learning from partial image inputs generated via masking or cropping – either by learning to predict the missing pixels, orlearning representational invariances to such augmentations – we propose PACMAC ,a two-stage adaptation algorithm for self-supervised ViTs.
PACMAC first performsin-domain SSL on pooled source and target data to learn task-discriminative features, and then probes the model’s predictive consistency across a set of partialtarget inputs generated via a novel attention-conditioned masking strategy, to identify reliable candidates for self-training.
Our simple approach leads to consistentperformance gains over competing methods that use ViTs and self-supervisedinitializations on standard object recognition benchmarks.
Introduction
Deep models struggle to generalize to visual domains that deviate from the ones on which they weretrained [1].
Unsupervised domain adaptation [2, 3, 4, 5, 6, 7, 8, 9] seeks to adapt models trained onlabeled source domains to unseen and unlabeled target domains but most existing approaches focus onadapting convolutional neural network (CNN) architectures initialized with supervised representations,typically from ImageNet [10].
In this work, we seek to “modernize” domain adaptation by focusingon adapting state-of-the-art architectures for object recognition – Vision Transformers (ViTs) [11]– initialized with modern pretraining strategies based on self-supervised learning (SSL).
Vision Transformer (ViT)
[11] architectures have recently gained traction as an effective alternativeto CNNs for computer vision tasks [12], achieving impressive performance despite fewer inductivebiases.
With their in-built self-attention mechanisms and improved calibration under distribution shiftover their CNN counterparts [13], ViTs may be particularly well-suited to domain adaptation [14, 15].
Similarly, self-supervised representation learning (SSL) is rapidly replacing supervised learningas the de-facto pretraining strategy for deep networks, due to improved scalability (unlabeleddata is easier to collect) and generality (domain-specific SSL is often preferable to one-fits-all
ImageNet pretraining [16, 17]).
Many successful methods that optimize proxy objectives based onpretext tasks [18, 19, 20], instance discrimination [21, 22, 23, 24], self-distillation [25], and imagereconstruction [26], have been proposed.
Despite the increasing popularity of ViTs and SSL, to our knowledge no prior work has focusedon adapting self-supervised ViTs.
Recent work has proposed additional self-supervised contrastivelearning on the pooled source and target domain as a strong initialization for DA methods [27, 28],but focuses on adapting CNNs.
We follow these works to perform additional in-domain pretraining, finding that it leads to learning task-discriminative features with ViTs as well.
Interestingly, Shen etal.
[28] find that such contrastive pretraining learns features that disentangle domain and classinformation and can generalize to the target without being domain invariant, subverting traditionalwisdom in UDA.
We experimentally find that this causes several off-the-shelf DA methods ( e.g.based on domain adversarial learning) to sometimes fail at adapting ViTs initialized with SSLrepresentations, suggesting the need for specialized solutions.
Our work attempts to fill this gap.
Concurrent work has also studied the problem of adapting ViTs [14, 15], but focuses on supervisedrather than self-supervised initializations.
Crucially, none of these prior works propose an adaptationstrategy specifically designed for adapting self-supervised initializations, whereas we explicitlyincorporate components of the SSL pretraining in our adaptation algorithm for better transfer.
Concretely, we observe that several recent state-of-the-art SSL methods for ViTs focus on learningfrom partial inputs [22, 23, 26, 24, 25] generated via masking or cropping strategies.
In the contextof vision transformers, such methods learn to either reconstruct the missing information [26, 29] or tobe invariant to such cropping [25].
We find that the predictive consistency of a model initialized withsuch representations across partial inputs generated via masking acts as an effective self-supervisedreliability measure on the unlabeled target domain.
Further, rather than masking inputs randomly,we leverage the ViT attention mechanism to perform attention-conditioned masking : we generate aset of disjoint masks that only keep highly attended patches from the input image.
We then probethe reliability of a given target instance by measuring the model’s predictive consistency across suchmasked images and the original image, and mark target instances with high consistency as reliable.
We call our selection strategy Probing Attention-Conditioned Masking Consistency ( PACMAC ).
We further mark instances with high confidence as reliable, and then selectively self-train the modelagainst the current prediction on target instances identified as reliable via either scheme.
This leadsto an easy to implement selective self-training adaptation algorithm that outperforms competingmethods on standard benchmarks.
We make the following contributions:
• We propose an algorithm to adapt self-supervised ViTs to unseen domains that i) performsin-domain SSL on the pooled source and target data, and ii) self-trains the model on targetinstances identified to be reliable by probing the model’s predictive consistency across a setof attention-conditioned masks applied to the target image, in combination with confidence.• Our approach outperforms more complex DA methods from prior work at adapting selfsupervised ViTs on the OfficeHome [30], DomainNet [31], and VisDA [32] adaptationbenchmarks for object recognition.
Related work
Unsupervised domain adaptation (UDA).
UDA seeks to transfer a model trained on a labeled sourcedomain to an unlabeled target domain across distribution shift.
A variety of successful approachesbased on aligning feature spaces via optimizing domain divergence measures [4, 3] and distributionmatching via domain adversarial learning [6, 5, 7] (often incorporating additional pixel-level matchingconstraints), have been proposed.
Recently, selective self-training [33, 9] on the model’s predictionson target data has emerged as a simple and effective UDA technique.
With the upsurge in adoption of Vision Transformer (ViT [11]) architectures for object recognition,some recent works specifically focus on adapting ViTs [14, 15].
These methods leverage the ViTattention mechanism for incorporating patch-level transferability [14] and category-level featurealignment [15].
However, these works focus on adapting models initialized with supervised ImageNetweights.
In this work, we propose an adaptation algorithm designed for self-supervised ViTs.
Self-supervised learning.
Self-supervised learning (SSL) leverages unlabeled data to learn representations that can be transferred efficiently to downstream tasks.
Several approaches based onpretext tasks [18, 19, 20], masked-image modeling [26, 34], context prediction [35] and instancediscrimination [22, 23], have been proposed to learn strong semantic representations in the absence oflabeled data.
Some of the state-of-the-art SSL methods are designed for ViTs: DINO [25] performsself-distillation by training a student network that is given a strongly augmented image as input tomatch the output of a teacher network that sees a global view of the image.
MAE [26] is a maskedimage modeling (MIM) approach that learns to predict a complete image given only a fraction of theimage patches as input.
Recently, ADIOS [36] improves MIM methods by learning an adversarialmasking function to obscure salient regions of the image.
A concurrent work, MSN [29] improves upon DINO by passing in a masked version of an augmented image to the student network.
We deriveinspiration from the local-to-global self-supervisory signal used by these methods in designing ouradaptation algorithm, but instead of learning reconstruction or invariance to missing information,measure the model’s predictive confidence under targeted missing information as a self-supervisedprobe to determine reliability.
Self-supervised learning for domain adaptation.
While most UDA methods focus on adaptingmodels initialized with supervised initializations, typically from ImageNet, a few recent works havestudied self-supervised domain adaptation [29, 27].
Kim et al.
[27] propose CDS, a pretrainingstrategy for domain adaptation that makes use of in-domain contrastive learning in conjunction withcross-domain matching as a superior initialization alternative to ImageNet pretraining.
Shen et al.
[28]study the transferability of representations learned via additional in-domain contrastive learning onthe source and target domain, finding that such pretrained features can be transferred effectively tothe target by simply finetuning on the source, despite not being domain invariant.
However, thesemethods restrict their experiments to ResNet [37] architectures while we work focus on adapting
ViTs.
Crucially, neither of these works propose a DA algorithm catered to the model’s self-supervisedinitialization, whereas we do so explicitly.
Approach
3.1 Notation
Let X and Y denote input and output spaces.
In unsupervised domain adaptation (UDA) we aregiven access to labeled source instances ( x S , y S ) ∼ P S ( X , Y ) , and unlabeled target instancesx T ∼ P T ( X ) , where S and T correspond to source and target domains.
The goal is to learn a modelf
= h ( ϕ ( . ))
( ϕ denotes the encoder and h denotes classifier), parameterized by Θ with minimumerror on the target dataset.
In our experiments, Θ is parameterized as a Vision Transformer or
ViT [11]), which takes as input a linear embedding of of N image patches in addition to a class tokenembedding.
These N+1 embeddings are then appended with positional encodings and passed throughseveral transformer layers, each of which comprises of a sequence of multi-headed self-attention(with M attention heads), multilayer perceptron, and layernorm modules.
Features from the finalencoder layer are typically fed to a classifier layer and used to predict the output.
We consideradapting models trained for C -way image classification: the inputs x are images, and labels y arecategorical variables y ∈ { 1 , 2 , .., C } .
Let p Θ
( y | x ) denote the final probabilistic output from themodel.
For a target instance x T ∼ P T ( X ) , we compute a “pseudolabel” as ˆ y = argmax p Θ ( y | x T ) .
3.2 Preliminaries
Self-supervised learning (SSL) for ViTs.
We consider SSL methods for ViTs that learn from partialinputs, under two popular formulations: i) Masked Image Modeling, wherein the model is given apartial image and trained to predict the missing content (at a pixel or token level) via learning a maskedor denoising autoencoder [26, 38, 39, 34].
ii) Joint-embedding networks, which learn a model thatproduces similar features for different views of a given image, across strong cropping and additionalaugmentation [25, 29].
Let m ( x T ) denote the partial image generated from target image x T undertransformation m ( . ) which could correspond to either a masking or augmentation strategy.
Broadlyspeaking, SSL corresponds to minimizing an objective of the form L SSL ( ϕ ( m ( x T )) , ϕ ( x T )) , whichcould correspond to a reconstruction or invariance based objective operating on encoded features.
Self-training for unsupervised DA.
Self-training for UDA typically involves training the modelon its predictions on unlabeled target data (“pseudolabels”), optimizing either a conditional entropyminimization [40] or cross-entropy objective.
However, under a domain shift many of the model’spredictions may initially be incorrect [41, 42, 9] (especially under severe distribution shift), andunconstrained self-training may lead to amplifying model errors.
To combat this, a few recent works have proposed selective self-training on predictions that have ahigh likelihood of being correct.
Recent works propose to identify such reliable predictions eitherbased on model confidence [33, 43], predictive consistency under data augmentation [9, 44], orcombinations of the two [45].
Let r ( x T ) denote a binary reliability value for target instance x T .
Weoptimize the following selective self-training objective on target data L SST :
[Equation]
In this work, we propose an alternative selection criterion designed for self-supervised representationsbased on predictive consistency under attention-conditioned masking, which we now introduce.
3.3 PACMAC : Probing Attention-conditioned Masking
Consistency for UDA
Following prior work [28, 27], we first learn task-discriminative features by optimizing the followingin-domain self-supervised pretraining objective L IDP on pooled source and target data:
[Equation]
Next, we perform selective self-training.
For a target instance x T ∼ P T , we generate a committee ofk masked versions.
However, applying random masks may lead to capturing irrelevant backgroundfeatures; instead, we propose attention-conditioned masking (see Algo. 1).
We first obtain themodel’s per-patch attention score ˆ a T over the original image (in practice, we obtain the patchwiseself-attention from the last encoder layer of the class token with respect to all image patches for eachattention head, and average across the M heads).
We then initialize all k masks with zeros (hideall patches).
Next, we sort patches in descending order of attention scores and perform a greedyround-robin assignment to the k masks to only unhide highly attended patches, until the desiredmasking ratio is satisfied.
We then apply these masks to the original image to generate masked images{ m 1 ( x T ) , m 2 ( x T ) , ..., m k ( x T ) } .
We make predictions for each of these k masked versions (following prior work [37] we only feed inunmasked patches), and measure consistency between the model’s prediction for the original imageand for each of the k masked versions.
If the model’s prediction for all masked versions matches itsprediction on the original image, we consider the instance as “reliable”, and as “unreliable” if not.
Further, we also consider instances as reliable if the model’s predictive confidence on the originalimage is higher than a threshold T .
Let 1 [ . ] denote an indicator function.
Formally, we assign targetinstance reliability r ( x T ) as:
[Equation]
Algorithm 2 lists the steps involved in computing the PACMAC objective.
Without loss of generality,we minimize the cross-entropy between the model’s prediction on the last consistent masked imagem k ( x T ) and its consistent pseudolabel.
We empirically find this to act as an effective data augmentation strategy and provide a stronger learning signal than when using the model’s prediction on theoriginal unmasked image (see Sec. 4.5).
We additionally optimize a cross-entropy loss over labeledsource examples.
For loss weight α , the full L PACMAC objective is:
[Equation]
1: Input: x T , per-patch attention ˆ a T , masking ratio mr , committee size k 2: L ← (1 − mr ) × len( ˆ a T ) 3: ˆ S T ← argsort( ˆ a T )
▷ Sort patch-wise ascending order 4: for i ← 1 to k
do M
i ← zeros_like
(ˆ a ) ▷ Initialize
, 0 is hide, 1 is keep 5: i ←6: while i < L do 7: i ← i +8: for j ← 1 to k do 9: p ← ˆ S T .pop_last
() ▷ Pop next-most attended patch index 10:
M pj ←▷
Greedy round-robin assignment 11: for j ← 1 to k do m j ( x T ) =
M j ⊙ x
T 12: Return { m 1 ( x T ) , . . .
, m k ( x T ) }
1: Input: SrcLoader, TgtLoader, model parameters Θ , masking ratio mr , committee size k , confidence threshold T , loss weight α 2: for epoch ← 1 to MAX_EPOCH do 3: for x S , y S in SrcLoader x T
TgtLoader do 4: ˆ p T , ˆ y T ← max p Θ ( y | x T )
▷ Get top prediction original image 5: ˆ a T ← att ( p Θ ( y | x T ))
▷ Get per-patch attention 6: { m 1 ( x T ) , . . .
, m k ( x T ) } ← AttentionConditionedMasking
( x T , ˆ a T , mr , k ) 7:
C ← { m i ( x T ) | ˆ y T = argmax p Θ ( y | m i ( x T )) } ki = 8:
L SST
=9: if len (C) =
k OR ˆ p T > T then ▷ Consistent or Confident 10:
L SST ← L CE (
Θ y m
Minimize L CE ( x S , y S )
+ α L SST
Experiments
We first describe our datasets and metrics (Sec. 4.1), implementation details (Sec. 4.2), and baselines(Sec. 4.3).
Next, we present results (Sec. 4.4), method ablations (Sec. 4.5), and analysis (Sec. 4.6).
4.1 Datasets and metrics
We evaluate PACMAC on three classification benchmarks for domain adaptation: i) OfficeHome [30]is a classification-based benchmark comprising of 12 shifts spanning 65 categories of objects foundin home and office environments.
It consists of 4 domains: Real-world ( Rw ), Clipart ( Cl ), Product( Pr ), and Art ( Ar ).
ii) DomainNet [31] is a large benchmark for adapting object recognition models.
Matching prior work [28, 9], we use the subset of DomainNet proposed in Tan et al .
[33] for ourexperiments, which reports performance over 12 shifts comprising 40 common classes from 4domains:
Real ( R ), Clipart ( C ), Painting ( P ), and Sketch ( S ).
iii)
VisDA2017 [32] is a large imageclassification benchmark for synthetic → real adaptation with >200k images from 12 classes.
Metric.
Matching prior work
we report standard accuracy on the target test set as our metric.
4.2 Implementation details
We use a ViT-base [11] architecture with 16x16 image patches.
We use official codebases for
MAE [26] and DINO [25] and initialize with checkpoints pretrained on ImageNet1K [10].
Wepretrain on the combined source and target domain for 800 epochs (MAE) and 200 epochs (DINO).
For pretraining, we linearly scale the learning rate to 4 × 10 − 4 (MAE) and 5 × 10 − 5 (DINO) during a 40epoch warmup phase followed by a cosine decay.
We use the AdamW [46] optimizer.
For PACMAC , weuse k
= 2 , mr = 0 .
75 , T = 50% , and α =
[Table]
4.3 Baselines and SSL strategies
DA baselines: Lacking baselines designed for adapting self-supervised ViTs, we compare againstdiverse DA methods proposed for CNNs with supervised initializations: based on domain adversariallearning (CDAN [7]), minimizing classifier confusion (MCC [49]), self-training (SENTRY [45]), andcontrastive learning (Shen et al.
We also benchmark a concurrent method for adapting ViTs(TVT [14]) with supervised initializations.
1) CDAN [7]: CDAN improves upon domain adversariallearning with multilinear conditioning, by capturing cross-covariance between feature representationsand classifier predictions to improve discriminability.
Recent work [50] finds CDAN to outperformmore recent DA methods when combined with new architectures like ViTs.
MCC [49]: Minimum
Classifier Confusion is a non-adversarial DA method that aligns domains by minimizing pairwiseclass confusion on the target domain estimated from model predictions.
3) SENTRY [9] : SENTRYmeasures model predictive consistency across randomly augmented versions of each target imageand selectively minimizes entropy to increase predictive confidence on highly consistent instances,while maximizing it to decrease confidence on highly inconsistent ones.
Shen et al.
[28] : Proposescontrastive learning on the pooled source and target domain followed by finetuning on source labels.
We adapt their approach by using SSL strategies designed for ViTs (described below), and by startingfrom ImageNet SSL weights rather than from scratch.
5) TVT [14] : Transferable Vision Transformer(TVT) injects a learned transferability measure into the transformer attention blocks, in addition toperforming global domain-adversarial alignment and discriminative clustering.
SSL Strategies: As discussed in Sec. 3, PACMAC is designed as an adaptation strategy for selfsupervised representations.
We report benchmark performance on top of two popular SSL strategiesfor ViTs (see supp.
for more details): 1) MAE [26] : Masked Autoencoding (MAE) is a selfsupervised learning strategy that learns a visual transformer autoencoder to reconstruct imagesgiven only a random subset of patches from the original image.
By masking out large portions ofimages ( ∼ 75%), the MAE encoder is shown to learn strong representations that can be effectivelyfinetuned to downstream tasks.
2) DINO [25] : Self-Distillation with No Labels is a visual selfsupervised learning strategy that passes two transformed versions of each image to a student andteacher network respectively, and trains the student network to predict the output of the teacher.
Insupp.
for completeness we also benchmark PACMAC on top of a supervised ImageNet initialization.
4.4 Results
[Table]
Tables1, 2, and 3, present results.
We observe:
▷ Several existing DA methods underperform on top of SSL representations.
On both OfficeHome and DomainNet, we observe existingDA methods (CDAN [7], MCC [49], TVT [14]) to frequently underperform even the source model on average, despite hyperparameter tuning.
The fact that many existing DA methods struggle with such initializations suggests the need for specialized solutions, particularly as SSLpretraining becomes more common.
SENTRY [45], Shen et al.
[28],and PACMAC however offer consistent improvements.▷ PACMAC improves upon prior work across benchmarks and SSL initializations.
On OfficeHome(12 shift average), PACMAC improves over the next-best method by 1.3% (MAE init.) and 0.2% [Table] [Table] [Table]
Shen et al.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
SENTRY [9]
[Table]
[Table] (DINO). On DomainNet, we observe average gains of 1.1% (MAE) and 0.4% (DINO), and 1.7%(MAE) on VisDA. We note here that PACMAC and SENTRY [9] both make use of selective self-training on reliableinstances identified via predictive consistency, but differ in important ways: i) PACMAC matches thedesign of its SSL pretraining and measures consistency across masked rather than random augmentedimages, ii) PACMAC uses attention-conditioning to generate masks rather than random selection.
Thesedifferences lead to its improved performance, despite being considerably simpler (2 losses, Eq. 4,and no diversity regularizers, class-balancing on the source and target, or entropy maximization).
4.5 Ablating PACMAC
[Table]
In Tables 4- 5 we ablate PACMAC with a DINO initialization on OfficeHome Cl
→ Pr.
We observe:▷ In-domain pretraining helps.(4a)
We report two metrics: cross-domain k-Nearest Neighbor (k=7)accuracy on the target domain using source domain embeddings from the trained encoder (kNNcolumn), and target accuracy after finetuning on the source (FT column).
Across MAE and DINOinitializations, we observe additional in-domain pretraining on the pooled source and target domain toimprove finetuning performance (Row 2 v/s 1, +6.2% FT acc.
with MAE, +2.7% with DINO).
With
MAE, we additionally try pretraining only on source or target domains and find them to underperform
S+T pretraining (not in table).
We also try S+T pretraining with MAE by learning separate decodersto reconstruct source and target images, and find that to underperform no pretraining (Row 3 v/s 1,-2.3% ).
Interestingly, we find pretraining to improve kNN accuracy in all cases except Row 3 with
MAE, indicating better domain alignment in encoder feature space.
With DINO, we observe higherkNN accuracies than MAE, but see a slight drop after in-domain pretraining.▷ PACMAC
benefits from S+T pretraining (4b
We apply PACMAC to ImageNet SSL features without
S+T pretraining and observe worse accuracy ( Row 3 v/s 2, -2.4% with MAE, -2.7% with DINO).
▷ Combining masking consistency and confidence is an effective selection measure.
In Table 4c,we first self-train on all (Row 1) or only confident (model confidence > 50%, Row 2) target examples.
We find both to underperform selection based on our proposed attention-conditioned maskingconsistency scheme (Row 3, +5.1% and +2.4% ).
We further combine our consistency measure withconfidence and find selecting instances marked as reliable by atleast one measure to perform best(Row 5).
In Row 6, as we report upper-bound performance of an oracle method which only self-trains
[Table] [Table]
[Table]
on correct pseudolabels, and find it to achieve a high accuracy of 94%. We also try self-training onthe original rather than masked image (not in table) and find it to underperform by 3.2%.▷
PACMAC ’s gains are not simply due to a stronger initialization.
To verify this, we swap out
PACMAC ’s attention-conditioned masking consistency scheme with the committee consistency acrossaugmentation scheme used by the next best performing method, SENTRY [9].
We match originalhyperparameters and use a committee size of 3, RandAugment [47] parameters of N=3 and M=2.0,and majority voting.
[-Averaged over 
 OfficeHome shifts, we-]We find this obtains an 71.8% v/s 74.0% ( +2.2% ) with PACMAC .
This indicatesthe increased efficacy of our selection strategy over SENTRY’s for adapting self-supervised ViTs.▷ Attention-conditioning helps, as does committee consistency.
In Table 5a, we vary the committee size k and masking strategy (random v/s attention-conditioned masking).
As seen, attentionconditioning consistently improves upon random masking across k ( +0.5 to 1.8% ).▷
Ablating masking ratio and confidence threshold.
In Tables 5b and 5c, we vary the maskingratio mr and confidence threshold T , and find mr =75% and T =50% to work best.
[Figure]
[Figure]
[Table]
MAE 8.
[Table]
[Table] [Table] DINO [Table]
[Table] [Table] [Table] [Table]
Sup.
[Table] [Table]
[Table]
Evaluating reliability estimation.
In Fig.
2, we evaluate our proposed reliability estimation schemeacross MAE and DINO initializations.
We observe a high precision (70-80% across both) acrossepochs, indicating that our method correctly identifies reliable instances with a low false-positiverate (per-class analysis in supp.).
We also find that the percentage of target instances selected forself-training increases over time, and is particularly high with DINO pretraining.
Understanding SSL initialization In Tab. 6, we contrast self-supervised (MAE and DINO) and supervised ImageNet initialization, by reporting the error of linear classifiers [28] (averaged over all OfficeHome shifts) trained to distinguish class token features from: C2.
Different domains.
We observehigher error for the supervised init, indicating better domain alignment.
C3.
Same class but differentdomains.
Low error with DINO indicates less per-class domain alignment.
C4.
Different classes butsame domain.
High error with MAE indicates more inter-class confusion.
C5. Different domains andclasses. C6. We define a domain alignment (DA) score = C3-max(C4, C5) – higher means that crossdomain examples from a different domain but same class are on average closer than examples fromdifferent classes and either the same or different domains.
Intuitively, a high DA score will translateto better performance for CDAN, which aligns logits-conditioned source and target features 1 . We findthat SSL inits indeed have lower DA scores, possibly explaining why methods like CDAN struggle. O r i g i n a l M a s k e d
M a s k e d
A tt e n t i o n c a nd l e c a nd l e c a nd l e b a tt e r i e s b a tt e r i e s b a tt e r
i e s p
e n c il s p oo n s p oo n n o t e b oo k n o t e b oo k p o s t i t
s p e n c il c a l e nd a r r u l e r f il e c a b i n e t
o v e
n c o m pu t
e r
GT candle paper clip eraser fridge post-its pen f l o w e r s f l o w e r s f l o w e r s flowers
[Figure]
Visualizing attention-conditioned masking consistency.
In Fig.
3, we visualize our proposedattention-conditioned masking consistency scheme for random target images from OfficeHome.
Asseen, attention-conditioning ensures that the masked images focus on disjoint, highly attended regionsof the target image (Row 2).
Columns 1-5 illustrate examples for which our selection strategy iscorrectly able to identify reliable (Cols 1-2) and unreliable instances (Cols 3-5).
Cols 6-7 denotefailure cases, with the first showing a false negative (correctly classified instance being misidentifiedas unreliable) and the second denoting a false positive.
Effect of in-domain pretraining on per-class kNN accuracy.
In supplementary, we report per-classcross-domain kNN accuracy on the OfficeHome Cl
→
Pr shift before and after in-domain pretrainingacross MAE and DINO initializations. We find that accuracy improves on several classes in bothcases, particularly with MAE. While prior work has observed benefits from additional in-domainpretraining for CNNs [28, 27], we corroborate this finding with self-supervised ViTs. In the supplementary, we analyze the effect of in-domain pretraining on out-of-distribution confidencecalibration (observing inconsistent trends across shifts) , and include t-SNE [51] visualizations of thefeature space learned by encoders before and after in-domain pretraining.
Limitations
PACMAC requires additional i) in-domain pretraining and ii) forward passes over masked target images(but
no additional backpropagation), which makes it slower.
Further, PACMAC ’s effectiveness atidentifying reliable instances varies across categories (see supp.).
In our paper we only experimentwith ViTs trained for image classification, and the generality of our approach across architecturesand tasks is not established.
Finally, the performance of DA methods with SSL initializations stilllags behind supervised initializations on standard benchmarks like OfficeHome and DomainNet(comparison in supp.).
We hypothesize that this may be due to strong category overlap between
ImageNet and these benchmarks and may not translate to other datasets, but this requires further study.
Conclusion
We focus on adapting modern vision architectures (ViTs) initialized with state-of-the-art pretraining(self-supervised learning or SSL).
Our method PACMAC first performs SSL on source and target data,and then uses predictive consistency across partial versions of target images generated via an attentionconditioned masking strategy to judge reliability for selective self-training.
Despite its simplicity,
PACMAC outperforms prior work at adapting self-supervised ViTs trained for object recognition.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSec. 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] See
supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Sec.
4.2 and supplementary.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplementary.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Sec 4.1.
(b) Did you mention the license of the assets?
[Yes] See supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See supplementary.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See supplementary.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]