A NY DA: A NYTIME D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised domain adaptation is an open and challenging problem in computer vision.
While existing research shows encouraging results in addressing crossdomain distribution shift on common benchmarks, they are often limited to testing under a specific target setting.
This can limit their impact for many real-world applications that present different resource constraints.
In this paper, we introduce a simple yet effective framework for anytime domain adaptation that is executable with dynamic resource constraints to achieve accuracy-efficiency trade-offs under domain-shifts.
We achieve this by training a single shared network using both labeled source and unlabeled data, with switchable depth, width and input resolutions on the fly to enable testing under a wide range of computation budgets.
Starting with a teacher network trained from a label-rich source domain, we utilize bootstrapped recursive knowledge distillation as a nexus between source and target domains to jointly train the student network with switchable subnetworks.
Extensive experiments on several diverse benchmark datasets well demonstrate the superiority of our proposed approach over state-of-the-art methods.
I NTRODUCTION
Unsupervised Domain Adaptation (UDA) which aims to adapt models trained on a labeled source domain to an unlabeled target domain has attracted intense attention in recent years.
However, recent successful UDA approaches (Carlucci et al., 2019; Ganin et al., 2016; Li et al., 2020a;
Prabhu et al., 2021; Sun et al., 2019; Tan et al., 2020;
Tzeng et al., 2015; 2017) often rely on complicated network architectures and are limited to testing under a specific target setting, which may not be particularly suitable for applications across a wide range of platforms that present different resource constraints (see Figure 1a).
While adapting the trained model independently for all testing scenarios in the target domain with drastically different resource requirements looks like a possible option at the first glance, it is not efficient and economical, because of time-consuming training and benchmarking for each of these adaptation settings.
Preferably, we want to be able to adjust the model, without the need of re-training or re-adaptation in the target domain, to run in high accuracy mode when resources are sufficient and switch to low accuracy mode when resources are limited.
Motivated by this, in this paper, we investigate the problem of anytime domain adaptation where we have labeled training data from a source domain but no labeled data in the target domain and in addition testing at a resource setting with wide range of variation ( e.g. , see Figure 1b).
Specifically, we aim to train a single network using both labeled source and unlabeled target data that can directly run at arbitrary resource budget while being invariant to distribution shifts across both domains.
This is an extremely relevant problem to address as it will provide a distinct opportunity for a more practical and efficient domain adaptation to favor different scenarios with different resource budgets.
Recently, anytime prediction (Cai et al., 2019; Huang et al., 2018; Jie et al., 2019) that train a network to carry out inference under varying budget constraints have witnessed great success in many vision tasks.
However, all these methods assume that the models are trained and tested using data coming from some fixed distribution and lead to substantially poor generalization when the two data distributions are different.
The twin goals of aligning two domains and operating at different constrained computation budgets bring in additional challenges for anytime domain adaptation.
To this end, we propose a simple yet effective method for anytime domain adaptation, called AnyDA , by considering domain alignment in addition to varying both network (width and depth) and input (resolution) scales to enable testing under a wide range of computation budgets.
Such variation over width, depth and resolution enables tighter as well as finer coupling of efficiency-computation trade-off than prior works that only focus on one or two out of the three dimensions (Li et al., 2021a;
Yang et al., 2020; Yu et al., 2019b) for in-domain data.
In particular, we adopt a switchable network where individual subnetworks executable at variable computation budget share parameters with the full-budget network, known as the supernet .
However, inability to leverage on the higher capacity of complex networks may, in effect, cause such a model to severely underperform, leading to suboptimal performance across different resource budgets.
To alleviate this, we propose to distill (Ba & Caruana, 2014; Hinton et al., 2015) richer alignment information from higher capacity models to networks with limited computation.
In particular, our proposed AnyDA , adopts two switchable networks as teacher and student, that interact and learn from each other.
The student subnetworks are trained recursively to fit the output logits of an ensemble of larger subnetworks of the teacher.
Such recursive distillation within a single network with only adaptive width is shown to improve generality and reduce performance gaps between high and low capacity networks (Li et al., 2021a).
Starting with the labeled source data, we build the teacher from past iterations of the student network as an exponential moving average (ema) of the student.
The bootstrapped teacher provides the targets to train the student for an enhanced representation.
Once the target data is available, the bootstrapped recursive distillation not only brings the target features close to the source but also transfers the learned knowledge to a smaller network for efficient inference.
Moreover, we harness the categorical information by leveraging self-supervision through a pseudo-label loss on the student supernet to ensure a discriminative latent space for the unlabelled target images.
Interestingly, without using any component for explicit domain alignment ( e.g. , a domain discriminator), we show that our approach trades the performance gracefully with decreasing budgets of the subnetworks.
Our extensive experiments on 4 benchmark datasets show very minimal drop in performance across a wide range of computation budget ( e.g. , a maximum drop of only 1 . 1 % is observed when the range of computation budget gets 8 × small during testing in the Office-31 dataset (Saenko et al., 2010)).
Our work forges a connection between two literatures that have evolved mostly independently: anytime prediction and domain adaptation .
This connection allows us to leverage the progress made in unsupervised representation learning to address the very practical problem of anytime domain adaptation.
To summarize, our key contributions include:
• We introduce a novel approach for anytime domain adaptation, that is executable with dynamic resource constraints to achieve accuracy-efficiency trade-offs under domain-shifts.
We achieve this by training two networks as teacher and student with switchable depth, width and input resolutions to enable testing under a wide range of computation budgets.
• We propose a bootstrapped recursive distillation approach to train the student subnets with the knowledge from the teacher network that not only brings the target features close to the source but also transfers the learned knowledge to a smaller network for efficient inference.
• We perform extensive experiments on 4 benchmark datasets and demonstrate that AnyDA achieves superior performance over the state-of-the-art baselines, more significantly at lower computation budgets.
We also include comprehensive ablation studies to depict the importance of each module of our proposed framework.
R ELATED W ORKS
Efficient Domain Adaptation.
UDA has been dominated by methods minimizing some measure of domain discrepancy (Shen et al., 2018; Sun & Saenko, 2016;
Tzeng et al., 2015) or maximizing domain confusion (Ganin et al., 2016; Long et al., 2018; Pei et al., 2018; Tzeng et al., 2017) to generate domain-invariant features.
Leveraging image translation (Hoffman et al., 2018;
Murez et al., 2018) or style transfer (Dundar et al., 2020; Zhang et al., 2018) is also another popular trend.
Self-supervised approaches like solving pretext tasks (Carlucci et al., 2019; Mei et al., 2020;
Sahoo et al., 2020; Sun et al., 2019) and contrastive learning (Li et al., 2020a; Prabhu et al., 2021; Sahoo et al., 2021; Tan et al., 2020) have also recently enjoyed huge success in aligning domains.
In spite of the growing development of traditional UDA (Kouw & Loog, 2019; Zhao et al., 2020), the challenging problem of efficient domain adaptation remains largely underexplored.
Authors in (Jiang et al., 2020; Li et al., 2021b) have proposed a multi-scale early-exit architecture (Huang et al., 2018) with DANN (Ganin et al., 2016) as the domain adaptation method.
We, on the other hand, employ recursive knowledge distillation from an ensemble of teacher subnetworks that produces consistency supervision for the student subnetworks.
The proposed approach eliminates the need for resource hungry spatial augmentation and further reduces the training burden by eliminating the domain discriminator which is unwieldy to train.
The contemporary work SlimDA (Meng et al., 2022), attaches a weight-sharing slimmable network to a domain symmetric adaptation network (SymNet) (Zhang et al., 2019b).
The stochastic distillation uses an empirical confidence score for the subnets requiring the bi-classifier of SymNet to be replaced by a tri-classifier.
Ours, on the other hand, is model agnostic showing strong transferability across backbones.
In addition, unlike early-exit architectures with varying depth only or SlimDA with varying width only, we train a network with switchable depth, width and input resolution.
To the best of our knowledge, this work is the first attempt to address efficient domain adaptation under anytime prediction framework with switchable depth, width and input resolution with domain specific batch normalization.
Anytime Neural Networks.
Anytime neural networks (Hu et al., 2019; Jie et al., 2019; Larsson et al., 2017) are becoming increasingly attractive due to its computational efficiency.
While MSDNet (Huang et al., 2018) makes early exits to meet varying resource demands, MutualNet (Yang et al., 2020) trains a single network to achieve accuracy-efficiency tradeoffs at runtime.
Skipping unimportant channels (Chen et al., 2019) and layers (Wang et al., 2018; Wu et al., 2018), dynamic routing of multiple inference paths (Liu & Deng, 2018; Li et al., 2020b; McGill & Perona, 2017) and input dependent adaptation (Chen et al., 2020) are investigated for efficient inference on many applications.
Slimmable networks (Yu et al., 2019b) and it’s variants (Yu & Huang, 2019; Li et al., 2021a) train a model to support different width multipliers.
Adjusting width, depth and kernel sizes simultaneously to achieve better accuracy-efficiency trade-off is also proposed in (Cai et al., 2019;
Han et al., 2020).
Typically, anytime prediction works under the assumption of no distribution shift between train and test data.
While our approach is inspired by these, in this paper, we propose a framework that provides budget adaptive anytime predictions under change of domains.
Knowledge Distillation.
Distilling knowledge by mimicking output (Hinton et al., 2015; Bucilu ˇ a et al., 2006) and intermediate features (Romero et al., 2014) from an ensemble of teachers or selfdistilling (Furlanello et al., 2018; Ji et al., 2021; Lee et al., 2020; Zhang et al., 2019a) knowledge from previous iterations have obtained compact models without sacrificing the performance.
Recent works have explored self-supervised learning with knowledge distillation (Caron et al., 2021; Fang et al., 2021; Tian et al., 2020; Xu et al., 2020) enabling model compression and performance gains without explicit supervision.
However, these works rely on a teacher trained on data from the same domain while our teacher is bootstrapped from the student that is trained with both source and target data stabilizing the feature alignment process between the two domains.
M ETHODOLOGY
Anytime domain adaptation aims to achieve better generalization and low inference latency across different platforms in unlabeled target domain by transferring knowledge from a labelled source domain.
Formally, we have a set of N s labelled source images D s =
{ ( x is , y is ) }
N s i =1 and a set of N t unlabelled target images D t
= { x it } N t i =1 , with a common label space L .
The source and target samples have different data distributions.
We also have a set of n computation budgets B = {
b 1 , b 2 , ..., b n } for inference with b i <
b j when i < j .
Our goal is to learn a single domain adaptive model executable at a wide range of computation budget on the target domain with minimal performance drop.
3. P RELIMINARIES
Budget Configurations.
To make a single network executable at different budgets, we consider degrees of freedom both at the level of input ( resolution ) and network ( width , depth ), denoted by r , w , and d respectively.
r takes positive integral values, while w and d are numbers in the interval (0 , 1] .
Let R = { r 1 , r 2 , ..., r a } , W = { w 1 , w 2 , ..., w b } , and D = { d 1 , d 2 , ..., d c } denote the sets of possible values of r , w , and b .
Following (Yang et al., 2020; Yu et al., 2019b), given a configuration tuple <r x , w y , d z > ∈ R×W ×D we obtain a subnetwork (or subnet ) by executing only the first w y × 100% filters and d z × 100% blocks in each layer and take input images of r x × r x resolution consequently utilizing a specific budget b i ∈B .
Let the set of all subnets for a network G be denoted as
G ∗ =
{ s 1 , s 2 , ..., s n } , where s i is a subnet with budget b i .
The network with the highest computation budget executable at the highest width, depth and input resolution is denoted as the supernet , while the one with the lowests is called the minnet .
Rest of the networks are simply addressed as subnet .
We compute the budget in terms of Multiply-Accumulate operations (MAC), following (Li et al., 2021b), and provide the detailed budget calculation for a given tuple in Table 4 of the Appendix F.
Domain-Specific Switchable Batch Normalization.
Independently reducing internal covariate shift through batch normalization (Ioffe & Szegedy, 2015) for each of the subnets is crucial for learning conditional information specific to the network configurations (Yu et al., 2019b).
Additionally, batch normalization parameters have been used to encode domain-specific information for unsupervised domain adaptation (Chang et al., 2019).
For these reasons, in AnyDA , we learn batch normalization statistics individually for each domain as well as for each of the subnets.
3. A PPROACH O VERVIEW
Figure 2 provides an overview of the proposed approach.
We adopt a teacher-student framework consisting of two switchable networks with the same architecture, namely G θ stu and G θ tea parameterized respectively by θ stu and θ tea , which are configured into n subnets with increasing budgets following B .
The student network is fed with both source and target images, while the teacher network takes only target images.
Given an input image, each of the student subnets s stui and teacher subnets s teai predicts probability distributions p stui and p teai over the label space L by utilizing a maximum budget of b i per prediction.
The student supernet, subnets, and minnet are trained using supervised learning on the source data.
For target data, we propose a bootstrapped recursive distillation approach to train the student subnets with knowledge obtained from the teacher network.
Specifically, the student minnet is trained to match its output with the average prediction of the teacher subnets, while the student subnets are trained to match their outputs with the prediction of the teacher supernet.
Teacher network parameters are updated as exponential moving average of that of the student network.
The teacher network only gets target data as input and provides the cross-domain knowledge essential for domain alignment to the lower computation budget student networks.
Additionally, to ensure a discriminative latent space for the unlabelled target images, we harness the categorical information by leveraging self-supervision through a pseudo-label loss on the student supernet.
We now describe each of our proposed components individually in detail.
3. A NYTIME D OMAIN A DAPTATION
Bootstrapped Recursive Distillation.
Self-distillation (Caron et al., 2021) leverages multiple views of the same input to train a student network to match the output of a teacher network.
To this end, we propose a recursive distillation learning paradigm where we emulate multiple views of an input through the outputs of the budgeted subnets.
Each of the individual subnets encodes unique information specific to the configured width , depth and resolution providing us with diverse knowledge for a given input.
For each of the subnets in G θ stu ∗ and G θ tea ∗ the predicted logits are normalized with softmax for any input x to output a probability distribution over the label space L as follows:
[Equation]
where, s stuj and s teaj are any subnets in G θ stu ∗ and G θ tea ∗ respectively.
τ stu , τ tea > 0 are temperature parameters controlling the sharpness of the output distributions.
We allow the student network to observe data from both source and target domain while the teacher network sees data from target only.
This is crucial to alleviate the distribution shift by encouraging “target-to-source” correspondences for distillation from the teacher to the student.
Ensemble of teacher networks is shown to generate more accurate and general soft-labels for distillation (Shen et al., 2019;
Shen & Savvides, 2020).
So, we consider the average prediction of the teacher subnets s tea 2 to s tean − 1 as a representative of the diverse knowledge learned by them.
Additionally, this reduces the effect of noisy subnets while distilling.
The average prediction is obtained by averaging the logits and then softmaxing.
Now, for a target image x t ∈ D t , let the corresponding softmaxed logits from the student subnets be { p stut 1 , p stut 2 , ..., p stutn } , the same from the teacher subnets be { p teat 1 , p teat 2 , ..., p teatn } and p teatavg be the average prediction of the subnets s teato s tean − 1 .
We formulate the recursive distillation loss as,
[Equation]
where, H( a, b ) = − (cid:80) |L| l =1 a ( l ) log( b ( l ) ) is cross-entropy loss and is minimized with respect to student parameters θ stu only.
The first term in Eqn.
2 trains student minnet to mimic average prediction of teacher subnets which helps to gain noise robustness using the ensemble especially when distilling to very low capacity networks.
The second term encourages student subnets to mimic teacher supernet on target data, enabling a recursive flow of knowledge.
The teacher parameters θ tea are updated as: θ tea ← λθ tea +
[Equation]
1 − λ ) θ stu , where λ represents the momentum hyperparameter.
The teacher network is maintained as exponential moving average of historical parameters over training iteration allowing to bootstrap on previous representations, as in (Grill et al., 2020).
Target Pseudo-Labels.
While the bootstrapped recursive distillation loss helps in learning low budget networks with simultaneous domain alignment, relying only on the source domain for categorical information can be sub-optimal for the target domain.
Therefore, we use a pseudo-label loss on the student network to harness categorical information from the target domain through self-supervision.
For a target image x t ∈ D t the pseudo-label loss is formulated as:
[Equation]
where, ˆ y t = arg max( p stutn ) is the pseudo-label obtained from the student supernet s stun for x t .
Optimization.
While training, we follow the sandwich rule (Yu & Huang, 2019) for better convergence behavior and overall performance.
Specifically, in every iteration, we train the student minnet, supernet, and two randomly selected subnets.
This is inspired from the fact that performances at all budget are bounded by that of the model at smallest budget (minnet) and largest budget (supernet).
Thus, optimizing performance lower bound and upper bound can implicitly optimize all subnetworks of different capacities.
Additionally, this is computationally more efficient compared to training all the subnetworks in each iteration.
Formally, at a given iteration with mini-batches β s ⊂ D s and β t ⊂ D t , we optimize the following loss function:
[Equation]
where, λ cls , λ rd , and λ pl are loss coefficients.
L cls denotes the cross-entropy loss on labeled source images on all the subnets (ref.
Figure 2).
Also, note that because of the sandwich rule, the second term of L rd (ref.
Eqn. 2) now applies on the two selected subnets only.
Following this, the teacher network parameters are updated using ema in every iteration.
Warm-up using Source Data.
Absence of labels in the target domain may quickly make the networks learn a degenerate solution if we do not take care with proper initialization (Tzeng et al., 2017).
Additionally, random initialization is prone to provide high-confident noisy pseudo-labels leading to catastrophic outcomes.
Therefore, before training AnyDA with Eqn. 4, we warmup the student network with the L cls loss on the source data.
Once the warmup is completed, we copy the learned weights θ s to θ t to provide the same initialization to both the networks.
Inference.
Once the networks are trained, we only use the student network for inference.
Given any computational budget b i ∈ B , we first obtain all possible configuration tuples in R×W ×D forming subnets of budget b i .
E.g. in our experimental setup, given a budget constraint of 0 .
× 10 9 MACs, we have the tuples < 224 , 0 . 9 , 0 . 5 > and < 160 , 0 . 9 , 1 . 0 > satisfying the requirement.
We then test all of these subnets on a validation set, choose the best performing subnet and report its accuracy.
E XPERIMENTS
Datasets.
We evaluate the performance of our proposed approach using 4 benchmark datasets, namely, Office-31 (Saenko et al., 2010), DomainNet (Peng et al., 2019), Office-Home (Venkateswara et al., 2017) and ImageCLEF-DA (Long et al., 2017).
Office-31 contains a total of 4 , 652 images belonging to 31 categories from 3 distinct domains: Amazon ( A ), Webcam ( W ) and Dslr ( D ).
DomainNet is the largest available benchmark dataset containing images from 6 domains: Infograph ( inf ), Quickdraw ( qdr ), Real ( rel ), Sketch ( skt ), Clipart ( clp ), and Painting ( pnt ).
600 K images are distributed among 345 categories.
Office-Home is a challenging dataset containing images fromdifferent domains: Artistic ( Ar ), Clipart ( Cl ), Product ( Pr ), and Real-World images ( Rw ) belonging to 65 categories.
The ImageCLEF-DA dataset has around 1,800 images belonging to 12 categories in 3 domains: Caltech-256 ( C ), ImageNet ILSVRC 2012 ( I ) and Pascal VOC 2012 ( P ).
Baselines and Implementation Details.
We compare our approach with the following baselines:
ResNet (He et al., 2016) models with varying depth (ResNet-18 to ResNet-152), MobileNetV3 (Howard et al., 2019),
MSDNet (Huang et al., 2018), REDA (Jiang et al., 2020) equipped with DANN (Ganin & Lempitsky, 2015) as the domain adaptation method, TCP (Yu et al., 2019a) and ResNet (18, 34, and 50) trained with knowledge distillation (Ba & Caruana, 2014) from ResNet101 as the teacher.
We also compare with DDA (Li et al., 2021b).
For the baselines and DDA, we quote the numbers reported in the paper itself except for OfficeHome and ImageCLEF-DA for which we use the publicly available source code.
We use ResNet-50 (He et al., 2016) architecture as the full network.
For budget configuration, we use R = { 224 , 192 , 160 , 128 } , W = { 1 . 0 , 0 . 9 } , and D = { 1 . 0 , 0 . 5 } , providing us 16 subnets with computational budgets ranging from roughly 0 .
2 × 10to 2 .
× 10 9 MACs.
We further divide the range of budgets into 8 intervals of equal size.
For each interval, the best performing subnet is used to report the accuracy.
Warmup using source data was done on top of networks initialized with Imagenet pretrained weights.
In addition, following the general practice we use a validation set to obtain best hyperparameters.
Additional implementation details including hyperparameter values are provided in the Appendix F.
4. R ESULTS AND A NALYSIS
Office-31.
Figure 3a shows the results on the Office-31 dataset.
AnyDA achieves the best average accuracy of 85 .
1% at a budget of 0 .
7 × 10 9 MACs.
Baseline methods specifically tailored for efficient domain adaptation ( e.g. , DDA and REDA) outperform other baselines like MSDNet, and
ResNets.
While comparing with DDA and REDA (Li et al., 2021b) over the same budget range,
AnyDA performs significantly better, specifically for low-budget networks.
E.g., we outperform
DDA by almost 8 % at a budget of 0 .
2 × 10 9 MACs, while REDA by 6 . 9% .
AnyDA ’s performance, unlike others, does not experience a drastic drop when the available budget is decreased.
This behavior clearly shows that our bootstrapped recursive distillation is not only able to train robust low-capacity models, but also ensures simultaneous domain alignment.
DomainNet.
On the large-scale DomainNet dataset (ref Figure 3b) also, AnyDA outperforms DDA and other baselines.
Similar to Office-31, AnyDA does especially well at very low computation [Figure] [Figure] [Table]
budgets. Our average accuracy at the lowest budget of 0 .
25 × 10 9 MACs is 24 .
4% which is about 6 .
6 % more than DDA at a slightly higher lowest budget of 0 .
3 × 10 9 MACs.
It is worth noting that
DDA used 7 layered ResNet-152 model in its S7 variant as the backbone while AnyDA uses a lower complexity backbone, ResNet-50 with comparatively weak representation learning ability.
Similarly, when compared to the next best approach (MobileNet-v3) available at the same lowest budget as ours, the performance is better by 7 . 4% .
At the highest budget ( 1 . 6 × 10 9 MACs) available for
DDA, AnyDA achieves 1 % improved performance of 29 . 4% over it.
The performance improvement is continued as more computation in the anytime framework is available and reaches 29 .
7% at our highest budget of 2 .
× 10 9 MACs.
To better illustrate that AnyDA can effectively enhance domain alignment at the highest budgets of DDA S4 and S7 architectures, we provide a finegrained comparative analysis on all 30 tasks of DomainNet in Table 1.
It can be seen that even though AnyDA uses a low capacity backbone (ResNet-50 vis-a-vis ResNet-152), our approach can exploit the goodness of the higher subnetworks for domain alignment in a resource constrained scenario.
Office-Home.
In Office-Home (ref Figure 3c), we again outperform DDA for all the budgets.
Our average accuracy at the lowest budget of 0 .
25 × 10 9 MACs outperforms DDA by 30 % , while by 7 . 9% at 0 .
7 × 10 9 MACs of budget, and achieves a best accuracy of 64% at 2 .
× 10 9 MACs.
ImageCLEF-DA.
Our performance in ImageCLEF-DA (ref Figure 3d) is also in accordance with the trend in the rest of the datasets.
We outperform DDA in all budgets significantly.
At the lowest budget of 0 . 25 × 10 9 MACs AnyDA is better by an absolute margin of 23 .
1% , while by 4 . 7% at 0 .
7 × 10 9 MACs.
As in other datasets, the performance of AnyDA is minimally hampered with decreasing computation budgets, showing the efficacy of our approach for low-capacity networks.
4. C OMPARISON WITH S LIM DA
We also compared our performance with SlimDA (Meng et al., 2022), most close to ours.
Due to unavailability of the SlimDA code at the time of submission, we could compare the performance on [Table] the three datasets Office-31, Office-Home and ImageCLEF-DA on which the reported results were obtained.
SlimDA builds on top of more recent and stronger domain adaptation approach SymNet (Zhang et al., 2019b) by adding stochastic ensemble distillation with it.
For a fair comparison, we also adopted AnyDA with the SymNet architecture by simply equipping the ResNet50 backbone of SymNet with switchable depth, width and input resolution and employing the bootstrapped recursive distillation therein without needing to change the original biclassifier design of SymNet.
Table 2, shows AnyDA is better than SlimDA in all datasets across various computation budgets while being competitive at lowest compared budget of 110 × on Office-31.
On the other hand, the full budget comparison shows an absolute improvement of 1 . 9% on Office-31 dataset.
This experiment shows the easy adaptability of AnyDA with other sophisticated domain adaptation frameworks towards efficient domain adaptation under budget constraints.
We also combine SymNet (Zhang et al., 2019b), with anytime prediction (US-Net (Yu & Huang, 2019) with slimmable width, depth and resolution as ours) naively in three possible baseline settings on Office-31 and Office-Home to verify the performance of directly combining existing domain adaptation and anytime prediction techniques.
Table 3 in Appendix A shows that a direct combination of these two existing methods fails miserably for the lower budget subnets even with additional pseudo-labeling added on top of it for harnessing categorical information from the target domain.
This clearly corroborates the importance of the proposed components in AnyDA towards learning a robust network executable at different computation budget as well as alleviating the domain shift.
4. A BLATION S TUDIES
We perform ablation studies on Office-31 dataset (unless otherwise specified) to test the effectiveness of different components of AnyDA and different variations of bootstrapped recursive distillation.
Training Supernet with Only Source Data.
This ablation tests the naive scenario of employing a single high capacity network for anytime domain adaptation with a goal to set a lower bound of the performance.
We train only the supernet (a ResNet-50 model only) with source data and test it in all the computation budgets in the target domain.
Figure 6y shows that the performance is poor as expected.
Especially, the lower subnets suffer the most with fluctuations in the performance.
Effect of Teacher Selection for Distillation.
We verify the advantage of recursive distillation by comparing with two other variants.
In the first variation, the recursive distillation loss encourages all student subnets including the minnet to mimic the highest capacity teacher supernet by modifying the first term in Eqn.
2 from H( p stut 1 , p teatavg ) to H( p stut 1 , p teatn ) .
From Figure 6z, we see consistent performance drop across all budgets by around 1% compared to the proposed approach where the minnet learns from the average of the teacher subnets.
We conjecture that the progressively lower capacity and thus continually increasing gap of the student subnets with the teacher supernet can cause convergence hardship.
In the second variation, we study the effect of mimicking the average of the soft logits of all the teacher subnets, supernet and minnet.
Figure 6aa shows that the performance drops by around 2% consistently across all computation budgets showing that our proposed recursive distillation strategy makes better use of the knowledge from multiple teacher subnets.
Training by Non-Recursive Distillation.
Here, we experiment with non-recursive distillation where each student subnet learns from its companion subnet in the teacher.
This is done by encouraging each p stu ti to mimic the corresponding soft logit p tea ti from the teacher resulting in L rd ( x t ) to be (cid:80) ni =1 H
( p stuti , p teati ) in Eqn.
As seen in Figure 6ab, this results in a drastic performance drop especially for the lower subnets showing the importance of the flow of information progressively from the top of the ladder to the bottom.
Interestingly, the importance of recursive distillation can be appreciated from the fact that without it, the performance drops even below the naive approach of training a single supernet with only source data (compare with Figure 6y).
Another variant of non-recursive learning is not to learn from the bootstrapped teacher, instead try to mimic its nearest [Figure] superior subnet from the student itself.
Such inplace distillation (ID) works well with no distribution shift (Yu & Huang, 2019).
However such a strategy (ID w/o bootstapping) naturally fails for the current task as it does not use the information from supervised source data warmup.
Thus, Figure 6ac shows significant performance drop across all budgets.
A fairer comparison would be to employ the teacher for inference which starting with source data warmup updates itself by exponential moving average of the student weights.
However, Figure 6ad shows that even such an inplace distillation with bootstapping (ID w/ bootstapping) does not also work well for anytime domain adaptation.
Ablation on Distillation Loss.
We investigate the effectiveness of cross-entropy loss (ref.
Eqn. 2) by replacing it with KL Divergence loss traditionally used in knowledge distillation.
Figure 6ae shows that training with cross-entropy loss surpasses the performance of training with KL Divergence loss by a high margin especially for the lower subnets (around 1.6 % average accuracy drop in minnet).
Ablation on Batch Normalization.
Using batch normalization is crucial for better convergence of both domain adaptation (Chang et al., 2019) as well as slimmable networks (Yu et al., 2019b).
We ran a variant of AnyDA where switchable batch normalization layers were used for the different subnets but these were not different for the source and target domain data.
The performance degraded by roughly 1% across all computation budgets on average (ref.
Figure 6af).
We also tried using a single domain-specific batch-normalization for all the subnets but the testing accuracy was extremely poor.
These results help us conclude the need for both domain as well as subnet specific batch normalization to achieve the twin goals of domain generalization and computation efficiency.
Need for Explicit Domain Discriminator.
Using domain discriminators to confuse domains has been a popular choice for domain adaptation.
However, our experiments in Figure 4i show that in anytime setting, adding an explicit domain discriminator (Ganin & Lempitsky, 2015) may not offer any advantage, rather the presence of many subnetworks and as a result, multi-scale features can deteriorate the discriminability of the framework.
Role of Pseudo-Labeling.
We remove the pseudo-label loss L pl and observe that AnyDA without target pseudo-labels decreases the performance by more than 1% on average (ref.
Figure 4j) indicating the advantage of pseudo-labeling in enhancing discriminability by successfully leveraging label information from target domain.
Additional results and analysis are included in the Appendix G.
C ONCLUSION
In this paper, we introduce a novel approach for anytime domain adaptation by considering domain alignment with switchable depth, width and input resolutions to achieve accuracy-efficiency trade-offs in the target domain for different resource constraints.
In particular, we adopt a teacherstudent framework with bootstrapped recursive distillation to bring the target features close to the source and also to transfer the learned knowledge for efficient inference.
We also leverage selfsupervision via pseudo-labeling on the student supernets to ensure a discriminative latent space for the unlabelled target images.
We demonstrate the effectiveness of our proposed approach on four benchmark datasets, outperforming several competing methods.
A C OMPARISON WITH N AIVE C OMBINATION OF A NYTIME P REDICTION AND D OMAIN A DAPTATION
In Table 3, we combine one existing state-of-the-art domain adaptation (DA) method, namely
SymNet (Zhang et al., 2019b), with anytime prediction (AP) (US-Net (Yu & Huang, 2019) with slimmable width, depth and resolution as ours) naively in three settings: (1) Domain adaptation +
Anytime prediction (inference only): in this experiment, we first train a network using the SymNet domain adaptation method.
Once trained, we used the adapted model directly for anytime inference to obtain its performance at various computation budgets on the target domain. (2) Domain adaptation + Anytime prediction (w/ Pseudo-labeling): in this experiment, similar to point (1) above, we obtain an adapted network using the SymNet domain adaptation technique on source and target data. After that, we further train the network in an anytime fashion using US-Net on the target data using self-supervision through pseudo-labels (PL). Note that since the target data is unlabeled, we need to use pseudo-labels to train on them using existing anytime networks.
Anytime prediction + Domain Adaptation: in this experiment, we first train a network for anytime prediction on the labeled source data.
Then, we train it using the SymNet domain adaptation approach.
We perform the above experiments on Office-31 and Office-Home datasets and report the results in the tables below.
We compare the performance with our proposed approach AnyDA using the same SymNet backbone network for a fair comparison.
[Table]
We have the following observations from Table 3, (1) DA + AP (inference only): simply performing anytime inference on a domain adapted model performs poorly as compared to AnyDA even when using the full network with the highest budget (1.8% lower in Office-31, 5.9% lower in OfficeHome) and fails miserably for the lower budget subnets (e.g. 57.2% lower in Office-31, 46.7% lower in Office-Home in the lowest budget configuration); (2) DA + AP (w/ PL): as can be seen, the performance improves as compared to point (1) because of the additional anytime training on the target data.
But, the the huge drop in performance from the highest budget network to the lower budget subnets is still significant (e.g. 49.5% and 39.9% lower than AnyDA in the lowest budget network); (3) While this variation performs better at lower budgets, the performance at the highest budget is lower than the other two variants, showing that anytime training without exploiting target data can give a poor initialization for anytime domain adaptation.
Our approach outperforms
AP+DA variant by 14.6% and 12.2% at the highest budget, while significantly outperforming it by more than 30% at the lower budgets for the Office-31 dataset.
Similar observations can be made in the Office-Home dataset as well.
To summarize, all the findings above corroborate the importance of the proposed components in AnyDA towards learning a robust network executable at different computation budget as well as alleviating the domain shift.
B P ERFORMANCE USING LARGER BACKBONE
In Figure 5, we perform additional experiments using on Office-31 dataset in order to show the effectiveness of our approach in larger backbones.
We train DDA using the largest available supported backbone of S7.
As can be seen,
AnyDA achieves the best average accuracy of 86.4% at a budget of 3 .
9 × 10 9 MACs.
At the highest comparable budget with
DDA of 1 . 6 × 10 9 MACs, AnyDA significantly outperforms by 4.2%.
AnyDA’s performance, unlike others, does not experience a drastic drop when the available budget is decreased.
The improvement obtained at the lower budgets is even significantly more, e.g., at the corresponding lowest comparable budgets, AnyDA outperforms DDA by 16.2% at 0 .
3 × 10 9 MACs and ResNet by 5.9% at 0 .
9 × 10 9 MACs.
This behavior clearly shows that our bootstrapped recursive distillation is not only able to train robust low-capacity models, but also ensures simultaneous domain alignment with larger backbones.
C F EATURE V ISUALIZATION UNDER D IFFERENT B UDGETS
Figure 6 shows the t-SNE visualizations on four adaptation tasks (A → W, D
→ A, A → D, and D → W) from the Office-31 dataset.
The figure shows the clustering of the target features at various computation budget subnets from the supernet in the left to the minnet in the right for AnyDA.
As can be seen from Figure 6, the clustering is fairly consistent and discriminative across the subnets till the network with budget 0 .
6 x 10 9 MACs while slightly slackening for the very low budgets (after 5th column), showing the effectiveness of the bootstrapped recursive distillation in learning discriminative feature space at different budget configurations.
[Figure]
D D ISCUSSION ON U TILITY OF A NYTIME D OMAIN A DAPTATION
The focus of our work is on learning a single network which is domain invariant and can be executed at different computational budgets (MACs) in multiple devices with different budget requirements [Figure] Training a single network with the ability to be executed at different budgets is more feasible and efficient than training separate networks of corresponding budget configurations. In many practical applications, once the model is trained, it is an extremely important to perform inference many times (without retraining) due to highly dynamic deployment environments (train once but inference many).
Additionally, computation cost of training a network for AnyDA is more than that of a conventional network due to joint training of all the subnets.
However, we follow sandwich rule to reduce the training-time computation overhead by almost 4 times as compared to forward-passing through all the subnets. Focussing on both training-time as well as inference-time efficiency is an interesting research topic, which would be an exciting future work. E D ATASET D ETAILS We evaluate the performance of our approach on four benchmark datasets, namely, (1) Office31 (Saenko et al., 2010), (2) Office-Home (Venkateswara et al., 2017),
(3) DomainNet (Peng et al., 2019) and (4) ImageCLEF-DA (Long et al., 2017).
Below we provide them in details.
Office-31.
This dataset contains 4 , 110 images distributed among 31 different categories and collected from three different domains: Amazon ( A ), Webcam ( W ) and Dslr ( D ), resulting in 6 transfer tasks.
Amazon images are collected from merchant websites and possess a clear white background.
Webcam images are captured using a web-cam and are of lower resolution.
Dslr images are lownoise high resolution images.
The dataset is imbalanced across domains with 2 , 817 images belonging to Amazon, 795 images to Webcam, and 498 images to Dslr, making Amazon a larger domain as compared to Webcam and Dslr.
The dataset is publicly available to download at: https://people.eecs.berkeley.edu/˜jhoffman/domainadapt/#datasets_code .
Office-Home.
This dataset contains 15 , 588 images distributed among 65 different classes and collected from four different domains: Art ( Ar ), Clipart ( Cl ), Product ( Pr ), and RealWorld ( Rw ), resulting in 12 transfer tasks.
The Art domain has images of paintings, sketches and artistic depictions.
The Clipart domain consists of clipart images, while Product domain has clear background images.
The RealWorld domain has regular images captured with a camera.
The dataset is split across domains with 2427 images belonging to Art, 4365 images to Clipart, 4439 images to Product, and 4347 images to RealWorld.
The dataset is publicly available to download at: http://hemanthdv.org/OfficeHome-Dataset/ .
DomainNet.
This dataset is one of the largest domain adaptation benchmark datasets available containing around 0 .
6 million images.
The images are distributed among 345 different categories and are collected from six different domains: Infograph ( inf ), Quickdraw ( qdr ), Real ( rel ), Sketch ( skt ), Clipart ( clp ), and Painting ( pnt ), resulting in 30 transfer tasks.
We use the cleaned version of the dataset which is split across the domains with 51 , 605 images belonging to Infograph, 172 ,images to Quickdraw, 172 , 947 images to Real, 69 , 128 images to Sketch, 48 , 129 images to Clipart, and 72 , 266 images to Painting.
The dataset is publicly available to download at: http://ai.bu.edu/M3SDA/ .
ImageCLEF-DA.
The ImageCLEF-DA dataset is a benchmark dataset used in the ImageCLEF domain adaptation challenge of 2014 .
Following the standard practise, we carried out our experiments considering the three domains, Caltech-256 ( C ), ImageNet ILSVRC 2012 ( I ) and Pascal VOC( P ).
There are a total of 12 categories in each domain with each category having around 50 images.
The dataset is publicly available at: http://imageclef.org/2014/adaptation/ .
F I MPLEMENTATION D ETAILS
The pseudo-code for the training of AnyDA is shown in Algorithm 1.
Training Details.
For Eqn. 1, we use τ stu = 0 .
1 and τ tea = 0 . 04 .
In Eqn. ?? , we use a momentum hyperparameter value of λ = 0 .
In Eqn. 3, a threshold value of τ pl = 0 . 9 was used for Office31 and Office-Home dataset, while τ pl = 0 .
4 for the DomainNet dataset.
While in Eqn.
we Data: source data D s and target data D t , hyperparameters Networks: Student network G θ stu ∗ = { s stu
, s stu s stun } and Teacher network G tea
= { s tea
, s
., s tean } 1: warmup the networks G θ stu ∗ using source data D s 2: θ tea = θ stu # load mini-batch and iterate over dataloader 3: for s , t in loader do 4: obtain predictions p t i tea for all s teai in G θ tea ∗ 5: compute p teatavg # Sandwich rule 6: stu subnets ← minnet, two randomly sampled subnets, supernet from G θ stu ∗ # Iterate over the student subnets 7: for s stui in stu subnets do 8: obtain predictions p stusi , p stuti 9: compute cross-entropy loss for p stusi 10: compute recursive distillation loss for p stuti using H(.) 11: compute pseudo-label loss for p stu ti , if s stu i is supernet 12: end for 13: compute L cls , L rd , L pl 14: optimize G θ stu using gradient descent # Update teacher network using ema 15: update teacher network: θ tea ← λθ tea + (1 − λ ) θ stu 16: end for 17: def H( p s , p t ): # Stop gradient 18: p p
.detach() 19: mean
) use λ cls = 1 , 15 , 64 , λ rd = 1 , 1 , 0 . 5 for Office-31, Office-Home and DomainNet, respectively, λ pl = 0 .
1 for all the datasets.
We perform warm-up using source data for 100 , 100 , 30 epochs for
Office-31, Office-Home and DomainNet, respectively.
The proposed approach AnyDA is trained for 30 , 100 , 20 epochs, respectively.
Different input configurations are handled by our network as: depth : for a depth of d , we only execute first d × 100% blocks in each layer.
E.g. layer-1 of
ResNet-50 has 3 blocks, so for d =0.5, we execute only the first ⌊ 3 × 0 .
5 ⌋ =1 block of layer-1.
width :
Similarly, for a width of w , we only execute first w × 100% filters/weights for all layers.
resolution : we consider R = { 224 , 192 , 160 , 128 } which are compatible with convolution kernel size, padding and strides of ResNet-50 network.
Following (Yu & Huang, 2019), we use a single linear layer as our classifier.
Specifically, for a linear layer of shape d × n ( n = # classes), given an width scale of 0 .
9 , the subset weights of shape 0 .
9 d × n is used.
We use a per-gpu batch size of 64 ( 32 source + 32 target) for all the experiments.
We use a learning rate of 2 e - 4 for Office-31 and Office-Home, while 3 e - 5 for DomainNet.
We follow cosine annealing to update the learning rate.
We report the average classification accuracy.
We additionally use information maximization on target data as a regularizer.
All the experiments were performed using 4 NVIDIA Tesla V100 GPUs.
Budget Calculation.
In Table 4, we show the budget values for all the 16 configuration tuples.
Specific budgets at which our network is executed during inference are not pre-set, instead only a range of budgets (minimum and maximum) for defining subnets during training is used in our current work.
As can be seen, we obtain 8 intervals of equal size from the 16 configurations.
After training, our trained model (the student network) is executable at various budget configurations.
The goal is to find the best configuration under a particular resource constraint. We achieve this by using a query table. For example, in ResNet50, we sample network width from { 1, 0.9 } , network depth from { 1, 0.5 } and sample input resolution from { 224, 192, 160, 128 } . We test all these widthdepth-resolution configurations on a validation set and choose the best one under a given budget at inference.
Since there is no re-training, the whole process is once for all.
For handling budgets that fall outside of this range, one can train a supernet with a very large backbone or by considering extreme low values of width, depth and resolution: we leave this as an interesting future work.
Code. Please refer to AnyDA code.zip in the supplementary material for our code submission.
We will make the code public upon acceptance.
[Table]
[Figure]
G A DDITIONAL E XPERIMENTS
[Figure]
Effect of Hyperparameters.
We study the effect of the coefficient hyperparameters λ rd and λ pl .
For Office-31, we consider values of λ rd and λ pl an order above and below the best value used in our experiments and show the obtained results in Fig.
We observe that λ rd = 1 . 0 and
λ pl = 0 .gives the best performance. [Figure] [Figure] [Table]
[Figure]
Ablation Studies on OfficeHome.
Similar to the main paper, we perform ablation studies on the
OfficeHome dataset, and report all the findings in Fig.
For all the cases, AnyDA outperforms the corresponding ablation setup, which affirms the importance of the proposed bootstrapped recursive distillation method along with the other components for anytime domain adaptation.
Benefit of Recursive Distillation.
It was shown in (Yang et al., 2020) that decreasing the lower bound of computation budget can result in decrease of the model performance at higher budgets.
In this experiment, we tested by running AnyDA with increased lower bound of the computation from 0 .
2 × 10 9 macs to 0 .
5 × 10 9 macs for Office-31 and studied the performance in Fig.
We observe that the increased budget at lower end of the subnetworks provides a very similar performance with slight improvements.
This shows the effectiveness of the recursive distillation method in learning robust representations even if we have very low budget subnetworks which can not drag down the performance of the higher end subnetworks, rather the lower end subnetworks perform almost as good as the higher end ones.
H B ROADER I MPACT
Our research on anytime domain adaptation can help reduce burden of collecting large-scale supervised data in many real-world applications by transferring knowledge from auxiliary datasets.
At the same time, it can have a positive impact on many applications that require customization of a single deep neural network in the target domain to meet the dynamically changing demand.
Our research on anytime domain adaptation can also reduce the memory and power consumption, leading to a high impact on the environment as AI systems become more prevalent.
Potential negative impacts share many of the pitfalls associated with standard deep learning models such as vulnerability to adversarial attacks and dataset bias, and lack of interoperability, etc.