Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts
Anonymous Author(s)
AffiliationAddress email
Abstract
In this paper, we tackle the problem of domain shift.
Most existing methodsperform training on multiple source domains using a single model, and the sametrained model is used on all unseen target domains.
Such solutions are sub-optimalas each target domain exhibits its own speciality, which is not adapted.
Furthermore,expecting the single-model training to learn extensive knowledge from the multiplesource domains is counterintuitive.
The model is more biased to learning onlydomain-invariant features and may result in negative knowledge transfer.
In thiswork, we propose a novel framework for unsupervised test-time adaptation, which isformulated as a knowledge distillation process to address domain shift.
Specifically,we incorporate with Mixture-of-Experts (MoE) as teachers, where each expert isseparately trained on different source domains to maximize their speciality.
Givena test-time target domain, a small set of unlabeled data is sampled to query theknowledge from MoE. As the source domains are correlated to the target domains,a transformer-based aggregator then combines the domain knowledge by examiningthe interconnection among them.
The output is treated as a supervision signal toadapt a student prediction network toward the target domain.
We further employmeta-learning to enforce the aggregator to distill positive knowledge and the studentnetwork to achieve fast adaptation.
Extensive experiments demonstrate that theproposed method outperforms the state-of-the-art and validates the effectiveness ofeach proposed component.
Introduction
The emergence of deep models has achieved superior performance [26, 34].
Such unprecedentedsuccess is built on the strong assumption that the training and testing data are highly correlated (i e.,they are both sampled from the same data distribution).
However, the assumption normally does nothold in real-world settings as the training data is infeasible to cover all the ever-changing deploymentenvironments [33].
Reducing such distribution correlation is known as distribution shift, whichsignificantly hampers the performance of deep models.
Although human is more robust against thedistribution shift, artificial learning-based systems suffer more from performance degradation.
One line of research aims to mitigate the distribution shift by exploiting some unlabeled data froma target domain is known as unsupervised domain adaptation (UDA) [20, 41, 22].
The unlabeleddata is an estimation of the target distribution [72].
Therefore, UDA normally adapts to the targetdomain by transferring the source knowledge via a common feature space with less effect fromdomain discrepancy [66, 40].
However, UDA is less applicable for real-world scenarios as repetitivelarge-scale training is required for every target domain.
In addition, collecting the data samples froma target domain in advance might be unavailable as the target could be unknown during training.
Domain generalization (DG) [44, 24, 6] is an alternative line of research but more challenging as it assumes the prior knowledge of the target domains is unknown.
DG methods leverage multiple sourcedomains for training and directly use the trained model on all unseen domains.
As the domain-specificinformation for the target domains is not adapted, a generic model is sub-optimal [57, 13].
Recently, an adaptive method, ARM [72], incorporates test-time adaptation with DG.
Metalearning [21] is utilized for training the model as an initialization such that it can get updatedusing the unlabeled data from each target domain before making predictions.
However, we observedthat ARM only trains a single model, which is counterintuitive for the multi-source domain setting.
There is a certain amount of correlation among the source domains while each of them also exhibitsits own specific knowledge.
When the number of source domains rises, data complexity dramatically increases, which impedes the exploration of the dataset thoroughly.
Furthermore, real-worlddomains are not always balanced in data scales [33].
Therefore, the single-model training is morebiased toward the domain-invariant features and dominant domains instead of the domain-specificfeatures [11].
In this work, we propose to formulate the test-time adaptation as the process of knowledge distillation [28] from multiple source domains.
Concretely, we propose to incorporate the concept of
Mixture-of-Experts (MoE), which is a natural fit for the multi-source domain settings.
The MoEmodels are treated as a teacher and separately trained on the corresponding domain to maximizetheir domain speciality.
Given a new target domain, a few unlabeled data are collected to query thefeatures from expert models.
A transformer-based knowledge aggregator is proposed to examinethe interconnection among queried knowledge and aggregate the correlated information toward thetarget domain.
The output is then treated as a supervision signal to update a student predictionnetwork to adapt to the target domain.
The adapted student is then used for subsequent inference.
Weemploy bi-level optimization as meta-learning to train the aggregator at the meta-level to improvegeneralization.
The student network is also meta-trained to achieve fast adaptation via a few samples.
Furthermore, we simulate the test-time out-of-distribution scenarios during training to align thetraining objective with the evaluation protocol.
The proposed method also provides additional advantages over ARM: 1)
Our method provides alarger model capability to improve the generalization power; 2) Despite the higher computationalcost, only the adapted student network is kept for inference, while the MoE models are discardedafter adaptation.
Therefore, our method is more flexible in designing the architectures for the teacheror student models.
(e.g., designing compact models for the power-constrained environment); 3) Ourmethod does not need to access the raw data of source domains but only needs their trained models.
So, we can take advantage of private domains in a real-world setting where their data is inaccessible.
We name our method as Meta - D istillation of MoE (Meta-DMoE).
Our contributions are manifold:
• We propose a novel unsupervised test-time adaptation framework that is tailored for multiple
sources domain settings.
Our framework employs the concept of MoE to allow each expert model to explore each source domain thoroughly.
We formulate the adaptation process as
knowledge distillation via aggregating the positive knowledge retrieved from MoE. • The alignment between training and evaluation objectives via meta-learning improves the adaptation, hence the test-time generalization.• We conduct extensive experiments to show the superiority of the proposed method among
the state-of-the-arts and validate the effectiveness of each component of Meta-DMoE.
• We validate that our method is more flexible in real-world settings where computational power and data privacy are the concerns.
Related work
Domain shift Unsupervised Domain Adaptation (UDA) has been popular to address domain shiftby transferring the knowledge from the labeled source domain to the unlabeled target domain.
It isachieved by developing domain-invariant via minimizing statistical discrepancy across domains [5,47, 59].
Adversarial learning is also applied to develop indistinguishable feature space [22, 41, 46].
The main limitation of UDA is the assumption of the co-existence of source and target data, which isinapplicable when the target domain is unknown in advance.
Furthermore, most of the algorithmsfocus on unrealistic single-source-single-target adaptation as source data normally come from multipledomains.
To drop the dependence on source domain data, algorithms toward source-free DA are closer to the real-world applications [39, 35, 68].
Splitting the source data into various distinctdomains and exploring the unique characteristics of each domain and the dependencies among themfurther strengthen the robustness [74, 64, 65].
Domain generalization (DG) is another line of research to alleviate the domain shift.
The generalsetting is to train a model on multiple domains and expect it to perform well on unseen targetdomains.
Similar to DA methods, learning the domain-invariant feature representation is alsoeffective [24, 37, 43].
Data augmentation strategies in data or feature space are also promising [52, 63].
However, for most DG methods, the same generic trained model is deployed to all unseen domains,which discards their domain speciality and yields sub-optimal solutions.
Test-time adaptation (TTA) TTA has also been popular to address the domain shift. The main ideais to obtain a supervision signal at test time to update the model before making a prediction.
Sun etal.
[57] use rotation prediction to update the model during inference.
Chi et al.
[13] reconstruct theinput images to achieve internal-learning to better restore the blurry images.
TTA is also related topersonalization as the adaptation process captures unique information [38].
Meta-learning Meta-learning is an active area of research.
The existing methods can be categorisedas model-based [51, 48, 8], metric-based [54], and optimization-based [21].
Meta-learning aims totrain a model to achieve learning to learn. It is realized by episodic learning at the task level. Suchbi-level optimization has been wildly applied in different tasks, such as coupling the performance oftwo tasks to achieve test-time adaptation [13], unsupervised adaptation for domain shift [72].
Mixture-of-Experts The goal of MoE [31] is to decompose the whole training set into many subsets,which are independently learned by different models.
It has been successfully applied in imagerecognition models to improve the accuracy [1].
MoE is also popular in scaling up the architectures.
As each expert is independently trained, sparse selection methods are developed to select a subsetof the MoE during inference to increase the network capacity [36, 19, 25].
In contrast, our methodutilizes all the experts to extract and combine the knowledge for valid knowledge transfer.
Preliminaries
In this section, we describe the problem setting and discuss the adaptive model.
We mainly followthe test-time unsupervised adaptation as in [72].
Specifically, we define a set of N source domains
D S =
{D S i } Ni =1 and M target domains D T
= {D T j } Mj =1 .
The physical definition of a domainvaries and depends on the applications or data collection methods.
It could be a specific dataset, useror location.
Let x ∈ X
and
y ∈ Y denote the input and corresponding label, respectively.
Each of thesource domains contains the data in the form of input-output pairs: D S i
= { ( x z S , y z S ) }
Z i z =1 .
In contrast,each of the target domains contains only the unlabeled data: D T
j = { ( x k T ) } K j k
=1 .
For well-designeddatasets, (e.g. [27, 16]), all the source or target domains have the same number of data samples.
Suchcondition is not ubiquitous for real-world scenarios (i.e. Z i 1 ̸
=
Z i 2 if i 1 ̸
= i 2 and K j 1 ̸ = K j 2 ifj 1 ̸ =
j 2 ) where data imbalance always exists [33].
It further challenges the generalization with abroader range of real-world distribution shifts instead of finite synthetic ones.
Generic domain shifttasks focus on the out-of-distribution setting where the source and target domains are non-overlapping(i.e.
D S ∩ D T = ∅ ), but the label spaces of both domains are the same (i.e. Y S =
Y T ).
Conventional DG methods perform training on D S and make minimal assumption on the testingscenarios [56, 3, 29].
Therefore, the same generic model is directly applied to all target domains D T ,which leads to non-optimal solutions [57].
In fact, for each D T j , some unlabeled data are readilyavailable which provide certain prior knowledge for that target distribution.
Adaptive Risk Minimization (ARM) [72] assumes that a batch of unlabeled input data x approximates the input distributionp x which provides useful information about p y | x .
Based on the assumption, an unsupervised test-timeadaptation [48, 23] is proposed.
The fundamental concept is to adapt the model to the specific domainusing x .
Overall, ARM aims to minimize the following objective L ( · , · ) over all training domains:
[Equation]
x
[Equation]
θ
[Equation]
where θ ′ = h ( x , θ ; ϕ ) .
[Equation]
y is the labels corresponding to x . f
( x ; θ ) denotes the prediction function parameterized by θ .
h ( · ; ϕ )is an adaptation function parameterized by ϕ .
It receives the original parameter θ of f and theunlabeled data x to adapt θ to θ ′ .
[Figure]
[Figure]
The goal of ARM is to learn both ( θ, ϕ ) .
To mimic the test-time adaptation (i.e., adapt beforeprediction), it follows the episodic learning as in meta-learning [21].
Specifically, each episodeprocesses a domain by performing unsupervised adaptation using x and h ( · ; ϕ ) in the inner loop toobtain f ( · ; θ ′ ).
The outer loop evaluates the adapted f ( · ; θ ′ ) using the true label to perform metaupdate.
ARM is a general framework that can be incorporated with existing meta-learning approacheswith different forms of adaptation module h ( · ; · ) [21, 23].
However, several shortcomings are observed with respect to the generalization.
The episodic learningprocesses one domain at a time, which has clear boundaries among the domains.
The overall settingis equivalent to the multi-source domain setting, which is proven to be more effective than learningfrom a single domain [43, 73] as most of the domains are correlated to each other [2].
However, it iscounterintuitive to learn all the domain knowledge in one single model as each domain has specializedsemantics or low-level features [53].
Therefore, the single-model method in ARM is sub-optimaldue to: 1) some domains may contain competitive information, which leads to negative knowledgetransfer [55].
It may tend to learn the ambiguous feature representations instead of capturing all thedomain-specific information [67]; 2) not all the domains are equally important [64], and the learningmight be biased as the data in each domain is imbalanced in real-world [33].
Proposed approach
In this section, we introduce a simple yet effective framework that is more tailored for domaingeneralization tasks to harness its multi-domain characteristic.
Specifically, we learn a set of mixtureof-experts (MoE) that specialize in different domains.
We explicitly formulate the test-time adaptationas a knowledge transfer process to distill the knowledge from MoE. The proposed method is learnedvia meta-learning to mimic the test-time out-of-distribution scenarios and ensure positive knowledgetransfer. We name our approach Meta-distillation from MoE (Meta-DMoE).
4.1 Meta-distillation from mixture-of-experts
Overview Fig.
1 shows the method overview.
We wish to explicitly transfer valid knowledgefrom various domains to elevate the generalization of unseen domains.
Concretely, we define MoEas M =
{M i } Ni =1 to represent the domain-specific models.
Each M i is separately trained usingstandard supervised learning on source domain D S i to learn its discriminative features.
We proposethe test-time adaptation as the unsupervised knowledge distillation [28] to learn the knowledge from
MoE.
Therefore, we treat M as the teacher and aim to distill its knowledge to a student predictionnetwork f ( · ; θ ) to achieve adaptation.
To do so, we sample a batch of unlabeled x from a targetdomain, and pass it to M to query their domain-specific knowledge {M i ( x ) } Ni =1 .
That knowledgeis then forwarded to a knowledge aggregator A ( · ; ϕ ) .
The aggregator is learned to explore theinterconnection among domain knowledge and yield knowledge composition towards that targetdomain.
The output of A ( · ; ϕ ) is treated as the supervision signal to update f ( x ; θ ) .
Once the adapted θ ′ is obtained, f ( · ; θ ′ ) is used to test the rest of the data in that domain.
The overall frameworkfollows the effective few-shot learning where x is treated as unlabeled support set [62, 54, 21].
[Table]
Training Meta-DMoE
Properlytraining the ( θ, ϕ ) is critical to improve the generalization on unseen domains.
First, A ( · , ϕ ) performs as a mechanism that explores and mixes the input knowledge.
Thus, it is not supposedto be biased to any training data.
Second, the conventional distillation process requires large numbers of data samples and learning iterations [28, 2].
The repetitive large-scale training is inapplicable in real-world applications.
To mitigate the aforementionedchallenges, we follow the metalearning paradigm [21].
Such bilevel optimization enforces the
A ( · , ϕ ) to learn beyond any specific knowledge [71] and allowsthe student prediction networkf ( · ; θ ) to achieve fast adaptation.
Specifically, We first split the data samples in each source domain D S i into disjoint support and query sets.
The unlabeled support set ( x SU ) is used to perform adaptation via knowledge distillation, while the labeled query set ( x Q , y Q ) is used to evaluate the adaptedparameters to explicitly test the generalization on unseen data.
The student prediction network f ( · ; θ ) can be decoupled as a feature extractor θ e and classifier θ c .
Unsupervised knowledge distillation can be achieved via the softened output [28] or intermediatefeatures [70] from M .
The former one allows the whole student network θ =
( θ e , θ c ) to be adaptive,while the latter one allows partial or complete θ e to adapt to x , depending on the features utilized.
We follow [45] to only adapt θ e in the inner loop while keeping the θ c fixed.
Thus, the adaptationprocess is achieved by distilling the knowledge via the aggregated features:
[Equation]
where α denotes the adaptation learning rate, M e is the feature extractor of MoE models whichextracts the features before the classifier, and ∥·∥ 2 measures the L 2 distance.
The goal is to obtain anupdated θ ′
e such that the extracted features of f ( x SU ; θ ′ e ) is closer to the aggregated features.
Theoverall learning objective of Meta-DMoE is to minimize the following expected loss:
[Equation]
, where θ [Equation] where L CE is the cross-entropy loss.
Alg.
1 demonstrates our full training procedure.
To smooth themeta gradient and stabilize the training, we process a batch of episodes before each meta-update.
Since the training domains overlap for the MoE and meta-training, we simulate the test-time out-ofdistribution by excluding the corresponding expert model in each episode.
To do so, we multiplethe features by 0 to mask them out.
M ′ e in L11 of Alg.
1 denotes such operation.
Therefore, theadaptation is enforced to use the knowledge that is aggregated from other domains.
4.2 Fully learned explicit knowledge aggregator
To explicitly aggregate the knowledge from distinct domains requires exploring the relation amongthem to ensure the relevant knowledge transfer.
Prior works design more specific hand-engineeredtechniques to combine the knowledge or choose data samples that are closer to the target domainfor knowledge transfer [2, 74].
A superior alternative is to replace the hand-designed pipelines withthe fully learned solutions, including learning to learn algorithms using meta-learning [15, 9].
Thus we follow the same trend and allow the aggregator A ( · ; ϕ ) to be fully meta-learned without manymanual designs except defining its architecture.
We observe that the self-attention mechanism quite meets our needs where interaction amongdifferent domain knowledge can be computed.
Therefore, we use a naive transformer encoder as theaggregator [18, 61].
The transformer encoder consists of multi-head self-attention and multi-layerperceptron blocks with layernorm [4] applied before each block, and residual connection appliedafter each block.
We refer the readers to the supplementary material for the detailed architecture andcomputation.
We concatenate the output features from the MoE models in the domain dimensionas Concat [ M 1 e ( x ) , M 2 e ( x ) , ..., M Ne ( x )] ∈ R N × d , where d is the feature dimension.
The aggregator A ( · ; ϕ ) processes the input tensor to obtain the aggregated feature F ∈ R d , which is used as asupervision signal for test-time adaptation.
4.3 More constrained real-world settings
In this section, we investigate two critical settings for real-world applications but have drawn lessattention from the prior works: limitation on computational resources and data privacy.
Constraint on computational cost In real-world deployment environments, the computational powermight be highly constrained (e.g., smartphones).
It requires fast inference and compact models.
However, the reduction in learning capabilities greatly hinders the generalization as some methodsutilize only a single model regardless of the data complexity.
On the other hand, when the numberof domain data scales up, methods relying on adaptation on every data sample [72] will experienceinefficiency.
In contrast, our method only needs to perform adaptation once for every unseen domain.
Only the final f ( · ; θ ′ ) is used for inference.
To investigate the impact on generalization caused byreducing the model size, we replace the models (only f ( · ; θ ) for us) with MobileNet V2 [50].
Data privacy regulation Large-scale training data is normally collected from various venues.
However, some venues may have privacy regulations enforced.
Their data might not be accessible but themodels that are trained using the private data are available.
To simulate such an environment, we splitthe training source domains into two splits: private domains ( D S pri ) and public domains ( D S pub ).
We use D S pri to train MoE models and D S pub for the subsequent meta-training.
Since ARM andother methods only utilize the data as input, we train them on D S pub .
We conduct the experimentswith data privacy on top of the power-constrained scenario and show the superiority of the proposedmethod in the experiment section.
For details of the settings, please refer the supplementary material.
Experiements
5.1 Datasets and implementation details
Datasets and evaluation metrics.
In this work, we mainly evaluate our method on the real-worlddomain shift scenarios.
Drastic variation in deployment conditions normally exists in nature, such asa change in illumination, background, and time.
It shows a huge domain gap between deploymentenvironments and imposes challenges to the algorithm’s robustness.
Thus, we test our methodson the large-scale distribution shift benchmark WILDS [33], which reflects a diverse range of realworld distribution shifts.
Following [72], we mainly perform experiments on five image testbeds,iWildCam [10], Camelyon17 [7],RxRx1 [58] and FMoW [14] and PovertyMap [69].
In eachbenchmark dataset, a domain represents a distribution over data that is similar in some way, suchas images collected from the same camera trap or satellite images taken in the same locations.
Wefollow the same evaluation metrics as in [33] to compute: accuracy, Macro F1, worst-case (WC)accuracy, Pearson correlation (r), and its worst-case counterpart.
Network architecture.
We follow WILDS [33] to use ResNet18 & 50 [26] or DenseNet101 [30] forthe expert models {M i } Ni =1 and student network f ( · , ; θ ) .
Also, we use a single-layer transformerencoder block[61] as the knowledge aggregator A ( · ; ϕ ) .
To investigate the resource-constrained andprivacy-sensitive scenarios, we utilize MobileNet V2 [50] with a width multiplier of 0.25.
Pre-training domain-specific models.
WILDS benchmark is highly imbalanced in data size, andsome classes have empty input data set; we found that using every single domain to train an expertis unstable, and sometimes it cannot converge.
Inspired by [42], we propose to cluster the trainingdomains into N super domains and use each super-domain to train the expert models.
Specifically, we [Table] set N = { 10 , 5 , 3 , 4 , 3 } for iWildCam, Camelyon17, RxRx1, FMoW and Poverty Map, respectively.
We use ImageNet [17] pre-trained model as the initialization and separately train the models using
Adam optimizer [32] with a learning rate of 1 e − 4 and a decay of 0.96 per epoch.
Meta-training and testing.
To improve the convergence speed, we first pre-train the aggregatorand student network using supervised learning [12].
After that, the model is further trained using Alg. 1 for 15 epochs with a fixed learning rate of 3 e − 4 for α and 3 e − 5 for β .
During meta-testing,we use Line 13 of Alg. 1 to adapt before making a prediction for every testing domain.
For bothmeta-training and testing, we perform one gradient update for adaptation on the unseen target domain.
For all experiments, we tune the hyperparameters using the validation split and do the final evaluationon the test split.
We refer the readers to the supplementary materials for more detailed information.
5.2 Main results
In this section, we compare the proposed method with prior approaches showing on WILDS leaderboard [33], including non-adaptive methods: CORAL [56], ERM [33], IRM [3], Group DRO [49]and adaptive methods used in ARM [72] (CML, BN and LL).
We directly copy the available resultsfrom the WILDS leader-board or their corresponding paper.
As for the missing ones, we conductexperiments using their provided source code with default hyperparameters.
Table 1 reports thecomparison with the state-of-the-art.
Our proposed method performs well across all datasets andincreases both worst-case and average accuracy compared to other methods.
Our proposed methodachieves the best performance on 4 out of 5 benchmark datasets.
ARM [72] applies meta-learningapproach to learn how to adapt to unseen domains with unlabeled data.
However, their method isgreatly bounded by using a single model to exploit knowledge from multiple source domains.
Instead,our proposed method is more fitted to multi-source domain settings and meta-trains an aggregatorthat properly mixtures the knowledge from multiple domain-specific experts.
As a result, our methodoutperforms ARM-CML, BN and LL by 9.5%, 9.8%, 8.1% for iWildCam, 8.5%, 4.8%, 8.5% for
Camelyon17 and 14.8%, 25.0%, 22.9% for FMoW in terms of average accuracy.
Furthermore, wealso evaluate our method without masking the in-distribution domain in MoE models (Ours w/o mask)during meta training (Line 10-11 of Alg. 1), where the sampled domain is overlapped with MoE.
It violates the generalization to unseen target domains during testing.
As most of the performancedropped, it reflects the importance of aligning the training and evaluation objectives.
Visualization of adapted feature . To evaluate the capability of adaptation via learning discriminativerepresentations on unseen target domains, we compare the t-SNE [60] feature visualization using thesame test domain sampled from iWildCam and Camelyon17 datasets.
ERM utilizes single model andstandard supervised training without adaptation, therefore, we set it as the baseline.
Figure 2 showsthe comparison, where each color denotes a class and each point represents a data sample.
It is clearthat
our method obtains better clustered and more discriminative features.
5.3 Results under constrained real-world settings
Constraint on computational cost Computational power is always limited in real-world deploymentscenarios, such as edge devices.
Efficiency and adaptation ability should be both considered.
Thus,we replace our student model and the models in other methods with MobileNet V2.
As reported
[Figure]
[Table] in Table [Table] our proposed method still outperforms prior methods.
Since the MoE model is onlyused for knowledge transfer, our method is more flexible in designing the student architecture fordifferent scenarios.
We also report multiply-Accumulate operations (MACS) for inference and timecomplexity on adaptation.
As ARM needs to make adaptation before inference on every example, itsadaptation cost scales linearly with the number of examples.
Our proposed method performs better inaccuracy and requires much less computational cost for adaptation, as reported in Table
[Table] [Table]
[Table]
Constraint on privacy regulation On top ofcomputational limitations, privacy-regulated scenarios are common in the real world.
It introduces new challenges as the raw data is inaccessible.
Our method does not need to access theraw data but the trained models, which greatlymitigates such regulation.
Thus, as shown in Table 4, our method does not suffer from much performance degradation compared to other methods that require access to the private raw data.
5.4 Ablation studies
In this section, we conduct ablation studies oniWildCam to analyze various components ofthe proposed method.
We also seek to answerthe two key questions: 1) Does the number ofexperts affect the capability of capturing knowledge from multi-source domains? 2) Does metalearning perform better than standard supervisedlearning under the knowledge distillation framework?
Number of domain-specific experts
Instead of using a single network, our method exploits multiple experts to store domain-specificknowledge separately. Increasing the number of experts improves the capability of fully exploring the speciality of each domain. Therefore, theadaptation to unseen target domain is also enhanced.
The experiments in Table 5 also validate thebenefits of using more domain-specific experts.
Training scheme To verify the effectiveness of meta-learning, we investigate three training schemes:random initialization, pre-train, and meta-train.
To pre-train the aggregator, we add a classifier layerto its aggregated output and follow the standard supervised training scheme.
For fair comparisons,we use the same testing scheme, including the number of updates and images for adaptation.
Table 6reports the results of different training scheme combinations.
We observe that the randomly initializedstudent model struggles to learn with only a few-shot data.
And the pre-trained aggregator bringsweaker adaptation guidance to the student network as the aggregator is not learned to distill.
Incontrast, our bi-level optimization-based training scheme enforces the aggregator to choose morecorrelated knowledge from multiple experts to improve the adaptation of the student model.
Therefore,the meta-learned aggregator is more optimal (row 1 vs. row 2).
Furthermore, our meta-distillationtraining process simulates the adaptation in testing scenarios, which aligns with the training objectiveand evaluation protocol.
Hence, for both meta-trained aggregator and student models, it gainsadditional improvement (row 3 vs. row 4).
[Table]
wards test-time adaptation
[Table] [Table]
[Table]
Aggregator and distillation methods We analyze the importance of the various architecturechoices of the knowledge aggregator in Table 7.
We found that the fully learned aggregator iscrucial for mixing domain-specific features andoutperforms other hand-designed aggregationoperators: max and average pooling.
Anotherimportant design choice in our proposed framework in the form of knowledge: distilling theteacher model’s logits, intermediate features, orboth.
We show evaluation results of those threeforms of knowledge in Table
Discussion
We present Meta-DMoE, a framework for adaptation towards domain shift using unlabeled examples at test time.
We formulate the adaptationas a knowledge distillation process and devise ameta-learning algorithm to guide the student prediction network to fast adapt to unseen target domains via transferring the aggregated knowledgefrom multiple sources domain-specific models.
We demonstrate that Meta-DMoE is the state-ofthe-art on four challenging benchmarks.
And itis competitive under two constrained real-worldsettings with a limited computational budget anddomain data privacy regulation.
Limitations.
As discussed in Section 5.4, MetaDMoE can improve the capacity to capture complex knowledge from multi-source domains byincreasing the number of experts.
However,to compute the aggregated knowledge fromdomain-specific experts, every expert modelneeds to have one feed-forward pass.
As a result,the total computational cost of adaptation scales linearly with the number of experts.
Furthermore,to add or remove any domain-specific expert, both the aggregator and the student network requireto be re-trained from scratch.
Thus, enabling a sparse-gated Meta-DMoE to encourage efficiencyand scalability could be a valuable direction for future studies, where a gating module determines asparse combination of domain-specific experts to be used for each target domain.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 1.
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seesupplemental material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tal material
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See 5.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental material
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
a) If your work uses existing assets, did you cite the creators?
[Yes] See [33]
(b) Did you mention the license of the assets?
[Yes] See supplemental material
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]