Towards Data-Free Domain Generalization
Abstract
In this work, we investigate the unexplored intersection of domain generalization and data-free learning.
In particular, we address the question: How can knowledge contained in models trained on different source data domains be merged into a single model that generalizes well to unseen target domains, in the absence of source and target domain data?
Machine learning models that can cope with domain shift are essential for for real-world scenarios with often changing data distributions.
Prior domain generalization methods typically rely on using source domain data, making them unsuitable for private decentralized data.
We define the novel problem of Data-Free Domain Generalization (DFDG), a practical setting where models trained on the source domains separately are available instead of the original datasets, and investigate how to effectively solve the domain generalization problem in that case.
We propose DEKAN, an approach that extracts and fuses domain-specific knowledge from the available teacher models into a student model robust to domain shift.
Our empirical evaluation demonstrates the effectiveness of our method which achieves first state-of-the-art results in DFDG by significantly outperforming ensemble and data-free knowledge distillation baselines.
Introduction
Deep learning methods have achieved impressive performance in a wide variety of tasks where the data is independent and identically distributed.
However, real-world scenarios usually involve a distribution shift between the training data used during development and the test data faced at deployment time.
In such situations, deep learning models often suffer from a performance degradation and fail to generalize to the out-of-distribution (OOD) data from the target domain [62, 66, 17, 21].
For instance, this domain shift problem is encountered when applying deep learning models on MRI data from different clinical centers that use different scanners [10].
Domain Adaptation (DA) approaches [71, 73] assume access to data from the source domain(s) for training as well as target domain data for model adaptation.
However, data collection from the target domain can sometimes be expensive, slow, or infeasible, e.g. self-driving cars have to generalize to a variety of weather conditions [80] and object poses [3] in urban and rural environments from different countries.
In this work, we focus on the Domain Generalization (DG) [5, 48] setting, where a model trained on multiple source domains is applied without any modification to unseen target domains.
In the last decade, a plethora of DG methods requiring only access to the source domains were proposed [86].
Nevertheless, the assumption that access to source domain data can always be granted does not hold in many cases.
For instance, General Data Protection Regulation (GDPR) prohibits the access to sensitive data that might identify individuals, e.g. bio-metric data or other confidential information.
Likewise, some commercial entities are not willing to share their original data to prevent competitive disadvantage.
Furthermore, as datasets get larger, their release, transfer, storage and management can become prohibitively expensive [39].
To circumvent the concerns related to releasing the original dataset, the data owners might want to share a model trained on their data instead.
In light of increasing data privacy concerns, this alternative has recently enjoyed a surge of interest [44, 7, 50, 37, 33, 28, 78, 1].
Although Data-Free Knowledge Distillation (DFKD) methods were developed to transfer knowledge from a teacher model to a student model without any access to the original data [39, 44, 7, 50, 78, 9], only single-teacher scenarios with no domain shift were studied.
On the other hand, Source-Free Domain Adaptation (SFDA) approaches were proposed to tackle the domain shift problem setting where one [37, 33, 28, 70, 11] or multiple [1] models trained on source domain data are available instead of the original dataset(s).
Nonetheless, they require access to data from the target domain.
In this work, we investigate the unstudied intersection of Domain Generalization and Data-Free
Learning. Data-Free Domain Generalization (DFDG) is a problem setting that assumes only access to models trained on the source domains, without requiring data from source or target domains.
Hereby, the goal is to have a single model able to generalize to unseen domains without any modification or data exposure, as it is the case in DG.
To the best of our knowledge, we are the first to address this problem setting.
Works addressing related problems are discussed in Appendix A.
Our contribution is threefold: Firstly, we introduce and define the novel and practical DFDG problem setting.
Secondly, we tackle it by proposing a first and strong approach that merges the knowledge stored in the domain-specific models via the generation of synthetic data and distills it into a single model.
Thirdly, we demonstrate the effectiveness of our method by empirically evaluating it on two DG benchmark datasets.
Approach 2.1 Problem statement
Let D is and D jt denote the datasets available from the source and target domains respectively with i = 1 , .., I and j = 1 , .., J .
Hereby, I and J denote the number of source and target domains respectively.
In the Domain Generalization (DG) [5, 48] problem setting, the goal is to train a model on the source domain data D is in a way that enables generalization to a priori unavailable target domain data D jt , without any model modification at test time.
We consider the source-data-free scenario of this problem where the source domain datasets D is are not accessible, e.g., due to privacy, security, safety or commercial concerns, and models trained on these domain-specific datasets separately are available instead.
We refer to the source domain models as teacher models T i as in the knowledge distillation literature [22].
We assume that the teacher models were trained without the prior knowledge that they would be used in a DFDG setting, i.e., their training does not involve any domain shift robustness mechanism.
Hence, the application scenarios where the source domain data is not accessible anymore, e.g., was deleted, are also considered.
We refer to this novel learning scenario as Data-Free
Domain Generalization (DFDG) .
The major difference with Source-Free Domain Adaptation (SFDA) [37, 33, 28] is the absence of target domain data D jt in DFDG.
The DFDG problem is a prototype for a practical use case where a model robust to domain shifts is needed and models trained on the same task but different data domains are available.
This problem definition is motivated by the question: How can we amalgamate the knowledge from multiple models trained on different domains into a single model that is able to generalize to unseen target domains without any data exposure?
2.2 Domain Entanglement via Knowledge Amalgamation from Domain-Specific Networks
We propose Domain Entanglement via Knowledge Amalgamation from domain-specific Networks (DEKAN).
Our approach tackles the challenges of DFDG in 3 stages: Knowledge extraction, fusion and transfer.
In the first stage, Intra-Domain Data-Free Knowledge Extraction , we extract the knowledge from the different source domain teacher models separately. Hereby, we generate domainspecific synthetic datasets via inceptionism-style [46] image synthesis, i.e., we initialize random noise images ˆ x and optimize them to be recognized as a sample from a pre-defined class by a trained domain-specific model.
In particular, we apply the data-free knowledge distillation method described in [78, 83] to invert each domain-specific teacher separately.
In the second stage, Cross-Domain
Data-Free Knowledge Fusion , DEKAN generates cross-domain synthetic data by leveraging all pairs of inter-domain model-dataset combinations.
Here, the cross-domain examples are optimized to be recognizable by teacher models trained on different domains.
In the final stage, Multi-Domain Knowledge Distillation , DEKAN transfers the extracted knowledge from the domain-specific teachers to a student model via knowledge distillation using the generated data.
At test time, i.e., deployment phase, the resulting student model is evaluated on target domain data without any modification.
Details about the first stage as well as DEKAN’s complete algorithm can be found in Appendix B.
In the following, we focus on the second and third stages. In the second stage, we propose a technique to merge the knowledge from two domains by generating cross-domain synthetic images that capture class-discriminative features present in the two domains, and match the distribution of intermediate features extracted by a domain-specific model from images of another domain.
Let T a and T b denote the teacher models, and D ag and D bg the synthetic data generated in the first stage (Appendix B.0.1), specific to two domains a and b .
We generate synthetic images D abg by minimizing the cross-domain inversion loss L abCD , that we formulate as [Equation] [Equation] where L C denotes the classification loss, e.g., cross-entropy, L R an image prior regularization,
L CDM the cross-domain feature moment matching loss, and α 1 and α 2 weighting coefficients.
L R penalizes the l 2 -norm and the total variation of the image to ensure the convergence to valid natural images [42, 52, 46, 78].
We incentivize the generated images to contain class-discriminative features from both domains by minimizing the classification loss using both teachers.
We hypothesize that images that can be recognized by models trained on different domains capture more domain-agnostic semantic features than those generated by inverting a single domain-specific model as done in prior works [78].
In addition, the cross-domain feature distribution matching loss L abCDM optimizes the cross-domain synthetic images D abg so that their feature distribution matches the distribution of the features extracted by T a , the model trained on domain a , for images D bg synthesized from domain b .
Note that L abCDM ̸ =
L baCDM and that using the model T b and the data generated by inverting T a in the first stage, i.e., D ag , would yield the cross-domain images D bag that are different from D abg .
Formally,
[Equation]
L abCDM minimizes the l 2 -norm between the BN-statistics of the synthetic data, µ l (ˆ x ) and σ 2 l (ˆ x ) , and target statistics, at each BN layer l .
Here, the target statistics, ba ˆ µ l and ba ˆ σ 2 l , are computed in a way that involves knowledge from different domains.
In particular, they result from feeding the synthetic data specific to domain b through the teacher model trained on data from domain a , and computing the first two feature moments, i.e., mean and variance, for each BN layer.
The intention behind this is to synthesize images that capture the features learned by the model on domain a that are activated and recognized when exposed to images from domain b .
We hypothesize that such images would encompass domain-agnostic semantic information that would be useful for training a single model resilient to domain shift in the next stage.
We relax L CDM by allowing the BN-statistics of the synthetic input to fluctuate within a certain interval.
Here, we compute the relaxation constants ba
δ l and ba γ l as the ϵ CD percentile of the distribution of differences between the stored BN-statistics, i.e., computed on the original domain a images, and those computed using the images D bg synthesized from the domain b teacher model in the first stage.
Note that ϵ CD = 100% corresponds to synthesized images ˆ x yielding the BN-statistics from domain a , i.e., stored in model T a , would not be penalized, i.e., L abCDM = 0 .
This stage can be viewed as a domain augmentation, since the synthesized images D abg do not belong neither to domain a nor to domain b . The synthesis of cross-domain data is applied to all possible domain pairs.
In the final DEKAN stage, the domain-specific and cross-domain knowledge, which is captured in the synthetic data generated in the first and second stages respectively, is transferred to a single student model S . To this end, we use knowledge distillation [22], i.e., we train the student model to mimic the predictions of the teachers for the synthetic data. As described in Equation 3, we minimize the Kullback-Leibler divergence D KL between the predictions of the student S and the teacher(s) corresponding to the synthetic image ˆ x .
In particular, if the data example is domain-specific, i.e., it was generated in the first DEKAN stage, the predictions of the corresponding teacher are used as soft labels to train the student. For the cross-domain synthetic images that were generated in the second stage, the average predictions of the two corresponding teachers is used instead. The aggregation of the prediction distributions of two domain-specific teacher models contributes to the knowledge amalgamation across domains. [Equation]
The conducted experiments 3 aim to tackle the following key questions: ( a ) How does DEKAN compare to leveraging the domain specific models directly to make predictions on data from unseen domains?
( b ) How does our approach compare to data-free knowledge distillation methods applied to each domain separately?
( c ) How much does the unavailability of data cost in terms of performance?
We design baseline methods to address the novel DFDG problem, and compare them with DEKAN.
The first category of baselines applies the available domain-specific models on the data from the target domains (Question ( a ) ).
We consider two ensemble baselines that aggregate the predictions of these models, e.g., by taking the average of the model predictions ( AvgPred ), or by taking the prediction of the most confident model, i.e., the model with the lowest entropy ( HighestConf ).
Besides, we implement oracle methods that evaluate each of the domain-specific models separately on the target domain and then report the results of the best model ( BestTeacher ).
Furthermore, we propose a baseline that applies an improved version [83] of DeepInversion (DI) [78] on each of the models separately to generate domain-specific synthetic images used to then train a student model via knowledge distillation ( Multi-DI ; Question ( b ) ).
Note that Multi-DI is equivalent to the application of DEKAN’s first and third stage.
Finally, we compare DEKAN to an upper-bound baseline that uses the original data from the source domains to train a single model via Empirical Risk Minimization ( ERM ) [68, 20], a common domain generalization baseline (Question ( c ) ) [Table]
We evaluate DEKAN and the baselines on two DG benchmark datasets, PACS [30] and Digits, which comprises images from MNIST [29], MNIST-M [15], SVHN [51] and USPS [24].
Table 1 shows the results of DEKAN and the baselines.
Hereby, the column name refers to the unseen target domain, i.e., the 3 other domains are the source domains used to train the teacher models.
The test accuracy is computed on the test set of the target domain.
DEKAN outperforms all data-free baselines on both datasets on average, setting a first state-of-the-art performance for the novel DFDG problem.
We find that generative approaches, i.e., Multi-DI and
DEKAN, outperform the ensemble methods on average, suggesting that training a single model on data from different domains enables a better aggregation of knowledge than the aggregation of domain-specific model predictions. Most importantly, DEKAN substantially outperforms Multi-DI, highlighting the importance of the synthesized cross-domain images.
This is especially the case for the challenging domains, i.e., the domains where all the methods yield the lowest performance. In particular, the generation of cross-domain synthetic data leads to performance improvements of 6 . 4% and 3 . 8% on the Sketch and Cartoon PACS domains respectively, as well as a 2 . 4% increase on the
SVHN domain of Digits.
Additionally, we note the positive knowledge transfer across domains on the PACS dataset, as all the multi-domain methods outperform the oracle BestTeacher baseline that uses a single domain-specific teacher model, i.e., the teacher that achieves the highest performance on a validation set from the target domain.
Finally, it is worth noting that while DEKAN significantly reduces the gap between the best data-free baseline and the upper-bound baseline that uses the original data, there is still potential for improvement.
Conclusion
This work addressed the unstudied intersection of domain generalization and data-free learning, a practical setting where a model robust to domain shifts is needed and the available models were trained on the same task but with data from different domains.
We proposed DEKAN, an approach that fuses domain-specific knowledge from the available teacher models into a single student model that can generalize to data from a priori unknown domains.
Our empirical evaluation demonstrated the effectiveness of our method which outperformed ensemble and data-free knowledge distillation baselines, hence achieving first state-of-the-art results in the novel and challenging data-free domain generalization problem.
A Related Work
Our method addresses the Data-Free Domain Generalization (DFDG) problem.
To the best of our knowledge, we are the first to address this problem.
In the following, we discuss approaches to related problem settings.
A.1 Domain Generalization
Domain Generalization (DG) approaches can be broadly classified into three categories.
Domain alignment methods attempt to learn a domain-invariant representation of the data from the source domains by regularizing the learning objective.
Variants of such a regularization include the minimization across the source domains of the maximum mean discrepancy criteria (MMD)
[19, 32], the minimization of a distance metric between the domain-specific means [67] or covariance matrices [65], the minimization of a contrastive loss [47, 79, 41, 26], or the maximization of loss gradient alignment [61, 59].
Other works use adversarial training with a domain discriminator model [16, 34] for the same purpose.
Another category of works leverages meta-learning techniques, e.g., the bi-level optimization scheme proposed in [12], to optimize for quick adaptation to different domains [31], or to learn how to regularize the output layer [4].
A combination of meta-learning and embedding space regularization is proposed in [10].
Another line of works augment the training data to tackle DG.
On the one hand, some approaches perturb the source domain data by computing inter-domain examples [74, 76, 72] via Mixup [82], by randomizing the style of images [49], by computing adversarial examples [18] using a class classifier [63, 69, 55] or a domain classifier [60], or corrupting learned features to incentivize new feature discovery [14].
On the other hand, CNNs are trained to generate new images from the source domains [56, 64, 6] or from novel domains [43, 87].
Other works perturb intermediate representations of the data [23, 88, 14].
We refer to [86] for a more extensive overview of DG approaches.
Unlike standard DG approaches that require access to the source domain datasets, our method merges the domain-specific knowledge from models trained on the source domains into a single model resilient to domain shift, while preserving data privacy.
A.2 Knowledge Distillation
Knowledge distillation (KD)
[22] was originally proposed to compress the knowledge of a large teacher network into a smaller student network.
Several KD extensions and improvements [58, 81, 75, 2, 53] enabled its application to a variety of scenarios including quantization [45, 54], domain adaptation [84, 85], semantic segmentation [38], and few-shot learning [57, 8].
While these methods rely on the original data, Data-Free Knowledge Distillation (DFKD) methods were recently developed [39, 44, 50, 7].
Hereby, knowledge is transferred from one [44, 50, 7, 9, 78, 40, 83] or multiple [36] teacher(s) to the student model via the generation of synthetic data, either by optimizing random noise examples [50, 78, 83] or by training a generator network [44, 7, 9, 40].
Nevertheless, the aforementioned DFKD methods focus on scenarios without any domain shift, i.e. the student is evaluated on examples from the same data distribution used for training the teacher.
In the DFDG problem setting we address, the student is trained from multiple teachers that are trained on different source domains in a way that enables generalization to data from unseen target domains.
We propose a baseline that extends the usage of a recent DFKD method [83] to the DFDG setting, and compare it to our approach (Section 3).
A.3 Source-free domain adaptation
The recently addressed Source-Free Domain Adaptation problem [37, 33, 28] assumes access to one or multiple model(s) trained on the source domains, as well as data examples from a specific target domain.
Proposed approaches to tackle it include the combination of generative models with a regularization loss [33], a feature alignment mechanism [77], or a weighting of the target domain samples by their similarity to the source domain [28].
SHOT [37] employs an information maximization loss along with a self-supervised pseudo-labeling, and is extended to the multi-source scenario via source model weighting [1].
BUFR [11] aligns the target domain feature distribution with the one from the source domain.
Another line of works leverage Batch Normalization (BN) [25] layers by replacing the BN-statistics computed on the source domain with those computed on the target domain [35], or by training the BN-parameters on the target domain via entropy minimization [70].
While these approaches rely on the availability of data from a known target domain, we address the DFDG scenario where the model is expected to generalize to a priori unknown target domain(s) without any modification or exposure to their data.
We also note that some methods [28, 37, 11] modify the training procedure on the source domain, which would not be possible in cases where the data is not accessible anymore.
B More details about DEKAN
In the following, we introduce the first DEKAN stage in more detail.
The second and third stages are described in Section 2.2.
DEKAN’s training procedure is described in Algorithm 1.
B.0.1 Intra-Domain Data-Free Knowledge Extraction
In this stage, we extract the domain-specific knowledge from the available teacher models T i separately by generating domain-specific synthetic datasets D ig .
For this, we apply [83], an improved version of the data-free knowledge distillation method DeepInversion (DI) [78] that enables the generation of more diverse images.
Hereby, we use inceptionism-style [46] image synthesis, also called DeepDream, i.e., we initialize random noise images ˆ x and optimize them to be recognized as a sample from a pre-defined class by a trained model.
This process is also referred to as Inversion [13, 78].
Following [78, 83], uniformly sample labels y and optimize the corresponding random images ˆ x by minimizing the domain-specific inversion loss L DS given by
[Equation]
where L C denotes the classification loss, e.g., cross-entropy, L R an image prior regularization, L M a feature moment matching loss, and λ 1 and λ 2 weighting coefficients.
L R penalizes the l 2 -norm and the total variation of the image to ensure the convergence to valid natural images [42, 52, 46, 78].
L M , also called moment matching loss [40], optimizes the synthetic images so that their feature distributions captured by batch normalization (BN) layers match those of the real data used to train the teacher model.
Formally,
[Equation]
L M minimizes the l 2 -norm between the BN-statistics of the synthetic data, i.e., mean µ l (ˆ x ) and variance σ 2 l (ˆ x ) , and those stored in the trained teacher model, ˆ µ l and ˆ σ l 2 , at each BN layer l [78].
In order to increase the diversity of the generated images, we relax this optimization by allowing the BN-statistics computed on the synthetic images to deviate from those stored in the model within certain margins, as introduced in [83].
These deviation margins are defined by relaxation constants for mean and variance, denoted by δ l and γ l respectively.
The latter are computed as the ϵ DS percentile of the distribution of differences between the stored BN-statistics and those computed using random images, as proposed in [83].
We note that the higher the value of the hyperparameter ϵ DS , the higher the relaxation.
We apply this data-free inversion step to each domain-specific model T i separately, yielding domainspecific synthetic datasets D ig that are correctly classified by their respective model and match the distribution of the features extracted by it.
B.0.2 Algorithm
Algorithm 1 summarizes the 3 stages of the DEKAN’s training procedure.
We note that the updates of the syntehtic data and the student model parameters θ are performed using gradient-based optimization, specifically Adam [27] in our case.
Explicit update rule formulas and iteration over the synthetic data batches are omitted for simplicity of notation.
Require: T 1 ..
I : I Domain-specific teacher models //
First stage: Intra-Domain Knowledge Extraction 1: for i ← 1 to I do 2:
Initialize the domain-specific synthetic dataset D ig : Images ˆ x ∼ N (0, I ) and arbitrary labels 3: while not converged do 4:
Update D ig by minimizing the domains-specific inversion loss L DS (Eq. 4) using T i 5: end while 6: end for //
Second stage: Cross-Domain Knowledge Fusion 7: for i ← 1 to I do 8: for j ← 1 to I and i ̸
= j do 9:
Initialize the cross-domain synthetic dataset D ijg : Images ˆ x ∼ N (0, I ) and arbitrary labels 10: while not converged do 11:
Update D ijg by minimizing the cross-domain inversion loss L ijCD (Eq. 1) using T i , T j and D jg 12: end while 13: end for 14: end for // Third stage: Multi-Domain Knowledge Distillation 15: Initialize the student model S θ randomly or from a pre-trained model 16:
Concatenate the domain-specific and cross-domain synthetic datasets into one dataset D g 17: while not converged do 18:
Randomly sample a mini-batch B = { ˆ x, y } from D g 19:
Update θ by minimizing the knowledge distillation loss L KD (Eq. 3) using B and T 1 ..
I 20: end while 21: return Domain-generalized student model S θ