L EARNING TO G ENERALIZE ACROSS D OMAINS ON S INGLE T EST S AMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
We strive to learn a model from a set of source domains that generalizes well to unseen target domains.
The main challenge in such a domain generalization scenario is the unavailability of any target domain data during training, resulting in the learned model not being explicitly adapted to the unseen target domains.
We propose learning to generalize across domains on single test samples.
We leverage a meta-learning paradigm to learn our model to acquire the ability of adaptation with single samples at training time so as to further adapt itself to each single test sample at test time.
We formulate the adaptation to the single test sample as a variational Bayesian inference problem, which incorporates the test sample as a conditional into the generation of model parameters.
The adaptation to each test sample requires only one feed-forward computation at test time without any ﬁnetuning or self-supervised training on additional data from the unseen domains.
We conduct extensive ablation studies to demonstrate the effectiveness and advantages.
Further, our model achieves at least comparable – and often better – performance than state-of-the-art methods on multiple benchmarks for domain generalization 1 .
I NTRODUCTION
Despite their widespread adoption and success in academia and industry alike, deep convolutional neural networks suffer from a fundamental ﬂaw: they have insufﬁcient generalizability to test data that is out of their training distributions (Recht et al., 2019).
Improving the generalization of machine learning methods therefore remains a challenging problem (Moreno-Torres et al., 2012; Recht et al., 2019; Krueger et al., 2021).
To deal with the distribution shift, domain adaptation (Saenko et al., 2010;
Long et al., 2015; Lu et al., 2020; Li et al., 2021) and domain generalization (Muandet et al., 2013; Li et al., 2017; 2020) have been extensively investigated.
In domain adaptation, the key assumption is that the target data, either labeled or unlabeled, is accessible during training , which allows the model to be adapted to the target domains.
However, this assumption does not necessarily hold in realistic scenarios where the target domain data is not available.
By contrast, domain generalization strives to learn a model on source domains that can generalize well to unseen target domains without any access to the target domain data during training.
We aim for domain generalization.
Previous domain generalization approaches have successfully explored domain-invariant learning (Muandet et al., 2013; Motiian et al., 2017b;
Dou et al., 2019; Chattopadhyay et al., 2020) or domain augmentation (Shankar et al., 2018; Qiao et al., 2020;
Zhou et al., 2020a) to handle the unknown domain shift between the source domains and target domains.
However, since those models are trained on source domains, there will always be an “adaptivity gap” when applying them to target domains without further adaptation (Dubey et al., 2021).
Thus, it is necessary to train models that are able to further adapt to target domains without ever using target data during training.
Recently, further adapting a pre-trained model using target domain data by ﬁne-tuning or selfsupervised training at test time has shown effectiveness in improving performance on domain generalization (D’Innocente et al., 2019; Sun et al., 2020; Wang et al., 2021).
To achieve proper adaptation to the target domain, these methods typically rely on extra ﬁne-tuning operations on target data or extra networks (Dubey et al., 2021) at test time. Although some of these methods can achieve adaptation with a single test sample (D’Innocente et al., 2019; Sun et al., 2020; Banerjee et al., 2021),
[Table]
batches of target data are required for good performance. This prevents these methods from being able to effectively adapt to the target domain when provided with very few target samples or when exposed to test samples from multiple target domains without domain identiﬁers.
Different from these works, we aim to learn the ability of adaptation to a single sample such that the model can further adapt to each test sample from any target domain
[Table]
to each single target sample. In this paper, we propose learning to generalize on single test samples for domain generalization.
Using metalearning (Li et al., 2017), we train our model to acquire the ability to adapt with each single sample on source domains such that it can then adapt to each individual sample in the target domain to handle the domain shift at test time.
The rationale is that a single target sample provides target information that can be used to achieve the model adaptation to the sample itself. Hence, our method adapts the model to each individual target sample, rather than the entire target domain. An illustration of the method is shown in Figure 1.
Unlike previous methods, our model does not need to be ﬁne-tuned using target data or train extra networks when generalized to the unseen target domains.
Once it is trained on source domains, our model is able to adapt to each single target sample with just a feed-forward computation at test time, without the need for any extra target data.
Note that not a single target sample is seen during training, which does not forgo the strict source and target separation setting of domain generalization.
Moreover, in contrast to previous test-time adaptation methods (Sun et al., 2020; Wang et al., 2021; Dubey et al., 2021), our method does not need to re-adapt the model to each target domain.
Our model effortlessly handles test samples from multiple domains without knowing their domain identiﬁers.
To be speciﬁc, we build our model under the meta-learning framework and formulate the single test sample generalization as a variational inference problem.
In the training stage, we divide the source domains into several meta-source domains and a meta-target domain and explore the adaptive model by incorporating the information of the meta-target sample into the generation of the model parameters.
For any given meta-target sample, we propose a variational distribution generated by this sample and the meta-source data to approximate the model distribution obtained by the meta-target data.
In the test phase, the adapted models for the samples from the real target domains are generated on the ﬂy by the variational model distribution.
The random splits of the meta-source and meta-target domains expose the model to domain shifts and mimic the real generalization process from source domains to the target domain.
Thus, the model is endowed with the ability to adapt to any unseen sample by end-to-end training with the data from source domains only.
By doing so, our method does not need to introduce any extra target data or ﬁne-tuning operations on the target domain.
A comparison of the training/test-time settings among different domain generalization (and domain adaptation) methods is summarized in Table 1.
We demonstrate the effectiveness of our single test sample generalization by conducting experiments on commonly-used benchmarks for domain generalization.
Our ablation studies under different domain generalization settings demonstrate the advantages of our method in comparison to other recent adaptive methods.
R ELATED W ORK
Domain adaptation To handle the domain shift between source and target domains, considerable efforts have been devoted to developing methods for domain adaptation (Long et al., 2015;
Lu et al., 2020; Hoffman et al., 2018; Kumar et al., 2010; Tzeng et al., 2017; Luo et al., 2019).
However, the assumption that target data is accessible during training is often invalid.
Recently, some even more challenging settings were proposed in domain adaptation, e.g., few-shot (Motiian et al., 2017a) and one-shot domain adaptation (Dong & Xing, 2018;
Luo et al., 2020), but there are still a few or one target sample available during training.
In this work, we focus on the more strict setting, domain generalization, that assumes not a single target sample is accessible during training. We construct our model under a meta-learning framework to learn the ability of adaptation to single samples across (at least two) source domains. At test time, no more learning is needed and we generate an adapted model for each individual target sample by only a forward pass, without any extra ﬁne-tuning operation.
Domain generalization The problem of domain generalization was introduced by Blanchard et al.
(2011), and formally posed by Muandet et al.
Muandet et al.
and Ghifary et al.
both considered domain-invariant learning by minimizing the dissimilarity of features across source domains.
Following these works, many methods explored domain invariant learning for domain generalization (Motiian et al., 2017b; Rojas-Carulla et al., 2018; Li et al., 2018b; Seo et al., 2020;
Zhao et al., 2020).
Alternatively, augmenting the source domain data has been explored to simulate more domain shift during training (Shankar et al., 2018;
Volpi et al., 2018; Zhou et al., 2020a; Qiao et al., 2020;
Zhou et al., 2020b).
Our method tackles domain shift by learning to adapt to each unseen target sample, which is orthogonal to domain-invariant and domain augmentation methods.
Domain meta-learning Meta-learning methods have also been studied for domain generalization (Balaji et al., 2018; Du et al., 2020; Xiao et al., 2021; Du et al., 2021).
Li et al.
(2018a) introduced the model agnostic meta-learning
(Finn et al., 2017) into domain generalization.
Dou et al.
(2019) further enhanced the model agnostic meta-learning with global and local regularization.
We follow the experimental settings of previous methods that split source domains into meta-source and metatarget domains.
By mimicking the adaptation process at training time, our model learns to learn the adaptable classiﬁers from meta-source domains and the meta-target sample.
Test-time adaptation Recently, several methods have proposed to adapt a model trained on source data to unseen target data at test time (D’Innocente et al., 2019;
Sun et al., 2020; Wang et al., 2021; Dubey et al., 2021).
Sun et al.
(2020) trained their model on source domains with both supervised and self-supervised objective functions, and then ﬁne-tuned it on target data with only the self-supervised one.
Wang et al.
(2021) further improved this method by removing the self-supervised module, while ﬁne-tuning the model with an entropy minimization loss.
Differently, Dubey et al.
(2021) trained a domain embedding network with unsupervised source data, which is utilized to generate the target domain embedding by hundreds of unlabeled target samples to adapt the model at test time, without ﬁne-tuning or further training.
These adaptive methods all need extra steps, e.g., ﬁne-tuning the model or training embedding networks, and batches of target samples for good adaptation.
In contrast, our method learns to adapt to each target sample using just its own information without any extra target samples and ﬁne-tuning operations.
The adaptation is achieved by generating the model for each target sample with only one forward pass.
Another recent work utilized test-time augmentation to improve robustness at test time (Chai et al., 2021).
It will also be interesting if the model augments the test-time sample to source domains, e.g., adapting the target sample to source space, while our method aims to adapt the model trained by source domains to each target sample.
M ETHODOLOGY
In domain generalization, several domains are deﬁned as different data distributions in the joint space X × Y , where X and Y denote the input space and the label space.
The domains are divided into non-overlapping source domains, collectively denoted as S and the target domains T .
During training, only data from source domains is accessible.
Data from target domains is never seen.
Our goal is to learn a model on the source domains only that generalizes well to the target domains.
As we focus on homogeneous domain generalization (Zhou et al., 2021) in this work, all domains share the same label space.
Since the target data is inaccessible during training, we design a meta-learning framework with episodic training (Li et al., 2019; Balaji et al., 2018) that endows the model with the capability of learning to generalize on single test samples from any unseen domain.
To this end, we generate the source domains S and target domain T according to the domain annotations and select one of the available source domains as the meta-target domain T (cid:48) in each iteration, while treating the others as the meta-source domains S (cid:48) . In this case, the model is exposed to different domain shifts to learn the ability to handle it.
After training, the learned model is applied to the target domain T for evaluation.
Variational learning We develop our model under the probabilistic framework.
Assume we have a single sample x t (cid:48) from the meta-target domain T (cid:48) , and we would like to predict its class label y t (cid:48) .
We consider the conditional predictive log-likelihood log p ( y t (cid:48) | x t (cid:48) , T (cid:48) ) . By incorporating the model θ t (cid:48) into the prediction distribution, we have
[Equation]
where we condition the prediction on T (cid:48) to leverage information from the meta-target data and ( x t
[Equation]
, y t (cid:48) ) denotes the input-output pair of the meta-target sample.
We refer to p ( θ t (cid:48) |T (cid:48) )
as the meta-prior distribution since it is conditioned on the meta-target data.
However, in practice, the labeled data from the target domain is inaccessible, leading to an intractable distribution p ( θ t |T ) during inference.
To address the problem, we propose to approximate the model function p ( θ t |T ) with the accessible source data by the variational inference approach, which provides a framework to approximate intractable distributions via optimization.
By training under the meta-learning framework, our method will learn the ability to approximate the model functions of unseen target domains by source domain data.
Variational domain generalization We introduce a variational distribution q ( θ t (cid:48) |S (cid:48) ) to approximate the true posterior over the model parameters θ .
By incorporating q ( θ t (cid:48) |S (cid:48) ) into eq.
(1), we derive a lower bound of the conditional predictive log-likelihood:
[Equation]
The introduced Kullback-Leibler (KL) divergence term is minimized to reduce the distance between q ( θ t (cid:48) |S (cid:48) ) and p ( θ t (cid:48) |T (cid:48) ) , which encourages the model θ inferred by q ( θ t (cid:48) |S (cid:48) ) from meta-source domains to be generalizable to the meta-target domain.
Although eq.
(2) reduces the difference between the variational distributions q ( θ t (cid:48) |S (cid:48) ) and p ( θ t (cid:48) |T (cid:48) ) , q ( θ t (cid:48) |S (cid:48) ) still has limited capacity of handling the meta-target samples due to “adaptivity gap” (Dubey et al., 2021).
Since only little information about the meta-target domain T (cid:48) and the meta-target sample x t (cid:48) is available in S (cid:48) , there is no guarantee that the variational distribution q ( θ t (cid:48) |S (cid:48) ) will be adapted to the meta-target domain T (cid:48) .
To reduce the “adaptivity gap”, the model needs to acquire the ability to effectively use the target information.
Thus, we further propose to learn to adapt the model θ t (cid:48) by taking more information of the meta-target data into account.
Single test sample generalization In domain generalization, especially for real applications, when given data from an unseen target domain, the only accessible information of the target data is from the target sample, without any annotations.
We therefore propose to utilize the information contained in the given target sample to construct the adapted model.
Instead of utilizing q ( θ t (cid:48) |S (cid:48) )
as the variational distribution, we incorporate the meta-target sample x t (cid:48) into the variational distribution q ( θ t (cid:48) | x t (cid:48) , S (cid:48) ) to approximate the true posterior distribution.
Given an unlabeled sample x t (cid:48) from the meta-target domain T (cid:48) , we have a new evidence lower bound (ELBO) as follows:
[Equation]
By conditioning on the given sample x t (cid:48) and the source data S (cid:48) , the adapted variational distribution q ( θ t (cid:48) | x t (cid:48) , S (cid:48) ) contains both the categorical information provided by the meta-source data and the domain information given by the meta-target sample.
This to a large extent guarantees the inferred model to be discriminative and adapted to the given meta-target domain sample.
Moreover, the KL divergence in eq.
(3) further acts as a regularizer to guide the variational distribution to be similar to the parameter distribution p ( θ t (cid:48) |T (cid:48) )
obtained through the meta-target data, which is assumed to be the most suitable parameter for the meta-target sample x t (cid:48) .
This makes the model better adapted to the sample from the meta-target domain.
Based on the meta-learning framework and eq.
(3), our model learns the ability to adapt the metasource model to each meta-target instance across different domain shifts. The learned ability is exploited to handle domain shifts at test time by adapting the source model to each target instance.
During inference, our method directly uses the variational distributions q ( θ t | x t , S ) as the adapted model generated by the information of the given target sample x t and the data from all source domains S .
Thus, the method is able to generalize to each unseen target sample without the requirement of any extra data from the target domains or ﬁne-tuning operations.
The derivations of eq.
(2) and eq.
(3), as well as more discussions on their tightness are provided in Appendix A.
Learning to adapt classiﬁers
The variational inference framework in eq.
(3) indicates that we can learn to adapt all model parameters θ to the target sample with its own information.
This could, however, be computationally infeasible due to the large number of parameters in the network.
Thus, for computational efﬁciency and feasibility we utilize the features extracted by the backbone network as the data information to construct an adapted classiﬁer, which contains much fewer parameters.
To do so, we divide the model θ into a feature extractor φ and a classiﬁer w .
The feature extractor is shared across domains, while the classiﬁer is trained to be adapted to each single sample by eq.
Both the meta-prior distribution p ( w t (cid:48) |T (cid:48) ) and the variational posterior distribution q ( w t (cid:48) | x t (cid:48) , S (cid:48) ) of the classiﬁer w t (cid:48) are generated by amortized inference using the amortization technique (Kingma & Welling, 2013; Gershman & Goodman, 2014; Gordon et al., 2019).
The amortization technique provides a natural way to generate model parameters by feature representations, which is one appropriate solution for our method to incorporate the target feature into the construction of the adapted variational posterior distribution.
Moreover, the amortization networks can be trained and evaluated together with the model, without introducing extra training steps and ﬁne-tuning operations to achieve the adaptation ability.
Speciﬁcally, we use the center features of samples in each class from the meta-target domain and meta-source domains as the representatives of T (cid:48) and S (cid:48) .
We generate the meta-prior distribution p ( w t (cid:48) |T (cid:48) ) by taking the center features of T (cid:48) as input to the amortized inference network θ a .
For the variational distribution q ( w t (cid:48) | x t (cid:48) , S (cid:48) ) , the input to the amortized inference network is the concatenation of the center features of the meta-source domain S (cid:48) with the features of the target sample x t (cid:48) for adaptation.
The amortized inference network θ a is shared for the inference of both the meta-prior and posterior distributions.
However, the direct concatenation operation is not necessarily optimal as the input for the amortized inference network.
We propose to generate the parameter distribution using the meta-source domains and then adapt it with the given meta-target sample.
To do so, we ﬁrst generate a classiﬁer distribution p ( w s (cid:48) |S (cid:48) ) using the center features of the meta-source data S (cid:48) .
After that, we take p ( w s (cid:48) |S (cid:48) ) as a prior and combine it with the meta-target sample to estimate the variational posterior, as follows:
[Equation]
This establishes a hierarchical variational inference of the posterior distribution.
We show in our experiments that the hierarchical inference yields better results than the direct concatenation.
In practice, we approximate the integral by:
[Equation]
where w ( (cid:96) ) s (cid:48)
∼ p ( w s (cid:48) |S (cid:48) ) and L is the number of Monte Carlo samples.
The negative KL divergence term in eq.
(3) tries to close the gap between the variational adaptive distribution q ( w t (cid:48) | x t (cid:48) , S (cid:48) ) and the meta-prior distribution p ( w t (cid:48) |T (cid:48) ) , aiming to make the variational distribution adapt better to the target data.
However, there is no guarantee that the classiﬁer produced by the meta-prior is discriminative enough for classiﬁcation.
To address this issue, we introduce an intermediate supervision based on the meta-prior distribution to make it as predictive as possible, by maximizing:
[Equation]
This is also implemented as a cross-entropy loss using Monte Carlo sampling from the meta-prior.
An illustration of our architecture is provided in Figure 2.
We also provide an algorithm in Appendix C.
Empirical objective function By incorporating eq.
(5) and eq.
(6) into eq.
(3), we obtain the ﬁnal empirical objective function:
[Equation]
where we use Monte Carlo sampling: w [Equation] m
[Equation]
t
[Equation]
As illustrated in Figure 3, at training time (a), we train the model episodically by randomly dividing the source domains S into meta-source domains S (cid:48) and a meta-target domain T (cid:48) .
At inference time (b), given a single sample x t from the target domain T , we use this sample to adapt the w s generated by the source domains.
The adaptation is achieved by generating the adapted classiﬁers w t for each target sample with only one forward pass using the learned amortization inference network θ a .
E XPERIMENTS
Datasets We conduct our experiments on four widely used benchmarks for domain generalization: PACS, Ofﬁce-Home, rotated MNIST, and Fashion-MNIST. We report the mean classiﬁcation accuracy on target domains.
PACS (Li et al., 2017) is a widely used dataset consisting of 9,991 images of seven classes from four domains, i.e., photo , art-painting , cartoon and sketch .
We use the same training and validation split as in (Li et al., 2017) and follow the “leave-one-out” protocol from (Li et al., 2017; 2019; Carlucci et al., 2019).
Ofﬁce-Home (Venkateswara et al., 2017) has 15,500 images ofcategories.
The images are also from four domains, i.e., art , clipart , product and real-world .
We use the same experimental protocol as for PACS.
Rotated MNIST and Fashion-MNIST are utilized in
Piratla et al.
We follow their recommended settings and randomly select a subset of 2, images from MNIST and 10,000 images from Fashion-MNIST.
The subset of images is rotated by 0 ◦ through 90 ◦ in intervals of 15 ◦ , creating seven domains.
We use the subsets with rotation angles from 15 ◦ to 75 ◦ as the source domains, and images rotated by 0 ◦ and 90 ◦ as the target domains.
For comparison, we evaluate our method on both in-distribution and out-of-distribution data.
The implementation details on these datasets are provided in Appendix D.
Beneﬁt of single test sample generalization
We ﬁrst investigate the effectiveness of our single test sample generalization on the PACS dataset.
To demonstrate the beneﬁts of all components of our method, we consider four settings: (i) the “variational amortized classiﬁer” with eq.
(2) as the objective function, (ii) our single test sample generalization method without supervision on
[Table]
We plot the test samples and classiﬁers in the same 2D plane.
Different shapes denote different categories.
Samples are in red, classiﬁers in blue.
The baseline method utilizes the same classiﬁer for all samples from each domain, while our method generates an adapted classiﬁer for each target sample.
The two methods use the same samples.
In our single test domain generalization, the test sample is the closest to the adapted classiﬁer of its own class, indicating better classiﬁcation ability.
the meta-prior distribution (eq. 6), (iii) single test sample generalization without the hierarchical architecture (eq. 5), and (iv) the entire single test sample generalization method, with the objective function in eq.
Results based on the ResNet-18 backbone are shown in Table 2.
In Appendix B, we also conduct experiments to show the beneﬁts of our meta-learning settings in Table 5 and 6 and provide some single-source domain experiments in Table 8.
By comparing rows 1 and 3 of Table 2, it is clear that our single test sample generalization brings a considerable improvement in accuracy over the invariant amortized classiﬁer, especially on the “art-painting” and “cartoon” domains.
These results demonstrate the beneﬁts of introducing the information of the given target sample into the model generation.
The comparison also demonstrates that our model learns the capability to adapt to the target sample using its own information.
Comparing rows 2 and 4 shows the beneﬁts of the intermediate supervision in eq.
(6), which provides a more discriminative meta-prior distribution during training to learn more adapted variational distribution, especially on the “art-painting”, “cartoon” and “sketch” domains.
Results in the last two rows of
Table 2 show that the hierarchical architecture for the variational adaptive classiﬁer q ( w t (cid:48) ) provides a more adapted model for each target sample and achieves the best performance on all domains.
Visualization of single test sample generalization To illustrate the beneﬁt of incorporating the test sample into classiﬁer generation, we visualize the classiﬁers of the “variational amortized classiﬁer” (baseline), our classiﬁers adapted by the single target samples, and the target features in Figure 4.
We treat the vectors of different categories in the classiﬁer as prototypes with the same dimension as the test features and map the classiﬁer vectors and features into the same 2D space. In this sense, the distance between classiﬁer vectors and features reﬂects their similarity.
We use the same sample features in the visualizations of the two methods.
As shown in the left ﬁgure (baseline), the classiﬁer parameters of the “invariant amortized classiﬁer” are ﬁxed for each target sample during inference.
Since the samples are out of the training distributions, they are easy to misﬁt the ﬁxed classiﬁer trained on source domains, leading to failed predictions.
In contrast, as shown in the three ﬁgures on the right, our method generates a speciﬁc, adapted classiﬁer for each test target sample.
For instance, the test sample ( ) has the shortest distance to the adapted classiﬁer of its own class ( ), producing the correct predictions.
More visualizations, including failure cases, are provided in Appendix E.
Single test sample generalization vs. Tent 2 Next we conduct an experiment on rotated-MNIST to compare our method with Tent by Wang et al.
(2021), which is a recent test-time adaptation method [Figure] u r a c y A cc u r a c y
[Table] that ﬁne-tunes the model with batches of target samples. We follow the setting in (Piratla et al., 2020) and generalize seven domains by rotating the samples by 0 ◦ to 90 ◦ in intervals of 15 ◦ .
To make the target domains more different from each other, we use images rotated by 0 ◦ , 45 ◦ and 90 ◦ as target domains, 15 ◦ , 30 ◦ , 60 ◦ and 75 ◦ as source domains.
For fair comparisons, we train the base model of Tent using data from all source domains, which is the same amount of train and validation data as in our method.
Then, the trained base model is adapted by the target samples as in the original paper of Tent.
The other settings are the same as our methods. As shown in Figure 5, we design two settings for Tent for the comparisons: “single target domain” (left) and “multiple target domains” (right).
In the “single target domain” setting, we adapt the base model to each of the target domains separately using Tent and take the average accuracy of different domains.
For “multiple target domains”, Tent is required to adapt to all three target domains jointly without the domain identiﬁers.
Since our method is able to adapt to each sample, the performance is consistent under both settings.
As expected, Tent achieves good adaptation for large batch sizes (e.g., 128) from a single target domain.
However, when there are only very few samples, and ultimately just one sample, the adaptation of Tent starts to suffer, and may even hurt accuracy.
Tent, like other test-time adaptive domain generalization methods, relies on a batch of samples to adapt to the target domain.
In other words, the test samples have known domain identiﬁers.
However, in practical settings, the test samples might come from different domains and the domain identiﬁers may be inaccessible.
Thus, we also explore the effectiveness under the “multiple target domains” setting, as shown in Figure(right).
In this scenario, the adaptability of test-time adaptation methods like Tent, starts to decline, even with more samples.
In contrast, our method learns the capability to adapt to each sample from any target domain, overcoming the lack of target samples while being robust under multiple [Table] target domains.
We also compare our method with Tent on PACS and Ofﬁce-Home.
The results are shown in Table 3 and 4.
We perform Tent with a baseline model trained on all source domain data, which is then ﬁne-tuned by 128 target samples for 100 iterations.
The conclusion is the same as on rotated MNIST.
Our method achieves better performance.
One reason can be that our model adapts the model to each individual target sample rather than the entire target domain.
Thus, different adaptations are conducted on different target samples.
Alternatives are sensitive to the chosen batch of samples, which is not necessarily generalizable to the entire domain, especially when the batch is small.
Another reason is our choice for meta-learning. By mimicking the adaptation process under the meta-learning framework, the model learns the ability to adapt to a single target sample. Without meta-learning, as shown in Table 6 in Appendix B, the performance drops. Comparisons with State of the Art Finally, we compare our method with the state-of-the-art on four datasets.
The results on PACS and Ofﬁce-Home are reported in Tables 3 and 4, respectively.
The results on rotated-MNIST and Fashion-MNIST are provided in Table 7 in Appendix B.
PACS In Table 3, we conduct experiments with both ResNet-18 and ResNet-50 backbones.
Our method achieves good performance using the ResNet-18 backbone.
For each individual domain, we are competitive with the state of the art and slightly better on the “cartoon” domain when using
ResNet-18.
On the “art-painting” and “cartoon” domains with ResNet-50, our method also achieves good performance.
Although it delivers competitive, or sometimes even better, performance on most domains, our method is not as good on the “sketch” domain.
One reason might be that the images and features from this domain carry less information than other domains, leading to less adaptive models.
Ofﬁce-Home As shown in Table 4, our method again achieves good overall performance with both the ResNet-18 and ResNet-50 backbones.
With ResNet-18 as the backbone, we outperform other methods by a good margin on the “clipart” domain, while delivering competitive performance on the other domains.
When utilizing ResNet-50 as the backbone, our method achieves good performance on all four domains.
Overall, compared with other methods, our method shows larger advantages with ResNet-50 than ResNet-18 on both PACS and Ofﬁce-Home.
This is reasonable since we generate the adaptive model using the features of the given target sample.
The features extracted by ResNetcontain more information, leading to a more adapted model for the given target sample.
C ONCLUSION AND F UTURE W ORK
We propose learning to generalize by single test samples for domain generalization.
Our model is able to adapt to any unseen target sample by leveraging the information of the given sample to generate the model.
We formulate the adaptation process as a variational inference problem and train the model under the meta-learning framework.
After training on the source data, our method can adapt to each sample from the unseen target domains without any ﬁne-tuning operations or extra target data.
Ablation studies and further experiments on several benchmarks show the effectiveness of our method.
Like other meta-learning-based domain generalization methods, our method utilizes the domain shift between source domains to mimic the shift between source and target domains, which requires at least two source domains during meta-training.
This limits the generalization ability when only one source domain is available.
An appealing solution is to augment the source domain at training time, which we leave as our future work.
E THICS S TATEMENT
With the capability to handle the domain shift without the accessibility of target data during training and ﬁne-tuning operations, our method has potential positive impacts in the applications that often face unseen domains in practice, e.g., medical imaging and automatic driving.
Accordingly, our method would also potentially face some negative social impacts accompanying with applications, e.g., lack of fairness with the model trained by incomplete data and the privacy of patients in medical imaging.
R EPRODUCIBILITY S TATEMENT
To reproduce our method, the details of the method and loss functions are provided in Section 3.
We also provide the derivations and discussions about the evidence lower bound used in our method in
Appendix A.
The datasets used in this paper and the detailed implementations of the method are shown in Appendix D. An illustration of the architecture is shown in Figure 2.
We also provide a link for the code of our method in Abstract.
A D ERIVATIONS
Derivation of variational domain generalization In domain generalization, since the target domain data is inaccessible, the distribution p ( θ t |T ) is intractable.
We leverage the meta-learning setting to mimic the domain generalization process.
In particular, we episodically divide the source domains into meta-source domains S (cid:48) and a meta-target domain T (cid:48) and learn the ability to generalize to the target data.
Thus, during training, we generate the parameter distribution by the meta-source data S (cid:48) to approximate the distribution p ( θ t (cid:48) |T (cid:48) ) generated by the meta-target data.
Then at inference, the model is able to approximate the distribution p ( θ t |T ) with the parameters generated by the source data S .
To do so, we introduce a variational distribution q ( θ t (cid:48) |S (cid:48) ) to approximate the true posterior over the model parameter θ .
The full derivation of eq.
(2) in the main paper is as follows:
[Equation]
Derivation of single test sample generalization Since there is little information available about the target domain or target data in the variational distribution q ( θ t (cid:48) |S (cid:48) ) in eq.
(2), the resultant model is not adaptive to the target domain data.
To generate a model adapted to the target sample
[Equation]
we introduce information of the target data
[Equation]
and incorporate the meta-target sample x t (cid:48) into the variational distribution q ( θ t (cid:48) | x t (cid:48) , S (cid:48) ) .
The detailed derivation is:
[Equation]
Comparison of the two evidence lower bounds The main difference between eq.
(8) and eq.
(9) is the variational posterior q ( θ t (cid:48) ) .
Compared with eq.
(8), our method (eq. 9) introduces the target sample x t (cid:48) into the generation of the variational posterior, which therefore achieves more adaptive classiﬁers for the target sample.
Moreover, the tightness of the bound is also related to the variational posterior.
The approximation gap between our bound (eq. 3) and the objective log-likelihood corresponds to the KL divergence between the variational posterior q ( θ t (cid:48) | x t (cid:48) , S (cid:48) ) and the true posterior p ( θ t (cid:48) | x t (cid:48) , y t (cid:48) , T (cid:48) ) , as shown in the following equation:
[Equation]
Compared with eq.
(2), whose approximation gap is formulated as D KL [ q ( θ t
[Equation]
|S
[Equation]
|| p ( θ
[Equation]
| x t
[Equation]
, y t
[Equation]
, T
[Equation]
)] , our method takes the given sample x t (cid:48) as one of the conditions, leading to a tighter lower bound. [Table]
[Table]
B E XTRA E XPERIMENTS B. A BLATIONS OF M ETA - LEARNING SETTINGS
Our method is constructed under a meta-learning framework and trains the model to learn the ability to adapt with each single sample.
The meta-learning setting mimics the domain shift among source and target domains by domain shift within the source domains.
To achieve that, we divide the source domains into meta-source and meta-target domains according to their domain annotations and explore the adapted model on each single meta-target sample with the meta-source information.
To demonstrate the importance of mimicking the domain shift within source domains and the signiﬁcance of the meta-learning setting, we conduct two extra experiments as shown in Table 5 and Table 6.
The experiments are conducted on PACS with ResNet-18 as the backbone.
To show the importance of mimicking the domain shift in training time, we randomly split the images from all source domains ( D s 1 , D s 2 , D s 3 ) into three new source domains D ns 1 , D ns 2 , and D ns 3 , which means each of the three new source domains D nsi has samples from all of the original source domains ( D s 1 , D s 2 , D s 3 ).
Thus, the domain shift between these new source domains is too small to mimic the domain shift between source and target domains.
We train our method on these new source domains and the results are shown in the ﬁrst row in Table 5.
The performance of random split by images decreases compared with the annotation-based one, especially on “art-painting” and “sketch” domains, indicating the importance of constructing appropriate domain shifts within source domains under the meta-learning framework.
To show the importance of the meta-learning framework, we also implement a non-meta-learning version of our method.
We remove the splitting of meta-source and meta-target domains and directly generate the adapted classiﬁers by features from all source domains and the feature representations of the single sample.
The prediction of source sample x s in training time and target sample x t in test time are as follows:
[Equation]
The results are shown in Table 6.
Without the meta-learning framework, the model is difﬁcult to learn the ability to adapt to unseen domains and tends to overﬁt to the source ones.
Thus the performance on all domains is worse than the meta-learning-based model.
B. C OMPARISON ON ROTATED MNIST AND F ASHION -MNIST
On rotated MNIST and Fashion-MNIST, for fair comparison, we use ResNet-18 as the backbone following (Piratla et al., 2020) and evaluate the method on both in-distribution and out-of-distribution sets.
Our method achieves best performance on both datasets as shown in Table 7, especially for the out-of-distribution setting.
Moreover, our method demonstrates less performance drop from in-distribution to out-of-distribution data in comparison to other methods.
[Table] [Table] [Table] B. S INGLE SOURCE DOMAIN GENERALIZATION
As our method learns the adaptation ability by mimicking the domain shift during training
[Table]
it requires at least two source domains to support the meta-learning scheme, which is a limitation of the method.
We conduct an experiment that uses SVHN as the single source domain and MNIST as the target.
We generate source domains by two simple methods: random image split and clustering of images (K-means).
The results in Table 8 show that our method performs slightly better with larger domain shift simulated in the single source domain.
To learn better adaptation ability, auxiliary methods are needed to generate larger domain shifts, for example by domain augmentation techniques like Qiao et al.
C A LGORITHM OF S INGLE S AMPLE G ENERALIZATION
We describe the the detailed training and test algorithm of our single sample generalization method in Algorithm D I MPLEMENTATION D ETAILS We evaluate our method on PACS with both ResNet-18 and ResNet-50 (He et al., 2016) pretrained on
ImageNet (Deng et al., 2009) as the backbone.
During training, we use Adam optimization and train the model for 10,000 iterations.
The learning rate of the backbone is set to 0.00005 for ResNetand 0.00001 for ResNet-50, while the learning rate of the network for generating the classiﬁer is set to 0.0001 consistently.
The batch size is 128.
To generate the classiﬁer, in each iteration we select 10 samples from each category in each meta-source domain for generating the center features S (cid:48) , and the same number of samples from the meta-target domain for T (cid:48) .
The model with the highest validation accuracy is utilized for evaluation on the target domain.
Most of the experimental settings and hyperparameters on Ofﬁce-Home are the same as on PACS.
Since there are more categories in Ofﬁce-Home, the number of samples per category per domain is set to 5 to ﬁt the memory footprint. The learning rate of the backbone is set to 0.00001 for both ResNet-18 and ResNet-50. For fair comparisons, we evaluate the rotated MNIST and Fashion-MNIST with ResNet-18, following (Piratla et al., 2020).
The other experimental settings are also the same as PACS.
We train all models on an
NVIDIA Tesla V100 GPU.
The detailed information about other hyperparameters are summarized in Table 9. TRAINING TIME [Table]
for
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
=
[Equation]
m = [Equation] CrossEntropy [Equation]
+
[Equation] / Compute loss as eq [Equation] Update parameters
[Table]
[Table]
[Table] descent step end for TEST TIME Require : Source domains S
= { D s } Ss =1 ; target samples { x t } Tt =1 from the target domain T ; trained model with weight Θ = { φ , θ a , ψ } . µ w s , σ w s = f ψ ( S ) ; w s ∼ N ( µ w s , σ w s ) . // Compute p ( w s |S ) µ w ( n ) t , σ w ( n ) t = f θ a ( w s , φ ( x t )) ; w ( n ) t ∼ N ( µ w ( n ) t , σ
[Table] n [Table]
Compute [Table] [Table] ) return y t = w ( n ) t · x t .
[Table]
E V ISUALIZATIONS
More comparisons with baseline To further demonstrate the effectiveness of our method, we provide more visualizations of the classiﬁer parameters and feature representations from different domains in Figure.
The left column shows the visualizations of the “variational amortized classiﬁer” (baseline), and the three columns on the right show our results.
The same conclusion can be drawn here as in the main paper.
Our method is able to generate adapted classiﬁers for each test sample, providing better predictions than the ﬁxed classiﬁers in the baseline method.
Failure cases To gain insights into our method, we also provide some failure cases in Figure 7.
Our method gets confused when samples have objects of different categories in the same image, as shown in the ﬁrst row.
Although failing in these cases, our model provides correct predictions for the other objects in the image.
In the visualizations, the classiﬁer with the same category of the labeled object is also close to the feature, which demonstrates the effectiveness of our method.
In the middle two rows, our method struggles with the samples that contain multiple objects or complex backgrounds. A possible reason is that to take full advantage of the given sample, we utilize multi-level features of the target sample to generate the adapted classiﬁer. However, the complex background and multiple P h
o t o
Baseline
Single test sample generalization
A r t - p a i n t i n g
C a r t oo n
S k e t c h
Classifiers:
Elephant
Dog
Giraffe
Guitar
Horse
House
Person
A dog sample
A giraffe sample
A horse sample
A dog sample
A horse sample
A person sample
A giraffe sample
A person sample An elephant sample
A house sample An elephant sample
[Figure]
sample objects in the image bring too much noisy information, leading to less adapted classiﬁers for the given sample.
A solution can be extracting information of the target sample selectively to reduce noise in the features.
Moreover, in the “sketch” domain, as shown in the last row, the samples usually have less information than other domains.
This makes our method more sensitive to the given information.
When there is noisy information, e.g., the chair in the left ﬁgure, or object with unobvious features, e.g., the elephant in the right ﬁgure, the method fails to generate proper classiﬁers for the sample.
F M ORE COMPARISONS WITH T ENT
We provide more detailed comparisons with Tent Wang et al.
(2021) in Figure 8 to further show the effectiveness of our method.
The three rows in Figure 8 show the accuracy of different settings on 0 ◦ , 45 ◦ , and 90 ◦ target domain, respectively.
The mean accuracy is shown in Figure 5 in the main paper.
The same as in the main paper, the left column shows results under the “single target domain” setting, while the right column shows the “multiple target domains” setting.
In the “single target domain” setting, Tent is utilized to adapt the base model trained on source domains to each target domain independently.
By contrast, in the “multiple target domains” setting, the base model is adapted and evaluated on three target domains jointly, without the domain identiﬁers.
The base model is adapted by Tent using different number of samples, e.g., 1, 32, 128, with different optimization steps, e.g., 1, 10, 100.
The same conclusion can be drawn here as in the main paper.
Compared with Tent, our [Figure] method adapts the model to each target sample individually by a single and efﬁcient feed-forward pass.
Thus, our performance is independent to the settings and optimization steps.
Speciﬁcally, as shown in the left column, Tent achieves better performance with both larger numbers of samples and ﬁne-tuning steps, especially on 0 ◦ and 90 ◦ domains.
However, with fewer target samples being available, and ultimately just one sample, the adaptation of Tent starts to suffer, or even hurt the accuracy with more optimization steps.
When adapted to three target domains jointly, as shown in the right column, the adapted performance is similar with the “single target domain” settings with few target samples.
With more target samples, the performance degrades, worse than the left column.
Accuracy on 0 ◦ domain improves slightly, while on 45 ◦ and 90 ◦ domain even drops as the adaptation is affected by samples from other domains.
In contrast, since our method adapts the model to each target sample individually, we achieve consistently good performance regardless of how many domains the test samples cover. This demonstrates the beneﬁt of our single test generalization.
[Figure]
Number of samples Number of samples
A cc u r a c y A cc u r a c y
Number of samples
Number of samples Number of samples
Number of samples
A cc u r a c y A cc u r a c y A cc u r a c y
A cc u r a c y
[Figure]