Improving Domain Generalization with Interpolation Robustness
Abstract
We address domain generalization (DG) by viewing the underlying distributional shift as performing interpolation between domains.
We devise an algorithm to learn a representation that is robustly invariant under such interpolation and term it as interpolation robustness .
We investigate the failure aspect of DG algorithms when availability of training data is scarce.
Through extensive experiments, we show that our approach significantly outperforms the recent state-of-the-art algorithm,
DIRT [Nguyen et al., 2021] and the baseline DeepAll on average across different sizes of data on PACS and VLCS datasets.
Introduction
Domain generalization (DG) aims at learning to generalize well to an unseen test distribution given data from multiple source distributions.
In general, there are three main approaches to DG [Shen et al., 2021].
The first approach is to learn a domain-invariant representation [Arjovsky et al., 2019, Li et al., 2018, Nguyen et al., 2021] from the training domains.
This stems from the assumption that there exists several properties that are common across all domains.
Learning such a representation captures the essence common to all the domains while simultaneously eliminating any non-generalizable features from individual domains.
The next offer training strategies that can efficiently amalgamate multipledomain-specific networks, including meta learning [Li et al., 2017a] and ensemble learning [Wang et al., 2020a].
Finally, selectively augmenting the training data by generating new data from existing data [Zhou et al., 2021, Yao et al., 2022].
However, in critical applications, available data is often scarce, with limited labels and having shifts in its distribution.
For example, in medical image analysis it is very expensive to get expert annotations or perform intensive experiments to obtain ground truths [Castro et al., 2020].
Furthermore, there are differences in the distribution between data during training versus testing.
Few-shot learning techniques [Song et al., 2022, Fei-Fei et al., 2006, Fink, 2004] have been developed to mitigate the limitation of availability of labelled data.
However, this has not been addressed in the domain generalization arena.
Most existing DG algorithms assume the availability of adequate training data.
They use additional generative models to perform density transformations [Nguyen et al., 2021,
Robey et al., 2021] between source domains to enforce the invariance condition across them.
This approach however quickly becomes unfavourable with less data.
We show with our experiments in a limited data setting that a recent state-of-the-art domain-invariant representation method [Nguyen et al., 2021] performs poorly when applied to a limited data setting.
In this paper, we address the DG problem, in a limited data setting.
Our key perspective is that we view distributional shifts as an interpolation process between domains, and by learning a representation that is invariant and robust under such interpolation, we can improve the domain-invariant representation and the out-of-distribution generalization.
We term this as Interpolation Robustness .
We realize this insight by performing interpolation between latent representations of a pair of inputs from two different domains that have the same class label.
We then enforce every intermediate representation in the interpolation path to have the same class label.
The interpolation operation exposes the model to a novel set of representations that lie between any pair of domains for the same class label.
As interpolation operation yields intermediate representations of an input as its domain gets changed, our objective is to find a representation that remains label-invariant to interpolation between any pair of domains in the training data.
In summary:
• We propose interpolation robustness , a novel perspective to DG that aims at learning a representation that is robustly invariant to interpolation operations between different source domains, creating an effective model-agnostic framework that can be used along with any of the existing DG algorithms that use domain-invariant representation; • We show that our algorithm outperforms state-of-art using PACS and VLCS datasets in DG with limited data.
Related Works
Enforcing label invariance on density transformations between a pair of training domains has been a recent area of research in state of the art works [Nguyen et al., 2021, Robey et al., 2021, Zhou et al., 2020] in DG.
In general, large generative models (GANs) are used to perform this transformation.
Then a classifier is made to classify the images from original domain and the transformed domain to be same.
We hypothesize that the precursor step of training a generative model may not be as effective when the data available to train the generative model is limited.
In contrast, performing interpolation to transform data from one domain to another exposes the model to unseen data that is present in between the interpolation path but does not require training of additional models.
Prior works [Yao et al., 2022, Gong et al., 2019] synthesize intermediate images between two domains to improve generalization by augmenting the interpolated images to the classifier during training.
Instead, we perform interpolation in the latent space and propose a representation that is also robust against the interpolation operation on the images.
Mixup techniques [Wang et al., 2020b, Yan et al., 2020] also perform interpolation among data samples in order to improve generalization.
However, the labels are also linearly interpolated with the data instead of our invariant approach.
Interpolation Robustness
Problem setting.
We consider the standard setting of domain generalization (DG) where multiple labeled source domains D = { ( x 1 , y 1 , d 1 ) , . . .
, ( x N , y N , d N ) } ⊂ R n ×Y ×
[ D ] are available during training, where x i ∈ R n is the input,
y i ∈ Y is the label of x i , d i ∈ [ D ] := { 1 , . . .
, D } is the domain of ( x i , y i ) , and D denotes the number of training domains.
Let P dom be the domain distribution and
P d be the data distribution of domain d .
The goal of a DG learner is to learn from D such that it generalizes well to an unseen data { ( ¯ x i , ¯ y i ) } i ∈
[ M ] in a novel domain that is different from any of the training domains.
Let N d and n y be the total number of samples in each domain d ∈
[ D ] and in each class y ∈ Y respectively.
Our key idea is to learn a representation that is invariant under the interpolation operation under any pair of source domains.
We assert that the mechanism behind the distribution shift between any pairs of domains is the interpolation operation.
We argue that a model that is robust against interpolation between the training domains can be generalized to unseen data.
Moreover, if the intermediate images in the path of the interpolation are semantically valid, the model will be exposed to unseen data.
This can be thought of as implicit data augmentation.
In general, natural images lie on a non-convex manifold, making interpolation tricky
[Chen et al., 2019].
Instead of performing the interpolation in the pixel space, we train a neural network to unfurl the image manifold to a latent space, and use that latent space for smooth interpolation.
Having a robust representation to perform classification directly is more efficient than augmenting the interpolated images and re-training the model.
It also doesn’t require generation of high resolution intermediate images that are on the same level as original data.
3.1 Model Design
In this section, we describe our model design to realize interpolation robustness.
It consists of three main components.
Encoder ( E ) .
We consider the encoder as E ϕ : X → Z (with parameter ϕ ).
It maps data from the high-dimensional input space into a low-dimensional latent space.
E must create the latent representation space Z such that the representations z obtained from inputs that share the same label should lie clustered together and z ’s obtained from inputs having different labels should lie spread apart.
This is achieved with the help of the classifier module below.
Classifier ( C ) .
We consider the classifier as
C θ : Z → Y which is parameterized by θ .
C θ ( z ) predicts the class of all samples x that produce the latent representation z via the encoder E ϕ .
In order for C to associate the representation z to its correct label, we minimize the following loss function:
[Equation]
where L : Y × Y → R is the training loss (e.g: cross entropy) and the expectation is taken with respect to x ∼ P d ( x | y ) , y ∼ P d ( y ) , d ∼ P dom .
Interpolator ( Z ) .
The interpolator module takes as input any two instances x and x ′ and outputs a latent representation
Z ( x , x ′ ; ϕ, w ) :=
E ϕ ( x ) + w ( E ϕ
( x ′ )
− E ϕ ( x )) that is interpolated from the latent representation z
=
E ϕ ( x ) and z ′ =
E ϕ ( x ′ ) of the two inputs, respectively.
Here w ∈ [0 , 1] is the interpolation weight.
Given two instances x and x ′ of the same class y , we force the interpolated latent samples { Z ( x , x ′ ; ϕ, w ) } w ∈ [0 , 1] to be class-invariant under the interpolation, i.e.
{ Z ( x , x ′ ; ϕ, w ) } w ∈ [0 , 1] should yield the same label y as its reference points x and x ′ .
To encode such behavior, we define the linear interpolation loss (INT).
[Equation]
where the expectation is taken with respect to
[Equation]
Overall, our final loss function is:
[Equation]
where λ is the regularization hyperparameter.
We also include a parametric interpolation within the latent space inspired from [Chen et al., 2019].
This leads to an additional module T ψ such that: Z ( x , x ′ ; ϕ, w, ψ ) :
= E ϕ ( x ) + wT ψ ( E ϕ ( x ′ ) −
E ϕ ( x )) .
Here, T ψ : Z → Z is parameterized by ψ .
It is trained jointly with E and C to minimize the loss in equation (2).
In order for the traversal to be a valid interpolation, the interpolated sample obtained when w = 1 should be the same as the target E ϕ ( x ′ ) .
This constraint is also added to the loss to get the parametric version of equation (2) below.
[Equation]
Experiments
We perform our experiments on PACS [Li et al., 2017b] and VLCS
[Ghifary et al., 2015] datasets.
PACS dataset consists of a total 9,991 images of 7 different classes from four domains: art painting (A), cartoon (C), photo (P), sketch (S).
VLCS contains 10, [Table]
4.1 Experimental Settings
Sampling for Limited Data Training data is modified into limited data settings via two ways.
In the uniform sampling approach, given a particular (class, domain) pair, a fixed number n i ( n i <
N d , ∀ d ∈
[ D ] , n i < n y , ∀ y ∈
[ Y ] ) is sampled from each such pairs.
Samples in the new training set will be uniformly distributed among domains and classes.
As most real data does not occur that way, we also follow proportional sampling approach that doesn’t modify P d ( y ) during the sampling process.
We split the original training data ( 100% ) into a subset of size n j while keeping P d ( y ) intact.
Validation and test data remain the same.
Implementation Details We chose E as Resnet-18 [He et al., 2016] network for PACS and
Alexnet [Krizhevsky et al., 2012] for VLCS following DIRT.
C is a single layer dense network preceded by a ReLU layer activation.
T is a 3 layer convolutional neural network.
We follow the experimental set up of DIRT in assigning the hyperparameters of the networks which are dimension of Z : 256, learning algorithm:
SGD, learning rate: 0.001, momentum: 0.9, minibatch size: 64, weight decay: 0.001 and λ : 1.
We train every method for 100 epochs and report the average overdifferent seeds.
Baselines We choose one of the baselines to be DeepAll which is one of the commonly used baselines in DG literature [Carlucci et al., 2019, Dou et al., 2019, Nguyen et al., 2021] and uses only equation (1) while training on the aggregated dataset of all training domains.
Then we consider
DIRT, the model proposed in [Nguyen et al., 2021] which enforces invariance on z via domain transformations in the form of an additional regularization term to DeepAll.
We introduce our models as : DNT and DRINT.
DNT: DeepAll+INT in which loss (INT: 2) is augmented to the DeepAll model.
The next model DRINT:
DIRT+INT is where INT is augmented to the DIRT model.
We demonstrate that on an average, presence of INT in both the models increase their performance.
4.2 Experimental Results
We use leave-one-domain-out evaluation scheme where one of the domains is chosen as the target domain and kept aside for evaluation while the model is trained on the aggregated set of other domains.
Results on the individual domains are provided in Tables 4, 5, 7, 6 in Appendix A.2.
Mean accuracy and standard error over 3 runs are reported.
We report the average prediction accuracy obtained over all the domains in Table 1 for VLCS and PACS.
Entries under the column 100% 1 correspond to the accuracy obtained for the complete training data.
We chose n i : 70 , 50 & 20 for uniform sampling approach for PACS.
For VLCS we have n i : 20 .
As the dataset size is reduced across the columns, there is an increase in accuracy from under 1% to over 3% between DIRT & DRINT and DeepAll & DNT for both datasets.
We have repeated the experiments with the proportional sampling approach.
We split the datasets into 20% , 10% , 5% ( n j ) of its original size while keeping the distribution of its classes and domains unchanged.
As we move towards the right over columns under n j , we can observe an increase in accuracy from under 1% to over 3% between DIRT & DRINT for both datasets.
DeepAll has slightly better results at first, but DNT fares better with limited data.
Results in Tableare from linear interpolation. We provide results on parametric interpolation in Table 2.
We note an improvement of roughly 0 .
1% for DNT and 0 .
7% for ERINT over linear interpolation on PACS.
[Table]
Conclusion
We introduce a framework for learning a representation that is invariant to the interpolation operation.
We demonstrate that the learned representation is robust to domain generalization in comparison to
DeepAll and DIRT.
Additionally, we show that the efficacy of our method increases compared to the baseline methods as there is further shortage in the training data
A Appendix A.1 Visualization of the representation
In this section we visualize the 256 dimensional latent space Z in a 2 D space using TSNE algorithm.
Data from domains S,L,V of the VLCS data is trained on DeepAll, DNT, DIRT and DRINT algorithms where n j = 5% .
Around 300 data points ( x ) are sampled from D such that there aresamples per domain per class.
Then, the representation z i is computed as z i = E ϕ ( x ′ ) .
For each class y ∈
[ Y ] , two samples: ( x ′ , y, d ′ ) and ( x ′′ , y, d ′′ ) are drawn from D , ( d ′ , d ′′ )
i.i d. ∼ P dom and d ̸ = d ′ .
First z ′ =
E ϕ ( x ′ ) and z ′′ =
E ϕ ( x ′′ ) are computed.
Then z j is computed via linear interpolation between z ′ and z ′′ by varying w from 0 to 1 at a span of 0 .
We assign y as the original label of each of the interpolated samples based on the classes of the reference points x ′ and x ′′ .
The following plots are obtained after running the TSNE algorithm on z (comprising of both z i and z j ) by reducing its dimension from 256 to 2 .
Each of the colors in the plot represents a different class: 0 : bird, 1 : car, 2 : chair, 3 : dog, and 4 : person.
The different domains of the dataset are represented by the following markers: • : L , ▼ : S , ▲ : V .
The interpolated samples are represented by the marker + .
Main figures.
Figures 2, 3, 4 and 5 contain z ’s obtained from the algorithms DeepAll, DNT, DIRT and DRINT respectively.
Sub figures on the left column.
The colors corresponding to each z point present in the images are based on their true label y .
Each of the interpolated z j ’s corresponding have been assigned the color based on the class y their reference points were sampled from. Sub figures on the right column. Colors assigned to z i and z j are based on the label values y
Discussion.
It can be seen that the presence of interpolation loss (figures 3 and 5) in DNT and
DRINT models has made the prediction of data to be closer to the true labels than without its presence in DeepAll and DIRT (figures 2 and 4) respectively.
We also compute the mismatch error between the true labels and the predicted labels for each of the models in Table 3.
It is % of unequal predictions to that of the original labels.
From Table 3, it can be seen that the presence of interpolation loss always reduces the mismatch error.
[Figure]
[Figure]
A.2 Further Results
We report the test accuracy of individual domains in both PACS and VLCS dataset in this section.
Every reported accuracy is the average over 3 different runs.
Standard error has been reported as well.
90% of the data is used as training data and 10% as the validation data for the results of every dataset.
The test accuracy is reported on the model with the best validation accuracy and loss after epoch 50 .
[Figure]
[Figure] [Table] [Table] [Table]
[Table] [Table]