O RIENT : Submodular Mutual Information Measures for Data Subset Selection under Distribution Shift
Anonymous Author(s)
AffiliationAddress email
Abstract
Real-world machine-learning applications require robust models that generalizewell to distribution shift settings, which is typical in real-world situations.
Domainadaptation techniques aim to address this issue of distribution shift by minimizingthe disparities between domains to ensure that the model trained on the sourcedomain performs well on the target domain.
Nevertheless, the existing domain adaptation methods are computationally very expensive.
In this work, we aim to improvethe efficiency of existing supervised domain adaptation (SDA) methods by using asubset of source data that is similar to target data for faster model training.
Specifically, we propose O RIENT , a subset selection framework that uses the submodularmutual information (SMI) functions to select a source data subset similar to the target data for faster training.
Additionally, we demonstrate how existing robust subsetselection strategies, such as G LISTER , G RAD M ATCH , and C RAIG , when used witha held-out query set, fit within our proposed framework and demonstrate the connections with them.
Finally, we empirically demonstrate that SDA approaches liked -SNE, CCSA, and standard Cross-entropy training, when employed together with
O RIENT , achieve a) faster training and b) better performance on the target data.
Introduction
The recent success of deep learning frameworks in applications such as image classification [10],speech recognition [21], and object detection [14] stems primarily from the availability of largeamounts of labeled data.
However, due to enormous labeling costs and the need for specialists inspecific domains such as medical imaging, it is not always possible to obtain vast amounts of labeleddata in all situations.
On the contrary, training deep models on limited amounts of data can lead topoor performance due to overfitting [2].
Consequently, where obtaining large amounts of labeled datais difficult for the target domain, closely related domain (source domain) is used to train the model.
However, this may result in a deep model with suboptimal performance on target domain as a resultof the distribution shift [1, 5, 4, 49], i.e., change of data distribution from source domains to targetdomains.
A change in the data distribution often renders the features learned by the model on thesource domain irrelevant for the target domain.
To address the problem of distribution shift, many domain adaptation [53, 41] and domain generalization [40] techniques have been proposed in recent years.
Domain adaptation methods assume thatsome target domain information is available (usually target data), whereas domain generalization methods do not.
Domain adaptation (DA) methods can be categorized into unsupervised [16, 13, 33, 51](using unlabeled target data), semi-supervised [17, 58, 43, 47] (using labeled and unlabeled targetdata), and supervised [39, 38, 57, 37, 20] (using labeled target data) methods.
UDA methods that do not require any labeled target data assume access to large volumes of unlabeled target data.
Whenproviding the same amount of target data, SDA methods are more effective than UDA methods [39].
As it is not difficult to obtain a few samples (as less as 2 examples per class) of labeled target data inpractice, SDA methods are an attractive approach in scenarios with limited target data.
Hence, in thiswork we are explicitly focused on the SDA setting.
Recently, various effective SDA methods are proposed [39, 57, 37].
However, they are usuallycompute intensive.
For example, using one of the state-of-the-art SDA method, d -SNE [57], to traina ResNet50 model [19] on the Office-Home dataset [52] with 3022 samples of the a source dataand 338 samples a target data for 300 epochs takes > 18 hours using a GTX 1080 Ti GPU.
To putthat in perspective, training a ResNet50 model on the larger CIFAR100 dataset with 45000 trainingsamples for 300 epochs on the same machine for standard cross-entropy loss takes only 14 hours.
Theincrease in training time not only increases energy consumption and CO2 emissions but also restrictsthe usefulness of SDA methods in resource-constrained environments.
We address this problem byseeking an answer to the following question: Can we improve the efficiency of SDA methods bytraining on subsets of source data to ensure faster adaptation and reduction in training time?
To this end, we propose O RIENT , a subset selection framework that utilizes Submodular Mutual
Information (SMI) measures [22, 18] to select subsets of source data that are similar to target data fortraining.
Our key insight is that by training the model on data points that are similar to target data, wecan speed up the training.
The O RIENT framework, illustrated in Figure 1, complements existing
SDA methods and can be effectively combined with any of them, enabling us to achieve a reductionin training times, energy costs, and CO2 emissions.
The key contributions of our paper include: (1) O RIENT , an SMI-based subset selection framework for efficient and effective supervised domain adaptation.
In Section 3, we illustrate that
O RIENT unifies three existing subset selection methods: CRAIG [35], GLISTER [26], and G RADM ATCH [25].
Specifically, these methods fit into O RIENT ’s framework when used with a held-outvalidation set from the target domain.
(3) We empirically show the effectiveness of O RIENT whenused with existing SDA methods like CCSA [39], d -SNE [57], and Standard Cross-entropy trainingon two real-world datasets: Office-31 [42] and Office-Home [52].
Our experiments also demonstratethat model learn better class discrimination features with O RIENT , resulting in better classificationperformance.
1.1 Related work
SDA When the labeled training data is scarce for a domain τ (or a task), a powerful way to boostperformance is by pretraining a model on a related domain and fine-tuning it in the target domain[15, 59, 9].
I SDA techniques have been investigated for various distribution shift settings where thetarget domain ( τ t = { X t , Y t } ) has different marginal or conditional distributions than the sourcedomain ( τ s = { X s , Y t } ).
In this work, we consider covariate shift [1], wherein the marginaldistibution is different (cid:0)
i.e. P ( X s ) ̸
= P ( X t ) (cid:1) , but the conditional distribution remains the same 71 (cid:0) i.e. P ( Y s | X s ) =
P ( Y t | X t ) =
P ( Y | X ) (cid:1) .
A notable body of work for covariate shift SDA hasfocused on identifying a latent feature space that is domain-invariant [39, 38, 57, 37, 20].
This lineof work train a network consisting of a feature extractor and a classifier end-to-end.
The featureextractor learns a non-linear transformation of the samples from different domains to a shared latentspace, and the classifier learns to assign class labels.
Different researchers have proposed differentoptimization objectives and loss functions to encourage domain confusion and class separability.
Tzeng et al.
[50] designed a domain-confusion loss to optimize for domain invariance and match thedistribution over classes in the source domain to the soft label in the target domain.
Motiian et al.[39] propose a classification and contrastive semantic alignment loss (CCSA).
CCSA uses contrastiveloss with Siamese Network to encourage the same class samples from different distributions to benearby in the latent space (semantic alignment loss), and the different class samples from differentdistributions to be far apart (separation loss).
Few-shot adversarial domain adaptation (FADA)
[38]proposes to learn similar latent space with adversarial training.
Domain-adaptation using stochastic neighborhood embedding ( d -SNE) [57] proposes to maximizes the smallest distance between thesamples of different classes and minimizes the largest distance between the samples of the same classwhile being domain invariant.
Another approach, Second- or Higher-order Transfer of Knowledge(So-HoT) [30], aims to align the with-in class scatters and maintain separation of between-classscatters.
Finally, Morsing et al.
[37] propose domain adaptation using graph embedding (DAGE) andshow that CCSA and d -SNE can also be expressed as graph embedding methods [20].
Subset Selection methods Submodular function [34, 12] is one of the effective approach of datasubset selection which is used in variety of applications [6].
For example, it has been employedfor efficient training in speech recognition [55, 54], machine translation [29], computer vision [23],supervised classification [35, 26, 25], semi-supervised classification [27], active learning methods [56,46, 3, 26], and hyper-parameter optimization [28].
Another widely used method for selecting subsetsis coreset construction.
The coresets [11] are weighted subsets of the data that approximate thesemantic characteristics of the entire dataset (e.g., loss, marginal probability, etc.).
A number ofrecent coreset selection-based methods [36, 26, 25, 27] have demonstrated the promise to efficientlyand robustly train deep models.
Coresets are used for other deep learning applications like continuallearning, active learning and data summarization [7, 48] have used.
Subset selection plays a significant role in robust learning under realistic scenarios, such as imbalances or rare classes, out-of-distribution(OOD) data, or redundancy.
Kothawade et al.
[31] employed SMI measures for active learning in realistic scenarios tackling imbalance, OOD, and redundancy.
PRISM [32] employed SMI measures for targeted data summarization.
GLISTER [26] and G RADM ATCH [25] shows that using a clean held-out validation set with SMI mitigates the label noise andclass imbalance in the training set.
Mirzasoleiman et al.
[36] showed the data samples with clean labels cluster together, whereas the ones with noisy labels spread out in the gradient space, thus makingk-medoid clustering an effective method for reducing noise in labeled data.
Furthermore, Killamsettyet al.
[27] tackles OOD and imbalance in the unlabeled data in semi-supervised learning settingthrough data subset selection.
We show how these previous subset selection strategies including amodified version of C RAIG [35] were also using instantiations of SMI measures in Section 3.
Preliminaries
In this section, we introduce submodular functions, SMIs and SDA loss functions.
Submodular Functions:
Let D be a set of n data points D = { 1 , · · · , n } and f : 2 D → R bea set function returning real-value for any subset of set D .
A function f is submodular [12] iff ( A ∪ x ) − f
( A ) ≥ f
( B ∪ x ) − f ( B ) , ∀ x ∈ D , ∀ A ⊆ B ⊆ D and x / ∈ B .
Submodular functions,when maximized for a cardinality constraint using a simple greedy algorithm, admit a constant factorof 1 − 1 e .
This can be done in near-linear time using a stochastic-greedy algorithm.
Submodular Mutual Information (SMI): The SMI between two sets A, B ⊆ D , instantiated witha submodular function f , is defined as I f
( A ; B ) =
f ( A ) + f ( B ) − f
( A ∪ B ) [22].
In this work, weaim to select data points A ⊂ D that maximize the SMI between A and B .
Intuitively, this allowsselection of data points that are similar to B while being diverse.
SDA Loss Functions: For supervised domain adaptation, specialized loss functions are introducedas described in the previous section.
Here we highlight two state-of-the-art domain adaptation lossfunctions, CCSA and d -SNE, which are used in our experiments.
In CCSA, the classifier is modeled as a composition of two functions– h ◦ g .
Here, g : X → Z is afeature extractor that transforms the input from the feature space X to an embedding space Z , andh : Z → Y is a predictor function.
The CCSA loss for supervised domain adaptation is defined as,
[Equation]
where L CE ( h ◦ g ) is the cross-entropy loss for multi-class classification, L SA is a semantic alignmentloss encouraging the samples from different domains but same label to map nearby in the embeddingspace, and L S is a separation loss encouraging the samples from different domains and differentlabels to map far away in the embedding space. [Table]
[Equation]
COM
Appendix Eq.
[Table]
The d -SNE loss function for SDA is defined as
[Equation]
where L s CE and L t CE are the cross-entropy loss on the source and the target domain examples,respectively, and ˜ L minimizes the largest distance between the samples from different domains withsame label and maximizes the smallest distance between the samples from different domain withdifferent label in the embedding space.
SDA loss functions are further elaborated in Section A.4.
O RIENT
For our SDA setting, we assume a small amount of labeled data (as less as 2 examples per class) isavailable from the target domain τ t and sufficient labeled data is available from the source domain τ s .
We denote the source dataset as D s =
( x i , y i )
| D S | i =1 and the target dataset as D t = ( x i , y i ) | D t | i =1 .
Thesource dataset is used for training the model M using an SDA loss function L .
We will introduceother necessary notations in the remainder of the paper, as necessary.
A complete list of notations ispresented in Appendix (A.1).
First, we extend the SMI measure to incorporate data subsets from different domains.
For a sourcedomain subset A ⊆ D s and the target domain data set D t , we denote SMI measure as I f
( A ; D t ) .
Essentially, assuming that both the domains form a set D , and A ⊆ D s ⊂ D and D t ⊂ D .
With Sdenoting the similarity matrix defined over a subset A and D t , Table 5 presents the mathematicalexpression of the SMI functions for different instantiations of submodular functions ( f ) [32].
Wedefer the readers to Kothawade et al.
[32] for more details.
Note that the FLMI, GCMI and COMonly need the pairwise similarities of the data points in A and D t .
Consequently, the similarity kernelis of size
| A | × | D t | , making it very efficient to optimize.
For efficient SDA, we want to select a subset A of the source domain τ s such that training a modelon A results in a classifier that is proficient on the target domain τ t .
To this effect, we use thegradients χ of the current model to represent the data points and use it to compute the similarity matrix.
Specifically, we define the pairwise similarity between two data points as the cosine similaritybetween the gradients,
[Equation]
where, χ i = ∇ θ L ( x i , y i ) and χ j = ∇ θ L ( x j , y j )
Given a set size b , we select a diverse subset A of the source data D s that is similar to the target data
D t by maximizing the following SMI measure,
[Equation]
I f
( A ; D t ) is an instance of cardinality constrained non-monotone submodular maximization andwe can achieve a 1 e approximation using the randomized greedy algorithm [8].
However, as thenumber of parameters in the deep learning model can be extremely high, our gradients χ can be veryhigh dimensional.
Following the success of targeted subset selection approaches, GLISTER [26],
SIMILAR [31], CRAIG [36] and BADGE [3], we circumvent this problem by using last-layergradient approximations to represent the data point in the similarity matrix S .
Further, we select anew subset every L epochs as the model is trained, and the subsets chosen are adapted accordingly.
Algorithm: We present the complete training procedure of O RIENT in Algorithm 1.
We firstrandomly initialize the training subset A and the model parameters θ in line 1 and 2 , respectively.
For each epoch, we update the model parameters by optimizing a gradient-descent (GD) based loss Lon the current training subset A in line 4 .
After a fixed interval of L epochs, we update the subset A( line 5–11 ).
To do this, we compute the gradients χ s and χ t for all the datapoints in D s and D t , inline 6 and 7 ,respectively.
Next, we use the gradients to compute the similarity matrix S in line 8 , asdescribed in Equation 1.
In line 9 we instantiate the SMI function I f with S and update the subset Ain line 10 using Equation 2.
Our implementation is made available online 1 .
Algorithm
Figure 1 illustrates the O RIENT workflow.
The general framework of O RIENT can incorporateany gradient decent (GD) based SDA technique.
It can train any model with loss function L byusing a subset of the source data, selected every L epochs.
In our experiments, we demonstratethe effectiveness of our method in conjunction with standard cross-entropy loss as well as twostate-of-the-art domain adaptation losses, CCSA [39] and d-SNE [57].
Connections to Previous Work
O RIENT generalizes three approaches, GLISTER [26], G RAD M ATCH [25], and a slightly adaptedversion of CRAIG [35], which were initially proposed for efficient and robust subset selection.
(a.1)
A → D (a.2)
A → W (b.1)
D → A (b.2)
D → W (c.1)
W → A (c.2)
W → D
Specifically, these subset selection approaches maximize a submodular function corresponding to adifferent instantiations of SMI functions for subset selection.
Following theorems summarize theconnection of SMI functions with these subset selection approaches.
Theorem 1 When the outer level loss of the discrete bi-level optimization problem of GLISTER ishinge loss, logistic loss, and perceptron loss, then the optimization problem becomes an instance of
Concave over Modular (COM) SMI function.
Theorem 2 When the optimization problem of G RAD M ATCH is used to match gradients of a held-outvalidation set, it becomes an instance of the summation of GCMI and a diversity function.
Theorem 3 When the optimization problem of C RAIG is adapted to match gradients of a held-outvalidation set, it becomes an instance of the FLMI function with η = 0 .
We present the proofs of these theorems in Appendix (A.3).
In conclusion, SMI measures have beenused in previous subset selection strategies for robust learning to deal with the class imbalance andnoisy labels in training data sets and achieved comparable performance to current state-of-the-artrobust supervised learning approaches.
Experimental Evaluation
Our experiments explicitly aim at answering the following questions,
Q1:
Does the proposed O RIENT approach substantially reduce the training time while maintaining comparable performance to training on complete source dataset?
Q2:
Can the proposed O RIENT approach augment the existing domain adaptation approaches?
(a.1)
A → C (a.2)
A → P (a.3)
A → R (b.1)
C
→ A (b.2)
C → P (b.3)
C → R (c.1) P → A (c.2)
P → C (c.3)
P → R (d.1)
R → A (d.2) R → C (d.3) R → P
We evaluate our proposed approach on two domain adaptation datasets: Office-31 [42] and OfficeHome [52]. Office-31 dataset consists of 4110 images of 31 object categories from three differentdomains: Amazon (A), DSLR (D), and Webcam (W). For our experiments we used all the imagesfrom the source domain in the training set ( D s ) and two examples of each category in the queryset ( D t ).
Office-Home dataset consists of 10812 images of 65 object categories from four differentdomains: Art (A), Clipart (C), Product (P), and Real-world (R). Here, the target data ( D t ) consists ofabout 20% of the images from the target domain. [Table]
[Table]
To answer the first question, we use O RIENT to train a ResNet50 architecture using the stochasticgradient descent (SGD) algorithm with the momentum of 0 .
9 and the weight decay ratio of 5 e 4 . Wecompare our approach against three baselines: Full —training on the source data and the target data,i.e. D s ∪ D t ; Random —a random subset of the source dataset and the target set, i.e. R ∪ D t , R = RandomSubset( D s ) ; and CRAIG —a coreset of the source dataset.
We use standard categoricalcross-entropy loss function (for multi-class classification) and five different instantiations of thesubmodular mutual information function: Facility Location Mutual Information ( ORIENT (FLMI) ),
Graph Cut Mutual Information ( ORIENT (GCMI) ), Log Determinant Mutual Information ( ORIENT(LDMI) ), GLISTER ( ORIENT (G) ), and GradMatch ( ORIENT (GM) ).
Figure 2 presents the scatter plot of the speed up against the prediction accuracy of Office-31 datasetusing cross-entropy loss..
In all our experiments, we use 0 .
3 fraction of the source data as the subsetand see > 2 .
5 × speed up.
Figure 3 presents the convergence curves of the Office-Home datasetusing cross-entropy loss.
It is evident from these charts that our proposed approach O RIENT canachieve better or comparable performance to the Full training in significantly less amount of time,indicating better efficiency.
This analysis helps us answer our Q1.
O RIENT reduces the training timesubstantially, the speed up achieved is reciprocal to the fraction of the subset used, without tradingthe performance.
Rather, in some combinations, we observe that O RIENT outperforms Full.
Figure 4 presents the bar plots of the speed up when O RIENT (FLMI) is used in conjunction withd -SNE loss function on Office-31 and Office-Home datasets.
Here, we see a consistent 3 × speed upas compared to full model training with d -SNE loss.
These bar plots demonstrate that our proposed [Figure] [Figure] approach O RIENT can augment existing domain adaptation approaches and substantially reducethe training times.
Additionally, Tables 2 and 3 present the test accuracy while using d -SNE and
CCSA loss in conjunction with O RIENT (FLMI) on Office-31 and Office-Home datasets, respectively.
It’s evident that augmenting existing domain adaptation approaches with O RIENT achieves betteror comparable performance to the Full training.
These two observations help us answer our Q2 inthe affirmative.
Precise values of prediction accuracy and training time for all the experiments areprovided in Appendix (A.5).
Figure 5 presents the GradCam [45] class-activation maps of trained models on the Office-Homedataset
( P → R setting) using d -SNE loss for both Full and O RIENT (FLMI).
These activation mapsshow that the model trained with the O RIENT framework learned better class discrimination featuresthan the model trained with Full.
This might explain why the O RIENT framework performs bettersometimes than Full.
To summarize, our experiments on two adaptation datasets suggest that our proposed approach O RIENT can substantially reduce the training time while maintaining comparable performance to trainingon complete source data.
[-In comparison to the O RIENT using remaining SMI functions,i.e., O RIENT (GM), O RIENT (G), and O RIENT (FLMI), O RIENT (FLMI) consistently achieves the bestperformance versus speed-up trade-off. We further present synthetic experiments to provide intuitionson how different SMI functions selects data subsets in Appendix A.
 To summarize, our experiments on two adaptation datasets suggest that our proposed approach O RIENT can substantially reduce the training time while maintaining comparable performance to trainingon complete source data.-] Additionally, our experiments using d -SNE and CCSA loss functionssuggest that O RIENT can be used in conjunction with existing SDA methods to achieve significantspeed ups in training time, while maintaining or improving the prediction accuracy.
In particular,two instantiations of our proposed approach, O RIENT (FLMI) and O RIENT (GM) consistently achievecomparable or better performance compared to Full training while being ∼ 3 × faster to train.
Conclusion, Limitations, and Broader Impact
We introduce O RIENT , a subset selection framework based on SMI functions for supervised domainadaptation.
The submodularity of SMI functions allows us to use scalable greedy algorithms to selectthe data subsets efficiently.
In addition, we demonstrate how O RIENT is a unified framework thatintegrates previous approaches based on subset selection for robust learning.
Empirically, we showthat O RIENT is very effective for SDA.
Specifically, it achieves ∼ 3 × speed up over existing SDAapproaches like d -SNE and CCSA while achieving comparable or better performance.
Our findingsconfirm that O RIENT has a significant social impact by making existing SDA algorithms significantlyfaster and more energy-efficient, reducing CO2 emissions and energy consumption incurred duringtraining, thus contributing to a Green AI [44].
The main limitation of our work is that although
O RIENT significantly reduces the training time, it requires more memory to store the similarity kernelrequired for subset selection.
This makes running O RIENT harder on devices with low memory.