Domain Adaptation under Open Set Label Shift
Anonymous Author(s)
AffiliationAddress email
Abstract
We introduce the problem of domain adaptation under Open Set Label Shift (OSLS)where the label distribution can change arbitrarily and a new class may arrive during deployment, but the class-conditional distributions p p x | y q are domain-invariant.
OSLS subsumes domain adaptation under label shift and Positive-Unlabeled
(PU)learning.
The learner’s goals here are two-fold: (a) estimate the target label distribution, including the novel class; and (b) learn a target classifier.
First, we establishnecessary and sufficient conditions for identifying these quantities.
Second, motivated by advances in label shift and PU learning, we propose practical methodsfor both tasks that leverage black-box predictors.
Unlike typical Open Set Domain Adaptation (OSDA) problems, which tend to be ill-posed and amenable only toheuristics
, OSLS offers a well-posed problem amenable to more principled machinery.
Experiments across numerous semi-synthetic benchmarks on vision, language,and medical datasets demonstrate that our methods consistently outperform OSDAbaselines, achieving 10 – 25% improvements in target domain accuracy.
Finally, weanalyze the proposed methods, establishing finite-sample convergence to the truelabel marginal and convergence to optimal classifier for linear models in a Gaussian setup.
Code is available at https://github.com/Neurips2022Anon .
Introduction
Suppose that we wished to deploy a machine learning system to recognize diagnoses based on theirclinical manifestations.
If the distribution of data were static over time, then we could rely on thestandard machinery of statistical prediction.
However, disease prevalences are constantly changing,violating the assumption of independent and identically distributed (iid) data.
In such scenarios,we might reasonably apply the label shift assumption, where prevalences can change but clinicalmanifestations cannot.
When only the relative proportion of previously seen diseases can change,principled methods can detect and correcting for label shift on the fly [56, 78, 45, 4, 1, 27].
But whatif a new disease, like COVID-19, were to arrive suddenly?
Traditional label shift adaptation techniques break when faced with a previously unseen class.
Adistinct literature on Open Set Domain Adaptation (OSDA) seeks to handle such cases [51, 5, 14,70, 43, 73, 58, 59, 25]).
Given access to labeled source data and unlabeled target data, the goal in OSDA is to adapt classifiers in general settings where previous classes can shift in prevalence (andeven appearance), and novel classes separated out from those previously seen can appear.
Mostwork on OSDA is driven by the creation of and progress on benchmark datasets (e.g., DomainNet,
OfficeHome).
Existing OSDA methods are heuristic in nature, addressing settings where the rightanswers seem intuitive but are not identified mathematically.
However, absent assumptions on: (i)the nature of distribution shift among source classes and (ii) the relation between source classes andnovel class, standard impossibility results for domain adaptation condemn us to guesswork [8].
In this work, we introduce domain adaptation under Open Set Label Shift (OSLS), a coherentinstantiation of OSDA that applies the label shift assumption but allows for a new class to show up in the target distribution.
Formally, the label distribution may shift between source and targetp s p y q ‰ p t p y q , but the class-conditional distributions among previously seen classes may not (i.e.,@ y P t 1 , 2 , . . .
, k u , p s p x | y q “ p t p x | y q ).
Moreover, a new class y “ k ` 1 may arrive in the targetperiod.
Notably, OSLS subsumes label shift [56, 67, 45] (when p t p y “ k ` 1 q “ 0 ) and learning from
Positive and Unlabeled (PU) data [20, 42, 24] (when k “ 1 ).
As with label shift and PU learning, ourgoals are two-fold.
Here, we must (i) estimate the target label distribution p t p y q (including the novelclass prevalence); (ii) train a p k ` 1 q -way target-domain classifier.
First, we characterize when the parameters of interest are identified (Sec. 4).
Namely, we define a(necessary) weak positivity condition, which states that there exists a subset of each label’s supportthat has zero probability mass under the novel class and that the submatrix of p p x | y q consisting onlyof rows outside the novel class’s support is full rank.
Moreover, we prove that weak positivity aloneis not sufficient.
We introduce two sufficient conditions: strong positivity and separability , either ofwhich (independently) ensures identifiability.
Focusing on cases with strong positivity, we show that OSLS reduces to k PU learning problems(Sec. 5).
However, we demonstrate that straightforward applications of this idea fail because (i) biasaccumulates across the k mixture proportion estimates leading to grossly underestimating the novelclass’s prevalence; and (ii) naive combinations of the k PU classifiers are biased and inaccurate.
Thus motivated, we propose the PULSE framework, which combines methods from Positive and
Unlabeled learning and Label Shift Estimation, yielding two-stage techniques for both label marginalestimation and classification (Sec. 6).
Our methods build on recent advances in label shift [45, 4, 1, 27]and PU learning [38, 35, 29], that leverage appropriately chosen black-box predictors to avoid thecurse of dimensionality.
PULSE first estimates the label shift among previously seen classes, andthen re-samples the source data to formulate a single PU learning problem between (reweighted)source and target data to estimate fraction of novel class and to learn the target classifier.
In particular,our procedure builds on the BBE and CVIR techniques proposed in Garg et al.
PULSE is simpleto implement and compatible with arbitrary hypothesis classes (including deep networks).
We conduct extensive semi-synthetic experiments adapting seven benchmark datasets spanningvision (CIFAR10, CIFAR100, Entity30), natural language (Newsgroups-20), biology (Tabula Muris),and medicine (DermNet, BreakHis) (Sec. 7).
Across numerous data modalities, draws of the labeldistributions, and model architectures, PULSE consistently outperforms generic OSDA methods,improving by 10 – 25% in accuracy on target domain.
Moreover, PULSE outperforms methods thatnaively solve k PU problems on both label distribution estimation and classification.
Finally, we analyze our framework (Sec. 8).
First, we extend Garg et al.
[29]’s analysis of BBE toderive finite-sample error bounds for our estimates of the label marginal.
Next, we develop newanalyses of the CVIR objective [29] that PULSE relies in the classification stage.
Focusing on a
Gaussian setup and linear models optimized by gradient descent, we prove that CVIR converges toa true positive versus negative classifier in population.
Addressing the overparameterized settingwhere parameters exceed dataset size, we conduct an empirical study that helps to elucidate why, onseparable data, CVIR outperforms other consistent objectives, including nnPU [38] and uPU [21]
Related Work(Closed Set) Domain Adaptation (DA)
Under DA, the goal is to adapt a predictor from asource distribution with labeled data to a target distribution from which we observe only unlabeledexamples.
DA is classically explored under two distribution shift scenarios [67]: (i) Covariateshift [78, 74, 19, 18, 32] where p p y | x q remains invariant among source and target; and (ii) Labelshift [56, 45, 4, 1, 27, 77] where p p x | y q is shared across source and target.
In these settings mosttheoretical analysis requires that the target distribution’s support is a subset of the source support [8].
However, recent empirically work in DA [48, 49, 68, 69, 80, 79, 26, 64] focuses on settings motivatedby benchmark datasets (e.g., WILDS [57, 39], Office-31 [55] OfficeHome [71], DomainNet [52])where such overlap assumptions are violated.
Instead, they rely on some intuitive notion of semanticequivalence across domains.
These problems are not well-specified and in practice, despite carefulhyperparameter tuning, these methods often do not improve over standard empirical risk minimizationon source data alone for practical, and importantly, previously unseen datasets [57].
Open Set Domain Adaptation (OSDA) OSDA [51, 9, 62] extends DA to settings where alongwith distribution shift among previously seen classes, we may observe a novel class in the targetdata.
This setting is also known as universal domain adaptation [73, 59].
Rather than making preciseassumptions about the nature of shift between source and target as in OSLS, the OSDA literature isprimarily governed by semi-synthetic problems on benchmark DA datasets (e.g. DomainNet, Office31 and OfficeHome).
Numerous OSDA methods have been proposed [5, 14, 70, 43, 73, 58, 59, 25, 11].
At a high level, most OSDA methods perform two steps: (i) align source and target representationfor previously seen classes; and (ii) train a discrimination to reject novel class from previously seenclasses.
The second step typically uses novelty detection heuristics to identify novel samples.
PU Learning Positive and Unlabeled (PU) learning is the base case of OSLS.
Here, we observelabeled data a single source class and unlabeled target data contains data from both the novel classand the source class.
In PU learning, our goals are: (i) Mixture Proportion Estimation (MPE), i.e.,determining the fraction of previously seen class in target ; and (ii) PU classification, i.e., learningto discriminate between the novel and the positive (source) class.
Several classical methods wereproposed for both MPE [24, 22, 63, 36, 6, 7] and classification [24, 23, 21].
However, classical MPEmethods do not scale to high-dimensional settings [53].
More recent methods alleviate these issues byoperating in classifier output space [29, 35].
For classification, traditional methods fail when deployedwith models classes with high capacity due to their capacity of fitting random labels [75].
Recentmethods [29, 38, 16], avoid over-fitting by employing regularization or self-training techniques.
Other related work A separate line of work looks at the problem of Out-Of-Distribution (OOD)detection [34, 31, 41, 37, 50, 76].
Here, the goal is to identify novel examples, i.e., samples that lieout of the support of training distribution.
The main different between OOD detection and OSDA isthat in OOD detection we do not have access to unlabeled data containing a novel class.
Recently,
Cao et al.
[13] proposed open-world semi-supervised learning, where the task is to not only identifynovel classes in target but also to separate out different novel classes in an unsupervised manner.
Our work takes a step back from the hopelessly general OSDA setup, introducing OSLS, a wellposed OSDA setting where the sought-after parameters can be identified.
Open Set Label Shift
Notation For a vector v P R d , we use v j to denote its j th entry, and for an event E , we let I r E sdenote the binary indicator of the event.
By | A | , we denote the cardinality of set A .
Let X
P
R
d be the input space and Y “ t 1 , 2 , . . .
, k ` 1 u be the output space for multiclass classification.
Let P s and P t be the source and target distributions and let p s and p t denote the correspondingprobability density (or mass) functions.
By E s and E t , we denote expectations over the source andtarget distributions.
We assume that we are given a loss function ℓ : ∆ k ˆ Y Ñ R , such that ℓ p z, y q isthe loss incurred by predicting z when the true label is y .
Unless specified otherwise, we assume that ℓis the cross entropy loss.
As in standard unsupervised domain adaptation, we are given independentlyand identically distributed (iid) samples from labeled source data tp x 1 , y 1 q , p x 2 , y 2 q , . . .
, p x n , y n qu „
P ns and iid samples from unlabeled target data t x n ` 1 , x n ` 2 , . . .
, x n ` m u „ P mt .
Before formally introducing OSLS, we describe label shift and PU learning settings.
Under labelshift, we observe data from k classes in both source and target where the conditional distribution remain invariant (i.e., p s p x | y q “ p t p x | y q for all classes y P r 1 , k s ) but the target label marginal maychange (i.e., p t p y q ‰ p s p y q ).
Additionally, for all classes in source have a non-zero support , i.e., forall y P r 1 , k s , p s p y q ě c , where c ą 0 .
Under PU learning, we possess labeled source data from apositive class and unlabeled target data from a mixture of positive and negative class with a goal oflearning a positive-versus-negative classifier on target.
We now introduce the OSLS setting:
Definition 1 (Open set label shift) .
Define Y t “ Y and Y s “ Y zt k ` 1 u .
Under OSLS, the labeldistribution among source classes Y s may change but the class conditional p p x | y q for those classesremain invariant between source and target, and the target domain may contain a novel class, i.e.,p s p x | y “ j q “ p t p x | y
“ j q @ j P Y s and p s p
y “ k ` 1 q “ 0 .
Additionally, we have non-zero support for all k (previously-seen) labels in the source distribution,i.e., for all y P Y s , p s p y q ě c for
some c ą 0 .
Note that the label shift and PU learning problems can be obtained as special cases of OSLS.
Whenno novel class is observed in target (i.e., when p t p y “ k ` 1 q “ 0 ), we recover the label shift problem,and when we observe only one class in source (i.e., when k “ 1 ), the OSLS problem reduces to PUlearning.
Under OSLS, our goal naturally breaks down into two tasks: (i) estimate the target labelmarginal p t p y q for each class y P Y ; (ii) train a classifier f : X Ñ ∆ k to approximate p t p
y
| x q .
Identifiablity of OSLS
We now introduce conditions for OSLS, under which the solution is identifiable.
Throughout thesection, we will assume access to population distribution for labeled source data and unlabeled targetdata, i.e., p s p x, y q and p t p x q is given.
To keep the discussion simple, we assume finite input domain
X which can then be relaxed to continuous inputs.
We relegate proofs to App.
B.
We first make a connection between target label marginal p t p
y q estimation and learning
the target classifier p t p y | x q showing that recovering p t p y q is enough to identify p t p y | x q .
In population,given access to p t p y q , the class conditional p t p x | y “ k ` 1 q can be obtained in closed form asp p t p x q´ ř kj “ 1 p t p y “ j q p s p x | y “ j q q{ p t p y “ k ` 1 q .
We can then apply Bayes rule to obtain p t p y | x q .
Henceforth,we will focus our discussion on identifiability of p t p y q which implies identifiability of p t p y | x q .
Infollowing proposition, we present weak positivity , a necessary condition for p t p y q to be identifiable.
Proposition 1 (Necessary conditions) .
Assume p t p y q ą 0 for all y P Y t .
Then p t p y q is identifiedonly if p t p x | y “ k ` 1 q and p s p x | y q for all y P Y s satisfy weak positivity, i.e., there must exists asubdomain X wp Ă X such that: (i)
p t
p
X wp | y “ k ` 1 q “ 0 ; and (ii) the matrix r p s p x | y qs x P X wp ,y P Y sis full column-rank.
Intuitively, Proposition 1 states that if the target marginal doesn’t lie on the vertex of the simplex∆ k , then their must exist a subdomain X wp where the support of novel class is zero and within X wp ,p t p y q for source classes is identifiable.
While it may seem that existence of a subdomain X wp isenough, we show that for the OSLS problem, existence doesn’t imply uniqueness.
In App.
B.1, weconstruct an example, where the weak positivity condition is not sufficient.
In that example, we showthat there can exist two subdomains X wp and X 1 wp satisfying weak positivity, both of which lead toseparate solutions for p t p y q .
Next, we extend weak positivity to two stronger conditions, either ofwhich (alone) implies identifiability.
Proposition 2 (Sufficient conditions) .
The target marginal p t p y q is identified if for all y P Y zt k ` 1 u ,p t p x | y “ k ` 1 q and p s p x | y q satisfy either: (i)
Strong positivity, i.e., there exists X sp Ă X suchthat p t p X sp | y “ k ` 1 q “ 0 and the matrix r p s p x | y qs x P X sp ,y P Y s is full-rank and diagonal; or (ii)
Separability, i.e., there exists X sep Ă X , such that p t p X sep | y “ k ` 1 q “ 0 , p s p X sep q “ 1 , and thematrix r p s p x | y qs x P X sep ,y P Y s is full column-rank.
Strong positivity generalizes the irreducibility condition [10], which is sufficient for identifiabilityunder PU learning, to k PU learning problems.
Note that while the two conditions in Proposition 2overlap, they cover independent set of OSLS problems.
Informally, strong positivity extends weakpositivity by making an additional assumption that the matrix formed by p p x | y q on inputs in X wp isdiagonal and the separability assumption extends the weak positivity condition to the full input domainof source classes instead of just X wp .
Both of these conditions identify a support region of X whichpurely belongs to source classes where we can either individually estimate the proportion of eachsource classes (i.e., under strong positivity) or jointly estimate
To extend our identifiability conditions for continuous distributions, the linear independence conditions on the matrix r p s p x | y qs x P X sep ,y P Y s has the undesirable property of being sensitive to changes onsets of measure zero.
We may introduce stronger notions of linear independence as in Lemma 1 of Garg et al.
We discuss this in App.
B.2.
Reduction of OSLS to k PU Problems
Under the strong positivity condition, the OSLS problem can be broken down into k PU problems asfollows: By treating a given source class y j P Y s as positive and grouping all other classes
togetheras negative we observe that the unlabeled target data is then a mixture of data from the positive andnegative classes.
This yields a PU learning problem and the corresponding mixture proportion isthe fraction p t p y “ j q (proportion of class y j ) among the target data.
By iterating this process forall source classes, we can solve for the entire target label marginal p t p y q .
Thus, OSLS reduces to kinstances of PU learning problem.
Formally, note that p t p x q can be written as:
[Equation]
individually for all j P Y s .
By repeating this reduction for all classes, we obtain k separate PUlearning problems.
Hence, a natural choice is to leverage this structure and solve k PU problemsto solve the original OSLS problem.
In particular, for each class j P Y s , we can first estimate itsprevalence p p t p
y “ j q in the unlabeled target.
Then the target marginal for the novel class is givenby p p t p y “ k ` 1 q “ 1 ´ ř ki “ 1 p p t p
y “ i q .
Similarly, for classification, we can train k PU learningclassifiers f i , where f i is trained to classify a source class i versus others in target.
An example isclassified as belonging to the class y “ k ` 1 , if it rejected by all classifiers f i as other in target.
Weexplain this procedure more formally in App. A.1.
This reduction has been mentioned in past work [60, 72].
However, to the best of our knowledge, noprevious work has empirically investigated both classification and target label marginal estimationjointly.
Sanderson and Scott [60] focuses only on target marginal estimation for tabular datasets and
Xu et al.
[72] assumes that the target marginal is known and only trains k separate PU classifiers.
In our work, we perform the first large scale experiments to evaluate efficacy of the reduction of the
OSLS problem to k -PU problems.
With plugin state-of-the-art PU learning algorithms, we observethat this naive reduction doesn’t scale to datasets with large number of classes because of erroraccumulation in each of the k MPEs and k one-versus-other PU classifiers.
To mitigate the erroraccumulation problem, we propose the PULSE framework in the next section.
The PULSE Framework for OSLS
We begin with presenting our framework for OSLS problem under strong positivity condition.
First,we explain the structure of OSLS that we leverage in PULSE framework and then elaborate designdecisions we make to exploit the identified structure.
Overview of PULSE framework Rather than simply dividing each OSLS instance into k PUproblems, we exploit the joint structure of the problem to obtain a single PU learning problem.
Tobegin, we note that if only we could apply a label shift correction to source, i.e., re-sample sourceclasses according to their relative proportion in the target data, then we could subsequently considerthe unlabeled target data as a mixture of (i) the (reweighted) source distribution; and (ii) the novelclass distribution (i.e., p t p x | y “ k ` 1 q ).
Formally, we have
[Equation]
where p 1 s p x q is the label-shift-corrected source distribution, i.e., p 1 s p x q “ ř j P Y s w p
j q p s p x, y “ j q ,where w p j q “ p p t p
y “ j q { ř k p t p
y
“ k q q{ p s p y “ j q for all j P Y s .
Intuitively, p 1 t p j q “ p t p y “ j q { ř k p t p
y “ k qis re-normalized label distribution in target among source classes and w p j q ’s are the importanceweights.
Hence, after applying a label shift correction to the source distribution p 1 s p x q , we havereduced the OSLS problem to a single PU learning problem, where p 1 s p x q plays the part of thepositive distribution and p t p x | y “ k ` 1 q acts as negative distribution with mixture coefficients1 ´
p t
p
y
“ k ` 1 q and p t p y “ k ` 1 q respectively.
We now discuss our methods (i) to estimate theimportance ratios w p y q ; and (ii) to tackle the PU learning instance obtained from OSLS.
Label shift correction: Target marginal estimation among source classes While traditionalmethods for estimating label shift breakdown in high dimensional settings [78], recent methodsexploit black-box classifiers to avoid the curse of dimensionality [45, 4, 1].
However, these recenttechniques require overlapping label distributions, and a direct application would require demarcationof samples from p 1 s p x q sub-population in target, creating a cyclic dependency.
Instead, to estimate therelative proportion of previously seen classes in target, we leverage the k PU reduction described in Sec.
5 with two crucial distinctions. First, we normalize the obtained estimates of fraction previouslyseen classes to obtain the relative proportions in p 1 s p y q . In particular, we do not leverage the estimatesof previously seen class proportions in target to directly estimate the proportion of novel class whichavoids issues due to error accumulation.
Second, we exploit a k -way source classifier f s trainedon labeled source data instead of training k one-versus-other PU classifiers. We tailor the recentlyproposed Best Bin Estimation (BBE) technique from Garg et al.
We describe the modified BBEprocedure in App.
C (Algorithm 2).
After estimating the relative fraction of source classes in target(i.e. , p p 1 t p j q “ p p t p y “ j q { ř k P Y s p p t p y “ k q for all j P Y s ), we re-sample the source data according to p p 1 t p y qto mimic samples from distribution p 1 s p x q .
PU Learning: Separating the novel class from previously seen classes After obtaining a PUlearning problem instance, we resort to PU learning techniques to (i) estimate the fraction of novelclass p t p
y “ k ` 1 q ; and (ii) learn a binary classifier
f d p x q to discriminate between label shift correctedsource p 1 s p x q and novel class p t p x | y “ k ` 1 q .
With traditional methods for PU learning involvingdomain discrimination, over-parameterized models can memorize the positive instances in unlabeled,assigning them confidently to the negative class, which can severely hurt generalization on PNdata [38, 29].
Rather, we employ Conditional Value
Ignoring Risk (CVIR) loss proposed in Garg et al.[29] which was shown to outperform alternative approaches.
First, we estimate the proportion of novelclass p p t p
y “ k ` 1 q with BBE. Next, given an estimate p p t p y “ k ` 1 q , CVIR objective discards thehighest loss p 1 ´ p p t p
y “ k ` 1 qq fraction of examples on each training epoch, removing the incentiveto overfit to the examples from p 1 s p x q .
Consequently, we employ the iterative procedure that alternatesbetween estimating the prevalence of novel class p p t p
y “ k ` 1 q (with BBE) and minimizing the CVIR loss with estimated fraction of novel class.
We detail this procedure in App. C (Algorithm 3).
Combining PU learning and label shift correction Finally, to obtain a p k ` 1 q -way classifier f t p x qon target we combine discriminator f d and source classifier f s with importance-reweighted label shiftcorrection.
In particular, for all j P Y s , r f t p x qs j “ p f d p x qq w p
j q¨r f s p x qs j ř k P Y s w p
k
q¨r f s p x qs k and r f t p x qs k ` 1 “1 ´ f d p x q .
Overall, our approach outlined in Algorithm 1 proceeds as follows:
First, we estimatethe label shift among previously seen classes.
Then we employ importance re-weighting of sourcedata to formulate a single PU learning problem to estimate the fraction of novel class p p t p
y “ k ` 1 qand to learn a discriminator f d for the novel class.
Combining discriminator and label shift correctedsource classifier we get p k ` 1 q -way target classifier.
We analyse crucial steps in PULSE in Sec. 8.
Our ideas for PULSE framework can be extended to separability condition since (3) continues to hold.
However, in our initial experiments, we observe that techniques proposed under strong positivitywere empirically stable and outperform methods developed under separability.
This is intuitive formany benchmark datasets where it is natural to assume that for each class there exists a subdomainthat only belongs to that class.
We describe this in more detail in App.
C.1.
[Table] [Table]
ExperimentsBaselines
We compare PULSE with several popular methods from OSDA literature.
Whilethese methods are not specifically proposed for OSLS
[Table]
they are introduced for the more general
OSDA problem.
In particular, we make comparions with DANCE [59], UAN [73], CMU [25],
STA [46], Backprop-ODA (or BODA)
We use the open source implementation availableat https://github.com/thuml .
For alternative baselines, we experiment with source classifierdirectly deployed on the target data which may contain novel class and label shift among sourceclasses (referred to as source-only ).
We also train a domain discriminator classifier for source versustarget (referred to as domain disc. ).
This is adaptation of PU learning baseline[24] which assumesno label shift among source classes.
Finally, per the reduction presented in Sec. 5, we train k PUclassifiers (referred to as k-PU ).
We include detailed description of each method in App. F.1.
Datasets We conduct experiments with seven benchmark classification datasets across vision, naturallanguage, biology and medicine.
For each dataset, we simulate an OSLS problem as described in nextparagraph.
For vision, we use CIFAR10, CIFAR100 [40] and Entity30 [61].
For language, we experiment with Newsgroups-20 ( http://qwone.com/~jason/20Newsgroups/ ) dataset.
Additionally,inspired by applications of OSLS in biology and medicine, we experiment with Tabula Muris [17](Gene Ontology prediction), Dermnet (skin disease prediction https://dermnetnz.org/ ), and
BreakHis [66] (tumor cell classification).
These datasets span language, image and table modalities.
We provide interpretation of OSLS problem for each dataset along with other details in App.
F.2.
OSLS Setup To simulate an OSLS problem, we experiment with different fraction of novel classprevalence, source label distribution, and target label distribution.
We randomly choose classes thatconstitute the novel target class.
After randomly choosing source and novel classes, we first splitthe training data from each source class randomly into two partitions.
This creates a random labeldistribution for shared classes among source and target.
We then club novel classes to assign them a new class (i.e. k ` 1 ).
Finally, we throw away labels for the target data to obtain an unsupervised DAproblem.
We repeat the same process on iid hold out data to obtain validation data with no target labels.
Training and Evaluation
We use Resnet18 [33] for CIFAR10, CIFAR100, and Entity30.
Fornewsgroups, we use a convolutional architecture.
For Tabular Muris and MNIST, we use a fullyconnected MLP.
For Dermnet and BreakHis, we use Resnet-50.
For all methods, we use the samebackbone for discriminator and source classifier.
For kPU, we use a separate final layer for each classwith the same backbone.
We use default hyperparameters for all methods.
For OSDA methods, weuse default method specific hyperparameters introduced in their works.
Since OSDA methods do notestimate the prevalence of novel class explicitly, we use the fraction of examples predicted in classk ` 1 as a surrogate.
We train models till the performance on validation source data (labeled) ceasesto increase.
Unlike OSDA methods, note that we do not use early stopping based on performanceon held-out labeled target data.
To evaluate classification performance, we report target accuracyon all classes, seen classes and the novel class.
For novel class prevalence estimation, we reportabsolute difference between the true and estimated marginal.
We open-source our code and by simplychanging a single config file, new OSLS setups can be generated and experimented with.
We provideprecise details about hyperparameters, OSLS setup for each dataset and code in App.
F.3.
Results Across different datasets, we observe that PULSE consistently outperforms other methodsfor the target classification and novel prevalence estimation (Table 1).
For detection of novel classes(Acc (Novel) column), kPU achieves superior performance as compared to alternative approachesbecause of its bias to default to p k ` 1 q th class.
This is evident by the sharp decrease in performance onpreviously seen classes.
For each dataset, we plot evolution of performance with training in App.
F.4.
We observe more stability in performance of PULSE as compared to other methods.
We observe that with default hyperparameters, popular OSDA methods significantly under performas compared to PULSE.
We hypothesize that the primary reasons underlying the poor performanceof OSDA methods are (i) the heuristics employed to detect novel classes; and (ii) loss functionsincorporated to improve alignment between examples from common classes in source and target.
Todetect novel classes, a standard heuristic employed in popular OSDA methods involves thresholdinguncertainty estimates (e.g., prediction entropy, softmax confidence [73, 25, 59]) at a predefinedthreshold κ .
However, a fixed κ , may not for different datasets and different fractions of the novel class.
In App.
F.5, we ablate by (i) removing loss function terms incorporated with an aim to improve sourcetarget alignment; and (ii) vary threshold κ and show improvements in performance of these methods.
In contrast, our two-stage method PULSE, first estimates the fraction of novel class which then guidesthe classification of novel class versus previously seen classes avoiding the need to guess κ .
Ablations Different datasets, in our setup span different fraction of novel class prevalence rangingfrom 0 .
22 (in CIFAR10) to 0 .
64 (in Tabula Muris).
For each dataset, we perform more ablations onthe novel class proportion in App. F.6.
For kPU and PULSE, in the main paper, we include results with
BBE and CVIR [29].
In App.
F.8, we perform experiments with alternative PU learning approachesand highlight the superiority of BBE and CVIR over other methods.
Moreover, since we have accessto unlabeled target data, we experiment with SimCLR [15] pre-training on the mixture of unlabeledsource and target dataset.
We include setup details and results in App.
F.7.
While pre-trained backbonearchitecture improves performance for all methods, PULSE continues to dominate other methods.
Analysis of PULSE Framework
In this section, we analyse key steps of our PULSE procedure for target label marginal estimation (Step3, 5 Algorithm 1) and learning the domain discriminator classifier (Step 5, Algorithm 1).
Due to spaceconstraints, we present informal results here and relegate formal statements and proofs to App.
D.
Theoretical analysis for target marginal estimation Building on BBE results from Garg et al.
[29],we present finite sample results for target label marginal estimation.
When the data satisfies strongpositivity, we observe that source classifiers often exhibit a threshold c y on softmax output of eachclass y P Y s above which the top bin (i.e., r c y , 1 s ) contains mostly examples from that class y .
We giveempirical evidence to this claim in App.
D.1.
Then, we show that the existence of (nearly) pure topbin for each class in f s is sufficient for Step 3 in Algorithm 1 to produce (nearly) consistent estimates:
Theorem 1 (Informal) .
Assume that for each class y P Y s , there exists a threshold c y such that forthe classifier f s , if r f s p x qs y ą c y for any x then the true label for that sample x is y .
Then, we have
[Equation]
The proof technique simply builds on the proof of Theorem 1 in Garg et al.
By assuming thatwe recover close to ground truth label marginal for source classes, we can also extend the aboveanalysis to Step 5 of Algorithm 1 to show convergence of estimate p p t p
y “ k ` 1 q to true prevalencep t p y “ k ` 1 q .
We discuss this further in App.
D.3.
Theoretical analysis of CVIR in population While the CVIR loss was proposed in Garg et al.
[29],no analysis was provided for convergence of the iterative gradient descent procedure.
In our work,we show that in population on a separable Gaussian dataset, CVIR will recover the optimal classifier.
We consider a binary classification problem where we have access to positive distribution (i.e., p p ),unlabeled distribution (i.e., p u : “ αp p ` p 1 ´ α q p n ), and mixture coefficient α .
Making a parallelconnection to Step 5 of PULSE, positive distribution p p here refers to the label shift corrected sourcedistribution p 1 s and p u refers to p t “ p t p
y “ k ` 1 q p t p x | y “ k ` 1 q`p 1 ´ p t
p
y
“ k ` 1 qq p 1 s p x q .
Ourgoal is to recover the classifier that discriminates p p versus p n (parallel p 1 s versus p t p¨|
y “ k ` 1 q ).
First we introduce some notation.
For a classifier f and loss function ℓ (i.e., logistic loss), define VIR α p f q “ inf t τ P R : P x „ p u p ℓ p x, ´ 1; f q ď τ q ě 1 ´ α u .
Intuitively, VIR α p f q identifies athreshold τ to capture bottom 1 ´ α fraction of the loss ℓ p x, ´ 1 q for points x sampled from p u .
Additionally, define CVIR loss as L p f, w q “ α E p p r ℓ p x, 1; f qs` E p u
r w p x q ℓ p x, ´ 1; f qs for classifierf and some weights w p x q P t 0 , 1 u .
Formally, given a classifier f t at an iterate t , CVIR procedureproceeds as follows:
[Equation]
We assume that x are drawn from two half multivariate Gaussian with mean zero and identitycovariance, i.e., x „ p p ô x “ γ 0 θ opt ` z | θ T opt z ě 0 , and x „ p n ô x “ ´ γ 0 θ opt ` z | θ T opt z ă0 , where z „ N p 0 , I d q . Here γ 0 is the margin and θ opt P R d is the true separator.
Here, we haveaccess to distribution p p , p u “ αp p ` p 1 ´ α q p n , and the true proportion α .
Theorem 2 (Informal) .
In the data setup detailed above, a linear classifier f p x ; θ q “ σ ` θ T x ˘ trainedwith CVIR procedure as in (4) - (5) will converge to an optimal positive versus negative classifier.
The proof uses a key idea that for any classifier θ not separating positive and negative data perfectly,the gradient in (5) is non-zero.
Hence, convergence of the CVIR procedure (implied by smoothness of CVIR loss) implies converge to an optimal classifier.
For separable datasets in general, we can extendthe above analysis with some modifications to the CVIR procedure.
We discuss this in App.
D.4.
Empirical investigation in overparameterized models As noted in our ablation experimentsand in Garg et al.
[29], domain discriminator trained with CVIR outperforms classifiers trainedwith other consistent objectives (nnPU [38] and uPU [21]).
While the above analysis highlightsconsistency of CVIR procedure in population, it doesn’t capture the observed empirical efficacy of
CVIR over alternative methods in overparameterized models.
In the Gaussian setup described above,we train overparameterized linear models to compare CVIR with other methods.
We discuss preciseexperiments and results in App.
E, but highlight the key takeaway here.
First, we observe that whena classifier is trained to distinguish positive and unlabeled data, early learning happens [47, 3, 28],i.e.
, during the initial phase of learning classifier learns to classify positives in unlabeled correctlyas positives.
Next, we show that post early learning rejection of large fraction of positives fromunlabeled training in equation (4) crucially helps CVIR.
Conclusion
In this work, we introduce OSLS a well-posed instantiation of OSDA that subsumes label shift and PUlearning into a framework for learning adaptive classifiers.
We presented identifiability conditions for
OSLS and proposed PULSE, a simple and effective approach to tackle the OSLS problem.
Moreover,our extensive experiments demonstrate efficacy of PULSE over popular OSDA alternatives whenthe OSLS assumptions are met.
We would like to highlight the brittle nature of benchmark drivenprogress in OSDA and hope that our work can help to stimulate more solid foundations and enablesystematic progress in this area.
In future work, we hope to bridge the gap between the necessary andsufficient identifiability conditions.
While we empirically investigate reasons for CVIR’s efficacyin overparameterized models, we aim to extend our theory to overparameterized settings in future.
Finally, we hope that our open source code and benchmarks will foster further progress on OSLS.