Unsupervised Learning under Latent Label Shift
Anonymous Author(s)
AffiliationAddress email
Abstract
What sorts of structure might enable a learner to discover classes from unlabeled data?
Traditional approaches rely on feature-space similarity and heroic assumptions on the data.
In this paper, we introduce unsupervised learning under
Latent Label Shift (LLS), where we have access to unlabeled data from multipledomains such that the label marginals p d p y q can shift across domains but the classconditionals p p x | y q do not.
This work instantiates a new principle for identifyingclasses: elements that shift together group together.
For finite input spaces, we establish an isomorphism between LLS and topic modeling: inputs correspond towords, domains to documents, and labels to topics.
Addressing continuous data,we prove that when each label’s support contains a separable region, analogous toan anchor word, oracle access to p p
d
| x q suffices to identify p d p y q and p d p y | x q upto permutation.
Thus motivated, we introduce a practical algorithm that leveragesdomain-discriminative models as follows: (i) push examples through domain discriminator p p
d | x q ; (ii) discretize the data by clustering examples in p p d | x q space;(iii) perform non-negative matrix factorization on the discrete data; (iv) combinethe recovered p p y | d q with the discriminator outputs p p d | x q to compute p d p y | x q @ d .
With semi-synthetic experiments, we show that our algorithm can leverage domaininformation to improve state of the art unsupervised classification methods.
Wereveal a failure mode of standard unsupervised classification methods when dataspace similarity does not indicate true groupings, and show empirically that ourmethod better handles this case.
Our results establish a deep connection betweendistribution shift and topic modeling, opening promising lines for future work.
Introduction
Discovering systems of categories from unlabeled data is a fundamental but ill-posed challenge inmachine learning.
Typical unsupervised learning methods group instances together based on featurespace similarity.
Accordingly, given a collection of photographs of animals, a practitioner might hopethat, in some appropriate feature space, images of animals of the same species should be somehowsimilar to each other.
But why should we expect a clustering algorithm to recognize that dogs viewedin sunlight and dogs viewed at night belong to the same category?
Why should we expect thatbutterflies and caterpillars should lie close together in feature space?
In this paper, we offer an alternative principle according to which we might identify a set of classes:we exploit distribution shift across times and locations to reveal otherwise unrecognizable groupingsamong examples.
For example, if we noticed that whenever we found ourselves in a location wherebutterflies are abundant, caterpillars were similarly abundant, and that whenever butterflies werescarce, caterpillars had a similar drop in prevalence, we might conclude that the two were tied to thesame underlying concept, no matter how different they appear in feature space.
In short, our principlesuggests that latent classes might be uncovered whenever instances that shift together group together .
pairs
Domain Domain
Domain r
D o m a i n
D i sc r
i m i n a t
o r
Cluster
NMF
Formalizing this intuition, we introduce the problem of unsupervised learning under Latent Label
Shift (LLS).
Here, we assume access to a collection of domains d P t 1 , . . .
, r u , where the mixtureproportions p d p y q vary across domains but the class conditional distribution p p x | y q is domaininvariant.
Our goals are to recover the underlying classes up to permutation, and thus to identifyboth the per-domain mixture proportions p
d p y q and optimally adapted per-domain classifiers p d p y | x q .
The essential feature of our setup is that only the true y ’s, as characterized by their class-conditionaldistributions p p x | y q , could account for the observed shifts in p d p x q .
We prove that under mildassumptions, knowledge of this underlying structure is sufficient for inducing the full set of categories.
First, we focus on the tabular setting , demonstrating that when the input space is discrete andfinite, LLS is isomorphic to topic modeling [8].
Here, each distinct input x maps to a word eachlatent label y maps to a topic and each domain d maps to a document.
In this case, we can applystandard identification results for topic modeling [20, 4, 27, 33, 12] that rely only on the existenceof anchor words within each topic (i.e., for each label y there is at least one x in the support of y ,that is not in the support of any y 1 ‰ y ).
Here, standard methods based on Non-negative Matrix
Factorization (NMF) can recover each domain’s underlying mixture proportion p d p y q and optimalpredictor p d p y | x q [20, 33, 27].
However, the restriction to discrete inputs, while appropriate for topicmodeling, proves restrictive when our interests extend to high-dimensional continuous input spaces.
Then, to handle high-dimensional inputs, we propose Discriminate-Discretize-Factorize-Adjust(DDFA) , a general framework that proceeds in the following steps: (i) pool data from all domains toproduce a mixture distribution q p x, d q ; (ii) train a domain discriminative model f to predict q p d | x q ;(iii) push all data through f , cluster examples in the pushforward distribution, and tabularize thedata based on cluster membership; (iv) solve the resulting discrete topic modeling problem (e.g., via
NMF), estimating q p y, d q up to permutation of the latent labels; (v) combine the predicted q p d | x q andq p y, d q to estimate p d p y q and p d p y | x q .
In developing this approach, we draw inspiration from recentworks on distribution shift and learning from positive and unlabeled data that (i) leverage black boxpredictors to perform dimensionality reduction [40, 23, 24]; and (ii) work with anchor sets , separablesubsets of continuous input spaces that belong to only one class’s support [52, 41, 21, 6, 24].
Our key theoretical result shows that domain discrimination ( q p d | x q ) provides a sufficient representation for identifying all parameters of interest.
Given oracle access to q p d | x q (which is identifiedwithout labels), our procedure is consistent.
Our analysis reveals that the true q p d | x q maps all pointsin the same anchor set to a single point mass in the push-forward distribution.
This motivates ourpractical approach of discretizing data by hunting for tight clusters in q p d | x q space.
In semi-synthetic experiments, we adapt existing image classification benchmarks to the LLS setting,sampling without replacement to construct collections of label-shifted domains.
We note that traininga domain discriminative classifier is a difficult task, and find that warm starting the initial layers of ourmodel with pretrained weights from unsupervised approaches can significantly boost performance.
Weshow that warm-started DDFA outperforms state-of-the-art (SOTA) unsupervised approaches whendomain marginals p d p y q are sufficiently sparse.
In particular, we observe improvements of as muchas 30% accuracy over unsupervised SOTA on CIFAR-20.
Further, on subsets of FieldGuide dataset,where similarity between species and diversity within a species leads to failure of unsupervisedlearning, we show that DDFA recovers the true distinctions.
To be clear, these are not apples-to- apples comparisons: our methods are specifically tailored to the LLS setting.
The takeaway is that thestructure of the LLS setting can be exploited to outperform the best unsupervised learning heuristics.
Related Work
Unsupervised Learning Standard unsupervised learning approaches for discovering labels often relyon similarity in the original data space [42, 50].
While distances in feature space become meaninglessfor high-dimensional data, deep learning researchers have turned to similarity in a representationsspace learned via self-supervised contrastive tasks [44, 19, 26, 11], or similarity in a feature spacelearned end-to-end for a clustering task [9, 10, 47, 57].
Our problem setup closely resemblesindependent component analysis (ICA), where one seeks to identify statistically independent signalcomponents from mixtures [34].
However, ICA’s assumption of statistical independence amongthe components does not obtain in our setup.
In topic modeling [8, 4, 33, 12, 46], documents aremodeled as mixtures of topics, and topics as categorical distributions over a finite vocabulary.
Earlytopic models include the well-known Latent Dirichlet Allocation (LDA) [8], which assumes thattopic mixing coefficients are drawn from a Dirichlet distribution, along with papers with morerelaxed assumptions on the distribution of topic mixing coefficients (e.g., pLSI)
The topicmodeling literature often draws on non-negative Matrix Factorization (NMF) methods [45, 53], whichdecompose a given matrix into a product of two matrices with non-negative elements [18, 16, 25, 28].
In both Topic Modeling and NMF, a fundamental problem has been to characterize the preciseconditions under which the system is uniquely identifiable [20, 4, 33, 12].
The anchor condition (alsoreferred to as separability) is known to be instrumental for identifying topic models [4, 12, 33, 20].
In this work, we extend these ideas, leveraging separable subsets of each label’s support (the anchorsets) to produce anchor words in the discretized problem.
Existing methods have attempted to extendlatent variable modeling to continuous input domains by making assumptions about the functionalforms of the class-conditional densities, e.g., restricting to Gaussian mixtures [50, 49].
A second lineof approach involves finding an appropriate discretization of the continuous space [56].
Distribution Shift under the Label Shift Assumption The label shift assumption, where p d p y q canvary
but p p x | y q cannot, has been extensively studied in the domain adaptation literature [51, 54, 60,40, 29, 23] and also features in the problem of learning from positive and unlabeled data [22, 7, 24].
For both problems, many classical approaches suffer from the curse of dimensionality, failing inthe settings where deep learning prevails.
Our solution strategy draws inspiration from recent workon label shift [40, 1, 5, 23] and PU learning [7, 41, 52, 24] that leverage black-box predictors toproduce sufficient low-dimensional representations for identifying target distributions of interest(other works leverage black box predictors heuristically [36]).
Key differences: While PU learningrequires identifying one new class for which we lack labeled examples provided that the positiveclass contains an anchor set [24], LLS can identify an arbitrary number of classes (up to permutation)from completely unlabeled data, provided a sufficient number of domains.
Domain Generalization
The related problem of Domain Generalization (DG) also addresseslearning with data drawn from multiple distributions and where the domain identifiers play a keyrole [43, 2].
However in DG, we are given labeled data from multiple domains, and our goal is tolearn a classifier that can generalize to new domains.
By contrast, in LLS, we work with unlabeleddata only, leveraging the problem structure to identify the underlying labels.
Unsupervised Learning under Latent Label Shift
Notation For a vector v P R p , we use v j to denote its j th entry, and for an event E , we let I r E sdenote the binary indicator of the event.
By | A | , we denote the cardinality of set A .
With r n s , wedenote the set t 1 , 2 , . . . , n u .
We use r A
s i,j to access the element at p i, j q in A .
Let X be the inputspace and Y “ t 1 , 2 , . . .
, k u be the output space for multiclass classification.
We assume throughoutthis work that the number of true classes k is known.
Throughout this paper, we use capital lettersto denote random variables and small case letters to denote the corresponding values they take.
Forexample, by X we denote the input random variable and by x , we denote a value that X may take.
We now formally introduce the problem of unsupervised learning under LLS.
In LLS, we assumethat we observe unlabeled data from r domains.
Let R “ t 1 , 2 , . . .
, r u be the set of domains.
By p d ,we denote the probability density (or mass) function for each domain d P R .
D
Y
X
Definition 1 (Latent label shift) .
We observe data from r domains.
While the label distribution candiffer across the domains, for all d, d 1 P R and for all p x, y q P X ˆ Y , we have p d p x | y q “ p d 1 p x | y q .
Simply put, Definition 1 states that the conditional distribution p d p x | y q remains invariant acrossdomains, i.e., they satisfy the label shift assumption.
Thus, we can drop the subscript on this factor,denoting all p d p x | y q by p p x | y q .
Crucially, under LLS, p d p y q can vary across different domains.
Under LLS, we observe unlabeled data with domain label tp x 1 , d 1 q , p x 2 , d 2 q , . .
, p x n , d n qu .
Ourgoal breaks down into two tasks.
Up to permutation of labels, we aim to (i) estimate the label marginalin each domain p d p y q ; and (ii) estimate the optimal per-domain predictor p
d
p y | x q .
Mixing distribution Q A key step in our algorithm will be to train a domain discriminative model.
Towards this end we define Q , a distribution over X ˆ Y ˆ R , constructed by taking a uniform mixtureover all domains.
By q , we denote the probability density (or mass) function of Q .
Define Q suchthat q p x, y | D “ d q “ p d p x, y q , i.e., when we condition on D “ d we recover the joint distributionover X ˆ Y specific to that domain d .
For all d P R , we define γ d “ q p d q , i.e., the prevalence of eachdomain in our distribution Q .
Notice that q p x, y q is a mixture over the distributions t p d p x, y qu d P R ,with t γ d u d P R as the corresponding mixture coefficients.
Under LLS (Definition 1), X does notdepend on D when conditioned on Y (Fig. 2).
Additional notation for the discrete case To begin, we setup notation for discrete input spaces with|X| “ m .
Without loss of generality, we assume that X “ t 1 , 2 , . . .
, m u .
The label shift assumptionallows us to formulate the label marginal estimation problem in matrix form.
Let Q X | D be an m ˆ rmatrix such that r Q X
| D
s i,d “ p d p X “ i q , i.e., the d -th column of Q X | D is p d p x q .
Let Q X | Y be anm ˆ k matrix such that r Q X | Y s i,j “ p p X
“ i | Y “ j q , the j -th column is a distribution over X given
Y “ j .
Similarly, define Q Y | D as a k ˆ r matrix whose d -th column is the domain marginal p d p y q .
Now with Definition 1, we have p
[Equation]
p
[Equation]
Since this is true @
d P R , we can express this in a matrix form as Q X | D “ Q X | Y Q Y | D .
Additional assumptions Before we present identifiability results for the LLS problem, we introducefour additional assumptions required throughout the paper:
A.1 There are at least as many domains as classes, i.e., |R| ě |Y| .
A.
The matrix formed by label marginals (as columns) across different domains is full-rank,
i
e., rank p Q Y | D q “ k .
A.3 Equal representation of domains,
i
e., for all d P R , γ d “ 1 { r .A.
Fix ϵ ą 0 .
For all y P Y , there exists a unique subdomain A y Ď X , such that q p A y q ě ϵ
and x P A y if and only if the following conditions are satisfied: q p x | y q ą 0 and q p x | y 1 q “ 0for all y 1 P Y zt y u .
We refer to this
assumption as the ϵ -anchor sub-domain condition.
We now comment on the assumptions.
A.1–A.2 are benign, these assumptions just imply that thematrix Q Y | D is full row rank.
Without loss of generality, A.3 can be assumed when dealing withdata from a collection of domains.
When this condition is not satisfied, one could just re-sample datapoints uniformly at random from each domain d .
Intuitively, A.4 states that for each label y P Y , wehave some subset of inputs that only belong to that class y .
To avoid vanishing probability of thissubset, we ensure at least ϵ probability mass in our mixing distribution Q .
The anchor word conditionis related to the positive sub-domain in PU learning, which requires that there exists a subset of X inwhich all examples only belong to the positive class [52, 41, 21, 6].
Theoretical Analysis
In this section, we establish identifiability of LLS problem.
We begin by considering the case wherethe input space is discrete and formalize the isomorphism to topic modeling.
Then we establishthe identifiability of the system in this discrete setting by appealing to existing results in topic modeling [33].
Finally, extending results from discrete case, we provide novel analysis to establishour identifiability result for the continuous setting.
Isomorphism to topic modeling Recall that for the discrete input setting, we have the matrixformulation: Q X | D “ Q X | Y Q Y | D .
Consider a corpus of r documents, consisting of terms from avocabulary of size m .
Let D be an R m ˆ r matrix representing the underlying corpus.
Each column of
D represents a document, and each row represents a term in the vocabulary.
Each element r D s i,jrepresents the frequency of term i in document j .
Topic modeling [8, 32, 33, 4] considers eachdocument to be composed as a mixture of k topics.
Each topic prescribes a frequency with which theterms in the vocabulary occur given that topic.
Further, the proportion of each topic varies acrossdocuments with the frequency of terms given topic remaining invariant.
We can state the topic modeling problem as: D “ CW , where C is an R m ˆ k matrix, r C s i,jrepresents the frequency of term i given topic j , and W is an R k ˆ r matrix, where r W s i,j representsthe proportion of topic i in document j .
Note that all three matrices are column normalized.
Theisomorphism is then between document and domain, topic and label, term and input sample, i.e.,
D “ CW ” Q X | D “ Q X | Y Q Y | D .
In both the cases, we are interested in decomposing aknown matrix into two unknown matrices.
This formulation is examined as a non-negative matrixfactorization problem with an added simplicial constraint on the columns (columns sum to 1) [3, 27].
Identifiability of the topic modeling problem is well-established [20, 4, 27, 33, 12].
We leveragethe isomorphism to topic modeling to extend this identifiability condition to our LLS setting.
Weformalize the adaptation here:
Theorem 1.
(adapted from Proposition 1 in Huang et al.
[33]) Assume A.1, A.2 and A.4 hold(A.4 in the discrete setting is referred to as the anchor word condition).
Then the solution to
Q X | D “ Q X | Y Q Y | D is uniquely identified up to permutation of class labels.
We refer readers to Huang et al.
[33] for a proof of this theorem.
Intuitively, Theorem 1 states that ifeach label y has at least one token in the input space that has support only in y , and A.1, A.2 hold,then the solution to Q X | Y , Q Y | D is unique.
Furthermore, under this condition, there exist algorithmsthat can recover Q X | Y , Q Y | D within some permutation of class labels [33, 27, 3, 4].
Extensions to the continuous case We will prove identifiability in the continuous setting, when
X “ R p for some p ě 1 .
In addition to A.1–A.4, we make an additional assumption that we haveoracle access to q p d | x q , i.e., the true domain discriminator for mixture distribution Q .
This is impliedby assuming access to the marginal q p x, d q from which we observe our samples.
Formally, we definea push forward function f such that r f p x qs d “ q p d | x q , then push the data forward through f to obtainoutputs in ∆ r ´ 1 .
In the proof of Theorem 2, we will show that these outputs can be discretized in afashion that maps anchor subdomains to anchor words in a tabular, discrete setting.
We separatelyremark that the anchor word outputs are in fact extreme corners of the convex polytope in ∆
r ´ 1 whichencloses all f p x q mass; we discuss this geometry further in App.
G.
After constructing the anchorword discretization, we appeal to Theorem 1 to recover Q Y | D .
Given Q Y | D , we show that we canuse Bayes’ rule and the LLS condition (Definition 1) to identify the distribution q p y | x, d q “ p d p y | x qover latent variable y .
We formalize this in the following theorem:
Theorem 2.
Let the distribution Q over random variables X, Y, D satisfy Assumptions A.1–A.4.
Assuming access to the joint distribution q p x, d q , and knowledge of the number of true classes k , weshow that the following quantities are identifiable: (i) Q Y | D , (ii) q p
y
| X “ x q , for all x P X thatlies in the support (i.e. q p x q ą 0 ); and (iii) q p y | X “ x, D “ d q , for all x P X and d P R such thatq p x, d q ą 0 .
Before presenting a proof sketch for Theorem 2, we first present key lemmas (we include their proofsin App. C).
Lemma 1.
Under the same assumptions as Theorem 2, the matrix Q Y | D and f p x q “ q p d | x q uniquelydetermine q p y | x q for all y P Y and x P X such that q p x q ą 0 .
Lemma 1 states that given matrix Q Y | D and oracle domain discriminator, we can uniquely identifyq p y | x q .
In particular, we show that for any x P X , q p d | x q can be expressed as a convex combinationof the k columns of Q D | Y (which is computed from Q Y | D and is column rank k ) and the coefficientsof the combination are q p y | x q .
Combining this with the linear independence of the columns of Q D | Y ,we show that these coefficients are unique.
In the following lemma, we show how the identifiedq p y | x q can then be used to identify q p y | x, d q :
Lemma 2.
Under the same assumptions as Theorem 2, for all y P Y , and x P X such that q p x, d q ą 0 .the
matrix Q Y | D and q p y | x q uniquely determine q p y | x, d q .
To prove Lemma 2, we show that we can combine the conditional distribution over the labels givena sample x P X with the prior distribution of the labels in each domain to determine the posteriordistribution over labels given the sample x and the domain of interest.
Next, we introduce a keyproperty of the domain discriminator classifier f :
Lemma 3.
Under the same assumptions as Theorem 2, for all x, x 1 in anchor sub-domain, i.e.,x, x 1 P A y for a given label y P Y , we have f p x q “ f p x 1 q .
Further, for any y P Y , if x P A y , x 1 R A y ,then f p x q ‰ f p x 1 q .
Lemma 3 implies that the oracle domain discriminator f maps all points in an anchor subdomain,and only those points in that anchor subdomain to the same point in f p x q “ q p d | x q space.
We cannow present a proof sketch for Theorem 2 (full proof in App.
C):
Proof sketch of Theorem 2.
The key idea of the proof lies in proposing a discretization such that somesubset of anchor subdomains for each label y in the continuous space map to distinct anchor words indiscrete space.
In particular, if there exists a discretization of the continuous space X that for anyy P Y , maps all
x P A y to the same point in the discrete space, but no x R A y maps to this point, thenthis point serves as an anchor word.
From Lemma 3, we know that all the x P A y and only the x P A yget mapped to specific points in the f p x q space.
Pushing all the x P X through f , we know from A.4that there exists k point masses of size ϵ , one for each f p
A y q in the f p x q “ q p d | x q space.
We can nowinspect this space for point masses of size at least ϵ to find at most O p 1 { ϵ q such point masses amongwhich are contained the k point masses corresponding to the anchor subdomains.
Discretizing thisspace by assigning each point mass to a group (and non-point masses to a single additional group),we have k groups that have support only in one y each.
Thus, we have achieved a discretizationwith anchor words.
Further, since the discrete space arises from a pushforward of the continuousspace through f , the discrete space also satisfies the latent label shift assumption A.1.
We now use
Theorem 1 to claim identifiability of Q Y | D .
We then use Lemmas 1 and 2 to prove parts (ii) and (iii).
DDFA Framework
Motivated by our identifiability analysis, in this section, we present an algorithm to estimate
Q Y | D , q p y | x q , and q p y | x, d q when X is continuous by exploiting domain structure and approximating the true domain discriminator f .
Intuitively, q p y | x, d q is the domain specific classifier p d
p y | x q andq p y | x q is the classifier for data from aggregated domains.
Q Y | D captures label marginal for individual domains.
A naive approach would be to aggregate data from different domains and exploit recentadvancements in unsupervised learning [57, 47, 9, 10].
However, aggregating data from multiple domains loses the domain structure that we hope to leverage.
We highlight this failure mode of the traditional unsupervised clustering method in Sec.
We remark that DDFA draws heavy inspiration fromthe proof of Theorem 2, but we do not present a guarantee that the DDFA solution will converge tothe identifiable solution.
This is primarily due to the K-means clustering heuristic we rely on, whichempirically offers effective noise tolerance, but theoretically has no guarantee of correct convergence.
Discriminate We begin Algorithm 1 by creating a split of the unlabeled samples into the trainingand validation sets.
Using the unlabeled data samples and the domain that each sample originatedfrom, we first train a domain discriminative classifier p f .
The domain discriminative classifier outputsa distribution over domains for a given input.
This classifier is trained with cross-entropy loss topredict the domain label of each sample on the training set.
With unlimited data, the minimizer ofthis loss is the true f , as we prove in App.
D. To avoid overfitting, we stop training p f
when the crossentropy loss on the validation set stops decreasing.
Note that here the validation set also only containsdomain indices (and no information about true class labels).
Discretize We now push forward all the samples from the training and validation sets through thedomain discriminator to get vector p
f p x i q for each sample x i .
In the proof of Theorem 2, we argue thatwhen working with true f , and the entire marginal q p x, d q , we can choose a discretization satisfyingthe anchor word assumption by identifying point masses in the distribution of f p x q and filtering toinclude those of at least ϵ mass.
In the practical setting, because we have only a finite set of datapoints and a noisy p f , we use clustering to approximately find point masses.
We choose m ě k and
[Table] [Table]
[Table]
recover m clusters with any standard clustering procedure (e.g. K-means).
This clustering procedureis effectively a useful, but imperfect heuristic: if the noise in p f is sufficiently small and the clusteringsufficiently granular, we hope that our m discovered clusters will include k pure clusters, each ofwhich only contains data points from a different anchor subdomain which are tightly packed aroundthe true
f p A y q for the corresponding label y .
Clustering in this space is superior to a naive clusteringon the input space because close proximity in this space indicates similarity in q p d | x q .
Let us denote the learned clustering function as c , where c p x q is the cluster assigned to a datapointx .
We now leverage the cluster id c p x i q of each sample x i to discretize sample into a finite discretespace r m s .
Combining cluster id with the domain source d i for each sample, we estimate p Q c p X q| Dby simply computing, for each domain, the fraction of its samples assigned to each cluster.
Factorize We apply an NMF algorithm to p Q c p X q| D to obtain estimates of p Q c p X q| Y and p Q Y | D .
Adjust We begin Algorithm 2 by considering a test point p x 1 , d 1 q .
To make a prediction, if we hadaccess to oracle f and true Q Y | D , we could precisely compute q p y | x 1 q (Lemma 1).
However, in placeof these true quantities, we plug in the estimates p f and p Q Y | D .
Since our estimates contain noise, theestimate p q p y | x 1 q is found by left-multiplying p f p x 1 q with the pseudo-inverse of p Q D | Y , as opposedto solving a sufficient system of equations.
As our estimates p f and p Q D | Y approach the true values,the projection of p f p x 1 q into the column space of p Q D | Y tends to p f p x 1 q itself, so the pseudo-inverseapproaches the true solution.
Now we can use the constructive procedure introduced in the proof of
Lemma 2 to compute the plug-in estimate p q p y | x 1 , d 1 q “ p p d 1 p y | x 1 q .
Algorithm input p Q Y | D , p f, p x
[Equation]
d
[Equation]
q 1: Populate p Q D s
Ð
[Equation] [Equation]
Ð
[Equation]
q
[Equation] [Equation]
X “
[Equation]
D
[Equation]
output : p
[Equation]
p
[Equation] [Equation]
pred
Experiments
Baselines We select the unsupervised classification method SCAN as a state-of-the-art baseline [57].
SCAN pretrains a ResNet [30] backbone using SimCLR [11] and MoCo [31] setups (pretext tasks).
SCAN then trains a clustering head to minimize the SCAN loss (refer [57] for more details) 1 .
Wemake sure to evaluate SCAN on the same potentially class-imbalanced test subset we create for eachexperiment.
Since SCAN is fit on a superset of the data DDFA sees, we believe this gives a slightdata advantage to the SCAN baseline (although we acknowledge that the class balance for SCANtraining is also potentially different from its evaluation class balance).
To evaluate SCAN, we usethe public pretrained weights available for CIFAR-10, CIFAR-20, and ImageNet-50.
We also train
SCAN ourselves on the train and validation portions of the FieldGuide2 and FieldGuide28 datasetswith a ResNet18 backbone and SimCLR pretext task.
We replicate the hyperparameters used for
CIFAR training.
Datasets First we examine standard multiclass image datasets CIFAR-10, CIFAR-20 [38], and
ImageNet-50 [17] containing images from 10, 20, and 50 classes respectively.
Images in these datasetstypically focus on a single large object which dominates the center of the frame, so unsupervisedclassification methods which respond strongly to similarity in visual space are well-suited to recovertrue classes up to permutation.
These datasets are often believed to be separable (i.e., single true labelapplies to each image), so every example falls in an anchor subdomain (satisfying A.4).
Motivated by the application of LLS problem, we consider the FieldGuide dataset 2 , which containsimages of moths and butterflies.
The true classes in this dataset are species, but each class containsimages taken in immature (caterpillar) and adult stages of life.
Based on the intuition that butterfliesfrom a given species look more like butterflies from other species than caterpillars from their ownspecies, we hypothesize that unsupervised classification will learn incorrect class boundaries whichdistinguish caterpillars from butterflies, as opposed to recovering the true class boundaries.
Dueto high visual similarity between members of different classes, this dataset may indeed have slightoverlap between classes.
However, we hypothesize that anchor subdomain still holds, i.e., thereexist some images from each class that could only come from that class.
Additionally, if we haveaccess to data from multiple domains, it is natural to assume that within each domain the relativedistribution of caterpillar to adult stages of each species stay relatively constant as compared toprevalence of different species.
We create two subsets of this dataset: FieldGuide2, with two species,and FieldGuide28, with 28 species.
LLS Setup The full sampling procedure for semisynthetic experiments is described in App.
E.
Roughly, we sample p d p y q from a symmetric Dirichlet distribution with concentration α { k , where kis the number of classes and α is a generation parameter that adjusts the difficulty of the syntheticproblem, and enforce maximum condition number κ on Q Y | D .
Small α and small κ encouragessparsity in Q Y | D , so each label tends to only appear in a few domains.
Larger parameters encouragesp d p y q to tend toward uniform.
We draw from test, train, and valid datasets without replacement tomatch these distributions, but discard some examples due to class imbalance.
Training and Evaluation
The algorithm uses train and validation data consisting of pairs of imagesand domain indices.
We train ResNet50 [30] (with added dropout) on images x i with domain indicesd i as the label, choose best iteration by valid loss, pass all training and validation data through p f , andcluster pushforward predictions p
f p x i q into m ě k clusters with Faiss K-Means [37].
We compute thep Q c p X q| D matrix and run NMF to obtain p Q c p X q| Y , p Q Y | D .
To make columns sum to 1, we normalizecolumns of p Q c p X q| Y , multiply each column’s normalization coefficient over the corresponding row
ofp Q Y | D (to preserve correctness of the decomposition), and then normalize columns of p Q Y | D .
Some
NMF algorithms only output solutions satisfying the anchor word property [3, 39, 27].
We found thestrict requirement of an exact anchor word solution to lead to low noise tolerance.
We therefore usethe Sklearn implementation of standard NMF [13, 55, 48].
We instantiate the domain discriminator as ResNet18, and preseed its backbone with SCAN [57]pre-trained weights or [57] contrastive pre-text weights.
We denote these models DDFA (SI) and
DDFA (SPI) respectively.
We predict class labels with Algorithm 2 .
With the Hungarian algorithm,implemented in [14, 58], we compute the highest true accuracy among any permutation of these labels(denoted “Test acc”).
With the same permutation, we reorder rows of p P Y | D , then compute the averageabsolute difference between corresponding entries of p Q Y | D and Q Y | D (denoted “ Q Y | D err”).
[Table]
Results On CIFAR-10, we observe that DDFA alone is incapable of matching highly competitivestate-of-the-art baseline SCAN performance—however, in suitably sparse problem settings (smallα ), it comes substantially close, indicating good recovery of true classes.
Due to space constraints,we include CIFAR-10 results in App.
F. DDFA (SI) combines SCAN’s strong pretrain with domaindiscrimination fine-tuning to outperform SCAN in the easiest, sparsest setting and certain densersettings.
On CIFAR-20, baseline SCAN is much less competitive, so our DDFA(SI) dominatesbaseline SCAN in all settings except the densest (Table 1).
These results demonstrate how addingdomain information can dramatically boost unsupervised baselines.
On FieldGuide-2, DDFA (SPI) outperforms SCAN baselines across all problem settings and domaincounts (Table 4); in sparser settings, the accuracy gap is 20-30%.
In this dataset, SCAN performs onlyslightly above chance, reflecting perhaps a total misalignment of learned class distinctions with truespecies boundaries.
We do not believe that SCAN is too weak to effectively detect image groupingson this data; instead we acknowledge that the domain information available to DDFA (SPI) (andnot to SCAN) is informative for ensuring recovery of the true class distinction between species asopposed to the visually striking distinction between adult and immature life stages.
Results frommore domains are available in App.
F.
On FieldGuide-28 (Table 5), DDFA outperforms SCAN when
Q Y | D is sufficiently sparse (sampled with α : 0 .
5 or α : 3 ), with the highest observed accuracydifference ranging above 30-40%.
Conclusion
Our theoretical results demonstrate that under LLS, we can leverage shifts among previously seendomains to recover labels in a purely unsupervised manner, and our practical instantiation of the
DDFA framework demonstrates both (i) the practical efficacy of our approach; and (ii) that genericunsupervised methods can play a key role both in clustering discriminator outputs, and providingweights for initializing the discriminator.
We believe that this work is just the first step in a newdirection for leveraging structural assumptions together with distribution shift to perform unsupervisedlearning.
Within the LLS setup, several components of the DDFA framework warrant further investigation:(i) the deep domain discriminator can be enhanced in myriad ways; (ii) for clustering discriminatoroutputs, we might develop methods specially tailored to our setting to replace the current genericclustering heuristic; (iii) clustering might be replaced altogether with geometrically informed methodsthat directly identify the corners of the polytope; (iv) the theory of LLS can be extended beyondidentification to provide statistical results that might hold when q p d | x q can only be noisily estimated,and when only finite samples are available for the induced topic modeling problem; (v) when thenumber of true classes k is unknown, we may develop approaches to estimate this k .
B Proofs of Lemmas
In this section, we present several new lemmas which are required to prove Theorem 2, and provideproofs.
We also provide proofs for Lemmas 1, 2, and 3.
Lemma 4.
Let distribution Q over random variables X, Y, D satisfy A.1–A.4.
Then for all y P Y ,q p
y q ą 0 .
That is, all labels have nonzero probability under Q .
Proof of Lemma 4.
Proof by contradiction.
Let y P Y with q p y q “ 0 .q p
y q “ ÿ d P R q p d q q p y
[Equation]
D “
[Equation]
“ ÿ d P R
[Equation]
“ ÿ d
[Equation] [Equation]
[Equation]
ÿ d P
[Equation]
q p
[Equation]
[Equation]
Since q p y | D “ d q ě 0 for all d P R , we see that if q p y q “ 0 , then q p y | D “ d q “ 0 for all d P R .
Then r Q Y | D s y,d “ 0 for all d P R .
Then there is a row (row d ) in the matrix Q Y | D in which everyentry is 0, so Q Y | D cannot be full row rank k .
This violates assumption A.2.
Then by contradictionwe have shown q p y q ą 0 .
Lemma 5.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1–A.4.
Let x
P Xsuch that q p x q ą 0 .
Then if x P A y for some y P Y , we have that q p y | X “ x q “ 1 , and for all y 1 P
Y zt y u , q p y 1 |
X “ x q “ 0 .
The converse is also true:
if q p y | X “ x q “ 1 for some y P Y andq p y 1 | X “ x q “ 0 @ y 1 P Y zt y u , then we know that x P A y .
Proof of Lemma 5.
We prove directions one at a time.
Forward direction.
Assume x P A
y .q
p x q “ ÿ y 2 P Y q p y
2 q q p x | Y “ y 2 q q p x q “ q p y q q p x | Y “ y q ` ÿ y 1 P Y
zt y u
q p y 1 q q p x | Y “ y 1 q q p x q “ q p y q q p x | Y “ y q ` ÿ y 1 P Y
zt y u
q p y 1 q p 0 q q p x q “ q p y q q p x | Y “ y q
Recalling q p x | y q ą 0 (by A.4) and q p y q ą 0 (by Lemma 4), we know that q p x q “ q p y q q p x | Y “y q ą 0 .
Then q p y | X “ x q “ q p y q q p x | Y “ y q q p x q “ q p x q q p x q “ 1 (Bayes’ rule).
Because probabilitiessum to 1, q p y | X “ x q ` ř y 1 P Y
zt y u
q p y 1 | X “ x q “ 1 .
Then because q p y | X “ x q “ 1 , we have :ř y 1 P Y
zt y u
q p y 1 | X “ x q “ 0 .
Then for all y 1 P Y zt y u , it must be that q p y 1 | X “ x q “ 0 .
Then we haveshown q p y | X “ x q “ 1 , and for all y 1 P Y zt y u ,
q p y 1 | X “ x q “ 0 .
Converse.
Assume q p y | X “ x q “ 1 and for all y 1 P Y zt y u , q p
y
| X “ x q “ 0 .
We recall thatq p x q ą 0 .
Also, q p y q ą 0 by Lemma 4.
Then q p x | Y “ y q “ q p y | X “ x q q p x q q p y q “ p 1 q q p x q q p y q ą 0 .
Lety 1 P Y zt y u .
Then q p x | Y “ y 1 q “ q p y 1 | X “ x q q p x q q p y 1 q “ p 0 q q p x q q p y 1 q “ 0 .
Then because q p x | Y “ y q ą 0and @
y 1 P Y zt y u , q p x | Y “ y 1 q “ 0 , we see that x P A y .
Lemma 6.
Let random variables X, Y, D and distribution Q satisfy Assumptions A.1–A.4.
Then,the matrix Q D | Y , defined as an r ˆ k matrix whose elements are r Q D | Y s i,j “ Q p D “ i | Y “ j q ,and in which each column is a conditional distribution over the domains given a label, has linearlyindependent columns.
Furthermore, Q D | Y can be computed directly from only Q Y | D .
Proof of Lemma 6.
Let random variables X, Y, D and distribution Q satisfy Assumptions A.1–A.4.
Each
[Equation]
Since each y th column of Q D | Y is a probability distribution that sums to 1, and rq p y q is constantdown each y th column, we can obtain Q D | Y by simply taking Q J Y | D , in which each r Q J Y
| D s d,y “r Q Y | D s y,d “ q p y | D “ d q , and normalizing the columns so they sum to 1.
The matrix Q Y | D has linearly independent rows by Assumption A.2.
Then Q J Y | D has linearlyindependent columns.
Scaling these columns by a nonzero value does not change their linearindependence, so the columns of Q D | Y are also linearly independent.
Then matrix Q D | Y has linearly independent columns, and can be computed by taking Q J Y | D andnormalizing its columns.
Lemma 7.
Let random variables X, Y, D and distribution Q satisfy Assumptions A.1–A.4.
Letd P R , x P X , y P Y .
Then q p d | X “ x, Y “ y q “ q p d | Y “ y q .
Proof of Lemma 7.
[Equation]
Proof of Lemma 1.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
Let x P X with q p x q ą 0 , and y P Y .
Assume we know Q Y | D and r f p x qs d “ q p d | X “ x q .
With Q Y | D , we know q p y | D “ d q for all y, d .
Also, with the oracle domain discriminator f , we are able to obtain q p d | X “ x q for all x, d .
For all x P X , d P R , q p d | X “ x q “
[Equation] [Equation]
, using Lemma 7 .
Define the vector-valued function g : X Ñ R k such that r g p x qs y “ q p y | X “ x q for all x Psupp Q p X q .
Q D | Y is a matrix of shape r ˆ k , with r Q D
| Y s i,j “ Q p D “ i | Y “ j q .
It can becomputed from Q Y | D and has linearly independent columns—both facts shown in Lemma 6.
Then r f p x qs d “ q p d | X “ x q “ Q D | Y r d, : s g p x q , a product between the d th row vector of Q D | Y andthe column vector
g p x q .
Then f p x q “ Q D | Y g p x q .
This system is a linear system with r ě k equations.
Recalling that Q D | Y has k linearly independentcolumns, we can select any k linearly independent rows of Q D | Y to solve the equation for the true,unique solution for the unknown vector g p x q .
Another way to describe this is with the pseudo-inverse:g p x q “ p Q D | Y q : f p x q .
Then we have r g p x qs y “ q p y | X “ x q for all y P Y .
Proof of Lemma 2.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
Let x P X , d P R with q p x, d q ą 0 , and y P Y .
Assume we know matrix Q Y | D and q p y 1 | X “ x q , @
y 1 P Y .
We can compute Q D | Y from Q Y | D via
Lemma 6.
[Equation]
Using Lemma 7, q p d | X “ x, Y “ y q “ q p d | Y “ y q .
We apply this property.
[Equation]
The denominator q p d | X “ x q is constant across all values of y , so we can write that q p y | X “ x, D “d q 9 q p d | Y “ y q q p y | X “ x q and normalize to find the probability:
[Equation]
We know q p d | Y “ y q as r Q D | Y s d,y , and every q p d | Y “ y 1 q , where y 1 P Y zt y u , as r Q D | Y s d,y 1 .
Wealso know q p y | X “ x q and every q p y 1 |
X “ x q where y 1 P Y zt y u , by the precondition assumptions.
Then we can compute q p y | X “ x, D “ d q .
Proof of Lemma 3.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
Recall f : R p Ñ R r is a vector-valued oracle function such that r f p x qs d “ q p d | X “ x q for all x Psupp Q p X q .
Also let us recall that Q D | Y is defined as an r ˆ k matrix whose elements r Q D
| Y s i,j “
Q p D “ i | Y “ j q , and each column is a conditional distribution over the domains given a label.
Ithas linearly independent columns by Lemma 6.
First recognize that for all d P R , x P X such that q p x q ą 0 ,
[Equation]
X “ x q , using the equality from Lemma 7.
Then we can write f p x q “ ř y 2 P Y q p y 2 | X “ x q Q D | Y r : , y 2 s , where Q D | Y r : , y 2 s is the y 2 th column of
Q D | Y .
Now we could also rewrite f p x q “ Q D | Y r Q p Y “ 1 | X “ x q ... Q p Y “ k | X “ x qs J .
We now prove two key components of the lemma.
Let y P Y .
Let x P A
y
such that q p x q ą 0 .
Points in same anchor sub-domain map together.
Let x
1 P
A y such that q p x 1 q ą 0 .
We now seekto show that f p x q “ f p x 1 q .
Recall that x, x 1 P A y .
By Lemma 5, q p y | X “ x q “ q p y | X “ x 1 q “ 1 .
Also by lemma 5, @
y 2 P Y zt y u , q p y 2 |
X “ x q “ q p y 2 | X “ x 1 q “ 0 .
Then for all y 2 P Y ,q p y 2 | X “ x q “ q p y 2 | X “ x 1 q .
Therefore, @
d P R ,
[Equation]
Then f p x q “ f p x 1 q .
Point outside of the anchor sub-domain does not map with points in the anchor sub-domain .
Let x 0 R
A
y
such that q p x 0 q ą 0 .
We now seek to show that f p x q ‰ f p x 0 q .
Because x 0 R A y withq p x 0 q ą 0 , and because A y contains all x such that q p x q ą 0 , q p y | X “ x q “ 1 , and q p y 1 | X “ x q “ 0for all y 1 P Y zt y u , then by Lemma 5, it must be that one of the following cases is true:
• Case 1: q p y | X “ x 0 q ‰ 1• Case 2: q p y 1 | X “ x 0 q ą 0 for some y 1 P Y zt y u .
In all circumstances, there exists some y 2 P Y : q p y 2 | x 0 q ‰ q p y 2 | x q .
Then,
[Equation]
Because Q D | Y has linearly independent columns (shown in Lemma 6), we now know that
[Equation]
So f p x q ‰ f p x 0 q .
C Proof of Theorem 2
Proof of Theorem 2.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
Recall f : X Ñ R r is a vector-valued oracle function such that r f p x qs d “ q p d | X “ x q for allx P supp Q p X q .
It is known because we know the marginal q p x, d q .
Let y P Y .
Then by Lemma 3, fsends every x P
A y (and no other x R A y ) to the same value.
We overload notation to denote thisas f p
A y q .
Then Q p f p X q
“ f p A y qq “ Q p X P A y q ě ϵ .
Then in the marginal distribution of f p X qwith respect to distribution Q , there is a distinct point mass on each f p A y q , with mass at least ϵ .
Because we know the marginal q p x, d q , we know the marginal q p x q , so we can obtain the distributionof f p X q with respect to distribution Q .
If we analyze the marginal distribution of f p X q with respectto distribution Q , and recover all point masses with mass at least ϵ , we can recover no more than
O p 1 { ϵ q such points.
We set m P Z ` so that the number of points we recovered is m ´ 1 .
We denote a mapping ψ : R r Ñ r m s .
This mapping sends each value of f p x q corresponding to apoint mass with mass at least ϵ to a unique index in t 1 , ..., m ´ 1 u .
It sends any other value in R p tom .
We note that the ordering of the point masses might have p m ´ 1 q ! permutations.
Notice that the point mass on each f p
A y q must be recovered among these m ´ 1 masses.
Recall thatfor all y P Y , f p x q “ f p
A y q if and only if x P
A y .
Then for all y P Y , ψ p f p x qq “ ψ p f p A y qq if andonly if x P A y , because ψ does not send any other value in R r besides f p A y q to ψ p f p
A y qq .
For convenience, we now define a mapping c : X Ñ r m s such that c “ ψ ˝ f .
We will also abusenotation here to denote c p
A y q “ ψ p f p
A y qq .
Then c p X q is a discrete, finite random variable thattakes values in r m s .
As c is a pushforward function on X , c p X q satisfies the label shift assumptionbecause X does (i.e., when conditioning on Y , the distribution of c p X q is domain-invariant).
We might now define a matrix Q c p X q| D in which each entry r Q c p X q| D
s i,d “ Q p c
p
X q “ i | D “ d q .
We recall that we know the number of true classes k .
Then we know that there is a (possibly unique)unknown decomposition of the following form:
[Equation]
“ d q , using the label shift property .
To express this decomposition in matrix form, we write Q c p X q| D “ Q c p X q| Y Q Y | D .
Now we make 734 observations about the unknown Q c p X q| Y .
For all y P Y ,
[Equation]
Then for each y P Y , the row of Q c p X q| Y with row index c
p
A y q is positive in the y th column, and zeroeverywhere else.
Restated, for each y P Y , there is some row with positive entry exactly in y th column.
This is precisely the anchor word assumption for a discrete, finite random variable.
We already know that Q Y | D is full row-rank, so because Q c p X q| Y satisfies the anchor word assumption, we canidentify Q Y | D up to permutation of rows by Theorem 1.
In other words, when we set the constraintthat the recovered Q c p X q| Y must have k columns and satisfy anchor word and the recovered Q Y | D 741 must have k rows and be full row-rank, any solution to the decomposition Q c p X q| D “ Q c p X q| Y Q Y |
Dmust identify the ground truth Q Y | D , up to permutation of its rows.
D Minimizing Cross-Entropy Loss yields Domain Discriminator
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
We here examinethe behavior of the cross-entropy loss, in the infinite data case (when we can work with expectationsover the entire distribution instead of empirical expectations over a finite set of datapoints).
Definethe vector-valued function z : R Ñ R r such that z p d q is a one-hot vector of length r , such thatr z p d qs i “ 1 , iff d “ i .
Then we write the cross-entropy loss with targets as true domains as
[Equation]
« ´
[Equation]
ÿ
[Equation]
ff
[Equation] [Equation]
X « ´ i “ r ÿ i “r z p D qs
[Equation]
f
[Equation]
q ff
L
[Equation]
“ E X r
´ i “ r ÿ i “
E
[Equation]
|
[Equation]
qs
[Equation]
log pr f p
X
qs i
qss
L CE “ E X «
´ i “ r ÿ i “log pr f p X
qs i q E D |
X rr z p
D qs
i s ff
L CE “ E X «
´ i “ r ÿ i “log pr f p X
qs
i qp 1 ˆ Q p D
“ i | X q
[Equation]
In order to find the minimizer of the cross entropy loss over the class of all functions from R p Ñr 0 , 1 s r , we formulate the following objective with the Lagrange constraint:
[Equation]
Setting partial derivative with respect to r f p
X qs r to 0 , we get ´
Q p D “ i | X q r f ‹ p
X qs i ` λ “ 0 and r f ‹ p
X qs i “1 λ Q p D “ i |
X q .
From KKT condition, the optimal solution lies on constraint surface, giving:i “ r ÿ i “r f ‹ p
X qs
i “i “ r ÿ i “ λQ p D “ i | X q “ λ i “ r ÿ i “
Q p D “ i | X q “ λ “λ “
Finally, we get r f ‹ p
X qs i “ Q p D “ i | X q , so the optimal f ˚ by the cross entropy loss as defined willin fact recover the oracle domain discriminator.
E Additional Experimental Details
Our code is available at https://github.com/latentlabelshift-anonymous/latentlabelshift .
Here we present the full generation procedure for semisynthetic exampleproblems, and discuss the parameters.
Choose a Dirichlet concentration parameter α ą 0 , maximum condition number κ ě 1(with respect to 2-norm), and domain count r ě k .2.
For each y P r k s , sample p d p y q „ Dir p αk 1 k q .3.
Populate the matrix Q Y | D with the computed p d p y q s. If cond p Q Y | D q ě k , return to step 2and re-sample.4.
Distribute examples across domains according to Q Y | D , for each of train, test, and validsets.
This procedure entails creating a quota number of examples for each (class, domain)pair, and drawing datapoints without replacement to fill each quota.
We must discard excessexamples from some classes in the dataset due to class imbalance in the Q Y | D matrix.
Dueto integral rounding, domains may be slightly imbalanced.
Conceal true class information and return p x i , d i q pairs.
It is important to note the role of κ and α in the above formulation.
Although they are unknownparameters to the classification algorithm, they affect the sparsity of the Q Y | D and difficulty of theproblem.
Small α encourages high sparsity in p d p y q , and large α causes p d p y q to tend towards auniform distribution.
We observe an example of the effects of α in Fig.
κ has a strong effect onthe difficulty of the problem.
Consider the case when k “ 2 .
When κ “ 1 , the only potential Q Y | Dmatrices are I 2 up to row permutation (which means that domains and classes are exactly correlated,so the domain indicates the class and the problem is supervised).
In the other limit, if κ Ñ `8 , wemay generate Q Y | D matrices that are singular, breaking needed assumptions for domain discriminatoroutput to uniquely identify true class of anchor subdomains.
κ also helps control the class imbalance(if a row of Q Y | D is small, indicating that the class is heavily under-represented across all domains,the condition number will increase).
E.1 FieldGuide-2 and FieldGuide-28 Datasets
The dataset and description is available at https://sites.google.com/view/fgvc6/competitions/butterflies-moths-2019 .
From this data we create two datasets FieldGuide-2and FieldGuide-28.
For FieldGuide-28 we select the 28 classes which have 1000 datapoints in thetraining file.
Since the test set provided in the website does not have annotations, we manually create a test set by sampling 200 datapoints from training file of each of the 28 classes.
Therefore, wefinally have 22400 training points and 5600 testing points.
The FieldGuide-2 dataset is created byconsidering two classes from the created FieldGuide-28 dataset.
83 0 .ȷ
(a) α : 0 . 5 , κ :„ 0 .
63 0 .ȷ
(b) α : 3 , κ :„ 0 . 42 0 .0 .
0 .ȷ
(c) α : 10 , κ :
E.2 Hyperparameters and Implementation Details: SCAN baseline
In all cases, we initialize the SCAN [57] network with the clustering head attached, sample dataaccording to the Q D | Y matrix, and predict classes.
With the Hungarian algorithm, implemented in [14, 58], we compute the highest true accuracy amongany permutation of these labels (denoted “Test acc”).
• CIFAR-10 and CIFAR-20 Datasets [38]We use ResNet-18 [30] backbone with weights trained by SCAN-loss and obtained from the
SCAN repo
https://github.com/wvangansbeke/Unsupervised-Classification .We use the same transforms present in the repo for test data.
• ImageNet-50 Dataset [17]
[Figure]
We use ResNet-50 backbone with weights trained by SCAN-loss and obtained from the
SCAN repo.
We use the same transforms present in the repo for test data.
• FieldGuide-2 and FieldGuide-28 Datasets
[Figure]
For each of the two datasets, we pretrain a different SCAN baseline network (includingpretext and SCAN-loss steps) on all available data from the dataset.
The backbone for eachis ResNet-18.
For training the pretext task, we use the same transform strategy used in the repo for CIFAR10 data (with mean and std values as computed on the Fieldguide-28 dataset, and crop size224).
For training SCAN, we resize the smallest image dimension to 256, perform a randomhorizontal flip and random crop to size 224.
We also normalize.
For validation we resizesmallest image dimension to 256, center crop to 224, and normalize.
E.3 Hyperparameters and Implementation Details: DDFA (RI)
This is the DDFA procedure with random initialization.
The bulk of this procedure is described in Section 6, but for completeness we reiterate here.
We train ResNet-50 [30] (with random initialization and added dropout) based on the implementationfrom https://github.com/kuangliu/pytorch-cifar on images x
i with domain indices d i asthe label, choose best iteration by valid loss, pass all training and validation data through p f , andcluster pushforward predictions p
f p x i q into m ě k clusters with Faiss K-Means [37].
We compute thep Q c p X q| D matrix and run NMF to obtain p Q c p X q| Y , p Q Y | D .
To make columns sum to 1, we normalizecolumns of p Q c p X q| Y , multiply each column’s normalization coefficient over the corresponding rowof p Q Y | D (to preserve correctness of the decomposition), and then normalize columns of p Q Y | D .
Some NMF algorithms only output solutions satisfying the anchor word property [3, 39, 27].
Wefound the strict requirement of an exact anchor word solution to lead to low noise tolerance.
Wetherefore use the Sklearn implementation of standard NMF [13, 55, 48].
We predict class labels with Algorithm 2 .
With the Hungarian algorithm, implemented in [14, 58],we compute the highest true accuracy among any permutation of these labels (denoted “Test acc”).
With the same permutation, we reorder rows of p Q Y | D , then compute the average absolute differencebetween corresponding entries of p Q Y | D and Q Y | D (denoted “ Q Y | D err”).
In order to make hyperparameter choices for final experiments, such as the choice of the NMF solver,clustering algorithm, and learning rate, we consulted CIFAR-10 and CINIC-10 (similar to an extensionof CIFAR-10)
[15] final test task accuracy.
We were also able to confirm our intuition that minimumvalidation loss on domain discriminator training corresponds to good final task performance.
Finalevaluation runs on CIFAR-10 were made with the best hyperparameters found here.
We acknowledgethat this may lead to test-set overfitting on CIFAR-10, but point out that on all other datasets, weconsulted only validation loss on domain discriminator training when adjusting the hyperparameters.
Final evaluation runs used the following fixed hyperparameters:
Common Hyperparameters
Architecture: ResNet-50 with added dropout
Faiss KMeans number of iterations (niter):
Faiss Kmeans number of clustering redos (nredo):
Learning Rate: 0.
Learning Rate Decay: Exponential, parameter 0.
SKlearn NMF initialization: random
Dataset-Specific Hyperparameters
• CIFAR-10 Dataset
Training Epochs:
Number of Clusters ( m ):
• CIFAR-20 Dataset
Training Epochs:
Number of Clusters ( m ): 60• ImageNet-50 DatasetNot evaluated.
• FieldGuide-2 Dataset
Training Epochs: 100Number of Clusters ( m ): 10• FieldGuide-28 Dataset
Not evaluated.
E.4 Hyperparameters and Implementation Details: DDFA (SI) and DDFA (SPI)
This is the DDFA procedure with SCAN initialization.
DDFA (SI) uses the SCAN pretext + SCANloss pretraining steps, while DDFA (SPI) uses only the SCAN pretext step.
The procedure is identical to the standard DDFA procedure, except that SCAN [57] pre-trainedweights or SCAN [57] contrastive pre-text weights are used to initialize the domain discriminatorbefore it is fine-tuned on the domain discrimination task.
Hyperparameters used also differ.
When SCAN pretrained weights are available, we use those.
When they are not, we train SCANourselves.
Much like SCAN (RI), we used CIFAR-10 final test acuracy to choose hyperparameters and makealgorithm decisions.
For all other datasets, we consulted only validation domain discrimination loss.
Final evaluation runs used the following fixed hyperparameters:
Common Hyperparameters
Faiss KMeans number of iterations (niter):
Faiss Kmeans number of clustering redos (nredo):
Learning Rate: 0.
Learning Rate Decay: Exponential, parameter 0.
SKlearn NMF initialization: random
Dataset-Specific Hyperparameters
• CIFAR-10 Dataset
Architecture: ResNet-18Pre-seed: Weights trained with SCAN pretext and SCAN-loss on entirety of CIFAR-10(from SCAN repo).Training Epochs:
Number of Clusters ( m ):
Transforms used: Same as SCAN repo.
• CIFAR-20 Dataset
Architecture:
ResNet-
Pre-seed: Weights trained with SCAN pretext and SCAN-loss on entirety of CIFAR-20(from SCAN repo).
Training Epochs:
Number of Clusters ( m ):
Transforms used: Same as SCAN repo.
• ImageNet-50 Dataset
Architecture:
ResNet-
Pre-seed: Weights trained with SCAN pretext and SCAN-loss on entirety of ImageNet-50(from SCAN repo).Training Epochs:
Number of Clusters ( m ):
Transforms used: Same as SCAN repo.
• FieldGuide-2 Dataset
Architecture:
ResNet-
Pre-seed: Weights trained with SCAN pretext on entirety of FieldGuide-2 (trained by us).Training Epochs:
Number of Clusters ( m )
Transforms used for pretext: Same strategy as CIFAR-10 in SCAN repo with appropriate
mean, std, and crop size 224.
Transform
used for SCAN:
Resize to 256, Random horizontal flip, Random crop to 224,
normalize
Learning rate used for SCAN: 0.001 (other hyperparameters were same as in SCAN repofor
CIFAR-
Note :
During one of the random seeds of training, test data transforms were mismatched withtrain transforms (specifically, missing the Resize(256) transform on test only).
We considerthis to disadvantage our approach for that random seed as compared to the SCAN baseline,which uses the proper transforms in all seeds.
We report these results regardless due to thefact that our approach still competes effectively even despite the transform disadvantage.
This random seed is the one displayed in Section 6 of the main paper.
The results shownin App.
F are different results, with the Resize(256) included (and are therefore the best,fair-footing evaluation comparison between our method and baseline).
• FieldGuide-28 DatasetArchitecture:
ResNet-18Pre-seed: Weights trained with SCAN pretext on entirety of FieldGuide-28 (trained by us).Training Epochs: 60Number of Clusters ( m ):
Transforms used for pretext: Same strategy as CIFAR-10 in SCAN repo with appropriatemean, std, and crop size 224.
Transform used for SCAN:
Resize to 256, Random horizontal flip, Random crop to 224,normalize
Learning rate used for SCAN: 0.01 (other hyperparameters were same as in SCAN repo for
CIFAR-10)
Note :
During one of the random seeds of training, test data transforms were mismatched withtrain transforms (specifically, missing the Resize(256) transform on test only).
We considerthis to disadvantage our approach for that random seed as compared to the SCAN baseline,which uses the proper transforms in all seeds.
We report these results regardless due to thefact that our approach still competes effectively even despite the transform disadvantage.
This random seed is the one displayed in Section 6 of the main paper.
The results shownin App.
F are different results, with the Resize(256) included (and are therefore the best,fair-footing evaluation comparison between our method and baseline).
F Additional Experimental Results
[Table]
DDFA (RI) 0.
DDFA (SI) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SI) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SI) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SI) 0.
[Table] [Table]
D err
Test acc
Q Y
| D err
Test acc
Q Y
| D err
SCAN 0.
DDFA (RI) 0.
DDFA (SPI) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SPI) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SPI) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SPI) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SPI) 0.
G Discussion of Convex Polytope Geometry
The geometric properties of topic modeling for finite, discrete random variables has been exploredin depth in related works ([33, 20, 12]).
The observation that columns in Q X | D are convex combinations of columns in Q X | Y leads to a perspective on identification of the matrix decomposition asidentification of the convex polytope in R m which encloses all of the columns of Q X | D (the cornersof which correspond to columns of Q X | Y under certain identifiability conditions).
Here, we briefly discuss an interesting but somewhat different application of convex polytopegeometry.
Instead of a convex polytope in R m with corners as columns of Q X | Y , we concernourselves with the convex polytope in R r with corners as columns in Q D | Y , which must enclose allvalues taken by the oracle domain discriminator f p x q for x P X , q p x q ą 0 .
Let us assume that Assumptions A.1–A.4 are satisfied.
We recall the oracle domain discriminator fdefined such that r f p x qs d “ q p d | X “ x q .
Let x
P
X
“ R p .
Now, since the r values q p d | X “ x q for
[Table]
[Table]
d P t 1 , 2 , ..., r u together constitute a categorical distribution, each of these r values lie between 0and 1 , and also their sum adds to 1.
Therefore the vector f p x q lies on the simplex ∆ r ´ 1 .
We nowexpress f p x q as a convex combination of the k columns of Q D | Y .
We denote these column vectors
Q D | Y r : , y s for each y P Y “ r k s each such
As an intermediate step in the proof of Lemma 3 given in App.
B, we showed that each f p x q is alinear combination of these columns of Q D | Y with coefficients q p y | X “ x q for all y P Y .
That is,we can rewrite f p x q “ Q D | Y r Q p Y “ 1 | X “ x q ... Q p Y “ k | X “ x qs J
Since the coefficients in the linear combination are probabilities which, taken together, form acategorical distribution, they lie between 0 and 1 and sum to 1.
Thus, for all x P X with q p x q ą 0 ,f p x q can be expressed as a convex combination of the columns of Q D | Y .
Therefore, for any x withq p x q ą 0 , f p x q lies inside the k ´ vertex convex polytope with corners as the columns of Q D | Y(which are linearly independent by Lemma 6).
This polytope is embedded in ∆ r ´ 1 .
Now consider x in an anchor sub-domain, that is x P
A y for some y P Y .
We know that if q p x q ą 0 ,q p y | X “ x q “ 1 , q p y 1 | X “ x q “ 0 for all y 1 ‰ y
(Lemma 5).
Since the q p y | X “ x q are now one- hot, we have that f p x q “ Q D | Y r : , y s for x P A y .
In words, this means that f p A y q is precisely the y thcolumn of Q D | Y .
It follows that the domain discriminator maps each of the k anchor sub-domainsexactly to a unique vertex of the polytope.
The situation is described in Fig.
We could now recover the columns of Q D | Y , up to permutation, with the following procedure:
Push all x P X through f .
Find the minimum volume convex polytope that contains the resulting density of pointson the simplex.
The vectors that compose the vertices of this polytope are the columns ofQ D | Y , up to permutation.
Note that from Assumption A.4, we are guaranteed to have a region of the input space with at leastϵ ą 0 mass that gets mapped to each of the vertices when carrying out step (i).
Therefore, ourdiscovered minimum volume polytope must enclose all of these vertices.
Since no mass will existoutside of the true polytope, requiring a minimum volume polytope will ensure that the recoveredpolytope fits the true polytope’s vertices precisely (as any extraneous volume outside of the truepolytope must be eliminated).
Then step (ii) recovers Q D | Y , up to permutation of columns.
Havingrecovered Q D | Y , we can use Lemmas 1 and 2 to recover q p y | x, d q .
This procedure is a geometric alternative to the clustering approach outlined in Algorithm 1 .
Inpractice, fitting a convex hull around the outputs of a noisy, non-oracle estimated domain discriminatormay be computationally expensive, and noise may lead this sensitive procedure to fail to recover thetrue vertices.
H Ablation Study on Number of Clusters
We conduct an ablation on the choice of m , the parameter indicating how many clusters to find in theq p
d | x q space.
We use the CIFAR-20 dataset with 20 domains and employ DDFA (SI) and SCANmodels, following the same hyperparameters as outlined in App.
E, except for modifying the choiceof m for DDFA (SI).
Results are obtained as the average of three random seeds.
[Figure]
Test accuracy vs of clusters m, on CIFAR20 with 20 domains, across problem settings (alpha
[Figure]
The number of true classes is 20 in CIFAR-20.
As seen in Fig.
5, when m is chosen to be 10, violatingthe typical constraint that m ě k , we can still solve for the solution, but we get poor performance,seeing a drop in accuracy as much as 20% from a better-chosen value of m .
Choosing m directlyequal to or slightly larger than k provide the best performance, with a slope-off in performance atvery large m .
The trend is roughly mirrored in Fig.
6, which shows how the reconstruction error modifies over thesame change in m .
Under all settings, using m “ 10 ă k clusters provides a poor reconstruction,
[Table]
while the best reconstruction is found with m roughly equal to k or slightly larger.
Performance
[Table]
degrades as m grows very large, although the effect is very slight for the alpha = 0.5 setting.
[Table]
Intuitively, these results show that breaking the m ě k condition not only violates the theoretical
[Table]
identifiability, but also leads to poorer empirical performance; choosing very large m can also lead to
[Table]
degraded performance, likely due to propagation of inaccuracies in the finite-sample estimation ofthe
p Q c p X q| D matrix.
[Table]
[Figure]
Q Y
D err alpha
[Figure]
I Ablation Study with a Naïve Feature Space
One might ask whether the semantic meaning of the domain discrimination space is necessary in
DDFA; might we exchange the domain discrimination step in Algorithm 1 for a naive step in whichwe simply pass the input through an arbitrary feature extractor and then proceed to clustering in thisspace?
The first remark we make is that the domain discriminator does not purely provide a clusteringrepresentation; its semantic meaning is also important for the computation of the final domainadjusted p d p y | x q prediction, as a reliable estimate of q p d | x q , in conjunction with the estimate of the
Q Y | D matrix, allow us to estimate p d p y | x q via Algorithm 2.
Without this semantic meaning, ourclass prediction can be based only on a coarse prediction at the level of the cluster , not the individualdatapoint.
However, if we are still determined to use an alternate representation, it is indeed possible to do so.
We illustrate a variant of DDFA using a naïve representation in Algorithms 3 and 4, and then evaluatethis procedure on CIFAR-20 as an ablative study on DDFA.
We compare naïve results with standard
DDFA results, and with a traditional SCAN baseline, in Table 6.
Naïve Representation Variant of DDFA
The only major changes from the original DDFA for
Algorithm 3 are the removal of the need to train any p f domain discriminator, the use of the arbitraryrepresentation space ϕ before clustering, and the output of the clustering discretization function cas well as p Q c p X q| Y (which are both discarded in the original procedure).
We need c and p Q c p X q| Ybecause in Algorithm 4 we will use them for domain-adjusted class prediction.
Algorithm 4 includes significant changes from the DDFA procedure.
Since we do not have theestimate of q p d | x q to use, we cannot directly reason about how different locations in the representationspace induced by ϕ correspond to different probabilities of class labels.
However, because we havep Q c p X q| Y and p Q Y | D , two outputs of the NMF decomposition in Algorithm 3, we can calculate acoarse prediction over labels y for each cluster, and then assign the same prediction to each point in input k ě r ě k x i d i
Pr n q p c q “ a | D “ b q Ð ř i Pr n s I r c p x i q“ a, d i “ b s ř j Pr n s I r d j “ b s 5:
Populate p Q c p X q| D as r p Q c p X q| D
s a,b Ð p q p c
p
X
q “ a | D “ b q 6: p Q c p X q| Y p Q Y | D
Ð NMF p p Q c p X q| D
q output p Q c p X q| Y , p Q Y | D , clustering discretization function c input
4: p q p y | c p q “ x
Ð p q p c p x 1 q| Y “ y q p q p y | D “ d 1 q ř y 2 P Y p
q p c p x 1 q| Y “ y 2 q p q p y 2 | D “ d 1 q 5: y pred Ð arg max
y Pr k s p q p y | c p X q “ c p x 1 q , D “ d 1 q
[Equation]
that cluster.
To obtain the closed-form for this coarse prediction
p p
d p y | c p x qq used in Algorithm 4, weuse the following derivation:
[Equation]
By label shift, q p c p x q| y, d q “ q p c p x q| y q , then “
[Equation]
Combining Algorithms 3 and 4 allows us to empirically evaluate the behavior of an ablation on DDFAwhich does not use any domain discriminator.
For a reasonable comparison, we need a meaningfulnaïve representation space.
We use a SCAN pretrain backbone for ResNet-18, and remove the lastlinear layer in the ResNet-18 backbone in order to expose a 512-dimension representation space.
Since clustering in high-dimensional spaces often performs poorly, we also map this 512-dimensionrepresentation down to only r (the number of domains) dimensions using two different commondimensionality reduction methods: Independent Component Analysis (ICA) [35] and Principal
Component Analysis (PCA) [59].
These smaller-dimension clustering problems provide a closercomparison to the dimensionality of the clustering problem in the DDFA (SI) procedure, for whichwe employ m clusters.
Note: we use ICA and PCA implementations from scikit-learn [48].
SCAN, DDFA (RI), and DDFA (SI) experiment details are the same as explained in App.
E; in factthese are the same results presented for CIFAR-20 in Sec. 6.
In general, we can see that all variants of the Naïve/ablated DDFA procedure perform worse than DDFA (SI) in all problem settings, over both metrics of interest.
All variants of the Naïve procedureare also worse than DDFA (RI) in both metrics of interest when α is 0.5 or 3, although they matchor outpace it slightly in some of the hardest settings ( α “ 10 ).
It is worth pointing out that this ispredominantly due to the fact that DDFA (RI) performs notably poorly at harder settings.
The SCAN baseline outpaces these Naïve approaches, in terms of classification accuracy, in allproblem settings.
However, the Naïve approaches always achieve better Q Y | D reconstruction error.
[Table]
SCAN 0.
DDFA (RI) 0.
DDFA (SI) 0.
Naïve 0.
Naïve (ICA) 0.
Naïve (PCA) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SI) 0.
Naïve 0.
Naïve (ICA) 0.
Naïve (PCA) 0.
SCAN 0.
DDFA (RI) 0.
DDFA (SI) 0.
Naïve 0.
[Table]