Unsupervised Learning under Latent Label Shift
Abstract
What sorts of structure might enable a learner to discover classes from unlabeled data?
Traditional unsupervised learning approaches risk recovering incorrect classes based on spurious dataspace similarity.
In this paper, we introduce unsupervised learning under Latent Label Shift (LLS), where the label marginals p d p y q shift but the class conditionals p p x | y q do not.
This setting suggests a new principle for identifying classes: elements that shift together across domains belong to the same true class.
For finite input spaces, we establish an isomorphism between LLS and topic modeling; for continuous data, we show that if each label’s support contains a separable region, analogous to an anchor word, oracle access to p p d | x q suffices to identify p d p y q and p d p y | x q up to permutation.
Thus motivated, we introduce a practical algorithm that leverages domaindiscriminative models as follows: (i) push examples through domain discriminator p p
d | x q ; (ii) discretize the data by clustering examples in p p d | x q space; (iii) perform non-negative matrix factorization on the discrete data; (iv) combine recovered p p y | d q with discriminator outputs p p
d
| x q to compute p d p y | x q @ d .
In semi-synthetic experiments, we show that our algorithm can use domain information to overcome a failure mode of standard unsupervised classification in which data-space similarity does not indicate true groupings.
1. Introduction
Discovering systems of categories from unlabeled data is a fundamental but ill-posed challenge in machine learning.
Typical unsupervised learning methods group instances together based on feature-space similarity.
Accordingly, given
Published at the ICML
Workshop on Spurious Correlations, Invariance, and Stability .
Baltimore, Maryland, USA.
Copyright 2022 by the author(s).
a collection of photographs of animals, a practitioner might hope that, in some feature space, images of animals of the same species should be somehow similar to each other.
But why should we expect a clustering algorithm to recognize that dogs viewed in sunlight and dogs viewed at night belong to the same category?
Why should we expect that butterflies and caterpillars lie close together in feature space?
In this paper, we offer an alternative principle according to which we might identify a set of classes: we exploit distribution shift across times and locations to reveal otherwise unrecognizable groupings among examples.
For example, if we noticed that whenever we found ourselves in a location where butterflies are abundant, caterpillars were similarly abundant, and that whenever butterflies were scarce, caterpillars had a similar drop in prevalence, we might conclude that the two were tied to the same underlying concept, no matter how different they appear in feature space.
In short, our principle suggests that latent classes might be uncovered whenever instances that shift together group together .
Formalizing this intuition, we introduce the problem of unsupervised learning under Latent Label Shift (LLS).
Here, we assume access to a collection of domains d P t 1 , . . .
, r u , where the mixture proportions p d p y q vary across domains but the class conditional distribution p p x | y q is domaininvariant.
Our goals are to recover the underlying classes up to permutation, and thus to identify both the per-domain mixture proportions p d p y q and optimally adapted per-domain classifiers p d p y | x q .
We prove that under mild assumptions, knowledge of this underlying structure is sufficient for inducing the full set of categories.
First, we focus on the tabular setting , demonstrating that when the input space is discrete and finite, LLS is isomorphic to topic modeling (Blei et al., 2003).
In this case, we can apply standard identification results for topic modeling (Donoho & Stodden, 2003; Arora et al., 2012b; Gillis &
Vavasis, 2014; Huang et al., 2016; Chen et al., 2021) that rely only on the existence of anchor words within each topic (i.e., for each label y i there is at least one x in the support of y i , that is not in the support of any y j ‰ y i ).
Here, standard methods based on Non-negative Matrix Factorization (NMF) can recover each domain’s underlying mixture proportion p d p y q and optimal predictor p d p y | x q .
However, the restriction to discrete inputs, while appropriate for topic modeling, proves restrictive when our interests extend to high-dimensional continuous input spaces.
Then, to handle high-dimensional inputs, we propose
Discriminate-Discretize-Factorize-Adjust (DDFA) , a general framework that proceeds in the following steps: (i) pool data from all domains to produce a mixture distribution q p x, d q ; (ii) train a domain discriminative model f to predict q p d | x q ; (iii) push all data through f , cluster examples in the pushforward distribution, and tabularize the data based on cluster membership; (iv) solve the resulting discrete topic modeling problem (e.g., via NMF), estimating q p y, d q up to permutation of the latent labels; (v) combine the predicted q p d | x q and q p y, d q to estimate p d p y q and p d p y | x q .
In developing this approach, we draw inspiration from recent works on distribution shift and learning from positive and unlabeled data that (i) leverage black box predictors to perform dimensionality reduction (Lipton et al., 2018; Garg et al., 2020; 2021); and (ii) work with anchor sets , separable subsets of continuous input spaces that belong to only one class’s support (Scott, 2015; Liu & Tao, 2016; du Plessis et al., 2016)
Our key theoretical result shows that domain discrimination ( q p d | x q ) provides a sufficient representation for identifying parameters of interest.
Given oracle access to q p d | x q (which is identified without labels), our procedure is consistent.
Our analysis reveals that the true q p d | x q maps all points in the same anchor set to a single point mass in the push-forward distribution.
This motivates our approach of discretizing data by hunting for tight clusters in p q p d | x q space.
In semi-synthetic experiments, we adapt existing image classification benchmarks to the LLS setting.
We note that training a domain discriminative classifier is a difficult task, and find that warm starting the initial layers of our model with pretrained weights from unsupervised approaches can significantly boost performance.
We show that warm-started
DDFA outperforms state-of-the-art (SOTA) unsupervised approaches when domain marginals p d p y q are sufficiently sparse.
In particular, we observe improvements of as much as 30% accuracy over unsupervised SOTA on CIFAR-20.
Further, on subsets of FieldGuide dataset, where similarity between species and diversity within a species leads to failure of unsupervised learning, we show that DDFA recovers the true distinctions.
To be clear, these are not apples-toapples comparisons: our methods are specifically tailored to the LLS setting.
The takeaway is that the structure of the
LLS setting can be exploited to outperform the best unsupervised learning heuristics, in particular when these heuristics detect classes by exploiting spurious similarities.
2. Problem Formulation
For a vector v P R p , we use v j to denote its j th entry.
For an event E , we let I r E s denote the binary indicator of the event.
We use r A
s i,j to access the element at p i, j q in A .
Let X be the input space and Y “ t 1 , 2 , . . .
, k u the output space for classification.
We assume throughout this work that the number of true classes k is known.
We use capital letters (e.g., X ) to denote random variables and small case letters (e.g., x ) to denote the values they take.
We now introduce the problem of unsupervised learning under LLS.
We assume that we observe unlabeled data from R “ t 1 , 2 , . . .
, r u domains.
By p d , we denote the probability density (or mass) function for each domain d P R .
Definition 2.1 (Latent label shift) .
We observe data from r domains.
While the label distribution among these domains can change, for all d, d 1 P R and for all p x, y q P X ˆ Y , we have p d p x | y [Equation]
Simply put, Definition 2.1 states that the conditional distribution p d p x | y q remains invariant across domains, i.e., they satisfy the label shift assumption.
Crucially, under LLS, p d p y q can vary across domains, and we observe unlabeled data with domain label tp x 1 , d 1 q , p x 2 , d 2 q , . . .
, p x n , d n qu .
We now aim to (i) estimate the label marginal in each domain
p d p y q ; and (ii) estimate the optimal per-domain predictor p d p y | x q , upto some permutation of labels.
Mixing distribution Q A key step in our algorithm will be to train a domain discriminative model.
Towards this end we define Q , a distribution over X ˆ Y ˆ R , constructed by taking a uniform mixture over all domains.
By q , we denote the probability density (or mass) function of Q .
Define Q such that q p x, y | D “ d q “ p d p x, y q .
For all d P R , we define γ d “ q p d q , i.e., the prevalence of each domain in our distribution Q .
Notice that q p x, y q is a mixture over the distributions t p d p x, y qu d P R , with mixture weights t γ d u d P R .
Notation for the discrete case To begin, we setup notation for discrete input spaces.
Without loss of generality, we assume that X “ t 1 , 2 , . . .
, m u .
The label shift assumption allows us to formulate the label marginal estimation problem in matrix form.
Let Q X | D be an m ˆ r matrix such that r Q X | D s i,d “ p d p X “ i q , i.e., the d -th column of
Q X | D is p d p x q .
Let Q X | Y be an m ˆ k matrix such that r Q X | Y s i,j “ p p X
“ i | Y “ j q , the j -th column is a distribution over X given Y “ j .
Similarly, define Q Y | D as a k ˆ r matrix whose d -th column is the marginal p d p y q .
With Definition 2.1, we have p d p x q “ ř y p
d p x, y q “ ř y p
d p x | y q p d p y q “ ř y p p x | y q p d p
y q
Since this is true @
d P R , we have the matrix form as Q X | D “ Q X | Y Q Y | D .
Assumptions We introduce four additional assumptions:
A.1 As many domains as classes,
i
e., |R| ě |Y| .
A.2 Matrix formed by label marginals (as columns) across domains is full-rank, i.e., rank p Q Y | D q “ k . A.3 Equal representation of domains, i.e., γ d “ 1 { r .
A.
For all y P Y , there exists a unique subdomain A y Ď X , such that q p A y q ě ϵ and x P A y if and only if
the following conditions are satisfied: q p x | y q ą 0 and q p x | y 1 q “ 0 for all y 1 P Y zt y u .
This is the ϵ -anchor sub-domain condition.
A.1–A.2 are benign, these assumptions just imply that the matrix Q Y | D is full row rank.
Without loss of generality,
A.3 can be assumed when dealing with data from a collection of domains.
Intuitively, A.4 states that for each label y P
Y , we have some subset of inputs that only belong to that class y .
The anchor word condition is related to the positive sub-domain in PU learning, which requires that there exists a subset of X in which all examples only belong to the positive class (Scott, 2015; Liu & Tao, 2016;
du Plessis et al., 2016).
3. Theoretical Analysis
In this section, we establish identifiability of LLS problem.
We begin by considering the case where the input space is discrete and formalize the isomorphism to topic modeling.
Then we establish the identifiability of the system in this discrete setting by appealing to existing results in topic modeling (Huang et al., 2016).
Finally, extending results from discrete case, we provide novel analysis to establish our identifiability result for the continuous setting.
Isomorphism to topic modeling
Recall that for the discrete input setting, we have the matrix formulation:
Q X | D “ Q X | Y Q Y | D .
Consider a corpus of r documents, consisting of terms from a vocabulary of size m .
Let D be an R m ˆ r matrix such that r D s i,j represents the frequency of term i in document j .
Topic modeling (Hofmann, 1999;
Blei et al., 2003) considers each document to be composed as a mixture of k topics.
Given a topic, each term has a probability of occurring that is document-invariant, but the proportion of topics themselves vary across documents.
This can be expressed as: D “ CW , where C is an R m ˆ k matrix, r C s i,j is the probability of term i given topic j , and
W is an R k ˆ r matrix, where r W s i,j is the proportion of topic i in document j .
The isomorphism is then: document ” domain, topic ” label, term ” input-sample, and
D “ CW ” Q X | D “ Q X | Y Q Y | D .
We leverage this isomorphism to extend identifiability conditions of the topic modeling problem (Donoho & Stodden, 2003; Chen et al., 2021) to our LLS-setting.
We formalize the adaption here:
Theorem 3.1.
(adapted from Proposition 1 in Huang et al.
Assume A.1, A.2 and A.4 hold (A.4 in the discrete setting is referred to as the anchor word condition).
Then the solution to Q X | D “ Q X | Y Q Y | D is uniquely identified.
We refer readers to Huang et al.
for a proof of this theorem.
Theorem 3.1 states that if each label y has at least one token in the input space that has support only in y , and
A.1, A.2 hold, then the solution to Q X | Y , Q Y | D is unique.
Furthermore, under this condition, there exist algorithms that can recover Q X | Y , Q Y | D within some permutation (Arora et al., 2012a;b; Gillis & Vavasis, 2014).
Extensions to the continuous case We will prove identifiability in the continuous setting, when X “ R p for some p ě 1 .
In addition to A.1–A.4, we make the assumption that we have oracle access to q p d | x q , i.e., the true domain discriminator for mixture distribution Q .
This is implied by assuming access to the marginal q p x, d q from which we observe samples.
We formalize this extension in the following theorem:
Theorem 3.2.
Let the distribution Q over X, Y, D satisfy
Assumptions A.1–A.4.
Assuming access to the joint distribution q p x, d q , and the true number of classes k , we show that the following quantities are identifiable: (i) Q Y | D , (ii) q p y | X “ x q , for all x P X that lies in the support (i.e. q p x q ą 0 ); and (iii) q p y | X “ x, D “ d q , for all x P X and d P R such that q p x, d q ą 0 .
We present a proof sketch for Theorem 3.2 in App.
B, and a full proof in App.
D.
The core idea is to show that with access to an oracle domain discriminator f p x q “ q p d | x q , we can construct a map from a continuous space which satisfies assumption A.4, A.1 to a discrete space which satisfies anchor word condition.
We also provide a geometric perspective on identifiability in the continuous setting in App. H.
4. DDFA Framework
Motivated by our identifiability analysis, in this section, we present an algorithm to estimate Q Y | D , q p y | x q , and q p y | x, d q when X is continuous by exploiting domain structure and approximating the true domain discriminator f .
Intuitively, q p y | x, d q is the domain specific classifier p d
p y | x q and q p y | x q is the classifier for data from aggregated domains.
Q Y | D captures label marginals for individual domains.
A naive approach would be to aggregate data from different domains and exploit recent advancements in unsupervised learning (Van Gansbeke et al., 2020; Park et al., 2020; Caron et al., 2018; 2019).
However, aggregating data from multiple domains loses the domain structure that we hope to leverage.
We highlight this failure mode of the unsupervised clustering method in Sec. 5.
Discriminate We begin Algorithm 1 by creating a split of the unlabeled samples into the training and validation sets.
Using the unlabeled data samples and the domain that each sample originated from, we first train a domain discriminative classifier p f .
The domain discriminative classifier outputs a distribution over domains for a given input.
This classifier is trained with cross-entropy loss to predict the domain label of each sample on the training set.
With unlimited data, the minimizer of this loss is the true f , as we prove in App.
E.
To avoid overfitting, we stop training p f
when the cross-entropy loss on the validation set stops decreasing.
Note that here the validation set also only contains domain labels (and no information about true labels).
input r ě k ě 1 , tp x i , d i qu i Pr n s „ q p x, d q ,
A of
4: Cluster
5: c p x i q Ð Cluster id of p f p x i q 6: p q p c p X q “ a | D “ b q Ð ř i Pr n s I r c p x i q“ a, d i “ b s ř j Pr n s I r d j “ b s 7:
For all a, b, r p Q c p X q| D
s a,b Ð p q p c
p
X
q “ a | D “ b q 8: p Q c p X q| Y , p Q Y | D
Ð NMF p p Q c p X q| D
q output p Q Y | D , p f input p Q Y | D , p f, p x 1 , d 1 q „ q p x, d q 1: Populate p Q D | Y as r p Q D | Y s d,y Ð r p Q Y | D s y,d ř d 2“ r d 2“ 1 r p Q Y | D s y,d2:
Assign p q p y | x 1 q Ð ” p p Q D | Y q : p
f p x 1 q ı y 3:
Assign p q p y | x 1 , d 1 q Ð r p Q D | Y s d 1 ,y p q p y | x 1 q ř y
Pr k s r p Q D
| Y s d 1 ,y 2 p q p y
| x 1 q 4: y pred Ð arg max y Pr k s p q p y | x 1 , d 1 q output : p q p y | x 1 , d 1 q “ p p d 1 p y | x 1 q , p q p y | x 1 q , y pred Discretize We now push forward all the samples from the training and validation sets through the domain discriminator to get vector p f p x i q for each sample x i . In the proof of Theorem 3.2, we argue that when working with true f , and the entire marginal q p x q , we can choose a discretization satisfying the anchor word assumption by identifying point masses in the distribution of f p x q and filtering to include those of at least ϵ size.
In the practical setting, because we have only a finite set of data points and a noisy p f , we use clustering to approximately find point masses.
We choose m ě k and recover m clusters with any standard clustering procedure (e.g. K-means).
This clustering procedure is effectively a useful, but imperfect heuristic: if the noise in p f is sufficiently small and the clustering sufficiently granular, we hope that our m discovered clusters will include k pure clusters, each of which only contains data points from a different anchor subdomain which are tightly packed around the true f p A y q for the corresponding label y .
Clustering in this space is superior to a naive clustering on the input space because close proximity in this space indicates similarity in q p d | x q .
Let us denote the learned clustering function as c , where c p x q is the cluster assigned to a datapoint x .
We now leverage the cluster id c p x i q of each sample x i to discretize sample into a finite discrete space r m s .
Combining cluster id with the domain source d i for each sample, we estimate p Q c p X q| D by simply computing, for each domain, the fraction of its samples assigned to each cluster.
Factorize
We apply an NMF algorithm to p Q c p X q| D to obtain our estimates of p Q c p X q| Y and p Q Y | D .
Adjust We begin Algorithm 2 by considering a test point p x 1 , d 1 q .
To make a prediction, if we had access to oracle f and true Q Y | D , we could precisely compute q p y | x 1 q (Lemma B.1).
However, in place of these true quantities, we plug in the estimates p f and p Q Y | D .
Since our estimates contain noise, the estimate p q p y | x 1 q is found by left-multiplying p f p x 1 q with the pseudo-inverse of p Q D | Y , as opposed to solving a sufficient system of equations.
As our estimates p f and p Q D | Y approach the true values, the projection of p f p x 1 q into the column space of p Q D | Y tends to p f p x 1 q itself, so the pseudo-inverse approaches the true solution.
Now we can use the constructive procedure introduced in the proof of
Lemma B.2 to compute the plug-in estimate p q p y | x 1 , d 1 q “ p p d 1 p y | x 1 q .
5. Experiments
[Table]
Our approach is well-suited to problems in which (i) we have data from several data domains (e.g. wildlife images collected from different regions, or medical patient profiles collected in a variety of different timeframes), (ii) we expect that the class-conditional data distribution is domaininvariant, and (iii) we expect that for each true class, there are some datapoints that could only belong to that class (satisfying the anchor subdomain property).
We simulate such a problem in this section, and reveal that in this case our approach can help avoid failure modes of traditional unsu- pervised classification and recover true class boundaries
. Datasets
To explore a failure mode of unsupervised classification, we use FieldGuide ( https://sites.
google.com/view/fgvc6/competitions/ butterflies-moths-2019 ), which contains images of moths and butterflies.
Each species is a class, and each class contains images from youth (caterpillar) and adult stages of life.
We intuit that butterflies from a given species look more like butterflies from other species than caterpillars from their own species, and expect that unsupervised methods will learn incorrect class boundaries which distinguish caterpillars from butterflies, instead of true species boundaries.
We assume each class has an anchor sub-domain, i.e., some images can only belong to that species.
We build
FieldGuide-2 and FieldGuide-28 subsets, with two andspecies, respectively.
Results from FieldGuide-2 and benchmarks CIFAR-10, CIFAR-20 (Krizhevsky & Hinton, 2009), and ImageNet-50 (Deng et al., 2009) are available in App.
G.
LLS Setup Full semi-synthetic experiment setup is found in App.
F. We sample p d p y q from a symmetric Dirichlet distribution with concentration α { k and enforce max condition number κ on Q Y | D .
Small α and small κ encourage sparsity in Q Y | D , so each label tends to only appear in a few domains.
Larger α encourages each p d p y q to tend toward uniform.
We draw from test, train, and valid datasets without replacement to match these distributions.
Baseline
Unsupervised classification method SCAN (Van Gansbeke et al., 2020) is trained on train and valid splits of each dataset, then evaluated on sampled test subsets.
Training and Evaluation
The domain discriminator is ResNet-18, pretrained with SCAN contrastive pre-text weights.
We train this network on images x i to predict domain indices d i , choose best iteration by valid loss, pass train and valid data through p f , and cluster pushforward predictions p
f p x i q .
We compute the p Q c p X q| D matrix and run
NMF to obtain p Q c p X q| Y , p Q Y | D .
We predict class labels with Algorithm 2 , then compute the highest true accuracy among any permutation of these labels (denoted “Test acc”).
Permuting rows of p Q Y | D to match this label permutation, we compute the average absolute difference between corresponding entries of p Q Y | D and Q Y | D (denoted “ Q Y | D err”).
See App.
F for full details, as well as setup for other base domain discriminator models and initialization strategies.
Results On FieldGuide-28 (Table 1), DDFA outperforms
SCAN when Q Y | D is sufficiently sparse (sampled with α : 0 .
5 or α : 3 ), with the highest observed accuracy difference ranging above 30-40%.
We do not claim that SCAN is too weak to find image groupings on this data; instead we acknowledge that the domain information available to
DDFA (and not to SCAN) is helpful for finding the true class distinctions between species as opposed to spurious distinctions between adult and immature life stages.
Results from all experiments are available in App.
G.
6. Conclusion and Future Work
Our theoretical results demonstrate that under LLS, we can leverage shifts among previously seen domains to recover correct class distinctions in a purely unsupervised manner.
We believe that this work is just the first step in a new direction for leveraging structural assumptions together with distribution shift to perform unsupervised learning.
Several components of our DDFA framework warrant further investigation: (i) the deep domain discriminator can be enhanced in myriad ways; (ii) for clustering discriminator outputs, we might develop methods specially tailored to our setting to replace the current generic clustering heuristic; (iii) clustering might be replaced altogether with geometrically informed methods that directly identify the corners of the polytope; (iv) the theory of LLS can be extended beyond identification to provide statistical results that might hold when q p d | x q can only be noisily estimated, and when only finite samples are available for the induced topic modeling problem; (v) when the number of true classes k is unknown, we may develop approaches to estimate this k .
A. Related Work
Unsupervised Learning Standard unsupervised learning approaches for discovering labels often rely on similarity in the original data space (MacQueen et al., 1967; Reynolds, 2009).
While distances in feature space become meaningless for highdimensional data, deep learning researchers have turned to similarity in a representations space learned via self-supervised contrastive tasks (Doersch et al., 2015; Noroozi & Favaro, 2016; Gidaris et al., 2018; Chen et al., 2020), or similarity in a feature space learned end-to-end for a clustering task (Caron et al., 2018; 2019; Park et al., 2020; Van Gansbeke et al., 2020).
Our problem setup closely resembles independent component analysis (ICA), where one seeks to identify statistically independent signal components from mixtures (Hyv¨arinen & Oja, 2000).
However, ICA’s assumption of statistical independence among the components does not obtain in our setup.
In topic modeling (Papadimitriou et al., 2000;
Blei et al., 2003; Arora et al., 2012b; Huang et al., 2016; Chen et al., 2021), documents are modeled as mixtures of topics, and topics as categorical distributions over a finite vocabulary.
Early topic modeling approaches include the well-known
Latent Dirichlet Allocation (LDA) which assumes that topic mixing coefficients are drawn from a Dirichlet distribution (Blei et al., 2003), as well as papers with relaxed assumptions on the distribution of topic mixing coefficients (pLSI) (Hofmann, 1999; Papadimitriou et al., 2000).
The topic modeling literature often draws on non-negative Matrix Factorization (NMF) methods (Paatero & Tapper, 1994; Seung & Lee, 2001), which decompose a given matrix into a product of two matrices with non-negative elements (Girolami & Kab´an, 2003; Gaussier & Goutte, 2005; Ding et al., 2008; de Paulo Faleiros & de Andrade Lopes, 2016).
In both Topic Modeling and NMF, a fundamental problem has been to characterize the precise conditions under which the system is uniquely identifiable (Donoho & Stodden, 2003; Arora et al., 2012b; Huang et al., 2016; Chen et al., 2021).
The anchor condition (also referred to as separability) is known to be instrumental for identifying topic models (Donoho & Stodden, 2003; Arora et al., 2012b; Huang et al., 2016; Chen et al., 2021).
In this work, we extend these ideas, leveraging separable subsets of each label’s support (the anchor sets) to produce anchor words in the discretized problem.
Existing methods have attempted to extend latent variable modeling to continuous input domains by making assumptions about the functional forms of the class-conditional densities, e.g., restricting to Gaussian mixtures (Reynolds, 2009; Prabhudesai et al., 2018).
A second line of approach involves finding an appropriate discretization of the continuous space (Tian, 2018).
Distribution Shift under the Label Shift Assumption The label shift assumption, where p d p y q can vary but p p x | y q cannot, has been extensively studied in the domain adaptation literature (Saerens et al., 2002; Storkey, 2009; Zhang et al., 2013; Lipton et al., 2018; Garg et al., 2020) and also obtains in the problem of learning from positive and unlabeled data (Elkan & Noto, 2008; Bekker & Davis, 2020; Garg et al., 2021).
For both problems, many classical approaches suffer from the curse of dimensionality, failing in the settings where deep learning prevails.
Our solution strategy draws inspiration from recent work on label shift (Lipton et al., 2018;
Alexandari et al., 2019;
Azizzadenesheli et al., 2019; Garg et al., 2020) and
PU learning (Scott, 2015; Liu & Tao, 2016; Bekker & Davis, 2020; Garg et al., 2021) that leverage black-box predictors to produce sufficient low-dimensional representations for identifying target distributions of interest (other works leverage black box predictors heuristically (Ivanov, 2019)).
Key differences: While PU learning requires identifying one new class for which we lack labeled examples provided that the positive class contains an anchor set (Garg et al., 2021), LLS can identify an arbitrary number of classes (up to permutation) from completely unlabeled data, provided a sufficient number of domains.
Domain Generalization
The related problem of Domain Generalization (DG) also addresses learning with data drawn from multiple distributions and where the domain identifiers play a key role (Muandet et al., 2013; Arjovsky et al., 2019).
However in DG, we are given labeled data from multiple domains, and our goal is to learn a classifier that can generalize to new domains.
By contrast, in LLS, we work with unlabeled data only, leveraging the problem structure to identify the underlying labels.
B. Proof Sketch
In this section we present a proof sketch for Theorem 3.2.
We begin by first presenting key lemmas (we include their proofs in App. C).
The full proof of Theorem 3.2 is in App.
D.
Lemma B.1.
Under the same assumptions as Theorem 3.2, the matrix Q Y | D and f p x q “ q p d | x q uniquely determine q p y | x q for all y P Y and x P X such that q p x q ą 0 .
Lemma B.1 states that given matrix Q Y | D and oracle domain discriminator, we can uniquely identify q p y | x q .
In particular, we show that for any x P X , q p d | x q can be expressed as a convex combination of the k columns of Q D | Y (which is computed from Q Y | D and is column rank k ) and the coefficients of the combination are q p y | x q .
Combining this with the linear independence of the columns of Q D | Y , we show that these coefficients are unique.
In the following lemma, we show how the identified q p y | x q can then be used to identify q p y | x, d q :
Lemma B.2.
Under the same assumptions as Theorem 3.2, for all y P Y and x P X such that q p x, d q ą 0 , the matrix
Q Y | D and q p y | x q uniquely determine q p y | x, d q .
To prove Lemma B.2, we show that we can combine the conditional distribution over the labels given a sample x P X with the prior distribution of the labels in each domain to determine the posterior distribution over labels given the sample x and the domain of interest.
Next, we introduce a key property of the domain discriminator classifier f :
Lemma B.3.
Under the same assumptions as Theorem 3.2, for all x, x 1 in the same anchor sub-domain, i.e., x, x 1 P A y for a given label y P Y , we have f p x q “ f p x 1 q .
Further, for any y P Y , if x P A y , x 1 R A y , then f p x q ‰ f p x 1 q .
Lemma B.3 implies that the oracle domain discriminator f maps all points in an anchor subdomain, and only those points in that anchor subdomain to the same point in f p x q “ q p d | x q space.
We can now present a proof sketch for Theorem 3.2 (full proof in App.
D):
Proof sketch of Theorem 3.2.
The key idea of the proof lies in proposing a discretization such that some subset of anchor subdomains for each label y in the continuous space map to distinct anchor words in discrete space.
In particular, if there exists a discretization of the continuous space X that for any y P Y , maps all
x P A y to the same point in the discrete space, but no x R A y maps to this point, then this point serves as an anchor word.
From Lemma B.3, we know that for each y P Y , there is a unique point in f p x q space to which all x P
A y are mapped and to which no x 1 R A y is mapped.
Pushing all the x P X through f , we know from A.4 that there exists k point masses of size ϵ , one for each f p
A y q in the f p x q “ q p d | x q space.
We can now inspect this space for point masses of size at least ϵ to find at most O p 1 { ϵ q such point masses among which are contained the k point masses corresponding to the anchor subdomains.
Discretizing this space by assigning each point mass to a group (and non-point masses to a single additional group), we have k groups that have support only in one y each.
Thus, we have achieved a discretization with anchor words.
Further, since the discrete space arises from a pushforward of the continuous space through f , the discrete space also satisfies the latent label shift assumption A.1.
We now use Theorem 3.1 to claim identifiability of Q Y | D .
We then use Lemmas B.1 and B.2 to prove parts (ii) and (iii).
C. Proofs of Lemmas
In this section, we present several new lemmas which are required to prove Theorem 3.2, and provide proofs.
We also provide proofs for Lemmas B.1, B.2, and B.3.
Lemma C.1.
Let distribution Q over random variables X, Y, D satisfy A.1–A.4.
Then for all y P Y , q p
y q ą 0 .
That is, all labels have nonzero probability under Q .
Proof of Lemma C.1.
Proof by contradiction.
Let y P Y with q p y q “ 0 .
[Equation]
Since q p y | D “ d q ě 0 for all d P R , we see that if q p y q “ 0 , then q p y | D “ d q “ 0 for all d P R .
Then r Q Y | D s y,d “ 0 for all d P R .
Then there is a row (row d ) in the matrix Q Y | D in which every entry is 0, so Q Y | D cannot be full row rank k .
This violates assumption A.2.
Then by contradiction we have shown q p y q ą 0 .
Lemma C.2.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1–A.4.
Let x
P
X
such that q p x q ą 0 .
Then if x P A y for some y P Y , we have that q p y | X “ x q “ 1 , and for all y 1 P Y zt y u ,
q p y 1 | X “ x q “ 0 .
The converse is also true: if
q p y | X “ x q “ 1 for some y P Y and q p y 1 | X “ x q “ 0 @
y 1 P Y zt y u
, then we know
that x P A y
Proof of Lemma C.2.
We prove directions one at a time.
• Forward direction.
Assume x P A y .
[Equation]
q
Recalling q p x | y q ą 0 (by A.4) and q p y q ą 0 (by Lemma C.1), we know that q p x q “ q p y q q p x | Y “ y q ą 0 .
Then q p y | X “ x q “ q p y q q p x | Y “ y q q p x q “ q p x q q p x q “ 1 (Bayes’ rule).
Because probabilities sum to 1, we write q p y | X “ x q` ř y 1 P Y zt
y u q p y
| X “ x q “ 1 .
Then because q p y | X “ x q “ 1 , we have : ř y 1 P Y
zt y u
q p y 1 | X “ x q “ 0 .
Then for all y 1 P
Y zt y u , it must be that q p y 1 | X “ x q “ 0 .
We have shown q p y | X “ x q “ 1 , and for all y 1 P Y zt y u ,
q p y 1 | X “ x q “ 0 .
• Converse
Assume q p y | X “ x q “ 1 and for all y 1 P Y zt y u , q p y 1 | X “ x q “ 0 .
Recall q p x q ą 0 .
Also, q p y q ą 0 by Lemma C.1.
Then q p x | Y “ y q “ q p y | X “ x q q p x q q p y q “ p 1 q q p x q q p y q ą 0 .
Let y 1 P Y
zt y u .
Then q p x | Y “ y 1 q “ q p y 1 | X “ x q q p x q q p y 1 q “ p 0 q q p x q q p y 1 q “ 0 .
Now
because q p x | Y “ y q ą 0 and @ y 1 P Y zt y u , q p x | Y “ y 1 q “ 0 , we know x P A y .
Lemma C.3.
Let random variables X, Y, D and distribution Q satisfy Assumptions A.1–A.4.
Then, the matrix
Q D | Y , defined as an r ˆ k matrix whose elements are r Q D | Y s i,j “ Q p D “ i | Y “ j q , and in which each column is a conditional distribution over the domains given a label, has linearly independent columns.
Furthermore, Q D | Y can be computed directly from only Q Y | D .
Proof of Lemma C.3.
Let random variables X, Y, D and distribution Q satisfy Assumptions A.1–A.4.
Each r Q D |
[Equation]
d,
[Equation]
Since each y th column of Q D | Y is a probability distribution that sums to 1, and rq p y q is constant down each y th column, we can obtain Q D | Y by simply taking Q J Y | D , in which each r Q J Y
| D s d,y “ r Q Y | D s y,d “ q p y | D “ d q , and normalizing the columns so they sum to 1.
The matrix Q Y | D has linearly independent rows by Assumption A.2.
Then Q J Y | D has linearly independent columns.
Scaling these columns by a nonzero value does not change their linear independence, so the columns of Q D | Y are also linearly independent.
Then matrix Q D | Y has linearly
Lemma C.4.
Let random variables X, Y, D and distribution Q satisfy Assumptions A.1–A.4.
Let d P R , x P X , y P Y .
Then q p d | X “ x, Y “ y q “ q p d | Y “ y q .
Proof of Lemma C.4.
[Equation]
Proof of Lemma B.1.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
Let x
P
X with q p x q ą 0 , and y P Y .
Assume we know Q Y | D and r f p x qs d “ q p d | X “ x q .
With Q Y | D , we know q p y | D “ d q for all y, d .
Also, with the oracle domain discriminator f , we are able to obtain q p d | X “ x q for all x, d .
For all x P X , d P R , q p d | X “ x q “ ÿ y 1 P Y
q
p
d | X “ x, Y “ y 1 q q p y
[Equation]
| X “ x q “ ÿ y 1 P Y q p d | Y “ y 1 q q p y 1 |
X “ x q , using Lemma C.4 .
Define the vector-valued function g : X Ñ R k such that r g p x qs y “ q p y | X “ x q for all x P supp Q p X q .
Q D | Y is a matrix of shape r ˆ k , with r Q D
| Y s i,j “ Q p D “ i | Y “ j q .
It can be computed from Q Y | D and has linearly independent columns— both facts shown in Lemma C.3.
Then r f p x qs d “ q p d | X “ x q “ Q D | Y r d, : s g p x q , a product between the d th row vector of Q D | Y and the column vector g p x q .
Then f p x q “ Q D | Y g p x q .
This system is a linear system with r ě k equations.
Recalling that Q D | Y has k linearly independent columns, we can select any k linearly independent rows of Q D | Y to solve the equation for the true, unique solution for the unknown vector g p x q .
We can also describe this with the pseudo-inverse: g p x q “ p Q D | Y q :
f p x q .
Then we have r g p x qs y “ q p y | X “ x q for all y P Y .
Proof of Lemma B.2.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
Let x P X , d P R with q p x, d q ą 0 , and y P Y .
Assume we know matrix Q Y | D and q p y 1 | X “ x q , @
y 1 P Y .
We can compute Q D | Y from Q Y | D via Lemma C.3.
[Equation]
Using Lemma C.4, q p d | X “ x, Y “ y q “ q p d | Y “ y q .
We apply this property.
[Equation]
The denominator q p d | X “ x q is constant across all values of y , so we can write that q p y | X “ x, D “ d q 9 q p d | Y “ y q q p y | X “ x q and normalize to find the probability:
[Equation]
We know q p d | Y “ y q as r Q D | Y s d,y , and every q p d | Y “ y 1 q , where y 1 P Y zt y u , as r Q D | Y s d,y 1 .
We also know q p y | X “ x q and every q p y 1 |
X “ x q where y 1 P Y zt y u , by the precondition assumptions.
Then we can compute q p y | X “ x, D “ d q .
Proof of Lemma B.3.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
Recall f :
R p Ñ
R r is a vector-valued oracle function such that r f p x qs d “ q p d | X “ x q for all x
P supp Q p X q .
Also let us recall that Q D | Y is defined as an r ˆ k matrix whose elements r Q D
| Y s i,j “ Q p D “ i | Y “ j q , and each column is a conditional distribution over the domains given a label.
It has linearly independent columns by
Lemma C.3.
First recognize that for all d P R , x P X such that q p x q ą 0 ,
[Equation]
q , using the equality from Lemma C.4.
Then we can write f p x q “ ř y 2 P Y q p y 2 | X “ x q Q D | Y r : , y 2 s , where Q D | Y r : , y 2 s is the y 2 th column of Q D | Y .
Now we could also rewrite f p x q “ Q D | Y r Q p Y “ 1 | X “ x q ... Q p Y “ k | X “ x qs J .
We now prove two key components of the lemma.
Let y P Y .
Let x P A
y
such that q p x q ą 0 .
• Points in same anchor sub-domain map together
Let x 1 P
A
y
such that q p x 1 q ą 0 .
We now seek to show that f p x q “ f p x 1 q .
Recall that x, x 1 P A y .
By Lemma C.2, q p y | X “ x q “ q p y | X “ x 1 q “ 1 .
Also by lemma C.2, @
y 2 P Y zt y u , q p y 2 |
X “ x q “ q p y 2 | X “ x 1 q “ 0 .
Then for all y 2 P Y , q p y 2 | X “ x q “ q p y 2 | X “ x 1 q .
Therefore, @
d P R ,
[Equation]
1 q “ r f p x 1 qs d .
Then f p x q “ f p x 1 q .
• Point outside of the anchor sub-domain does not map with points in the anchor sub-domain .
Let x
R
A
y
such that q p x 0 q ą 0 .
We now seek to show that f p x q ‰ f p x 0 q .
Because x 0 R A y with q p x 0 q ą 0 , and because A y contains all x such that q p x q ą 0 , q p y | X “ x q “ 1 , and q p y 1 | X “ x q “ 0 for all y 1 P Y zt y u , then by Lemma C.2, it must be that one of the following cases is true: – Case 1: q p y | X “ x 0 q ‰– Case 2: q p y 1 | X “ x 0 q ą 0 for some y 1 P Y zt y u .
In all circumstances, there exists some y 2 P Y : q p y 2 | x 0 q ‰ q p y 2 | x q .
Then,
[Equation]
Because Q D | Y has linearly independent columns (shown
in Lemma C.
3), we now know that
[Equation]
So f p x q ‰ f p x 0 q .
D. Proof of Theorem 3.
Proof of Theorem 3.2.
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
Recall f : X Ñ R r is a vector-valued oracle function such that r f p x qs d “ q p d | X “ x q for all x P supp Q p X q .
It is known because we know the marginal q p x, d q .
Let y P Y .
Then by Lemma B.3, f sends every x P
A y (and no other x R A y ) to the same value.
We overload notation to denote this as f p A y q .
Then Q p f p X q
“ f p A y qq “ Q p X P A y q ě ϵ .
Then in the marginal distribution of f p X q with respect to distribution Q , there is a distinct point mass on each f p A y q , with mass at least ϵ .
Because we know the marginal q p x, d q , we know the marginal q p x q , so we can obtain the distribution of f p X q with respect to distribution Q .
If we analyze the marginal distribution of f p X q with respect to distribution Q , and recover all point masses with mass at least ϵ , we can recover no more than O p 1 { ϵ q such points.
We set m P Z ` so that the number of points we recovered is m ´ 1 .
We denote a mapping ψ : R r Ñ r m s .
This mapping sends each value of f p x q corresponding to a point mass with mass at least ϵ to a unique index in t 1 , ..., m ´ 1 u .
It sends any other value in R p to m .
We note that the ordering of the point masses might have p m ´ 1 q ! permutations.
Notice that the point mass on each f p
A y q must be recovered among these m ´ 1 masses.
Recall that for all y P Y , f p x q “ f p A y q if and only if x P A y .
Then for all y P Y , ψ p f p x qq “ ψ p f p A y qq if and only if x P A y , because ψ does not send any other value in R r besides f p A y q to ψ p f p
A y qq .
For convenience, we now define a mapping c : X Ñ r m s such that c “ ψ ˝ f .
We will also abuse notation here to denote c p
A y q “ ψ p f p
A y qq .
Then c p X q is a discrete, finite random variable that takes values in r m s .
As c is a pushforward function on X , c p X q satisfies the label shift assumption because X does (i.e., when conditioning on Y , the distribution of c p X q is domain-invariant).
We might now define a matrix Q c p X q| D in which each entry r Q c p X q| D
s i,d “ Q p c
p
X q “ i | D “ d q .
We recall that we know the number of true classes k .
Then we know that there is a (possibly unique) unknown decomposition of the following form:
[Equation]
using the label shift property .
To express this decomposition in matrix form, we write Q c p X q| D “ Q c p X q| Y Q Y | D .
Now we make observations about the unknown Q c p X q| Y .
For all y P Y ,
[Equation]
Then for each y P Y , the row of Q c p X q| Y with row index c
p
A y q is positive in the y th column, and zero everywhere else.
Restated, for each y P Y , there is some row with positive entry exactly in y th column.
This is precisely the anchor word assumption for a discrete, finite random variable.
We already know that Q Y | D is full row-rank, so because Q c p X q| Y satisfies the anchor word assumption, we can identify Q Y | D up to permutation of rows by Theorem 3.1.
In other words, when we set the constraint that the recovered Q c p X q| Y must have k columns and satisfy anchor word and the recovered Q Y | D must have k rows and be full row-rank, any solution to the decomposition Q c p X q| D “ Q c p X q| Y Q Y | D must identify the ground truth
Q Y | D , up to permutation of its rows.
E. Minimizing Cross-Entropy Loss yields Domain Discriminator
Let distribution Q over random variables X, Y, D satisfy Assumptions A.1-A.4.
We here examine the behavior of the crossentropy loss, in the infinite data case (when we can work with expectations over the entire distribution instead of empirical expectations over a finite set of datapoints).
Define the vector-valued function z : R Ñ R r such that z p d q is a one-hot vector of length r , such that r z p d qs i “ 1 , iff d “ i .
Then we write the cross-entropy loss with targets as true domains as
[Equation]
In order to find the minimizer of the cross entropy loss over the class of all functions from R p Ñ r
[Equation]
1 s r
[Equation]
we formulate the following objective with the Lagrange constraint:
[Equation]
D “ i | X q ff ` λ ˜ i
“ r ÿ i “r f p X qs i
[Equation]
¸
Setting partial derivative with respect to r f p
X qs r to 0 , we get ´
Q p D “ i | X q r f ‹ p
X qs i ` λ “ 0 and r f ‹ p
X qs i “ 1 λ Q p D “ i |
X q .
From KKT condition, the optimal solution lies on constraint surface, giving: i “ r ÿ
[Equation]
Finally, we get r f ‹ p
X qs i “ Q p D “ i | X q , so the optimal f ˚ by the cross entropy loss as defined will in fact recover the oracle domain discriminator.
F. Additional Experimental Details
Our code is available at https://github.com/manleyroberts/lls-ddfa .
Here we present the full generation procedure for semisynthetic example problems, and discuss the parameters.
[Figure]
Choose a Dirichlet concentration parameter α ą 0 , maximum condition number κ ě 1 (with respect to 2-norm), and domain count r ě k . 2.
For each y P r k s , sample p d p y q „ Dir p αk 1 k q . 3.
Populate the matrix Q Y | D with the computed p d p y q s. If cond p Q Y | D q ě k , return to step 2 and re-sample.
Distribute examples across domains according to Q Y | D , for each of train, test, and valid sets.
This procedure entails creating a quota number of examples for each (class, domain) pair, and drawing datapoints without replacement to fill each quota.
We must discard excess examples from some classes in the dataset due to class imbalance in the Q Y | D matrix.
Due to integral rounding, domains may be slightly imbalanced.
Conceal true class information and return p x i , d i q pairs.
It is important to note the role of κ and α in the above formulation.
Although they are unknown parameters to the classification algorithm, they affect the sparsity of the Q Y | D and difficulty of the problem.
Small α encourages high sparsity in p d p y q , and large α causes p d p y q to tend towards a uniform distribution.
We observe an example of the effects of α in
Fig.
κ has a strong effect on the difficulty of the problem.
Consider the case when k “ 2 .
When κ “ 1 , the only potential
Q Y | D matrices are I 2 up to row permutation (which means that domains and classes are exactly correlated, so the domain indicates the class and the problem is supervised).
In the other limit, if κ Ñ `8 , we may generate Q Y | D matrices that are singular, breaking needed assumptions for domain discriminator output to uniquely identify true class of anchor subdomains.
κ also helps control the class imbalance (if a row of Q Y | D is small, indicating that the class is heavily under-represented across all domains, the condition number will increase).
F.1. FieldGuide-2 and FieldGuide-28 Datasets
The dataset and description is available at https://sites.google.com/view/fgvc6/competitions/ butterflies-moths-2019 .
From this data we create two datasets FieldGuide-2 and FieldGuide-28.
For FieldGuide28 we select the 28 classes which have 1000 datapoints in the training file.
Since the test set provided in the website does not have annotations, we manually create a test set by sampling 200 datapoints from training file of each of the 28 classes.
Therefore, we finally have 22400 trainging points and 5600 testing points.
The FieldGuide-2 dataset is created by considering two classes from the created FieldGuide-28 dataset.
F.2. Hyperparameters and Implementation Details: SCAN baseline
We select the unsupervised classification method SCAN as a state-of-the-art baseline (Van Gansbeke et al., 2020).
SCAN pretrains a ResNet (He et al., 2016) backbone using SimCLR
(Chen et al., 2020) and MoCo (He et al., 2020) setups (pretext tasks).
SCAN then trains a clustering head to minimize the SCAN loss.
SCAN code is available at https: //github.com/wvangansbeke/Unsupervised-Classification .
To evaluate SCAN, we use the public pretrained weights available for CIFAR-10, CIFAR-20, and ImageNet-50.
We also train SCAN ourselves on the train and validation portions of the FieldGuide2 and FieldGuide28 datasets with a ResNet18 backbone and SimCLR pretext task.
We replicate the hyperparameters used for CIFAR training.
We make sure to evaluate SCAN classification on the same potentially class-imbalanced test subset we create for each experiment.
Since SCAN is fit on a superset of the data DDFA sees, we believe this gives a slight data advantage to the
SCAN baseline (although we acknowledge that the class balance for SCAN training is also potentially different from its evaluation class balance).
With the Hungarian algorithm, implemented in (Crouse, 2016; Virtanen et al., 2020), we compute the highest true accuracy among any permutation of predicted class labels (denoted “Test acc”).
• CIFAR-10 and CIFAR-20 Datasets (Krizhevsky & Hinton, 2009)
We use ResNet-18 (He et al., 2016) backbone with weights trained by SCAN-loss and obtained from the SCAN repo https://github.com/wvangansbeke/Unsupervised-Classification .
We use the same transforms present in the repo for test data.
• ImageNet-50 Dataset (Deng et al., 2009)
We use ResNet-50 backbone with weights trained by SCAN-loss and obtained from the SCAN repo.
We use the same transforms present in the repo for test data.
• FieldGuide-2 and FieldGuide-28 Datasets
For each of the two datasets, we pretrain a different SCAN baseline network (including pretext and SCAN-loss steps) on all available data from the dataset.
The backbone for each is ResNet-18.
For training the pretext task, we use the same transform strategy used in the repo for CIFAR-10 data (with mean and std values as computed on the Fieldguide-28 dataset, and crop size 224).
For training SCAN, we resize the smallest image dimension to 256, perform a random horizontal flip and random crop to size 224.
We also normalize.
For validation we resize smallest image dimension to 256, center crop to 224, and normalize.
F.3. Hyperparameters and Implementation Details: DDFA (RI)
This is the DDFA procedure with random initialization.
We train ResNet-50 (He et al., 2016)
(with random initialization and added dropout) based on the implementation from https://github.com/kuangliu/pytorch-cifar on images x i with domain indices
d i as the label, choose best iteration by valid loss, pass all training and validation data through p f , and cluster pushforward predictions p
f p x i q into m ě k clusters with Faiss K-Means (Johnson et al., 2019).
We compute the p Q c p X q| D matrix and run NMF to obtain p Q c p X q| Y , p
Q Y | D .
To make columns sum to 1, we normalize columns of p Q c p X q| Y , multiply each column’s normalization coefficient over the corresponding row of p Q Y | D (to preserve correctness of the decomposition), and then normalize columns of p Q Y | D .
Some NMF algorithms only output solutions satisfying the anchor word property (Arora et al., 2012a; Kumar et al., 2013;
Gillis & Vavasis, 2014).
We found the strict requirement of an exact anchor word solution to lead to low noise tolerance.
We therefore use the Sklearn implementation of standard NMF (Cichocki & Phan, 2009; Pedregosa et al., 2011; Tan & F´evotte, 2012).
We predict class labels with Algorithm 2 .
With the Hungarian algorithm, implemented in (Crouse, 2016; Virtanen et al., 2020), we compute the highest true accuracy among any permutation of these labels (denoted “Test acc”).
With the same permutation, we reorder rows of p Q Y | D , then compute the average absolute difference between corresponding entries of p
Q Y | D and Q Y | D (denoted “ Q Y | D err”).
In order to make hyperparameter choices for final experiments, such as the choice of the NMF solver, clustering algorithm, and learning rate, we consulted CIFAR-10 and CINIC-10 (similar to an extension of CIFAR-10)
(Darlow et al., 2018) final test task accuracy.
We were also able to confirm our intuition that minimum validation loss on domain discriminator training corresponds to good final task performance.
Final evaluation runs on CIFAR-10 were made with the best hyperparameters found here.
We acknowledge that this may lead to test-set overfitting on CIFAR-10, but point out that on all other datasets, we consulted only validation loss on domain discriminator training when adjusting the hyperparameters
. Final evaluation
runs used the following fixed hyperparameters:
Common Hyperparameters
Architecture: ResNet-50 with added dropout
Faiss KMeans number of iterations (niter):
Faiss Kmeans number of clustering redos (nredo):
Learning Rate: 0.
Learning Rate Decay: Exponential, parameter 0.
SKlearn NMF initialization: random
Dataset
-Specific Hyperparameters
• CIFAR-10 Dataset Training Epochs:Number of Clusters ( m ):• CIFAR-20 Dataset Training Epochs:Number of Clusters ( m ):• ImageNet-50 Dataset Not evaluated.
• FieldGuide-2 and FieldGuide-28 Datasets Not evaluated.
F.4. Hyperparameters and Implementation Details: DDFA (SI) and DDFA (SPI)
In Sec. 5, we used DDFA in the results section as a shorthand for DDFA (SPI).
This is the DDFA procedure with SCAN initialization.
The procedure is identical to the standard DDFA procedure, except that SCAN (Van Gansbeke et al., 2020) pre-trained weights or SCAN (Van Gansbeke et al., 2020) contrastive pre-text weights are used to initialize the domain discriminator before it is fine-tuned on the domain discrimination task.
Hyperparameters used also differ.
When SCAN pretrained weights are available, we use those.
When they are not, we train SCAN ourselves.
Much like SCAN (RI), we used CIFAR-10 final test acuracy to choose hyperparameters and make algorithm decisions.
For all other datasets, we consulted only validation domain discrimination loss.
Final evaluation runs used the following fixed hyperparameters:
Common
Hyperparameters
Faiss KMeans number of
iterations
(niter):
Faiss Kmeans number of clustering redos (nredo)
Learning
Rate: 0.
Learning Rate Decay: Exponential, parameter 0.
SKlearn NMF initialization: random
Dataset-Specific Hyperparameters • CIFAR-10 Dataset Architecture: ResNetPre-seed: Weights trained with SCAN pretext and SCAN-loss on entirety of CIFAR-10 (from SCAN repo).
Training Epochs
Number of Clusters ( m ):Transforms used: Same as SCAN repo.
• CIFAR-20 Dataset Architecture: ResNetPre-seed: Weights trained with SCAN pretext and SCAN-loss on entirety of CIFAR-20 (from SCAN repo).
Training Epochs:Number of Clusters ( m ):Transforms used:
Same as SCAN repo.
• ImageNet-50 Dataset Architecture:
ResNetPre-seed: Weights trained with SCAN pretext and SCAN-loss on entirety of ImageNet-50 (from SCAN repo).
Training Epochs:Number of Clusters ( m ):Transforms used:
Same as SCAN repo.
• FieldGuide-2 Dataset Architecture: ResNetPre-seed: Weights trained with SCAN pretext on entirety of FieldGuide-2 (trained by us).
Training Epochs
[Table]
Number of Clusters ( m ):Transforms used for pretext: Same strategy as CIFAR-10 in SCAN repo with appropriate mean, std, and crop size 224.
Transform used for SCAN:
Resize to 256, Random horizontal flip, Random crop to 224, normalize Learning rate used for SCAN:
0.001 (other hyperparameters were same as in SCAN repo for CIFAR-10) • FieldGuide-28 Dataset Architecture: ResNetPre-seed: Weights trained with SCAN pretext on entirety of FieldGuide-28 (trained by us).
Training Epochs:Number of Clusters ( m ):Transforms used for pretext: Same strategy as CIFAR-10 in SCAN repo with appropriate mean, std, and crop size 224.
Transform used for SCAN:
Resize to 256, Random horizontal flip, Random crop to 224, normalize Learning rate used for SCAN: 0.01 (other hyperparameters were same as in SCAN repo for CIFAR-10)
G. Additional Experimental Results
[Table] [Table]
[Table] [Table]
[Table]
H. Discussion of Convex Polytope Geometry
The geometric properties of topic modeling for finite, discrete random variables has been explored in depth in related works ((Donoho & Stodden, 2003; Huang et al., 2016;
Chen et al., 2021)).
The observation that columns in Q X | D are convex combinations of columns in Q X | Y leads to a perspective on identification of the matrix decomposition as identification of the convex polytope in R m which encloses all of the columns of Q X | D (the corners of which correspond to columns of
Q X | Y under certain identifiability conditions).
Here, we briefly discuss an interesting but somewhat different application of convex polytope geometry.
Instead of a convex polytope in R m with corners as columns of Q X | Y , we concern ourselves with the convex polytope in R r with corners as columns in Q D | Y , which must enclose all values taken by the oracle domain discriminator f p x q for x P X , q p x q ą 0 .
Let us assume that Assumptions A.1–A.4 are satisfied.
We recall the oracle domain discriminator f defined such that r f p x qs d “ q p d | X “ x q .
Let x
P
X
“ R p .
Now, since the r values q p d | X “ x q for d P t 1 , 2 , ..., r u together constitute a categorical distribution, each of these r values lie between 0 and 1 , and also their sum adds to 1.
Therefore the vector f p x q lies on the simplex ∆ r ´ 1 .
We now express f p x q as a convex combination of the k columns of Q D | Y .
We denote these column vectors Q D | Y r : , y s for each y P Y “ r k s .
Note that each such vector also lies in the ∆ r ´ 1 simplex.
As an intermediate step in the proof of Lemma B.3 given in App.
C, we showed that each f p x q is a linear combination of these columns of Q D | Y with coefficients q p y | X “ x q for all y P Y .
That is, we can rewrite f p x q “
Q D | Y r Q p Y “ 1 | X “ x q ... Q p Y “ k | X “ x [Equation]
Since the coefficients in the linear combination are probabilities which, taken together, form a categorical distribution, they lie between 0 and 1 and sum to 1.
Thus, for all x P X with q p x q ą 0 , f p x q can be expressed as a convex combination of the columns of Q D | Y .
Therefore, for any x with q p x q ą 0 , f p x q lies inside the k ´ vertex convex polytope with corners as the columns of Q D | Y (which are linearly independent by Lemma C.3).
This polytope is embedded in ∆ r ´ 1 .
Now consider x in an anchor sub-domain, that is x P
A y for some y P Y .
We know that if q p x q ą 0 , q p y | X “ x q “ 1 , q p y 1 | X “ x q “ 0 for all y 1 ‰ y (Lemma C.2).
Since the q p y | X “ x q are now one-hot, we have that f p x q “ Q D | Y r : , y s for x P A y .
In words, this means that f p A y q is precisely the y th column of Q D | Y .
It follows that the domain discriminator maps each of the k anchor sub-domains exactly to a unique vertex of the polytope.
The situation is described in Fig.
We could now recover the columns of Q D | Y , up to permutation, with the following procedure:
Push all x P X through f . 2.
Find the minimum volume convex polytope that contains the resulting density of points on the simplex.
The vectors that compose the vertices of this polytope are the columns of Q D | Y , up to permutation.
[Figure]
Note that from Assumption A.4, we are guaranteed to have a region of the input space with at least ϵ ą 0 mass that gets mapped to each of the vertices when carrying out step (i).
Therefore, our discovered minimum volume polytope must enclose all of these vertices.
Since no mass will exist outside of the true polytope, requiring a minimum volume polytope will ensure that the recovered polytope fits the true polytope’s vertices precisely (as any extraneous volume outside of the true polytope must be eliminated).
Then step (ii) recovers Q D | Y , up to permutation of columns.
Having recovered Q D | Y , we can use Lemmas B.1 and B.2 to recover q p y | x, d q .
This procedure is a geometric alternative to the clustering approach outlined in Algorithm 1 .
In practice, fitting a convex hull around the outputs of a noisy, non-oracle estimated domain discriminator may be computationally expensive, and noise may lead this sensitive procedure fail to recover the true vertices.