W EAKLY S UPERVISED D ISENTANGLEMENT WITH G UARANTEES
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning disentangled representations that correspond to factors of variation in real-world data is critical to interpretable and human-controllable machine learning.
Recently, concerns about the viability of learning disentangled representations in a purely unsupervised manner has spurred a shift toward the incorporation of weak supervision.
However, there is currently no formalism that identiﬁes when and how weak supervision will guarantee disentanglement.
To address this issue, we provide a theoretical framework to assist in analyzing the disentanglement guarantees (or lack thereof) conferred by weak supervision when coupled with learning algorithms based on distribution matching.
We empirically verify the guarantees and limitations of several weak supervision methods (restricted labeling, match-pairing, and rank-pairing), demonstrating the predictive power and usefulness of our theoretical framework.
I NTRODUCTION
Many real-world data can be intuitively described via a data-generating process that ﬁrst samples an underlying set of interpretable factors, and then—conditional on those factors—generates an observed data point.
For example, in image generation, one might ﬁrst generate the object identity and pose, and then build an image of this object accordingly.
The goal of disentangled representation learning is to learn a representation where each dimensions of the representation measures a distinct factor of variation in the dataset (Bengio et al., 2013).
Learning such representations that align with the underlying factors of variation may be critical to the development of machine learning models that are explainable or human-controllable (Gilpin et al., 2018; Lee et al., 2019;
Klys et al., 2018).
In recent years, disentanglement research has focused on the learning of such representations in an unsupervised fashion, using only independent samples from the data distribution without access to the true factors of variation (Higgins et al., 2017;
Chen et al., 2018a; Kim & Mnih, 2018;
Esmaeili et al., 2018).
However, Locatello et al.
(2019) demonstrated that many existing methods for the unsupervised learning of disentangled representations are brittle, requiring careful supervision-based hyperparameter tuning.
To build robust disentangled representation learning methods that do not require large amounts of supervised data, recent work has turned to forms of weak supervision (Chen & Batmanghelich, 2019; Gabbay & Hoshen, 2019).
Weak supervision can allow one to build models that have interpretable representations even when human labeling is challenging (e.g., hair style in face generation, or style in music generation).
While existing methods based on weaklysupervised learning demonstrate empirical gains, there is no existing formalism for describing the theoretical guarantees conferred by different forms of weak supervision (Kulkarni et al., 2015;
Reed et al., 2015; Bouchacourt et al., 2018).
In this paper, we present a comprehensive theoretical framework for weakly supervised disentanglement, and evaluate our framework on several datasets.
Our contributions are several-fold.
We formalize weakly-supervised learning as distribution matching in an extended space.
We propose a set of deﬁnitions for disentanglement that can handle correlated factors and are inspired by many existing deﬁnitions in the literature (Higgins et al., 2018; Suter et al., 2018; Ridgeway & Mozer, 2018).
Using these deﬁnitions, we provide a conceptually useful and theoretically rigorous calculus of disentanglement.
We apply our theoretical framework of disentanglement to analyze the theoretical guarantees of three notable weak supervision methods (restricted labeling, match pairing, and rank pairing) and experimentally verify these guarantees.
F ROM UNSUPERVISED TO WEAKLY SUPERVISED DISTRIBUTION MATCHING
Our goal in disentangled representation learning is to identify a latent-variable generative model whose latent variables correspond to ground truth factors of variation in the data.
To identify the role that weak supervision plays in providing guarantees on disentanglement, we ﬁrst formalize the model families we are considering, the forms of weak supervision, and ﬁnally the metrics we will use to evaluate and prove components of disentanglement.
We consider data-generating processes where S ∈ R n are the factors of variation, with distribution p ∗ ( s ) , and X ∈ R m is the observed data point which is a deterministic function of S , i.e., X = g ∗ ( S ) .
Many existing algorithms in unsupervised learning of disentangled representations aim to learn a latent-variable model with prior p ( z ) and generator g , where g ( Z ) d = g ∗ ( S ) .
However, simply matching the marginal distribution over data is not enough: the learned latent variables Z and the true generating factors S could still be entangled with each other (Locatello et al., 2019).
To address the failures of unsupervised learning of disentangled representations, we leverage weak supervision, where information about the data-generating process is conveyed through additional observations.
By performing distribution matching on an augmented space (instead of just on the observation of X ), we can provide guarantees on learned representations. [Equation]
[Figure]
We consider three practical forms of weak supervision: restricted labeling, match pairing, and rank pairing.
All of these forms of supervision can be thought of as augmented forms of the original joint distribution, where we partition the latent variables in two S =
( S I , S \ I ) , and either observe a subset of the latent variables or share latents between multiple samples.
A visualization of these augmented distributions is presented in Figure 1, and below we detail each form of weak supervision.
In restricted labeling , we observe a subset of the ground truth factors, S I in addition to X .
This allows us to perform distribution matching on p ∗ ( s I , x ) , the joint distribution over data and observed factors, instead of just the data, p ∗ ( x ) , as in unsupervised learning.
This form of supervision is often leveraged in style-content disentanglement, where labels are available for content but not style (Kingma et al., 2014; Narayanaswamy et al., 2017; Chen et al., 2018b; Gabbay & Hoshen, 2019).
Match Pairing uses paired data, ( x, x (cid:48) )
that share values for a known subset of factors, s I .
In many data modalities, certain factors of variation may be difﬁcult to prescribe as an explicit label, but it is easier to collect pairs of samples that share the same underlying factor (e.g., it may be easier to collect pairs of images of different people wearing the same glasses, than to explicitly deﬁne a label for glasses style).
Match pairing is a weaker form of supervision than restricted labeling, as the learning algorithm no longer depends on the underlying value s I .
Several variants of match pairing have appeared in the literature (Kulkarni et al., 2015; Bouchacourt et al., 2018; Ridgeway & Mozer, 2018), but typically focus on groups of observations in contrast to the paired setting we consider in this paper.
Rank Pairing is another form of paired data generation where the pairs ( x, x (cid:48) ) are generated in an i.i.d.
fashion, and an additional indicator variable y is observed that determines whether the corresponding latent s i is greater than s (cid:48)
i : y = 1 { s i ≥ s (cid:48) i } .
Such a form of supervision is effective when it is easier to compare two samples with respect to an underlying factor than to directly collect labels (e.g., comparing two object sizes versus providing a ruler measurement of an object).
Although supervision via ranking features prominently in the metric learning literature (McFee & Lanckriet, 2010; Wang et al., 2014), our focus in this paper will be on rank pairing in the context of disentanglement guarantees.
For each form of weak supervision, we can train generative models with the same structure as in
Figure 1, using data sampled from the ground truth model and a distribution matching objective.
For example, for match pairing, we train a generative model ( p ( z ) , g ) such that the paired random variable ( g ( Z I , Z \ I ) , g ( Z I , Z (cid:48) \ I )) from the generator matches the distribution of the corresponding paired random variable ( g ∗ ( S I , S \ I ) , g ∗ ( S I , S (cid:48)\ I )) from the augmented data distribution.
D EFINING D ISENTANGLEMENT
To identify the role that weak supervision plays in providing guarantees on disentanglement, we introduce a set of deﬁnitions that are consistent with our intuitions about what constitutes “disentanglement” and amenable to theoretical analysis.
Our new deﬁnitions decompose disentanglement into two distinct concepts: consistency and restrictiveness.
Different forms of weak supervision can enable consistency or restrictiveness on subsets of factors, and in Section 4 we build up a calculus of disentanglement from these primitives.
We discuss the relationship to prior deﬁnitions of disentanglement in Appendix A.
3. D ECOMPOSING D ISENTANGLEMENT INTO C ONSISTENCY AND R ESTRICTIVENESS
[Figure]
To ground our discussion of disentanglement in a concrete example, we shall consider an oracle that generates shapes, with the underlying factors of variation size ( S 1 ), shape ( S 2 ), and color ( S 3 ).
We now wish to determine whether Z 1 of our generative model disentangles the concept of size.
Intuitively, one way to check whether Z 1 of the generative model disentangles size ( S 1 ) is to visually inspect what happens as we vary Z 1 , Z 2 , and Z 3 , and see whether the resulting visualizations are consistent with Figure 2a.
In doing so, our visual inspection checks for two properties:
When Z 1 is ﬁxed, the size ( S 1 ) of the generated object never changes.
When Z 1 is changed, the change is restricted to the size ( S 1 ) of the generated object.
We thus argue that disentanglement decomposes into these two properties, which we refer to as generator consistency and generator restrictiveness .
We shall now formalize these two properties.
Let H be a hypothesis class of generative models from which we assume the true data-generating function is drawn.
Each element of the hypothesis class H is a tuple ( p ( s ) , g, e ) , where p ( s ) describes the distribution over factors of variation, the generator g is a function that maps from the factor space S ∈ R n to the observation space X ∈ R m , and the encoder e is a function that maps from X → S .
S and X can consist of both discrete and continuous random variables.
We impose a few mild assumptions on H (see Appendix I.1).
Notably, we assume every factor of variation is exactly recoverable from the observation X , i.e. e ( g ( S ))
=
S .
Given an oracle model h ∗ =
( p ∗ , g ∗ , e ∗ )
∈ H , we would like to learn a model h
= ( p, g, e ) ∈ H whose latent variables disentangle the latent variables in h ∗ .
We refer to the latent-variables in the oracle h ∗ as S and the alternative model
h
’s latent variables as Z .
If we further restrict h to only those models where g ( Z ) d
= g ∗ ( S ) are equal in distribution, it is natural to align Z and S via
S = e ∗ ◦ g
( Z ) .
Under this relation between Z and S , our goal is to construct deﬁnitions that describe whether the latent code Z i disentangles the corresponding factor S i .
Generator Consistency.
Let I denote a set of indices and p I denote the generating process
[Equation]
This generating process samples Z I once and then conditionally samples
Z I twice in an i.i.d.
fashion.
We say that Z I is consistent with S I
if
[Equation]
where e ∗ I is the oracle encoder restricted to the indices I .
Intuitively, Equation (3) states that, for any ﬁxed choice of Z I , resampling of Z \ I will not inﬂuence the oracle’s measurement of the factors
S I .
In other words, S I is invariant to changes in Z \ I .
An illustration of a generative model where Z 1 is consistent with size ( S 1 ) is provided in Figure 2b.
A notable property of our deﬁnition is that the prescribed sampling process p I does not require the underlying factors of variation to be statistically independent.
We characterize this property in contrast to previous deﬁnitions of disentanglement in Appendix A.
Generator Restrictiveness.
Let p \
I denote the generating process
[Equation]
We say that Z I is restricted to S I
if
[Equation]
Equation (6) states that, for any ﬁxed choice of Z \ I , resampling of Z I will not inﬂuence the oracle’s measurement of the factors S \ I . In other words, S \ I is invariant to changes in Z I .
Thus, changing Z I is restricted to modifying only S I .
An illustration of a generative model where Z 1 is restricted to size ( S 1 ) is provided in Figure 2c.
Generator Disentanglement.
We now say that Z I disentangles S I if Z I is consistent with and restricted to S I .
If we denote consistency and restrictiveness via Boolean functions C ( I ) and R ( I ) , we can now concisely state that
[Equation]
where D ( I ) denotes whether Z I disentangles S I .
An illustration of a generative model where Zdisentangles size ( S 1 ) is provided in Figure 2a.
Note that while size increases monotonically with
Z 1 in the ﬁgure for convenience of illustration, we wish to clarify that monotonicity is orthogonal to the concepts of consistency and restrictiveness.
3. R ELATION TO B IJECTIVITY -B ASED D EFINITION OF D ISENTANGLEMENT
Under our mild assumptions on H , distribution matching on g ( Z ) d = g ( S ) combined with generator disentanglement on factor I implies the existence of two invertible functions
f I and f \ I such that the alignment via S =
e ∗ ◦ g ( Z ) decomposes into
[Equation]
This expression highlights the connection between disentanglement and invariance , whereby S I is only inﬂuenced by Z I , and S \ I is only inﬂuenced by Z \ I .
However, such a bijectivity-based deﬁnition of disentanglement does not naturally expose the underlying primitives of consistency and restrictiveness , which we shall demonstrate in our theory and experiments to be valuable concepts for describing disentanglement guarantees under weak supervision.
3. E NCODER -B ASED D EFINITIONS FOR D ISENTANGLEMENT
Our proposed deﬁnitions are asymmetric—measuring the behavior of a generative model against an encoder.
So far, we have chosen to present the deﬁnitions from the perspective of a learned generator ( p, g ) measured against an oracle encoder e ∗ .
In this sense, they are generator-based deﬁnitions.
We can also develop a parallel set of deﬁnitions for encoder-based consistency, restrictiveness, and disentanglement within our framework simply by using an oracle generator ( p ∗ , g ∗ ) measured against a learned encoder e .
We only present consistency for brevity.
Encoder Consistency.
Let p ∗
I denote the generating process
[Equation]
This generating process samples S I once and then conditionally samples S I twice in an i.i.d. fashion.
We say that S I is consistent with Z I if
[Equation]
We now make two important observations.
First, a valuable trait of our encoder-based deﬁnitions is that one can check for encoder consistency / restrictiveness / disentanglement as long as one has access to match pairing data from the oracle generator .
This is in contrast to the existing disentanglement deﬁnitions and metrics, which require access to the ground truth factors (Higgins et al., 2017; Kumar et al., 2018; Kim & Mnih, 2018; Chen et al., 2018a; Suter et al., 2018; Ridgeway & Mozer, 2018; Eastwood & Williams, 2018).
The ability to check for our deﬁnitions in a weakly supervised fashion is the key to why we can develop a theoretical framework using the language of consistency and restrictiveness.
Second, encoder-based deﬁnitions are tractable to measure when testing on synthetic data, since the synthetic data directly serves the role of the oracle generator.
As such, while we develop our theory to guarantee both generator-based and the encoder-based disentanglement, all of our measurements in the experiments will be conducted with respect to a learned encoder.
We make three remarks on notations.
First, D ( i ) :
= D ( { i } ) .
Second, D ( ∅ ) evaluates to true.
Finally, D ( I ) is implicitly dependent on either ( p, g, e ∗ ) (generator-based) or ( p ∗ , g ∗ , e ) (encoderbased).
Where important, we shall make this dependency explicit (e.g., let D ( I ; p, g, e ∗ ) denote generator-based disentanglement).
We apply these conventions to C and R analogously.
A C ALCULUS OF D ISENTANGLEMENT
We note several interesting relationships between restrictiveness and consistency.
First, by deﬁnition, C ( I ) is equivalent to R ( \ I ) .
Second, we can see from Figures 2b and 2c that C ( I ) and R ( I ) do not imply each other.
Based on these observations and given that consistency and restrictiveness operate over subsets of the random variables, a natural question that arises is whether consistency or restrictiveness over certain sets of variables imply additional properties over other sets of variables.
We develop a calculus for discovering implied relationships between learned latent variables Z and ground truth factors of variation S given known relationships as follows.
Calculus of Disentanglement Consistency and Restrictiveness
[Equation]
Union Rules
[Equation]
Intersection Rules
[Equation]
Our calculus provides a theoretically rigorous procedure for reasoning about disentanglement.
In particular, it is no longer necessary to prove whether the supervision method of interest satisﬁes consistency and restrictiveness for each and every factor.
Instead, it sufﬁces to show that a supervision method guarantees consistency or restrictiveness for a subset of factors, and then combine multiple supervision methods via the calculus to guarantee full disentanglement.
We can additionally use the calculus to uncover consistency or restrictiveness on individual factors when weak supervision is available only for sets of variables.
For example, achieving consistency on S 1 , 2 and S 2 , 3 implies consistency on the intersection S 2 .
Furthermore, we note that these rules are agnostic to using generator or encoder-based deﬁnitions.
We defer the complete proofs to Appendix I.2.
F ORMALIZING W EAK S UPERVISION WITH G UARANTEES
In this section, we address the important question of how to distinguish when disentanglement arises from the supervision method and when it comes from model inductive bias.
This challenge was ﬁrst put forth by Locatello et al.
(2019), which noted that unsupervised disentanglement is heavily reliant on model inductive bias.
As we transition toward supervised approaches, it is crucial that we formalize what it means for disentanglement to be guaranteed by weak supervision.
Sufﬁciency for Disentanglement.
Let P denote a family of augmented distributions.
We say that a weak supervision method S :
H → P is sufﬁcient for learning a generator whose latent codes
Z I disentangle the factors
S I if there exists a learning algorithm
A : P → H such that for any choice of ( p ∗ ( s ) , g ∗ , e ∗ )
∈ H , the procedure A ◦ S ( p ∗ ( s ) , g ∗ , e ∗ ) returns a model ( p ( z ) , g, e ) for which both
D ( I ; p, g, e ∗ ) and D ( I ; p ∗ , g ∗ , e ) hold, and g ( Z ) d
= g ∗ ( S ) .
The key insight of this deﬁnition is that we force the strategy and learning algorithm pair ( S , A ) to handle all possible oracles drawn from the hypothesis class H .
This prevents the exploitation of model inductive bias, since any bias from the learning algorithm
A toward a reduced hypothesis class ˆ H ⊂ H will result in failure to handle oracles in the complementary hypothesis class H \ ˆ H .
The distribution matching requirement g ( Z ) d
= g ∗ ( S ) ensures latent code informativeness, i.e., preventing trivial solutions where the latent code is uninformative (see Proposition 6 for formal statement).
Intuitively, distribution matching paired with a deterministic generator guarantees invertibility of the learned generator and encoder, enforcing that Z I cannot encode less information than S I (e.g., only encoding age group instead of numerical age) and vice versa.
A NALYSIS OF W EAK S UPERVISION M ETHODS
We now apply our theoretical framework to three practical weak supervision methods: restricted labeling, match pairing, and rank pairing.
Our main theoretical ﬁndings are that: (1) These methods can be applied in a targeted manner to provide single factor consistency or restrictiveness guarantees.
(2) By enforcing consistency (or restrictiveness) on all factors, we can learn models with strong disentanglement performance.
Correspondingly, Figure 3 and Figure 5 are our main experimental results, demonstrating that these theoretical guarantees have predictive power in practice.
6. T HEORETICAL G UARANTEES FROM W EAK S UPERVISION
We prove that if a training algorithm successfully matches the generated distribution to data distribution generated via restricted labeling, match pairing, or rank pairing of factors S I , then Z I is guaranteed to be consistent with S I :
Theorem 1.
Given any oracle ( p ∗ ( s ) , g ∗ , e ∗ ) ∈ H , consider the distribution-matching algorithm A that selects a model ( p ( z )
, g, e ) ∈ H such that:
( g ∗ ( S ) , S I )
d = ( g ( Z ) , Z I ) ( Restricted Labeling ); or 2. (cid:16) g ∗ ( S I , S \ I ) , g ∗ ( S I , S (cid:48)\ I ) (cid:17) d = (cid:16) g
( Z I , Z \ I ) , g ( Z I , Z
(cid:48)\ I ) (cid:17) ( Match Pairing ); or 3.
( g ∗ ( S ) , g ∗ ( S (cid:48) ) , 1 { S I ≤ S (cid:48) I } )
d = ( g ( Z ) , g ( Z (cid:48) ) , 1 { Z I ≤ Z (cid:48) I } ) ( Rank Pairing ) .
Then (p, g ) satisﬁes C ( I ; p, g, e ∗ ) and e satisﬁes C
( I ; p ∗ , g ∗ , e ) .
Theorem 1 states that distribution-matching under restricted labeling, match pairing, or rank pairing of S
I guarantees both generator and encoder consistency for the learned generator and encoder respectively.
We note that while the complement rule C ( I ) =
⇒ R ( \ I ) further guarantees that Z \ I is restricted to S \
I , we can prove that the same supervision does not guarantee that Z I is restricted to S
I (Theorem 2).
However, if we additionally have restricted labeling for S \ I , or match pairing for S \ I , then we can see from the calculus that we will have guaranteed R ( I )
∧ C ( I ) , thus implying disentanglement of factor I .
We also note that while restricted labeling and match pairing can be applied on a set of factors at once (i.e. | I | ≥ 1 ), rank pairing is restricted to one-dimensional factors for which an ordering exists.
In the experiments below, we empirically verify the theoretical guarantees provided in Theorem 1.
6. E XPERIMENTS
We conducted experiments on ﬁve prominent datasets in the disentanglement literature: Shapes3D (Kim & Mnih, 2018), dSprites (Higgins et al., 2017), Scream-dSprites (Locatello et al., 2019), SmallNORB (LeCun et al., 2004), and
Cars3D (Reed et al., 2015).
Since some of the underlying factors are treated as nuisance variables in SmallNORB and Scream-dSprites, we show in Appendix C that our theoretical framework can be easily adapted accordingly to handle such situations.
We use generative adversarial networks (GANs, Goodfellow et al.
for learning ( p, g ) but any distribution matching algorithm (e.g., maximum likelihood training in tractable models, or VI in latent-variable models) could be applied.
Our results are collected over a broad range of hyperparameter conﬁgurations (see Appendix H for details).
Since existing quantitative metrics of disentanglement all measure the performance of an encoder with respect to the true data generator, we trained an encoder post-hoc to approximately invert the learned generator, and measured all quantitative metrics (e.g., mutual information gap) on the encoder.
Our theory assumes that the learned generator must be invertible.
While this is not true for conventional GANs, our empirical results show that this is not an issue in practice (see Appendix G).
We present three sets of experimental results: (1) Single-factor experiments, where we show that our theory can be applied in a targeted fashion to guarantee consistency or restrictiveness of a single factor.
(2) Consistency versus restrictiveness experiments, where we show the extent to which single-factor consistency and restrictiveness are correlated even when the models are only trained to maximize one or the other.
(3) Full disentanglement experiments, where we apply our theory to fully disentangle all factors.
A more extensive set of experiments can be found in the Appendix.
6.2. S INGLE -F ACTOR C ONSISTENCY AND R ESTRICTIVENESS
We empirically verify that single-factor consistency or restrictiveness can be achieved with the supervision methods of interest. Note there are two special cases of match pairing: one where S i is the only factor that is shared between x and x (cid:48) and one where S i is the only factor that is changed. We distinguish these two conditions as share pairing and change pairing , respectively.
Theorem
[Table]
0.92 0.91 0.87 0.91 0.910 1 2 3 4 5 Rank Pairing at Factor i     5 N o r m a li z e d C o n s i s t
e n c y S
c
o r e a t F a c t o r i 0.99 0.34 0.36 0.13 0.21 0.
0 1 2 3 4 5 Change Pair Intersection at Factor i     5 N o r m a li z e d R e s t
r
i c t i v e n e ss S
c
o r e a t F a c t o r i 1 0.84 0.83 0.8 0.79 0.
shows that restricted labeling, share pairing, and rank pairing of the i th factor are each sufﬁcient supervision strategies for guaranteeing consistency on S i .
Change pairing at S i is equivalent to share pairing at S \ i ; the complement rule C ( I ) ⇐⇒ R
( \ I ) allows us to conclude that change pairing guarantees restrictiveness.
The ﬁrst four heatmaps in Figure 3 show the results for restricted labeling, share pairing, change pairing, and rank pairing.
The numbers shown in the heatmap are the normalized consistency and restrictiveness scores .
We deﬁne the normalized consistency score as
[Equation]
This score is bounded on the interval [0 , 1] (a consequence of Lemma 1) and is maximal when C ( I ; p ∗ , g ∗ , e ) is satisﬁed.
This normalization procedure is similar in spirit to that used in Suter et al.
(2018)’s Interventional Robustness Score.
The normalized restrictiveness score ˜ r can be analogously deﬁned.
In practice, we estimate this score via Monte Carlo estimation.
The ﬁnal heatmap in Figure 3 demonstrates the calculus of intersection.
In practice, it may be easier to acquire paired data where multiple factors change simultaneously.
If we have access to two kinds of datasets, one where S I are changed and one where S J are changed, our calculus predicts that training on both datasets will guarantee restrictiveness on S
I ∩ J .
The ﬁnal heatmap shows six such intersection settings and measures the normalized restrictiveness score; in all but one setting, the results are consistent with our theory.
We show in Figure 7 that this inconsistency is attributable to the failure of the GAN to distribution-match due to sensitivity to a speciﬁc hyperparameter.
6.2. C ONSISTENCY VERSUS R ESTRICTIVENESS
[Equation]
[Table]
N o r m a li z e d R e s t r
i c t i v e n e ss S
c
{+o r e f o r F a c t o r0.0 0.2 0.4 0.
 0.8 1.0 Normalized Consistency Score for Factor 5 0. N o r m a li z e d R e s t r i c t i v e n e ss S+}
c
o r e f o r F a c t o r
We now determine the extent to which consistency and restrictiveness are correlated in practice.
In
Figure 4, we collected all 864 Shapes3D models that we trained in Section 6.2.1 and measured the consistency and restrictiveness of each model on each factor, providing both the correlation plot and scatterplots of ˜ c
( i ) versus
˜ r
( i ) .
Since the models trained in Section 6.2.1 only ever targeted the consistency or restrictiveness of a single factor, and since our calculus demonstrates that consistency and restrictiveness do not imply each other, one might a priori expect to ﬁnd no correlation in
[-o r e f o r F a c t o r0.0 0.2 0.4 0.
-]
Figure 4.
Our results show that the correlation is actually quite strong.
Since this correlation is not guaranteed by our choice of weak supervision, it is necessarily a consequence of model inductive bias.
We believe this correlation between consistency and restrictiveness to have been a general source of confusion in the disentanglement literature, causing many to either observe or believe that restricted labeling or share pairing on S i (which only guarantees consistency) is sufﬁcient for disentangling S i (Kingma et al., 2014; Chen & Batmanghelich, 2019; Gabbay & Hoshen, 2019;
Narayanaswamy et al., 2017).
It remains an open question why consistency and restrictiveness are so strongly correlated when training existing models on real-world data.
6.2. F ULL D ISENTANGLEMENT
[Figure]
If we have access to share / change / rank-pairing data for each factor, our calculus states that it is possible to guarantee full disentanglement.
We trained our generative model on either complete share pairing, complete change pairing, or complete rank pairing, and measured disentanglement performance via the discretized mutual information gap (Chen et al., 2018a;
Locatello et al., 2019).
As negative and positive controls, we also show the performance of an unsupervised GAN and a fully-supervised GAN where the latents are ﬁxed to the ground truth factors of variation.
Our results in Figure 5 empirically verify that combining single-factor weak supervision datasets leads to consistently high disentanglement scores.
C ONCLUSION
In this work, we construct a theoretical framework to rigorously analyze the disentanglement guarantees of weak supervision algorithms.
Our paper clariﬁes several important concepts, such as consistency and restrictiveness, that have been hitherto confused or overlooked in the existing literature, and provides a formalism that precisely distinguishes when disentanglement arises from supervision versus model inductive bias.
Through our theory and a comprehensive set of experiments, we demonstrated the conditions under which various supervision strategies guarantee disentanglement.
Our work establishes several promising directions for future research.
First, we hope that our formalism and experiments inspire greater theoretical and scientiﬁc scrutiny of the inductive biases present in existing models.
Second, we encourage the search for other learning algorithms (besides distribution-matching) that may have theoretical guarantees when paired with the right form of supervision.
Finally, we hope that our framework enables the theoretical analysis of other promising weak supervision methods. A PPENDIX
Our appendix consists of nine sections.
We provide a brief summary of each section below.
Appendix A: We elaborate on the connections between existing deﬁnitions of disentanglement and our deﬁnitions of consistency / restrictiveness / disentanglement. In particular, we highlight three notable properties of our deﬁnitions not present in many existing deﬁnitions. Appendix B: We evaluate our consistency and restrictiveness metrics on the 10800 models in the disentanglement lib , and identify models where consistency and restrictiveness are not correlated. Appendix C: We adapt our deﬁnitions to be able to handle nuisance variables.
We do so through a simple modiﬁcation of the deﬁnition of restrictiveness.
Appendix D: We show several additional single-factor experiments.
We ﬁrst address one of the results in the main text that is not consistent with our theory, and explain why it can be attributed to hyperparameter sensitivity.
We next unwrap the heatmaps into more informative boxplots.
Appendix E: We provide an additional suite of consistency versus restrictiveness experiments by comparing the effects of training with share pairing (which guarantees consistency), change pairing (which guarantees restrictiveness), and both .
Appendix F: We provide full disentanglement results on all ﬁve datasets as measured according to six different metrics of disentanglement found in the literature.
Appendix G: We show visualizations of a weakly supervised generative model trained to achieve full disentanglement.
Appendix H:
We describe the set of hyperparameter conﬁgurations used in all our experiments.
Appendix I: We provide the complete set of assumptions and proofs for our theoretical framework.
A C ONNECTIONS TO E XISTING D EFINITIONS
Numerous deﬁnitions of disentanglement are present in the literature (Higgins et al., 2017; 2018;
Kim & Mnih, 2018; Suter et al., 2018; Ridgeway & Mozer, 2018; Eastwood & Williams, 2018; Chen et al., 2018a).
We mostly defer to the terminology suggested by Ridgeway & Mozer (2018), which decomposes disentanglement into modularity , compactness , and explicitness .
Modularity means a latent code
Z i is predictive of at most one factor of variation S j .
Compactness means a factor of variation S i is predicted by at most one latent code Z j .
And explicitness means a factor of variation
S j is predicted by the latent codes via a simple transformation (e.g. linear).
Similar to Eastwood &
Williams (2018); Higgins et al.
, we suggest a further decomposition of Ridgeway & Mozer (2018)’s explicitness into latent code informativeness and latent code simplicity .
In this paper, we omit latent code simplicity from consideration.
Since informativeness of the latent code is already enforced by our requirement that g ( Z ) is equal in distribution to g ∗ ( S ) (see Proposition 6), we focus on comparing our proposed concepts of consistency and restrictiveness to modularity and compactness.
We make note of three important distinctions.
Restrictiveness is not synonymous with either modularity or compactness .
In Figure 2c, it is evident the factor of variation size is not predictable
any individual Z i (conversely, Z 1 is not predictable from any individual factor S i ).
As such, Z 1 is neither a modular nor compact representation of size , despite being restricted to size .
To our knowledge, no existing quantitative deﬁnition of disentanglement (or its decomposition) speciﬁcally measures restrictiveness.
Consistency and restrictiveness are invariant to statistically dependent factors of variation .
Many existing deﬁnitions of disentanglement are instantiated by measuring the mutual information between Z and S .
For example, Ridgeway & Mozer (2018) deﬁnes that a latent code Z i to be “ideally modular” if it has high mutual information with a single factor S j and zero mutual information with all other factors S \ j .
This presents a issue when the true factors of variation themselves are statistically dependent; even if Z 1 = S 1 , the latent code Z 1 would violate modularity if S 1 itself has positive mutual information with S 2 .
Consistency and restrictiveness circumvent this issue by relying on conditional resampling.
Consistency, for example, only measures the extent to which S
I is invariant to resampling of Z \ I when conditioned on Z I and is thus achieved as long as s I is a function of only z I —irrespective of whether s I and s \
I are statistically dependent.
In this regard, our deﬁnitions draw inspiration from Suter et al.
(2018)’s intervention-based deﬁnition but replaces the need for counterfactual reasoning with the simpler conditional sampling.
Consistency and restrictiveness arise in weak supervision guarantees .
One of our goals is to propose deﬁnitions that are amenable to theoretical analysis.
As we can see in Section 4, consistency and restrictiveness serve as the core primitive concepts that we use to describe disentanglement guarantees conferred by various forms of weak supervision.
B E VALUATING C ONSISTENCY AND R ESTRICTIVENESS ON D ISENTANGLEMENT -L IB M ODELS
To better understand the empirical relationship between consistency and restrictiveness, we calculated the normalized consistency and restrictiveness scores on the suite of 12800 models from disentanglement lib for each ground-truth factor. By using the normalized consistency and restrictiveness scores as probes, we were able to identify models that achieve high consistency but low restrictiveness (and vice versa). In Fig.
6, we highlight two models that are either consistent or restrictive for object color on the Shapes3D dataset.
[Figure]
In Fig.
6a, we can see that this factor consistenly represents object color, i.e. each column of images has the same object color, but as we move along rows we see that other factors change as well, e.g. object type, thus this factor is not restricted to object color.
In Fig.
6b, we see that varying the factor along each row results in changes to object color but to no other attributes.
However if we look across columns, we see that the representation of color changes depending on the setting of other factors, thus this factor is not consistent for object color.
C H ANDLING NUISANCE VARIABLES
Our theoretical framework can handle nuisance variables , i.e., variables we cannot measure or perform weak supervision on.
It may be impossible to label, or provide match-pairing on that factor of variation.
For example, while many features of an image are measurable (such as brightness and coloration), we may not be able to measure certain factors of variation or generate data pairs where these factors are kept constant.
In this case, we can let one additional variable η act as nuisance variable that captures all additional sources of variation / stochasticity.
Formally, suppose the full set of true factors is S ∪ { η } ∈ R n +1 .
We deﬁne η -consistency C η ( I ) =
C ( I ) and η -restrictiveness R η ( I ) =
R ( I ∪ { η } ) .
This captures our intuition that, with nuisance variable, for consistency, we still want changes to Z \ I ∪ { η } to not modify S I ; for restrictiveness, we want changes to Z I ∪ { η } to only modify S I ∪ { η } .
We deﬁne η -disentanglement as D η ( I ) =
C η ( I ) ∧ R η ( I ) .
All of our calculus still holds where we substitute C η ( I ) , R η ( I ) , D η ( I ) for C ( I ) , R ( I ) , D ( I ) ; we prove one of the new full disentanglement rule as an illustration:
Proposition 1.
[Equation]
Proof.
On the one hand, (cid:86) ni =1 C η ( i ) ⇐⇒ (cid:86) ni
=1 C ( i ) =
⇒ C
(1 : n ) =
⇒ R ( η ) .
On the other hand, (cid:86) ni =1 C ( i ) =
⇒ (cid:86) ni =1 D ( i ) =
⇒ (cid:86) ni =1 R ( i ) .
Therefore LHS =
⇒ ∀ i ∈ [ n ] , R ( i ) ∧
R ( η ) =
⇒ R η ( i ) .
The reverse direction is trivial.
In (Locatello et al., 2019), the “instance” factor in SmallNORB and the background image factor in Scream-dSprites are treated as nuisance variables.
By Proposition 1, as long as we perform weak supervision on all of the non-nuisance variables (via sharing-pairing, say) to guarantee their consistency with respect to the corresponding true factor of variation, we still have guaranteed full disentanglement despite the existence of nuisance variable and the fact that we cannot measure or perform weak supervision on nuisance variable.
[Table]
0.92 0.85 0.83 0.9 0.90 1 2 3 4 5 Rank Pairing at Factor i     5 N o r m a li z e d C o n s i s t
e n c y S
c
o r e a t F a c t o r i 0.99 0.2 0.39 0.19 0.13 0.
0 1 2 3 4 5 Change Pair Intersection at Factor i     5 N o r m a li z e d R e s t
r
i c t i v e n e ss S
c
o r e a t F a c t o r i 1 0.98 0.81 0.8 0.8 0.
N o e c
o r e f o r F a c t o r0 1 2 3 4 5 Restricted Labeling at Factor i 0.
N o r m a li z e d C o n s i s t
e n c y S
c
o r e f o r F a c t o r0 1 2 3 4 5 Restricted Labeling at Factor i 0.
N o r m a li z e d C o n s i s t
e n c y S
c
o r e f o r F a c t o r0 1 2 3 4 5 Restricted Labeling at Factor i 0.
N o r m a li z e d C o n s i s t
e n c y S
c
o r e f o r F a c t o r0 1 2 3 4 5 Restricted Labeling at Factor i 0.
N o r m a li z e d C o n s i s t
e n c y S
c
o r e f o r F a c t o r0 1 2 3 4 5 Restricted Labeling at Factor i 0.
N o r m a li z e d C o n s i s t
e n c y S
c
o r e f o r F a c t o r
[Table]
[Table] [Table]
[Table] [Table]
[Table]
t r
[Table]
i
[Table]
r
[Table]
r
[Table]
t r c t i
[Table]
e
[Table]
o r
[Table] [Table]
[Table] [Table]
[Table] [Table]
N o r m a li z e d R e s t r
i c t i v e n e ss S
c
o r e f o r F a c t o r
[Table]
N o r m a li z e d R e s t r
i c t i v e n e ss S
c
o r e f o r F a c t o r
[Table] [Table]
R
[Table]
t r i c t i v e n e ss S
c
o r e f o r F a c t o
[Table]
[Table] [Table]
N o r m a li z e d R e s t r
i c t i v e n e ss S c o r e f o r F a c t o r
[Table]
SmallNORB
[Table] [Table]
N o r m a li z e d R e s t r
i c t i v e n e ss S
c
o r e f o r F a c t o r   3 Supervision via Change Pairing at Factor
i 0.3 0.
N o r m a li z e d R e s t r
i c t i v e n e ss S
c
o r e f o r F a c t o r  2 Supervision via Change Pairing at Factor i 0.
N o r m a li z e d R e s t r
i c t i v e n e ss S
c
o r e f o r F a c t o r
Cars3D  2 Supervision via Change Pairing at Factor i 0.
N o r m a li z e d R e s t r
i c t i v e n e ss S
c
o r e f o r F a c t o r  2 Supervision via Change Pairing at Factor i 0.
N o r m a li z e d R e s t r
i c t i v e n e ss S
c
o r e f o r F a c t o r
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
n s i s t e n c y S c
r e f r
F a c t r
[Table]
n s i s t e n c y S c
r e f r
F a c t r
[Table]
e d C n s i s t
[Table]
f r
[Table]
t r
[Table]
n
[Table]
f
[Table]
f r
F a c t r    4 Supervisi n via Share Pairing at Fact r i −0.
−0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r    4 Supervisi n via Share Pairing at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r    4 Supervisi n via
[Table]
Pairing at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r   3 Supervisi n via
[Table]
at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r
SmallNORB   3 Supervisi n via Share Pairing at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r   3 Supervisi n via Share Pairing at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r
[Table]
n
[Table]
at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r  2 Supervisi n via Share Pairing at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r
Cars3D  2 Supervisi n via Share Pairing at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r  2 Supervisi n via Share Pairing at Fact r i 0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
e c y S
c
o r e f o r F a c t o rdSprites    4 Ra k Pairi g at Factor i 0.
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a c t o r    4 Ra k Pairi g at Factor i 0.
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a c t o r
[Table]
[Table]
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a c t o r
[Table]
[Table] [Table]
[Table] [Table]
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a c t o r
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
e f
[Table]
r F
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table]
N
[Table]
r m
[Table]
li
[Table]
e d C
o s i s t e c y S
c
o r e f o r F a c t o r
SmallNORB
[Table]
Ra k Pairi g at Factor
i 0.
[Table]
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a c t o r
[Table]
Ra k Pairi g at Factor
i 0.
[Table] [Table]
[Table]
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a
[Table]
t o r
[Table] [Table]
[Table] [Table]
N o r m a
[Table]
z e d C
o s i s t
e c y S c
o r e f o r F a c t o r  2 Ra k Pairi g at Factor i 0.
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a c t o r
Cars3D  2 Ra k Pairi g at Factor i 0.
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a c t o r  2 Ra k Pairi g at Factor
i 0.1 0.
N o r m a li z e d C
o s i s t
e c y S c
o r e f o r F a c t o r
E C ONSISTENCY VERSUS R ESTRICTIVENESS
[Figure]
[Table] [Table] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
Restrictiveness Sc re f r Fact r 0 −1.
−0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r
Cars3D ChangeShareB th 0.75 0.80 0.85 0.90 0.95 1.00 N rmalized Restrictiveness Sc re f r Fact r0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r
ChangeShareB th 0.82 0.84 0.86 0.88 0.90 0.92 0.94 0.96 N rmalized Restrictiveness Sc re f r Fact r0.
N r m a li z e d C n s i s t e n c y S c
r e f r
F a c t r
ChangeShareB th
[Table] [Table]
[Table]
[Figure]
[Table]
None Share Change Rank Full-Label 0. None Share Change Rank Full-Label 0.
None Share Change Rank Full-Label −0.
Scream-dSprites
None Share Change Rank Full-Label 0.
None Share Change Rank Full-Label −0.
None
[Figure]
Rank Full-Label 0.
[Figure] [Figure]
None Share Change Rank Full-Label 0.
SmallNORB
None Share
[Figure]
Rank Full-Label 0.
None Share
[Figure]
Rank Full-Label 0.
None
[Figure]
-Label 0.
None
[Figure]
Full-Label 0.
Cars3D
None
[Figure]
-Label 0.
[Figure]
-Label 0.
} in the ﬁrst column shows that distribution matching to all six change / share pairing datasets is particularly challenging for the models when trained on certain hyperparameter choices.
However, since consistency and restrictiveness can be measured in weakly supervised settings, it sufﬁces to use these metrics for hyperparameter selection.
We see in Figure 16 and Appendix G that using consistency and restrictiveness for hyperparameter selection serves as a viable weakly-supervised surrogate for existing fully-supervised disentanglement metrics.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
None Share Change Rank Full-Label 0. None Share Change Rank Full-Label 0.
None Share
[Figure]
Rank Full-Label 0.
Scream-dSprites
None Share Change Rank Full-Label 0.
None Share Change Rank Full-Label 0. None Share Change Rank Full-Label 0.
None
[Figure]
Full-Label 0.
[Figure]
-Label 0.
SmallNORB
None
[Figure]
Rank Full-Label 0.
None Share
[Figure]
Rank Full-Label 0.
[Figure]
-Label 0.
None Share
[Figure]
Rank Full-Label 0.775 0.
Cars3D
None
[Figure]
Full-Label 0.
None Share Change Rank Full-Label 0.
-labeled GAN (purple), as measured by normalized restrictiveness score of each factor (rows) across multiple datasets (columns).
Since restrictiveness and consistency are complementary, we see that the anomalies in Figure 14 are reﬂected in the complementary factors in this ﬁgure.
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
l I n f r m a t
i o n
[Table]
p
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
G F ULL D ISENTANGLEMENT V ISUALIZATIONS
As a demonstration of the weakly-supervised generative models, we visualize our best-performing match-pairing generative models (as selected according to the normalized consistency score averaged across all the factors).
Recall from Figures 2a to 2c that, to visually check for consistency and restrictiveness, it is important that we not only ablate a single factor (across the column), but also show that the factor stays consistent (down the row).
Each block of 3 × 12 images in Figuresto 21 checks for disentanglement of the corresponding factor.
Each row is constructed by random sampling of Z \ i and then ablating Z i .
H H YPERPARAMETERS
[Table]
[Table] [Table]
[Table]
Body Applied Separately to x and x (cid:48)
[Table] [Table]
For all models, we use the Adam optimizer with β 1 = 0 .
5 , β 2 = 0 .
and set the generator learning rate to 1 × 10 − 3 .
We use a batch size of 64 and set the leaky ReLU negative slope to 0 .
To demonstrate some degree of robustness to hyperparameter choices, we considered ﬁve different ablations:
Width multiplier on the discriminator network ( { 1 , 2 } ) 2.
Whether to add an extra fully-connected layer to the discriminator ( { True , False } ).
Whether to add a bias term to the head ( { True , False } ).
Whether to use two-time scale learning rate by setting encoder+discriminator learning rate multipler to ( { 1 , 2 } ).
Whether to use the default PyTorch or Keras initialization scheme in all models.
As such, each of our experimental setting trains a total of 32 distinct models.
The only exception is the intersection experiments where we ﬁxed the width multiplier to 1 .
To give a sense of the scale of our experimental setup, note that the 864 models in Figure 4 originate as follows:
32 hyperparameter conditions × 6 restricted labeling conditions.
32 hyperparameter conditions × 6 match pairing conditions.
32 hyperparameter conditions × 6 share pairing conditions. 32 hyperparameter conditions × 6 rank pairing conditions.
16 hyperparameter conditions × 6 intersection conditions.
I P ROOFS I. A SSUMPTIONS ON H
Assumption 1.
Let D ⊆
[ n ] indexes discrete random variables S D .
Assume that the remaining random variables S C
=
S \ D have probability density function p ( s C |
s D ) for any set of values s D where p ( S D = s D ) > 0 .
Assumption 2.
Without loss of generality, suppose S 1: n = [ S C , S D ] is ordered by concatenating the continuous variables with the discrete variables.
Let B ( s D ) =
[int(supp( p ( s C | s D ))) , s D ] denote the interior of the support of the continuous conditional distribution of S C concatenated with its conditioning variable s D drawn from S D .
With a slight abuse of notation, let B ( S ) =
(cid:83) s D : p ( s D > 0) B ( s D ) .
We assume B ( S ) is zig-zag connected , i.e., for any I, J ⊆
[ n ] , for any two points s 1: n , s (cid:48) 1: n ∈ B ( S ) that only differ in coordinates in I ∪ J , there exists a path { s t 1: n } t =0: T contained in B ( S ) such that
[Equation]
Intuitively, this assumption allows transition from s 1: n to s (cid:48) 1: n via a series of modiﬁcations that are only in I or only in J .
Note that zig-zag connectedness is necessary for restrictiveness union (Proposition 3) and consistency intersection (Proposition 4).
Fig.
22 gives examples where restrictiveness union is not satisﬁed when zig-zag connectedness is violated.
Assumption 3.
For arbitrary coordinate j ∈ [ m ] of g that maps to a continuous variable X j , we assume that g j ( s ) is continuous at s , ∀ s ∈ B ( S ) ; For arbitrary coordinate j ∈ [ m ] of g that maps to a discrete variable X j , ∀ s D where p ( s D ) > 0 , we assume that g j ( s ) is constant over each connected component of int(supp( p ( s C | s D )) .
Deﬁne B ( X ) analogously to B ( S ) .
Symmetrically, for arbitrary coordinate i ∈ [ n ] of e that maps to a continuous variable S i , we assume that e i ( x ) is continuous at x , ∀ x ∈ B ( X ) ; For arbitrary coordinate i ∈ [ n ] of e that maps to a discrete S i , ∀ x D where p ( x D ) > 0 , we assume that e i ( x ) is constant over each connected component of int(supp( p ( x C | x D )) .
Assumption 4.
Assume that every factor of variation is recoverable from the observation X .
Formally, ( p, g, e ) satisﬁes the following property
[Equation]
I. C ALCULUS OF D ISENTANGLEMENT I.2. E XPECTED -N ORM R EDUCTION L EMMA
Lemma 1.
Let x, y be two random variables with distribution p , f ( x, y ) be arbitrary function.
Then
E
[Equation]
Proof.
Assume w.l.o.g that E ( x,y ) ∼ p ( x,y ) f
( x, y ) = 0 .
[Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
22) =
RHS.
[Equation]
I.2. C ONSISTENCY U NION
Let L = I ∩
J , K = \
( I ∪ J ) , M = I − L , N = J − L .
Proposition 2.
C ( I ) ∧
C
( J ) =
⇒ C
( I ∪ J ) .
Proof.
[Equation]
For any ﬁxed value of z M , z L ,
[Equation]
by plugging in x = z N , y =
z K into Lemma 1.
Therefore
[Equation]
Similarly we have
[Equation]
As I ∩ N = ∅ , I ∪ N
=
I ∪ J , adding the above two equations gives us C
( I ∪ J ) .
I.2.
R ESTRICTIVENESS U NION
[Equation]
[Equation]
Similarly deﬁne index sets L, K, M, N .
Proposition 3.
Under assumptions speciﬁed in Appendix I.1, R ( I ) ∧ R ( J ) =
⇒ R
( I ∪ J ) .
Proof.
Denote f = e ∗ K ◦ g .
We claim that
[Equation] [Equation]
We ﬁrst prove the backward direction: When we draw z \ I ∼ p ( z \ I ) , z I , z (cid:48)
I ∼ p ( z I | z \ I ) , let Edenote the event that ( z I , z \ I ) / ∈ B
( Z ) , and E 2 denote the event that ( z (cid:48)
I , z \ I ) / ∈ B ( Z ) .
Reorder the indices of ( z I , z \ I ) as ( z C , z D ) .
The probability that ( z I , z \ I ) / ∈ B
( Z ) (i.e., z C is on the boundary of B ( z D ) ) is 0.
Therefore Pr[ E 1 ] =
Pr[ E 2 ] = 0 .
Therefore Pr[ E 1 ∪ E 2 ] ≤ Pr[ E 1 ] + Pr[ E 2 ] = 0 , i.e., with probability 1, (cid:107) f
( z I , z \ I ) − f ( z (cid:48)
I , z \ I ) (cid:107) 2 = 0 .
Now we prove the forward direction:
Assume for the sake of contradiction that ∃ ( z I , z \ I ) , ( z (cid:48)
I , z \ I ) ∈
B ( Z ) such that f ( z I , z \ I ) <
f ( z (cid:48)
I , z \ I ) .
Denote U
=
I ∩ D , V = I ∩ C ,
W = \ I ∩ D , Q = \ I ∩ C .
We have f ( z U , z V , z W , z Q ) <
f ( z (cid:48)
U , z (cid:48) V , z W , z Q ) .
Since f is continuous (or constant) at ( z U , z V , z W , z Q ) in the interior of B ([ z U , z W ]) , and f is also continuous (or constant) at ( z (cid:48) U , z (cid:48)
V , z W , z Q ) in the interior of B ([ z (cid:48) U , z W ]) , we can draw open balls of radius r >around each point, i.e., B r ( z V , z Q ) ⊂ B
([ z U , z W ]) and B r ( z (cid:48) V , z Q ) ⊂ B ([ z (cid:48) U , z W ]) , where
[Equation]
When we draw z \ I ∼ p ( z \ I ) , z I , z (cid:48)
I ∼ p ( z I | z \ I ) , let C denote the event that ( z I , z \ I ) =
( z ∗ V , z U , z
[Equation]
Q , z W ) , ( z (cid:48)
I ,
[Equation]
) =
[Equation]
# Q , z W )
where ( z
[Equation]
, z
[Equation]
∈
[Equation]
z
[Equation]
, z
[Equation]
) and
[Equation]
Since both balls have positive volume
[Equation]
Pr[ C ] > 0 .
However
[Equation]
(cid:107) f
( z I , z \ I ) − f ( z
[Equation]
I , z \ I )
[Equation]
2 > 0 whenever event C happens [Equation] which contradicts R ( I ) .
Therefore ∀ ( z I , z \
[Equation]
) , ( z
[Equation] [Equation]
We have shown that
[Equation] [Equation]
Similarly
[Equation]
Let the zig-zag path between ( z M , z L , z N , z K ) and ( z (cid:48)
[Equation]
∈
B ( Z ) be { ( z tM , z tL , z tN , z K ) } Tt =0 .
Repeatedly applying the equivalent conditions of R ( I ) and R ( J ) gives us
[Equation]
I. C ONSISTENCY AND R ESTIVENESS I NTERSECTION
Proposition 4.
Under the same assumptions as restrictiveness union, C ( I ) ∧ C ( J ) =
⇒ C
( I ∩ J ) .
Proof.
[Equation] [Equation]
[Equation] [Equation]
Proposition 5.
R ( I ) ∧
R ( J ) =
⇒ R
( I ∩ J ) .
Proof is analogous to Proposition 4.
I. D ISTRIBUTION MATCHING GUARANTEES LATENT CODE INFORMATIVENESS
Proposition 6.
If ( p ∗ , g ∗ , e ∗ ) ∈ H , and ( p, g, e ) ∈ H , and g ∗ ( S ) d = g ( Z ) , then there exists a continuous function r such that
[Equation]
Proof.
We show that r = e ∗ ◦ g satisﬁes Proposition 6.
By Assumption 4,
[Equation]
By the same reasoning as in the proof of Proposition 3,
[Equation]
Let s ∼ p ( s ) .
We claim that Pr[ E 1 ] = 1 , where E 1 denote the event that ∃ z ∈ B ( Z ) such that g ∗ ( s ) =
g ( z ) .
Suppose to the contrary that there is a measure-non-zero set S ⊆ supp ( p ( s )) such that ∀ s ∈ S , no z ∈ B ( Z ) satisﬁes g ∗ ( s ) =
g ( z ) .
Let X
= { g ( s ) : s ∈ S} .
As g ∗ ( S ) d = g ( Z ) ,
Pr s [ g ∗ ( s ) ∈ X ] =
Pr z [ g ( z ) ∈ X ] > 0 .
Therefore ∃Z ⊆ supp ( p ( z ))
−B ( Z ) such that X ⊆ { g ( z ) :
z ∈ Z ) } .
But supp ( p ( z ))
− B ( Z ) has measure 0.
Contradiction.
When we draw s , let E 2 denote the event that s ∈ B ( S ) .
Pr[ E 2 ] = 1 , so
Pr[ E 1 ∧ E 2 ] = 1 .
When
E 1 ∧ E 2 happens, e ∗ ◦ g ◦
e ◦ g ∗ ( s ) =
e ∗ ◦ g ◦ e ◦ g
( z ) =
e ∗ ◦ g
( z ) =
e ∗ ◦ g ∗ ( s )
= s .
Therefore
E
[Equation]
I. W EAK S UPERVISION G UARANTEE
Theorem 1.
Given any oracle ( p ∗ ( s ) , g ∗ , e ∗ ) ∈ H , consider the distribution-matching algorithm A that selects a model ( p ( z )
, g, e ) ∈ H such that: ( g ∗ ( S ) , S I ) d = ( g ( Z ) , Z I ) ( Restricted Labeling ); or 2. (cid:16) g ∗ ( S I , S \ I ) , g ∗ ( S I , S (cid:48)\ I ) (cid:17) d = (cid:16) g ( Z I , Z \ I ) , g ( Z I , Z
(cid:48)\ I ) (cid:17) ( Match Pairing ); or 3.
( g ∗ ( S ) , g ∗ ( S (cid:48) ) , 1 { S I ≤ S (cid:48) I } ) d = ( g ( Z ) , g ( Z (cid:48) ) , 1 { Z I ≤ Z (cid:48) I } ) ( Rank Pairing
Then (p, g ) satisﬁes C ( I ; p, g, e ∗ ) and e satisﬁes C
( I ; p ∗ , g ∗ , e ) .
Proof.
We prove the three cases separately:
Since ( x d , s I )
d = ( x g , z I ) , consider the measurable function [Equation]
[Equation]
We have
[Equation]
[Equation]
By the same reasoning as in the proof of Proposition 3,
[Equation]
Therefore
[Equation]
i
e., g satisﬁes C ( I ; p, g, e ∗ ) .
By symmetry, e satisﬁes C ( I ; p ∗ , g ∗ , e ) .
[Equation]
[Equation]
So g satisﬁes C ( I ; p, g, e ∗ ) .
By symmetry, e satisﬁes C ( I ; p ∗ , g ∗ , e ) .
Let I =
{ i } , f = e ∗
I ◦ g .
Distribution matching implies that, with probability 1 over random draws of Z, Z (cid:48) , the following event P happens:
[Equation]
Let W = \
I ∩ D , Q = \ I ∩ \ D .
We showed in the proof of Proposition 3 that
[Equation]
We prove by contradiction.
Suppose ∃ ( z I , z W , z Q ) ,
( z I , z (cid:48) W , z (cid:48) Q )
∈ B ( Z ) such that f ( z I , z W , z Q ) <
f
( z I , z (cid:48) W , z (cid:48) Q ) .
(a) Case 1: Z I is discrete.
Since f is constant both at ( z I , z W , z Q ) in the interior of B ([ z I , z W ]) , and at ( z I , z (cid:48) W , z (cid:48) Q ) in the interior of B ([ z I , z (cid:48) W ]) , we can draw open balls of radius r >around each point, i.e., B r ( z Q ) ⊂ B ([ z I , z W ]) and B r ( z (cid:48) Q )
⊂ B ([ z I , z (cid:48) W ]) , where
[Equation]
When we draw z, z (cid:48) ∼ p ( z ) , let C denote the event that this speciﬁc value of z I is picked for both z, z (cid:48) , and we picked z \
I ∈ B r
( z (cid:48) Q ) ,
z (cid:48)\ I ∈ B r ( z Q ) .
Since both balls have positive volume, Pr[ C ] > 0 .
However, P does not happen whenever event C happens, since z I =
z (cid:48)
I but f ( z ) >
f ( z (cid:48) ) , which contradicts Pr[ P ] = 1 .
(b) Case 2: z I is continuous.
Similar to case 1, we can draw open balls of radius r > 0 around each point, i.e.,
B r ( z I , z Q ) ⊂
B ( z W ) and B r ( z I , z (cid:48) Q ) ⊂
B ( z (cid:48) W ) , where ∀ ( z ∗ I , z
[Equation]
Q )
[Equation]
B r
[Equation]
[Equation]
[Equation]
[Equation]
Let H
= { ( z ∗ I , z ∗ Q )
∈ B r ( z I , z Q ) :
z ∗ I > =
z I } , H
= { ( z ∆ I , z ∆ Q )
∈ B r ( z I , z (cid:48) Q ) :
z ∆ I < =
z I } .
When we draw z, z (cid:48) ∼ p ( z ) , let C denote the event that we picked z (cid:48)
∈ H 1 × { z W } , z ∈ H 2 × { z (cid:48) W } .
Since H 1 , H 2 have positive volume, Pr[ C ] > 0 .
However, P does not happen whenever event C happens, since z I < =
z (cid:48)
I but f ( z ) >
f ( z (cid:48) ) , which contradicts Pr[ P ] = 1 .
Therefore we showed
[Equation]
i.e., g satisﬁes C ( I ; p, g, e ∗ ) . By symmetry, e satisﬁes C ( I ; p ∗ , g ∗ , e ) . I. W EAK S UPERVISION I MPOSSIBILITY R ESULT Theorem 2.
Weak supervision via restricted labeling, match pairing, or ranking on s I is not sufﬁcient for learning a generative model whose latent code
Z I is restricted to S I . Proof. We construct the following counterexample. Let n = m =
and I = { 1 } .
The data generation process is s 1 ∼ unif([0 , 2 π )) , ( s 2 , s 3 ) ∼ unif( { ( x, y ) : x 2 + y 2 ≤ 1 } ) , g ∗ ( s ) =
[ s 1 , s 2 , s 3 ] .
Consider a generator z 1 ∼ unif([0 , 2 π )) , ( s 2 , s 3 ) ∼ unif( { ( x, y ) : x 2 + y 2 ≤ 1 } ) , g ( z ) =
[ z 1 , cos ( z 1 ) z 2 − sin ( z 1 ) z 3 , sin ( z 1 ) z 2 + cos ( z 1 ) z 3 ] .
Then ( x d , s I )
d = ( x g , z I )
but R ( I ; p, g, e ∗ ) and R ( I ; p ∗ , g ∗ , e ) does not hold.
The same counterexample is applicable for match pairing and rank pairing.