D ISENTANGLEMENT A NALYSIS WITH P ARTIAL I NFORMATION D ECOMPOSITION
A BSTRACT
We propose a framework to analyze how multivariate representations disentangle ground-truth generative factors.
A quantitative analysis of disentanglement has been based on metrics designed to compare how one variable explains each generative factor.
Current metrics, however, may fail to detect entanglement that involves more than two variables, e.g., representations that duplicate and rotate generative factors in high dimensional spaces.
In this work, we establish a framework to analyze information sharing in a multivariate representation with Partial Information Decomposition and propose a new disentanglement metric.
This framework enables us to understand disentanglement in terms of uniqueness , redundancy , and synergy .
We develop an experimental protocol to assess how increasingly entangled representations are evaluated with each metric and conﬁrm that the proposed metric correctly responds to entanglement.
Through experiments on variational autoencoders, we ﬁnd that models with similar disentanglement scores have a variety of characteristics in entanglement, for each of which a distinct strategy may be required to obtain a disentangled representation
I NTRODUCTION
Disentanglement is a guiding principle for designing a learned representation separable into parts that individually capture the underlying factors of variation.
The concept is originally concerned as an inductive bias towards obtaining representations aligned with the underlying factors of variation in data (Bengio et al., 2013) and has been applied to controlling otherwise unstructured representations of data from several domains, e.g., images (Karras et al., 2019;
Esser et al., 2019), text (Hu et al., 2017), and audio (Hsu et al., 2019) to name just a few.
While the concept is appealing, deﬁning disentanglement is not clear.
After Higgins et al.
(2017), generative learning methods with regularized total correlation have been proposed (Kim & Mnih, 2018; Chen et al., 2018); however, it is still not clear if independence of latent variables is essential for better disentanglement (Higgins et al., 2018).
Furthermore, it is not obvious to measure disentanglement given true generative factors.
Towards understanding disentanglement, it is crucial to deﬁne disentanglement metrics , for which several attempts have been made (Higgins et al., 2017;
Kim & Mnih, 2018; Chen et al., 2018; Eastwood & Williams, 2018; Do & Tran, 2020; Zaidi et al., 2020); however, there are still problems to be solved.
Current disentanglement metrics may fail to detect entanglement involving more than two variables.
In these metrics, one ﬁrst measures how each variable explains one generative factor and then compares or contrasts them among variables.
With such a procedure, we may overlook multiple variables conveying information of one generative factor.
For example, let z
=
( z 1 , z 2 ) be a representation consisting of two vectors, where each variable in z 1 disentangles a distinct generative factor and zis a rotation of z 1 not axis-aligned with the factors.
Since any dimension of z 2 alone may convey little information of one generative factor, these metrics do not detect that multiple variables encode one generative factor redundantly.
Although this is a simple example, this kind of information sharing may arise in learned representations as well, if not the variables are linearly correlated.
In this work, we present a disentanglement analysis framework aware of interactions among multiple variables.
Our key idea is to decompose the information of representation into entangled and disentangled components using Partial Information Decomposition (PID), which is a framework in modern information theory to analyze information sharing among multiple random variables (Williams
[Equation]
I ( u ; v 1 )
I
[Equation]
u
[Equation]
u
[Equation]
v
[Equation]
[Equation]
[Figure]
& Beer, 2010).
As illustrated in Figure 1, the mutual information
I ( u [Equation] between a random variable u and a pair of random variables ( v 1 , v 2 ) is decomposed into four nonnegative terms: unique information U ( u ; v 1 \ v 2 ) and U ( u ; v 2 \ v 1 ) 1 , redundant information
R ( u ; v 1 , v 2 ) , and complementary (or synergistic ) information C ( u ; v 1 , v 2 ) .
While these partial information terms have no agreed-upon concrete deﬁnitions yet (Bertschinger et al., 2014; Finn &
Lizier, 2018; Lizier et al., 2018; Finn & Lizier, 2020; Sigtermans, 2020), we can derive universal lower and upper bounds of the partial information terms only with well-deﬁned mutual information terms.
We apply the PID framework to representations learned from data by letting u be a generative factor and v 1 , v 2 be one and the remaining latent variables, respectively.
The unique information of a latent variable intuitively corresponds to the amount of disentangled information, while the redundant and complementary information correspond to different types of entangled information.
We can quantify disentanglement and multiple types of entanglement through the framework, which enriches our understanding on disentangled representations.
Our contributions are summarized as follows.
• PID-based disentanglement analysis framework : We propose a disentanglement analysis framework that captures interactions among multiple variables with PID.
With this framework, one can distinguish two different types of entanglement, namely redundancy and synergy, which provide insights on how a representation entangles generative factors.
• Tractable bounds of partial information terms : We derive lower and upper bounds of partial information terms.
We formulate a disentanglement metric, called U NI B OUND , using the lower bound of unique information.
We design entanglement attacks , which inject entanglement to a given disentangled representation, and conﬁrm through experiments using them that U NI B OUND effectively captures entanglement involving multiple variables.
• Detailed analyses of learned representations : We analyze representations obtained by variational autoencoders (VAEs).
We observe that U NI B OUND sometimes disagrees with other metrics, which indicates multi-variable interactions may dominate learned representations.
We also observe that different types of entanglement arise in models learned with different methods.
This observation provides us an insight that we may require distinct approaches to remove them for disentangled representation learning.
P ROBLEM F ORMULATION AND N OTATIONS
Let x be a random variable representing a data point, drawn uniformly from a dataset D .
Assume that the true generative factors y =
( y 1 , . . .
, y K ) (cid:62) are available for each data point; in other words, we can access the subset D ( y ) ⊂
D of the data points with any ﬁxed generative factors y .
Let z
=
( z 1 , . . .
, z L ) (cid:62) be a latent representation consisting of L random variables.
An inference model is provided as the conditional distribution p ( z | x ) .
Our goal is to evaluate how well the latent variables z disentangle each generative factor in y .
The inference model can integrate out the input as p ( ·| y )
=
E p ( x | y )
[ p ( ·| x )]
= 1 |D ( y ) | (cid:80) x ∈D ( y ) p ( ·| x ) ; therefore, we only use y and z in most of our discussions.
We denote the mutual information between random variables u and v by I ( u ; v ) =
E (cid:104) log p ( u,v ) p ( u ) p ( v ) (cid:105) , the entropy of u by H ( u ) = − E [log p ( u )] , and the conditional entropy of u given v by H ( u | v ) =
− E [log p ( u | v )] .
We denote a vector of zeros by 0 , a vector of ones by 1 , and an identity matrix by
I .
We denote by N ( µ , Σ ) the Gaussian distribution with mean µ and covariance Σ and denote its density function by N ( · ; µ , Σ ) .
R ELATED W ORK
The importance of representing data with multiple variables conveying distinct information has been recognized at least since the ’80s (Barlow, 1989; Barlow et al., 1989; Schmidhuber, 1992).
The minimum entropy coding principle (Watanabe, 1981), which aims at representing data by random variables z with the sum of minimum marginal entropies (cid:80) (cid:96) H
( z (cid:96) ) , is found to be useful for unsupervised learning to remove the inherent redundancy in the sensory stimuli.
The resulting representation minimizes the total correlation and is called factorial coding .
Recent advancements in disentangled representation learning based on VAEs (Kingma & Welling, 2014) are guided by the same principle as minimum entropy coding (Kim & Mnih, 2018;
Chen et al., 2018; Gao et al., 2019).
Understanding better representations, which is tackled from the coding side as above, is also approached from the generative perspective.
It is often expected that data are generated from generative factors through a process that entangles them into high dimensional sensory space (DiCarlo &
Cox, 2007).
As generative factors are useful as the basis of downstream learning tasks, obtaining disentangled representations from data is a hot topic of representation learning (Bengio et al., 2013).
Towards learning disentangled representations, it is arguably important to quantitatively measure disentanglement.
In that regard, Higgins et al.
(2017) established a standard evaluation procedure using controlled datasets with balanced and fully-annotated ground-truth factors.
A variety of metrics have then been proposed on the basis of the procedure.
Among them, Higgins et al.
(2017) and
Kim & Mnih (2018) propose metrics based on the deviation of each latent variable conditioned by a generative factor.
In contrast, Mutual Information Gap (MIG) (Chen et al., 2018) and its variants (Do & Tran, 2020; Zaidi et al., 2020) are based on mutual information between a latent variable and a generative factor.
We extend the latter direction, considering multi-variable interactions.
Barlow (1989) discussed redundancy by comparing the population and the individual variables by their entropies, i.e., total correlation.
It is though less trivial to measure redundancy as an information quantity.
The PID framework (Williams & Beer, 2010) provides an approach to understanding redundancy among multiple random variables as a constituent of mutual information.
The framework provides some desirable relationships between decomposed information terms, while it leaves some degrees of freedom to determine all of them, for which several deﬁnitions have been proposed (Williams & Beer, 2010; Bertschinger et al., 2014; Finn & Lizier, 2018; 2020; Sigtermans, 2020).
The PID framework has been applied to machine learning models.
For example, Tax et al.
(2017) measured the PID terms for restricted Boltzmann machines using the deﬁnition of Williams & Beer (2010).
Yu et al.
(2021) took an alternative route, similar to our approach, where they measured linear combinations of PID terms by corresponding linear combinations of mutual information terms.
These work aims at analyzing the learning dynamics of models in supervised settings.
In contrast, we use the PID framework for analyzing disentanglement in unsupervised representation learning.
P ARTIAL I NFORMATION D ECOMPOSITION FOR D ISENTANGLEMENT
In this section, we analyze the current metrics and introduce our framework.
In Section 3.1, we introduce PID of the system we concern.
In Section 3.2, we investigate the current metrics in terms of multi-variable interactions.
In Section 3.3 and 3.4, we construct our disentanglement metric with bounds for PID terms.
We provide a method of computing the bounds in Section 3.5.
3. P ARTIAL I NFORMATION D ECOMPOSITION
We tackle the problem of evaluating disentanglement of a latent representation z relative to the true generative factors y from an information-theoretic perspective.
Let us consider evaluating how one generative factor y k is captured by the latent representation z .
The information of y k captured by z is measured using mutual information
I ( y k ; z ) =
H ( z ) − H
( z | y k ) .
In a desirably disentangled representation, we expect one of the latent variables z (cid:96) to exclusively capture the information of the factor y k .
To evaluate a given representation, we are interested in understanding how the information is distributed between a latent variable z (cid:96) and the remaining representation z \ (cid:96) =
( z (cid:96) (cid:48) )
(cid:96) (cid:48) (cid:54) =
(cid:96) .
This is best described by the PID framework, where the mutual information is decomposed into the following four terms.
[Equation]
Here, the decomposed terms represent the following non-negative quantities.
• Redundant information R ( y k ; z (cid:96) , z \ (cid:96) ) is the information of y k held by both z (cid:96) and z \ (cid:96) .
• Unique information U ( y k ; z (cid:96) \ z \ (cid:96) ) is the information of y k held by z (cid:96) and not held by z \ (cid:96) .
The opposite term U ( y k ; z \ (cid:96) \ z (cid:96) ) is also deﬁned by exchanging the roles
of z (cid:96)
and
z \ (cid:96) .
• Complementary information (or synergistic information) C ( y k ; z (cid:96) , z \ (cid:96) ) is the information of y k held by z =
( z (cid:96) , z \ (cid:96) )
that is not held by either z (cid:96) or z \ (cid:96) alone.
The following identities, combined with Eq.1, partially characterize each term.
[Equation]
The decomposition of this system is illustrated in Figure 2a.
We expect disentangled representations to concentrate the information of y k to a single latent variable z (cid:96) , and to let the other variables z \ (cid:96) not convey the information in either unique, redundant, or synergistic ways.
This is understood in terms of PID as maximizing the unique information
U ( y k ; z (cid:96) \ z \ (cid:96) ) while minimizing the other parts of the decomposition.
The above formulation is incomplete as one degree of freedom remains to determine the four terms with the three equalities.
Instead of stepping into searching for suitable deﬁnitions of these terms, we build discussions applicable to any such deﬁnitions that fulﬁll the above incomplete requirements 2 .
3. U NDERSTANDING D ISENTANGLEMENT M ETRICS FROM I NTERACTION P ERSPECTIVE
Current disentanglement metrics are typically designed to measure how each latent variable captures a factor and compare it among latent variables, i.e., output a high score when only one latent variable captures the factor well.
For example, the BetaVAE metric (Higgins et al., 2017) is computed by estimating the mean absolute difference (MAD) of two i.i.d.
variables following p ( z (cid:96) | y k ) =
E p ( x | y k ) p ( z (cid:96) | x ) for each (cid:96) by
Monte-Carlo sampling and training a linear classiﬁer that predicts k from the noisy estimations of the differences.
The FactorVAE metric (Kim & Mnih, 2018) is computed similarly, except that
the
MAD is replaced with variance (following normalization by population), and a majority-vote classiﬁer is used to eliminate a failure mode of ignoring one of the factors and to avoid depending on hyperparameters.
These metrics have the same goal of ﬁnding a mapping between (cid:96) and k by comparing the deviation of z (cid:96) when ﬁxing y k .
Since the deviation is computed for each z (cid:96) separately, these metrics do not count how each latent variable z
(cid:96) interacts with the other variables z \ (cid:96) .
Another example is MIG (Chen et al., 2018), which compares mutual information
I ( y k ; z (cid:96) ) for all (cid:96) and uses the gap between the maximum and the second maximum among them.
More precisely,
MIG is deﬁned by the following formula.
[Equation]
Here, dividing each summand by H ( y k ) balances the contribution of each factor when they are discrete.
The difference in mutual information is rewritten as a difference in unique information as
[Equation]
In that sense, MIG effectively captures the pairwise interactions between latent variables.
This metric still ignores interplays between more than two variables.
Figure 2b reveals that some of the redundant information R ( y k ; z (cid:96) , z \ (cid:96) ) is positively evaluated in MIG, which should have been considered as a signal of entanglement.
Note that there are several extensions to MIG (Do & Tran, 2020; Li et al., 2020; Zaidi et al., 2020); see Appendix B for detailed discussions on them
I ( y k ; z )
I ( y k ; z (cid:96) ) I ( y k ; z \ (cid:96) )
U ( y k ; z (cid:96) \ z \ (cid:96) )
R ( y k ; z (cid:96) , z \ (cid:96) )
U ( y k ; z \ (cid:96) \ z (cid:96) )
C ( y k
[Figure]
z (cid:96) , z \ (cid:96) )
[Figure]
I ( y k ; z )
I ( y k ; z (cid:96) ) I ( y k ; z \ (cid:96) )
I ( y k ; z (cid:96) (cid:48) )
U ( y k ; z (cid:96) \ z (cid:96) (cid:48) )
R ( y k ; z (cid:96) , z (cid:96) (cid:48) )
U ( y k ; z (cid:96) (cid:48)
\ z (cid:96) )
U ( y k ; z (cid:96) \ z \ (cid:96) )
R ( y k ; z (cid:96) , z \ (cid:96) )
U ( y k ; z \ (cid:96) \ z (cid:96) )
C ( y k ; z (cid:96) , z \ (cid:96) )
[Figure]
) where z (cid:96) (cid:48) is the latent variable chosen by MIG evaluation.
The green boxes are positively evaluated in MIG (the top colored line) and
U NI B OUND (the bottom colored line), while the red boxes are negatively evaluated in them.
Observe that MIG positively evaluates a part of the redundancy, namely R ( y k ; z (cid:96) , z \ (cid:96) )
− R ( y k ; z (cid:96) , z (cid:96) (cid:48) ) , as it does not take into account the interactions among strict supersets of { z (cid:96) , z (cid:96) (cid:48) } .
3. U NI B OUND : N OVEL D ISENTANGLEMENT M ETRIC
We can lower bound the unique information in any possible PID deﬁnitions by computable components, as we did in Eq.4.
To bound U ( y k ; z (cid:96) \ z \ (cid:96) ) instead of U ( y k ; z (cid:96) \ z (cid:96) (cid:48) ) , we replace z (cid:96) (cid:48) with z \ (cid:96) , obtaining
[Equation]
(5) where we use [ · ] + = max { 0 , ·} as the difference in mutual information is not guaranteed to be nonnegative.
The decomposed terms evaluated by the bound is illustrated in the lower part of Figure 2b.
It effectively excludes, from the positive term, the effect of interaction between z (cid:96) and any other latent variables.
In a similar way to MIG, we summarize this bound over all the generative factors to obtain the metric we call U NI B OUND .
[Equation]
Dividing each summand by the entropy H ( y k ) has the same role as in MIG; it makes the evaluation fair between factors and eases the comparison as the metric is normalized when y k is discrete.
3. O THER B OUNDS FOR P ARTIAL I NFORMATION T ERMS
The U NI B OUND metric is a handy quantity to compare representations by a single scalar, while PID itself may provide more ideas on how a given representation entangles or disentangles the factors.
To fully leverage the potential, we derive bounds for all the terms of interest, including redundancy and synergy terms, from both lower and upper sides.
Let II ( y k ; z (cid:96) ; z \ (cid:96) )
=
I ( y k ; z (cid:96) )
+ I ( y k ; z \ (cid:96) )
− I ( y k ; z ) be the interaction information of a triple ( y k , z (cid:96) , z \ (cid:96) ) .
Using nonnegativity of PID terms, we can derive the following bounds from Eq.1-2.
[Equation]
Note that all six bounds are computed by arithmetics on I ( y k ; z (cid:96) ) , I ( y k ; z \ (cid:96) ) , and I ( y k ; z ) .
We can summarize each lower bound in a similar way as we did in Eq.6 and summarize the corresponding upper bound using the same (cid:96) for each k as the lower bound.
While these bounds only determine the terms as intervals, they provide us enough insight into the type of entanglement dominant in the representation (redundancy or synergy).
3. E STIMATING B OUNDS BY E XACT L OG -M ARGINAL D ENSITIES
When the dataset is fully annotated with discrete generative factors and the inference distribution p ( z | x ) and its marginals p ( z (cid:96) | x ) , p ( z \ (cid:96) | x ) are all tractable (e.g., mean ﬁeld variational models), we can compute the bounds in a similar way as is done by Chen et al.
(2018) for MIG.
Let z S be either of z (cid:96) , z \ (cid:96) or z .
We denote by D ( y k ) the subset of the dataset D with a speciﬁc value of y k .
The mutual information I ( y k ; z S ) = − H ( z S | y k )
+ H ( z S ) can then be computed by the following formula.
[Equation]
Assuming that each generative factor y k is discrete and uniform, we employ stratiﬁed sampling over p ( y k , z S ) .
We approximate the expectation over p ( z S | y k ) by sampling x from the subset
D ( y k ) and then sampling z S from p ( z | x ) to avoid quadratic computational cost.
Following Chen et al.
(2018), we used the sample size of 10000 in experiments.
We use log-sum-exp function to compute log (cid:80) p ( z S | x ) = log (cid:80) exp(log p ( z S | x )) for numerical stability.
The PID bounds and the
U NI B OUND metric are computed by combining these estimations.
When the inference distribution p ( z | x ) is factorized, its log marginal log p ( z S | x ) is computed by just adding up the log marginal of each variable as (cid:80) (cid:96) ∈ S log p ( z (cid:96) | x ) .
Otherwise, we need to explicitly derive the marginal distribution and compute the log density.
For example, when p ( z | x ) is a Gaussian distribution with mean µ and non-diagonal covariance Σ , p ( z S | x ) is a Gaussian distribution with mean ( µ i ) i ∈ S and covariance (Σ ij )
i,j ∈ S .
Such a case arises for the attacked model we will describe in the next section.
Let M be the sample size for the expectations, and N = |D| be the size of the dataset.
Then, the computation of Eq.8 requires O ( MN ) evaluations of the conditional density p ( z S | x ) .
E NTANGLEMENT A TTACKS
To conﬁrm that the proposed framework effectively captures interactions among multiple latent variables, we apply it to adversarial representations that entangle any generative factors.
Instead of making a single artiﬁcial model, we modify a given model that disentangles factors well into a noised version that entangles the original variables.
We call this process an entanglement attack .
Let z ∈ R L be the representation deﬁned by a given model.
Our goal is to design a transform from z to an attacked representation ˜
z
so that disentanglement metrics fail to capture entanglement unless they correctly handle multi-variable interactions.
As we mentioned in Section 3, metrics that do not take into account the interaction among multiple latent variables may underestimate redundant information.
To crystallize such a situation, we ﬁrst design an entanglement attack to inject redundant information into multiple variables.
For completeness, we also design a similar attack to inject synergistic information as well.
4. R EDUNDANCY A TTACK
Let U be an L × L orthonormal matrix and
(cid:15) ∈ R L be a random vector following the standard normal distribution N ( 0 , I ) .
Using a hyperparameter α ≥ 0 , we deﬁne the redundancy attack by
[Equation]
The coefﬁcient α adjusts the degree of entanglement.
When α = 0 , the new representation just appends noise elements to a given representation, which does not affect the disentanglement.
Increasing α makes the additional dimensions less noisy, resulting in ˜ z red redundantly encoding the information of factors conveyed by the original representation.
The mixing matrix U chooses how the information of individual variables in z is distributed to the additional dimensions.
If we choose
U =
I , the dimensions are not mixed; thus considering one-to-one interaction between pairs of variables is enough to capture the redundancy.
To mix the dimensions, we can use U
= I − 2 L 11 (cid:62) instead, which is an orthonormal matrix that mixes each variable with the others.
To evaluate mutual information terms for MIG and U NI B OUND metrics after the attack, we need an explicit formula for the inference distribution p (˜
z red | x ) .
When the original model has a Gaussian inference distribution p ( z | x ) =
N ( z ; µ ( x ) , Σ ( x )) , the attacked model is a summation of linear transformation of z and a standard normal noise, which results in a Gaussian distribution
[Equation]
4. S YNERGY A TTACK
For completeness, we also deﬁne an attack that entangles representation by increasing synergistic information.
Using the same setting with an L × L orthonormal matrix U and a noise vector (cid:15) ∼
N ( 0 , I ) , we construct the synergy attack by
[Equation]
Here again, the coefﬁcient α adjusts the degree of entanglement.
When α = 0 , the attacked version just extends the original representation with independent noise elements.
By increasing α , the noise are mixed with the parts conveying the information of factors.
The attacked vector ˜ z syn fully conveys the information of factors in z regardless of α , as we can recover the original representation by z = ˜
z syn 1: L − α U ˜ z
syn L +1:2 L .
Note that most existing metrics correctly react to this attack since the information of individual variables is destroyed by the noise.
The upper bound of the unique information is one of the quantities that positively evaluate synergistic information and is expected to ignore the attack.
E VALUATION
We evaluated the metrics in a toy model in Section 5.1 and in VAEs trained on datasets with the true generative factors in Section 5.2.
We also performed detailed analyses of VAEs by plotting the PID terms of each factor, which is deferred to Appendix E due to the limited space.
5. E XACT A NALYSIS WITH T OY M ODEL
We consider a toy model with attacks deﬁned in Section 4 as a sanity check.
Suppose that data are generated by factors y ∼ N ( 0 , I ) , and we have a latent representation that disentangles them up to noise as z | y ∼ N ( y , σ 2 I ) , where σ > 0 .
With this simple setting, we can analytically compute MIG and U NI B OUND .
For example, when we set σ = 0 . 1 and K = 5 , we can derive the scores after the redundancy attack as MIG = 1 c log (cid:16)
× 1+0 .
65 α 2
+1 .
α 2 (cid:17) and U NI B OUND = 1 c log (cid:16) 101 × 1+0 .
α 2 1+1 .
α 2 (cid:17) , where c = log(2 πe ) .
As a function of α , U NI B OUND decreases faster than MIG.
The difference comes from the information distributed among the added dimensions of the attacked vector; while
U NI B OUND counts all the entangled information, MIG only deals with one of the added dimensions.
Indeed, the amount of the untracked entanglement remains in MIG after taking the limit of α → ∞ as MIG → 1 c log 65 , while U NI B OUND → 0 .
We can also compute the scores after the synergy attack as MIG =
U NI B OUND = 1 c log (cid:16) 1 + 1 α 2 +0 .(cid:17) , where both metrics correctly capture the injected synergy.
Refer to Appendix
C for the derivations of the exact scores for arbitrary parameter choice.
5. E MPIRICAL A NALYSIS WITH A NNOTATED D ATASETS
We used the D S PRITES and 3 DSHAPES datasets for our analysis.
The D S PRITES dataset consists of 737 , 280 binary images generated from ﬁve generative factors (shape, size, rotation, and x/y coordinates).
The 3 DSHAPES dataset consists of 480 , 000 color images generated from six generative factors (ﬂoor/wall/object hues, scale, shape, and orientation).
All the images have 64 × 64 pixels.
We used all the factors for y , encoded as a set of discrete (categorical) random variables.
We used variants of VAEs as methods for disentangled representation learning: β -VAE (Higgins et al., 2017), FactorVAE (Kim & Mnih, 2018), β -TCVAE
(Chen et al., 2018), and JointVAE (Dupont, 2018).
We trained all the models with six latent variables, one of which in JointVAE is
[Figure]
I
[Figure] S [Figure] a three-way categorical variable for D S PRITES and a four-way categorical variable for 3 DSHAPES .
We trained each model eight times with different random seeds and chose the best half of them for each metric to avoid cluttered results due to training instability.
We optimized network weights with
Adam (Kingma & Ba, 2015).
We used the standard convolutional networks used in the literature for the encoder and the decoder.
See Appendix D for the details of architectures and hyperparameters.
For disentanglement metrics, we compare BetaVAE metric (Higgins et al., 2017), FactorVAE metric (Kim & Mnih, 2018), MIG (Chen et al., 2018), and the proposed U NI B OUND metric.
We ﬁrst compared the models with each metric as shown in Figure 3a.
The trend is basically similar between U NI B OUND and the other metrics, while they disagree in some cases.
For example, FactorVAE achieves a higher MIG score for D S PRITES than β -VAE, while its U NI B OUND score is low.
As we saw in Figure 2b, such a case occurs when a part of the redundancy R ( y k ; z (cid:96) , z \ (cid:96) ) −R ( y k ; z (cid:96) , z (cid:96) (cid:48) ) is large.
This observation indicates that FactorVAE effectively forces each variable to encode information of distinct factors (i.e., one-vs-one redundancy is small), while it fails to avoid entangling the information over multiple variables (i.e., one-vs-all redundancy is large).
We can conﬁrm that the redundancy is indeed large in FactorVAE by computing the PID bounds.
In
Figure 3b, we plot the aggregated bounds of U ( y k ; z (cid:96) \ z \ (cid:96) ) , R ( y k ; z (cid:96) , z \ (cid:96) ) , and C ( y k ; z (cid:96) , z \ (cid:96) ) .
The plot reveals that FactorVAE tends to encode the factors redundantly into multiple latent variables.
To understand the tasks and models more deeply, we evaluated the PID bounds for each factor in
Figure 4.
We can see that the factors are captured by the models in various ways.
For example, βTCVAE succeeds to disentangle the position and scale factors in D S PRITES , while it encodes orientation synergistically.
It may reﬂect the inherent difﬁculty of disentangling this factor in D S PRITES , as the image of each shape corresponding to 0 ◦ orientation is chosen arbitrarily.
These observations help us to choose what kind of inductive biases to introduce into the models.
[Figure]
FactorVAE, on the other hand, tends to encode factors redundantly.
This indicates FactorVAE succeeds to make individual variables encode each factor, while it fails to prevent other variables from encoding the same information.
JointVAE also encodes factors redundantly.
While it fails to disentangle all the factors, this is the only model that encodes the shape and orientation to single latent variables.
This can be viewed as the effect of introducing a discrete variable into the representation.
We can understand large redundancy as the models failing to make variables independent enough.
The lower bound of redundancy in Eq.7 is related to independence as
[Equation]
Therefore, a high redundancy lower bound indicates large mutual information
I ( z k ; z \ (cid:96) ) ; i.e., the latent variables are highly dependent.
We conjecture that FactorVAE, which approximates the total correlation D KL ( p ( z ) (cid:107) (cid:81) (cid:96) p ( z (cid:96) )) by a critic, fails to make the critic capture the dependency in z enough in our experiments.
Since the MIG score is relatively high, the critic succeeds to capture pairwise dependency, while it fails to capture higher dimensional dependency.
The high redundancy in JointVAE can also be explained by the lack of independence; see Appendix F for details.
As in Figure 3a, some metrics have large deviations from the median.
This is caused by the randomness in training rather than in evaluation; see Appendix I for detailed analyses.
As we did for the toy model, we performed entanglement attacks in Figure 3c to assess its effect on each metric in learned representations.
We selected the best training trials of β -VAE and β -TCVAE.
Here, we only plot the results with the redundancy attack, as each metric already behaves well against the synergy attack.
The plot reveals that BetaVAE and FactorVAE metrics do not detect the redundancy injected by the attack.
MIG slightly decreases with the attack, which is not signiﬁcant against the score variation between learning methods as we observed in Figure 3a.
U NI B OUND strongly reacts against the attack, indicating that it effectively detects the injected redundancy.
C ONCLUSION
We established a framework of disentanglement analysis using Partial Information Decomposition.
We formulated a new disentanglement metric, U NI B OUND , using the unique information bounds, and conﬁrmed with entanglement attacks that the metric correctly responses to entanglement caused by multi-variable interactions which are not captured by other metrics.
U NI B OUND sometimes disagrees with other metrics on VAEs trained with controlled datasets, which indicates that multivariable interactions arise not only in artiﬁtial settings but in learned representations.
We found that
VAEs trained with different methods induce representations with a variety of ratios between PID terms, even if their disentanglement scores are close.
It is a major future work to develop learning methods of disentangled representations on the basis of these observations.
A CKNOWLEDGMENTS
We thank members of Issei Sato Laboratory and researchers in Preferred Networks for fruitful discussions and thank the reviewers for helpful comments to improve the work
A A DDITIONAL DIAGRAMS
[Equation]
B R ELATIONSHIPS TO O THER I NFORMATION -T HEORETIC M
There have been several metrics proposed in the literature for information-theoretic measurement of disentanglement.
For example, Do & Tran (2020) proposed four metrics, namely WSEPIN, WINDIN, RMIG, and
JEMMIG.
Among them, WSEPIN is computed based on the information gap
I ( x ; z (cid:96) | z \ (cid:96) ) =
I ( x ; z ) − I ( x ; z (cid:96) ) .
In the PID perspective, this quantity upper bounds the unique information of x held by z (cid:96) , i.e., I ( x ; z (cid:96) | z \ (cid:96) ) ≥ U ( x ; z (cid:96) \ z \ (cid:96) ) .
It is similar to the upper bound of unique information that we derived in Eq.7, as
[Equation]
In that sense, WSEPIN is an upper bound of unique information, where y k is replaced with x and the nonnegativity of redundant information is ignored.
As WSEPIN does not handle generative factors separately, it is useful when the generative factors are unknown, while our approach provides more detailed information of (dis)entanglement when the generative factors are available.
RMIG is an extension to MIG, where the conditioning between y k and x is inverted so that one can compute the quantity with uncontrolled dataset.
This approach may be extended for our framework as well, in a similar way to applying the inversion to MIG.
We did not use this extension as we only deal with controlled datasets in this paper to contrast our approach with a wider variety of metrics.
JEMMIG is also an extention to MIG, which provides a measurement of how each variable captures only one generative factor.
This aspect is also studied by Eastwood & Williams (2018) and Li et al.
As we used a set of simple latent variables each of which is either a one-dimensional real variable or a categorical variable with a small number of arms (three or four depending on the dataset), we expect that each variable does not capture much information of multiple generative factors.
Indeed, we did not observe any latent variable selected for more than two generative factors, and when a latent variable is selected for two generative factors, the overall disentanglement score is low so that the effect of duplicated selection is ignorable.
We still consider it an interesting future work to extend our approach for analyzing entanglement of multiple generative factors into a single latent variable.
C D ERIVATIONS FOR THE E XACT A NALYSIS WITH THE T OY M ODEL
In this section, we provide a sketch of analytically computing the metrics for the toy model we used in Section 5.1.
We use the following formulae
• Let u be a Gaussian vector with covariance Σ .
Then, its entropy is given by H ( u )
= 12 log(2 πe | Σ | ) , where |·| is the matrix determinant.
When we remove the (cid:96) -th element from u , the remaining vector u \ (cid:96) follows a Gaussian distribution whose covariance matrix Σ \ (cid:96) is obtained by removing the (cid:96) -th row and column from Σ .
The determinant of Σ \ (cid:96) is the cofactor of Σ at the (cid:96) -th diagonal element, i.e., (cid:12) (cid:12) Σ \ (cid:96) (cid:12) (cid:12) = | Σ | ( Σ − 1 ) (cid:96)(cid:96) .
With this formula, we can derive the entropy of u \ (cid:96) as H ( u \ (cid:96) )
=
H ( u ) + 12 log( Σ − 1 ) (cid:96)(cid:96) .
• Suppose that an invertible matrix Σ is written by blocks as Σ =
(cid:18)
A
B
C
D (cid:19)
If A and its Schur complement S := D −
CA − 1 B are invertible, the determinant and inverse of Σ are
[Table]
written as
[Equation]
Similarly, if D and its Schur complement T :
= A − BD − 1 C are invertible, then | Σ | = | D | · | T | , Σ
[Equation]
Provided that z is a Gaussian vector with covariance Σ , we can derive the entropies of the attacked vectors.
Recall that the redundancy and synergy attacks are deﬁned as follows.
[Equation]
where U is an orthonormal matrix, α > 0 , and (cid:15) ∼ N ( 0 , I ) .
Here, 0 is the zero matrix.
These again follow Gaussian distributions, whose covariance matrices are computed as follows.
[Equation]
Using Eq.13 and Eq.14, we obtain their determinants and inverses.
[Equation]
Therefore
[Equation]
we obtain the following entropies for each (cid:96) ∈ { 1 , . . .
, L } .
[Equation] [Equation]
Here, u (cid:96) and v (cid:96)
[Table]
Recall that, in the toy model
[Table]
the representation z is drawn from N ( y , σ 2 I )
[Table]
where the generative factors y follow the standard Gaussian distribution.
Therefore, the marginal distribution of the representation is p ( z ) =
N ( z ; 0 , (1 + σ 2 ) I ) .
When the representation is conditioned by a single factor y k , it follows p ( z | y k ) =
N ( z ; y k e k , (1 + σ 2 )
I − e k e (cid:62) k ) .
By substituting the covariance matrices of these distributions to Σ in Eq.15 and computing their differences, we obtain the mutual information terms as follows.
[Equation]
The metrics in Table 1 are computed from these quantities.
In addition, we can compute other partial information terms as follows.
[Equation]
We can observe that the redundancy and synergy terms effectively increase with the corresponding attacks.
D M ODEL A RCHITECTURES AND T RAINING H YPERPARAMETERS
The architectures of the encoder and decoder used in the experiments are listed in Table 2.
We used ReLU nonlinearity at each convolutional layer except for the ﬁnal output of each network.
In addition, we used for the critic in FactorVAE a feedforward network with ﬁve hidden layers each of which consists of 1,000 leaky ReLU units with the slope coefﬁcient of 0 . 2 .
[Table] [Table]
β
=γ =γ =β =
The hyperparameters used in training are listed in Table 3 and Table 4.
For hyperparameters not listed in the tables, we used the values suggested in the original papers.
E F ULL R ESULTS FOR F ACTOR - WISE PID A NALYSES
We illustrate the estimated bounds of PID terms for each factor in Figure 6, whose small version appeared in Figure 4.
We summarize these results in Table 5.
This table is made by observing and categorizing the plots in Figure 6 into some patterns as follows.
Note that we ignore the error bars here.
If the lower bound of the unique information is larger than the upper bounds of the redundant and complementary information, mark the plot as disentangled .
In this case, the model is determined as successfully disentangling the factor regardless of the concrete deﬁnition of PID.
Note that the model does not necessarily learn the factor completely; see the ﬁgure for how much the information of the factor is uniquely captured by a latent variable.
If the upper and lower bounds of the redundant information are larger than those of the complementary information by a margin, mark the plot as redundant .
In this case, the model entangles the factor in a redundant way.
As we analyzed in Section 5.2, this indicates that the latent variables are highly dependent.
If the upper and lower bounds of the complementary information are larger than those of the redundant information by a margin, mark the plot as synergistic .
In this case, the model entangles the factor in a synergistic way.
This may occur even if the latent variables are independent.
We may require additional inductive biases to help the model disentangle the factor.
Otherwise, mark the plot as ﬂat .
F P OSSIBLE E XPLANATIONS FOR H IGH R EDUNDANCY IN J OINT VAE
We observed in Table 5 and Figure 6 that JointVAE suffers from high redundancy in all the factors.
To explain this phenomenon, we review the training scheme of JointVAE (Dupont, 2018).
The βVA E
F ac t
o r
VA E β - T
C VA
E
J o i n t
VA E
[Figure]
[Table] [Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
JointVAE model consists of one categorical variable z 1 and L − 1 Gaussian variables z 2: L .
Let U ( z 1 ) be the uniform categorical distribution and p d ( x | z ) be the decoder to be learned simultaneously.
Then, the objective function of JointVAE is
[Equation]
| , which involves three hyperparameters: the regularization coefﬁcient γ , the capacity of the discrete variable c 1 , and the capacity of continuous variables c 2 .
Throughout training, γ is kept constant, while the capacities c 1 and c 2 are gradually increased and saturated at the predeﬁned maximum values ( C c and C z , respectively) in the middle of training.
As the capacities are positive at the end of training, this indicates that the KL terms, which include the total correlation of the latent variables (Kim & Mnih, 2018; Chen et al., 2018), are large in the trained model.
Therefore, as we discussed in Section 5.2, this model does not add pressure on the representation to be less redundant, which may explain the high redundancy 3 .
See Appendix G for the results with varying C z , whose results also support the above hypothesis as low capacities induce lower redundancy.
This training scheme is designed to align the amount of information captured by discrete and continuous variables, which seem to be successful as we observed, i.e., it effectively captures the shape factor in both datasets compared to the other methods.
G I NFLUENCE OF R EGULARIZATION H YPERPARAMETERS ON D ISENTANGLEMENT
We trained each model with varying hyperparameters ( β for β -VAE and β -TCVAE, γ for FactorVAE, and the capcity of continuous variables C z for JointVAE) and investigated how the metrics as well as PID terms are affected by the regularization strengths KL divergence
[Figure]
[Figure]
The results for D S PRITES and 3 DSHAPES are illustrated in Fig.7 and Fig.8, respectively.
From the left panels, we can observe that the U NI B OUND metric is positively correlated with the regularization strength 5 .
It indicates that the regularization method introduced by each model positively contributes to disentanglement in the PID perspective.
We also estimated PID bounds in the right panels.
They show some interesting effects of regularization against the types of entanglement.
For example, in JointVAE, the representation has low redundancy when the capacity of continuous variables is small, and the redundancy grows signiﬁcantly when we increase the capacity.
It indicates that a large capacity makes the model enforce each latent variable to capture details of the input images, ignoring how the information is redundantly captured by other variables.
H L ARGE F IGURES OF E XPERIMENTAL R ESULTS
We put large versions of Figure 3 and Figure 4 in Figure 9 and Figure 6, respectively, for ﬁner rendering.
I T RAINING AND E VALUATION S TABILITY
We illustrate the disentanglement scores of models trained with eight training seeds in Figure 10 and
Figure 11.
As each disentanglement metric involves sampling during evaluation, the evaluated score has some randomness even if we ﬁx the training random seed.
This plot reveals that the deviation caused by randomness in evaluating disentanglement metrics is much smaller than the deviation caused by randomness in training each model.
[Figure]
[Figure] [Figure]