T OWARDS B UILDING A G ROUP - BASED U NSUPERVISED R EPRESENTATION D ISENTANGLEMENT F RAMEWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
Disentangled representation learning is one of the major goals of deep learning, and is a key step for achieving explainable and generalizable models.
The key idea of the state-of-the-art VAE-based unsupervised representation disentanglement methods is to minimize the total correlation of the joint distribution of the latent variables.
However, these models lack theoretical guarantee.
The Group Theory based deﬁnition of representation disentanglement mathematically connects the data transformations to the representations using the formalism of group .
In this paper, built on the group-based deﬁnition and inspired by the n-th dihedral group , we ﬁrst propose a theoretical framework towards achieving unsupervised representation disentanglement.
We then propose a model based on existing VAEbased methods to tackle the unsupervised learning problem of the framework.
In the theoretical framework, we prove three sufﬁcient conditions on model, group structure, and data respectively in an effort to achieve, in an unsupervised way, disentangled representation per group-based deﬁnition.
With the ﬁrst two of the conditions satisﬁed and a necessary condition derived for the third one, we offer an option, from the perspective of the group-based deﬁnition, as an additional inductive bias for the existing VAE-based models.
Experimentally, we train 1800 models covering the most prominent VAE-based methods on ﬁve datasets to verify the effectiveness of our theoretical framework.
Compared to the original VAE-based methods, these Groupiﬁed VAEs consistently achieve better mean performance with smaller variances.
I NTRODUCTION
Learning independent and semantic representations of which individual dimension has interpretable meaning, usually referred to as disentangled representations learning, is critical for artiﬁcial intelligence research (Bengio et al., 2013).
Such disentangled representations are useful for many tasks: domain adaptation (Li et al., 2019; Zou et al., 2020), zero-shot learning (Lake et al., 2017), and adversarial attacks (Alemi et al., 2016), etc.
Intuitively, a disentangled representation should reﬂect the factors of variations behind the observed data of the world, and one latent unit is only sensitive to changes of an individual factor.
Due to the facts that obtaining the ground-truth labels requires signiﬁcant human effort and humans can learn those factors unsupervisedly, unsupervised representation disentanglement draws much attention from researchers recently.
A lot of methods are proposed base on some intuitions.
Most of the state-of-the-art methods (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2017) are based on Variational Autoencoder (VAE) (Kingma & Welling, 2013).
These methods are fully unsupervised and can be applied to a variety of complex datasets (Lee et al., 2020).
However, Locatello et al. (2019b) prove that disentanglement of existing VAE-based methods is fundamentally impossible to achieve without introducing additional inductive bias on both models and data.
Another stream of works (Chen et al., 2016; Lin et al., 2020; Khrulkov et al., 2021; Lee et al., 2020) leverages generative adversarial network (GAN) (Goodfellow et al., 2014) to achieve disentanglement.
In general, due to lack of theoretical guarantee, those methods are not interpretable, and there even exist trivial solutions (Higgins et al., 2018; Locatello et al., 2019b).
The research of symmetry in physics demonstrates that inﬁnitesimal transformations that conform to some symmetry groups on physical objects can reﬂect their nature (Anderson, 1972; Noether, 1915).
Recently, inspired by this research on symmetry, Higgins et al.
(2018) proposed a group-based deﬁnition of disentangled representation.
They argue that the symmetries, i.e., the transformations that change certain aspects of data and keep other aspects unchanged, ideally reﬂect the underlying data structure.
The group-based deﬁnition is a formal and rigorous mathematical deﬁnition of faithful and, ideally, interpretable representation of the generative factors of data, which is widely accepted (Greff et al., 2019; Mathieu et al., 2019; Khemakhem et al., 2020).
Subsequently, due to the fact that the deﬁnition is deﬁned by the world state (i.e., Ground Truth) and based on the assumption (CasellesDupr´e et al., 2019) that this deﬁnition should be useful for downstream tasks such as a Reinforcement
Learning, Caselles-Dupr´e et al.
, Quessard et al.
Painter et al.
(2020) propose environment-based (to provide world state) methods to learn such disentangled representations in
Reinforcement Learning settings.
These inspire us to ask the following question: how would the deﬁnition beneﬁt unsupervised representation disentanglement, and how to learn such a disentangled representation conforming to the deﬁnition in the setting of unsupervised representation learning?
In Group Theory, the n-th dihedral group 1 (Judson, 2020) is a set of all permutations of polygons vertices, forming a permutation group under the operation of composition (Miller, 1973).
The generators in an n-th dihedral group , i.e., ﬂip and rotation, can be regarded as the disentangled factors and also transformations.
In this paper, inspired by the n-th dihedral group , we answer the above questions and address the challenge by proposing a theoretical framework to make the deﬁnition practically applicable for unsupervised representation disentanglement.
We then propose a model to tackle the learning problem of the framework and verify its effectiveness.
We theoretically prove in Section 3.2 the three sufﬁcient conditions towards achieving disentangled representation per group-based deﬁnition, which are referred to as model, group structure, and data constraints, respectively.
With these conditions, we offer an option from the perspective of the deﬁnition as additional inductive bias.
The inductive bias encourages existing VAE-based models to satisfy the symmetry requirement that comes from the nature of factors.
Finally, we provide a learning model based on the existing VAE-based methods in an effort to fulﬁll the three conditions (with the model and group structure constraint and a necessary condition for the data constraint satisﬁed).
Our model consistently achieves statistically better performance in prominent metrics (higher means and lower variances) than corresponding existing VAE-based models on ﬁve datasets, demonstrating that the group-based deﬁnition together with our proposed framework further encourages disentanglement.
Our main contributions are summarized as follows:
• To our best knowledge, we are the ﬁrst to provide a theoretical framework to make the formal group-based mathematical deﬁnition of disentanglement practically applicable to unsupervised representation disentanglement.
• Our theoretical framework provides options from the perspective of group-based deﬁnition for additional inductive bias to existing VAE-based methods.
• We propose a learning model of the framework by deriving and integrating additional loss into existing VAE-based models, in an effort to make the learned representation conform to the group-based deﬁnition without relying on the environment (as done in Caselles-Dupr´e et al.
; Quessard et al.
Painter et al.
R ELATED W ORKS
Different deﬁnitions have been proposed for disentangled representation (Bengio et al., 2013; Higgins et al., 2018; Suter et al., 2019).
However, only the group-based deﬁnition proposed by Higgins et al.
(2018) focuses on the disentangled representation itself and is mathematically rigorous, which is well accepted (Caselles-Dupr´e et al., 2019;
Quessard et al., 2020; Painter et al., 2020; Diane Bouchacourt, 2021).
Nevertheless, Higgins et al.
(2018) do not propose a speciﬁc learning method based on their deﬁnition.
Before this rigorous deﬁnition was proposed, there had been some success in identifying generative factors in static datasets (without interaction with environment), such as β -VAE (Higgins et al., 2017),
Anneal-VAE (Burgess et al., 2018), β -TCVAE (Chen et al., 2018), and FactorVAE
(Kim
Mnih, 2018).
These VAE-based unsupervised methods are based on probabilistic inference.
Recently, Locatello et al.
(2019b) proved by Measure Theory that these methods theoretically have inﬁnite number of solutions (unidentiﬁable problem).
Other methods (Srivastava et al., 2020;
Shao et al., 2020; Kim et al., 2019; Lezama, 2018; Rezende & Viola, 2018) also suffer from the unidentiﬁable problem.
Therefore, we hope that bringing the group-based deﬁnition into this setting can help understand the alluded inductive bias that existing VAE-based unsupervised models lack, as suggested by Locatello et al. (2019b).
As pointed out in Quessard et al.
(2020), it is not straightforward to reconcile the probabilistic inference methods with the group-based deﬁnition framework.
Caselles-Dupr´e et al.
, Quessard et al.
Painter et al.
(2020) leverage the interaction with the environment (assuming it is available) as supervision instead of minimizing the total correlation as the VAE-based methods do.
Consequently, the effectiveness of these methods is limited to the datasets with the environment available.
Our framework learns a representation conforming to the group-based deﬁnition without relying on the environment.
Pfau et al.
(2020) propose a non-parametric method to unsupervisedly learn linear disentangled planes in data manifold under a metric.
However, as pointed out by the authors, the method does not generalize to held-out data and performs poorly when trying to disentangle directly from pixels.
To summarize, the existing probabilistic inference methods lack theoretical support, while the application scope of existing methods based on the group-based mathematical deﬁnition Higgins et al.
(2018) is very limited.
To the best of our knowledge, our work is the ﬁrst to reconcile the probabilistic generative methods with the inherently deterministic group-based deﬁnition framework of Higgins et al.
. T HE G ROUP - BASED F RAMEWORK F OR U NSUPERVISED R EPRESENTATION D ISENTANGLEMENT
Our goal is to explore the beneﬁt of the group-based deﬁnition for unsupervised representation disentanglement and learn such a disentangled representation.
The background of the group-based deﬁnition is provided in Section 3.1.
Section 3.2 presents the theoretical framework towards achieving unsupervised disentanglement, in which we derive three sufﬁcient conditions on the model, group structure, and data, respectively.
The conditions on the model and group structure provide options for the additional inductive bias of the existing VAE-based models.
3. G ROUP - BASED D EFINITION
We brieﬂy review the group-based deﬁnition of disentangled representation Higgins et al.
Considering a group G acting on world state space W (can be understood as ground-truth) of data space O and representation space Z via group action · W and group action · Z respectively.
For a mapping f =
b
◦ h , where b and h denote the data generative process and encoding, we state: the mapping f is equivariant between the actions on W and Z if
[Equation]
Deﬁnition 1 Assume G can be decomposed as G =
G 1 × G 2 × · · · × G m .
The set Z is disentangled with respect to G if: ( i ) group action of G on Z exits.
( ii ) the mapping f is equivariant between the actions on W and Z .
( iii )
There is a decomposition Z
= Z 1 × Z 2 × · · · × Z m such that each Z i is affected only by the corresponding G i .
It is challenging to apply the group-based deﬁnition to an unsupervised disentanglement setting in practice because the deﬁnition refers to the world state space W , the group action of G on W , and mapping b which are typically inaccessible in practice.
We tackle the challenge by re-framing the deﬁnition in a new framework in the following section.
3. P ROPOSED T HEORETICAL F RAMEWORK
Since when the representation is disentangled, one latent unit in the representation space is only sensitive to changes of an individual generative factor, we make the following assumptions: G is a direct product of m cyclic groups (as suggested by Higgins et al.
(2018) and for simplicity):
G =
( Z /n Z ) m =
Z /n
Z ×
Z /n
Z × · · · × Z /n Z , where n is the assumed total number of possible values for a factor and m is the total number of factors; we further assume Z is a set with the same elements in G .
Therefore, the group actions of G on Z can be set to be element-wise addition, i.e., g · z = g + z, ∀ z ∈ Z, g ∈ G .
For the generator of dimension i of G , g i = (0 , . .
. , 0) , g i only affects the i -th dimension of z by g i
+ z .
In addition, the action of each generator g i on w only affects a single dimension of w .
As we can seen from Equation 1 above, the group action is deﬁned on w , which is often not accessible, making it difﬁcult to apply the deﬁnition in practice.
Therefore, for the unsupervised setting, we would like to use permutations on the data space O (which only provides data without labels) to substitute the group actions on W .
Speciﬁcally, inspired by the n-th dihedral group (Dummit &
Foote, 1991), we construct a permutation group Φ , serving the role of an “agent” of G .
The actions of G on W can be performed by ϕ g ∈ Φ on O , which can be formulated as
[Equation]
where o denotes the data (e.g., image) corresponding to the world state w through the mapping function b .
If the above equation holds, we state that the “agent” permutation group Φ exists.
We ﬁrst give the conditions for the existence of this “agent” permutation group Φ , then derive the additional condition to achieve such disentanglement.
We accomplish these two objectives in Theorem 1 with the proof provided in Appendix B. Theorem 1 states that a general permutation group Φ on O can serve as an agent group ( agent group exists ) if and only if both ( i ) and ( ii ) are satisﬁed.
If the agent group exists, and its permutations (actions on O ) can be deﬁned by an autoencoder-like model as shown in the equation in ( iii ) , then Z is disentangled with respect to G .
Theorem 1 For the group G =
( Z /n Z ) m , a permutation group Φ on O , a representation space Z , a World State space W , and mapping b and h , Equation 2 holds if and only if ( i ) Φ is isomorphic to
G , and ( ii ) For each generator of dimension i of G , g i , there exists a ϕ i ∈
Φ , i = 1 , . . .
, m , such that ϕ
i · b ( w ) =
b ( g i · w ) , ∀ w ∈ W , and ϕ
i is a generator of Φ ; Further, if Equation 2 holds and ( iii ) ϕ g · b ( w ) = h − 1 ( g · f ( w ))
∀ w ∈ W, ϕ g ∈ Φ , then Z is disentangled with respect to G , where ϕ g is the corresponding element in Φ of g under the isomorphism.
In Theorem 1, ( i ) states that the relation between the elements (i.e., group structure) is preserved between Φ and G , and we denote it as the group structure constraint ; ( ii ) actually indicates a data constraint that all variations in the data can be generated by compositions of some basic permutation generators { ϕ i } i =1 ,...,m .
We denote it as the data constraint ; ( iii ) states that the permutations in the agent group Φ are deﬁned by encoding, action, and decoding, which is referred to as the model constraint .
Note that in Theorem 1, only the data constraint refers to the world state w .
Here is a sketch of the proof: data constraint is a special case of Equation 2 for a generator , and group structure constraint is a relation-preserving constraint on compositions of generators , and satisfying both constraints will thus result in that Equation 2 holds for any general element in Φ , and vice versa.
Moreover, we can derive Equation 1 for disentanglement when combining the model constraint and Equation 2.
The model constraint speciﬁes the way to permute the data.
When the data is permuted, its world state changes.
Therefore, how the world states transit between each other is modeled by the model constraint applied on the data.
The isomorphism between Φ and G ensures that the world state space
W and data space
O have the same symmetry.
In this way, the model applied on the data learns the transition of the world states.
Note that we aim to bring this group-based deﬁnition, which requires ground truth by default, into the unsupervised setting.
Now only the data constraint refers to the world states, and it seems almost impossible to derive a sufﬁcient condition for it without the labels.
We thus make a trade-off in which we use a necessary condition in the next section.
G ROUPIFIED VAE: A LEARNING METHOD OF THE FRAMEWORK
Let’s look closer into the three constraints, respectively.
Firstly, we consider the model constraint, ϕ g · o = h − 1 ( g · h ( o )) ∀ o ∈ O, ϕ g ∈ Φ , which suggests that the action of Φ on O can be implemented using an autoencoder-like network that performs encoding, action on its representation space, and decoding.
Given an autoencoder-like network with an encoder h and a decoder d , since d
[Figure]
Abel is approximately the inversion of h , the model constraint can be formulated as
[Equation]
together with further implementation of Φ as described in Section 4.1, the model constraint can be fulﬁlled.
Secondly, The data constraint requires that all variations in the data can be generated by compositions of some basic permutations generators .
Previous VAE-based works (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018;
Chen et al., 2018; Kumar et al., 2017) aim to generate the data with independent generative factors, which is in line with the data constraint .
Intuitively, if the VAE-based model can generate the data from statistical independent basic latent units and each unit corresponds to the basic permutation generator , the data constraint may be satisﬁed.
Based on the intuition above, we prove that if the world state is independently sampled per dimension, the minimization of total correlation is a necessary condition for the data constraint (see Appendix E).
Therefore, we can leverage existing VAE-based models to fulﬁll the data constraint to some extent for the unsupervised setting.
Lastly, to satisfy the group structure constraint , we derive a self-supervised
Isomorphism Loss which can be incorporated into the VAE-based model as described in Section 4.2.
4. I MPLEMENTATION OF G ROUP Φ
The key is to implement the group actions of G on Z into the VAE-based models, we need to map the representation z to a group that is isomorphic to G (cyclic representation space).
Therefore, we construct a function η to achieve this mapping.
Moreover, this mapping is required to be differentiable, in order for back-propagation to be adopted for optimization.
According to Group Theory, there is an isomorphism between G and the n-th root unity group : { exp((2 πiz ) /n )
| z ∈ Z m } , where n, m are the same as in G .
Therefore, the representation z can be mapped to z by the function η as z = η ( z ) =
exp((2 πiz ) /n )
(see Figure 1 (a)).
However, z can not be mapped to directly as it has complex numbers, but we can use Euler’s formula: exp((2 πiz ) /n ) =
sin((2 πz ) /n )
+ i cos((2 πz ) /n ) to map z to its real and imaginary part, i.e., vector sin((2 πz ) /n )
and cos((2 πz ) /n ) .
The two vectors are concatenated and fed to the decoder.
For ease of implementation, the permutation group Φ can be approximately generated by compositions of generators , i.e., Φ = < ϕ 1 , ϕ 2 , . . .
ϕ m > .
Recall that the generator ϕ i of group Φ is deﬁned as ϕ i · o = d ( g i · h ( o ))
=
d ( h ( o ) + g i ) , ∀ o ∈ O , where g i is generator of dimension i in G , as shown in Figure 1 (a).
For ϕ i , we implement g i · h ( o ) by adding 1 (without loss of generality) to the i -th dimension of h ( o ) , then make it cyclic by function η .
Similarly, for ϕ − 1 i , we add the value of n − 1 .
4. I MPLEMENTATION OF THE I SOMORPHISM
In this section, to satisfy the group structure constraint (isomorphism), we derive two equivalent constraints, which are then converted into an Isomorphism Loss L I .
Many groups are uniquely determined by the properties of the generators , e.g., group G = < a, b |
a 2 = b 2 = e, ab = ba > .
In addition, since the group Φ is isomorphic to G, Φ is also expected to be commutative and cyclic.
In light of this, we derive two constraints on generators that are equivalent to the isomorphism condition, as described in Theorem 2.
Please refer to Appendix C for the proof.
Theorem
The deﬁned permutation group Φ = < ϕ 1 , ϕ 2 , . .
. , ϕ m > is isomorphic to G =
( Z /n Z ) m if and only if: ( i ) for ∀ generators ϕ i , ϕ j ∈ Φ , 1 ≤ i, j ≤ m , we have ϕ i ϕ
j = ϕ
j ϕ i , and ( ii ) ∀ ϕ
i ∈
Φ , 1 ≤ i ≤ m , we have ϕ ni =
e , where e is the identity element of group Φ .
The ﬁrst constraint requires the group Φ to be an abelian group (Judson, 2020).
Therefore, we denote it as Abel constraint and the loss derived from it as the Abel Loss L a .
The second is a constraint on the order of elements.
We thus denote it as the Order constraint and the loss derived from it as the
Order Loss L o .
See Appendix F for a more detailed implementation.
Abel Loss.
For the Abel constraint: ∀ ϕ i , ϕ j ∈ Φ , 1 ≤ i, j ≤ m , we have ϕ
i ϕ j
= ϕ
j ϕ i .
We minimize (cid:107) ϕ
i · ( ϕ j · o ) − ϕ j · ( ϕ i · o ) (cid:107) , ∀ o ∈ O to meet the Abel constraint, as shown in Figure(b).
The Abel Loss is the sum of the losses of all combinations of two generators .
Denote the set of combinations as C =
{ ( i, j ) | 1 ≤ i, j ≤ m } .
The Abel Loss is deﬁned as follows
[Equation]
Order Loss.
For the Order constraint: ∀ ϕ i ∈ Φ , 1 ≤ i ≤ m , we have ϕ ni = e , where e is the identity element in group Φ (identity mapping).
Note that with n times composition of ϕ i , it is difﬁcult for the gradient to back-propagate.
We thus use an approximation that uses 2 times of composition instead.
When the autoencoder can do the reconstruction well, this approximation holds, see appendix E for details.
Similar to Abel Loss, we minimize (cid:107) ϕ
i · ( ϕ n − 1 i · o ) − o (cid:107) , ∀ o ∈ O to satisfy the Order constraint.
The whole process is illustrated in Figure 1 (b).
However, the equation is not symmetrical and leads to bias.
Therefore, we use the following symmetrical form instead:
[Equation]
With the above two loss functions optimized, the isomorphism condition is satisﬁed, which can be illustrated by Theorem 3.
Please refer to Appendix D for the proof.
Theorem
The following two conditions are equivalent: ( i ) ∀ ϕ
i , ϕ j ∈ Φ , 1 ≤ i, j ≤ m , we have ϕ
i ϕ j
= ϕ j ϕ i and ∀ ϕ i ∈ Φ , 1 ≤ i ≤ m , we have ϕ ni
=
e
( ii ) the Abel Loss function (Equation 4) and the Order Loss function (Equation 5) are optimized.
Since the Abel Loss and Order Loss are equally important for satisfying the isomorphism condition, we assign equal weight to them.
Thus, the Isomorphism Loss is L I =
L o + L a .
With the implementation of group Φ , the model constraint is satisﬁed.
We optimize the Isomorphism Loss to satisfy the group structure constraint .
To further satisfy the data constraint to some extent as described in Section 4, we leverage VAE-based models and optimize their original loss (that minimizes the total correlation), denoted as L V AE .
Therefore, the Total Loss is L =
L V AE + γ I L I , where γ I is the weight of Isomorphism Loss.
We denote the above VAE-based implementation as Groupiﬁed VAE.
E XPERIMENTS
We ﬁrst verify the effectiveness of Groupiﬁed VAE quantitatively in learning disentangled representations on several datasets and several VAE-based models.
Then, we show its effectiveness qualitatively on two typical datasets.
After that, we perform a case study on the dSprites dataset to analyze the effectiveness, and conduct ablation studies on the losses and hyperparameters.
For the performance comparison of two downstream tasks (abstract reasoning Van Steenkiste et al.
and fairness evaluation Locatello et al. (2019a)), and more comprehensive results, please see Appendix I.
5. D ATASETS AND B ASELINE M ETHODS
To evaluate our method, we consider several datasets: dSprites (Higgins et al., 2017), Shapes3D (Kim & Mnih, 2018), Cars3D (Reed et al., 2015), and the variants of dSprites introduced by Locatello et al.
(Locatello et al., 2019b): Color-dSprites and Noisy-dSprites.
Please refer to Appendix G for the details of the datasets.
We choose the following four baseline methods as representatives of the existing VAE-based models, which are denoted as Original VAEs.
We verify the effectiveness of our implementation based on those methods.
β -VAE (Higgins et al., 2017) introduces a hyperparameter β in front of the KL regularizer of the VAE loss.
It constrains the VAE information capacity to learn the most efﬁcient representation.
AnnealVAE (Burgess et al., 2018) progressively increases the bottleneck capacity so that the encoder learns new factors of variation while retaining disentanglement in previously learned factors.
FactorVAE (Burgess et al., 2018) and β -TCVAE (Chen et al., 2018)
both penalize the total correlation (Watanabe, 1960), but estimate it with adversarial training (Nguyen et al., 2010;
Sugiyama et al., 2012) and Monte-Carlo estimator respectively.
[Table]
We evaluate β -VAE, AnnealVAE, FactorVAE, and β -TCVAE on dSprites, Cars3d, Shapes3d, NoisydSprites, and Color-dSprites for 1800 settings.
These settings include different random seeds and hyperparameters, refer to Appendix G for the details.
We only show the ﬁrst three datasets here.
For more results, please refer to Appendix I.
5. Q UANTITATIVE E VALUATIONS
This section performs quantitative evaluations on the datasets and models introduced with different random seeds and different hyperparameters.
Then, we evaluate the performance of the Original and Groupiﬁed VAEs in terms of several popular metrics: BetaVAE score (Higgins et al., 2017),
DCI disentanglement Eastwood & Williams (2018) (DCI in short), MIG (Chen et al., 2018), and
FactorVAE score (Kim & Mnih, 2018).
We assign three or four hyperparameter settings for each model on each dataset.
We run it with ten random seeds for each hyperparameter setting to minimize the inﬂuence of random seeds.
Therefore, we totally run ((3 × 10 × 3 + 10 × 3 × 3) × 2) × 5 =models.
We evaluate each metric’s mean and variance for each model on each dataset to demonstrate the effectiveness of our method.
As shown in Table 1, these Groupiﬁed VAEs have better performance (numbers marked bold in Table 1) than the Original VAEs on almost all the cases.
On Shapes3d, the Groupiﬁed VAEs outperform the Original ones on all the metrics except for
BetaVAE scores, suggesting some disagreement between BetaVAE scores and other metrics.
Similar disagreement is also observed between the variances of MIG and other metrics on Cars3d.
Note that the qualitative evaluation in Appendix J shows that the disentanglement ability of Groupiﬁed VAEs is better on Shapes3d and Cars3d.
[Figure]
O r i g i n a l
G r oup
i ﬁ e d
5. Q UALITATIVE E VALUATIONS
We qualitatively show the Groupiﬁed VAEs achieve better disentanglement than the Original ones.
As shown in Figure 2, the traversal results of Groupiﬁed β -TCVAE on Shape3d and Car3d are less entangled.
For more qualitative evaluation, please refer to Appendix J. To verify that the Groupiﬁed
VAEs learn a cyclic representation space (where n = 10 ), we provide the traversal results of [0,18] with a step of 2 for both the Groupiﬁed and Original β -TCVAE on Shape3d in Figure 3.
We observe that the traversal results of Groupiﬁed VAEs are of high quality with a period of 10 (equal to n ).
However, the Original VAEs generate low-quality images without cyclicity.
For the comparison of the results on CelebA (real-world datasets), please see appendix J.
5. V ISUALIZATION OF THE L EARNED R EPRESENTATION S PACE
To understand how our theoretical framework helps the existing VAE-based models to improve the disentanglement ability, we take dSprites as an example, visualize the learned representation space, and show the typical score distributions of the metrics.
First, we visualize the space spanned by the three most dominant factors (x position, y position, and scale).
As shown in Figure 5 (for more results, please refers to Appendix L), the spaces learned by the Original
VAEs collapse, while the spaces of the Groupiﬁed VAEs only bend a little bit.
The main reason is that the Isomorphism Loss, serving as a self-supervision signal, suppresses the representation space distortion and encourages the disentanglement of the learned factors.
As Figure 4 shows, the Groupiﬁed VAEs consistently achieve better mean performance with smaller variances.
The isomorphism reduces the search space of the network so that the Groupiﬁed VAEs converge to the ideal disentanglement solution.
[Figure]
[Figure]
Factor Size n =n =n =n =w/o Abel w/o Order
Groupiﬁed
[Table]
[Figure]
The increase of C (a hyperparameter of AnnealVAE) results in a collapse of representation space of the Original VAE.
The collapse is suppressed by the Isomorphism Loss, which leads to better disentanglement.
5. A BLATION S TUDY
We perform an ablation study on the assumed total number of possible values for a factor (factor size) n , Abel Loss L a , and Order Loss L o .
We take the AnnealVAE trained on dSprites as an example.
We only consider the DCI disentanglement metric here.
We investigate the inﬂuence of factor size n .
Besides, to evaluate the effectiveness of the two constraints, the models with the Abel Loss alone or
Order Loss alone added are also evaluated.
In this setting, we ﬁx n to 10.
We compute the mean and variance of the performance for 30 settings of hyperparameters and random seeds.
Table 2 shows that the isomorphism plays a role of cycle consistency in the representation space, leading to better disentanglement.
The performance is robust to the factor size n , as the models learn to adapt to different n in the training process.
The models with only the Abel Loss or Order Loss applied have improved performance compared to the originals.
The former (Abel Loss) performs better than the latter, suggesting that commutativity plays a more important role.
Note that the number of factors m can be learned and is not a hyperparameter.
See Appendix F for details.
γ I is empirically set to 1 .
C ONCLUSION
In this paper, we have opened the possibility of applying group-based deﬁnition to unsupervised disentanglement by proposing a theoretical framework.
The group structure and model constraint in the framework provide effective inductive bias for existing VAE-based unsupervised disentanglement methods.
In addition, by establishing the feasibility of learning the representation conforming to the deﬁnition in unsupervised settings, we have exhibited the consistently better mean performance with lower variance attributed to the deﬁnition.
We believe our work constitutes a promising step towards unsupervised disentanglement with theoretical guarantee.
As to the limitation, we make a trade-off on the data constraint by proposing to use minimization of total correlation as a necessary condition, in exchange for no supervision.
In addition, the time complexity of Groupiﬁed VAEs is
O ( m 2 ) , where m is the number of factors.
For future work, a natural extension of our framework is to use lie group Hall (2015) (which is also a manifold) to extend our framework.
A P RELIMINARIES
Group : A set G together with a binary operation ◦ as: ◦ : G × G → G satisfying the following properties:
• Associativity: ∀ a, b, c ∈ G, s.t.
( a ◦ b ) ◦ c
= a ◦ ( b ◦ c ) .
• Identity: ∃ e ∈ G, s.t. ∀ a ∈ G, e ◦ a = a ◦ e = a. •
Inverse: ∀ a ∈ G, ∃ a − 1 ∈ G : a ◦ a − 1 = a − 1 ◦ a = e.
It is customary to represent a group with a set G and the binary operation ◦ as a pair ( G, ◦ ) .
When the binary operation is clear, we represent group ( G, ◦ ) as G and use multiplication to represent the binary operation ◦ , i.e., a ◦ b = ab, ∀ a, b ∈ G .
Group Action : Let ( G, ◦ ) be a group and P be a set.
By the group actions of ( G, ◦ ) on P , we mean a mapping: · P : G × P → P, (6) such that
• ∀ a, b ∈ G, p ∈ P, ( a ◦ b ) · p = a · ( b · p ) .
• e · p = p , where e is the identity element of G .
Symmetry Group and Permutation Group : Let Σ be a nonempty set, the bijections from Σ to itself are called Permutations .
S (Σ) denotes the set containing all the permutations on Σ .
S (Σ) forms a group under the binary operation: composition of functions, which is called Symmetry Group .
A subgroup of S (Σ) is called Permutation Group .
Abelian Group :
If the commutative law ( ∀ a, b ∈ G, a ◦ b = b ◦ a ) holds in a group ( G, ◦ ) , such a group is called an abelian group.
Subgroup :
If a subset H of a group G is itself a group under the operation of G , we say that H is a subgroup of G .
3      2      1      4      2      3      4      1𝑖𝑑 = 𝑓 (cid:2868) 𝑟 (cid:2868) (1432) =
𝑓 (cid:2868) 𝑟 (cid:2869) (13)(24) = 𝑓 (cid:2868) 𝑟 (cid:2870) (1234) = 𝑓 (cid:2868) 𝑟 (cid:2871) (14)(23) = 𝑓 (cid:2869) 𝑟 (cid:2868) (24) = 𝑓 (cid:2869) 𝑟 (cid:2869) (12)(34) = 𝑓 (cid:2869) 𝑟 (cid:2870) (13) = 𝑓 (cid:2869) 𝑟 (cid:2871)
Equivalence Relation : An equivalence relation on a set B is a subset U ⊂ B × B satisfying: (It is customary to represent ( a, b ) ∈ U as a ∼ b )
• Reﬂexive: ∀ a ∈ B, ( a, a ) ∈ U .
• Symmetric:
( a, b ) ∈ U ⇐⇒ ( b, a ) ∈ U .
• Transitive: ( a, b ) ∈ U and ( b, c ) ∈
U ⇒ ( a, c ) ∈ U .
Equivalence Class : Let ∼ be an equivalence relation on a set B .
We take an a ∈ B , and then the equivalence class containing a is the subset a = ∆ { b ∈ C | b ∼ a } ⊂ B .
Homomorphism Let ( G, · ) and ( H, ◦ ) be two groups.
A homomorphism f , from G to H , is a mapping f : G → H , such that f ( x · y ) =
f ( x ) ◦ f ( y ) , ∀ x, y ∈ G .
Isomorphism : A homomorphism f : G → H which is bijective is called an isomorphism.
Two groups are said to be isomorphic if there exists an isomorphism between them.
Related Groups :
( i ) Additive group of integers modulo n : Let n denote a positive integer.
We deﬁne an equivalence relation on Z as a ∼ b ⇔ a = b ( mod n ) ( a and b have the same remainder modulo n ).
The relation divides Z into n equivalent classes: 0 , 1 , . . . , n − 1 , where i represents the equivalent class containing i , i.e., i = { m ∈ Z | m =
i ( mod n ) } .
Let Z n = { 0 , 1 , . . .
, n − 1 } , and Z n forms a group under the binary operation: a + b = a + b , denoted by Z /n Z .
( ii ) Group of n -th root unity : Let n denote a positive integer and C n = { e 2 πian | 0 ≤ a ≤ n − 1 , a ∈ Z } be n roots of x n = 1 , and then C n forms a group under complex multiplication.
For the mapping: f : C n → ( Z n , +) deﬁned by f ( e 2 πian )
= a , we have
[Equation]
Therefore, f is a homomorphism, and thus is an isomorphism.
Congruence Class : Let n denote a positive integer, and we deﬁne an equivalence relation on Z as a ∼ b ⇔ a = b ( mod n ) .
The relation divides Z into n equivalent classes: 0 , 1 , . . . , n − 1 , which are congruence classes and the elements of the additive group of integers modulo n .
Subgroup Generated by B : If B is a nonempty subset of the group ( G, ◦ ) , the set deﬁned by < B > = { a ∈ G | a = a 1 ◦ a 2 ◦ · · · ◦ a n with either a i ∈ B or a − 1 i ∈ B } forms a subgroup of G , which is called the subgroup generated by B , e.g., G = < a, b | a 2 = e, ab = ba > in the toy example of Section 1.
Here a 1 , . . .
a n are called Generators , e.g., a, b in G .
Cyclic Group A group G is called cyclic if there is an element a ∈ G such that G = { a n | n ∈ Z } = < a > , e.g., Z /n
Z .
Such an element a is called a generator of G .
Symmetry :
A symmetry of a geometric ﬁgure is a rearrangement of the ﬁgure preserving the arrangement of its sides and vertices as well as its distances and angles.
Dihedral Group :
The permutation group formed by the symmetries of a regular n -sided polygon, denoted by D n .
D 4 as an example : The vertices of a square are numbered by { 1 , 2 , 3 , 4 } , which is analogous to an image dataset, the symmetries are analogous to image transformations.
We often abbreviate permutation 1 → 2 , 2 → 3 , 3 → 4 , 4 → 1 as (1234) .
The elements of group D 4 are shown in Figure 6, from which we know that all of the transformations are compounded by two basic permutations: horizontal ﬂip f and rotate 90 degrees clockwise r .
Please note that f and r are analogous to disentangled factors.
What’s more, the Group can be generated by these basic permutations f, r with some properties, i.e., D 4 = < f, r | f 2 = 1 , r 4 = 1 , fr = r − 1 f > .
The constraints f 2 = 1 , r 4 = 1 , fr = r − 1 f are analogous to the group constraints in this paper.
B P ROOF FOR T HEOREM
In our setting, the group-based deﬁnition is equivalent to the equivariant condition g · f ( w ) =
f ( g · w ) for the following reason.
Since the group actions of G on Z is the element wise addition, ( a ) the group action of G on Z exists and ( b ) group actions of G i only affect Z i (refer to Section 3.2).
These two conditions of the group-based deﬁnition thus hold.
Proof.
In the following, we prove in Step 1 that the necessary and sufﬁcient conditions of Φ playing the “agent” role ,
i.e., f ( g · w ) =
h ( ϕ · b ( w )) , are ( i ) Φ is isomorphic to G , and ( ii ) there exist
ϕ i ∈
Φ , s.t. ϕ
i · b ( w ) =
b ( g i · w ) , i = 1 , 2 , . . .
, m .
We then prove in Step
that if equation f ( g · w ) = h ( ϕ · b ( w )) and the deﬁnition of ϕ ∈ Φ hold, then Z is disentangled per group-based deﬁnition.
Step 1 :
In the following, we prove that the necessary and sufﬁcient conditions of f ( g · w ) =
h ( ϕ · b ( w )) are ( i ) and ( ii ) above.
Note that this holds even without a speciﬁc deﬁnition of Φ .
⇒ )
For a general permutation group Φ on O , we assume there exist ϕ i such that ϕ
i · b ( w ) =
b ( g i · w ) , i = 1 , 2 , . . .
, m .
Assume that there exists an isomorphism τ : G → Φ , where group
G =
( Z /n Z ) m .
From Group Theory, we know that there exists µ ∈ Aut (Φ) such that the following equation holds, where Aut (Φ) denotes the group of automorphisms of Φ , i.e., µ is an isomorphism µ : Φ → Φ .
We denote the composition of µ, τ as σ , and have
[Equation]
Since we have ϕ i · b ( w )
=
b ( g i · w ) , i = 1 , 2 , . . .
, m , the following equation holds,
[Equation]
Since both µ, τ are isomorphism, σ = µ ◦ τ is also an isomorphism, which indicates that ϕ
i is a generator of Φ .
For ∀ ϕ ∈ Φ , since σ is a bijection, there exists g ∈ G =
( Z /n Z ) m , such that σ ( g ) =
ϕ and g = (cid:80)
i k
i g i , k i ∈
Z , where g i , i = 1 , 2 , . .
, m denotes the generators in G .
In order to calculate ϕ · b ( w ) , we consider a speciﬁc example σ ( g 1 + g 2 ) .
[Equation]
Therefore, for the general element ϕ , we have
[Equation]
In general, we have ϕ · b ( w ) =
b ( g · w ) and thus we have h ◦ b ( g · w ) =
f ( g · w ) =
h ( ϕ · b ( w )) when taking h on both sides.
⇐ ) Here we prove that ( i ) G is isomorphic to Φ and ( ii ) there exists ϕ i such that ϕ i · b ( w )
=
b ( g i · w ) , i = 1 , 2 , . . .
, m are two necessary conditions of equation f ( g · w )
=
h ( ϕ · b ( w ))
=
h ( ϕ · x ) .
We take f − 1 on both sides of the equation.
For convenience, we rewrite g · w, ϕ · b ( w ) as g ( w ) and ϕ ( b ( w )) , then we have
[Equation]
Note that the notation ◦ here denotes the composition of functions.
We deﬁne the mapping τ between Φ and G as follows:
[Equation]
Note that b is a bijection, thus τ is a bijection.
We take ϕ
i , ϕ j ∈ Φ and we have
[Equation]
Therefore, τ is a homomorphism and thus is an isomorphism.
i.e., Φ is isomorphic to G .
Recall that group G has the form of G =
( Z /n Z ) m , where it is a direct product of m cyclic groups.
For ∀ ϕ ∈ Φ , we have
[Equation]
For generators g i ∈ G, i = 1 , 2 , . . .
, m and the corresponding ϕ i
= τ − 1 ( g i ) , we have a speciﬁc one derived from the above equation:
[Equation]
Q.E.D.
Step 2 : As discussed in Section 1 and Section 3, we deﬁne ϕ as ϕ · x = ϕ · b ( w ) =
h − 1 ( σ − 1 ( ϕ ) · h ( x ))
=
h − 1 ( g · f ( w )) , where σ is the same as in Step 1.
Since σ is a bijection, for ∀ ϕ ∈ Φ , σ − 1 ( ϕ ) uniquely exists, and thus ϕ is well-deﬁned.
We bring it into f ( g · w )
=
h ( ϕ · b ( w )) , and derive f ( g · w ) =
g · f ( w ) , i.e., the representation space Z is disentangled with respect to G .
Speciﬁcally, for the existing ϕ
i , i = 1 , 2 , . . .
, m , from equation 8, we have σ − 1 ( ϕ i )
=
g i and ϕ i · x = h − 1 ( g i · f ( w )) .
C P ROOF FOR T HEOREM
In order to easily distinguish a general element and a generator, we use φ to stand for the general element in Φ and ϕ i to stand for generators in Φ .
In addition, for mapping τ : G → Φ , notation ker ( τ ) is generally the inverse image of e , i.e., ker ( τ ) = { g ∈ G | τ ( g ) =
e } , where e is the identity element in Φ .
Proof.
Note that Section 3.4 describes the implementation of the elements in Φ , but there is no guarantee that Φ is a group.
Therefore, here we ﬁrst prove in Step 1 that set Φ = < ϕ 1 , ϕ 2 , . . .
, ϕ m | ϕ
i ϕ j
= ϕ
j ϕ i , ϕ ni
= e, i = 1 , 2 , . . .
, m and j = 1 , 2 , . .
, m > forms a group under the composition of functions, where e satisﬁes that eϕ i = ϕ
i , then prove in Step 2 the necessary and sufﬁcient condition for the isomorphism is to meet both the Abel constraint and the Order constraint.
Step 1 :
To prove a set forms a group under some operation, we only need to verify that the elements of the set satisfy the following three properties: 1.
Associativity 2.
Identity 3.
Inverse.
We ﬁrst verify the Associativity property: ∀ φ s , φ t , ϕ l ∈ Φ , s.t.
( φ s φ t ) φ l = φ s ( φ t φ l ) .
Recall that in Section 3.2, the representation space is a set with the same elements in G .
The group action of G on Z is the addition.
We have g · z ∈ Z, ∀ g ∈ G, z ∈ Z , and then we have d ( g · z ) ∈ O, ∀ g ∈ G, z ∈ Z .
Therefore, ϕ i is a permutation: ϕ i : O → O , i.e., ϕ i ∈ S ( O ) , where
S ( O ) is the symmetry group on images set O , then for ∀ ϕ
i , ϕ j , ϕ k ∈ { ϕ 1 , ϕ 2 , . . .
ϕ m } , we have
[Equation]
the generators of Φ thus satisfy the Associativity property.
However, whether the general elements of Φ satisfy the Associativity property is unknown.
We take ∀ φ s , φ t , φ l ∈ Φ , where φ t = (cid:81) i ϕ
t i
i , φ s = (cid:81) i ϕ
s i
i , φ l = (cid:81) i ϕ
l
i
i and t i , s i , l i ∈ Z , and we have
[Equation]
Therefore, the set of mappings Φ satisﬁes the Associativity property under the composition of functions.
We then verify the Identity property: ∃ e ∈ Φ , s.t. ∀
φ ∈ Φ , eφ = φe = φ .
For generators ϕ i , ϕ j ∈ { ϕ 1 , ϕ 2 , . . .
ϕ m } , we have ϕ ni ϕ j = eϕ
j
=
ϕ
j .
Therefore, for general elements, we take ∀ φ ∈ Φ , φ = (cid:81)
i ϕ k
i i , and we have:
[Equation] [Equation]
This states that the identity element of Φ is e = ϕ ni ∈ Φ .
Therefore, the set of mappings Φ satisﬁes the Identity property.
We ﬁnally verify the Inverse property: ∀ φ ∈ Φ , ∃ φ − 1 ∈ G : φφ − 1 = φ − 1 φ = e .
For the generators ϕ i ∈ { ϕ 1 , ϕ 2 , . . .
ϕ m } , we have ϕ ki ϕ n − k i = ϕ
ni = e, 1 ≤ k ≤ n, k ∈ Z .
For any general element φ ∈ Φ , φ = (cid:81) i ϕ k
i i , we have:
[Equation]
we denote (cid:81) i ϕ n − k
i i as φ − 1 and have φφ − 1 = e .
Similarly, we also have φ − 1 φ = e .
These two equations state that for any general element φ ∈ Φ , φ = (cid:81)
i ϕ k
i i , we have an inverse element φ − 1 =
(cid:81)
i ϕ n − k
i i in Φ .
Therefore, the set of mappings Φ satisﬁes the Inverse property.
To summarize, Φ is a group.
Q.E.D.
Step 2 :
In this step, we prove that the necessary and sufﬁcient condition for the isomorphism ( Z /n Z ) m ∼ < ϕ 1 , ϕ 2 , . . .
, ϕ m > is the satisfaction of ϕ i ϕ j = ϕ
j ϕ i , i = 1 , 2 , . . .
, m and j = 1 , 2 , . .
, m , and ϕ n i =
e, i = 1 , 2 , . . .
, m .
⇒ )
Assume the isomorphism is τ : G =
( Z /n Z ) m → Φ = < ϕ 1 , ϕ 2 , . .
. , ϕ m > , for generators ϕ
i , ϕ j ∈ { ϕ 1 , ϕ 2 , . . .
ϕ m } , there exist g α = (cid:80) i
α
i g i , g β =
(cid:80) i β i
g i ∈ G, s.t.
τ ( g α ) =
ϕ i , τ ( g β ) =
ϕ j , where g i , i = 1 , 2 , . . .
, m denotes the generators in G , which is the m-dimensional one-hot vector (0 , . . .
. , 0) ∈ G with 1 in position i and 0 elsewhere, and α i , β i ∈
Z .
Since G is an Abelian group, we have
[Equation]
For generators ϕ i ∈ { ϕ 1 , ϕ 2 , . . .
ϕ m } , we have g α = (cid:80)
i
α
i g i ∈ G, s.t.
τ ( g α ) =
ϕ
i , the n times composition of itself is
[Equation]
where e G , e are identity elements of G and Φ respectively.
In Equation 22, τ ( e G ) = e holds because τ is an isomorphism, and ker ( τ ) = { e G } .
The sufﬁciency is proven.
⇐ ) In the following, we prove that when two conditions are satisﬁed simultaneously, the mapping τ we deﬁne is an isomorphism.
Considering the mapping τ : Φ → G , deﬁned as
[Equation]
For general elements φ t , φ s ∈ Φ , where φ t = (cid:81) i ϕ
t i
i , φ s = (cid:81) i ϕ
s i
i and t i , s i ∈
Z , we have
[Equation]
Note that these terms can be merged since ϕ
i ϕ j
= ϕ
j ϕ i .
The summation on the right side of
Equation 24 is partitioned into two parts as follows
[Equation]
Consequently, we have τ ( φ t φ s ) =
τ ( φ t ) + τ ( φ s ) , which states that τ is a homomorphism.
Then, we only need to prove that mapping τ is bijective.
First, we prove τ is injective, i.e., ker ( τ ) = { e G } , after that, we prove τ is surjective, i.e., one can ﬁnd the inverse image of any element.
Since it is not hard to obtain τ ( φ )
=
τ ( e · φ )
=
τ ( e ) + τ ( φ ) , we have τ ( e ) =
=
e G , i.e., e ∈ τ − 1 ( e G ) .
Assume there is φ l = (cid:81)
i ϕ
l i
i , s.t. τ ( φ l ) =
e G , we have
[Equation]
are divisible
[Equation]
The equation above states that τ − 1 ( e G ) =
{ e } ( τ : Φ → Φ ) and mapping τ is injective.
We take ∀ g ∈ G, g = ( k 1 , k 1 , . . .
, k m ) ∈ ( Z /n Z ) m , and have
[Equation]
Hence, we have τ − 1 ( g ) =
(cid:81)
i ϕ k
i i ∈
Φ , which indicates that mapping τ is surjective.
Since mapping τ is bijective and homomorphism, τ is an isomorphism.
Q.E.D. D P ROOF FOR T HEOREM
Proof.
In the following, we prove that the necessary and sufﬁcient condition for satisfying Abel and
Order constraints is that both Abel and Order Loss are minimized.
⇒ )
For the condition: ∀ ϕ i , ϕ j ∈ Φ , 0 ≤ i, j ≤ m , we have ϕ
i ϕ j
= ϕ
j ϕ i .
The constraint on any image o ∈
O can be obtained by
[Equation]
For the set of combinations of factors C
= { ( i, j ) | 1 ≤ i, j ≤ m } and the set containing images O , we have
[Equation]
We obtain the Abel loss.
For the Order loss, we ﬁrst consider n times composition of the same mapping ϕ
i , we have
[Equation]
Therefore, we then have ϕ − 1 i
[Equation]
For a single image o ∈
O , we have
[Equation] [Equation]
Thus, for the set of factors and the set containing all images O , we have
[Equation]
minimized
[Equation]
To eliminate the bias of optimization
[Equation]
we optimize the symmetry form of the Order Loss
[Equation]
and we have
[Equation]
is optimized
[Equation]
The Order Loss is obtained.
⇐ ) When the Abel Loss L a is optimized, for ∀ o ∈ O , we have
[Equation]
[Equation]
Therefore, for ∀ ϕ i , ϕ j ∈ Φ , i = 1 , 2 , . . .
, m and j = 1 , 2 , . .
, m , we have ϕ
i ϕ j
= ϕ j ϕ i , and we obtain the Abel constraint.
When the Order Loss L o is minimized, for ∀ o ∈ O , we have
[Equation]
This implies that
[Equation]
Therefore, we obtain the Order constraint.
The Group constraints are satisﬁed.
E T HE D ATA C ONSTRAINT
In this section, we prove that if each dimension of the world state is independently sampled ( p ( w )
=
Π i p ( w i ) , where p denotes the probability mass function (for discrete random variable) or probability density function (for continuous random variable)), then the minimization of total correlation ( p ( z ) =
Π i p ( z i ) ) is a necessary condition to satisfy the data constraint (see Theorem 4 and Theorem 5 below).
Our goal is to learn a representation z conforming to the group-based deﬁnition of disentanglement with an encoder h and a decoder d .
Then the data constraint can be formulated as follows.
For a generator g i ∈ G , we have b ( g i · w ) =
ϕ
i · b ( w ) =
h − 1 ( g i · z ) , where z = f ( w ) and ϕ
i is the corresponding permutation of g i under the isomorphism between G and Φ .
We reorganize the formula and have h ◦ b ( g i · w ) =
f ( g i · w ) =
g i · z .
Theorem 4 Assume that the action of generator g i on z is g i
· z = z
+ g i (element-wise addition), and the action of each generator g i on w only affects a single dimension of w (see the assumptions in
Section 3.2).
Then the equation f
( g i · w ) =
g i · z, ∀ w ∈ W is equivalent to: for each i = 1 , 2 , . . .
, n , there exists a bijective function γ i s.t.
z i = γ
i ( w j ) for some j .
Proof.
⇒ )
Without loss of generality, we assume g i only affects the j -th dimension of w .
If the equation f ( g i · w ) =
g i · z, ∀ w ∈ W holds, it’s obvious that, for each i , there exists a bijective function γ i s.t.
z i = γ
i ( w j ) .
⇐ ) In the following, we use ( g i · w ) i (( g i · z ) i ) to denote the i -th dimension of vector g i · w ( g i · z ) .
If for i = 1 , . . . , n , the functions z i = γ i ( w j ) hold for some j , by using an index permutation
j = π ( i ) we can rewrite the function as z i =
γ i ( w π ( i ) ) .
Therefore, z can be formulated as follows
[Equation]
Please note that how the world state transits on dimension i dose not affect how disentangled the representation is.
Therefore, we deﬁne action of generator g i on w as follows.
[Equation]
Then, we take f on both sides of the above equation, and apply Equation 37, and have f
[Equation]
Note that it is obvious that the following theorem (Theorem 5) holds for discrete random variables and bijective functions.
However, since the world state space is dense and the encoder h and decoder d are differentiable in general, we also prove for the case where w and z are treated as continuous random variables.
Theorem 5 states that the minimization of total correlation ( p ( z )
=
Π i p ( z i ) ) is a necessary condition to make equations z i
= γ i ( w π ( i ) ) satisﬁed, where i = 1 , 2 , . . . , n and π ( i ) is an index permutation.
Theorem 5 For the independent random variables w 1 , w 2 , . . . , w n , considering the functions z i =
δ i ( w 1 , w 2 , . . . , w n ) , where i = 1 , 2 , . . . , n and each δ i is a bijective, differentiable function.
For i = 1 , . . . , n , if there exists an index permutation π ( i ) s.t. , we have ∂z π ( i ) ∂w
i (cid:54) = 0 and ∂z π ( i ) ∂w k = 0 , where k = 1 , 2 . . . , n but k (cid:54) =
i , then the new random variables z 1 , z 2 , . . .
, z n are independent.
Proof.
We treat the random variables w 1 , w 2 , . . .
, w n as an n -dimensional random vector W = ( w 1 , w 2 , . . . , w n ) .
Similarly, we write Z =
( z π (1) , z π (2) , . . .
, z π ( n ) ) , which is rearranged by index permutation π ( i ) .
According to Change of Variable Theorem For Random Vectors (Mood, 1950), we have
[Equation]
where J ( W ) is the jacobian matrix of Z w.r.t.
W , the ( i, j ) -th entry of it is ∂z π ( i )
∂w j .
Since for each i = 1 , . . . , n , we have ∂z π ( i ) ∂w
i (cid:54) = 0 and ∂z π ( i ) ∂w k = 0 , where k = 1 , 2 . . . , n but k (cid:54) =
i , then the jacobian matrix can be formulated as follows
[Equation]
According to Change of Variable Theorem For Random Variable ( Mood (1950)), we have
[Equation]
[Equation]
Bring Equation 41 and Equation 42 into Equation 40, we have
[Equation]
Please note that the inverse proposition of the theorem above does not hold.
As a counterexample, for zero mean independent gaussian random variables w 1 , w 2 with common variance σ 2 , new random variables z 1 = (cid:112) w 2 1 + w 2
(the norm of vector ( w 1 , w 2 ) ), z 2 = tan − 1 ( w 2 /w 1 ) (the angle between vector ( w 1 , w 2 ) and (0 , 1) ) are independent (Mood, 1950) but ∂z i ∂w j (cid:54)
=
for all i, j ∈ { 1 , 2 } .
In addition, the sufﬁciency would be satisﬁed for some speciﬁc settings, e.g., there are only two independent uniformly distributed random variables w 1 , w 2 , and the functions z 1 , z 2 are linear functions.
Combining Theorems 4 and 5, we have that the minimization of total correlation ( p ( z ) =
Π i p ( z i ) ) is a necessary condition to satisfy the data constraint.
F D ETAILS OF I MPLEMENTATION F. A BEL L OSS D ETAILS
As mentioned in the main paper, the Abel Loss of the VAE-based models is as follows:
[Equation]
j ) ∈ C (cid:107) ϕ
i · ( ϕ j · o ) − ϕ j · ( ϕ i · o ) (cid:107) , (44) where ϕ i · ( ϕ j · o ) =
ϕ i ( ϕ j ( o )) represents the top path of Figure 7 (a), and ϕ j · ( ϕ i · o ) =
ϕ j
( ϕ i ( o )) represents the bottom path of Figure 7 (a).
For better optimization, we constrain such consistency on their representation (straight dotted double arrow in Figure 7 (a)) instead of the reconstructed images.
Besides, we constrain the consistency between the representations of intermediate images (curved dotted double arrow in Figure 7 (a)).
[Figure]
F. O RDER L OSS D ETAILS
As mentioned in the main paper, the Order Loss of VAE-based models is as follows:
[Equation]
where ϕ i · ( ϕ − 1 i · o ) represents the lower path of Figure 7 (b), and ϕ − 1 i · ( ϕ i · o ) represents the upper path of Figure 7 (b).
Similar to the Abel Loss, we do not constrain such consistency on the reconstructed images for better optimization but on their representations instead (the long curved dotted line in Figure 7 (b)).
Besides, we constrain the consistency between the representations of intermediate images (short curved dotted lines in Figure 7 (b)).
F. T HE NUMBER OF FACTORS m
For the given VAE-based models, dimensional KL divergence (indicating the meaningful dimensions) increases during the training process.
Therefore, we use m dimensions of which the corresponding dimensional KL divergence KL i ≥ T , where T is a hyperparameter, which is empirically set toin our experiments.
G D ETAILS OF E XPERIMENTS G. D ATASET DETAILS
In all the experiments, we resize the images to 64 × 64 resolution.
We introduce all the datasets used in our paper in detail.
dSprites Higgins et al.
(2017): dSprites contains 737,280 binary 2D shapes (heart, oval and square) images with ﬁve ground truth factors: shape (3 values), scale (6 values), orientation (40 values), x-position (32 values), y-position (32 values).
Then we introduce the variants of dSprites (Color dSprites and Noisy dSprites) created by Locatello et al.
Locatello et al.
(2019b).
Color dSprites , the shapes of the images in dSprites are randomly colored.
Noisy dSprites , the background of the images in dSprites is noise.
Shapes3D Kim & Mnih (2018):
Shapes3D contains 480,000 images of 3D shapes with 6 ground truth factors: shape (4 values), scale (8 values), orientation (15 values), ﬂoor color (10 values), wall color (15 values), object color (10 values).
Cars3D Reed et al.
(2015): This dataset consists of 183 car CAD models, each rendered fromazimuth directions and 4 elevations.
Encoder
Decoder
Input: 64 × 64 × number of channels
Input: 10 or4 × 4 conv, 32 ReLU, stride
FC, 256 ReLU 4 × 4 conv, 32 ReLU, stride
FC, 256 ReLU 4 × 4 conv, 32 ReLU, stride
FC, 4 × 4 × 32 ReLU 4 × 4 conv, 32 ReLU, stride4 × 4 deconv, 32 ReLU, stride
FC 256 ReLU 4 × 4 deconv, 32 ReLU, stride FC 256 ReLU 4 × 4 deconv, 32 ReLU, stride
FC
×4 × 4 deconv, number of channels, stride the decoder is 10.
For Groupiﬁed VAE, the dimension is 20.
Note that the number of representation dimensions of Groupiﬁed VAE is still 10, which is the same as Original VAE, the comparison with
Original VAE is fair Watters et al.
G. A RCHITECTURE FOR ENCODER AND DECODER
We follow Locatello et al.
Locatello et al.
(2019b) to use the same architecture of VAEs in all of the experiments: the activation function used is ReLU except for the last layer of decoder, as shown in
Table 3.
For the details of the Discriminator in FactorVAE, please refer to Table 5 (a) and (c).
[Table]
[Table]
G. E XPERIMENT SETTINGS
We run using different hyperparameters and random seeds for every VAE-based model implemented by Pytorch Paszke et al.
As shown in Table 4, for β -VAE, we assign 3 choices for β andrandom seeds for both the Original and Groupiﬁed VAEs: 3 × 10 × 2 = 60 settings for each dataset.
Similarly
[Table]
we also assign 60 settings for FactorVAE and β -TCVAE.
For AnnealVAE, we assign three choices for C and 3 choices for the start and end pair, also assign 10 random seeds.
In summary, for all 5 datasets, we run (((3 × 10 × 2) × 3) + 3 × 3 × 10 × 2) × 5 = 1800 models.
For other hyperparameters, please refer to Table 5 (b)
[Table]
H R ELATION TO SOME PREVIOUS WORKS H. SYMMETRY - BASED DISENTANGLEMENT
Caselles-Dupr´e et al.
(2019) argue that the symmetry-based disentanglement requires interaction with the environments.
Speciﬁcally, for a given disentangled representation z w.r.t.
a world with some group action on W , there are multiple other worlds (same world states and symmetry) of a static dataset that have different group actions on each dimension of W .
For those worlds, the representation z is not disentangled, per the group-based deﬁnition.
However, in our work, we do not assume that a static dataset has a world equipped with some speciﬁc ﬁxed group action on W in advance, instead we use permutation group Φ as an agent to learn a world with proper group actions.
For this learned world, there is only one representation that satisﬁes the deﬁnition.
Therefore, we do not need an environment to provide group actions on W to determine which world it is.
Please see our proof in Theorem 1.
In the testing phase, since we input the images to the model directly to derive their representation, the disentanglement of the representation does not rely on how the world state transitions between each other (i.e., as a result of group actions on W ).
Therefore, our framework can learn such a disentangled representation without interaction with the environments.
H. L IE GROUP VAE
Zhu et al.
(2021) argue that the representation space being a vector space is sub-optimal since it requires the model to learn to discard different scales of variations.
They propose to use lie group as the representation space instead and use Hessian Penalty to encourage disentanglement.
Our framework is complimentary to their proposed method.
The lie group representation can be applied to extend our framework.
Here we leave it for future work.
In addition, our proposed representation mapped by the sine and cosine function ( exp ((2 πiz ) /n )
for a real vector z ) is also a lie group.
I M ORE Q UANTITATIVE R ESULTS AND S CORE D ISTRIBUTION I. Q UANTITATIVE R ESULTS
The performance of the Original and Groupiﬁed VAEs on all ﬁve datasets is shown in Table 6.
Our method outperforms the original one on most of the datasets in terms of nearly all the metrics
[Table]
We evaluate β -VAE, AnnealVAE, FactorVAE, and β -TCVAE on dSprites, Cars3d, Shapes3d, NoisydSprites, and Color-dSprites for 1800 settings.
These settings include different random seeds and hyperparameters.
I. A BSTRACT R EASONING & F AIRNESS
[Table]
As pointed out by Locatello et al.
Locatello et al.
(2019b), the disentangled representation’s downstream tasks should also be veriﬁed.
Therefore, we verify the effectiveness of the representations learned by the Groupiﬁed VAEs on
Shapes3d in two downstream tasks: abstract reasoning Van Steenkiste et al.
and fairness evaluation Locatello et al. (2019a).
As Tableshows, the performance of the abstract reasoning models ﬁne-tuned on the representation learned by the Groupiﬁed FactorVAEs is better than the original ones.
In terms of fairness evaluation, we can observe that the unfairness scores of the representation learned by the Groupiﬁed FactorVAEs are lower than the Original ones.
I. S CORE D ISTRIBUTION
The detailed distribution of the performance is shown in this section (demonstrated by the Violin
Plot Hintze & Nelson (1998)).
The performance distributions on dSprits, Car3d, Noisy dSprites,
Color-dSprites, and Shapes3d are shown in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12, respectively.
[Figure]
nn ea l
VA E
[Figure]
F ac t
o r
VA E
[Figure]
β - T
C VA
E
[Figure]
βVA E
[Figure]
A nn ea l
VA E
[Figure]
F ac t
o r
VA E
[Figure]
β - T
C VA
E
[Figure]
βVA E
[Figure]
A nn ea l
VA E
[Figure]
F ac t
o r
VA E
[Figure]
β - T
C VA
E
[Figure]
βVA E
[Figure]
A nn ea l
VA E
[Figure]
F ac t
o r
VA E
[Figure]
β - T
C VA
E
[Figure]
βVA E
[Figure]
A nn ea l
VA E
[Figure]
F ac t
o r
VA E
[Figure]
β - T
C VA
E
[Figure]
J M ORE Q UALITATIVE R ESULTS
We evaluate our methods qualitatively on two typical datasets: Cars3d and Shapes3d.
We visualize the traversal results of the Original and Groupiﬁed FactorVAE and β -TCVAE.
For every factor, we traverse ﬁve randomly sampled representations.
As shown in Figure 13 and Figure 14, the traversal results of the Groupiﬁed FactorVAE and β -TCVAE show that these models learn less entangled representations on Shapes3D (e.g., Orientation of FactorVAE and Scale and Shape of β -TCVAE).
Similarly, as shown in Figure 15 and Figure 16, the Groupiﬁed FactorVAE and β -TCVAE achieve better disentanglement ability on Car3d (e.g., Rotation of FactorVAE and Yaw of β -TCVAE).
For the real-world dataset, we show the qualitative comparison of the Groupiﬁed and original
AnnealVAE trained on CelebA.
As Figure 17 shows, for most of the factors, our model can extract cleaner semantics than the original model.
For example, the hair color is entangled with the face shape in the original model, but cleaner in the Groupiﬁed model.
[Figure]
The traversal range is (-2, 2).
[Figure]
The traversal range is (-2, 2).
O r i g i n a l
G r oup
i ﬁ e d
[Figure]
O r i g i n a l
G r oup
i ﬁ e d
[Figure] [Figure]
The traversal range is (-2, 2).
The factors learned by Groupiﬁed model are less entangled.
K M EANINGFUL D IMENSION V ISUALIZATIONS
When we assign some dimensions to Isomorphism Loss for Groupiﬁed AnnealVAEs, e.g., the ﬁrstdimensions, we have an interesting observation that the assigned dimensions are meaningful in
Groupiﬁed AnnealVAEs.
The KL divergence increases continuously on these assigned dimensions after the Isomorphism Loss is applied to them.
Note that the KL divergence loss in AnnealVAE indicates the amount of information encoded.
As Figure 18 shows, the KL divergence of assigned dimensions increases at the beginning of training, which means Isomorphism Loss results in that the assigned dimensions become meaningful.
Finally, the assigned ﬁrst-ﬁve dimensions learn to encode the semantics of x position, y position, scale, and orientations.
The possible reason is that the latent factors are learned and disentangled in the assigned dimensions due to the punishment of the Isomorphism Loss.
To illustrate that controllable dimensions in Groupiﬁed AnnealVAE are not an exception, we provide more visualizations.
The results of covering all hyperparameters settings with two random seeds are shown in Figure 18 to Figure 26, suggesting that their dimensions are controllable.
[Figure]
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So are (d), (e), and (f), which are results of a run with a different random seed.
[Figure]
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different amounts after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So as (d), (e), and (f), which are results of a run with a different random seed.
[Figure]
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So are (d), (e), and (f), which are results of a run with a different random seed.
[Figure]
The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a
The KL divergences in different dimensions are different amounts after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So as (d), (e), and (f), which have a different random seed.
[Figure]
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So are (d), (e), and (f), which are results of a run with a different random seed.
[Figure]
The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So are (d), (e), and (f), which are results of a run with a different random seed.
[Figure]
The KL divergences of the target dimension (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So are (d), (e), and (f), which are results of a run with a different random seed.
[Figure]
The KL divergences of the target dimension (0-4 dimension) increase one by one during training (
The KL divergences in different dimensions are different after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So are (d), (e), and (f), which are results of a run with a different random seed.
[Figure]
The KL divergences of target dimensions (0-4 dimension) increase one by one during training (a).
The KL divergences in different dimensions are different after training (b).
As the image traversal results (c) show, the meaningful dimensions are learned in 0-4 dims.
So are (d), (e), and (f), which are results of a run with a different random seed.
L M ORE R EPRESENTATION S PACE V ISUALIZATIONS
To illustrate that the Isomorphism Loss suppresses the representation space collapse in VAE-based methods is not an exception, we provide more visualizations of the representation space visualization of Groupiﬁed VAEs.
The Original and Groupiﬁed β -VAEs, AnnealVAEs, FactorVAEs
[Figure]
and βTCVAEs are shown in Figure 27, Figure 28, Figure
[Figure]
and Figure 30, respectively, which implies that the representation space of Groupiﬁed VAEs are organized better than the original ones.
[Figure]
[Figure] [Figure]
[Figure] [Figure]