Boxhead: A Dataset for Learning Hierarchical Representations
Abstract
Disentanglement is hypothesized to be beneﬁcial towards a number of downstream tasks.
However, a common assumption in learning disentangled representations is that the data generative factors are statistically independent.
As current methods are almost solely evaluated on toy datasets where this ideal assumption holds, we investigate their performance in hierarchical settings, a relevant feature of realworld data.
In this work, we introduce Boxhead , a dataset with hierarchically structured ground-truth generative factors.
We use this novel dataset to evaluate the performance of state-of-the-art autoencoder-based disentanglement models and observe that hierarchical models generally outperform single-layer VAEs in terms of disentanglement of hierarchically arranged factors
Introduction
A common assumption in representation learning is that high-dimensional observations are generated from lower-dimensional generative factors [1].
Disentanglement aims to identify these factors which have statistically independent and semantically meaningful sources [2, 3].
From a causal perspective, disentanglement aims to learn the independent causal mechanisms of the system [4–6].
The possible beneﬁts of disentangling such mechanisms have been investigated in the context of interpretability [7, 8], faster downstream learning [9], fairness [10–12], continual learning [13], and generalization [14–23].
State-of-the-art disentanglement methods are typically based on variational autoencoders (VAEs) with a single latent layer, assume independence of the latent factors [24], and are applied to toy datasets with independent generative factors [3, 25–29].
However, the generative process of real-world data is often hierarchical, i.e. variables describing low-level features are determined by higher-level variables.
VAE-based disentanglement methods with a hierarchical structure in the latent variables such as
VLAE [30] and pro-VLAE [31] have been shown to capture to some extent multiple hierarchical levels of detail in data.
However, they are not evaluated on datasets where we have a clear understanding of the hierarchical structure underlying the data.
This leads to ambiguities when assessing and analyzing the hierarchical structure learned by these models, as there is no ground truth to compare against.
Having a dataset with a known, yet simple hierarchical arrangement of generating factors of variation could enable a more controlled and sound quantitative evaluation.
With this paper we introduce such a hierarchically structured dataset 2 to accurately benchmark representation learning methods in this setting.
We use this dataset to evaluate prominent and representative VAE-based disentanglement approaches.
In addition, we introduce pro-LVAE, a hierarchical representation learning approach that is competitive with the other methods.
We summarize our contributions as follows:
• We introduce Boxhead , a new dataset for learning hierarchically structured representations that has a known hierarchical dependency structure in its factors of variation.
• We propose pro-LVAE, a modiﬁcation of Ladder VAEs (LVAEs)
[32, 33] with a training scheme similar to pro-VLAE.
• We present an empirical study on this new dataset, comparing pro-LVAE with various state-of-the-art disentanglement approaches ( β -VAE, β -TCVAE, pro-VLAE).
We evaluate all models in terms of (i) ELBO, (ii) disentanglement for higher-level (macro) FoVs
and (iii) disentanglement for lower-level (micro) FoVs.
Models with a hierarchical structure seem to outperform single-layer VAEs on each of these metrics.
Background
Variational Autoencoders (VAEs) are latent variable models that aim to model high-dimensional data x as arising from lower-dimensional latent representation z .
VAEs are optimized by deﬁning an approximate posterior distribution q φ ( z | x ) and maximizing the ELBO:
[Equation]
which is a lower bound to the log-likelihood log p θ ( x ) , where p ( z ) is typically a standard isotropic
Gaussian.
In this work, we will focus on VAEs with (i) one layer of latent variables, (ii) a hierarchy of latent variables given only by the neural architecture, and (iii) hierarchical latent variable models (generative models with a hierarchically structured latent space).
Disentangled representation learning methods aim to learn a representation where the coordinate axes are aligned with the data generative factors, also known as the factors of variation (FoV).
Common disentanglement models are single-layer VAEs.
They assume statistically independent generative factors and aim for achieving a statistically independent representation by regularizing the latent space.
The diagonal covariance prior and stochastic reconstruction loss forces local orthogonality of the coordinate axes, promoting disentanglement of the learned representations [34].
β -VAEs [27] aim at further encouraging independence in the learned factors by increasing the capacity constraint through β .
Rather than constraining the capacity, models such as β -TCVAE [3] and FactorVAE [26] explicitly minimize the total correlation of the latent representation, thus maintaining a relatively better generation quality.
It has also been argued that annealing the constraint on the capacity can be beneﬁcial for the reconstruction-disentanglement trade-off in β -VAEs [2].
Hierarchical disentanglement methods such as VLAE [30], pro-VLAE [31], and SAE [35], model factors that contribute the most to the reconstruction loss in the higher (macro feature) layers, and ﬁner (micro feature) details in the lower layers.
However, the latent variables are independent, i.e. the latent prior is fully factorized, so from a probabilistic perspective they are effectively standard VAEs.
Notably, pro-VLAE progressively introduces the latent layers during training [31].
This progressive training allows higher latent layers to converge before moving onto the next layers.
These higher layers (layer L being the highest) will learn the more abstract macro-level factors, i.e., factors that have a larger effect on the reconstruction loss.
This is similar in effect to AnnealedVAE [2, 34].
Finally, hierarchical latent variable models such as LVAE [32], BIVA [33], NVAE [36], SDNs [37], and Very Deep VAEs [38], are more expressive generative models in that their prior does not fully factorize, and they explicitly model statistical dependencies between latents at different layers.
Deﬁning hierarchy.
Hierarchy is a ubiquitous characteristic in real-world data.
However, the exact deﬁnition of hierarchy is ambiguous [8, 39–46].
One commonality is that base primitives are used to construct higher-level representations.
Roughly, models which focus on learning hierarchical relationships either utilize base primitives to construct higher-level representations [8, 40, 42, 45] or decompose a high-level object into base primitives [39, 41, 44].
The type of hierarchy we will focus on is better deﬁned in causality research as the relationship between micro and macro level factors.
Micro-level factors serve as ﬁner detailed factors and macro-level factors serve as the coarse-grained factors.
Both macro and micro level factors model the same given system and must be causally consistent [47].
Macro-level factors are the aggregate of a subset of micro-level factors.
The Boxhead dataset
To foster progress in the learning of hierarchically structured representation, we introduce Boxhead , a new dataset with hierarchical factor dependencies in which a cube is placed in a simple 3D scene (see Fig.
1 (left) for a representative sample).
This dataset was created with Open3D [48] and PyVista [49] and contains 10 discrete factors of variation: wall hue, ﬂoor hue, azimuth, cube size, cube hue, macro-level hue, and four micro-level hue factors.
The macro-level hue is the aggregate hue of the square in the center of the cube and the micro-level hues are the hues of the four smaller inscribed squares, their value is dependent on the macro-level hue value.
For the speciﬁc dataset dependency structure, see Figure 1 (right).
See Appendix B.1 for further details on the dataset.
To evaluate hierarchical dependencies, we generate a 64x64 sized image dataset with three separate variants.
Each variant has varying levels of dependencies between micro-level and macro-level factors.
These variants are named Boxhead 1, Boxhead 2, Boxhead 3 in order of strongest to weakest levels of dependencies.
See Appendix B.1 for further details on the dependencies.
As opposed to the 3DShapes dataset [50] which inspired Boxhead, our dataset does not purely have statistically independent factors and instead contains explicit hierarchical structure, an arguably important feature of realistic settings.
Compared to CelebA [51], Boxhead contains the ground truth labels of all factors and maintains simplicity for isolated testing of models, a crucial factor given that most disentanglement metrics require completely labeled data.
Pro-LVAE
We evaluate a set of well-known and commonly used VAE models to show the effects of adding hierarchical architectural components.
Single-layer disentanglement VAEs such as β -VAE or βTCVAE do not have hierarchical structure.
Pro-VLAE allows for hierarchical sharing of information through the encoder.
In addition, we introduce pro-LVAE , a novel hierarchical disentanglement model that additionally incorporates statistical dependencies across layers.
For pro-LVAE, we apply the same progressive introduction of latent layers as per pro-VLAE on an LVAE.
The objective we are maximizing for pro-LVAE with L latent layers at the s -th latent layer introduction step is:
[Equation]
where q ( z >l | x ) = (cid:81)
Li
=
l +1 q
( z i | x , z >i ) .
Here, β will constrain the amount of information encoded in the latent layer and will not limit information from the decoder.
Similar to pro-VLAE, the various latent layers must contribute different signals to the image reconstruction.
This allows for independence between different latent layers.
This is important as it means we can use the standard disentanglement metrics to evaluate disentanglement across the model.
See Fig.
4 in Appendix B for architecture details.
Experimental Results
Experimental setup.
To evaluate the beneﬁts of hierarchical inductive biases for disentanglement, we test four different models whose inductive bias allows varying levels of hierarchical information: β -VAE [27], β -TCVAE [3], pro-VLAE [31], pro-LVAE.
Since model capacity is important for learning micro-level factors, we train every model with small and large capacities.
Furthermore, we study four different values of β (1, 5, 10, 20) and run all experiments with ﬁve random seeds.
We train these models on the three variants of the Boxhead dataset.
This represents a total oftrained models.
3 For fairness, we use 12 latent variables across all models.
Ladder models such as pro-VLAE and pro-LVAE have 4 latents for each of their 3 latent layers.
We also anneal β for the single-layer VAEs and the top layer of the ladder models since the ladder models anneal the capacity by progressively introducing the latent layers.
We anneal the top layer of ladder models forsteps.
Additionally, since each of the two lower latent layers of the ladder model is progressively introduced over 5000 steps, we anneal the latent layer of the single-layer models over 10000 steps.
Evaluation metrics.
To measure how well the models disentangle the hierarchical factors, we use the DCI disentanglement metric [52] with 10000 data samples, and compute the score from the resulting importance matrix as detailed in Appendix A.
Following [53], we use Scikit-learn’s default gradient boosted trees.
See Appendix A for a detailed deﬁnition.
Since the micro- and macro-level Reproducing these experiments approximately 160 GPU days on Quadro RTX data generative factors have a varying impact on the pixel-level reconstruction loss, we split the factors into two separate groups: macro-level and micro-level factors.
The macro-level disentanglement score measures disentanglement among the macro-level factors.
These factors include ﬂoor hue, wall hue, cube hue, azimuth, scale.
Similarly, the micro-level disentanglement score measures disentanglement of the micro-level factors.
Disentanglement.
In Figure 2 we present the macro-level and micro-level disentanglement scores for all trained models across all three Boxhead variants.
Hierarchical models perform better than the single-layer models on the Boxhead variants on the micro-level disentanglement score and macro-level disentanglement score.
Micro-level disentanglement seems to mirror the underlying dataset structure, where the less correlated dataset variants are easier to disentangle.
This is in agreement with the ﬁndings in [12] when underlying FoV exhibit correlations.
Models of larger capacities (low β , larger architecture) are expected to model ﬁner details.
We ﬁnd that these models exhibit higher micro-level disentanglement scores on this dataset.
However, even low-capacity ladder models can disentangle micro-level factors better than high capacity single-layer models.
As β increases, single-layer model performance on the metrics degrades more than ladder models (see
Appendix C).
ELBO.
Similarly as above, the box plot in Figure 3 shows the respective ELBO scores across all models.
Overall we see a clear ordering in ELBO with respect to our three dataset variants.
Models trained on strongly correlated variants such as Boxhead 1 consistently have higher ELBO than the models trained on weaker correlated variants such as Boxhead 3.
Even though disentanglement is worse, stronger hierarchical dependencies are easier to model.
VAEs with hierarchical dependencies have higher ELBO scores compared to their single-layer counterparts.
As β increases, the ELBO score decreases for all models.
However, this happens at different values of β for different models (See Appendix C).
For higher-capacity VLAEs and LVAEs, the ELBO starts degrading at a β of around 10, low capacity VLAEs and LVAEs at a β of around 5, and the ELBO degrades at any β for single layer VAEs.
This shows that the ELBO for VAEs with hierarchical dependencies and higher capacities are more robust to higher values of β , which can provide better disentanglement.
The
ELBO is lower for single-layer VAEs because the capacity constraint is applied across a given layer in the model.
When enforcing higher capacity constraints on single layer VAEs, micro-level factors collapse when competing with macro-level factors.
On the other hand, VAEs with multiple latent layers allow factors to compete at their respective levels.
However, with high β (such as β = 20 ), posterior collapse starts to happen even with multiple latent layers.
Related Work
There have been several methods proposed towards achieving and deﬁning disentanglement.
InfoGAN [25] forms disentangled representations in GANs by maximizing the mutual information between the noise variables and the observations with an auxiliary network.
Locatello et al.
[53] evaluated state-ofthe-art single-layer VAE disentanglement approaches, showing that unsupervised disentanglement is impossible without inductive biases, and highly sensitive to hyperparameters and random seeds.
Shu et al.
[54] use consistency and restrictiveness to deﬁne disentanglement, qualities which also directly represent predictability, a quality seen in humans [55].
Recent work has shown that current disentanglement models particularly struggle on data with correlated factors [12] in the unsupervised setting but can be mitigated with weak labels [19].
There also exists recent work towards hierarchical disen- tanglement by Ross and Doshi-Velez [46].
Their deﬁnition of hierarchy differs from ours as they do not have an equivalent macro model of the complete system, which we represent through aggregation.
A macro factor in our deﬁnition is the aggregate representation of a subset of micro factors.
Conclusion
We introduced a new dataset to analyze the disentanglement of hierarchical factors and evaluated state-of-the-art disentanglement models to show the beneﬁts of hierarchical inductive biases.
We show that hierarchical models outperform single-layer VAEs in disentangling micro- and macrolevel factors, and suggest a better ELBO.
The performance of hierarchical models over single-layer models becomes more prominent as the capacity constraint ( β ) increases.
Future work includes identifying inductive biases for hierarchical latent space models to better disentangle hierarchical factors, especially micro-level factors which contribute less to the reconstruction loss.
An additional promising direction is the application of these hierarchical models for realistic environments within dynamic and interactive observation settings: learning varying levels of detail could be helpful towards disentanglement in scenarios where the observer moves closer to an object of interest.
Another interesting area of future research is on metrics for evaluating hierarchical relationships.
A Model Objectives and Metric Deﬁnition
β VAE Objective: The objective we are maximizing for β -VAE is deﬁned below, with β > 1 :
[Equation]
β TCVAE Objective: The β -TCVAE objective we are maximizing is:
L TCV AE =
E q ( z | x ) [
[Equation]
Pro-VLAE Objective: At the s th latent layer introduction step, the objective we are maximizing for pro-VLAE is:
[Equation]
Where L is the number of latent layers.
DCI Disentanglement Score:
The DCI disentanglement score measures if a latent element contains information about a given data generative factor.
The disentanglement score is calculated by the equation: (cid:80) i ρ i (1 − H ( P i )) .
Given the i th latent representation and the j th data generative factor,
P ij = R ij / (cid:80) K − 1 k =
R ik .
R is the importance matrix.
H ( P i ) calculates the entropy of P i .
ρ i is a multiplier used to account for irrelevant and dead units in the latent representation.
B Further Model and Dataset Details B.1 Boxhead Dataset Speciﬁcations
For all Boxhead datasets, the hue is bounded by [0, 1], using the HLV color scheme.
The distribution that generates macro-level hue is discretized into 7 values.
The distributions for the micro-level hues is discretized into 4 values.
Other factors of variation are discretized into 10 values.
Discretization happens on the distribution before the addition of information from other factors.
All Boxhead variants.
All boxhead variants have the following speciﬁcations:
• wall hue ∼ U (0 , 1) • ﬂoor hue ∼ U (0 , 1) • cube hue ∼ U (0 , 1) • scale ∼ U (1 , 1 . 25) • azimuth ∼ U (cid:16) − π 6 , π 6 (cid:17)
Boxhead 1.
The Boxhead 1 variant has the additional following speciﬁcations:
• macro-level hue ∼ N ( cube hue , 0 . 2) •
i th micro-level hue ∼ U ( macro-level hue − 0 .
1 , macro-level hue + 0 . 1) for i
= 1 , 2 , 3 ,
Boxhead 2.
The Boxhead 2 variant has the additional following speciﬁcations:
• macro-level hue ∼ U (0 , 1) •
i th micro-level hue ∼ N ( macro-level hue , 0 . 1)
for i = 1 , 2 , 3 ,
Boxhead 3.
The Boxhead 3 variant has the additional following speciﬁcations:
• macro-level hue ∼ U (0 , 1) •
i th micro-level hue ∼ N ( macro-level hue , 0 . 2)
for i = 1 , 2 , 3 ,
Network Architecture Speciﬁcations
Here, we detail the network architecture for the models.
All models have equivalent network layers.
Ladder models have additional ladder layer speciﬁcations.
Each ladder layer contains an encoding section and a decoding section.
There are two architectures; a small architecture and a large architecture.
Default activation is leaky ReLU, decoder output activation is sigmoid, encoder activation is linear (no change).
[Table]
[Table] [Table]
[Table]
C Additional Experimental Results