S TRUCTURE BY A RCHITECTURE : D ISENTANGLED R EPRESENTATIONS WITHOUT R EGULARIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the problem of self-supervised structured representation learning using autoencoders for downstream tasks such as generative modeling.
Unlike most methods which rely on matching an arbitrary, relatively unstructured, prior distribution for sampling, we propose a sampling technique that relies solely on the independence of latent variables, thereby avoiding the trade-off between reconstruction quality and generative performance inherent to VAEs.
We design a novel autoencoder architecture capable of learning a structured representation without the need for aggressive regularization.
Our structural decoders learn a hierarchy of latent variables, akin to structural causal models, thereby ordering the information without any additional regularization.
We demonstrate how these models learn a representation that improves results in a variety of downstream tasks including generation, disentanglement, and extrapolation using several challenging and natural image datasets
I NTRODUCTION
Deep learning has achieved strong results on a plethora of challenging tasks.
However, performing well on a highly speciﬁc dataset is usually insufﬁcient to satisfactorily solve real-world problems (Tan et al., 2018; Zhuang et al., 2019).
This has lead to a particular interest in consistently learning more structured representations with useful properties to help with a variety of downstream tasks (Bengio et al., 2013; Tschannen et al., 2018; Bengio et al., 2017; Tschannen et al., 2018; Radhakrishnan et al., 2018).
Here, deep learning provides a ﬂexible paradigm to train complex architectures based on autoencoders (Ballard, 1987; Lange and Riedmiller, 2010) which are typically latent variable models, thus allowing us to embed powerful inductive biases into the models to further structure the representations.
However, it is still largely an open question as to what kinds of structure in a representation are the most effective for generative modeling and how to learn such structures without supervision (Locatello et al., 2018; Khrulkov et al., 2021; Shu et al., 2019;
Chen et al., 2020; Nie et al., 2020; Mathieu et al., 2019; Vaswani et al., 2017; Kwon and Ye, 2021).
One direction that may contribute to an answer is causal modeling, as it focuses on the underlying (causal) mechanisms that generate the observations, instead of relying on (possibly spurious) correlations (Pearl, 2009; Peters et al., 2017; Sch¨olkopf, 2019;
Louizos et al., 2017; Mitrovic et al., 2020;
Shen et al., 2020).
With the versatility of deep learning on one hand, and the conceptual insights of causality on the other, our contributions herein include:
• We propose an architecture called the Structural Autoencoder (SAE), where the structural decoder emulates a general acyclic structural causal model to learn a hierarchical representation that can separate and order the underlying factors of variation in the data.
• We provide a sampling method that can be used for any autoencoder-based generative models which does not use an explicit regularization objective and instead relies on independence in the latent space.
• We investigate how well the encoder and decoder are able to extrapolate upon seeing novel samples.
We release our code at * anonymized * .
1. R ELATED W ORK
The most popular autoencoder based method is the Variational Autoencoder (VAE) (Kingma and
Welling, 2013a) and the closely related β VAE (Higgins et al., 2017).
These methods focus on matching the distribution in the latent space to a known prior distribution by regularizing the reconstruction training objective (Locatello et al., 2020a; Zhou et al., 2020).
Although this structure is convenient for generative modeling and even tends to disentangle the latent space to some extent, it comes at the cost of somewhat blurry images due to posterior collapse and holes in the latent space (Locatello et al., 2018;
Higgins et al., 2017; Burgess et al., 2018; Kim and Mnih, 2018; St¨uhmer et al., 2020).
To mitigate the double-edged nature of VAEs, less aggressive regularization techniques have been proposed such as the Wasserstein Autoencoder (WAE), which focuses on the aggregate posterior (Tolstikhin et al., 2018).
Unfortunately, WAEs generally fail to produce a particularly meaningful or disentangled latent space (Rubenstein et al., 2018), unless weak supervision is available (Han et al., 2021).
A more structured alternative is the Variational Ladder autoencoder (VLAE)
(Zhao et al., 2017) which separates the latent space into separate chunks each of which is processed at different levels of the encoder and decoder (called ”rungs”).
Our proposed architecture takes inspiration from the VLAEs but crucially we do not use the variational regularization, and instead use a sampling method based on hybridization (Besserve et al., 2018).
We also infuse the information from the latent variables using
Str-Tfm layers (see section 2.2) which are based on the Ada-IN layers from Style-GANs (Karras et al., 2019), which signiﬁcantly improves on the ladder rungs.
Lastly, unlike the VLAEs, the SAEs use a simple feed-forward CNN encoder (like all the other unstructured models), which simpliﬁes analysis without signiﬁcantly impacting the ability to structure the representation (see further discussion in section 4.2).
M ETHODS 2. C AUSAL R EPRESENTATION L EARNING
Graphical causal modeling represents the relationship between random variables
S i using a directed acyclic graph (DAG) whose edges indicate direct causation and structural equations of the form
S i
[Equation] [Equation]
(1) encoding the dependence of variable S i on its parents PA i in the graph and on an unexplained noise variable
U i .
The noises U 1 , . .
, U D are assumed to be jointly independent.
The DAG along with the mechanisms (1) is referred to as a Structural Causal Model (SCM) (Pearl, 2009).
Any joint distribution of the S i can be expressed as an SCM using suitable f i and U i .
However, the SCM entails additional assumptions regarding how statistical dependencies between the S i are generated by mechanisms (1), such that changes due to interventions can be modelled as well (e.g., by setting some U i or S i to constants).
Real-world observations are often not structured into meaningful causal variables and mechanisms to begin with.
E.g., images are high-dimensional, and it is hard to learn objects and their causal relationships from data (Lopez-Paz et al., 2017).
One may thus attempt to learn a representation consisting of causal variables or disentangled “factors” which are statistically independent (Higgins et al., 2017).
However, in an SCM it is not the S i that should be statistically independent, but the
U i .
For this reason, our representations will comprise the U i as latent variables, driving causal mechanisms via learned functions
f i (1).
This embeds an SCM into a larger model whose inputs and outputs may be high-dimensional and unstructured (e.g., images) (Suter et al., 2018a; Sch¨olkopf, 2019).
Given (high-dimensional) X
=
( X 1 , . . .
, X d ) , our goal is to learn a low-dimensional representation
U =
( U 1 , . . .
, U D )
( D (cid:28) d ) using an encoder R d → R D , and model the generative process (including the SCM) that produced X from the inferred latent variables using a decoder R D → R d .
If the causal graph of the true generative process were known, the topology of the decoder could be ﬁxed accordingly.
However, in the fully unsupervised setting, the generator must be capable of modeling a general SCM, so our architecture speciﬁes a causal ordering (while learning what information to embed in the speciﬁed ordering), while the edges between parents and children in the SCM are learned implicitly in the computation layers.
Speciﬁcally, the learned generative process, a.k.a. decoder, produces a reconstruction ˆ X of X by feeding each of the U i into subsequent computation layers.
Here, the root node S 1 in the DAG only depends on U 1 , while later S i depend on their noise U i and potentially their parents S j ( j < i ) .
Thus the depth in the network corresponds to a causal ordering (see Supplement for further discussion).
2. S TRUCTURAL D ECODERS
[Figure]
This model architecture is implemented by the structural decoder , using D StructuralTransform (Str-Tfm) layers placed evenly in between the convolution blocks.
From the corresponding latent variable U
i , the i th Str-Tfm layer estimates a scale α i and bias β i which are then used to pixelwise transform the intermediate features of the l th layer v l (as seen in Figure 1) much like in Ada-IN (Karras et al., 2019) except without the preceding normalization.
Each Str-Tfm layer thus acts like an f i in (1) by integrating the information from a latent variable
U i to transform the features PA i from earlier layers.
This variable decoder depth for each of the latent variables biases high-level non-linear information towards the earlier (and thereby deeper) latent variables, while the model capacity is reduced for the later variables so they can only capture relatively low-level linear features with respect to the data space.
Meanwhile, the random initialization of the MLPs in the Str-Tfm layers produce more distinct activations with respect to each of the latent variables than if they are all transformed linearly by same dense layers.
This architectural asymmetry between latent variables thereby encourages statistical independence and induces a relatively intuitive hierarchical structure of the latent space.
In theory, without some form of supervision or side information, the learned latent variables are not guaranteed to disentangle the true factors of variation, much less that the learned SCM matches the true one (Locatello et al., 2020b).
Instead, all we are guaranteed from training on observational data is that the model is optimized to reproduce the same observational distribution as the true generative process.
However, for datasets with independent factors of variation, the independent latent variables may align with the true factors, thus disentangling the factors in the representation.
Regardless of disentanglement, independence between latent variables also enables principled interventions on individual variables for generating new samples, using hybrid sampling, which is akin to intervening on the learned SCM for generative modeling.
2. H YBRID S AMPLING
For generative modeling, it is necessary to sample novel latent vectors that are transformed into (synthetic) observations using the decoder.
Usually, this is done by regularizing the training objective so the posterior matches some simple prior (e.g. the standard normal).
However, in practice, regularization techniques can fail to match the prior perfectly and actually exacerbate the information bottleneck, leading to blurry samples from holes in the learned latent distribution and unused latent dimensions due to posterior collapse (Dai and Wipf, 2019; St¨uhmer et al., 2020; Lucas et al., 2019;
Hoffman and Johnson, 2016).
Instead of trying to match some prior distribution in the latent space, we suggest an alternative sampling method that eliminates the need for any regularization of the loss.
Inspired by Besserve et al.
, we refer to it as hybrid sampling: the model stores a ﬁnite set of N ( = 128 in our case) latent vectors, selected uniformly at random from the training set.
To generate a new latent vector ˜ U , a value for each of the D latent variables ˜
U
i is selected independently from the N stored latent vectors { U ( j ) } Nj =1 uniformly at random, such that: ˜ U
i ← U ( j ) i where j ∼ { 1 , ..., N }
This allows the model to generate N D distinct latent vectors, and since we assume D (cid:28) d , N can easily be increased up to the full training set size without signiﬁcant memory costs.
Note that hybrid sampling is directly applicable to any learned representation as it does not affect training at all, however the ﬁdelity of generated samples will diminish if there are strong correlations between latent dimensions.
Consequently, the goal is to achieve maximal independence between latent variables without compromising on the ﬁdelity of the decoder (i.e. reconstruction error).
Hybrid sampling implicitly relies on the statistical independence of the latent variables since resampling the marginal of the aggregate posterior independently breaks any existing correlations.
Not only does this align well with the objectives of unsupervised disentanglement methods, but it is also consistent with the causal perspective of the latent variables as independent noises
U i driving an
SCM.
E XPERIMENTS
We train the proposed architectures and baselines on two smaller disentanglement benchmarking datasets (where d = 64 × 64 × 3 and the true factors are independent): 3D-Shapes (Burgess and Kim, 2018) and the three variants (”toy”, ”sim”, and ”real”) of the MPI3D Disentanglement dataset (Gondal et al., 2019), as well as two larger more realistic datasets (where d = 128 × 128 × 3 ): Celeb-A (Liu et al., 2015) and the Robot Finger Dataset (RFD) (Dittadi et al., 2020).
After training our models on a standard 70-10-20 (train-val-test) split of the datasets we evaluate the quality of the reconstructions based on the reconstruction loss (using binary cross entropy loss, same as the optimization objective) and the Fr ´ echet Inception Distance (FID) (Heusel et al., 2017) as in Williams et al.
The FID is able to capture higher level visual features and can be used to directly compare the reconstructed and generated sample quality, while the binary cross entropy is a purely pixelwise comparison.
Next we compare the performance of the hybrid sampling method to the prior based sampling.
Unlike the prior based sampling, which only makes sense for the models that use regularization (VLAE,
VAE, β VAE, and WAE), the hybrid sampling method can be applied to any latent variable model.
Finally we take a closer look at the learned representations to understand how the model architecture affects the induced structure and possibly disentanglement.
3. M ODELS
All models use the same CNN backbone for both the encoder and decoder with the same number of convolution layers and ﬁlters, each of which is followed by group normalization and a MISH nonlinearity (Misra, 2019), where the encoder uses max-pooling to down-sample the extracted features and the decoder uses bilinear up-sampling to up-sample features (see the appendix for details).
For the smaller datasets, the encoder and decoder have 12 convolution layers each withﬁlters, the latent space has 12 dimensions in total, and the models are trained for 100k iterations, while for CelebA each the encoder/decoder has 16 convolution layers each with twice as many ﬁlters, the latent space is 32 dimensional in total, and the models are trained for 200k iterations.
For the RFD dataset, the settings are the same as for CelebA, except that there are 256 channels per convolution layer.
We compare four kinds of autoencoder architectures.
The ﬁrst type is our Structural Autoencoders (SAE) which use a conventional encoder and a structural decoder with the latent space split evenly into 2, 3, 4, 6, or 12 variables for the smaller datasets and 16 variables for the larger ones corresponding to the labels SAE-2, SAE-3, SAE-4, SAE-6, SAE-12 and SAE-16 respectively.
The simplest ”baseline” architecture uses the traditional ”hourglass” architecture, in addition to a variety of different regularization methods: (1) unregularized autoencoders (”AE”), (2) Wasserstein-autoencoders (”WAE”)
(Tolstikhin et al., 2018), (3) VAEs (Kingma and Welling, 2013b), and a β VAE (Higgins et al., 2017)
(with the experimentally determined best setting of β = 2 ).
The next baseline architecture is called ”AdaAE” (and referred to as ”Adaptive” as it is identical to the SAE models except that all latent variables are passed to each of the Str-Tfm layers, so the architecture is effectively the same as using Ada-IN layers (Karras et al., 2019).
For the purposes of hybrid sampling, each latent dimension of these less structured baselines is treated as a separate latent variable.
The ﬁnal type of architecture we investigate is the Variational Ladder Autoencoder (Zhao et al., 2017) which also uses a structured architecture to learn a hierarchical representation, but unlike our Structural Autoencoders, VLAEs also use the variational regularization and use an encoder architecture that roughly mirrors the decoder.
Another subtle, but signiﬁcant difference between the
SAE and VLAE architecture is that the ladder rungs in the VLAE decoder concatenate the features from the latent variable of the corresponding rung in the encoder, instead of using the variable to scale and shift the convolution features directly.
Just like for the SAE models, we include variants of the VLAEs with 2, 3, 4, 6, 12, and 16 rungs.
3. E XTRAPOLATION
While the encoder and decoder are generally optimized jointly, the tight coupling combined with the inherent asymmetry in the respective learning tasks begs the question which of the two is the weaker link when it comes to generalization.
To this end, we investigate to what extent the encoder vs the decoder can extrapolate to novel observations.
First, both the encoder and decoder are trained jointly on a subset of 3D-Shapes where only three distinct shapes exist (instead of four, as the ball is missing) for 80k iterations.
Then either the encoder only, the decoder only, or both are trained for another 20k iterations on the full 3D-Shapes training dataset.
The reconstruction error for samples not seen during any part of training is compared for each of the variants and each of the architectures to identify how well the encoder extrapolates compared to the decoder.
R ESULTS
[Figure] [Figure]
[Figure]
In terms of reconstruction quality, the structural autoencoder architecture consistently outperforms the baselines (seen in ﬁgure 2).
As expected, unregularized methods like the SAE, AdaAE, and AE tend to have signiﬁcantly better reconstruction quality.
However, the structured architectures SAE and VLAE show improved results compared to their unstructured counterparts.
[Figure]
In comparing the SAE models to the AdaAE architecture, we see that there can be a slight penalty in reconstruction quality incurred from separately processing the latent variables.
However this is more than made up for in the quality of the generated samples (shown in ﬁgure 3), where the SAE models perform signiﬁcantly better than the baselines.
Even the regularized models such as VLAE and VAEs usually generate higher quality samples using the hybrid sampling than when sampling from the prior they were trained to match.
This can qualitatively be observed from ﬁgure 6b.
Surprisingly, the AdaAE architecture actually outperforms all other models on CelebA using hybrid sampling.
This may be explained by the severity of the information bottleneck experienced when embedding
CelebA into only 32 dimensions.
Since the AdaAE model produces the best reconstructions, the generated samples have a higher ﬁdelity even though any learned structure between latent dimensions is disregarded by the hybrid sampling.
In general, if we consider the distribution of the latent variables (i.e., the push-forward of the data distribution into the latent space), then sampling from a simple factorized prior can introduce at least two types of errors: (1) errors due to not taking into account statistical dependences among latent variables, and (2) errors due to sampling from ”holes” in the latent distribution if the prior does not match it everywhere.
Whenever (2) is the dominating source of error, hybrid sampling is preferred.
Furthermore, learning independent latent variables aligns with the aim towards disentangled representations, while minimizing the divergence between the posterior and prior results in a compromise that does not necessarily promote disentanglement.
4. H IERARCHICAL S TRUCTURE
To get a rough idea of how the representations learned using the structural decoders differ from more conventional architectures, ﬁgure 4 shows the one dimensional latent traversals (i.e., each row corresponds to the decoder outputs when incrementally increasing the corresponding latent dimension at a time from the min to the max value observed).
The traversals illustrate the hierarchical structure in the representation learned by the SAE models: the information encoded in the ﬁrst few latent variables can be more nonlinear with respect to the output (pixel) space, as the decoder has more layers to process that information, while the more linear information must be embedded
[Figure]
in the last few variables.
This results in a reliable ordering of ”high-level” information (such as object shape or camera orientation)
ﬁrst, followed by the ”low-level” information (such as color).
This means the structural decoder architecture biases the representation to separate and order the information necessary for reconstruction (and generation) in a meaningful way, and thereby ordering, and potentially even fully disentangling, the underlying factors of variation better.
Figure 5 evaluates how disentangled the representations are using common metrics.
The table shows the disentanglement scores of the same models discussed above, while the plot on the right sheds light on quality of the representations varies for ﬁve different random seeds (used to initialize the network parameters).
Most noteworthy is that the SAE-12 model consistently achieves very high disentanglement scores.
This shows, empirically, that the SAE architecture promotes independence between latent variables (especially SAE-12).
We may explain this as a consequence of splitting up the latent dimensions so that each variable has a unique parameterization in the decoder, making different latent variables less likely to be processed in the same way.
Unsurprisingly, when there are multiple latent dimensions per variable (like in SAE-6, SAE-4, etc.), the dimensions within a variable are entangled similarly to the baselines like AE or adaptive baselines.
Since all of these disentanglment metrics are computed on a dimension-by-dimension basis, the resulting scores are systematically underestimated.
Qualitatively, these SAE models still achieve the same consistent ordering of causal mechanisms, as can be seen from ﬁgure 4b.
For a real world demonstration of how well SAE models are able to order information in the latent space, ﬁgure 6a shows generated CelebA samples when varying only a quarter of the latent variables at a time.
The labels on the left are empirically chosen by the authors to describe roughly what kind of semantic information that quarter of the latent space appears to contain.
Although the the inductive biases are not strong enough to fully disentangle the factors of variation into individual latent dimensions.
The hierarchical representation reliably learns a diffuse kind of disentanglement where information pertaining to higher-level features tend to be encoded in the ﬁrst few dimensions while lower level factors of variation show up towards the last few dimensions.
SAE models achieve this structured disentanglement using the Str-Tfm layers as opposed to the standard
Ada-In layers.
Each Str-Tfm layer only has access to one of the latent variables which is not directly seen by any other part of the decoder.
In contrast, the Ada-In layers used by the AdaAE allows information from anywhere in the latent vector to leak into any part of the decoder.
Consequently, the AdaAE does not disentangle the representation at all (although it achieves impressive results for reconstruction nonetheless).
Model
DCI-D
IRS
MIG
ModExp
SAE0.
SAE0. SAE0.
VLAE0. VLAE0.
VLAE0.
AdaAE0.
AE 0.
VAE 0.
β VAE 0.
WAE 0.
[Figure]
not only does the SAE architecture consistently achieve high disentanglement scores, but its performance is also less sensitive to the random seed.
E x p r e ss i o n
O r i e n t a t i o n &
F a c i a l f e a t u r e s
H a
i r s t y l e
C
o l o r s / s t y l e (a) Here we use hybridized chunks of latent vec- tor to show how different aspects of the resulting generated image can be affected using our SAE-16 architecture and hybrid sampling technique.
di while are
This how the SAE architecture is able to order partially disentangled factors variation
S
[Figure]
y
[Figure]
A
[Figure]
a
[Figure]
V
[Figure]
V L A
[Figure]
[Figure]
Samples generated hybrid
4. E XTRAPOLATION
Perhaps unsurprisingly, none of models were particularly adept at zero-shot extrapolation to observations that were not in the initial training data, as is consistent with Schott et al.
Comparing the ﬁrst two columns on the right of ﬁgure 7, without any update, the reconstructed images ﬁlter out the novel information in the sample (in this case, ball shape), and instead reconstruct a similar sample seen during training (a cylinder).
If only the encoder is updated on some observations with the additional shape while the decoder is frozen, then the reconstruction performance increases somewhat, but some deformations and artifacts become visible in the reconstruction.
This suggests the frozen decoder struggles to adequately extrapolate, even when the encoder extends the representation to include the ball.
In contrast, when only the decoder is updated, the reconstructions qualitatively look much more similar to the original observations.
Although the encoder can be expected to generally ﬁlter out any information it has not been trained to encode into the latent space due to the bottleneck, the representation may still extrapolate somewhat provided the decoder can reconstruct any novel features.
This underscores the importance of focusing on carefully designing and training the decoder, as embodied by the SAEs, because the decoder is not able to extrapolate as well as the encoder.
Model
Neither
Encoder
Decoder
Both
SAE13. SAE13.
VLAE18.
VAE 12.
β VAE 15.
AE 11.
WAE 11.
S
[Figure]
E -
[Figure]
V L A EVA E
A E and decoder.
Furthermore, note that the SAE-12 generally outperforms the all variational baselines, suggesting the aggressive regularization of VAEs makes updating the representation more difﬁcult.
C ONCLUSION
While VAEs provide a principled approach to generative modeling with autoencoders, in practice, the regularization narrows the information bottleneck by penalizing any dependence of the posterior on the observation (Hoffman and Johnson, 2016;
Tolstikhin et al., 2018), resulting in a trade-off between reconstruction quality and matching the prior.
While this encourages a more compact and possibly even disentangled representation, it also tends to result in blurry generated samples and relatively poor reconstructions.
This motivated us to look for an alternate sampling method that does not require aggressive regularization as VAEs.
Our hybrid sampling technique relies on independence between latent variables instead of expecting the learned posterior to match a (relatively unstructured) prior.
This effectively uniﬁes the goals of achieving a disentangled and samplable representation, thereby enabling challenging downstream tasks such as controllable generation.
To that end, we propose the structural autoencoder architecture inspired by structural causal models, which orders information in the latent space, while also, as shown by our experiments, encouraging independence.
Notably, it does so without additional loss terms or regularization.
The SAE architecture produces high quality reconstructions and generated samples, improving extrapolation, as well as achieving a signiﬁcant degree of disentanglement across a variety of datasets.
While it is encouraging how far one can get with a suitable architectural bias, future work should assay whether the learned models can be structured further by more explicit forms of causal training.
For instance, we could explicitly encourage, that hybridization produce realistic samples using an adversarial discriminator, or that across domain shifts or actions/interventions external to the network, only a sparse set of the latent factors change (cf.
the Sparse Mechanism Shift Hypothesis, (Sch¨olkopf et al., 2021)).
E THICS S TATEMENT
Fortunately, our project for the most part does not have signiﬁcant ethical concerns.
However, the broader ﬁeld of representation learning and scene generation does, which we address here.
Statistical machine learning has in recent years focused heavily on optimizing the performance of systems on supervised i.i.d.
problems.
The ﬁeld of causal learning tries to mitigate the heavy reliance on single i.i.d.
datasets and instead attempts to build systems that transfer knowledge to solve many tasks.
This often comes in combination with an element of stronger interpretability of models (our paper above is an examples thereof).
In the long term, we hope that causal learning will help build machines that approach ”common sense” and enable rational and fair decision making in a variety of disciplines.
A small portion of our results uses the CelebA dataset, which is available for non-commercial research purposes (no license provided).
It does show images of individuals, and permission may not have been obtained; however, our understanding is that usage of these images is permitted for research purposes since the individuals are celebrities.
We will remove those experiments from the paper if this is considered inappropriate.
We also note that sophisticated generative models can be misused to synthesize or modify images, reinforcing the need to develop methods that allow to reliably detect manipulated content.
On the other hand, the interpretability that comes with the modularity of our method can yield insights into image analysis, and could therefore facilitate better explanations for model decisions.
R EPRODUCIBILITY S TATEMENT
Especially in projects that make use of deep learning, particular care must be given to the experimental design and supplementary materials to ensure reproducibility of the methods and results.
For the experimental design, we have conducted extensive hyperparameter searches to identify which high-level architectural choices consistently perform best.
We designed all the models (those using our proposed methods and all baselines) to use as many of the same hyperparameters as possible, including adjusting the number of trainable parameters to be comparable.
Speciﬁcally, we used a greedy propose-reject algorithm: each model agnostic hyperparameter setting (primarily architectural and optimization choices discussed in the supplement) proposed was tested on one dataset (in this case 3D-shapes) and only accepted if it improved validation set results for all models.
This same process was then repeated (to a lesser extent) on the other datasets.
Next, the model agnostic settings were frozen, and model speciﬁc hyperparameters (such as the depth and width of the MLPs in the
Str-Tfm layers, VLAE ladder rungs, or β in the β VAE) were optimized using the same criterion of the optimization objective on the validation set.
Finally, we include results on 5 replica models with different random seeds for the 3D-Shapes dataset to make sure our results are not outliers for each method.
To ensure that our experiments can be reproduced and that interested parties can extend our work, we have made all code pertaining to this project available online (although the link is anonymized for the review process).
For the reviewers, we include a copy of code as supplementary material.
Both the online repository as well as the code provided for review has been tested and includes a walk through guide (a.k.a. ”README”) to make installation and setup as easy as possible.
Although we do not provide any of the dataset we have used for our experiments, they are all freely available for download online (see the corresponding references for more information).
A A PPENDIX A. C AUSAL ORDERING
Suppose we are given (high-dimensional) X
=
( X 1 , . . .
, X d ) (think of X as an image with pixels
X 1 , . .
, X d ), from which we should construct S 1 , . . .
, S D ( D (cid:28) d ) as well as causal mechanisms
S i :=
f i ( PA i , U i ) , ( i = 1 , . . .
, D ) .
To this end, we ﬁrst use an encoder f enc : R d → R D taking X to a latent bottleneck representation comprising U
=
( U 1 , . . .
, U D ) .
The next step is a map f ( U ) implementing the structural assignments f 1 , . . .
, f D as a function of U .
We construct it as follows: we evaluate the f i of a root node i , i.e., f i depends only on U i .
In the step, we evaluate any node j which depends only on its U j and possibly other variables that have already been computed.
We iterate until there are no nodes left.
This terminates (since the graph is acyclic) and yields a unique f ( U ) , but the order π ( i ) in which the f i get evaluated need not be unique.
It is referred to as a causal or topological ordering (Peters et al., 2017), satisfying π ( i ) <
π ( j ) whenever j is a descendant of i .
This embeds the SCM into the network starting from the bottleneck U =
( U 1 , . . .
, U D ) with the U i feeding into subsequent computation layers according to a causal ordering.
This structure reﬂects the fact that the root node(s) in the DAG only depend on ”their” noise variables, while later ones depend on their noise and those of their parents, and so on.
Finally, we apply a decoder f
dec : R D → R d .
The system can be trained using reconstruction error to satisfy f dec ◦ f ◦ f enc
≈ id on the observed images.
Recall that for a causally sufﬁcient system, the set of noises U 1 , . .
, U n are assumed to be jointly independent.
If, in contrast, only a subset of the causal variables are modelled, then the noises will in the generic case be dependent.
We would expect that the architectural bias implemented by the structural decoder, however, may still be a sensible one.
A. T RAINING P ROCEDURE A.2. A RCHITECTURE D ETAILS
As described in the main paper, the basic convolutional backbone of all models is the same.
For the smaller datasets, 3D-Shapes and the MPI3D datasets (where observations are 64x64 pixels), the encoder and decoder each have 12 convolutional blocks.
Each block has a convolutional layer with 64 channels and a kernel size of 3x3 and stride of 1 (unless otherwise speciﬁed), followed by a group normalization layer and then a MISH nonlinearity (Misra, 2019).
In the encoder, the features are downsampled using a 2x2 Max Pooling layer right after the convolution every third layer starting with the ﬁrst one and the ﬁrst convolution layer uses a kernel size of 5x5.
In the decoder, every third convolution layer is immediately preceded by a 2x2 bilinear upsampling.
For our structured modules (SAE and AdaAE), the speciﬁed number of Str-Tfm layers are placed evenly in between the convolution blocks.
For SAE models, the latent space is always split evenly between Str-Tfm layers, and each layer uses a three hidden layer network to process the latent space segment into the scale and bias vectors which are then applied to all pixels individually of the features.
For the VLAE models, the inference and generative ladder rungs each also have a three hidden layers to process the features into and out of the separate latent space segments respectively.
While the latent space was always 12 dimensional for 3D-Shapes and MPI3D, for Celeb-A
we use a 32 dimensional latent space.
For Celeb-A, we also expand the 12 block backbone to 16 blocks and double the ﬁlters per convolution layer to 128.
The exact sizes and connectivity of the models can be seen in the conﬁguration ﬁles of a the attached code, but overall, each of the 3D-Shapes and MPI3D models have approximately 1-1.2M trainable parameters, while for CelebA the models have 6-7M parameters.
A.2. T RAINING D ETAILS
All models used the same training hyperparameters, which included using an Adam optimizer with a learning rate of 0.0005 and momentum parameters of β 1 = 0 .
9 and β 2 = 0 .
For the smaller datasets (3D-Shapes, MPI3D) the models were trained for 100k iterations and a batch size of 128, while for Celeb-A
the models were trained for 200k iterations and a batch size of 32.
The convolution was doubled and the learning rate was decreased by a factor of 10.
The models are implemented using Pytorch (Paszke et al., 2019) and were trained on the in-house computing cluster using Nvidia V100 32GB GPUs, so that training a single model takes about 3hours on the smaller datasets and 7-10 hours for CelebA.
For the β -VAEs, β ∈ { 2 , 4 , 6 , 8 , 16 } were tested on 3D-Shapes and MPI3D, and the model with the smallest loss on the validation set was used for subsequent analysis, which was β = 2 for all datasets.
A. A DDITIONAL R ESULTS
[Figure]
[Figure]
o n s t r u c t i o n F I
D S c
o r e
[Figure]
SAE-12 (dark blue), while the other regime starts at the β -VAE (pink) and reaches to the VLAE-12 (red).
These regimes separate the models that use the variational regularization loss from the models that only use a reconstruction loss (or a regularization on the aggregated posterior like the WAE).
Lastly, the
SAE-12 consistently performs best both in terms of reconstruction FID and the pixelwise loss.
F I D
S c
o r e 3D-Shapes Generated Samples
Sampling Method
Hybrid
Prior
Model
SAEVLAE-
AdaAEAE
VAE VAE
WAE
Sampling Method
Hybrid
Prior
Model
DCI-d
IRS
MIG
SAP
ModExp
DCI-c
SAE0.
SAE0. SAE0.
SAE0. SAE0.
VLAE0. VLAE0.
VLAE0.
VLAE0. VLAE0.
AdaAE0.
[Table]
[Figure] [Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table]
[Figure]
A.3. E XTRAPOLATION
We present results on a variant of the exptrapolation experiment discussed in section 4.2.
Instead of modifying the shape in the initial training dataset, we remove the two most extreme camera angles in either direction (removing 4/15 of the full dataset).
As seen from ﬁgure 12, in this setting the warping and generally lower ﬁdelity experienced by only updating the encoder compared to updating the decoder is very apparent.
Model
Neither
Encoder
Decoder
Both
SAE4.
VLAE6.
AdaAE4.
AE 4.
WAE 4.
VAE 5.
β VAE 5.
S A EInput
No Update
Encoder only Decoder only
Both
V L A EVA E
A d a A E
A.3. MPI3D-T OY
Model
DCI-d
IRS
MIG
SAP
ModExp
DCI-c
SAE0.
SAE0. SAE0.
SAE0. SAE0.
VLAE0. VLAE0.
[Table]
[Figure] [Figure]
[Table] [Figure]
[Table] [Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table] [Table]
[Table]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
[Figure]