Exploring the Latent Space of Autoencoders with Interventional Assays
Anonymous Author(s)
AffiliationAddress email
Abstract
Autoencoders exhibit impressive abilities to embed the data manifold into a lowdimensional latent space, making them a staple of representation learning methods.
However, without explicit supervision, which is often unavailable, the representation is usually uninterpretable, making analysis and principled progress challenging.
We propose a framework, called latent responses , which exploits the locally contractive behavior exhibited by variational autoencoders to explore the learned manifold.
More specifically, we develop tools to probe the representation using interventionsin the latent space to quantify the relationships between latent variables.
We extendthe notion of disentanglement to take the learned generative process into accountand consequently avoid the limitations of existing metrics that may rely on spurious correlations.
Our analyses underscore the importance of studying the causalstructure of the representation to improve performance on downstream tasks suchas generation, interpolation, and inference of the factors of variation.
Introduction
Autoencoders (AEs)
[1, 2] and its modern variants like the widely used variational autoencoders(VAEs)
[3], are a powerful paradigm for self-supervised representation learning for generativemodeling [4], compression [5], anomaly detection [6] or natural language processing [7].
Sinceautoencoders can learn low dimensional representations without requiring labeled data, they areparticularly useful for computer vision tasks where samples can be very high dimensional makingprocessing, transmitting, and search prohibitively expensive.
Here VAEs have shown impressiveresults, often achieving state-of-the-art results compared to other paradigms [8, 9, 10, 11].
The striking performance coupled with the relatively flexible approach has prompted an explosionof variants to learn a representation with some structure that is particularly conducive to a giventask [12, 13, 14].
In addition to a meaningful lower-dimensional representations of our world [15],the focus may be to improve generalization [16, 17, 18], increase interpretability by disentangling theunderlying mechanisms [19, 20, 21, 22], or even to enable causal reasoning [23, 24].
In designing ever more intricate training objectives to learn more specialized structures as part ofcomplicated model pipelines, it becomes increasingly important to gain a better understanding ofwhat the representation actually looks like to more quickly identify and resolve any weaknesses ofa proposed method.
Here the manifold learning community provides a principled formulation toanalyze and control the geometry of the data manifold learned by the representation [25, 26, 27, 28,29, 30, 31, 32, 33].
Developing tools to better understand the structure of the representation is notonly useful as a diagnostic to identify avenues for improving modelling and sampling [34, 35], but italso has crucial importance for fairness [36, 37] and safety [18, 38, 39].
Our focus here is on taking advantage of common properties of autoencoders to gain a deeperunderstanding of the structure of the representation.
We summarize our contributions as follows:
• We propose a framework, called latent responses , which exploits the locally contractivebehavior of autoencoders to distinguish the informative components from the noise in the latentspace and to identify the relationships between latent variables.• We develop tools to analyze how the data manifold is embedded in the latent space by estimatingthe extrinsic curvature which also enables semantically meaningful interpolations.• Where true labels are available, we use conditioned latent responses to assess how each truefactor of variation is encoded in the representation and introduce the Causal DisentanglementScore to quantify how disentangled the learned generative process is.
Background
Representation learning begins with a set of N observation samples x ∈ X ⊆ R D which originatefrom some unknown stochastic generative process with distribution x ∼ p ( X ) and support X .
Thisdata manifold X is embedded into a low-dimensional ( d << D ) latent space R d , and is modelled bythe support Z of an encoder f : X (cid:55)→ Z with the decoder learning the inverse mapping g : Z (cid:55)→ ˆ Xwhere after training ˆ X ≈ X .
Variational Autoencoders (VAEs) [3, 40] are a framework for optimizing a latent variable modelp ( X )
≈ (cid:82) Z p ( X | Z ; θ ) p ( Z )
d Z with parameters θ , typically with a fixed prior p ( Z ) =
N ( Z ; 0 , I ) ,using amortized stochastic variational inference.
A variational distribution q ( Z | X ; ϕ ) with parameters ϕ approximates the intractable posterior p ( Z | X ) .
The encoder and decoder are parameterizedsuch that q ( Z | X = x ; ϕ ) =
N ( Z ; f ϕ ( x ) , σ ϕ ( x )) , and E [ p ( X | Z
= z ; θ )]
= g θ ( z ) where f ϕ ( x )and σ ϕ ( x ) , and g θ ( z ) are neural networks which are jointly optimized using the reparameterization trick [3] to maximize the ELBO (Evidence Lower BOund) which is a lower bound to the loglikelihood:
[Equation]
Note that in practice, p ( X ) is unknown, and we only have access samples { x ( i ) } Ni , so p ( X ) isapproximated by π ( X = x ) = 1 N (cid:80) Ni =1 δ ( x − x ( i ) ) .
The first term in the objective corresponds toa reconstruction loss, while the second can be interpreted as a regularization term to encourage theposterior to match the prior.
2.1 Related Work
Representation geometry A closely related approach is manifold learning which aims to exploitthe geometry of the data manifold, usually by regularizing the geometry of the representation byestimating the intrinsic curvature of the data manifold [26, 27, 28, 29, 30], or by improving samplingand interpolation using the the Riemmanian metric [31, 32, 25, 33].
In comparison, our responsemaps estimate the extrinsic curvature, which focuses on the specific embedding, rather than being anintrinsic property of the dataset (see appendix A.4).
Interpretability and Disentanglement
Another general approach is to gain a better understandingof the representation by making it more interpretable, by, for example, disentangling the true factorsof variation [41, 19, 22, 42, 43, 44].
While our analysis is more similar to approaches that improvethe representation by focusing on the structure of the representation, for example by learning extrapost-hoc models in the latent space [45, 46, 47], we develop a metric to evaluate how disentangledthe learned generator is, rather than just the encoder.
Autoencoder Consistency VAEs have been investigated from an information theoretic viewpoint[48, 49] and with respect to training problems like posterior collapse [50] or the holes problem[51] to better understand common failure modes.
Similarly, mismatches between the encoder anddecoder [52, 53], have spurred research into increasing the self-consistency of autoencoders [54,55, 56, 57, 58].
Our latent response framework relies on a very similar approach and formulation,but crucially, thus far these methods focus on regularizing the training of the representation toimpose certain desired structure, while we focus on analyzing the structure that is learned rather thanmodifying the training objective, making our tools directly applicable to virtually all VAEs.
Latent Responses
On a high level, to explore the structure of how the data manifold is embedded in the latent space, itis necessary to separate the semantic information from the noise in the latent space.
So our goal is todecompose the latent variables Z into an endogenous S and exogenous U component, which, forsimplicity, we choose to relate to one another as shown in equation 2.
[Equation]
Conceptually, S should capture the semantic information necessary to to reconstruct the sample X ,while U is a local noise model in the latent space for a given observation X which does not meaningfully affect the semantics.
In the context of VAEs with a Gaussian prior, we propose equations 3and 4 which recovers the familiar posterior for VAEs q ( Z | X = x ; ϕ ) = q ( S | X ; ϕ ) q ( U | X ; ϕ ) = N ( f ϕ ( x ) , σ ϕ ( x
[Equation]
[Equation]
[Equation]
One implication of separating the deterministic and stochastic part of the encoder is a new perspectiveon the training signal for the decoder from the VAE objective.
Substituting our definitions for S and
U into the reconstruction loss term (shown in equation 5) reveals the decoder is trained to map allsamples from the posterior to match the same observation sample
x
( i ) .
As a result, the decoder learnsto filter out any exogenous component u from z around s , making the latent space locally contractivearound S to the extent of U .
Although we observe this as a byproduct of the VAE objective, thiscontractive behavior is even observed in unregularized autoencoders to some extent [59] suggestingthis may be a more fundamental feature of the inductive biases in deep autoencoders.
E
[Equation]
where s ( i ) =
f ϕ ( x ( i ) ) is the (deterministic) latent code corresponding to the observation x ( i ) .
Starting from some latent sample z ∼ p
( Z ) , we would like to separate the constituent exogenousu and endogenous s components.
If we had the matching x such that z ∼ q ( Z | X = x ; ϕ ) , thenthe separation would be trivial since, by definition s = f ϕ ( x ) and u = z − s .
However, since the VAE is optimized to reconstruct observations from the latent space, we approximate the missing xwith ˆ x = g θ
( z ) .
Subsequently, we encode the generated sample ˆ x to infer an approximation of s ,ˆ s = f ϕ
(ˆ x ) .
Now by expanding g θ around z and f ϕ around X to the first order, we are left with threeterms shown in equation 6 (neglecting higher order terms, full derivation in appendix A.2).
[Equation]
where ϵ = ˆ x − x is the reconstruction error, J f ϕ ( x ) is the jacobian of f ϕ evaluated at x and J g θ ( s )is the jacobian of g
θ evaluated at s .
Term 1 aligns with our conceptual interpretation of s as the endogenous information should remaininvariant to any exogenous information also contained in z .
Meanwhile the second and third termscorrespond to two different sources of error to this interpretation we must potentially take into account.
The term 2 derives from the encoder struggling to encode samples that were not seen during training,since π ( X ) ̸
= p ( ˆ X | Z ; θ ) p ( Z ) .
However, provided the reconstructions sufficiently faithful acrosslatent space (i.e. g θ ( s ) − x is small), this term may be ignored.
This error can be further mitigatedby training the encoder to be more robust with respect to the input (diminishing J f ϕ ( x ) ) with mildadditive noise on the input observations or additional regularization terms [54, 52, 58].
Finally,term 3 in equation 6 originates from the decoder having to filter out the stochastic exogenousinformation from z when decoding.
As discussed above in equation 5, the VAE objective alreadydirectly minimizes this term by training the decoder output to be invariant to noise samples u , thusdiminishing J g
θ ( s ) .
The bottom line is, as long as the decoder can filter out the exogenous information u from latent samples z and the encoder can recognize the resulting generated samples ˆ x , the endogenous information s is preserved.
We call this process of decoding and reencoding latent samples, h ϕθ = f
ϕ ◦ g θ thelatent response function.
Crucially, the latent response function allows us to extract the semanticinformation from the latent space without knowing how the information is encoded, so although wecan identify the structure of the representation, the representation is not necessarily interpretablewithout ground truth label information.
A statistical treatment of this phenomenon is discussed inthe appendix, however provided the error terms are sufficiently small the corresponding responsedistribution r ( ˆ Z | Z ; θ, ϕ ) is shown in equation 7.
[Equation]
One subtlety of our interpretation remains to be addressed: how ambiguities due to overlapping posterior distributions are resolved by latent responses. To match the prior, there will be overlap betweenposterior distributions which correspond to semantically different samples.
Statistically speaking,these ambiguities are captured by the higher moments of r ( ˆ Z | Z ; θ, ϕ ) .
Despite E r ( ˆ Z | Z ; θ,ϕ ) [ ˆ S ] ≈ S ,the variance can be interpreted to relate to the uncertainty of the encoder in the inference of theendogenous variable from the generated sample, suggesting a potential signal for anomaly detection [55].
From another perspective, a strong deviation between ˆ S and S implies there is someinconsistency between the encoder and decoder, which identifies the "holes" in the latent space [51],where the encoder has trouble recognizing the samples generated by the decoder.
3.1 Interventions
[Figure]
project the sample back onto the manifold. Note that this projection (green arrow) changes both the horizontal and vertical dimensions, from which we infer there is a non-trivial relationship between the horizontal and vertical.
To control the learned generative process, weneed to know how to manipulate specific semantic information in the representation.
Weconceptualize these manipulations as interventions where a chosen latent variable is modifiedwhile all others remain unchanged.
For example, given latent sample z , ∆ ( z j ← ˜
z
j ) ( z ) refersto the interventional sample where the j th latentvariable is resampled from the marginal of theaggregate posterior ˜
z
j ∼ q ( Z j ; ϕ ) .
When intervening on latent variables Z , thereare two possible outcomes: either the resultinggenerated sample is affected significantly, that isto say, the semantic information S is affected bythe intervention, or there is no significant changein which case only the noise was affected.
Inthe first case, we identify a specific kind of intervention to manipulate the learned generativeprocess.
Latent Response Matrix This brings us to the first practical tool we introduce based on latentresponses, which aims to describe to what extent the latent variables causally affect one anotherwith respect to the learned generative process.
Each element in the matrix M ∈ R d ∗ d quantifies thedegree to which an intervention in latent variable j causes a response in latent variable k as seen inequation 8, where h θϕk refers to the k th variable of the latent response (see figure 1).
[Equation]
Along the diagonal, M jj can be interpreted as quantifying the extent to which an intervention alongthe
j th latent variable is detectable at all.
As the value approaches 0, changes in the latent variable donot affect the generated sample in any way detectable by the encoder.
For VAEs, this is frequentlydue to posterior collapse [60, 61, 50, 62].
On the other hand, if the latent variable is maximallyinformative and the intervention ˜
z j is fully recoverable it implies negligible exogenous noise, sothen h ϕθj (˜ z ) ≈ ˜
z
j and h ϕθj ( z ) ≈
z j , and M jj approaches 1, since both ˜ z j and z j are sampledindependently from a standard normal, and the elements are normalized with a factor of 12 .
Perhaps even more interesting, the off-diagonal elements of M jk show to what extent an interventionon variable j affects variable k .
Consequently, the latent response matrix can be interpreted as aweighted adjacency matrix of a directed graph of the learned structural causal graph.
Note thatthere may be cycles in this graph if there is some non-trivial relationship between latent variables.
For example, the model could embed a periodic variable into two latent dimensions, such as j 1and j 2 , to keep the representation continuous.
In that case, we would expect an intervention alongdimension
j 1 to elicit a similar response in j 2 as the response of j 1 from an intervention on j 2 .
Consequently, M j 1 j 2 ≈ M j 2 j 1 > 0 suggests j 1 and j 2 should be treated jointly as a single latent 182 variable.
Consequently, the latent response matrix not only describes the causal structure of the learnedgenerative process, but it can also identify more complex relationships between latent variables forfurther analysis.
Conditioned Response Matrix While the latent response matrix M jk lets us quantitatively comparehow much each latent dimension affects another, without manual inspection or label information,the correspondence between the learned causal variables the true causal factors is, in general,unknown [19].
However, when we have access to the true generative process, or at least the ground truth labelsy ( i )
∈ Y ⊆
R d ∗ corresponding to the observation samples x
( i ) , then there is variant of the latentresponse matrix, termed the conditioned response matrix , to quantify how well the learned variablesmatch with the true ones, which is closely related to the disentanglement of the representation.
The key is to carefully select our interventions such that they only affect one true factor at a time, andthen evaluate to what extent these interventions for each of the latent variables are still detectable.
Intuitively, if a latent variable Z j only captures information pertaining to factor Y c , then if we selectinterventions that only change factor Y c ′ where c ′ ̸
= c , then interventions on Z j do not produce aresponse, so M jj ≈ 0 .
To condition the set of interventions on a specific factor Y c , we choose a subset of observationswhich are all semantically identical except for a single factor Y c , x ∼ p
( X | Y c , Y − c ) p ( Y c ) p
( Y − c )where p ( X | Y ) refers to the true generative process given label Y and Y − c refers to alltrue causal variables except Y c .
Then the latent response matrix is computed with interventions exclusively sampled from the resulting aggregate posterior of this subset q ( Z | Y − c ; ϕ )
= 203 (cid:82) q ( Z | X ; ϕ ) p ( X | Y c , Y − c ) p ( Y c )
d X d Y c .
[Equation]
In the context of controllable generation, the conditioned response matrix quantifies how much anintervention on each latent variable can affect each of the true factors of variation.
Ideally, eachtrue causal factor would only be manipulable by disjoint subsets of the latent variables, which iscommonly referred to as "disentangling" the factors of variation.
From the conditioned responsematrix, we can identify not only which latent variables contain information pertaining to each of thetrue factors of variation, but also which factors are affected by an intervention in each of the latentvariables.
Causal Disentanglement Score To more easily compare representations, we can aggregate theinformation in the conditioned response matrix into a single value to measure the degree to which therepresentation is able to disentangle the true factors of variation.
The causal disentanglement score(CDS) in equation 10 which allows each latent variable to causally affect a single factor, but penalizesany additional responses.
As written, the score is between 1 d ∗ and 1, but we re-scale it to [0 , 1] .
[Equation]
(10) (To our knowledge) none of the existing disentanglement metrics take the learned generative processinto account at all.
If the task is only to infer the true labels from the observations (as is common),then
the decoder is admittedly superfluous, which is why disentanglement methods generally focus onthe encoder.
However, in tasks such as controllable generation, where disentanglement is obviouslyvaluable, the behavior of the decoder is critical.
Here, the main risk in evaluating disentanglementfrom the encoder alone is that some latent variables may be correlated with true factors of variationwithout them having any causal effect on those factors when generating new samples.
These spurious correlations also potentially decrease the resulting disentanglement score, which may falsely penalizelarger representations.
The conditioned response matrix and associated CDS mirrors the responsibility matrix and disentanglement score introduced by [42].
However, crucially, the responsibility matrix identifies howpredictive each latent variable is for each true factor of variation.
Response Maps The last type of analysis we propose focuses on gaining a qualitative understandingof how the data manifold is embedded in the latent space.
Specifically, we exploit the ability to probethe latent manifold using the response function to map out the manifolds extent, including estimatingits extrinsic curvature.
From the definition of s = f ϕ ( x ) where s ∈ Z , x ∈ X and f ϕ is a smoothdeterministic function, we may interpret s as a projection of the data manifold X into the latent space.
Usually, our analysis of Z is limited to the observation samples of x we have access, from which manifold learning methods often estimate the intrinsic curvature of the data manifold X .
However, usinglatent responses, we can use any latent sample
z ∼ p
( Z ) to probe Z as long as ˆ s ≈ s .
Rearrangingthe terms gives us u ( z ) in equation 11, whose magnitude can be interpretted as the unsigned distancefunction to the latent manifold, which is evocative of the neural implicit functions [63, 63, 64]suggesting a variety of further tools we leave for future work.
[Equation]
Treating | u ( z ) | as an approximate distance function to the manifold, we compute the mean curvature
H of the latent manifold (see appendix for further discussion).
In practice, we estimate the necessarygradients by finite differencing across a 2D grid in the latent space, we call the resulting map theresponse map , visualized similarly as in [65].
Qualitatively, with our sign convention, high curvature corresponds to regions where u ( z ) is smalland locally convergent, and consequently where we find the data manifold.
Empirically, we also findthe divergence of u ( z ) , which is closely related to the mean curvature, and is useful for identifyingthe regions of the latent space where u diverges, which may be interpreted as possible "holes" in thelatent space.
When interpolating between latent samples z , ideally we would follow the geodesic of the datamanifold.
This is can done by estimating the Riemannian metric from the data samples and integratingan expensive ODE to find the geodesic connecting samples [25].
The Riemmanian metric relates tothe intrinsic curvature of the data manifold, which is independent of the embedding, consequentlyguaranteeing we find the geodesic independent of the representation.
However, in our case, we usethe latent responses to estimate the extrinsic curvature which does depend on the embedding, so theresulting path may not be optimal with respect to the underlying data manifold.
Nevertheless, weoptimize the path along the response map to stay in high curvature regions, thereby effectively arefinding a path in the latent space which stays near the data manifold.
Toy Example: The Double Helix
To illustrate how the latent response framework can be used to study the representation learned bya VAE, we show the process when learning a 2D representation for samples from a double helixembedded in R 3 .
Disregarding the additive noise, the data manifold has two degrees of freedom (datamanifold formally defined in the appendix).This analysis is largely independent of the precise neuralnetwork architecture, provided the model has sufficient capacity to learn a satisfactory representation(hyperparameters in the appendix).
Figure 2 provides an example for how the response maps can be used to trace the latent representationthrough the mean curvature and divergence.
Note that the magnitude of the response | u ( z ) | is notsufficient to identify the latent manifold since both the regions with maximal and minimal curvaturehave a minimal response.
The divergence shows most of the latent space has a slightly negativedivergence, implying most of the latent space converges, rather than diverges, which is consistentwith expectations.
The mean curvature shows what regions of the latent space the map converges toin yellow, from which we can recognize the structure of the learned latent manifold.
Finally, the rightmost plot shows the aggregate posterior q ( S | X ; ϕ ) of the 1024 training samples.
This toy example also motivates the value of meaningful interpolations as seen in figure 3.
Thepath in red shows the shortest euclidean path between the two samples in orange, but note that
the
[Figure]
Magnitude of Response
Divergence
Mean Curvature
Posterior 0.
resulting path in the observation space jumps from one strand to another twice.
Meanwhile, the paththe maximizes the estimated mean curvature is shown in green and produces a much more reasonablepath.
[Figure]
Experiments & Results
Experimental Setup We apply our new tools on a small selection of common benchmark datasets,including 3D-Shapes [66], MNIST [67], and Fashion-MNIST [68] 1 .
Our methods are directlyapplicable to any VAE-based model, and can readily be extended to any autoencoders.
Nevertheless,we focus our empirical evaluation on vanilla VAEs and some β -VAEs (denoted by substituting theβ , so 4-VAE refers to a β -VAE where β
= 4 ).
Specifically, here we mostly analyze a 4-VAE modelwith a d = 24 latent space trained on 3D-Shapes (except for table 1) referred to as Model A, andinclude results on a range of other models in the appendix.
All our models use four convolution andtwo fully-connected layers in the encoder and decoder.
The models are trained using Adam [69] witha learning rate of 0.001 for 100k steps (see appendix for details).
Qualitative Understanding The most direct way to get a better understanding the manifoldstructure is the visualization of the response maps, in particular with the mean curvature (see figure 4).
Unfortunately, since the curvature is estimated numerically using a grid of samples, the maps donot scale well to the whole latent space.
Here the latent response matrices help identify pairs ofrelated latent dimensions which can then be analyzed more closely with a response map.
Furthermore,currently, all these response maps are aligned to the axes of the latent space.
Although VAEs doalign information along the axes somewhat [70], any off-axis structure is missed since the off-axesresponses are completely ignored.
Since, an implicit requirement of disentangled representationsis that the information is axis-aligned [19], the response maps present the most striking results fordisentangled representations (see appendix for more examples).
[Figure]
Map
[Figure]
Aggregate Pos- terior 1.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
1.5 Dimension 13 1.
D i m e n s i o n
[Figure]
From the mean curvature response map in figure 5a, we see that the manifold is particularly nonlinearalong these two latent dimensions (13 and 14).
Consequently, there can be a dramatic differencebetween the geodesic (here approximated using the mean curvature), compared to the shortest path ineuclidean space as seen in figure 5.
Causal Disentanglement Table 1, unsurprisingly, shows the disentanglement increasing withincreasing β .
Our proposed CDS score correlates strongly with the other disentanglement metrics.
Perhaps noteworthy is that even though the CDS and DCI-D scores are computed in similar ways
[Table]
-VAEs trained on 3D-shapes with varying β .
For the models in the first four rows d = 12 , while d = 24 for the remaining four.
While the CDS generally correlates well with other metrics, notably, the DCI-D score is consistently slightly lower which may be due to spurious correlations between latent variables and the true factors.
[Figure]
for each factor of variation (row), while the conditioned response matrix measures the effect an intervention in each latent dimension (column) has provided that the intervention can changes a specific factor of variation (row). Note that from the conditional response matrix we see only dimensions 16 and 22 are causally linked to the floor hue, for which the structure is further visualized in figure 4.
(the vital difference being whether responsibility rests on a causal link or a statistical correlation),the DCI-D scores are consistently lower than the CDS scores.
This may be explained by the DCI-Dmetric taking additional spurious correlations between latent variables into account, while the CDSmetric focuses on the causal links, so the DCI-D has an undeservedly low score.
Causal Structure Closer comparison between the conditioned response matrices and the responsibility matrices reveal more how the DCI-D metric and CDS differ.
Figure 6 shows the responsibilitymatrix matches the conditioned response matrix for the most part.
The only exception being latentvariables 4 and 20.
Since the DCI framework identifies which latent variables are most predictive forthe true factors, it cannot distinguish between a correlation and a causal link.
In this case, the DCImetric recognized a correlation between dimension 4 and the "scale" factor.
However, from the latentresponse matrix (and the graph 6d) we see interventions on dim 20 have a significant effect on dim 4,but not vice versa.
Consequently, we identify dim 20 is a parent of dim 4 in the learned causal graph.
Since dim 20 is closely related with the "scale" factor the causal link to dim 4 results in dim 4 beingcorrelated with "scale".
The conditioned response matrix correctly identifies that it is dimension 20which primarily affects the scale, but indirectly also affects the shape through dimension 4.
Thisis a prime example of how causal reasoning can avoid misattributing responsibility due to spuriouscorrelations.
Conclusion
In this work, we have introduced and motivated the latent response framework including a variety oftools to better visualize and understand the representations learned by variational autoencoders.
Notably, the current analysis relies on a certain degree of axis-aligned structure in the latent space, whichmakes these tools especially useful for understanding the structure of disentangled representation.
Another limitation is that computing the latent response maps to, for example, improve interpolations,does not scale well for the large representations of high fidelity generative models [8].
Consequently,our experiments thus far have focused on synthetic datasets designed for evaluating disentanglementmethods.
However, latent responses do not require any ground truth label information, which isparticularly promising for better understanding representations of real datasets and consequentlyspeed up development of not just better performing representation learning techniques, but also moreinterpretable and trustworthy [71] models.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The disentanglement metrics don’t have error bars, as wefocus on the trends between samples rather than specific quantitative comparisons
(d) Did you include the total amount of compute
and the type of resources used (e g., type
of GPUs
, internal cluster, or cloud provider)? [Yes If you are using existing assets (e g., code, data, models) or curating/releasing new assets..
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
d) Did you discuss whether and how consent was obtained
from
people whose data
you’re using/curating? N/A All
datasets
we use are standard benchmarks
(e) Did you discuss whether the data you are using/
curating contains
personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants
and
screenshots , if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]