U NSUPERVISED D ISENTANGLEMENT WITH T ENSOR P RODUCT R EPRESENTATIONS ON THE T ORUS
Anonymous authors
Paper under double-blind review
A BSTRACT
The current methods for learning representations with auto-encoders almost exclusively employ vectors as the latent representations.
In this work, we propose to employ a tensor product structure for this purpose.
This way, the obtained representations are naturally disentangled.
In contrast to the conventional variations methods, which are targeted toward normally distributed features, the latent space in our representation is distributed uniformly over a set of unit circles.
We argue that the torus structure of the latent space captures the generative factors effectively.
We employ recent tools for measuring unsupervised disentanglement, and in an extensive set of experiments demonstrate the advantage of our method in terms of disentanglement, completeness, and informativeness.
The code for our proposed method is available as a supplementary material.
I NTRODUCTION
Unsupervised learning methods of disentangled representations attempt to recover the explanatory factors z of variation in the data.
The recovered representation, c , is expected to be: (i) disentangled , i.e., each element in c should vary one of the generative factors in z (ii) complete in the sense that all generative factors z can be controlled by the latent representation c , and (iii) informative , that is the ability to relate via a low capacity model (e.g., a linear model) the generative factors z and the latent representation c .
It has been shown that solving this task without further assumptions is impossible, since there is an inﬁnite number of bijective functions f that map between c and z = f ( c ) (Locatello et al., 2019) with all corresponding generative models having the same marginal distributions of the observations.
In this work, we propose to use representations that are a tensor product of elements, which take values on unit circles S 1 , claiming that this structure is suitable for the effective recovery of the underlying modes of variation.
This claim follows from entropy considerations.
One wishes to have a representation that has a low entropy of entanglement .
The entropy of entanglement is measured by evaluating the number of non-zero eigenvalues of the Schmidt decomposition of the representation.
If a representation can be described by only one term, which is an outer product of the orthogonal basis functions, it has the lowest possible entanglement entropy, zero.
If a representation requires the use of more than one term in the decomposition, it is entangled and its entanglement entropy can be quantiﬁed, e.g. by the
Von Neumann entropy constructed from non-zero eigenvalues.
The tensor product of n unit circles, S 1 , takes the shape of an n -torus T n = ( S 1 ) n , and has a low entanglement property.
This representation has the advantage that it can capture the periodicity structure of the generative factors, where each circle controls a different aspect of the original distribution.
Unlike other generative models, which rely on the Gaussian distribution as a prior, T n is a compact manifold, and therefore any function that acts on it, such as a decoder, has to interpolate only, but not extrapolate when generating new instances.
In this work, we present the T D -VAE, a Variational Auto Encoder whose latent space resides on the
T D torus manifold.
In an extensive set of experiments with four different datasets, we compare the torus latent representations with others proposed in the literature.
We show a clear advantage of the torus representation in terms of various measures of disentanglement, completeness, informativeness and propose a new metric, the DC-score, which assesses the combined disentanglement and com- pleteness performance.
We present a detailed quantitative and qualitative analysis that supports our method.
R ELATED W ORK
Generative models aim to create new instances that are indistinguishable from a given distribution.
Auto encoders (Kramer, 1991) achieve this by constructing the identity function using a composition of two components, an encoder and a decoder.
The encoder is tasked with compressing the input into a latent representation with a dimension smaller than the input’s, whereas the decoder is responsible for reconstructing the original input from the latent space.
Such auto encoders usually overﬁt to their training data and are very limited when tasked with generating new samples.
Instead of mapping each input instance to a constant vector, Variational
Auto Encoders (Kingma & Welling, 2013) (VAE) map each instance to a pre-deﬁned distribution, by minimizing both the encoder-decoder reconstruction loss and a KL-divergence term that depends on the prior distribution.
VAE’s are capable of generating new instances by sampling a latent vector from the predeﬁned distribution; however, the interpretation of latent space components is typically obscure and the mapping between them and the dataset properties may be complex.
In order to increase the latent space interoperability, various modiﬁcations of the original VAE were introduced.
These variants try to achieve a factorization of the latent space components with each component corresponding to a speciﬁc simple characteristic of the dataset.
In β -VAE (Higgins et al., 2016), the weight of the KL-divergence term is increased w.r.t.
the reconstruction term, which results in better factorization of the latent space.
However, this introduces a new hyper-parameter and departs from the theoretical derivation of the ELBO term.
For example, large β parameters may prevent the conditional distribution from modeling the data faithfully.
DIP-VAE (Kumar et al., 2017) introduces a regularizer that constrains the covariance matrix of the posterior distribution.
This term is added to the original VAE loss function, along with two hyper-parameters, and helps achieve better factorization.
Factor-VAE (Kim & Mnih, 2018) adds a Total Correlation (TC) approximation penalty over the produced latent codes and introduces an additional hyper-parameter.
2. D ISENTANGLEMENT M ETRICS
In order to evaluate the expressiveness of different generative models, a plethora of disentanglement deﬁnitions and metrics were suggested in the literature (Do & Tran, 2019; Eastwood & Williams, 2018), see Locatello et al.
for a comprehensive review.
In this work, we adopt the deﬁnitions and metrics introduced in Eastwood & Williams (2018) which provides a successful set of disentanglement metrics.
The authors of Eastwood & Williams (2018) introduce three metrics, Disentanglement , Completeness and Informativeness (DCI), to quantify the disentanglement properties of the latent space and its ability to characterize the generative factors of a dataset.
Given a dataset that was generated using a set of K generative factors, (cid:126)z ∈ R K , and that it is required to learn latent codes, (cid:126)c ∈ R D of
D -dimensional representation.
Ideally, in an interpretable representation, each generative factor z i would correspond to only one latent code c a .
It is also beneﬁcial if the mapping is linear, as the correlation between generative factors and codes can then be easily assessed.
Generating new instances that are indistinguishable from the original dataset distribution requires that the latent codes cover the whole range of the generative factors.
Furthermore, such a representation provides the ability to modify a speciﬁc property of a generated instance by directly tuning the corresponding latent code.
The DCI metrics aim to quantify the relationship between codes and generating factors by having a single number that characterizes the relative importance of each code, 1 c a , in predicting a factor, z i , which in turn deﬁnes an importance matrix R ai .
To construct the importance matrix, K regressors are trained to ﬁnd a mapping between z i and (cid:126)c , ˆ z i
= f i ( (cid:126)c ) .
In this work, we follow Eastwood &
Williams (2018) and infer the importance matrix using a lasso linear regressor’s weights, W ia , by
R
[Equation]
=
[Equation]
| .
Once the importance matrix R ai is obtained, the DCI metrics can be deﬁned explicitly.
The disentanglement is given by
[Equation]
where
[Equation]
High disentanglement means that each entry of the latent vector (cid:126)c corresponds to only one element in (cid:126)z (in a linear sense), that is, each code element, c a , affects one generating factor.
The disentanglement metric deﬁned above differs from Eastwood & Williams (2018) by a correction factor rank( R ) K .
When the rank of R is equal to the generative factor’s dimension, this correction equals 1, and does not affect the metric.
However, it does make a difference when the number of codes is smaller than what is needed to account for all generating factors.
Typically, one assumes that there are at least as many expressive codes as the number of factors.
When this is not the case, it means that even if we have disentanglement among the expressive codes, their number is not sufﬁcient, and our correction accounts for this.
Note that the correction has no inﬂuence if there are irrelevant code elements in addition to a sufﬁcient number of expressive ones.
The ρ factors handle cases where some of the code dependence on the factors is very weak (namely irrelevant codes), while other codes depend mainly on one factor and hence should not be of equal importance.
The completeness is deﬁned by
[Equation]
In contrast to the disentanglement metric, which considers a weighted mean over the generative factors, motivated by the fact that irrelevant units in (cid:126)c should be ignored, here all codes are treated equally.
A situation in which each factor z i is explained by only one element of the code c a results in high completeness.
The informativeness is the MSE between the ground-truth factors, z , and the predicted values, f j ( (cid:126)c
[Equation]
[Equation]
Disentanglement and completeness values range between zero and one, where one is the best value, while the best value for informativeness is zero.
Disentanglement and completeness alone do not provide sufﬁcient information on whether a representation factorizes properly.
For example, consider the case of a representation where only one generative factor is described by all the codes.
While this representation is completely disentangled, it is not complete, and is therefore meaningless.
In order to have a meaningful representation, both disentanglement and completeness need to be high.
Thus, we introduce a new score, called the
DC-score, which accounts for both disentanglement and completeness.
We deﬁne the DC-score as the geometric mean of the two metrics, DC-score = √DC .
This way, only cases where both scores are high will result in a high DC-score.
Furthermore, the score will favor cases where both disentanglement and completeness are comparable rather than having different values while having the same arithmetic mean.
M ETHOD
Assume a pre-determined set of vectors sampled from an unknown distribution χ , each vector containing K independent factors, (cid:126)z ∈ R K .
The training dataset contains samples x
I = F
( (cid:126)z I ) ,
I = 1 , . .
, N that are generated using a generative function F .
In unsupervised disentanglement receives
The learner has to recover a set of representation vectors (cid:126)c I that are linked to (cid:126)z I in a way that is bijective and disentangled, see Section 2.1.
Furthermore, in order to generate new samples, it is also necessary to be able to sample new representation vectors (cid:126)c new , and to obtain a generative function G such that the new generated samples are from the same distribution of F ( (cid:126)z ) , where (cid:126)z ∼ χ .
In this work, we view the latent representation vector (cid:126)c as the angles associated with a list of D two-dimensional unit vectors m a ∈ R 2 , 1 ≤ a ≤ D , i.e., ∀ a , (cid:107) m a (cid:107) = 1 .
Using the two-dimensional vectors we deﬁne
[Equation]
where ⊗ is the outer product operator, α a ∈ { 0 , 1 } and vec is the vectorization operation (equivalent to ﬂattening the tensor).
Next, deﬁne the operator V , that given m 1 , . .
, m D , concatenates v prod together with v orient , V ( m 1 , . . .
, m D ) =
[ v prod ; v orient ] .
The vector v = V ( m 1 , . . .
, m D ) resides in a vector subspace of R 2 D + D , deﬁned by only a set of D parameters.
The additional D elements of v orient are required to ensure that the mapping V is bijective.
For example, this can be seen by examining the case of D = 2 : let m 1 ≡ (cos θ 1 , sin θ 1 ) , m 2 ≡ (cos θ 2 , sin θ 2 ) , m (cid:48) 1 ≡ − m 1 and m (cid:48) 2 ≡ − m 2 .
Then m 1 ⊗ m 2 = m (cid:48) 1 ⊗
m (cid:48) 2 .
A natural way of acquiring a random point on the circle, S 1 , is by sampling two independent
Gaussians, ˆ m α k a ∼ N ( µ α k a , σ α k a ) .
Each tuple of these vectors is then normalized to have a unit norm,
[Equation]
Assume that the elements, ˆ m α k a , follow the normal distribution with a zero mean and a standard deviation of 1 , ˆ m α k a ∼ N (0 , 1) , then the vectors m a follow the uniform distribution on S 1 .
For the purpose of obtaining the distribution parameters of ˆ m α k a , the encoder e is applied on an instance, x ,
[Equation]
The reparametization trick (Kingma & Welling, 2013) is then used to obtain a set of normally distributed vectors.
Denote by S the sampling operator, we sample the coding vector as
[Equation]
where each component, ( a, α a ) of m , a = 1 . .
D , α = 0 , 1 is sampled from N ( µ αa , σ αa ) .
The decoder G then acts on V ( m 1 , . . .
, m D ) to generate a new instance, ˜ x .
The requirement that the generated instance is indistinguishable from the original distribution F ( χ ) translates to minimizing the ELBO loss function.
First, it maximizes the log-likelihood probability of ˜ x to be similar to the training sample it reconstructs by having an l2 loss term.
Second, to encourage a uniform distribution on the torus, it contains a KL-divergence term with respect to N (0 , 1 D ) .
The overall expression for the loss function used during the optimization of the networks e and G is [Equation]
[Equation]
where β is a hyperparameter introduced in β -VAE (Higgins et al., 2016) and r I ∼ N (0 , 1 D ) .
As the latent representation resides on the torus, sub-vectors 2 D components of m a may be described by D angles, θ a ∈ [0 , 2 π ] (the angles θ a are identiﬁed with the codes c a ).
In order to generate new instances, identify, m 0 a = cos θ a , m 1 a = sin θ a . and apply the decoder G ( V ( m )) to these elements.
The torus topology, T D , is both compact and periodic.
Since it is compact, for every sampled θ a , there is a nearby θ (cid:48) a from the training set, thus, the network G can be viewed as an interpolator.
This is in contrast to vectors sampled over R D , in which G has to extrapolate.
Furthermore, being periodic, this topology is able to exploit a periodic structure in the generative factors.
Consider a common generative factor associated with rotations.
A rotation cannot be represented by a single non-compact variable; therefore encoding a rotation in a latent space requires the entanglement of two components.
However, on the torus, only one compact dimension can be used to identify this generative factor.
(cid:18) m
[Figure]
11 (cid:19) ⊗ (cid:18) mm 12 (cid:19)
  m 01 mm 01 mm 11 mm 11 mmm  m 01 mmm~m
V ( ~m )
[Figure]
E XPERIMENTS
We compare our method to the leading VAE architectures from the current literature.
The baseline methods are (i) β -VAE (Higgins et al., 2016), (ii) DIP-VAE-II (Kumar et al., 2017), and (iii)
Factor-VAE (Kim & Mnih, 2018).
The code for all methods was taken from the Pytorch-VAE repository (Subramanian, 2020) (Apache License 2.0).
The network architectures for both the encoder, decoder and the Factor-VAE’s discriminator (which we do not use) follow those in Eastwood & Williams (2018), see Table 1.
Each horizontal line in Table 1 denotes a skip-connection.
All methods were trained using ADAM (Kingma & Ba, 2014) with a learning rate of 0.0001 and a batch size of 144 .
The hyperparameters used for DIP-VAE were λ diag = 10 and λ off-diag = 1 . 0 .
All methods were trained for a maximum number of 50 epochs, and the metrics were evaluated using the model that performs best on the validation set with respect to MSE.
Furthermore, each baseline method was evaluated using two latent representations sizes, D = 10 , 128 for all datasets.
For our method, T 8 -VAE, we use 8 circles, (cid:0)
S(cid:1) 8 .
This results in V ∈ R 136 , leaving the number of parameters in the decoder comparable to the other baseline methods.
Each experiment was evaluated with ﬁve different seeds and the mean value is reported.
In order to compute the DCI metrics, we used a linear lasso regressor based on the implementation of
Eastwood & Williams (2018) (MIT License) and Zaidi et al.
(2020) (Apache License 2.0).
For each generative factor, a lasso regressor ﬁts between the normalized codes obtained by the encoder, e , to a normalized factor, namely f i ( (cid:126)c ) for each z i .
For each factor, the α parameter for the regressor using 10-fold cross-validation from the set (cid:8) 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 8 ,(cid:9) , picking the one with the minimal the MSE error.
The weights of the lasso regressor are then used to construct the W ∈ R K × D matrix from which the metrics are computed as explained in section 2.1.
[Figure]
The Fr ´ echet Inception Distance (Heusel et al., 2017) (FID) is used to assess the quality of image generation.
Speciﬁcally, we use the implementation of Seitzer (2020)
(Apache License 2.0) and report the FID at the ﬁnal average pooling features of the Inception3 NN (Szegedy et al., 2016).
The teapots dataset (Moreno et al., 2016) contains
64 × 64 images of camera-centered teapots rendered using a set of 5 generative factors, azimuth that is sampled from ∼ U [0 , 2 π ) , elevation that is sampled from ∼ U (cid:2) 0 , π(cid:1) and three RGB channels, each sampled from U [0 , 1] .
All
VAEs are trained over a subset of 160 , 000 (the generative factors are unknown to all architectures), whereas the disentanglement metrics are evaluated on the remaining 40 , 000 images.
The Cars3D dataset (Reed et al., 2015)
contains 183 CAD models of different cars projected into 64 × 64 image with four elevations and 24 azimuths.
These three generative factors are split between a training set of 14054 images and a validation set of 3514 images.
The 2dshapes Dataset , introduced here and released to the public (MIT License), contains 200 ,64 × 64 images of four possible shapes, triangles, squares, pentagons and hexagons.
All shapes in the dataset are centered in the image and have a white background.
The factors controlling the images are the following: speciﬁc shape drawn from a discrete uniform distribution, a scale value between s ∼ U [20 , 40] , rotation angle θ ∼ U [0 , 2 π ] , and color RGB values r ∼ U [0 , 1] , g ∼ U [0 , 1] , b ∼ U [0 , 1] all sampled uniformly.
Since the shapes are symmetric, different angles could correspond to the same image.
The training set contains a subset of 160 , 000 instances, whereas the disentanglement metrics are evaluated on the remaining 40 , 000 validation instances.
The 3dshapes dataset (Burgess & Kim, 2018) contains 480 , 000 64 × 64 images of four possible objects, a cylinder, a tube, a sphere and a box rendered with 8 linearly spaced scale values, 10 hues linearly spaced, ﬂoor wall and object hue values and ﬁfteen different orientations, which amounts to six generative factors.
The dSprites Dataset (Matthey et al., 2017) contains 737 , 280 64 × 64 binary images of three shapes, a square, an ellipse and a heart with six different, equally spaced scale values, 40 different orientations, and 32 × 32 possible positions in the plane, amounting to ﬁve generative factors.
589 , 824 instances were used for training and evaluation was performed on the remaining 147 , 456 validation instances.
Results
Across all the datasets (Tables 2–6), our T 8 -VAE architecture obtains the highest DC-score, which combines the disentanglement and completeness scores, compared with the baselines.
In all of these cases the generated images are meaningful, as can be seen from the relatively low FID scores.
Furthermore, for teapots (Table 2), 2dshapes (Table 3) and dSprites (Table 6), we also achieve the best reconstruction MSE and FID scores.
In the 3dshapes dataset (Table 4) we get the best MSE score and comparable FID scores to β -VAE, which gets the highest score.
For the Cars3D dataset (Table 5) the results are quite different, even though our DC-score is highest.
The best performing method in terms of the MSE and FID scores is DIP-VAE-II 128, which is not among the best performing methods for the other datasets.
One plausible reason is that this dataset,
[Table] [Table]
although having only three generative factors, hides a much richer high-dimensional distribution.
The factors controlling the car model (183 in total) implicitly contain many characteristics of a car, which requires a high dimensional latent space.
Nonetheless, our method achieves the second best
FID score for this dataset.
Dependence on β and on D
In order to examine the sensitivity of our method to different tori dimensions, as well as the inﬂuence of the β parameter on KL-divergence, we examined the DC-score and reconstruction error on the Teapot dataset, using four topologies, T 4 , T 5 , T 6 and T 8 , and using ﬁve different β coefﬁcients, 0 , 1 , 3 , 6 , 9 ( 0 corresponds to no KL-divergence).
As can be seen in Fig. 4, removing the KL-divergence completely, leads to a decrease in the reconstruction error; however, the
DC-score also decreases.
The T 4 -VAE exhibits the worst reconstruction error - not surprisingly, as it can only encode four generative factors, whereas there are ﬁve generative factors in the dataset.
Both the T 5 -VAE, T 6 -VAE and T 8 -VAE have the capacity of encoding all generative factors, however, the encoder of the T 8 -VAE should ignore three of the circles whereas the encoders of the T 5 -VAE and
T 6 -VAE should ignore at most one circle.
Thus, when increasing β , the induced angles on T 8 tend to a more uniform distribution, which entangles between themselves.
This does not happen in T 5 -VAE and T 6 -VAE, since there is at most one extra component that can be entangled.
Heatmap analysis
Disentanglement results can be qualitatively evaluated through a visual inspection of the 2d heatmaps of the generative factors z i vs. the codes c a (the angles θ a in our case).
In
Figure 2 we show an example of the heatmap results for using our method with D = 6 , namely using a T 6 -VAE.
The disentanglement and completeness properties are clearly seen in the z 1 , z 2 , z 3 , zdependence on θ 5 , θ 2 , θ 0 , θ 1 respectively, while we see worse completeness for the z 0 factor.
Such images help us better understand cases where one of the scores is high and the other is low, or whether a speciﬁc regressor is suited for capturing the functional dependence.
D ISCUSSION
We proposed and analyzed a low entangled latent representation of the generating factors of a distribution.
The latent space has the topology of a torus which is a direct product of circles.
Each
0  9 0 .0 .0 .0 .0 .β
D C S c o r e
T
T T
T(a) 0      · 10 −β
M S E
T
T T
T(b)
β
F I D
T
T T
T(c)
[Table] [Table]
[Table] [Table]
circle controls one generative factor, and moving along the circle changes the corresponding aspect of the distribution.
Our latent space construction resembles the phase space structure of a classical, integrable physical system, whose dynamics can be described using action-angle canonical variables as motion on a phase space torus.
The dimension of the torus is the number of the system’s degrees of freedom, and each point on it speciﬁes the location in space and the velocity of the system at a given instant in time.
The coordinates of the point on each circle correspond to a particular periodic cycle to which the whole motion has been decomposed.
Analogously, the coordinates on each circle of a point in the latent space torus specify one of the basic properties of an element of the dataset to which the description of that element has been decomposed.
The possible intriguing relationship between the integrable structure of dynamical systems and its decomposition into its basic harmonic motions and the decomposition of the dataset into its basic constituents can be valuable in relating dynamical systems to the process of learning.
The task of unsupervised disentanglement learning is complicated by the fact that given any disentanglement representation there is an inﬁnite family of entangled representations that have the same marginal distribution (Locatello et al., 2019).
Therefore, an inductive bias is needed for the task.
Our solution in this work is based on the concept of entanglement entropy in quantum physics.
There are non-generic states in the Hilbert space that have low entanglement, and a special class of them are the product states, whose entanglement entropies are identically zero.
We employed these states for our disentanglement representation.
It is indeed easy to see that these provide a special representation, since any random unitary (or orthogonal) matrix acting on them will make our representation entangled and hence destroy the disentanglement properties of our solution.
Thus, in a sense, our inductive bias is the use of the special non-generic states (of measure zero) in the Hilbert space for our representation.
We introduced a DC-score to jointly quantify the disentanglement and completeness of the torus latent representation and calculated it, as well as various other measures in a large set of numerical experiments.
For the comparison we used ﬁve different types of datasets.
We varied the dimension of the latent space torus and showed a clear advantage of the torus latent description compared to others when the torus dimension is equal or greater than the number of generating factors.
In order to properly assess the cases where the number of expressive codes is lower than the number of generative factors, we introduced a correction to the disentanglement score, and hence to our DC-score.
One limitation of our proposed low entangled representation is in scaling to datasets containing a large number of generative factors.
The dimensionality of v increases exponentially with the dimension of the torus latent representation and is, therefore, limited in describing these situations.
This can perhaps be remedied by a different choice of embedding, e.g. working with a direct sum of tensor products, which increases polynomially.
Another limitation of the model may be in its ability to effectively describe a property of the dataset that is not naturally encoded in a circle, but is, rather, non-compact in nature.
While our torus latent representation performs better compared to other approaches, one may still wonder why it did not achieve the perfect DC-score 1 .
This can be explained by the fact that the generation of the datasets assumes a certain number of generating factors, while in fact there could be more, some indirectly combined.
For example, in a case where multiple underlying factors are combined into one explanatory factor, the disentanglement may overshoot, thereby reducing the obtained score.
R EPRODUCIBILITY S TATEMENT
A G ENERATIVE F ACTORS H EATMAPS
In this section we provide additional heatmap examples for different tori which provide a qualitative insight.
Figures 5 and 6 present the heatmaps for our method on T 4 and T 8 torus topologies, respectively, with their D , C and DC scores.
When the number of generative factors is larger than the number of torus dimensions
[Figure]
D , one can expect some mixing between the generative factors and the latent codes.
This can be observed for instance for T 4 in Fig.
As the dimension of the latent codes increases
[Figure]
see Fig.
[Figure]
the D , C and DC -score metrics improve.
This effect on the disentanglement results from the prefactor rank ( R ) K in equation 1, which depends on the ratio of the numbers of latent codes and generative factors when the latter is larger.
[Figure]
[Figure]
B O N THE C ONNECTION TO Q UANTUM P HYSICS
The assumption is there is a set of elementary generative factors
z i , i = 1 , .., n that are disentangled from each other, and from which the data distribution can be constructed.
The task is to learn these elementary factors and represent them by latent codes c a , a = 1 , ..., m .
Ideally we would have liked to have n = m and a bijective map f that associates to any c a one z i and vice versa.
We are working with a linear representation and since quantum mechanics is a linear theory it inspires our choice of the latent space representation.
In quantum mechanics states are represented by vectors in Hilbert space and the operators that act on them are linear.
In our case, the generative factors constitute a state and so do the latent codes.
The linear regressor relating them is a linear operator.
We represent each elementary factor
z i as a state (qubit) in a Hilbert space |
z i (cid:105) ∈ F and the complete set of elementary factors by a state (quantum register of n qubits) in the tensor product of these
Hilbert spaces | z 1 , ..., z n (cid:105) =
| z 1 (cid:105) · · · | z n (cid:105) ∈ H =
F ⊗ F · · · ⊗ F .
Note, that only a measure zero of n -qubit states can be represented in such a way as a product state and the reason that we can do that is the assumption that z i are disentangled.
Entanglement can be viewed as a measure of how far the state is from being a product state.
One way to measure it is by taking the supremum on the absolute value of the inner product of the state with all the possible product states.
Another measure is the Von-Neumann entropy that is calculated by constructing a density matrix ρ as the outer product of | z 1 , ..., z n (cid:105) , decomposing z i to e.g. two subsets A and B such that ρ ∈ H A ⊗ H B and tracing out part B , ρ A =
Tr B ρ .
The eigenvalues of ρ A encode the entanglement between A and B .
Speciﬁcally, the Von-Neumann entropy is S V N = − Trρ A log ρ A
and it vanishes iff the state
| z 1 , ..., z n (cid:105) is a product state.
Returning to our case, we construct the latent space representation using a tensor product since it should capture the above structure of the generative factors.
The measure of disentanglement D a quantiﬁes it by the − P log P term which is the analog of the Von-Neumann entropy.
Note that there is a difference between our case and quantum mechanics: in quantum mechanics an elementary qubit state is represented by two angles ( θ, φ ) while in our case and elementary latent code is one angle parametrizing a circle of the torus.
One may inquire what would have gone wrong in our analysis if we chose that space of latent codes to be another curved compact manifold such as a sphere.
In such a case we should have covered the manifold with local patches such that there would be a linear map between the generative factors and the latent codes on each patch and a transition function between them.
Such a representation would be entangled and will fail, e.g. on the sphere near the pole since rotating by the azimuthal angle would not change any property.