Disentangling Images with Lie Group Transformations and Sparse Coding
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
Discrete spatial patterns and their continuous transformations are two important regularities in natural signals.
Lie groups and representation theory are mathematical tools used in previous works to model continuous image transformations.
On the other hand, sparse coding is an essential tool for learning dictionaries of discrete natural signal patterns.
This paper combines these ideas in a Bayesian generative model that learns to disentangle spatial patterns and their continuous transformations in a completely unsupervised manner.
Images are modeled as a sparse superposition of shape components followed by a transformation parameterized by n continuous variables.
The shape components and transformations are not predefined but are instead adapted to learn the data’s symmetries.
The constraint is that the transformations form a representation of an n -dimensional torus.
Training the model on a dataset consisting of controlled geometric transformations of specific MNIST digits shows that it can recover these transformations along with the digits.
Training on the full MNIST dataset shows that it can learn the basic digit shapes and the natural transformations such as shearing and stretching contained in this data.
This work provides the simplest known Bayesian mathematical model for building unsupervised factorized representations.
1. Introduction
A major challenge for both models of perception and unsupervised learning is to form disentangled representations of image data, in which variations along the latent dimensions explicitly reflect factors of variation in the visual world.
An important dichotomy among these factors of variation is the distinction between discrete patterns vs. continuous transformations (Mumford and Desolneux, 2010), the former referring to factors such as local shape features or objects and the latter typically referring to geometric transformations such as translation, scaling, and rotation.
Importantly, these factors are not overtly measurable but rather entangled in the pixel values of an image.
Learning to disentangle the shapes and transformations inherent in image data is an important task that many previous works have attempted to address using architectures such as bilinear models, manifold models, and modified VAEs (Tenenbaum and Freeman, 2000; Grimes and Rao, 2005; Olshausen et al.,
; DiCarlo and Cox, 2007; Cadieu and Olshausen, 2011; Bengio et al., 2013; Cheung et al., 2014; Dupont, 2018).
One class of previous approaches has used Lie groups to model image transformations to varying degrees of generality (Rao and Ruderman, 1999;
Miao and Rao, 2007; Culpepper and Olshausen, 2009; Sohl-Dickstein et al., 2010; Cohen and Welling, 2014, 2015; Gklezakos and Rao, 2017).
A Lie group can be thought of as a parametric family of continuous transformations, and is a natural tool not only for modelling image transformations but also for learning disentangled representations more generally (Higgins et al., 2018; Pfau et al., 2020).
However, all of these previous approaches focus solely on learning transformations but not the discrete patterns in the dataset.
On the other hand, sparse coding is a widely known unsupervised algorithm for learning a dictionary of discrete spatial patterns from which images are composed (Olshausen and Field, 1997).
Neurons in their hidden layer have been shown to recapitulate receptive field properties of V1 neurons after training on natural images, suggesting that the early human visual system may be employing the computational strategies of sparse coding.
However, sparse coding does not model image transformations explicitly, and as a result, information about both shape and transformations are entangled in its representations.
We propose a novel unsupervised algorithm, Lie Group Sparse Coding (LSC), that combines the advantages of sparse coding and Lie group learning.
Particularly, our work is inspired by the work of Cohen and Welling (2014), which introduces the mathematical framework of representation theory to disentanglement learning.
We build on this framework by including a latent representation of shape via sparse coding, which is inspired by the work of Gklezakos and Rao (2017).
The proposed algorithm infers both the sparse representation of shape and its transformation from an image.
It learns the shape dictionary and transformation operators from the data through an iterative process akin to expectation-maximization.
We demonstrate our model’s capacity to disentangle representations of shape and transformation from controlled datasets where the ground truth is known and from the full MNIST dataset where both the underlying shape categories and factors of variation due to style are unknown and must be learned from the data.
This work provides the simplest known Bayesian mathematical model for building unsupervised factorized representations.
2. Preliminaries
Sparse coding seeks to learn a dictionary of templates { Φ i } , such that each image I can be described by a sparse linear combination of these templates
I = (cid:80) i Φ
i α
i .
However, as mentioned in the introduction, sparse coding does not model image transformations explicitly.
Transforming an image changes its sparse code α =
[ α 1 , · · · , α K ] T in a non-equivariant manner, meaning form and transformations are entangled in the sparse code representation.
To address this problem, we add an operator T ( s ) that explicitly models the transformations, so that images are now represented as transformations of patterns generated from the sparse coding model, I = T ( s )
Φ α , where Φ =
[Φ 1 , · · · , Φ K ].
Inspired by the work of Cohen and Welling (2014), we choose to model the transformations
T ( s ) as actions of compact, connected, commutative (CCC)
Lie groups on images.
Many transformations - including rotations, rigid motion, and translations - can be understood as Lie groups, or more informally, a group of continuous symmetries of a space.
CCC Lie groups are a small subset of Lie groups that have a simple mathematical structure and are equivalent to n -dimensional tori (Dwyer and Wilkerson, 1998).
By the Peter-Weyl theorem, these transformations can be decomposed as T ( s ) =
W R ( s ) W T , where W is an orthogonal matrix, ω l ∈ Z n , and R ( s ) is a block diagonal matrix containing 2D-rotation subblocks of the form
[Equation]
In practice, as we show in Section 4, this parametrization supports the learning of various common transformations such as translation, rotation, shearing, stretching, etc.
The dependence on the parameter s takes a simple form of a block diagonal matrix with 2x2 blocks which allows for efficient inference and learning.
Interested readers may refer to Appendix A for detailed explanations of the theory behind the parametrization T ( s )
=
W R ( s ) W T .
A minor downside of restricting our attention to CCC Lie groups is that it imposes several theoretical constraints on the class of learnable transformations: Compactness enforces the transformations as periodic; connectedness precludes discrete transformations like reflections; commutativity excludes non-commutative transformations such as 3D rotations.
Fortunately, while many transformations violate these constraints in theory, we demonstrate in Sectionthat they can still be approximately learned in practice.
3. Algorithm
3.1 Probabilistic Model.
Let I ∈
R D be the input image, where D is even, and let
L ≤ D/ 2.
We model I as
[Equation]
where W ∈ R D × 2 L is a matrix with orthonormal columns (i.e. W T W = 1 ), Φ ∈ R D × K is the dictionary with each column having unit L2 norm, and R ( s ) is the block diagonal matrix with the subblocks R l ( s ) defined in Equation (1), where l = 1 , · · · , L .
The random variables in the model are s , α , and ϵ , which are all independent of each other.
The transformation parameter s ∈ R n is a random vector whose components s i are i.i.d.
with s i ∼ Unif(0 , 2 π ).
The sparse code α ∈ R K is also a random vector whose components α k are i.i.d.
with α k ∼ Exp( λ ), the exponential distribution.
The random noise ϵ is i.i.d.
Gaussian with variance σ 2 and zero mean.
Given an image I , we infer the transformation parameters s and sparse code α according to their posterior distribution.
Given a large ensemble of images, we aim to learn the parameters θ = { W , Φ } by maximizing their log-likelihood.
The inference and learning procedures are presented in 3.2.
The weights ω l ∈ Z n , which appear in the block diagonal rotational matrix R ( s ), are chosen to evenly tile the n -dimensional domain, with the constraint that if ω l is chosen then − ω l is omitted.
This is because, as shown in the derivation of T ( s ) =
W R ( s ) W in
Appendix C, each 2x2 block in R ( s ) is obtained by combining ω l terms with opposite signs in Equation (6).
A multiplicity m ≥ 1 is then assigned to the weights, meaning each ω l is repeated m times.
Finally, we select the first L ω l sorted by ascending frequency
|| ω
l
|| 2 .
This model combines aspects of sparse coding (Olshausen and Field, 1997) and the work by Cohen and Welling (2014) on learning irreducible representations of commutative Lie groups.
If the term W R ( s ) W T is replaced by the identity matrix, the model is identical to the sparse coding model of Olshausen and Field (1997).
If Φ α is replaced by a transformed image I ′ , and n = 1 (scalar s ), the model is identical to the one-parameter transformation model by Cohen and Welling (2014).
Algorithm θ =
{ W , Φ } ← { W 0 , Φ 0 } ▷ Initialize model parameters while W , Φ not converged do
Get normalized image batch
I α ← α▷ Initialize sparse coefficients for i ∈ { 1 , · · · , T } do ▷ Compute ˆ α = arg max α P θ ( α | I )
Compute P θ ( s |
I , α ) ∆ α ← E s ∼ P θ ( s |
I , α ) [ ∇ α ln P θ ( I | s , α )]
+ ∇ α ln P θ ( α ) ▷ Compute gradient for α α ← FISTA update( α , ∆ α )
▷ Update α using FISTA end
ˆ α ← α
Compute P θ ( s | I , ˆ α ) ∆ Φ ← E s ∼ P θ ( s | I , ˆ α ) [ ∇ Φ ln P
θ ( I | s , ˆ α )]
▷ Compute approximate gradient for Φ Φ ← Normalize( Φ + ∆ Φ )
▷ Update Φ and normalize columns ∆ W ← E
s ∼ P θ
( s | I , ˆ α ) [ ∇ W ln P
θ
( I | s , ˆ α )]
▷ Compute approximate gradient for W
W ← RiemannianAdam
( W , ∆ W )
▷ Update W with RiemannianAdam optimizer end 3.2 Inference and Learning.
The inference and learning procedure is outlined in Algorithm 1.
The general idea is as follows: to learn the model parameters θ = { W , Φ } , we perform gradient ascent on their log-likelihood using the approximate gradient ∇
[Equation]
(3) where ˆ α = arg max α P θ ( α | I ) is the MAP estimate of the hidden variable α
(see Appendix E for derivation details).
Notice moreover that
[Equation]
and that the right hand side Equation (4) is just our approximate gradient in Equation (3).
Hence, another interpretation is that we replaced the original objective function ln P θ ( I ) by the approximate objective function ln P
θ
( I | ˆ α ), which is much more computationally tractable than the original (see Appendix D for an explicit formula for ln P
θ ( I | α ), and
Appendix E for derivation details).
The approximation step assumes that the posterior distribution P ( α | I ) is sharply peaked around ˆ α , meaning P ( α | I )
≈
δ ( α − ˆ α ), where δ ( x ) is the Dirac delta function.
This is the same approximation used in the sparse coding algorithm by Olshausen and Field (1997).
Also note the similarity between this approach and the EM algorithm, as each gradient step partially maximizes the expectation of the log likelihood with respect to the posterior distribution of the hidden variable given the current parameters.
On the other hand, the MAP estimate ˆ α is computed by gradient ascent on the log-posterior, whose gradient is computed via
[Equation]
(see Appendix E for derivation details).
Note that both Equation (3) and Equation (5) require inferring the posterior distribution of the transformation variable P θ ( s |
I , α ).
We show how this posterior distribution can be computed in Appendix D.
One may wonder whether the posterior distribution P ( s | I , α ) can be approximated by δ ( s − ˆ s ), just as we did for α in Equation (3).
This would allow us to avoid computing the full distribution and use the point estimate ˆ s = arg max s P θ ( s |
I , α ) (optimized using gradient descent) in order to update the model parameters.
We find empirically that using this approach leads to worse convergence of the model parameters.
We believe there are two reasons for this: first, during the initial phase of training, the posterior distribution of s has many local extrema and hence using a single point estimate ˆ s is a bad approximation; second, the presence of many local extrema during initial stages means it is easy to get stuck in a local minimum using gradient descent.
Furthermore, when the number of transformation parameters is small (which is the case for the experiments in this paper), it is faster to compute the full distribution of s than performing gradient descent to find ˆ s , as the full distribution can be computed in a highly parallelized manner but gradient descent cannot.
Gradients computation involves the expectation term ¯ R =
E s ∼ P θ
( s |
I , α )
[ R ( s )] (see Appendix E for details), which is obtained by numerically integrating (cid:82) s P θ ( s |
I , α ) R ( s ) with
N samples along each dimension.
An efficient way of computing this quantity using Fast
Fourier Transform is detailed in Cohen and Welling (2015), although numerical integration is adequate for our purposes.
We used FISTA to greatly speed up the inference of α by around a factor of 10 (Beck and Teboulle, 2009).
While the objective ln P θ ( α | I ) is not guaranteed to be convex in α , which violates one of the theoretical assumptions of FISTA, we find that in practice α always converges well.
Details of the usage of FISTA in LSC is provided in
Appendix F.
As mentioned in section 3.1, there are two hard constraints on the model parameters
W , Φ : the columns of W must be orthonormal, and the columns of Φ must have unit norms.
The constraint on W comes from that we only want to learn orthogonal representations, while the unit norm constraint on Φ prevents Φ from growing without bound (see Olshausen and Field (1997)).
We optimized Φ by performing projected gradient descent, normalizing each column of Φ after each gradient step.
For W , we used the Riemannian ADAM optimizer to optimize W on the Stiefel manifold (the manifold of matrices with orthogonal columns).
We used the Riemmanian ADAM implementation from the python package geoopt (Kochurov et al., 2020).
Using Riemannian ADAM instead of projected gradient descent, as is done in
Cohen and Welling (2014), was empirically found to speed up convergence by around 3times.
4. Experiments
To demonstrate that our algorithm can successfully disentangle different forms and transformation factors, we first train the model on two synthetic datasets in which the generative models are fully known.
We set K = 10 and n = 2, meaning there are 10 dictionary templates and 2 latent dimensions for the transformation parameter s .
In the first dataset, we select one image from each of the 10 digit classes in 28x28 MNIST, then applyrandom 2D translations to each of the 10 selected images, totaling 60000 images.
Both vertical and horizontal translations are drawn uniformly between − 7 and 7 pixels.
In the second dataset, instead of 2D translations, we apply 6000 random rotations and scaling to the 10 images.
Rotation is drawn uniformly between − 75 ◦ and 75 ◦ , while scaling is drawn uniformly between 0 . 5 and 1 . 0.
Figure 4 in Appendix G shows 80 images from each dataset.
For each dataset, LSC can learn the 10 digits and the two operators that generated it (training details in Appendix I).
Figure 1 shows the learned W matrices.
The learned dictionary Φ is shown within Figure 2.
Notice that each of the learned dictionary template Φ i corresponds to one of the digits.
Latent traversals of the two operators are shown at the bottom of Figure 2, in which we select 2 random images from the test set and apply the learned operator T ( s ) with varying s to those images.
It is clear from the figure that the learned transformations are exactly the 2D translation operators and the rotation + scaling operators, respectively.
Strikingly, even though the rotation + scaling dataset contains only rotations between − 75 ◦ and 75 ◦ , the model learns the full 360 ◦ rotation.
This ability to generalize and correctly extrapolate the transformation present in the dataset is a feature of the Lie group structure that is built into LSC.
One might notice a slight mixture of rotation and scaling in the latent traversal plots in Figure 2, which may seem to suggest that the algorithm failed to disentangle rotation and scaling completely.
However, we note that there is no reason to require a one-to-one correspondence between ( s 1 , s 2 ) and (rotation, scaling) since any other linear combination of these dimensions still allows these transformations to be performed perfectly.
[Figure]
The inference process is demonstrated at the top of Figure 2.
An image I is given to the model, which then performs the inference given in section 3.2 to yield the MAP estimate of the sparse coefficients
ˆ α and the posterior distribution of the transformation parameter P ( s | I , ˆ α ).
A reconstruction of the input is then computed as ˆ I = T (ˆ s )
Φ ˆ α , where ˆ s = arg max s P ( s |
I , ˆ α ) is the MAP estimate of the s .
It can be seen from the figure that the inferred α is 1-sparse.
The posterior distribution of s is sharply peaked at a particular value.
We next trained our model on the full MNIST dataset to demonstrate its capacity to disentangle shape and transformations on data where the actual transformations are less clear and difficult to ascertain from first principles (see Appendix I for training details).
The left side of Figure 3 are analogous to Figure 2.
We see that LSC learns 9 out of thedigits with its dictionary Φ , while the latent traversal plots show that the model has learned horizontal stretching and shearing transformations.
Figure 1 also shows the columns of the learned W matrix.
The reason that the dictionary did not learn the digit “1” is because, during inference, it always uses the learned horizontal stretching transform to “squeeze” a “0” into a “1”, so that a separate “1” template is not necessary.
In our experiments we found
[Figure]
that one could learn the “1” template if a prior on s with a narrow peak near 0 is used instead of a uniform prior, with the intuition being that the narrow prior prevents large horizontal stretching transformations from being used to squeeze a “0” into a “1”.
We compare LSC to sparse coding alone by training it on MNIST as well with the same number of dictionary elements.
The learned dictionary Φ , as well as the inference process, is shown on the right of Figure 3.
For comparison, the same two example inputs I were used in inference for both LSC and sparse coding.
As can be seen, the reconstruction ˆ I is significantly degraded without a transformation model.
The inferred sparse representation ˆ α is also less sparse and less interpretable than LSC – e.g., the ‘8’ is described as a combination of ‘2’,‘0’ and ‘9’.
Also, note that the dictionary Φ learned by sparse coding requires more than one template to capture the different poses of a digit, such as the slanted ‘1’ in the 4th dictionary element and the upright ‘1’ in the 7th element.
We demonstrate the improvement of LSC over sparse coding quantitatively in Table 1.
We train both algorithms on the same datasets while ensuring that both models use the same dictionary size and sparsity cost.
After training for 20 epochs, we evaluate the two algorithms on a test set and calculate the reconstructed images’ SNR (signal-to-noise ratio).
LSC outperforms sparse coding in all settings, and the improvement in SNR is over 10 times on the 2D translation dataset.
This is particularly remarkable considering that we set the dimension of the transformation parameter s in LSC to be n = 2, meaning LSC only has two more degrees of freedom than sparse coding during the inference process.
5. Related Works
This work is inspired by Gklezakos and Rao (2017), who proposed the Transformational
Sparse Coding algorithm (TSC) that combines ideas from Lie group theory with sparse coding like LSC.
A significant difference is that in TSC, the group representation is predefined rather than learned; more concretely, the transformations in the generative model are fixed
[Figure]
[Table]
K = 10, λ =
[Table] [Table]
±
[Table]
K =
[Table]
λ =
[Table]
K = 10, λ = 10 .
[Table] [Table]
K = 100, λ = 1 .
[Table]
to be the family of 2D affine transformations
[Table]
whereas LSC allows for the learning of any transformations as long as they form a representation of a CCC Lie group.
Another difference is that in TSC, images are modeled as being a combination of transformed variants of root templates.
In contrast, in LSC, images are modeled as transformed versions of a combination of templates.
In other words, TSC focuses on the transformations of local features, whereas
LSC focuses on the global image transformations.
The inference and learning of transformations in our algorithm are based on the TSA algorithm (Toroidal Subgroup Analysis) by Cohen and Welling (2014) which learns the representation of a one-parameter subgroup of the maximal torus.
There are two main differences: first, TSA learns image transformations given pairs of images with the same shape, whereas LSC learns representations of both shape and transformation without any prior knowledge of the shapes contained in the images; second, LSC generalizes the oneparameter subgroup representation learned in TSA to the representation of an arbitrary
N -dimensional torus, allowing for the learning of a wider variety of transformations.
Cohen and Welling (2015) later extended their work to learning 3D object rotations using the group representation of SO (3), although it again only learns the transformations but not the discrete spatial patterns.
LSC grew out of works on separating form and transformation using bilinear models.
Bilinear models assume a generative process in which two vectors, one encoding form, and the other encoding transformation, combine multiplicatively, meaning the output is linear with respect to both vectors (Rao and Ballard, 1998;
Tenenbaum and Freeman, 2000; Grimes and
Rao, 2005; Olshausen et al., 2007; Memisevic and Hinton, 2010).
Some bilinear models do not explicitly learn a transformation operator, but instead learn transformed versions of shape templates.
As a result, the transformations learned will not easily generalize to new shapes (Tenenbaum and Freeman, 2000; Grimes and Rao, 2005).
Other bilinear models do learn an explicit transformation operator, but because of the difficulty of inferring and learning large transformations, only local transformations are learned (Rao and Ballard, 1998; Olshausen et al., 2007).
Attempts to learn larger transformations led to works on learning Lie group transformations.
Typically, these transformations are learned from pairs of transformed images or a sequence/set of transforming images (Rao and Ruderman, 1999; Miao and Rao, 2007; Culpepper and Olshausen, 2009; Sohl-Dickstein et al., 2010).
Early works learn the generator, or the Lie algebra, directly.
Still, because of the computational intractability of gradient descent with respect to the matrix exponential, first-order Taylor expansion of the matrix exponential is used during learning, limiting the ability to learn from images with large transformations (Rao and Ruderman, 1999; Miao and Rao, 2007).
An important innovation of Sohl-Dickstein et al.
(2010) was to diagonalize the generator, which allows for tractable Lie group learning from large transformations.
This idea was generalized by Cohen and Welling (2014) using representation theory, allowing for the learning of complex, large transformations such as 3D rotation (Cohen and Welling, 2015).
This was an important generalization since describing the set of transformations on images as a representation of a Lie group rather than a Lie group itself, as is done in previous work, allows for the use of representation theory to simplify computations.
For instance, the representation theory of the N -dimensional torus is used in deriving the simple parameterization of the transformation operator in this paper.
We leave many related works that modify deep neural network architectures and objective functions to Appendix B.
6. Discussion
In this work, we study the problem of disentangling factors of variation in images, specifically discrete patterns vs. continuous transformations, which remains an open theoretical problem.
To approach the problem, we combine Lie Group transformation learning and sparse coding within a Bayesian model.
We show how spatial patterns and transformations can be learned as separate generative factors and inferred simultaneously.
We wish to emphasize that the contribution is primarily in theory rather than a specific application.
When the model is trained on synthetic datasets containing known geometric transformations, the digits are learned by the dictionary templates, and the applied transformations are learned by the transformation operator, providing a proof of concept demonstration of the feasibility of combining Lie Group transformation learning and sparse coding.
We wish to emphasize two main points about this work.
Firstly, building on the foundational work of Cohen and Welling (2014), we show that incorporating the appropriate mathematical structure for describing transformations (Lie groups) enables a model to learn to disentangle shape and transformations in a network with a computationally simple structure.
Although in principle a generic multilayer neural network could learn to approximate the inferential computations in this model, we conjecture that it would lead to a more complicated structure (when evaluated in terms of the number of weights and layers) and less robust performance in terms of its ability to generalize outside the training set.
The representation theory of Lie groups leads us to a parameterization of the transformation operator that is both computationally efficient and effective at learning various transformations.
Secondly, our Bayesian framework provides an advantage for learning a joint form and transformation model.
While the usual approach is to jointly optimize the form and transformation parameters using gradient descent, a Bayesian approach reveals that it is better to integrate out the transformation parameters when optimizing the sparse coefficients.
Empirically, we found that this approach achieves better convergence than the joint optimization, allowing the algorithm to reliably learn the correct transformations and shape dictionary.
It is a promising future direction to use Resonator networks (Frady et al., 2020) for simultaneously factorizing α and s .
There are two main limitations to our model.
Our current way of computing the gradient of α is not scalable to a large number of transformation parameters.
It involves the expectation (cid:82) s P θ ( s |
I , α ) R ( s ), where the number of samples of s needed to compute the integral scales exponentially with n , or the dimension of s .
A potential direction to address this issue is to find a simple distribution approximating P θ ( s |
I , α ) that can be used for importance sampling.
Another possibility is using MCMC methods.
The second limitation to our model comes from the constraints on the transformations that can be learned, including orthogonality, compactness, connectedness, and commutativity.
Though we showed in our experiments that some of these theoretical constraints are not quite problematic, there are still important transformations that cannot be learned as a result, such as global variations in contrast or a combination of rotation and translation.
One possibility is to extend our current method and learn representations of a larger class of Lie groups, but it is unclear whether simple parameterizations exist for such groups.
To address this issue, one possible future direction is to learn the representation of an arbitrary Lie group by learning the corresponding Lie algebra.
Finally, the key idea of using a trainable Lie Group transformer module instead of a predefined transformation module like the spatial transformer (Jaderberg et al., 2015) may be highly useful for separating the transformation in deep neural networks.
This is another interesting future direction.
Acknowledgments
Research conducted by Yubei Chen at Redwood Center and BAIR was funded in part by
NSF-IIS-1718991 and NSF-DGE-1106400.
Bruno Olshausen’s contributions were funded in part by NSF-IIS-1718991, NSF-DGE-1106400, and DARPA’s Virtual Intellgence Processing program (SUPER-HD).
Ho Yin Chau’s contributions were funded in part by FAFSA.
We thank Christian Shewmake for providing many helpful feedbacks during the preparation of this paper.
Appendix A. The theory behind the parametrization
Many transformations - including rotations, rigid motion, and translations - can be understood as Lie groups, or more informally, a group of continuous symmetries of a space.
Some examples include the groups of n -dimensional rotations SO ( n ) and rigid motions SE ( n ).
When a Lie group deforms data, such as rotating an image, this constitutes the action of that Lie group G on the vector space of data.
If this action is linear, this amounts to a representation of G - an instantiation of G as a set of linear operators.
More formally, a representation of G on a vector space V is a group homomorphism ρ : G → GL ( V ) which maps each element of G to an invertible linear transformation on V .
In this paper, we consider the representation of compact, connected, commutative (CCC)
Lie groups.
We choose CCC Lie groups because they have straightforward representations.
The Peter-Weyl theorem states that any unitary representation of a compact Lie group can be written as a direct sum of its irreducible representations; informally, a representation decomposes into its atomic parts - the irreducibles - which cannot be further decomposed.
Since all irreducible representations of an n -dimensional torus (hence of CCC Lie groups, as they are equivalent) are of the form ρ ( e i s )
=
e i ω · s for any ω ∈ Z n (Kamnitzer, 2011), where e i s =
[ e is 1 , e is 2 , · · · , e is n ] T is a point on T n , any unitary representation a CCC Lie group is diagonal up to a unitary change of basis:
[Equation]
Since the data lives in the real vector space R D , we restrict ρ to be real by choosing the i ω l in the diagonal matrix to come in conjugate pairs.
After some simplification, this leads to the form W R ( s ) W T (see Appendix C for details).
In fact, any orthogonal representation of T n can be written as W R ( s ) W T (see Theorem 1 in Appendix C for details).
In practice, we reduce the dimensionality of the model and improve efficiency by reducing the number of columns of W to 2 L < D .
A minor downside of restricting our attention to CCC Lie groups is that it imposes several theoretical constraints on the class of learnable transformations: Compactness enforces the transformations as periodic; connectedness precludes discrete transformations like reflections; commutativity excludes non-commutative transformations such as 3D rotations.
Fortunately, while many transformations violate these constraints in theory, we demonstrate in Sectionthat they can still be approximately learned in practice.
Appendix B. More related works from deep learning
A large body of work has appeared in recent years that modifies existing deep neural network architectures such as GAN and VAE to learn disentangled representations in an unsupervised manner (Cheung et al., 2014;
Chen et al., 2016; Higgins et al., 2017; Dupont, 2018; Kim and Mnih, 2018;
Chen et al., 2018).
While these models are more general and potentially more powerful than LSC due to their many convolutional layers, they are also substantially more complex (in terms of the number of layers) than LSC, which requires only one layer for transformation and one layer for sparse coding.
The compactness of
LSC is because we explicitly designed a layer to model Lie group transformations, which is the mathematical structure needed for the problems we are trying to solve.
We speculate that a model with multiple such transformation layers could capture a broader range of image transformations than a generic multilayer convnet.
There are also important works in supervised deep learning that include a specialized module to handle image transformation.
Spatial Transformer Networks by Jaderberg et al.
(2015) uses a differentiable transformer module that models affine transformations.
Capsules use a group of neurons to collectively encode the probability of an object’s presence and the pose/transformation of such an object (Hinton et al., 2011; Sabour et al., 2017; Hinton et al., 2018).
Representation theory is used to develop new neural network layers that are equivariant to input transformations (Cohen and Welling, 2016, 2017; Cohen et al., 2018, 2019).
Appendix C. Orthogonal representations of T n
Theorem 1 Any D -dimensional real, orthogonal representation of T n can be written in the form W R ( s ) W T where:
R ( s ) is a D × D block-diagonal matrix with J
+ 1 blocks for some J ≤ D/2.
The first J blocks of R ( s ) are of the form (cid:20) cos
(cid:0) ω T s (cid:1) − sin (cid:0) ω
T s (cid:1) sin (cid:0) ω
T s (cid:1) cos (cid:0) ω T s (cid:1) (cid:21) where ω ∈ Z n .
We refer to this as a rotation block.
The last block of R ( s ) is a ( D − 2 J ) ×
( D − 2 J ) identity matrix.
W is a D × D orthogonal matrix.
If D is even, then R ( s ) is the block diagonal matrix with subblocks R i ( s ) defined in Equation (1) in the main text, since in that case D − 2 J is even, so the identity matrix block can be written as a direct sum of rotation blocks with ω = 0 .
Proof
Any unitary representation ρ of T n in C D can be expressed in the form ρ ( e i s ) =
V e Σ ( s ) V H (Equation (6) in the main text), where Σ ( s ) is a diagonal matrix with diagonal ( − i ω T 1 s , · · · , − i ω TD s ).
We will show that for every ω j ̸ = 0 there is a ω k such that ω k = − ω j .
First, we assume that ω j ̸ = 0 for all j .
Our first step is to show that there exists some open ball B ⊂ R n such that e −
i ω
Tj s have non-zero imaginary component for all j and for all s ∈ B .
Consider the polynomial f :
[Equation]
where U is a suitably small open subset of R n such
that ω
Tj s ∈ ( − π, π ) for all j .
f ( s ) is a non-zero function since ω j ’s are all non-zero by assumption, which implies there must be some open ball B ⊂ U such that f ( s ) ̸ = 0 for all s ∈ B (if not, then the zero set f − 1 ( { 0 } ) is dense in U .
But by continuity of f , f − 1 ( { 0 } ) is a closed set, so f − 1 ( { 0 } ) =
f − 1 ( { 0 } )
=
U , meaning f is a zero function, contradiction).
This implies that for all j and for all s ∈ B , ω
Tj s ̸ = 0, so e − i ω
Tj s ̸ = 1.
Moreover, since ω Tj s ∈ ( − π, π ), e − i ω
Tj s ̸ = − 1.
Since e − i ω
Tj s lies on the unit circle but e −
i ω
Tj s ̸∈ { 1 , − 1 } , we conclude e − i ω
Tj s must have non-zero imaginary component for all j and for all s ∈ B .
Now we show that for every ω j ̸ = 0 there is a ω k such that ω k = − ω j .
Consider the polynomial g : B ⊂ R n → R :
[Equation]
where we continue to assume that ω j ̸ = 0 for all j .
By Lemma 3, every eigenvalue e − i ω
Tj s of ρ ( e i s ) has a conjugate eigenvalue.
Since e − i ω
Tj s has non-zero imaginary part if s ∈ B , for every j there must be a k ̸ =
j such that e − i ω
Tk s =
e
i ω
Tj s since e − i ω
Tj s cannot be conjugate to itself.
Therefore, for all s ∈ B at least one of the factors is 0, so g ( s ) = 0 for all s ∈ B .
Lemma 4 implies that the g ( s ) is the zero polynomial and hence ω
j ∗ = − ω k ∗ for some j ∗ , k ∗ where j ∗ ̸ = k ∗ .
Since conjugate eigenvalues have the same multiplicity by Lemma 3, we may remove the term ( ω j ∗ + ω k ∗ ) T s from g ( s ) without changing the fact that g ( s ) = 0.
Then, we repeat the above procedure until all ω ’s have been paired.
Relaxing the assumption that ω j ̸ = 0 for all j , we may apply the above argument by restricting our attention to only the set of non-zero ω ’s. We conclude that for every ω j ̸ = 0 there is a k such that ω j = − ω k
Therefore, the non-zero ω ’s come in pairs ( ω , − ω ).
If there are J such pairs, WLOG we assume ω 2 j = − ω
2 j − 1 for 1 ≤ j ≤ J and ω k = 0 for k > 2 J .
Since the columns of
V in ρ ( e i s )
=
V e Σ ( s ) V H are the eigenvectors of ρ ( e i s ) and the eigenvalues e i ω
j come in conjugate pairs, Lemma 3 implies that WLOG we can assume the first 2 J columns of V come in conjugate pairs.
Moreover, we may also assume the last D − 2 J columns also come in conjugate pairs since eigenvectors of real eigenvalues also come in conjugate pairs by
Lemma 3.
Hence, WLOG V 2 j = V
j − 1 for all j .
Next, we show that the representation takes the form W R ( s ) W T .
Construct a block diagonal matrix U such that the first J blocks are 2 × 2 blocks of the form
[Equation]
and the last block is just a ( D − 2 J ) ×
( D − 2 J ) identity matrix.
Using the fact that U is a unitary matrix, we can expand ρ ( e i s ) to get
[Equation]
U ) H
For the first J blocks, we restrict our attention to a single 2 × 2 block of U H Σ ( s ) U .
Using ω 2 j = − ω
2 j − 1 , we simplify:
[Equation]
The exponential of this block is the 2 × 2 rotation matrix:
[Equation]
and hence the first J 2 × 2 blocks of e U H Σ ( s ) U are just 2 × 2 rotation blocks.
On the other hand, the ( D − 2 J ) × ( D − 2 J ) block of U H Σ( s ) U is a zero matrix, so the ( D − 2 J ) ×
( D − 2 J ) block in e U H Σ( s ) U is the identity matrix.
Thus, e U H Σ ( s ) U =
R ( s ).
Next we show that V U is a real orthogonal matrix and hence if we let W
=
V U then ρ ( e i s )
=
W R ( s ) W T as desired.
Since U H is block-diagonal, when computing the matrix product U H V H we can individually consider each block of U H multiplying its corresponding rows in V H .
Recall that the columns of V come in conjugate pairs.
Hence, restricting our attention to one pair of rows in ( V U ) H = U H V H , we get
[Equation]
so all columns of V U are real.
Because V U is a product of unitary matrices, it is also unitary, and V U must be real orthogonal
Thus we have ρ ( e i s )
=
W R ( s ) W T .
Remark
The converse to the theorem is also true, namely that if ρ is a map from
T n to GL ( D, R ) such that ρ ( e i s )
=
W R ( s ) W T , then it is a D -dimensional real orthogonal representation of T n .
This follows from the fact that W R ( s ) W T is orthogonal and
R ( s 1 ) R ( s 2 ) =
R ( s 1 + s 2 ) .
Finally, we note that all orthogonal representations of T n are actually special orthogonal representations of T n .
This is also easy to see, since we know det( ρ (1))
= det( 1 ) = 1 , so that if there exists some e i s ∈ T n such that det (cid:0) ρ ( e i s ) (cid:1) = − 1 , then due to continuity of det and ρ ( ρ is smooth by definition of representations of Lie groups), there must exist some e i s ′ such that det (cid:16) ρ ( e i s ′ )
(cid:17) = 0 , which is impossible.
Lemma
If A is a real matrix and λ is a complex eigenvalue of A with eigenvector v , then λ is also an eigenvalue with eigenvector v .
If λ is an eigenvalue with multiplicity n , then λ is also an eigenvalue with multiplicity n .
Proof Since the eigenvalues of A are the roots of its characteristic polynomial char ( A ) = det( A − λI ), if A is real then char ( A ) is a real polynomial.
Factorizing over C :
[Equation]
Suppose r 1 is a complex root with multiplicity n > 1.
As p (¯ z ) =
p ( z ) for a real polynomial, 0 = char ( A )( r 1 ) =
char ( A )( ¯ r 1 ), so r 1 is also a root and hence an eigenvalue.
Removing the factors ( z − r 1 )( z − r 1 ), we repeat the same argument n times to conclude that r 1 is also an eigenvalue with multiplicity n .
If λ is a complex eigenvalue of A with eigenvector v , then Av = λv .
Taking the conjugate of both sides shows:
[Equation]
Lemma
For any p ( x ) = p ( x 1 , · · · , x n ) ∈ R [ X 1 , · · · , X n ] , if p ( s ) =
for all s in an open set S then p ( x ) is the zero polynomial.
Proof Let t ∈ S .
Then, p ′ ( x ) = p ( x − t )
=
on some open S ′ set containing 0 .
If two smooth functions coincide over some open set U , then their partial derivatives coincide on
U .
Hence, every partial derivative of p ′ ( x ) is 0.
As evaluating the partial derivatives of p ′ ( x ) at 0 will return its coefficients, every coefficient is 0 and p ′ ( x ) is the zero polynomial.
As p ( x ) = p ′ ( x + t ), p ( x ) is also the zero polynomial.
Appendix D. Explicit formulae for ln P ( I | α ) and ln P ( s | I, α )
Although the main results presented in this paper assume a uniform prior on s , a more general prior on s can be used, and we will derive the formula using this more general prior.
This prior is actually the conjugate prior for our likelihood function, which is desirable as it gives a simple functional form for ln P ( I | α ).
Using this more general prior is likely to be useful in problems where the true prior distribution of the transformation variables is known and can be well-approximated by this prior.
Specifically, the general prior on s takes the form:
[Equation]
which is a distribution from the exponential family with natural parameter:
[Equation]
sufficient statistics:
[Equation]
and normalization constant:
[Equation]
Note that one can recover the uniform prior on s by simply taking κ l = 0 for all l .
We also note that this is only a slightly more generalized version of the prior discovered by Cohen and Welling (2014), and that the following derivation of ln P ( I | α ) is also entirely due to
Cohen and Welling (2014), with only slight modifications to adapt to our new model.
According to our model I = W R ( s ) W T Φ α + ϵ
[Equation]
where ϵ ∼ N (0 , σ 2 1 )
[Equation]
we have:
[Equation]
Now
[Equation]
for convenience
[Equation]
define u = W Φ α and v = W I .
Moreover
[Equation]
define
[Equation]
such that
[Equation]
where
[Equation]
Then:
[Equation]
where the integrands in step 4 and 5 are equal because η
[Equation]
Note that the parameter ˆ η determines the posterior distribution of s , which is given by
[Equation]
since P ( s | I , α ) ∝ P
( I | s , α ) P ( s ) ∝ exp (cid:0) ˆ η T T ( s ) (cid:1) (the last step can be seen from the derivation of ln P ( I | α ) where P ( I | s , α ) P ( s ) is the integrand).
In order to compute P ( s | I , α ), we just compute ˆ η and then apply the formula.
Appendix E. Gradients of the model
In this section we provide details of the derivations for Equation (3), Equation (4), and
Equation (5), as well as concrete expressions for the gradients.
Notice that the derivations for Equation (3), Equation (4), and Equation (5) all employ the same ”trick
[Equation]
Equation (3):
[Equation]
Equation (4):
[Equation]
Equation (5):
[Equation]
where in the last step we applied the Equation (4) with ˆ α replaced by α .
Next we provide concrete expressions for the various gradients:
[Equation]
where ϵ
= I − T ( s )
Φ α and ¯ R =
E s ∼ P θ
( s |
I , α )
[ R ( s )].
[Equation]
where ˆ ϵ =
I − T ( s )
Φ ˆ α .
[Equation]
where ˆ
[Equation]
As seen, the gradient for W is quite difficult to compute.
In our implementation, we used an approximation that results in a much simpler expression for the gradient for W derived above, by assuming independence between the term T ( s ) and ˆ ϵ , which are both dependent on the random variable s .
For consistency, we also applied the same approximation to both α and Φ gradients.
More explicitly, we use the following approximate gradients:
[Equation] [Equation]
where ¯ T =
E s ∼ P θ
( s | I , ˆ α )
[ T ( s )] and ˆ¯ ϵ =
E s ∼ P θ
( s | I , ˆ α )
[ˆ ϵ ].
We found by chance that the approximate gradient works better than the exact gradient in practice.
However, we currently do not have a theory for why the approximate gradient works better.
Appendix F. Usage of FISTA in LSC
FISTA is a method for fast gradient descent when the objective function is a sum of a smooth convex function f and a non-smooth convex function g (Beck and Teboulle, 2009).
It is typically applied to problems such as the traditional sparse coding, where the objective is the sum of the smooth convex function ||
I − Φ α || 22 and non-smooth convex sparsity cost ||
α || 1 .
As an extension of sparse coding, we would like to use FISTA in order to speed up convergence for α as well.
The main problem is that our new objective is possibly a non-convex function of α , and as a result the theoretical guarantees of FISTA may not apply.
Fortunately, we find that despite the lack of theoretical guarantees, FISTA still works very well in LSC.
In LSC, we directly applied FISTA with constant step size to perform the optimization problem
[Equation]
where g corresponds to the non-smooth convex function assumed in the FISTA paper.
The only free parameter is the choice of step size, which, accoridng to FISTA, should be set as 1 /L
( f ) if f ( α ) were convex, where L ( f ) is a Lipschitz constant of f .
In our case, we set the step size as 1 . 5 || 1 σ 2 Φ T W W T Φ || , where || · || is the spectral norm of the matrix.
To understand this choice of step size, we begin by noting that
[Equation]
where h ( s , α ) ≡ −∇ α ln P
θ ( I | s , α )
= − 1 σ 2 Φ T T ( s ) T ( I − T ( s ) Φ α ) is the score.
Hence ∇ 2 α f ( α ) =
∇ α
[Equation]
Since Φ T W W T Φ is positive semidefinite and − Cov( h ( s , α )) is negative semidefinite, the sum is not guaranteed to be positive semidefinite.
Hence f ( α ) is not necessarily convex.
However, it does imply that
[Equation]
If f ( α ) were convex, then this would mean that a Lipschitz constant for f would be || 1 σ 2 Φ T W W T Φ || + M , where M is a bound for || Cov( h ( s , α ))
|| .
M is difficult to compute, but in practice we find that setting the step size as 1 . 5 || 1 σ 2 Φ T W W T
Φ || works well, which suggests M is usually small in comparison to the first term.
This is not surprising, especially since during the later stages of training the variance of s is usually very small, and hence the covariance of h ( s , α ) is also expected to have a small spectral norm.
Appendix G. Training datasets
Example images from the 2D translation and the rotation + scaling datasets are shown in
Figure 4.
Appendix H. More examples of image reconstruction and latent traversals
In Figure 5 and Figure 6 we provide more examples of the image reconstructions and latent traversals shown in Figure 2 and Figure 3 respectively.
Appendix I. Training details
For training on the 2D translation and the rotation + scaling dataset, the hyperparameters used for the model is detailed in Table 2.
[Figure]
[Figure] [Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
default
When trained on MNIST, the only change is that the multiplicity of ω is 2 instead of 1.
For computing the SNRs in Table 1, we used N = 100 instead of N = 50 to obtain higher quality reconstruction.
Moreover, a warmup period is applied to the learning rates of both α and W , where the learning rates are linearly increased from 0 to the specified values in Tableover 3 epochs.
This greatly reduces the trial-to-trial fluctuation of image reconstruction quality, and its usage is highly recommended as there are no observed drawbacks.
The SNRs for sparse coding are obtained using an improved version of the algorithm in Olshausen and
Field (1997), where the main modification is the use of FISTA in optimizing α .
To ensure fair comparison between LSC and sparse coding, the following hyperparameters for both models are set to be equal: B, K, σ 2 , and λ .
Notice that when these hyperparameters are set equal, the loss function for LSC coincides with the loss function for sparse coding in the limiting case where all the ω l are 0 (in which case the transformation T ( s ) will just be the identity matrix) and W is full rank.
The learning rate for Φ in sparse coding algorithm is manually optimized for best possible SNRs, and we find η Φ = 0 .
05 works best.
All training is done on a Nvidia GTX 1080Ti GPU.