L EARNING I DENTITY -P RESERVING T RANSFORMATIONS ON D ATA M ANIFOLDS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many machine learning techniques incorporate identity-preserving transformations into their models to generalize their performance to previously unseen data.
These transformations are typically selected from a set of functions that are known to maintain the identity of an input when applied (e.g., rotation, translation, ﬂipping, and scaling).
However, there are many natural variations that cannot be labeled for supervision or deﬁned through examination of the data.
As suggested by the manifold hypothesis, many of these natural variations live on or near a lowdimensional, nonlinear manifold.
Several techniques represent manifold variations through a set of learned Lie group operators that deﬁne directions of motion on the manifold.
However theses approaches are limited because they require transformation labels when training their models and they lack a method for determining which regions of the manifold are appropriate for applying each speciﬁc operator.
We address these limitations by introducing a learning strategy that does not require transformation labels and developing a method that learns the local regions where each operator is likely to be used while preserving the identity of inputs.
Experiments on MNIST and Fashion MNIST highlight our model’s ability to learn identity-preserving transformations on multi-class datasets.
Additionally, we train on CelebA to showcase our model’s ability to learn semantically meaningful transformations on complex datasets in an unsupervised manner.
I NTRODUCTION
A goal of many machine learning models is to accurately identify objects as they undergo natural transformations – a task that humans are adept at.
According to the manifold hypothesis, natural variations in high-dimensional data lie on or near a low-dimensional, nonlinear manifold (Fefferman et al., 2016).
Additionally, the manifolds representing different classes are separated by low density regions (Rifai et al., 2011a).
Natural physical laws govern the possible transformations that objects can undergo and many of the identity-preserving transformations (e.g., changes in viewpoint, color, and lighting) are shared among classes of data.
Sharing of transformations between classes enables increased efﬁciency in deﬁning data variations – a model can represent a limited set of transformations that can describe a majority of variations in many classes.
Several machine learning models incorporate speciﬁc identity-preserving transformations that are shared among a large number of classes to generalize the performance of their model to unseen data.
These include equivariant models that incorporate transformations like translation and rotation into intermediate network layers (Cohen & Welling, 2016;
Cohen et al., 2018) and data augmentation techniques that apply known identity-preserving variations to data while training (Cubuk et al., 2019; Ho et al., 2019; Lim et al., 2019; Sohn et al., 2020; He et al., 2020; Chen et al., 2020).
However, many datasets have natural transformations shared among classes that are not easily prespeciﬁed from intuition, making it critical that we develop a model that can learn both 1) a representation for these transformations without explicit transformation labels and 2) the context for which locations in the data space each transformation is likely to be relevant.
Manifold learning strategies estimate the low-dimensional manifold structure of data.
A subset of these techniques learn to transform points on the manifold through nonlinear Lie group operators (Rao & Ruderman, 1999; Miao & Rao, 2007; Culpepper & Olshausen, 2009;
Sohl-Dickstein et al., 2010; Cohen & Welling, 2014; Hauberg et al., 2016; Connor & Rozell, 2020; Connor et al., 2021).
Lie group operators represent inﬁnitesimal transformations which can be applied to data through an exponential mapping to transform points along a manifold, and a manifold can be globally deﬁned by a set of operators that each move in different directions along it (Hoffman, 1966;
Dodwell, 1983).
A Lie group operator model is well-suited for representing natural data variations because the operators can be learned from the data, applied to data points to transform them beyond their local neighborhoods, and used to estimate geodesic paths.
While the Lie group operator models have many beneﬁts, previous approaches demonstrate the two shortcomings noted above.
First, to learn Lie group operators that represent a data manifold, pairs of training points are selected which lie within a neighborhood of one another.
The training objective encourages efﬁcient paths between these nearby points and the choice of training point pairs inﬂuences the types of manifold transformations that are learned.
Recent papers incorporating Lie group operators into machine learning models have either used predeﬁned operators that represent known transformations groups (e.g., the 3D rotational group SO (3) (Falorsi et al., 2019)), required transformation labels for selecting point pairs when training (Connor & Rozell, 2020), or randomly selected pairs of points from the same class (Connor et al., 2021).
To learn an effective model with datasets having no labeled transformation structure, we require a point pair selection strategy that identiﬁes points that are related through the transformations the model aims to learn.
Second, existing Lie group operator models have lacked a method for determining which regions of the manifold are appropriate for each operator, meaning that every operator is equally likely to be used at every point on the manifold.
This is a ﬂawed assumption because, while many transformations are shared between classes, there are also data variations that are unique to speciﬁc data classes.
Additionally, in a dataset with several manifolds (each representing a different class), there is a limited extent to which a transformation can be applied without moving a point onto another manifold.
The main contributions of this paper are the development of methods to address the two critical shortcomings of Lie group operator-based manifold models noted above.
Speciﬁcally, motivated by ﬁnding perceptually similar training samples without transformation labels, we ﬁrst introduce a point pair selection strategy to learn a manifold representation of natural variations shared across multiple data classes without requiring transformations labels.
Second, we develop a method that uses a pretrained classiﬁer (measuring identity-preservation of transformed samples) to learn the local regions where each operator is likely to be used while preserving the identity of transformed samples.
This approach enables us to analyze the local structure of the data manifold in the context of the learned operators and to describe the invariances of the classiﬁer.
We demonstrate the efﬁcacy of these strategies in the context of the Manifold Autoencoder (MAE) model (Connor & Rozell, 2020) to learn semantically meaningful transformations on MNIST (LeCun et al., 1998), Fashion
MNIST (Xiao et al., 2017), and CelebA (Liu et al., 2015).
B ACKGROUND
Manifold Learning Manifold learning models estimate the low-dimensional structure of highdimensional data by utilizing the property that local neighborhoods on the manifold are approximately linear.
Traditional techniques represent the manifold through a low-dimensional embedding of the data points (Tenenbaum et al., 2000; Roweis & Saul, 2000; Belkin & Niyogi, 2003; Maaten & Hinton, 2008) or through estimates of linear tangent planes that represent local directions of manifold motion (Doll´ar et al., 2007; Bengio & Monperrus, 2005; Park et al., 2015).
While these traditional manifold learning approaches are useful for understanding low-dimensional data structure, in many cases the input data space is an inefﬁcient representation of the data.
For example, data in the pixel space suffers from the curse of dimensionality and cannot be smoothly interpolated while maintaining identity (Bengio et al., 2005).
Many approaches have used neural networks to learn a low-dimensional latent space in which manifold models can be incorporated.
The contractive autoencoder (CAE) estimates manifold tangents by minimizing the the Jacobian of the encoder network, encouraging invariance of latent vectors to image space perturbations (Rifai et al., 2011c;b;a;
Kumar et al., 2017).
Several methods estimate geodesic paths in the latent space of a trained variational autoencoder (VAE) model (Arvanitidis et al., 2018; Chen et al., 2018; Shao et al., 2018;
Arvanitidis et al., 2019) and use this approach to learn VAEs with priors that are estimated using the
Riemannian metrics computed in the latent space (Arvanitidis et al., 2021; Kalatzis et al., 2020).
Lie Group Operators A Lie group is a group of continuous transformations which also deﬁnes a manifold by representing inﬁnitesimal transformations that can be applied to input data (Hoffman, Dodwell, 1983).
Several methods incorporate Lie groups into neural networks to represent data transformations that are identity-preserving within the model (Cohen & Welling, 2014; Cohen et al., 2018; Cosentino et al., 2021).
A prevalent strategy is to learn a dictionary of Lie group operators that are mapped to a speciﬁc group element through the matrix exponential expm( · ) (Rao & Ruderman, 1999; Ham & Lee, 2006; Miao & Rao, 2007; Culpepper & Olshausen, 2009; SohlDickstein et al., 2010; Cohen & Welling, 2014; Connor & Rozell, 2020; Connor et al., 2021).
In these models, each operator Ψ m , called a transport operator , describes a single direction along the manifold and is parameterized by a single coefﬁcient
c m .
Given an initial data point z , the transport operators deﬁne a generative model where transformations can be derived from sampling sparse coefﬁcients c m ∼ Laplace (0 , ζ ) :
[Equation]
where (cid:15) ∼ N (0 , σ 2 (cid:15) I ) .
The manifold autoencoder (MAE) incorporates the transport operator model into the latent space of an autoencoder to learn a dictionary of operators that represent the global, nonlinear manifold structure in the latent space (Connor & Rozell, 2020).
This model has been shown to effectively learn reusable operators with transformation supervision, and it will provide the context for demonstrating the effectiveness of the methods developed in this paper.
Disentanglement
One class of methods that also aim to identify factors of variation in data is disentanglement methods which learn features that each vary one independent characteristic of the data (Higgins et al., 2018).
In a supervised learning scenario, there are many disentangling techniques that separate data style from content by encouraging similarity between the content features in samples from the same class (Tenenbaum & Freeman, 2000; Reed et al., 2014; Mathieu et al., 2016; Bouchacourt et al., 2018) or encouraging features to map to known class (Cheung et al., 2015) or transformation labels (Hinton et al., 2011; Kulkarni et al., 2015; Yang et al., 2015). There are also techniques that can disentangle latent representations without labels like InfoGAN (Chen et al., 2016) and β -VAE (Higgins et al., 2017). The goal of our work is distinct from disentanglement work because, rather than identifying independently varying factors of variation, we aim to learn non-linear operators that correspond to transformations on the data manifold (which may not vary independently).
Our approach is advantageous because it can faithfully represent longer transformation paths, learn variations while maintaining image reconstruction, and change the number of learned variations by increasing or decreasing the number of learned operators.
The MAE learns a low-dimensional latent representation of the data by deﬁning an encoder function f : X → Z that maps high-dimensional data points x ∈ R D to low-dimensional latent vectors z ∈ R d and a decoder function g : Z → X that maps the latent vectors back into the data space (Connor & Rozell, 2020).
Transport operators Ψ are incorporated into the latent space to learn manifoldbased transformations.
Before learning the transport operators, the autoencoder is pretrained to extract a latent representation of the data using the traditional autoencoder reconstruction objective.
After pretraining, the autoencoder weights are ﬁxed and the operators are trained with the following objective, which encourages the learning of transport operators that generate efﬁcient paths between the latent vectors z and z 1 (coinciding with f ( x 0 ) and f ( x 1 ) ) that are nearby on the manifold:
[Equation]
where γ, ζ > 0 .
Objective (2) is minimized via an alternating minimization scheme.
Speciﬁcally, at each training iteration, points pairs x 0 and x 1 are selected in the input space and encoded into the latent space z 0 and z 1 .
Then, coefﬁcients c are inferred between the encoded latent vectors (by minimizing (2) with respect to c ) to estimate the best path between z 0 and z 1 .
After inference, the coefﬁcients are ﬁxed and a gradient step is taken to minimize (2) with respect to transport operator weights.
Once learned, these operators represent different types of motion that traverse the manifold and they can be combined to generate natural paths on the manifold.
After ﬁtting transport operators to the latent space of a ﬁxed autoencoder network, there is a ﬁnal ﬁne-tuning training phase that updates the network weights and transport operators simultaneously using a joint objective that combines the autoencoder reconstruction loss with L Ψ .
This ﬁne-tuning step addresses the potential of a mismatch between the data manifold and the learned latent structure by adapting the latent structure to ﬁt the transport operators learned between the selected training point pairs.
We empirically ﬁnd that breaking up training into these three phases increases the stability with which the transport operators can be learned. Given the context of this MAE model, in the following sections we describe our contributions.
3. U NSUPERVISED T RANSFORMATION L EARNING
There are many possible strategies for selecting training point pairs for a Lie group operator model and the choice of a strategy can dictate the types of learned transformations.
Point pairs may be selected as random samples from the dataset or from within the same class.
These points, however, are likely to be outside of local manifold neighborhoods and may not provide representations of the natural, perceptually smooth variations.
Point pairs can also be selected as nearest neighbors in the pixel or autoencoder latent space (denoted with h ( x ) =
x and h ( x ) =
f ( x ) in (3), respectively):
[Equation]
However, semantic transformations of interest may not be exhibited through pixel similarity nor through unsupervised autoencoder feature similarity.
In some applications there is additional information that can aid in point pair selection, such as rotation angle, semantic label, or time index in a temporal sequence.
However, for most complex datasets, information about the transformations of interest is not available.
This necessitates a strategy for selecting training point pairs which can be used to learn natural transformations without requiring additional transformation supervision.
We generalize transport operator training to incorporate an unsupervised learning strategy that can be applied to a wide array of datasets.
In particular, we assume access to a classiﬁer pretrained on a large-scale dataset (likely taken from a model zoo) with similar statistics as our training dataset of interest (Nguyen et al., 2020).
In this approach, we select point pairs by setting h ( x ) from (3) equal to the penultimate layer of the classiﬁer, similar to the previously proposed perceptual loss (Johnson et al., 2016).
It has been shown that high-level features in such classiﬁer networks correspond to semantic characteristics of the data and ﬁnding points nearby in this feature space can result in image pairs that are perceptually similar without being exactly the same (Johnson et al., 2016;
Yosinski et al., 2015).
Incorporating the classiﬁer features as a supervision strategy for point pair selection enables our model to learn transport operators without requiring transformation labels or class labels.
We show that this metric leads to learned operators that correspond to semantically identiﬁable transformations.
Discussion of different point pair strategies can be found in Appendix E.
3. L EARNING L OCAL T RANSPORT O PERATOR S TATISTICS TO E NCOURAGE I DENTITY P RESERVATION
A trained transport operator model provides a dictionary of operators that can be applied globally across the entire data space.
This model has ﬂexibility to deﬁne local manifold characteristics through the coefﬁcients c which specify the combination of operators to apply to a given point.
The standard prior on the coefﬁcients is a factorial Laplace distribution parameterized by a single scale parameter ζ , which encourages transformations to be deﬁned by a sparse set of operators: [Equation]
[Equation]
By setting a ﬁxed prior on coefﬁcients across the entire manifold, there is an implicit assumption that the manifold structure is the same over the whole dataset and every operator is equally likely to be applied at every point.
However, this is a ﬂawed assumption because not all transformations in one class of data are present in all other classes and within classes there may be regions with different manifold structure.
Failing to capture the local statistics of transport operator usage could result in transformed points that depart from the data manifold and change their identity.
To address this limitation, we introduce a coefﬁcient encoder network which maps latent vectors to scale parameters that specify the Laplace distribution on the coefﬁcients at those latent points.
The goal of this network is to estimate local coefﬁcient distributions that maximize the identitypreservation of points (as measured by a pretrained classiﬁer) transformed with operators characterized by sampled coefﬁcients as in (1).
[-where T Ψ ( c ) = expm (cid:16)(cid:80) Mm =
 Ψ m c m (cid:17) is the transformation matrix corresponding to the transport operators (parameterized by coefﬁcients c ).-] Given labeled observations ( z , y ) and a pretrained classiﬁer on the latent space r ( · ) , we aim to learn a network q φ ( c | z ) that outputs parameters that can be used to produce augmented samples (cid:98) z without changing the classiﬁer output r ( (cid:98) z ) =
y .
To maximize the likelihood of augmented input data maintaining the same classiﬁer output, we adapt the concept of consistency regularization from semi-supervised learning (Bachman et al., 2014; Sohn et al., 2020).
In our context, we have a pretrained classiﬁer and a dictionary of transport operators and we aim to ﬁnd a distribution on the coefﬁcients at individual input points that result in consistent classiﬁcation outputs when T Ψ ( c ) is applied.
The speciﬁc objective can be chosen from a variety of loss functions that encourage similarity in classiﬁer probability outputs.
We speciﬁcally minimize the KL-divegence between the classiﬁer output r ( z ) and the estimated probability of the transformed output r ( (cid:98) z ) .
Unfortunately, the consistency regularization objective can be trivially minimized by setting c = 0 , resulting in an identity transformation and the same classiﬁer output.
Therefore, we want to encourage the largest coefﬁcient values possible while maintaining the identity of our initial data point.
This motivates the addition of a KL-divergence regularizer that encourages the distribution q φ ( c | z ) to be similar to a speciﬁed Laplace prior with a ﬁxed scale ζ like in (4).
Our ﬁnal objective for training the coefﬁcient encoder network is: (a more detailed derivation can be found in Appendix A)
[Equation]
The coefﬁcient encoder introduces a principled way to build identity-preservation directly into our model and to identify local manifold characteristics throughout the latent space.
Additionally, it can quantify the extent with which operators can be applied in different parts of the latent space without changing class prediction, indicating transformations to which the classiﬁer is invariant.
Likewise, points or region that have small encoded coefﬁcient scale weights indicate closeness to class boundaries and can undergo only small transformations without changing identity.
3. D ECREASING C OMPUTATIONAL C OMPLEXITY OF C OEFFICIENT I NFERENCE
To compute the gradient for the matrix exponential during training and inference, previous works have used a linear approximation (Rao & Ruderman, 1999), learned the operators in a factored form (Sohl-Dickstein et al., 2010), or used analytic gradients that are not favorable for parallel computations (Culpepper & Olshausen, 2009; Connor & Rozell, 2020; Connor et al., 2021).
The main computational bottleneck for learning transport operators is the coefﬁcient inference step, where (2) is minimized with respect to c .
The (cid:96) 1 norm term is non-smooth, leading to slower convergence when using subgradients found via automatic differentiation (Boyd & Vandenberghe, 2004).
In this work we employ a forward-backward splitting scheme (Beck & Teboulle, 2009) that alternates between automatic differentiation of the matrix exponential term, and proximal gradients for the (cid:96)norm (Parikh & Boyd, 2014).
This signiﬁcantly speeds up the training process and allows for scaling up the learning of transport operators to complex datasets.
To the best knowledge of the authors, this is the ﬁrst work to demonstrate learning of Lie group operators on datasets as large and complex as CelebA. Additional details on the inference algorithm are provided in Appendix B.
A NALYSIS
For our experiments, we examine the ability of our proposed approaches to enable the MAE model to learn natural transformations of datasets where the underlying identity-preserving transformations are not easily identiﬁable.
We also highlight the beneﬁts of incorporating the coefﬁcient encoder network that captures the transport operator local usage statistics by encoding coefﬁcient scale values that best maintain the identity of latent vectors.
We work with three datasets: MNIST (LeCun et al., 1998), Fashion MNIST (Xiao et al., 2017), and CelebA (Liu et al., 2015).
We select MNIST and Fashion MNIST because they contain several classes that share natural transformations but they do not have transformation labels.
We select
CelebA to highlight our ability learn natural transformations in a larger, more complex dataset.
As a classic dataset used in papers that aim to disentangle dataset features (Higgins et al., 2017; Chen et al., 2016; Hu et al., 2018; Lin et al., 2019), CelebA contains semantically meaningful natural transformations that may be amenable to qualitative labeling.
In all experiments, we followed the general training procedure put forth previously (Connor &
Rozell, 2020) by separating the network training into three phases: the autoencoder training phase, the transport operator training phase, and the ﬁne-tuning phase.
The ﬁne-tuning phase adapts the autoencoder latent space to ﬁt the learned transformations in the event that that pretrained latent space does not match the desired data manifold.
We select training point pairs that are nearest neighbors in the feature space of the ﬁnal, pre-logit layer of a ResNet-
(He et al., 2015) classiﬁer pretrained on
ImageNet (Russakovsky et al., 2015).
After completely training the MAE, we ﬁx the autoencoder network weights and transport operator weights and train the coefﬁcient encoder network with the objective derived in Section 3.2.
Additional details on the datasets, network architectures, and training procedure are available in the Appendix.
We compare against the contractive autoencoder (CAE) (Rifai et al., 2011c) and β -VAE (Higgins et al., 2017), two other methods for incorporating data structure into the latent space..
The CAE represents another technique that learns a manifold representation in a neural network latent space.
In their model, the manifold is represented by estimated tangent planes at latent point locations.
The β -VAE learns to disentangle factors of variation along latent dimensions through an increase in the weighted penalty on the KL-divergence term in the VAE objective.
We choose these methods because they also learn natural dataset variations in the latent space without transformation labels.
4. L EARNING N ATURAL D ATA V ARIATIONS
First, we show how well the perceptual point pair selection strategy described in Section 3.1 enables the MAE model to learn natural data variations in datasets and we highlight the usefulness of a nonlinear manifold model for generating latent space paths.
Fig.
1 shows the paths generated by transport operators trained in this model.
The image in the middle column in each block of images is the reconstructed version of the input image x 0 .
The images to the left and right of the middle show the reconstructed images generated by an individual learned operator applied to the encoded latent vector z 0 : z c = expm( Ψ m c ) z 0 , c = − N c , ..., N c .
We see an individual operator generates a similar transformation across multiple inputs, and in many cases, the transformations induced by the transport operators are semantically meaningful.
We also show that the perceptual point pair selection strategy is effective over a range of datasets.
While many of the operators can be assigned a semantic label through qualitative visual inspection, the CelebA dataset has attribute labels for the images which enable a quantitative analysis of the connection between learned transport operators and dataset attributes.
To classify attributes, we ﬁne-tune a ResNet-18 pretrained on ImageNet with 16 classiﬁcation heads for attributes including smile, beard, hair color, and pale skin (Mao et al., 2020).
With this classiﬁer, we are able to identify transport operators that correspond to speciﬁc dataset attributes.
Fig.
2 shows the classiﬁcation outputs of the attribute classiﬁer for example transport operators.
Our model learns operators that vary hair color and skin paleness as well as several others which are shown in the Appendix.
In Fig.
1 we compare transport operator paths to those generated by the CAE and the β -VAE.
The
CAE-generated paths are the directions of motion on the tangent planes estimated at individual points.
The β -VAE paths are generated by varying the value of one latent dimension while the others remain ﬁxed.
While our method and the β -VAE learn several qualitatively similar transformations, our method is capable of doing so without signiﬁcantly sacriﬁcing reconstruction performance.
In the Appendix, we include examples of the transformations generated by each learned operator.
A primary motivation for learning the data manifold is to generate new views of data that cannot arise from pre-deﬁned functions.
To highlight the ability of our model to generate new views that preserve identity, we compare both interpolated and extrapolated paths from the MAE to those from the CAE, β -VAE, and a traditional auto-encoder in Fig.
Interpolated paths offer insight in the smooth variations between two data points, while extrapolated paths show the capacity to generate
[Figure]
[Figure]
new views of data.
For MAE, the paths are estimated by inferring coefﬁcients c ∗ between two latent points z 0 and z 1 and then generating the path:
z t = expm (cid:16)(cid:80)
Mm =
Ψ m c ∗ m t (cid:17) z 0 .
When the path multiplier t is between 0 and 1 that indicates interpolation and path multipliers beyond 1 indicate extrapolation.
For all other methods, Eucilidean distance between features is used for both interpolation and extrapolation.
In these ﬁgures, the ﬁrst block of images corresponds to the interpolated path with the selected ﬁnal point x 1 surrounded in an orange box.
The second block of images corresponds to the extrapolated paths.
To quantify the identity preservation of each transformation, we input each generated image into a pretrained classiﬁer and plot the prediction of the correct class.
All four methods perform interpolation effectively but our trained model estimates the extrapolated paths more accurately.
Fig.
4 shows how that accuracy varies during extrapolation sequences for 4000 samples.
The MAE is better at generating extrapolated outputs that maintain class identity.
The lower classiﬁcation accuracy in fashion MNIST is due to both a more challenging dataset and the lower resolution of autoencoder image outputs when compared to input images.
[Figure]
[Figure]
4. L EARNING L OCAL M ANIFOLD S TRUCTURE
After the MAE is trained we have a dictionary of transport operators that describe manifold transformations and a network with a latent space that is adapted to the manifold.
We then train the coefﬁcient encoder network to estimate the coefﬁcient scale weights as a function of points in the latent space.
To visualize how the use of the transport operators varies over the latent space we generate an Isomap embedding (Tenenbaum et al., 2000) of latent vectors and color each point by the encoded scale parameter for coefﬁcients associated with each of the transport operators.
Fig.
5a shows these embeddings for MNIST data.
Each operator has regions of the latent space where their use is concentrated.
By training the coefﬁcient encoder to maximize the similarity between the classiﬁcation outputs for an input sample and for a transformed version of that sample, the network aids in identifying which transport operators can be applied to inputs in regions of the data space without changing the identity of the input.
This helps signiﬁcantly with data augmentation where the goal is to create new samples with in-class variations.
To highlight this beneﬁt, in Fig.
6 we show samples augmented by applying transport operators with randomly sampled coefﬁcients to an input latent vector.
In each block of images, the leftmost image (in a green box) is the input image and the images to the right are decoded augmentations.
The top row shows samples augmented with transport operators controlled by coefﬁcients sampled from Laplace distributions with encoded coefﬁcient scale weights.
The bottom row shows samples augmented with transport operators controlled by coefﬁcients sampled from Laplace distributions with a ﬁxed scale parameter.
While both strategies generate some realistic variations of the data, using the encoded scale weights improves identity-preservation of the transformed output.
The augmentations with the encoded scale weights are better at maintaining the identity of the sampled points.
[Figure]
Because the coefﬁcient encoder uses a pretrained classiﬁer to determine the coefﬁcient distributions in the latent space, the resulting encoded scale weights can inform about the types of manifold transformations the classiﬁer is invariant to.
Fig.
5b shows the average coefﬁcient scale weights for each class (rows) and each transport operator (columns) for MNIST.
From this we can identify classes for which the classiﬁer is both sensitive and robust to transformations, represented by small and large scale weights respectively.
We can also examine which classes share the use of the same transformations.
The images to the right in Fig.
5b show transport operators being applied to samples with high encoded scale weights (in a yellow box) and samples with low encoded scale weights (in a blue box).
We can examine the characteristics of transformations that are better suited to some classes than others.
For instance, operator 3 in Fig.
5b increases the curve at the bottom of digits.
This is a natural transformation for classes 3, 5, and 6 which all have higher coefﬁcient scale weights for this operator, but when this is applied to a 1, that makes it look like an 8.
C ONCLUSION
In this work we develop methods to improve the effectiveness and utility of Lie group operator models for learning manifold representations of datasets with complex transformations that cannot be labeled.
We do this by introducing a perceptual point pair selection strategy for training operators and by developing a method that uses a pretrained classiﬁer to learn local regions where operators are likely to be used while preserving the identity of transformed samples.
We demonstrate the efﬁcacy of our approach in learning natural dataset variations with the MAE.
While this is a powerful model, users should be mindful of the biases that are introduced through training with speciﬁc supervision methods when drawing conclusions about learned data transformations.
This work presents a promising technique for learning representations of natural data variations that can improve model robustness.
In future work we can address some limitations by expanding this work to consider methods for localizing manifold structure with limited or no class labels and improving the reconstruction ﬁdelity by applying the model within more complex generative model embeddings.
S UPPLEMENTARY M ATERIAL
[Equation]
Our motivation in training an encoder to learn the coefﬁcient statistics is to understand the regions of the manifold in which speciﬁc transport operators are applicable and the invariances of a pretrained classiﬁer to the learned transport operator-induced transformations.
In this section, we will outline a derivation of our coefﬁcient encoder training objective from a variational inference perspective by learning a variational posterior for the coefﬁcients using a deep neural network (Kingma & Welling, 2013; Rezende et al., 2014).
Given observations ( z i , y i ) for i = 1 , . . .
, N , and a pretrained classiﬁer r ( · ) , we aim to learn a distribution q φ ( c | z ) from which we can sample to deﬁne the transformation T Ψ ( c ) that can be applied to z while maintaining class identity.
In other words, we want the augmented (cid:98) z
= T Ψ ( c ) z
+ (cid:15) to result in r ( (cid:98) z ) =
y .
To encourage identity-preservation of vectors transformed with transport operators that are controlled by sampled coefﬁcients, we maximize the likelihood of a class output for an observation under the system model diagrammed in Figure 7.
This system connects the observed latent vector z to an output y through a transformation deﬁned by c .
We follow the derivation used in Rezende &
Mohamed (Rezende & Mohamed, 2016) which introduces the variational posterior to estimate the log likelihood of the data.
Consider a parameterized distribution for the conditional likelihood of our observations:
[Equation]
In equation 6 and equation 7, we marginalize over the coefﬁcients c as required under our system model in Figure 7.
In equation 9, we lower bound the conditional likelihood with a variational lower bound derived from Jensen’s inequality.
Finally, we use the reparameterization trick to deﬁne (cid:98) z as a deterministic function of c , z , and the parameter-free random variable (cid:15) in equation 11.
To sample from q φ ( c | z ) when computing the expectations, we ﬁrst use the reparameterization trick (Connor et al., 2021; Kingma & Welling, 2013; Rezende et al., 2014) to deﬁne the sampled coefﬁcients (cid:98) c as a function of a uniform random variable u ∼ Unif (cid:0) − 1 2 , 1 2 (cid:1) M :
[Equation]
(12) where l φ is a deﬁned mapping from a uniform distribution to a Laplace distribution and the Laplace scale parameters are deﬁned by the output of the coefﬁcient encoder
h φ that we aim to learn.
Using the reparameterization of (cid:98) c and (cid:98) z , we can deﬁne the expectation:
[Equation]
Our pre-trained classiﬁer deﬁnes p θ which allows us to specify ﬁnal objective using of r ( · ) .
Using a
KL-divergence as the likelihood function between the ground truth labels
y i and the classiﬁer output for the augmented inputs
r ( (cid:98) z ( j ) i ) and simplifying the model by setting (cid:15) to 0 , we get:
[Equation]
where in practice we use a single sample J =
and we estimate D KL (cid:16)
y i | r ( (cid:98) z
( j ) i ) (cid:17)
≈
D KL (cid:16) r
( z ( j ) i ) | r ( (cid:98)
z ( j ) i ) (cid:17) for test samples for which we may not have class labels.
These practical simpliﬁcations result in the objective in 5 which we want to minimize.
The KL divergence between the q φ ( c | z ) and p ζ ( c ) has a closed form expression (Gil, 2011):
[Equation]
B C OEFFICIENT I NFERENCE D ETAILS
To perform coefﬁcient inference, previous Lie group operator methods (Connor & Rozell, 2020;
Connor et al., 2021) use the analytic gradient proposed in (Culpepper & Olshausen, 2009) with a conjugate gradient descent solver.
This method requires an eigenvalue decomposition for computing the gradient which is not favorable for parallel computations.
Alternatively, we use the PyTorch implementation of the matrix exponential that allows for automatic differentiation.
Furthermore, to handle the non-smooth (cid:96) 1 norm in objective (2), we apply a proximal gradient step in a forwardbackward splitting scheme (Beck & Teboulle, 2009).
For the (cid:96) 1 norm, the proximal gradient has a known, closed-form solution in the soft-thresholding function (Parikh & Boyd, 2014):
[Equation]
Let ∇ c 12 (cid:13)(cid:13)(cid:13) z 1 − expm (cid:16)(cid:80)
Mm =
Ψ m c m (cid:17) z 0 (cid:13)(cid:13)(cid:13) 2 2 ≈ ∇ (cid:101) f
( c ) be a numerical approximation to the gradient of the (cid:96) 2 term, found through automatic differentiation.
Given initial coefﬁcient values c 0 drawn from an isotropic Gaussian with variance 4 × 10 − 4 , our gradient descent step is: [Equation]
[Equation]
These steps are iterated upon until either a max iteration count is reached, or the change in coefﬁcients, (cid:107)
c k +1 − c k (cid:107) 2 , falls below some threshold.
For all experiments we use a max iteration count of 800 and a tolerance of 1 × 10 − 5 .
For our step-size, we use α k = (0 . 985) k
α 0 with α 0 = 1 × 10 − 2 .
We experimented with different acceleration methods (Kingma & Ba, 2015; Beck & Teboulle, 2009), and found that in many cases they resulted in worse performance.
The main parameter to select for inference is the sparsity-inducing parameter on the coefﬁcient prior, ζ .
This value is sensitive to the dataset, latent dimension, and γ hyper-parameter value.
In practice, we ﬁx a γ value and pick ζ to be as high as possible, inducing sparsity in the inferred coefﬁcients, while maintaining good reconstruction performance of (cid:98) z 1 .
If ζ is too large, that will result in all coefﬁcients going to zero, preventing proper reconstructive performance.
On the other hand, setting ζ to be too small will result in all operators being used to represent the transformation between each point pair, preventing any meaningful structure from being learned during training.
Parameters used in speciﬁc training runs are included in their respective Appendix sections.
When compared against the coefﬁcient inference implementation from (Connor & Rozell, 2020), we perform inference for a batch of 100 samples in an average of 1.67 seconds over 100 trials whereas the inference from (Connor & Rozell, 2020) took an average of 101.01 seconds.
We also compare the speed-up over a baseline that strictly uses automatic differentiation (subgradients) and compare the beneﬁts from moving to GPU hardware in Figure 8.
We use the proximal gradient (GPU) method from this ﬁgure.
Experiments were run on a machine with an Intel i7-6700 CPU with 4.00 GHz and a Nvidia TITAN RTX.
C T RAINING S TRATEGY
In all experiments, we follow the general training procedure put forth previously (Connor & Rozell, 2020).
We train the MAE with three training phases: the autoencoder training phase, the transport operator training phase, and the ﬁne-tuning phase.
During the autoencoder training phase, the network weights are updated using a reconstruction loss objective: E AE = (cid:107) x − ˆ x (cid:107) 22 .
During the transport operator training phase, the network weights are ﬁxed and the transport operators are trained between pairs of points using the objective (2).
Pairs of images x 0 , x 1 are chosen using the perceptual point pair selection strategy described in Section 3.1.
The images are then encoded into the latent space z 0 , z 1 .
For each batch, the ﬁrst step is to infer the coefﬁcients between all pairs of latent vectors.
Coefﬁcient inference is best performed when the entries of the latent vectors are close to the range [ − 1 , 1] .
Because of this, we deﬁne a scale factor that can be applied to encoded latent vectors to reduce the magnitude of their entries prior to performing coefﬁcient inference.
In practice, we inspect the latent vector magnitudes after the autoencoder training phase and choose a scale that will adjust the magnitudes of the latent vector entries to be in the range [ − 1 , 1] .
This does not have to be a precise range for the latent vector magnitudes but instead is a practical guideline.
Coefﬁcient inference is performed on the scaled latent vectors as described in Sections 3.3 and
Appendix B.
After the coefﬁcients are inferred for a batch, the weights on the dictionary elements are updated.
This phase of training is performed until the loss values reach a plateau and the dictionary magnitudes plateau.
The ﬁne-tuning training phase begins after the transport operator training phase is complete.
In this phase, both the network weights and the transport operator weights are updated using the joint objective:
[Equation]
(19) where E Ψ is deﬁned in (2).
Using this objective, we alternate between taking steps on the transport operator weights while the network weights are ﬁxed and taking steps on the network weights while the transport operator weights are ﬁxed.
Additionally, during the ﬁne-tuning phase we incorporate occasional steps in which we update the network weights using only the reconstruction loss to ensure effective image reconstruction.
In most cases, it is necessary to reduce the γ parameter in front of the Frobenius norm dictionary regularizer prior to ﬁne-tuning or the dictionary magnitudes will reduce to zero.
We report the γ we use for transport operator training and ﬁne-tuning in the experimental details sections below.
It may also be necessary to decrease the network learning rate during ﬁne-tuning.
The coefﬁcient encoder training requires a network that is trained to classify data from our selected dataset.
We train this classiﬁer using training data from a given dataset.
For datasets with worse autoencoder reconstruction quality, we train the classiﬁer in the latent space.
Otherwise we train the classiﬁer on images in the data space.
With the MAE and classiﬁer trained, we train the coefﬁcient encoder network following the strategy described in Section 3.2 and
Appendix A.
To train the CAE, we use the same autoencoder architecture used with the MAE with the addition of a Frobenius norm regularizer on the encoder Jacobian, weighted by a selected λ value (different from the λ in equation 19).
The Jacobian is computed using PyTorch automatic differentiation.
We ﬁnd the Jacobian norm decreases to the same value irrespective of our choice of λ , leading us to choose λ = 1 .
For the β -VAE we use the same architectures outlined in (Higgins et al., 2017) with β = 10 for CelebA and β = 5 for MNIST and FMNIST.
We ﬁnd that setting β any higher results in poor reconstructive performance.
Scripts for training both comparison methods are included in the code repository.
Hyper-parameter tuning for all experiments was performed on the Georgia Tech Partnership for
Advanced Computing Environment (PACE) clusters (PACE, 2017).
Experiments were performed using a Nvidia Quadro RTX 6000.
Runs training the CAE, and β -VAE on CelebA were all run on a separate machine with a Nvidia TITAN RTX.
D P ARAMETER S ELECTION
The MAE model has several hyperparameters that must be tuned and we will provide guidance to determining ideal parameter values for our experiments and future experiments.
First we will describe some signs to look out for to identify if a run is succeeding or failing.
One indicator that we compute is the transport operator difference which is:
[Equation]
where (cid:98) Ψ m is the dictionary after the gradient step is taken.
A gradient step should decrease the transport operator objective meaning the E Ψ diff value should be positive for an effective step.
If there are many gradient steps that result in negative E Ψ diff values, that indicates that the parameters are not optimal or that the learning rate is too large.
This is an important metric to observe during ﬁne-tuning to determine whether to select a smaller γ or smaller network learning rate.
Signs of failure of a training run with selected training parameters:
• All operator magnitudes reduce towards zero.
• All inferred coefﬁcients between point pairs are zero.
• Most of the operators generate transformation paths with latent values that increase quickly to inﬁnity.
• Many steps have negative values for E Ψ diff .
• The operator magnitudes increase which results in unstable training steps with NaN values in the computed objective.
Dictionary regularizer parameter
The dictionary regularizer parameter γ is the weight on the
Frobenius norm term in equation 2.
This objective term serves two purposes.
First, it balances the effect of the coefﬁcient sparsity regularizer with the parameter ζ .
If ζ is large, the sparsity regularizer encourages small coefﬁcient magnitudes and one way to achieve that while still effectively inferring paths is to increase the magnitude of the operators.
The Frobenius norm term must have a large enough inﬂuence to counterbalance this force or the operators will increase in magnitude to the point of being unstable.
If a run is becoming unstable, we recommend decreasing ζ or increasing γ .
The second purpose of the dictionary regularizer term is to identify which operators are necessary to represent the transformations on the data manifold.
If an operator is not being used to represent a transformation between z 0 and z 1 then the dictionary regularizer reduces its magnitude to zero.
Therefore, during training we are able to estimate the model order based on how many dictionary elements remain non-zero.
In our tests we vary γ between 2 × 10 − 8 and 2 × 10 − 4 .
If γ is too small, it will not counterbalance the coefﬁcient sparsity term and the dictionaries will grow to unstable magnitudes.
If γ is too large it will reduce the magnitude of all operators to zero.
During the ﬁnetuning steps, we have often found it necessary to reduce the γ because, with a larger γ , both the operator magnitudes and the latent vector magnitudes can decrease substantially which leads to an ineffective manifold model.
Coefﬁcient sparsity parameter
The coefﬁcient sparsity parameter ζ in equation 2 controls the sparsity of the coefﬁcients that are used to estimate paths between z 0 and z 1 .
We run tests with values of ζ between 0.005 and 2.
From dataset to dataset the ideal value varies.
If ζ values are too small then all of the operators are used to represent all of the paths between point pairs.
The ζ should be increased so fewer than M coefﬁcients are used for each inferred path.
When ζ is too large, all the coefﬁcients go to zero during inference.
This means there is no path inferred between z 0 and zbecause of overweighting the sparsity constraint.
Number of dictionary elements As mentioned above, the dictionary regularizer acts as a model order selection tool so our strategy for selecting number of dictionary elements M is to increase the number of dictionary elements until some of their magnitudes begin reducing to zero during training.
This indicates that some of the operators are not necessary for representing transformations.
Latent dimension The latent dimension of the autoencoder is selected to ensure quality reconstructed image outputs.
Relative weight of reconstruction and transport operator objectives The parameter λ determines the weight of the reconstruction term relative to the transport operator term.
We observe good performance of the model for λ between 0.5 and 0.75.
E C OMPARING P OINT P AIR S ELECTION S TRATEGIES
In the past, transport operators have been trained using point pairs that were selected randomly from the same class (Connor et al., 2021) or with some knowledge of transformation labels (Connor &
Rozell, 2020).
In this work, we establish a method that learns natural transformations without requiring transformation labels using a perceptual point pair selection strategy.
The perceptual loss metric (which compares the the features values from the penultimate layer of a pretrained classiﬁer model) enables us to select point pairs that may share semantically meaningful transformations without being exactly the same.
Fig.
9 shows examples of nearest neighbors selected using pixel similarity, similarity in the autoencoder latent space, and the perceptual loss metric.
This highlights how the perceptual loss metric can be useful for identifying inputs with similar qualitative characteristics.
For instance, in the set of images on the left, the perceptual loss metric identiﬁes another bald man as a nearest neighbor which has similar characteristics of the initial image even though the exact image looks quite different.
When examining the success of learned operators, one characteristic we care about is how well the operators maintain the stability of generated paths.
We consider generated paths stable if the latent vectors do not expand quickly to inﬁnity.
We have deﬁned this as a useful characteristic for identity-preservation of applied operators – if operators expand latent vectors to inﬁnity, that will very likely lead to a change in identity.
Each of the operators can be viewed as the dynamics matrix of a continuous time linear dynamical systems model
and we can analyze their stability as dynamical systems by observing their eigenvalues (Strogatz, 2000).
In continuous time linear dynamical systems, there are three classiﬁcations of system stability.
If a system is stable, the real parts of all eigenvalues are below zero.
In a stable system, the magnitude of an input shrinks as time progresses forward.
If a system is unstable, the real part of at least one eigenvalue is above zero.
In unstable systems, the magnitude of an input expands to inﬁnity as time progresses forward.
Finally, if a system is marginally stable, then the magnitude of an input remains constant and the real parts of all of its eigenvalues are zero 1 (Strogatz, 2000).
The transport operator model is unique because operators can represent dynamics matrices that deﬁne the natural data variations which do not have the same directionality as temporal systems (where time moves forward).
Therefore transport operators can be applied with positive or negative coefﬁcients and the traditional view of stability of temporal systems no longer applies.
To maintain the stability of generated paths, the operators need to generate cyclic transformation paths that neither increase nor decrease the latent vector magnitudes.
This corresponds to marginal stability in linear dynamical systems.
A marginally stable system has only imaginary eigenvalues with no real parts (Strogatz, 2000).
Therefore, we can identify transport operators approaching marginal stability by investigating the magnitudes of the real parts of their eigenvalues.
The maximum real magnitude of an eigenvalue of a dynamics matrix drives the speed with which the generated paths expand to inﬁnity.
Therefore this maximum real magnitude is the focus of our investigation into the stability of paths generated by learned operators.
When the maximum magnitude of the real part for all eigenvectors associated with an operator is close to zero, that indicates that the operator is closer to marginal stability.
Therefore, we quantify the stability of transport operator generated paths by looking at the maximum magnitude of real parts of eigenvalues associated with each transport operator.
Fig.
10a shows the sorted maximum magnitude of the real parts of eigenvalues in each of the 16 operators learned in the experiment in Section 4.1 which learns natural MNIST variations.
The blue x’s are associated with the operators trained using the perceptual point pair selection strategy and the red dots are associated with the operators trained using a simple strategy of selecting point pairs as nearest neighbors in the latent space.
With the exception of one operator, all the operators trained using latent space similarity for supervision have larger maximum magnitudes of real eigenvalue components than the operators trained using the perceptual loss metric.
This indicates that the operators trained using the latent space similarity to select training point pairs are farther from marginal stability and can be seen as less stable by our deﬁnition of transport operator stability.
To view the effect of these learned operators more intuitively, we plot the values of the latent dimensions as individual operators are applied in Fig.
Each line in these plots shows the inﬂuence of the operators on a single latent dimension (Note that this experiment utilizes a latent dimension of 10).
Fig.
11a shows the paths generated by transport operators trained with the perceptual point pair selection strategy .
The plots with straight lines (i.e., operators 3, 4, 7, 12, 14, and 16) indicate that those operators had their magnitudes reduced to zero during training and therefore they have no impact on the latent vectors when applied.
Most of the operators learned using the perceptual point pair selection strategy are close to cyclic except for operator 2 (the operator with the large real component magnitude in Fig. 10a).
By contrast, several of the operators trained with point pairs selected
[Figure]
as nearest neighbors in the latent space extend to inﬁnity (Fig. 11b).
This can explain the larger maximum real value magnitudes in Fig. 10a.
This analysis leads us to conclude that the perceptual point pair selection strategy has a greater potential to yield identity-preserving transport operators.
[Figure]
F MNIST E XPERIMENT D ETAILS
The MNIST dataset is made available under the terms of the Creative Commons Attribution-Share
Alike 3.0 license.
We split the MNIST dataset into training, validation, and testing sets.
The training set contains 50,000 images from the traditional MNIST training set.
The validation set is made up of the remaining 10,000 images.
The traditional MNIST testing set is used for our testing set.
The input images are normalized so their pixel values are between 0 and 1.
The network architecture used for the autoencoder is shown in Table 1.
The training parameters for the transport operator training phase and the ﬁne-tuning phase are shown in Tables 2 and 3.
Prior to training the coefﬁcient encoder for the MNIST dataset, we train a classiﬁer on the labeled
MNIST image data which we use to encourage identity-preservation during coefﬁcient encoder training.
The training parameters for the coefﬁcient encoder are shown in Table 4.
The image classiﬁer we use is based on the simple LeNet architecture with two convolutional layers and three fully connected layers (LeCun et al., 1998)
[Table] [Table]
[Table] [Table]
[Table] [Table]
G MNIST E XPERIMENT A DDITIONAL R ESULTS
Here we show additional experimental details and results for the MNIST experiment.
Fig.
12 shows the magnitude of all 16 operators after the ﬁne-tuning phase.
Six of the operators have their magnitudes reduced to zero.
Fig.
13 shows the paths generated by transport operators trained on MNIST data.
[Figure]
H F ASHION -MNIST E XPERIMENT D ETAILS
The Fashion-MNIST dataset is made available under the terms of the MIT license.
We split the
Fashion MNIST dataset into training, validation
[Table]
and testing sets.
The training set contains 50, images from the Fashion-MNIST training set.
The validation set is made up of the remaining 10, images.
The traditional Fashion-MNIST testing set is used for our testing set.
The input images are normalized so their pixel values are between 0 and 1.
The network architecture used for the autoencoder is the same as in the MNIST experiment and it is shown in Table 1.
The training parameters for the transport operator training phase and the ﬁne-tuning phase are shown in Tablesand 6.
Prior to training the coefﬁcient encoder for the Fashion-MNIST dataset
[Table]
we train a classiﬁer the latent vectors associated with labeled Fashion-MNIST data which we use to encourage identitypreservation during coefﬁcient encoder training.
The training parameters for the coefﬁcient encoder are shown in Table 7.
The latent vector classiﬁer has a simple architecture of Linear(512), ReLU,
Linear(10), Softmax.
[Table] [Table]
[Table]
I F ASHION MNIST E XPERIMENT A DDITIONAL R ESULTS
Here we show additional experimental details and results for the Fashion-MNIST experiment.
Fig.
14 shows the magnitude of all 16 operators after the ﬁne-tuning phase.
Six of the operators had their magnitudes reduced to zero.
To visualize how the use of the transport operators varies over the latent space
[Table]
we generate an
Isomap embedding (Tenenbaum et al., 2000) of latent vectors and color each point by the encoded scale parameter for coefﬁcients associated with each of the transport operators.
Fig.
15a shows these embeddings for Fashion-MNIST data.
Each operator has regions of the latent space where their use is concentrated.
Fig.
15b shows the average coefﬁcient scale weights for each class (rows) and each transport operator (columns) for Fashion-MNIST.
There are some classes like trouser and sandal (classes 1 and 5) which have large encoded coefﬁcient scale weights for most of the transport operators.
This means they are robust to many natural transformations.
Other classes like coat and shirt (classes 4 and 6) have smaller encoded coefﬁcient scale weights which means they are sensitive to many transformations.
The images to the right in Fig.
15b show transport operators being applied to samples with high encoded scale weights (in a yellow box) and samples with low encoded scale weights (in a blue box).
Fig.
16 shows the paths generated by non-zero transport operators trained on Fashion-MNIST data.
[Figure] [Figure]
J C ELEB A E XPERIMENT D ETAILS
The CelebA dataset is publicly available for non-commercial research purposes.
We split the CelebA dataset into training and testing sets.
The training set contains the ﬁrst 150,000 images accompanied with the entire test set.
The input images are normalized so their pixel values are between 0 and 1.
The network architecture used for the autoencoder is shown in Table 8.
The training parameters for the transport operator training phase and the ﬁne-tuning phase are shown in Table 9.
The attribute classiﬁer is a
ResNetmodel adapted from https://github.com/d-li14/face-attribute-prediction with 16 classiﬁer heads after the layer with 512 hidden units.
Each classiﬁer head corresponds to a single attribute and is modeled as a fully-connected linear layer with 256 hidden units, followed by batch normalization, dropout, and ReLU layers.
Afterwards, two logits are output for a 0/1 prediction for each classiﬁer output.
The training procedure, including the dynamic loss weighting, follows (Mao et al., 2020).
[Table] [Table]
K C ELEB A E XPERIMENT A DDITIONAL R ESULTS
Here we show additional experimental results for the CelebA experiment.
Fig.
17 shows the paths generated by the 40 transport operators trained on celebA data.
Each row represents a different operator acting on the same input image.
Images in the middle column of the image block are the reconstructed inputs and images to the right and left are images decoded from transformed latent vectors in positive and negative directions, respectively.
Fig.
18 shows the classiﬁcation outputs of the attribute classiﬁer for example transport operators.
These two operators that vary smiling, beard, and sunglasses attributes.
[Figure] [Figure]
Fig.
19 shows an interesting feature in our learned operators – many of them generate cyclic paths that begin and end at nearly the same point.
Also, in these cyclic sequences, the transformations seem to lead to a change in gender.
The image sequence in Fig.
19a shows the path generated by a single operator.
The image in the middle is the reconstructed input image and the images to the left and right are the paths generated by negative and positive coefﬁcients respectively.
This operator changes the hairline and quantity of bangs.
As we apply the transport operator with a negative coefﬁcient (to the left of center), the woman gains bangs and then becomes a man with a moustache on the far left of the image.
As we apply the transport operator with a positive coefﬁcient gets higher and then she becomes a with a high and eventually, on the right the woman transforms into a man with bangs, similar to the on
The similarity between generated images left is notable because this indicates the transformation path is nearly closed.
Fig.
19b shows the change in ﬁve of the 32 latent dimensions over the generated path.
These paths have a nearly cyclic structure.
This is particularly interesting because in (Connor & Rozell, 2020) they learn closed transformation paths by selecting point pairs on known cyclic transformation paths and highlight the beneﬁt of the transport operator model for representing these types of paths.
In this case, we learn this cyclic path with only perceptual point pair supervision.
Additionally, this identiﬁes a beneﬁt of the transport operator approach over other models of the manifold in a neural network latent space.
We can learn nonlinear paths that keep the generated points in the same neighborhood in the latent space without extending to inﬁnity which is inevitable when linear paths that are used to represent natural transformations.
[Figure]