C LOSED -L OOP D ATA T RANSCRIPTION TO AN LDR VIA M INIMAXING R ATE R EDUCTION
Anonymous authors
Paper under double-blind review
A BSTRACT
This work proposes a new computational framework for automatically learning a closed-loop transcription between multi-class multi-dimensional data and a linear discriminative representation (LDR) that consists of multiple multi-dimensional linear subspaces.
In particular, we argue that the optimal encoding and decoding mappings sought can be formulated as the equilibrium point of a two-player minimax game between the encoder and decoder .
A natural utility function for this game is the so-called rate reduction , a simple information-theoretic measure for distances between mixtures of subspace-like Gaussians in the feature space.
Our formulation avoids expensive evaluating and minimizing approximated distances between arbitrary distributions in either the data space or the feature space.
To a large extent, conceptually and computationally this new formulation uniﬁes the beneﬁts of Auto-Encoding and GAN and naturally extends them to the settings of learning a both discriminative and generative representation for complex multiclass and multi-dimensional real-world data.
Our extensive experiments on many benchmark datasets demonstrate tremendous potential of this framework: under fair comparison, visual quality of the learned decoder and classiﬁcation performance of the encoder is competitive and often better than existing methods based on GAN,
VAE or a combination of both.
I NTRODUCTION AND R ELATED W ORK
One of the most fundamental tasks in machine learning is to learn and model complex distributions (or structures) of real-world data, such as images or texts, from a set of observed samples.
By “learn and model”, one typically means that we want to establish a (parameteric) mapping between the distribution of the real data, say x ∈ R D , and a more compact random variable, say z ∈ R d : f ( · , θ ) : x ∈ R D
(cid:55)→ z ∈ R d or the inverse g ( · , η ) : z ∈ R d (cid:55)→ x ∈ R D , (1) where z has certain standard structure or distribution (e.g. normal distributions).
The so-learned representation or feature z would be much easier to use for either generative or discriminative purposes.
Be aware that the support of the distribution of x (and that of z ) can be low-dimensional hence the above map(s) may not be so well deﬁned off the support in the space R D (or R d ) .
L EARNING G ENERATIVE M ODELS VIA A UTO -E NCODING OR GAN
Auto-Encoding.
In the machine learning literature, roughly speaking, there have been two representative approaches to such a distribution-learning task.
One is the classic “Auto Encoding” (AE) approach (Kramer, 1991; Hinton & Zemel, 1993) that aims to simultaneously learn an encoding mapping f from x to z and an (inverse) decoding mapping g from z back to x :
[Equation]
Here we use bold capital letters to indicate a matrix of ﬁnite samples X
=
[ x 1 , . . .
, x n ] ∈ R D × n of x and their mapped features Z =
[ z 1 , . . .
, z n ] ⊂ R d × n , respectively.
Typically, one wishes for two properties: ﬁrstly, the (empirical) distribution of the mapped samples Z , denoted as ˆ p ( z | X , θ ) , is close to certain desired distribution p ( z ) , say some much lower-dimensional disentangled multivariate
Gaussian; 1 and, secondly, the decoded samples ˆ X are “similar” or close to the original X .
In the feature space z , to enforce the learned distribution ˆ p ( z ) to be close to a target p ( z ) , one normally minimizes a certain “distance” between the two distributions, say the KL-divergence : min D KL (ˆ p, p ) .
However, it is very difﬁcult, often computationally intractable, to precisely compute or minimize such a distance D between two arbitrary degenerate distributions in high-dimensional spaces.
So in practice, one typically chooses to minimize instead certain approximate bounds (surrogates) of the distance, such as a variational bound in variational auto-encoding (VAE) (Kingma & Welling, 2013; Zhao et al., 2017) and the earth mover’s (Wasserstein) distance .
In this work, we will show that if we impose speciﬁc requirements on the (distribution of) learned feature z (e.g. being subspace-like Gaussians), natural distances can be given in closed-form.
GAN.
Compared to measuring distribution distance in the (often controlled) feature space z , a much more challenging issue is how to effectively measure the distance between the decoded samples ˆ X and the original X in the data space x .
For instance, for visual data such as images, despite extensive studies in the computer vision and image processing literature (Wang et al., 2004), it remains elusive to ﬁnd a good measure for similarity of real images that is both efﬁcient to compute and effective in capturing visual quality and semantic information of the images equally well.
Generative Adversarial
Nets (GAN) (Goodfellow et al., 2014) offers an ingenious idea to resolve this difﬁculty by utilizing a powerful discriminator d , usually modeled and learned by a deep network, to discern differences between the generated samples ˆ X and the real ones X :
[Equation]
Such a discriminator minimizes the Jensen-Shannon divergence, between the data X and ˆ X . But as shown in (Arjovsky et al., 2017), the JS-divergence can be highly ill-conditioned to optimize when the distributions are low-dim. So in practice, one may choose to replace the JS-divergence with the earth mover’s distance or other variants.
The original GAN aims to directly learn a mapping g , called a generator, from a standard distribution (say a low-dim Gaussian random ﬁeld) to the real (visual) data distribution in a high-dim space.
However, distributions of real-world data can be rather sophisticated and often contain multiple classes and multiple factors in each class (Bengio et al., 2013).
That makes learning the mapping g rather challenging in practice, suffering difﬁculties such as mode-collapse (Srivastava et al., 2017).
As a result, many variants of GAN have been subsequently developed in order to improve the stability and performance in learning multiple modes and disentangling different factors in the data distribution, such as Conditional GAN (Mirza & Osindero, 2014; Sohn et al., 2015;
Mathieu et al., 2016; Van den Oord et al., 2016;
Wang et al., 2018), InfoGAN (Chen et al., 2016; Tang et al., 2021), or Implicit Maximum Likelihood Estimation (IMLE) (Li & Malik, 2018; Li et al., 2020)
In particular, to learn a generator for multi-class data, prevalent conditional GAN literature requires label information as inputs during training (Mirza & Osindero, 2014;
Odena et al., 2017; Dumoulin et al., 2016; Brock et al., 2018).
Recently Wu et al.
(2019b;a) has proposed to train a k -class GAN by generalizing the two-class cross entropy to a ( k + 1) -class cross entropy.
In this work, we will introduce a more reﬁned 2 k -class measure for the k real and k generated classes.
In addition, to avoid features for each class to collapse to a singleton, we will use the so-called rate reduction measure that promotes multi-dimension in the learned features (Yu et al., 2020).
Another line of research is about how to stablize the training of GAN.
SN-GAN (Miyato et al., 2018) has shown spectral normalization on the discriminator is rather effective, which we will adopt in our work.
PacGAN (Lin et al., 2018) shows that the training stability can be signiﬁcantly improved by packing a pair of real and fake images together for the discriminator.
In this work, we show how to naturally generalize this idea to discriminating an arbitrary number of pairs of real and decoded samples.
Also, Wu et al. (2019a) has shown that optimizing the latent features leads to state of the art visual quality.
There are strong reasons to believe that their method essentially utilizes the
Compressed Sensing principle to implicitly exploit low-dimensionality of the feature distribution.
Our framework will explicitly impose and exploits such low-dimensional structures of the learned feature distribution.
Combination of AE and GAN.
Although AE (VAE) and GAN have started with somewhat different motivations, they have evolved into popular and effective frameworks for learning and modeling complex distributions of many real-world data such as images.
Many recent efforts tend to combine both Auto-Encoding and GAN to generate more powerful generative frameworks for more diverse data sets, such as Larsen et al.
; Rosca et al.
; Srivastava et al.
; Bao et al.
; Huang et al.
; Ulyanov et al.
; Vahdat & Kautz (2020).
As we will see, in our framework, AE and GAN can be naturally interpreted as two segments of a closed-loop data transcription process.
But unlike GAN or VAE, the distribution of the feature z is learned from the data x and its low-dim support in R d is explicitly modeled as a union of discriminative subspaces.
1. L EARNING L INEAR D ISCRIMINATIVE R EPRESENTATION VIA R ATE R EDUCTION
Recently, Chan et al.
(2021) has proposed a new objective for deep learning that aims to learn a linear discriminative representation (LDR) for multi-class data.
The basic idea is to map distributions of real data, potentially on multiple nonlinear submanifolds, to a family of canonical models consisting of multiple incoherent (or orthogonal) linear subspaces.
To some extent, this generalizes nonlinear
PCA (Kramer, 1991) to the more general/realistic settings where we simultaneously apply multiple nonlinear PCAs to data on multiple nonlinear submanifolds.
Unlike conventional discriminative methods that only aim to predict class labels, the LDR aims to learn the likely multi-dimensional distribution of the data hence is potentially suitable for both discriminative and generative purposes.
It has been shown that this can be achieved via maximizing the so-called “rate reduction” objective.
MCR 2 .
More precisely, consider a set of data samples X
=
[ x 1 , . . .
, x n ] ∈ R D × n from k different classes and we use Π j , j = 1 , . . .
, k to denote the memberships of the samples in k classes X
= ∪ kj =1 X j .
One seeks a continuous mapping f ( · , θ ) :
x (cid:55)→ z from X to an optimal representation Z =
[ z 1 , . . .
, z n ] ⊂ R d × n that maximizes the following coding rate reduction objective, known as the MCR 2 principle (Yu et al., 2020):
[Equation]
R c ( Z ,(cid:15) | Π ) , (4) where α = dn(cid:15) 2 , α j = d tr ( Π j ) (cid:15) 2 , γ j = tr ( Π j ) n for j = 1 , . . .
, k .
In this paper, for simplicity we denote ∆ R ( Z , Π , (cid:15) ) as ∆ R ( Z ) assuming Π , (cid:15) are known and ﬁxed.
The ﬁrst term R ( Z | (cid:15) ) is the coding rate of the whole feature set Z (coded as a Gaussian source) with a prescribed precision (cid:15) ; the second term R c ( Z | Π , (cid:15) ) is the average coding rate of the k subsets of features Z j = f ( X j ) (each coded as a Gaussian).
As it has been shown by Yu et al.
(2020), maximizing the difference between the two terms will “expand” the whole feature set while “compressing and linearizing” features of each of the k classes. If the mapping f maximizes the rate reduction, it maps the features of different classes into independent (orthogonal) subspaces in R d .
Figure 1 illustrates a simple example of data with k =classes (on two submanifolds) mapped to two incoherent subspaces (solid black lines).
D ATA T RANSCRIPTION VIA R ATE R EDUCTION 2. LDR T RANSCRIPTION
One issue with this one-sided LDR learning is that maximizing the above objective tends to maximize the dimension of the learned subspace for features in each class.
2 To verify whether the learned features are good, we may consider learning a decoder g ( · , η ) :
z (cid:55)→ x from the representation
Z = f
( X , θ ) back to the data space x : ˆ X =
g ( Z , η ) , and check how close X and ˆ X are or how close their features Z and ˆ
Z are.
The overall pipeline can be illustrated by the following diagram:
[Equation]
where the overall model has parameters:
Θ = { θ, η } .
Notice that in the above process, the segment from X to ( ˆ X , X ) resembles a typical Auto-Encoding process although, as we will soon see,
our
MCR 2 -based encoder f plays an additional role as a discriminator.
The segment from Z to ( ˆ Z , Z ) draws resemblance to the typical GAN process although, in our context, the distribution of the latent variable z will be learned from the data x , which often is a distribution with multiple modes and each mode with multiple factors.
Here, in the speciﬁc context of rate reduction, we name this special auto-encoding process LDR
Transcription since the maximal rate reduction principle explicitly transcribes the data X , via f , to features Z on a linear discriminative representation (LDR), which can be subsequently decoded back to the data space ˆ X , via g .
Hence, the encoding and decoding maps f and g together form a closed-loop process, as illustrated in Figure 1.
We wish this closed-loop transcription process to have the following good properties:
• Injectivity: the generated ˆ x = g ( f ( x , θ ) , η ) ∈ ˆ X to be as close to (ideally the same as) the original data x ∈ X , in terms of certain measure of similarity or distance.
• Surjectivity: for all mapped images z = f
( x ) ∈ Z of the training data x ∈ X , there are decoded samples ˆ z
=
f
( g ( z , η )
, θ ) ∈
ˆ Z close to (ideally the same as) z . [Figure]
M EASURING D ISTANCES IN THE F EATURE S PACE AND D ATA S PACE
Contractive Measure.
For the second item in the above wishlist, as the representations in the feature space z are by design linear subspaces or (degenerate) Gaussians, we have geometrically or statistically meaningful metrics for both samples and distributions in the feature space z .
For example, we care about distance between distributions between the features of the original data Z and the transcribed ˆ Z .
Since the features of each class, Z j and ˆ Z j , are subspace/Gaussian-like, their “distance” can be measured by the rate reduction, with (4) restricted to two sets of equal size):
[Equation]
(6) which, as shown in Fig. 1, measures the space volume between a pair of black lines and blue lines, as per interpretation of rate reduction in (Yu et al., 2020).
Then for the “distance” of all, say k , classes, we simply sum the rate reduction for all pairs:
[Equation]
where Z j = f ( X j , θ ) and ˆ Z j = f
( ˆ X j , θ ) .
Obviously, a main goal of the learned decoder g ( · , η ) is to minimize the distance between these distributions.
Notice that if the encoder f preserves (i.e. injective for) the intrinsic structures of the original data X , 3 this criterion essentially aims to ensure there will be some decoded sample ˆ x close to every data sample x – hence the decoder should be “surjective.”
According to the ideas of IMLE (Li & Malik, 2018), such a requirement could effectively help avoid mode-collapsing or mode-dropping.
Contrastive Measure.
For the ﬁrst item in our wish-list, however, we normally do not have a natural metric or “distance” for similarity of samples or distributions in the original data space x . 4 To alleviate this difﬁculty, we can measure the similarity or difference between ˆ X and X through their mapped images ˆ Z and Z in the feature space (again assuming f is structure-preserving).
If we are interested in discerning any difference in the distributions of the original and transcribed samples, we may view the MCR 2 feature encoder f
( · , θ ) as a “discriminator” to magnify any difference between all pairs of X j and ˆ X j , by simply maximizing, instead of minimizing, the same quantity in (7):
[Equation]
That is, a “distance” between X and ˆ X can be measured as the maximally achievable rate reduction between all pairs of classes in these two sets.
In a way, this measures how well or bad the decoded ˆ X aligns with the original data X – hence measuring the goodness of “injectivity”.
Notice that such a discriminative measure is consistent with the idea of GAN (Goodfellow et al., 2014).
Nevertheless, here the MCR 2 -based discriminator f naturally generalizes to cases when the data distributions are multi-class and multi-modal, and the discriminativeness is measured with a more reﬁned measure – the rate reduction, instead of the typical two-class loss (e.g. cross entropy) used in GANs.
One may wonder the reason why we need the mapping f ( · , θ ) to function as a discriminator between X and ˆ X by maximizing
[Equation]
Figure 2 gives a simple illustration: there might be many decoders g such that f ◦ g is an identity mapping f ◦ g
( z ) =
z for all z in the subspace S z in the feature space.
However, g ◦ f is not necessarily an auto-encoding map for x in the original distribution S
x
(here for simplicity drawn as a subspace).
That is,
[Equation]
S x (let alone g ◦ f ( x ) = x ).
One should expect, without careful control of the image of g , this would typically be the case especially when the support of the distribution of x is extremely low-dimensional in the original high-dimensional data space (e.g. in the millions for images).
2. E NCODING AND D ECODING AS A T WO -P LAYER M INI M AX G AME
Comparing (7) and (8), we see the roles of the encoder f ( · , θ ) and the decoder g ( · , η ) naturally as a “ a two-player game ”: while the encoder f tries to magnify the difference between the original data and the transcribed; the decoder g aims to minimize the difference.
Now for convenience, let us deﬁne the “closed-loop encoding” function:
[Equation]
Ideally, we want this function to be very close to f ( x , θ ) .
With this notation, combining (7) and (8), a closed-loop notion of “distance” between X and ˆ X can be computed as an equilibrium point to the following Min-Max program for the same utility in terms of rate reduction:
[Equation]
Notice that this only measures the difference between (features of) the original data and its transcribed version.
It does not measure how good the representation Z (or ˆ Z ) is for the multiple classes within
X (or ˆ X ).
To this end, we may combine the above distance with the original MCR 2 -type objectives (4): namely, the rate reduction ∆ R ( Z ) and ∆ R ( ˆ Z ) for the learned LDR Z for X and ˆ Z for the decoded ˆ X .
The overall multi-class Min-Max program for learning the LDR model is:
[Equation]
Notice that, without the terms associated with the generative part h or with all such terms ﬁxed as constant, the above objective is precisely the original MCR 2 objective.
Also, notice that the minimax objective function depends only on the data
X hence one can learn the encoder and decoder (parameters) without the need of sampling any other distribution (as needed by GANs)!
As a special case, if X only has one class, the above Min-Max program reduces 6 to a special “binary” form between X and ˆ X :
[Equation]
Sometimes, even when X contains multiple classes/modes, one could still view all classes together as one class.
Then the above binary objective is to align the union distribution of all classes with their decoded ˆ X .
This is typically a simpler task to achieve than the multi-class one (11) since it does not require to learn a more reﬁned multi-class LDR for the data, as we will later see in experiments.
Notice that this binary case resembles formulation of the original GAN (3) by viewing X and ˆ X as two classes { 0 , 1 } .
Nevertheless, instead of using cross entropy, our formulation adopts a more reﬁned rate reduction measure, which has been shown to promote diversity in the learned representation (Yu et al., 2020).
2. D ISCUSSIONS ABOUT THE M INIMAX O BJECTIVE
There are many outstanding questions that one may wonder about the above formulation: When does the game min θ max η T X ( θ, η ) have a well-deﬁned Nash-equilibrium point ( θ (cid:63) , η (cid:63) ) ?
What good characteristics the associated optimal encoder f (cid:63) and g (cid:63) have?
For instance, under what conditions, will their composition be or close to be a true auto-encoding g (cid:63) ◦ f (cid:63) ≈ Id, and will the learned features Z (cid:63) =
f ( X , θ (cid:63) ) form orthogonal or highly incoherent subspaces?
In this paper, we will not be able to give complete rigorous answers to these questions.
The main purpose of this paper is to provide compelling empirical evidence that answers to the above questions are highly likely to be positive under fair broad conditions (see Section 3).
Here we will give some preliminary justiﬁcation based on characteristics of each team in T X ( θ, η ) and will also give reasons why a rigorous characterization of the optimality conditions is mathematically challenging.
Let us ﬁrst consider maximizing the ﬁrst two rate reduction terms in T X ( θ, η ) , assuming η ﬁxed and Z ( θ ) and ˆ Z ( θ ) are aligned, i.e.,
∆ R (cid:0) Z j ( θ ) , ˆ Z j ( θ ) (cid:1) reaches its minimum 0 for j ∈ [ k ] (to be discussed below).
Then they essentially maximize the rate reduction ∆ R ( Z ( θ ))
+ ∆ R ( ˆ Z ( θ )) for features of the k classes in Z ( θ )
=
f
( X , θ ) and ˆ Z ( θ )
=
h ( X , θ, η ) .
Based on the results in Yu et al.
(2020), a sufﬁcient condition for the rate reduction to reach a (local) maximum is when features of all classes belong to subspaces orthogonal to each other 8 : Z (cid:62)
i
Z j =
and ˆ Z (cid:62)
i ˆ Z j =
for i (cid:54)
=
j .
Now let us consider minimizing the k rate reduction terms in the third term of T X ( θ, η ) , assuming θ ﬁxed and the ∆ R ( ˆ Z ) achieves its maximum (as mentioned above).
Then, from the property of the log det( · ) function, we know that each of the k terms in the contractive measure: according to
Lemma 10 of Chan et al.
(2021), ∆ R (cid:0) Z j , h ( ˆ Z j ( η ) (cid:1) reaches its minimum ( 0 ) when the covariance matrices of Z j and ˆ Z j are the same, i.e., Z j Z (cid:62)
j = ˆ Z j ˆ Z (cid:62) j .
If one further assumes that the encoder is sufﬁciently injective, perfect alignment of Z j and ˆ Z j suggests perfect alignment of X j and ˆ X j .
Based on the above properties, one may show that when the k classes in each of Z and ˆ Z form k orthogonal subspace and each corresponding pair of Z j and ˆ Z j aligns perfectly, then one reaches a critical point 9 of the minimax game min θ max η T X ( θ, η ) .
It remains open whether such a critical point is a strict saddle point – a (local) Nash equilibrium.
There are more subtle questions that one could or should ask too, such as: how are the learned features
Z and ˆ Z distributed within the subspaces?
Are features for individual sample
z i ˆ z
i (hence x i and ˆ x i ) well aligned?
Results in Yu et al.
(2020) suggest that covariance of the features in each subspace needs to be nearly isotropic in order for a rate reduction term to be maximized.
When minimizing a rate reduction term, results about the so-called Brascamp-Lieb inequalities (Jonathan Bennett et al., 2007) suggest that a necessary condition for the minimum to be reached is when the distribution (density) within each subspace becomes Gaussian.
Hence from these results one may conjecture that the optimal distribution in each subspace for the minimax game is Gaussian too.
Although theoretically nothing is known about relationships between the features z i ˆ
z i of individual samples, (multi-class) experiments in next section suggest that they are actually very close.
Despite all these challenging open mathematical questions, one may notice that the aggregated effect of all the terms associated with the decoder g ( · , η ) is precisely “opposite” to the effect of the terms associated with the encoder f ( · , θ ) .
All together, the Min-Max program (11), or (12), aims to strike a good tradeoff between maximizing the expressiveness and discriminativeness of the learned representation as well as minimizing any unnecessary cost, in terms of the overall coding rate and errors in decoding, all measured in rate reduction (assuming features being subspace Gaussians).
E MPIRICAL V ERIFICATION ON R EAL -W ORLD I MAGERY D ATASETS
The experiment section serves three purposes: First, we empirically justify the new formulation by demonstrating good properties of the learned encoder, decoder, and representations.
Second, we compare our method with several representative methods from the GAN family and VAE family.
Finally, we evaluate the learned LDR through visualization and classiﬁcation tasks.
[Figure]
[Table]
Datasets.
We provide extensive qualitative and quantitative experimental results on the following datasets: MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), CelebA (Liu et al., 2015), LSUN bedroom (Yu et al., 2015), and ImageNet ILSVRC(Russakovsky et al., 2015).
The network architectures and implementation details can be found in
Appendix A.1 and corresponding appendix section for each dataset.
3. E MPIRICAL J USTIFICATION OF LDR T RANSCRIPTION
To empirically validate our new framework, we conduct experiments from a small low-variety dataset (MNIST), to a small dataset of diverse real-world objects (CIFAR-10), to higher resolution images (CelebA, LSUN-bedroom), to a large-scale image set (ImageNet).
The results are evaluated both quantitatively and qualitatively.
Implementation details and more results are given in the Appendix.
Comparison (IS and FID) with other formulations.
First, we conduct ﬁve experiments to fairly compare our formulation with GAN (Radford et al., 2015) and VAE(-GAN) (Larsen et al., 2016) on
MNIST and CIFAR-10.
Except for the objective function, everything else is exactly the same for all methods (e.g. networks, training data, optimization method).
These experiments are: 1).
GAN; 2).
GAN with its objective replaced by that of the LDR-Binary (12); 3).
VAE-GAN ; 4).
Binary LDR (12); and 5).
Multi-class LDR (11).
Some visual comparison is given in Fig.
IS (Salimans et al., 2016) and FID (Heusel et al., 2017) scores are summarized in Table 1.
As we see from the above Table 1, replacing cross-entropy with the Equation (12) can improve the generative quality.
The two LDR formulations are clearly on par with the others in terms of IS and signiﬁcantly better in FID.
Finally, with the same training datasets, quality of LDR-Multi is lower than
LDR-Binary.
This is expected as the multi-class task is more challenging.
Nevertheless, as we will see soon, images decoded by LDR-Multi align much better with their classes than Binary.
Visualizing correlation of features Z and decoded features ˆ Z .
We visualize the cosine similarity between Z and ˆ Z learned from the multi-class objective (11) on MNIST and CIFAR-10, which indicates how close ˆ z = f ◦ g
( z ) is from z .
Results in Fig 4 show that Z and ˆ Z are aligned very well within each class.
The block-diagonal patterns for MNIST are sharper than those for
CIFAR-10, as images in CIFAR-10 have more diverse visual appearances.
Visualizing auto-encoding of the data X and the decoded ˆ X .
We compare some representative X and ˆ X on MNIST, CIFAR-10 and ImageNet (10 classes) to verify how close ˆ x = g ◦ f ( x ) is to x .
The results are shown in Fig 5, and visualizations are created from training samples.
Visually, the auto-encoded ˆ x faithfully captures the visual features from its respective training sample x .
There also exist some auto-encoded images that are almost identical to the original.
We refer the reader to Appendix A.2 A.4 A.7 for
[Figure]
[Table]
more visualizations of results on these datasets, including similar results on transformed MNIST digits and real-life images decoded by randomly sampling the learned feature subspaces
[Table]
3. C OMPARISON TO E XISTING G ENERATIVE M ETHODS
Table 2 gives a quantitative comparison of visual quality of our method with others on CIFAR-10,
STL-
[Table]
and ImageNet.
In general, there is a large difference in terms of FID and IS scores between the GAN family and the VAE family of models.
SNGAN (Miyato et al., 2018) are commonly used methods in most generative applications while LOGAN (Wu et al., 2019a) is the state-of-the-art method on ImageNet in terms of FID and IS.
As we see, even if the rate reduction is not speciﬁcally designed nor engineered for visual quality 10 , our method is still rather competitive in terms of these metrics.
3. B ENEFITS OF THE L EARNED LDR T RANSCRIPTION M ODEL
As we have argued before, the learned LDR transcription model (including the feature z , the encoder f , and the decoder g ) can be used for both generative and discriminative purposes.
Decoding samples from the feature distribution.
Using the CIFAR-10 and CelebA datatsets, we visualize images decoded from samples of learned feature subspace.
For the CIFAR-10 dataset, for each class j , we ﬁrst compute the top-4 principal components of the learned features Z j (via SVD).
For each class j , we then compute |(cid:104) z ij , v lj (cid:105)| , the cosine similarity between the l -th principal direction v lj and feature sample z ij .
After ﬁnding the top-5 z ij according to |(cid:104) z ij , v lj (cid:105)| for each class j , we reconstruct images ˆ x ij =
g ( z ij ) .
Each row of Fig.
6 is for one principal component.
We observe that images in the same row share many common features; images in different rows differ signiﬁcantly in characteristics like shape, background, and style.
See Appendix A.4 for more visualization of principal components learned for all 10 classes of CIFAR-10.
For the CelebA dataset, we calculate the principal components of all learned features in the latent space.
Fig 7(a) shows some decoded images along these principal directions.
These components seem to clearly disentangle visual attributes/factors such as wearing a hat, changing hair color and wearing glasses.
More examples can be found in Appendix A.6.
The results are consistent with the property of
MCR 2 that promotes diversity of the learned features .
Fig. 7(b) shows interpolating features between pairs of training image samples, where for two training images x 1 and x 2 , we reconstruct based on their interpolated feature representations by ˆ x =
g
( αf ( x 1 )+(1 − α ) f
( x 2 )) , α ∈ [0 , 1] .
The decoded images show continuous morphing from one sample to another in terms of visual characteristics, as opposed to merely a superposition of the two images.
[Figure] [Figure]
Encoded features for classiﬁcation.
Notice that not only is the learned decoder good for generative purpose, but also the encoder is good for discriminative tasks.
In this experiment, we evaluate the discriminativeness of the learned LDR model by testing how well the encoded features can help classify the images.
We use features of the training images to compute the learned subspaces for all classes, then classify features of the test images based on a simple nearest subspace classiﬁer.
Results in Table 3 show that our model gives competitive classiﬁcation accuracy on MNIST, compared to some of best VAE-based methods.
We also tested the classiﬁcation on CIFAR-10, the accuracy is currently about 80 .
As expected, the representation learned with the multi-class objective is very discriminative and good for classiﬁcation tasks.
This demonstrates the learned LDR model is not only generative but also discriminative
[Table]
3. C ONCLUSION
This work provides a novel formulation for learning a both generative and discriminative representation for multi-class multi-dimensional real-world data.
We have provided compelling empirical evidence that the distribution of most data can be effectively mapped to an LDR, a union of incoherent subspaces.
The main purpose of this paper is to demonstrate the conceptual simplicity and practical potential of this new representation learning framework, instead of to strive for state of the art performance.
Nevertheless, with our preliminary implementation, LDR can be effectively learned for a variety of real-world datasets, from small to large, from domain-speciﬁc to diverse.
In addition, the so-learned decoder g already enjoys the beneﬁt of GAN for its good generative visual quality and the encoder f with the beneﬁt of AE for its discriminative property.
From our experience, the rate reduction based objective can be stably optimized across a wide range of datasets and network architectures without any additional regularizations or engineering tricks.
One may notice that there are many ways our method can be signiﬁcantly improved.
For one, in this work, we have simply adopted networks that were designed for GAN or AE, but they may not be optimal for the rate reduction type objectives.
Also notice that compared to GAN and AE, our method leads to an explicit model for the feature distribution: a mixture of incoherent subspace Gaussians.
Such an explicit model has the potential of making many subsequent tasks much easier and better: importance sampling for decoding, classiﬁcation, or even incremental learning.
We leave all these directions, together with all the open mathematical problems, for future investigation.
E THICS S TATEMENT
All authors agree and will adhere to the conference’s Code of Ethics.
We do not anticipate any potential ethics issues regarding the research conducted in this work.
R EPRODUCIBILITY S TATEMENT
Settings and implementation details of network architectures, optimization methods, and some common hyper-parameters are described in the Appendix A.1.
More detailed experimental settings for different datasets can be found in their corresponding Appendix section entitled with the name of the dataset.
We will also make our source code available upon request by the reviewers or the area chairs.
A A PPENDIX A. E XPERIMENT S ETTINGS AND I MPLEMENTATION D ETAILS
Network backbones.
For MNIST, we use the standard CNN models in Table 4 and Table 5, following the DCGAN architecture (Radford et al., 2015).
We resize the MNIST image resolution from 28 × 28 to 32 × 32 to ﬁt DCGAN architecture.
All α in lReLU (lReLU is short for Leaky-ReLU) of the encoder are set to 0.2.
We adopted conv ResNet architectures for CIFAR-10 in Tables 6 and 7, and STL-10 shown in Tablesand 9.
Each ResBlock up is same as Resnet, but add an up-sampler after the ﬁrst conv layer.
All batch normalization layers of ResBlock in encoder are replaced with spectral normalization layer.
Finally, we use the same architecture for CelebA, LSUN-bedroom, ImageNet-128 (see Tables 10 and 11) as all three datasets have the same 128 × 128 resolution.
Again, each ResBlock up is same as
Resnet, but add an up-sampler after the ﬁrst conv layer.
And all batch normalization layers in encoder are replaced with spectral normalization layer.
All experiments utilize this lightweight PyTorch library that provides implementations of popular state-of-the-art GANs and evaluation metrics.
[Table] [Table]
[Table] [Table]
[Table] [Table]
Training details.
Across all of our experiments
[Table]
we use Adam (Kingma & Ba, 2014) as our optimizer for all experiments
[Table]
with hyperparameters β 1 = 0 , β 2 = 0 . 9 .
The initial value of learning rate is 0.00015 and is scheduled with linear decay.
We choose (cid:15) 2 = 0 .
5 for both equation 11 and 12 in all
LDR experiments.
For all LDR-Multi experiments on ImageNet, we only choose 10 classes.
The details of the 10 classes as shown in Table 12.
Most experiments are trained on RTX 3090ti GPUs.
[Table] [Table]
[Table] [Table]
A. MNIST
Settings.
On MNIST dataset
[Table]
we train our model using DCGAN (Radford et al., 2015) architecture with our proposed models LDR-Multi (11) and LDR-Binary (12).
We set the learning rate to 10 − 4 , batch size to
[Table]
and training 15,000 iterations.
Due to the advantages of the LDR objective, we can achieve between-class discriminative representations while the within-class diversity of these representations can be preserved, which are shown in the following experimental results.
More results illustrating auto-encoding.
Here we give more reconstruction results, or ˆ X , of our
LDR-Multi and LDR-Binary models, compared to their corresponding original input X .
As shown in the Fig.8, for the LDR-Binary model, it can generate clean digit-like images but the decoded ˆ X might resemble digits from similar but different classes to the input data X since LDR-Binary tends to only align the distribution of all digits.
In contrast, with the LDR-Multi objective, the decoded ˆ X not only are coherent with the correct class with the input data X , but also show very clear one-to-one mapping between individual sample x and ˆ x although the objective 11 does not enforce that!
Comparing with the results from the VAE-GAN (Larsen et al., 2015), our decoded images preserve much better the individual characteristics of the original samples.
Images decoded from random samples on the learned multi-class LDR.
Since our LDR-Multi objective function maps input data of each class into a different (orthogonal) subspace in the feature space, we can generate images conditioned on each class by random sampling z in the subspace of each class and then decode them back to the input space as ˆ x .
To do random sampling in the learned subspace, we ﬁrst calculate the mean feature ¯ z j and the singular vectors v ij of the SVD (or principal components) of the learned features Z j of the class j of the training data, where index i represents the i th principal components.
We only use top r = 8 principal
[Figure]
components of each class on MNIST dataset.
These statistics of the subspace can be used for guiding the random sampling.
Then we sample z randomly along the principal components and around the mean feature as
[Equation]
where ¯ z j is the mean feature of class j , σ ij and v ij are the i -th singular value and principal component of class j , n i are i.i.d.
Gaussian N (0 , 1) random variables.
That is, the feature in each subspace/class is modeled by a r -dimensional multivariate Gaussian, with variances σ ij which characterize variances of the training data in the feature space.
Here, α is a hyper-parameter that controls the sampling range.
As for visualization of random generated images g ( z random j ) conditioned on the given class, we compare our method with some other conditional generation method such as ACGAN (Odena et al., 2017) and InfoGAN (Chen et al., 2016)
(For ACGAN and InfoGAN, we generate images conditioned on class labels with randomly sampled latent z according the procedures mentioned in their respective works).
Our model can give realistic and correct conditional generation results with high diversity in each class, while other methods may make mistakes in the generation between some similar classes such as classes 3 and 5 for InfoGAN.
[Figure]
A. T RANSFORMED MNIST
Settings.
In this experiment, we verify that our LDR-Multi model can preserve diverse data modes in the learned feature embeddings.
We construct a transformed MNIST dataset with 5 modes: normal, large(1.5 × ), small(0.5 × ), rotate 45 ◦ left, and rotate
Auto-encoding results.
Figure 10(b) gives the decoded results of the training data with different modes.
Even though the data are now much more diverse for each class, decoder learned from the
LDR-Multi objective can still achieve high sample-wise similarity to the original images.
[Figure]
Identifying different modes.
Similar to the earlier experiments of Fig.
6 for CIFAR-10 in the main paper, we ﬁnd the top principal components of features of each class Z j (via SVD) and generate new images using the learned decoder g from features of the training images aligned the best with these components.
In Fig 11, we select three classes 0, 1, 2 and visualize samples from top r = 8 principal components for each class.
Each row represents one principal components direction.
As it can be seen in the ﬁgure, the decoded images along each principal component shows similar mode and the modes along different component directions are rather incoherent.
All major modes of the original data can be identiﬁed as one of these principal directions.
This clearly shows that our LDR-Multi model can keep the different modes within each class of the data X j as the principal component directions of Z j , and these modes can also be retained in the decoded images ˆ X j .
[Figure]
. A. CIFAR-
Settings.
For all experiments on CIFAR-10, we follow the common training hyper-parameters in section A.1.
Beyond that, for each experiment, we run 450,000 iterations with mini-batch size 1600.
Images decoded from random samples on the learned multi-class LDR.
We sample z in the feature space randomly along the principal components and around the mean feature of each class Z j as in the MNIST case, according to equation (13).
The generated images from the sampled features are illustrated in Fig 12, one row per class.
As we see, the generator learned from the LDR-multi objective is capable of generating diverse images for each class.
Further, for visualization of random generated images g ( z random j ) conditioned on the given class, we also compare our method with some other conditional generation method such as ACGAN (Odena et al., 2017) and InfoGAN (Chen et al., 2016).
For all three experiments, we have randomly sampled 8 images per class in CIFAR-10.
For more complex dataset like CIFAR-10, our model can give more realistic conditional generation results for different classes with high diversity within each class.
[Figure]
Generating image along different PCA components for each class.
For each class, we ﬁrst compute top-10 principal components (singular vectors of the SVD) of Z and then for each of the top singular vectors, we display in each row the top-10 reconstructed image ˆ X whose Z are closest to the singular vector using method described in the main body of the paper, Section 3.3.
The results are given in Figure 13 below.
A. STL-
Settings.
For all experiments on STL-10, we follow the common training hyper-parameters in section A.1.
For LDR-Binary setting, we train 150,000 iterations.
For LDR-Multi setting, we initialize the weights from the 20,000-th iteration of LDR-Binary checkpoint and train for another 80,000 iterations (with the LDR-Multi objective). The IS and FID scores on the STL-10 dataset are reported in Table 13, on par or even better than existing methods such as SNGAN (Miyato et al., 2018) or DC-VAE (Parmar et al., 2021). Visualizing auto-encoding property for LDR-Binary. We visualize the original images x and their decoded ˆ x using the LDR model learned from LDR-Binary objective.
The results are shown in Figure 14 for STL-10.
A. C ELEB -A AND LSUN
To verify that our formulation works on images of higher-resolution, we conduct experiments on the
Celeb-A and LSUN datasets, which have a resolution of 128 × 128 .
Settings.
For all experiments on these datasets, we follow the common training hyper-parameters in Section A.1.
We choose a 300 mini-batch size for Celeb-A and LSUN.
Both of them are trained with the LDR-Binary formulation, and for 450,000 iterations. [Figure] [Figure] [Figure]
Generating image along different PCA components.
We calculate the principal components of the learned features Z in the latent subspace.
We manually choose 3 principle components which are related to hat, hair color, and glasses (see Fig 15).
The three components are 9-th, 19-th, and 23-th respectively from the overall 128 principal components.
These principal directions seem to clearly disentangle visual attributes/factors such as wearing hat, changing hair color, and wearing glasses.
Images generated from random sampling of the feature space.
We sample z randomly according to the following Gaussian model:
[Equation]
where ¯ z is the mean feature, σ i and v i are the i th singular value and singular vector, n i are i.i.d.
Gaussian N (0 , 1) random variables.
As before α is a hyper-parameter to control the sampling range.
We use top r=100 principle components for random sampling.
The random generated images are realistic and diverse.
(see Fig 16)
Visualizing auto-encoding property for LDR-Binary.
We visualize the original image x and their decoded ˆ x using the LDR model learned from LDR-Binary formulation.
The results are shown in
Fig 17 and Fig 18 for the Celeb-A dataset and the LSUN dataset, respectively.
The LDR-Binary formulation can give very good visual quality for ˆ x but cannot ensure sample to sample alignment.
Nevertheless, the decoded ˆ x seems to be similar in style to the original x .
We believe it manages to align only the dominant principal component(s) associated with the most salient visual attributes, say pose of the face for Celeb-A or layout of the room for LSUN, between features of X and ˆ X .
A. I MAGE N ET
Settings.
To verify that our formulation works on large-scale datasets, we train a model on the entire
ImageNet.
For all experiments on the ImageNet, we follow the common training hyper-parameters in
Section A.1.
We ﬁrst train our model with LDR-Binary (12) with a mini-batch size 1800 on the whole ImageNet
ILSVRC 2012 dataset.
The number of training iterations is 450,000. [Figure] [Figure]
[Table] After that [Table] we ﬁne-tune the Binary-pretrained model with LDR-Multi (11), on 10 selected classes.
Information about the 10 classes can be found in Table 12.
The ﬁne-tune mini-batch size is [Table] and we train another 35,000 iterations.
This experiment took 8 A100-SXM4 GPUs, each with 40GB of
CUDA memory for 120 GPU hours.
Note that our choice of mini-batch size is substantially larger than those commonly adopted in other works while training on the ImageNet (e [Table] g. 128 in Miyato et al.
We empirically observe that training with a larger mini-batch generates images of better quality and clearer class alignment.
This is consistent with the proposed LDR framework as the LDR-Multi objective explicitly encourages alignment of class distributions, therefore beneﬁting from a larger batch that better captures overall data distributions.
We leave a more rigorous study of the effect of batch size for future work.
Due to the heavy computation of such large batch size, we present the following intermediate results obtained at 35,000 iterations whereas most existing methods run with magnitude larger number of iterations.
Nevertheless, these intermediate results already verify the efﬁcacy of our framework.
In addition, we present the full version of the comparison with existing generative methods in Table 13.
We see the SI and FID scores for LDR-Multi degraded a little after the ﬁnetuning.
This is expected as learning a more reﬁned separation and alignment of 10 classes is a more challenging task thanclasses.
This is consistently observed from experiments on other datasets too.
Visualizing feature similarity for LDR-Multi.
We visualize the cosine similarity among features
Z of different classes learned from the LDR-Multi objective in Figure 19.
In addition, we provide the visualization of alignment between features Z and decoded features features ˆ Z .
These results demonstrate that not only the encoder has already learnt to discriminate between classes, Z and ˆ Z also are aligned clearly within each class.
Visualizing auto-encoding property for LDR-Multi.
We visualize the original image x and their decoded ˆ x using the LDR model ﬁne-tuned using LDR-Multi formulation.
The results are shown in
Fig 20 for the selected 10 classes in ImageNet.
The LDR-Multi formulation can give good visual quality for ˆ x as well as decent sample-to-sample alignment.
A. A BLATION S TUDY ON O BJECTIVE F UNCTIONS
To empirically validate the importance of the rate reduction ( ∆ R ) loss, we keep the closed-loop formulation but replace all rate reduction ( ∆ R ) loss terms in (11) with corresponding cross-entropy loss.
To replace the rate reduction ( ∆ R ) terms in the objective function (11) with cross-entropy, we introduce a linear mapping W ∈ R d × k to map Z ∈ R d × n from feature space to logits γ = Z (cid:62) W .
We then calculate the softmax cross-entropy function on logits γ and one hot label matrix Y .
Here H ( γ, Y ) = [Equation]
log e γij (cid:80) kj =1 e γ ij is the formulation of softmax cross-entropy function and
[Figure]
[Figure]
Y ∈ R n × k is one hot label matrix.
Then, we can replace the ﬁrst two terms of (11) ( ∆ R (cid:0) Z (cid:1) and
∆ R (cid:0) ˆ Z (cid:1) ) with H ( Z (cid:62) W , Y ) and H ( ˆ Z (cid:62) W , Y ) .
For the third term of (11), we extract j -th class one hot feature γ j = Z (cid:62) j W , ˆ γ j = ˆ Z (cid:62) j W from Z and ˆ Z , and deﬁne the distance D ( γ j , ˆ γ j ) =
e γj e γj + e ˆ γj of them.
For the third term of (11), we further introduce k linear layers as discriminators {D j } kj =1 for each class.
Then, we replace the third term with the GAN’s objective function as (cid:80) k
j =1 E [log D j ( Z j )]
+ E [log(1 − D j
( ˆ Z j ))] 11 .
Now, we have the cross-entropy version objective function (15) for closed loop framework.
We denote the closed loop framework with cross-entropy as
Closed-loop-CE.
[Figure]
[Equation]
We run the experiments on MNIST and CIFAR10.
The architectures of MNIST and CIFAR10 are given in Table 4 to Table 7 12 .
Results on MNIST.
The training hyper-parameters of LDR-Multi and Closed-loop-CE on MNIST are following Section A.1.
Comparisons between LDR-Multi and Closed-loop-CE are listed in
Figure 21, 22, and 23.
Figure 21(b) and 21(c) show the reconstructed images ˆ X from Closed-loop-CE and LDR-Multi.
Both methods can give sample-wise reconstruction results due to the transcription framework.
However, comparing training images whose features are best aligned with the principal components of class ‘2’ in Figure 22, we see that the principal components of CE features do not correspond to consistent visual attributes of the images, whereas ours do.
From the heatmaps in Figure 23(a) and 23(b), we see the features learned by rate reduction possess clear orthogonal subspace structures, whereas those learned by Closed-loop-CE do not.
Moreover,
Figure 23(c) and 23(d) show that the learned features of LDR-Multi have higher singular values for the top principal components of each class, corresponding to a more linearized and diverse feature distribution, whereas those by Closed-loop-CE do not.
Failed Attempts on CIFAR-10 with Cross Entropy.
The training hyper-parameters of Closed-loopCE on CIFAR10 follow Section A.1.
We do the grid search on three hyper-parameters: learning rate { 1 . 5 × 10 − 2 , 1 . 5 × 10 − 3 , 1 . 5 × 10 − 4 } , mini batch size (800 or 1600), and inner loop (1,2,3,4), conduct 24 experiments in total.
All cases of the Closed-loop-CE fail to converge or experience model collapse on the CIFAR-10 dataset.
[Figure]
[Figure]