Paper: Compressing Latent Space via Least Volume (ICLR-2024)

Reviewer comment (Experiments):
Limited set of experiments; the authors mainly check reconstruction performance at a given sparsity level. As auto encoders are primarily used for their feature learning capabilities, it would be worthwhile to see how the features learned through LV fare in several downstream tasks.

Author response:
Thanks for the suggestion. However, due to the time limit, we may not be able to include more comprehensive downstream tasks in our paper.

Reviewer comment (Evaluation):
In the experiment discussion, the authors mention that they use binary cross entropy for training, which deviates from their theory, but they perform evaluation with the L2 reconstruction error. Why is there this discrepancy? Furthermore, how is the binary cross-entropy used when you do not have binary data (e.g., CIFAR 10 and CelebA)?

Author response:
These images' pixel values are all normalized from [0,255] to [0,1], thus BCE can be employed as a reconstruction loss. BCE is not limited to the discrete binary classification case because gradient-wise for any label $y\in[0,1]$, we have $\frac{\partial BCE(p, y)}{\partial p} = 0 \Leftrightarrow p = y$, so minimizing it is equivalent to minimizing the other norm-based reconstruction losses like MSE and L2. 
We switched to BCE for the image autoencoder because it makes the AE converge much faster than MSE, probably because it assigns much higher gradient to wrongly reconstructed pixels.

Reviewer comment (Novelty):
Although the method is novel, the idea of minimizing the volume of the learned latent representations is not, see [PCA-AE](https://link.springer.com/article/10.1007/s10851-022-01077-z), [IRMAE](https://ar5iv.labs.arxiv.org/html/2010.00679). LV significantly differs from the mentioned papers, as it directly acts on the latent codes and does not require multistage training. Nonetheless, they should be mentioned.

Author response:
Thanks for mentioning these great works. We actually cited IRMAE in our paper, and we will add PCA-AE in our revision.

Reviewer comment (Novelty):
The authors mention [Sparse Feature Learning for Deep Belief Networks](https://papers.nips.cc/paper_files/paper/2007/hash/c60d060b946d6dd6145dcbad5c4ccf6f-Abstract.html), but do little to point out the similarities/differences to their method. The regularization term of the latent codes in that work is  
$L_\text{vol} = \sum\limits_{i=1}^{m} \log(1 + h^2(z_i))$,  
where $h$ is defined in Eq. 6 of the paper (notation was adjusted to fit the notation of the paper under review).  
When we regard $h$ as a homeomorphism that just produces new latent codes and center those new latent codes to 0 we get  
$\sqrt[m]{\exp(L_\text{vol})} = \sqrt[m]{\prod\limits_{i=1}^{m} (\sigma_i + 1)}$.  
This is equivalent to LV with $\eta = 1$. If the authors can point out what their contribution with regards to that already published regularization term is, it would in my opinion severely strengthen the case for this work.

Author response:
The student-t regularizer is not equivalent to the volume regularizer. First of all, for a mini-batch based training involving the student-t regularizer, the average over the batch samples is taken outside the regularizer term, i.e., $\frac{1}{n}\sum_j^n \sum_i^m \log(1+h^2(z_i^{(j)})) \neq \sum_i^m \log(1+ \frac{1}{n}\sum_j^nh^2(z_i^{(j)}))$. Second, even if the average is taken inside the term, the equation should instead be $\sqrt[m]{\exp(L_{\text{vol}})} = \sqrt[m]{\prod_i^m (\sigma_i^2+1)}$. In other words, the student-t regularizor is like minimizing the square of the volume, i.e., taking the variance rather than the STD as the side length of the cuboid. Just like the difference between L2 and L1, this makes it more inclined to reduce larger $\sigma_i$, which leads to the discrepancy between “vol” and “st” in Fig. 2.

Reviewer comment (Novelty):
In the related work section the authors describe the invariance of their method to translation of the latent codes. However in the next sentence it is stated that: "This equivalent latent set is then one that has zero-STDs along many latent dimensions, which is what this work tries to construct". It is not clear whether this refers to the translated latent set of LV or of aforementioned methods. In case it refers to LV, the zero-STDs are present also before translation (so it has *still* zero-STDs not only *then*). In case it refers to other methods the authors need to elaborate more on the claim that established methods already produce close to zero-STD latent codes in many latent-dimensions and how their method then differs from those established methods apart from allowing for translational invariance, which would not be a strong contribution by itself. This would also generally be a good addition to the related works section.

Author response:
Sorry for the confusion. It refers to the translated latent set of LV. What we are trying to convey here is that **in terms of the latent representation’s sparsity,** **the *absolute value* of a latent code dimension $z_i$ actually does not matter**, as what really matters is its STD $\sigma_i$. For a given latent dimension $z_i$, if it stays at 0 across all the data samples (thus making the latent code vector sparse in the traditional sense and leading to some improvement in downstream tasks), then the latent code vector should be equally sparse and good if this $z_i$ stays at any constant $c$ other than 0, because it can be easily made back to 0 with a simple translation that any complex neural network can learn. In other words, the sparsity of a latent code $\mathbf{z}$ should not be determined by how many $z_i = 0$ it has, but by how many $\sigma_i = 0$ it has.  
Our LV method is focused on the latter generalized definition of sparsity, and encourages sparsity this way. A sparse latent dimension obtained by LV does not necessarily have its mean equal to 0, but can be easily turned into a traditionally sparse dimension with a simple translation.

Reviewer comment (Theory):
In section B.3.3 in the appendix the authors claim that latent STD still correlates well with explained reconstruction if no Lipschitz constraint is applied because the decoder is naturally a Lipschitz function. While this is correct the Lipschitz constant of the decoder can be arbitrarily increased. It is not clear whether the argument of small updates of gradient-based optimization preserving the Lipschitz constant holds after cumulative gradient updates and sufficient training time. The observed effect might well be due to specific training dynamics and model initialization and, while it is interesting to report, a clear conclusion is hard to draw.

Author response:
Yes, we are just providing our hypothesis about this phenomenon. We would like to clarify it more in our revision in case it induces confusion.

Reviewer comment (Evaluation):
The autoencoders are trained with BCE loss on image datasets, but are evaluated in terms of L2 reconstruction. There is no justification for evaluating the trained models on a different metric than the training target. If there is a reason behind this the authors should clarify it.

Author response:
These images' pixel values are all normalized from [0,255] to [0,1], thus BCE can be employed as a reconstruction loss. BCE is not limited to the discrete binary classification case because gradient-wise for any label $y\in[0,1]$, we have $\frac{\partial BCE(p, y)}{\partial p} = 0 \Leftrightarrow p = y$, so minimizing it is equivalent to minimizing the other norm-based reconstruction losses like MSE and L2.  
The L2 loss is more like a metric in our theoretical framework for quantifying the accuracy of reconstruction. Just like we usually use CE as the loss function for classification problems, but we discuss the classification performance in terms of precision, recall, etc.  
We chose BCE as the loss function for image reconstruction because it leads to much faster convergence, reducing L2 error much faster than L2 itself and MSE.

Reviewer comment (Experiments):
To me, it seems like the main weakness is that the approach relies on several tuning parameters ($K$, $\eta$, $\lambda$) and it is unclear how to set these and how they interact with each other.

Author response:
We sincerely appreciate your comments and suggestions. In the following, we answer your questions point-by-point.

1. We did some sensitivity studies regarding $K, \eta, \lambda$ in Appendix B. In summary:
    1. $K$ is always first settled when constructing the decoder network and fixed afterwards, because the activation functions are fixed and the spectral normalization makes the linear layers 1-Lipschitz. 
    Its absolute value won’t affect LV’s dimension reduction ability, because if you can find a least-volume latent set $\mathcal{Z}$ with an encoder $e$ and a 1-Lipschitz decoder $g$, then for a new encoder $e'$ and a new $r$-Lipschitz decoder $g'$, you must be able to find a new least-volume $\mathcal{Z}'$ that is similar to $\mathcal{Z}$ with a scale factor of $1/r$ (i.e., $\mathcal{Z}' = \mathcal{Z}/r$), simply by making $g' = g \circ h$ and $e'= h^{-1}\circ e$, where $h(z') = r\cdot z'$. 
    In short, as long as it is bounded, you don’t need to worry about its value. 
    2. $\eta$ is studied in Sec. B.3.2 and Fig. B.6, which shows it indeed interpolates between $\sqrt[m]{\prod_i\sigma_i}$ and $\frac{1}{m}\|\sigma\|_1$. In practice, we almost always set it to 1, not only because it worked well enough in all cases we encountered, but also helps avoid gradient issue by lower bounding the denominator in Eqn. 4. 
    $K$ should have some interaction with $\eta$, because $K$ may scale up or down the latent set and thus scale $\sigma_i$, which changes the ratio between $\sigma_i$ and $\eta$.
    3. $\lambda$ is set after $K$ and $\eta$ are fixed. $K$ scales the latent set and may $K$scales its STDs $\sigma_i$$K$$\lambda$ much, because 1) $K$ and $\eta$ are fixed and we don’t adjust them frequently. 2) The scaling of $\sigma_i$ and $\eta$ should be cancelled out in Eqn. 4.

Reviewer comment (Experiments):
I miss experiments on the interaction between tuning parameters.

Author response:
We sincerely appreciate your comments and suggestions. In the following, we answer your questions point-by-point.

1. We did some sensitivity studies regarding $K, \eta, \lambda$ in Appendix B. In summary:
    1. $K$ is always first settled when constructing the decoder network and fixed afterwards, because the activation functions are fixed and the spectral normalization makes the linear layers 1-Lipschitz. 
    Its absolute value won’t affect LV’s dimension reduction ability, because if you can find a least-volume latent set $\mathcal{Z}$ with an encoder $e$ and a 1-Lipschitz decoder $g$, then for a new encoder $e'$ and a new $r$-Lipschitz decoder $g'$, you must be able to find a new least-volume $\mathcal{Z}'$ that is similar to $\mathcal{Z}$ with a scale factor of $1/r$ (i.e., $\mathcal{Z}' = \mathcal{Z}/r$), simply by making $g' = g \circ h$ and $e'= h^{-1}\circ e$, where $h(z') = r\cdot z'$. 
    In short, as long as it is bounded, you don’t need to worry about its value. 
    2. $\eta$ is studied in Sec. B.3.2 and Fig. B.6, which shows it indeed interpolates between $\sqrt[m]{\prod_i\sigma_i}$ and $\frac{1}{m}\|\sigma\|_1$. In practice, we almost always set it to 1, not only because it worked well enough in all cases we encountered, but also helps avoid gradient issue by lower bounding the denominator in Eqn. 4. 
    $K$ should have some interaction with $\eta$, because $K$ may scale up or down the latent set and thus scale $\sigma_i$, which changes the ratio between $\sigma_i$ and $\eta$.
    3. $\lambda$ is set after $K$ and $\eta$ are fixed. $K$ scales the latent set and may $K$scales its STDs $\sigma_i$$K$$\lambda$ much, because 1) $K$ and $\eta$ are fixed and we don’t adjust them frequently. 2) The scaling of $\sigma_i$ and $\eta$ should be cancelled out in Eqn. 4.

Reviewer comment (Reproducibility):
How do you tune the free parameters?

Author response:
We sincerely appreciate your comments and suggestions. In the following, we answer your questions point-by-point.

1. We did some sensitivity studies regarding $K, \eta, \lambda$ in Appendix B. In summary:
    1. $K$ is always first settled when constructing the decoder network and fixed afterwards, because the activation functions are fixed and the spectral normalization makes the linear layers 1-Lipschitz. 
    Its absolute value won’t affect LV’s dimension reduction ability, because if you can find a least-volume latent set $\mathcal{Z}$ with an encoder $e$ and a 1-Lipschitz decoder $g$, then for a new encoder $e'$ and a new $r$-Lipschitz decoder $g'$, you must be able to find a new least-volume $\mathcal{Z}'$ that is similar to $\mathcal{Z}$ with a scale factor of $1/r$ (i.e., $\mathcal{Z}' = \mathcal{Z}/r$), simply by making $g' = g \circ h$ and $e'= h^{-1}\circ e$, where $h(z') = r\cdot z'$. 
    In short, as long as it is bounded, you don’t need to worry about its value. 
    2. $\eta$ is studied in Sec. B.3.2 and Fig. B.6, which shows it indeed interpolates between $\sqrt[m]{\prod_i\sigma_i}$ and $\frac{1}{m}\|\sigma\|_1$. In practice, we almost always set it to 1, not only because it worked well enough in all cases we encountered, but also helps avoid gradient issue by lower bounding the denominator in Eqn. 4. 
    $K$ should have some interaction with $\eta$, because $K$ may scale up or down the latent set and thus scale $\sigma_i$, which changes the ratio between $\sigma_i$ and $\eta$.
    3. $\lambda$ is set after $K$ and $\eta$ are fixed. $K$ scales the latent set and may $K$scales its STDs $\sigma_i$$K$$\lambda$ much, because 1) $K$ and $\eta$ are fixed and we don’t adjust them frequently. 2) The scaling of $\sigma_i$ and $\eta$ should be cancelled out in Eqn. 4.

Reviewer comment (Writing):
On page 5, what is $\pi$? I couldn't figure it out from the context.

Author response:
2. $\pi: Z'\times c\to Z'$ is the projection map that strips off the dimensions correspond to $c$ and only keeps those correspond to $Z'$, i.e., removing all the latent dimensions that have constant values. As a toy example, $\pi: X_1\times X_2 \times X_3\to X_1\times X_2$ means for the vector $x = [4,5,6]$, we get $\pi(x)=[4,5]$. A quick reference can be found here: [https://en.wikipedia.org/wiki/Projection_(set_theory)](https://en.wikipedia.org/wiki/Projection_(set_theory)?oldformat=true)

Reviewer comment (Experiments):
The error bars in fig 2 are quite small. Does that imply that things are stable wrt choice of $\lambda$ or that the studied $\lambda$-range was too small?

Author response:
3. Yes, this means it is stable w.r.t. $\lambda$. We don’t think the range of $\lambda$ is small, because as you can see in Fig. B.4 in Sec. B.3.1, the largest $\lambda$ in each case is large enough to distort the reconstruction significantly.

Reviewer comment (Evaluation):
If the data sets are composed of distinct clusters, would this least volume regularization provide meaningful cluster structures?

Author response:
Yes. This is due to the homeomorphism that a well-trained autoencoder can establish between the dataset and the latent set (Theorem 2). If the dataset consists of disconnected components, then because the connectedness of a set is invariant under homeomorphism, these components will remain disconnected and form clusters in the latent space. The least volume regularization is just searching for a lower dimensional latent set among all the homeomorphic latent sets that an autoencoder can establish, without destroying that homeomorphism.

Reviewer comment (Evaluation):
Can this method be used to estimate the intrinsic dimensionality of a data set? How are the obtained latent set dimensions close to the exact data set dimensions (if known)?

Author response:
This depends on your definition of intrinsic dimensionality (ID). If you assume the dataset structure to be *locally Euclidean* of certain dimension and define the ID as that *local* dimension, then least volume cannot always retrieve that local ID, for instance in the following complex scenarios :  
    1. The dataset is the union of several manifolds of different local IDs.  
    2. The dataset is a manifold that cannot be embedded in a Euclidean space of the same dimension, e.g., sphere, Klein bottle.  
    3. The dataset is not a manifold, at least not locally Euclidean somewhere, e.g., the union of several *intersecting* manifolds.  
    4. The data manifold forms a non-trivial knot in the data space.  
      
    In these scenarios, traditional methods based on neighborhood graph (e.g. isomap) are probably better choices, though they may also have difficulty handling them. However, this doesn’t mean least volume is worthless, because what it tries to retrieve is the *global* ID, i.e., the dimensionality of the lowest dimensional Euclidean space that the whole dataset can be topologically embedded in, whereas the local methods are not designed to obtain this. Thus, LV is a *global, bidirectional* complement to those *local* methods. We found it generally works well in retrieving the global ID.

Reviewer comment (Experiments):
How do the overall outcomes, such as latent set dimensions, reconstruction losses, and the correlation between explained reconstruction and the standard deviations of latent dimensions, change when the dimension of the encoding latent space is altered?

Author response:
We didn’t tested this meticulously, but we didn’t experience any noticeable difference in the result when we double the latent dimensions.

Reviewer comment (Reproducibility):
Is the regularizer computed on minibatch? If so, how is the minibatch estimate reliable? How is this method scalable?

Author response:
Yes, it is evaluated on minibatch. We found that in general a larger batch size (≥64) helps stabilize the training more than a smaller one (≤32), probably because it provides a better estimation of the STD. So we set the batch size to 100. However, typical small batch size like 32 still works.

Reviewer comment (Experiments):
Is this regularization method only effective in the autoencoder setting? How can this regularization method be extended to other applications?

Author response:
It is not restricted to autoencoders. For instance, we can replace the decoder with a regressor and replace the reconstruction loss with the regression loss. In this case, the latent set is still forced to preserve information and thus won’t be compressed trivially. However, such use cases are theoretically more complicated than the autoencoding case and deserves a separate work. Specifically, for the current autoencoding case, we can readily resort to the theories about homeomorphism in general topology. For the general use case, though, we think it pertains to quotient map and quotient topology, which are more intricate topics. For now, to extend LV to the other applications, we can disengage the encoder and decoder and use them separately for different tasks, such as clustering and data generation.

Reviewer comment (Writing):
$\mathcal{H}$ is not defined in Section 2.3.

Author response:
R1(Minor Comments): $\mathcal{H}$ is the “a given set of $h$”, i.e., the set of $h$ that $g$ and $e$ have enough complexity to represent. We will rephrase it to make it clearer.

Reviewer comment (Presentation):
On page 19, Table B.8. -> Table B.6.

Author response:
R3(Minor Comments): Thanks for pointing this out. We will correct it.