Paper: $t^3$-Variational Autoencoder: Learning Heavy-tailed Data with Student's t and Power Divergence (ICLR-2024)

Reviewer comment (Experiments):
The authors claim that their idea is extendable to hierarchical models. Although there is a theoretical justification for this in the appendix, the paper would benefit from corresponding experimental studies.

Author response:
We have newly implemented the two-layer hierarchical model ($t^3$HVAE) and compared against the Gaussian HVAE on double resolution images. The results are presented in Section 4.2, Figure 5 and Table C3. We demonstrate that the increased hierarchical depth allows $t^3$HVAE to learn more sophisticated images with substantially higher clarity and sharper detail compared to Gaussian HVAE, further justifying the generality and effectiveness of our theoretical framework.

Reviewer comment (Experiments):
Have you considered providing experimental results for hierarchical architectures?

Author response:
We have newly implemented the two-layer hierarchical model ($t^3$HVAE) and compared against the Gaussian HVAE on double resolution images. The results are presented in Section 4.2, Figure 5 and Table C3. We demonstrate that the increased hierarchical depth allows $t^3$HVAE to learn more sophisticated images with substantially higher clarity and sharper detail compared to Gaussian HVAE, further justifying the generality and effectiveness of our theoretical framework.

Reviewer comment (Theory):
The transition from Eq(5) to Eq(19) is not straightforward for me. While intuitively, adopting the γ-power divergence to the KL-divergence might seem appropriate, a central question arises: Does the γ-power divergence between the two manifolds still lead to the ELBO? This needs to be addressed for better clarity.

Author response:
Thank you for pointing out the issue. We agree this is an important distinction that needs to be clarified, and the following discussion has been added to the manuscript in Section 3.2.

Due to the different expression for $\gamma$-power divergence and also the approximation step, the $\gamma$-loss does not function as a bound for log-likelihood, although joint minimization will still generally try to fit $p_\theta(x)$ to the true data distribution, maximizing likelihood while also keeping posteriors close. Its precise effect is a balance between reconstruction and regularization, which is explained throughout Section 3.3.

We emphasize that the central philosophy of our framework is viewing the ELBO *not* as a lower bound of likelihood (which naturally leads to modifying the KL regularizer term as most VAE variants do), but as a divergence between joint distributions (which naturally leads to modifying the entire divergence as in Eq.19). The $\gamma$-power or log-$\gamma$ divergences are reasonable choices due to their information geometric properties and viability of closed-form computations with respect to t-distributions.

But why does this work? Our experiments demonstrated that our framework is very effective even though we are not explicitly optimizing for likelihood. This can be explained in part by [1] which shows that $\gamma$-type divergence minimizers are M-estimators and proves asymptotic efficiency and robustness under heavy contamination, lending support to our approach as a solid alternative to MLE methods for statistical inference. Furthermore, for high-dimensional datasets $\gamma$ is quite small and the effect of changing divergence is simply less important compared to the effect of the coupled t-distributions (in particular the prior) whose degrees of freedom $\nu$ is small.

Reviewer comment (Reproducibility):
Incorporating the Student's t-distributions inherently increases the model's complexity. This can potentially introduce challenges in training and optimization. I recommend the authors delve into these potential challenges, discussing potential remedies and considerations for practitioners aiming to implement t3VAE.

Author response:
Unlike models such as DE-VAE or VAE-st which require numerical integration to calculate the KL divergence between t-distributions for every data point, the explicit form of the $\gamma$-loss does not create any significant computational bottlenecks or new instability issues (e.g. division by zero variance, exploding gradients) compared to the original ELBO of Gaussian VAE. The same holds for the t-based reparametrization trick or multivariate t-distribution sampling process as detailed in Appendix C.1, requiring very minor additional computation. We also observed virtually equal runtimes in practice. Nonetheless we agree that this discussion is needed in the paper, and have merged into the final paragraph in Section 4 to give a general analysis on training cost and hyperparameter selection.

Reviewer comment (Experiments):
No examples conducted on datasets where heavy tails are known to be especially relevant (e.g. economic datasets).

Author response:
Due to time constraints, we had to focus our efforts into implementing the hierarchical $t^3$VAE to learn very high dimensional images (experiments added to Section 4.2, Figure 5 and Table C3). We note that many non-image datasets that are considered heavy-tailed (economic, weather, etc) are low dimensional, while $t^3$VAE exhibits the best performance improvements over models such as Student-t VAE in higher dimensions. Nonetheless, we will continue to experiment on other datasets for the camera-ready version.

Reviewer comment (Novelty):
No discussion of, or comparisons to, similar developments in the normalizing flow literature (e.g. [1] and [2]).

Author response:
Works such as [4,5,6] implement heavy-tailed base distributions for normalizing flows, which is conceptually similar to using a heavy-tailed prior for VAEs. Of course in principle, any data distribution can in principle be approximated arbitrarily closely if the flow/decoder networks are powerful enough. However, these papers argue intuitively and empirically that t-distributions ensure better robustness and generalization, and we believe the same arguments carry over to our model.

We also take a step further (compared to prior-based alternatives) by also using t-distributions for the decoder and encoder, which enforces the power form Eq.(15) on the joint model distribution. If the decoder network $\mu_\theta$ is Lipschitz, this will explicitly ensure polynomial decay w.r.t. $x,z$. In fact, this control 'further down the line' seems to be key to $t^3$VAE's success as evidenced by our experiments against Student-t VAE, DE-VAE and VAE-st. This discussion has been added to the Related Works section.

Moreover, this raises an intriguing question post factum: can the tails of distributions generated by normalizing flows also be more strictly controlled down the line (not just modifying the base distributions) to improve performance? We leave this as an interesting direction for future work.

Reviewer comment (Reproducibility):
Can you provide an algorithm environment to show how this should be implemented at a glance? I appreciate the presentation and the derivation of the model, but any reader looking to quickly implement it is likely to have trouble if they do not thoroughly read the paper.

Author response:
Thank you for the helpful suggestion! We have added an implementation summary of our framework as Algorithm 1 in the main text.

Reviewer comment (Evaluation):
What do the MMD values themselves look like? These might be more useful to report than the p-values.

Author response:
When the RBF kernel $k(x,x') = \exp\left(-\frac{(x-x')^2}{2\sigma^2}\right)$ is implemented for the MMD bootstrap test, the hyperparameter $\sigma^2$ is automatically chosen as the median of $\{(x_i - x_j')^2\}_{i,j}$. In practice, $\sigma^2$ fluctuates between around 3 to 5. In this case, directly comparing MMD values may not be meaningful. The MMD values are also somewhat randomly distributed around -0.01~0.01 and not particularly informative on their own. Hence a bootstrap test must be run to obtain meaningful statistics, which is why we only report $p$-values.

Reviewer comment (Experiments):
Have you tried the model on any real datasets other than image sets?

Author response:
Due to time constraints, we had to focus our efforts into implementing the hierarchical $t^3$VAE to learn very high dimensional images (experiments added to Section 4.2, Figure 5 and Table C3). We note that many non-image datasets that are considered heavy-tailed (economic, weather, etc) are low dimensional, while $t^3$VAE exhibits the best performance improvements over models such as Student-t VAE in higher dimensions. Nonetheless, we will continue to experiment on other datasets for the camera-ready version.