Paper: Incorporating Domain Knowledge in VAE Learning via Exponential Dissimilarity-Dispersion Family (ICLR-2024)

Reviewer comment (Novelty):
The general derivation is quite similar to $\sigma$-VAE, except for the extension of MSE to different dissimilarity functions.

Author response:
The σ-VAE proposes an approach to resolve the balancing and tuning issues faced by β-VAE and indeed shares a similar concept with our paper. While the σ-VAE focuses on calibrating β in the decoder, our proposed method significantly differs by using dissimilarity functions to extend the functionality of VAEs themselves, expanding their theoretical framework while maintaining probabilistic consistency. Additionally, our method allows for the automatic adjustment of scale parameters, eliminating the need for manual tuning of the decoder's variance, unlike σ-VAE. Moreover, experiments have confirmed that our proposed method demonstrates equal or superior performance to σ-VAE. We believe these novelties represent a significant advancement in the extensibility of VAEs beyond what σ-VAE offers.

Reviewer comment (Evaluation):
When choosing MSE as the similarity metric, why does there exist a performance difference between $\sigma$-VAE and the proposed method in Tab.3?

Author response:
This aspect is one of the key points where the advantages of our proposed method are most evident. As stated in Viewpoint (i) of the introduction, the focus of our paper is the necessity of introducing a family of distributions to utilize prior information in VAEs, which already have a theoretical foundation. While previous studies have attempted to introduce additional mechanisms to balance reconstruction and regularization errors (e.g., $\sigma$-VAE, Rybkin et al., 2021), these approaches often deviated from the probabilistic framework, posing a challenge. Thus, our paper aims to resolve these traditional issues by attempting to learn using domain knowledge in the form of dissimilarity functions while maintaining the probabilistic model framework inherent in VAEs. Specifically, we propose a unified approach that balances reconstruction and regularization through variational inference backed by a probabilistic understanding of distribution families, rather than resolving issues through additional mechanisms or regularization.
The experimental results in Table 3 show that the advantages of $\sigma$-VAE can be replicated in comparison with β-VAE, and these benefits are demonstrated. Furthermore, our proposed method has shown superiority over these techniques, particularly demonstrating the effectiveness of automatic tuning including the dispersion $\gamma$ in complex datasets such as Food-101 and CIFAR-10. This indicates that the proposed method, compared to $\sigma$-VAE, provides a more nuanced capture of data characteristics, which can lead to differences in performance.

Reviewer comment (Experiments):
In Tab.5, different dissimilarities $d$ are chosen based on the validation set for different datasets. How are the hyper-parameters tuned for the baselines? It seems that $\beta$-VAE with different $\beta$ in Tab.3 performs better than some baselines. Why not compare to $\beta$-VAE with a hyper-parameter tuned on the validation set?

Author response:
First, regarding Table 5, the hyperparameters for both the baseline and proposed methods were determined based on the FID from the validation set. The dissimilarity functions selected in our method are detailed in Figure 4. 
As for Table 3, rather than optimizing β or evaluating the highest quality resolution, we focused on validating how different values of β affect performance. This allows us to directly observe the effects of changes in β. We compare these performances to those of the automatically determined γ. We will add to the revised manuscript the points that were insufficiently explained.

Reviewer comment (Theory):
There is no theoretical guarantee or analysis to quantify the approximation. Simply using a 1-D toy model is not sufficient to validate the claim.

Author response:
The main manuscript carries out validations on the quantification of approximations in a simple 1-D setting. Additionally, validations regarding the validity of γ<<1 are performed in appendix G of the original manuscript.

Reviewer comment (Experiments):
It is not clear how the proposed model compares to other generative models (e.g., GAN). Though the goal of the paper is to improve VAE, it is important to compare to other generative models to understand its significance.

Author response:
Indeed, the experimental results in Table 3 do not always show our proposed method as superior. However, the technical contribution of this paper is not to achieve the highest precision in a specific task but to derive a theoretical framework for the introduction of any dissimilarity into VAEs. The focus of the experiments in Table 3 is to verify the validity of the newly proposed EDDF family of distributions and their estimation algorithm within the unified framework across various datasets. The fact that our proposed method, which features automatic adjustment of the scale parameter, exhibits performance comparable to the existing literature confirms the effectiveness of our theory. Meanwhile, as pointed out, discussions on the relationship and applicability to other generative models are necessary. Thus, the revised manuscript includes additions to this aspect.

Reviewer comment (Writing):
The term ‘domain knowledge’ used in the paper is not fitting the presentation well. How the domain knowledge relates to the proposed EDDF decoder is not well defined at all.

Author response:
In this paper, data characteristics are interpreted and discussed as distribution. An example of the relationship between distribution and EDDF decoder is shown in Table 1. The novelty of this paper is that it is possible to incorporate various data distribution assumptions in a unified framework based on probability theory. As you mentioned, the term "domain knowledge" is abstract and lacks direct explanation of the correspondence with the data distribution, so we have revised the manuscript to clarify.

Reviewer comment (Evaluation):
What is the computational cost of the proposed method?

Author response:
As originally described in the architecture details in the appendix, the computation cost is about 1h using a single GPU (RTX 2080Ti). Other parameters are also originally described in the appendix.

Reviewer comment (Evaluation):
Can you give some insights or explanations on why the proposed model performs better on some of the datasets but worse on others in Table 3?

Author response:
Using the visual domain as an example, this experiment comprehensively examines the dissimilarity functions proposed in the past, not limited to Neural Network-friendly similarity measures. Since the proposed method includes the aspect of gradient descent applicability as an implicit requirement, the applicability of the dissimilarity function may be quantified in terms of differentiability in the domain domain. Therefore, one drawback of the proposed method is that its performance is highly dependent on the Neural Network-Friendly dissimilarity function. The indicators of unstable performance are mainly those proposed before the rise of Deep Learning, and are considered to include operations that are not stochastic gradient descent-friendly. In such cases, it is considered necessary to introduce a more stable method of gradient calculation, as it may affect the stable learning of the proposed method. However, please note that neural network-based LPIPS and other methods are differentiable and thus provide stable learning. The above suggests that a differentiable data structure is required for the data domain, which is a limitation in the application of the proposed method.

Reviewer comment (Theory):
Can you give any justifications on how the proposed distribution family encodes domain knowledge?

Author response:
In this paper, we have demonstrated the effectiveness of the theory primarily through quantitative considerations. We believe that qualitative evaluation is a better way to understand the effectiveness of encoding domain knowledge. Figures 3 and 4 in the Appendix show the qualitative evaluation results corresponding to Tables 3 and 5. It can be seen that the proposed method is able to reflect the style and details of the input images better than other methods on relatively complex datasets such as Food-101 and CelebA. This indicates that the proposed method encodes knowledge important to the domain.

Reviewer comment (Novelty):
The paper's contribution is limited, as dissimilarity functions in the context of generative models have been presented in the past.

Author response:
The novelty of our proposed method lies not just in suggesting a new dissimilarity function but in providing a unified framework for employing such functions within VAEs across various fields, along with a methodology for the automatic estimation of scale parameters. We have established a new theory that incorporates domain knowledge into VAEs in a probabilistically consistent manner, demonstrating the potential to integrate insights from various domains into VAEs as functional biases in a unified way.
As VAEs are one of the representative generative models alongside GANs and diffusion models and are used in various fields and practical challenges, the application range of our proposed theory is expected to expand broadly. This paper constructs and validates the foundation of this new theory. Although the current contribution of this paper has not yet extended to direct applications in GANs or diffusion models, we believe it provides valuable insights for the deep learning and generative modeling communities.

Reviewer comment (Novelty):
The ELBO in (25) is probably the leading paper contribution, but beyond this point, I do not see any other contribution that justifies a whole paper around this idea.

Author response:
The novelty of our proposed method lies not just in suggesting a new dissimilarity function but in providing a unified framework for employing such functions within VAEs across various fields, along with a methodology for the automatic estimation of scale parameters. We have established a new theory that incorporates domain knowledge into VAEs in a probabilistically consistent manner, demonstrating the potential to integrate insights from various domains into VAEs as functional biases in a unified way.
As VAEs are one of the representative generative models alongside GANs and diffusion models and are used in various fields and practical challenges, the application range of our proposed theory is expected to expand broadly. This paper constructs and validates the foundation of this new theory. Although the current contribution of this paper has not yet extended to direct applications in GANs or diffusion models, we believe it provides valuable insights for the deep learning and generative modeling communities.

Reviewer comment (Experiments):
How do you extend this idea to other generative models such as GANs or latent diffusion models?

Author response:
VAEs, GANs, and diffusion models each offer distinct advantages as generative models. In this paper, we propose enhancements to the versatility of VAEs, which have a probabilistic foundation. Meanwhile, GANs and diffusion models are garnering attention for high-quality image generation based on large datasets and substantial model sizes. It is conceivable that considering data characteristics and domain knowledge will become important in these models as well, where this paper can make a contribution.
Specifically, the stochastic processes inherent in diffusion models may offer a higher potential for applying the concepts of this theory. For instance, incorporating the concepts of this paper into the denoising process could be envisaged. We have added 'potential' as future work in the manuscript.

Reviewer comment (Theory):
One aspect I am unclear about is why the reconstruction loss is not just defined directly as $\mathbb{E}[d(\textbf{x}, \hat{\textbf{x}}(\textbf{z}))]$ (expectation wrt ${q_{\phi}(\textbf{x}, \textbf{z})}$)? Even the log-expected reconstruction loss in Eq. (24) can be considered as an upper bound of taking log of the dissimilarity measure $d$, i.e., $\log d(\textbf{x}, \hat{\textbf{x}}(\textbf{z}))$ (using Jensen's).

Author response:
**One aspect I am unclear about is why the reconstruction loss is not just defined directly .. (using Jensen's).**
In this paper, reconstruction loss was defined using a specific discrepancy measure d in order to capture the rich characteristics of the data in more detail. This measure captures the complexity of the data and allows for flexibility in the model. Using Jensen's inequality allows for more robust constraints to be applied to the loss function during the optimization process, resulting in more precise management of information transfer from latent variables to data variables.

Reviewer comment (Novelty):
One further note, the above perspective seems to fit in the narrative of "generalized variational inference" (See, e.g., "An optimization-centric view on Bayes' rule: reviewing and generalizing variational inference" Section 4.4.4, switching log-likelihood for a general dissimilarity function).

Author response:
**One further note, the above perspective seems to ... general dissimilarity function).**
The "generalized variational inference" perspective you have introduced is perfectly in line with our research motivation. From this perspective, our approach extends the framework of probabilistic variational inference to handle a wider range of discrepancy functions. This allows for better adaptability to specific data sets and tasks. Thank you for your deep insight.

Reviewer comment (Theory):
Eq. (20) is misleading as from Eq. (29) the equality should instead be an upper bound due to Jensen's.

Author response:
**Eq. (20) is misleading as from Eq. (29) the equality should instead be an upper bound due to Jensen's.**
Thank you for pointing this out. We have clarified the text to avoid misunderstandings.

Reviewer comment (Theory):
Why is the M / 2 factor not utilized in Eq. (24)?

Author response:
**Why is the M / 2 factor not utilized in Eq. (24)?**
This was a typographical error and has been corrected. The implementation has been reconfirmed as correct.

Reviewer comment (Writing):
Above Eq. (25) there is a mention of "well-trained autoencoder" with a condition on $D$. I believe this should be $\gamma$ instead?

Author response:
**Above Eq. (25) there is a mention of "well-trained autoencoder" ..  instead?**
The wording has been revised and clarified.

Reviewer comment (Theory):
Something to add to Section 3.1: It might be worth mentioning that exponential family distributions can be explicitly be characterized as a density $ \propto \exp(- B_f(\cdot, \textbf{m}))$ where $ B_f$ is a Bregman divergence, see, e.g., "Information Geometry and Its Applications" Section 2.7.

Author response:
**Something to add to Section 3.1: ...  Section 2.7.**
Thank you very much for your helpful comments. We have added them to the manuscript.

Reviewer comment (Writing):
The writing of the paper is somewhat obscure, especially in what concerns long sentences, cf below.

Author response:
**The writing of the paper is somewhat obscure, especially in what concerns long sentences, cf below.**
Thank you for your feedback. We have revisited the entire manuscript and clarified the sentence accordingly.

Reviewer comment (Theory):
The rational to consider the minimization, instead of the standard maximization, of the ELBO objective is not clear.

Author response:
**The rational to consider the minimization, instead of the standard maximization, of the ELBO objective is not clear.**
This was a word error in the explanation of the general introduction of VAEs. We apologize for any confusion caused.

Reviewer comment (Novelty):
The results seems to be too incremental, many previous works have proposed to replace the Gaussian distribution in VAE by some other distributions.

Author response:
**The results seems to be too incremental, many previous works have proposed to replace the Gaussian distribution in VAE by some other distributions.**
In this paper, we present not merely a substitution of simple distributions but a unified framework for dissimilarity functions in VAEs, along with methodologies to realize it while maintaining the validity of the probabilistic model. The differences between prior work and our proposed approach are primarily detailed in the related work section of the original manuscript; please refer to it. For instance, [Czolbe et al., "A Loss Function for Generative Neural Networks Based on Watson’s Perceptual Model," 2020](https://papers.nips.cc/paper/2020/hash/165a59f7cf3b5c4396ba65953d679f17-Abstract.html) and [Esser et al., "Taming Transformers for High-Resolution Image Synthesis," 2021](https://compvis.github.io/taming-transformers/) employ perceptual loss. However, these maintain a constant scale in the distribution, which fails to leverage the Bayesian properties of VAEs, leading to a balancing problem where the decoder cannot capture uncertainty. Moreover, these perceptual losses are introduced in an ad-hoc manner, making it challenging to tie them to the rich theoretical foundation that VAEs possess. 
Our proposed method suggests a versatile dissimilarity function that preserves the theoretical coherence inherent in VAEs, allowing for the introduction of any function deemed significant for the data domain under consideration, as discussed in viewpoint II. Also, while [Takida et al., "Preventing oversmoothing in VAE via generalized variance parameterization," 2022](https://www.sciencedirect.com/science/article/pii/S0925231222010591) is mentioned as prior work, their consideration is limited to Gaussian cases, making it difficult to incorporate data domain-specific dissimilarity functions.

Reviewer comment (Evaluation):
In 2 out of 5 cases in the described empirical results, the baselines from previous papers performed better (Table 5)

Author response:
**In 2 out of 5 cases in the described empirical results, the baselines from previous papers performed better (Table 5)**
Indeed, the results in Table 5 do not always favor our proposed method. However, the technical contribution of this paper is not about achieving the highest precision in image generation tasks, but about the theoretical derivation and validation of a unified framework for introducing arbitrary dissimilarity into VAEs. Therefore, the most notable aspect of our experiments is the verification of whether the EDDF family of distributions and its estimation algorithm operate validly. The fact that our proposed method performs comparably to existing literature validates the effectiveness of our theory.

Reviewer comment (Evaluation):
The standard deviations in the experiments are not reported.

Author response:
**The standard deviations in the experiments are not reported. Only two out of many metrics evaluating VAE quality are calculated. In particular, the reconstruction errors are not reported in the comparison tables.**
The primary focus of our experiments is to demonstrate a theory that incorporates domain knowledge into the probabilistic model characteristics while preserving its properties. To validate this theory, it is necessary to comprehensively evaluate dissimilarity functions proposed in the past. Using these dissimilarity metrics directly as evaluation criteria would lead to an unfair assessment and could shift the focus away from the main objective of the verification, potentially causing confusion. Therefore, we uniformly employed the current metrics for this purpose. Consequently, we believe that FID and KID are suitable unified metrics for the evaluations in this verification and have adopted them. Furthermore, we have validated the effectiveness of our approach across a broader range of datasets compared to previous literature. We will address the standard deviation below.

Reviewer comment (Evaluation):
Only two out of many metrics evaluating VAE quality are calculated. In particular, the reconstruction errors are not reported in the comparison tables.

Author response:
**The standard deviations in the experiments are not reported. Only two out of many metrics evaluating VAE quality are calculated. In particular, the reconstruction errors are not reported in the comparison tables.**
The primary focus of our experiments is to demonstrate a theory that incorporates domain knowledge into the probabilistic model characteristics while preserving its properties. To validate this theory, it is necessary to comprehensively evaluate dissimilarity functions proposed in the past. Using these dissimilarity metrics directly as evaluation criteria would lead to an unfair assessment and could shift the focus away from the main objective of the verification, potentially causing confusion. Therefore, we uniformly employed the current metrics for this purpose. Consequently, we believe that FID and KID are suitable unified metrics for the evaluations in this verification and have adopted them. Furthermore, we have validated the effectiveness of our approach across a broader range of datasets compared to previous literature. We will address the standard deviation below.

Reviewer comment (Writing):
On page 1, the phrase "VAEs with the commonly used settings practically suffer from performance degradation, such as reconstruction fidelity and generation naturalness" could be rephrased to "... insufficient reconstruction fidelity and..." since as it stands, both "reconstruction fidelity" and "generation naturalness" appear as undesired/bad qualities.

Author response:
**page 1 : "VAEs with the commonly used settings ... appear as undesired/bad qualities.**
Thank you for your comment. It has been corrected.

Reviewer comment (Writing):
On pages 2-3, the phrase "...with negative ELBO objective... whose maximization ..." is potentially confusing, since usually the negative ELBO objective is minimized.

Author response:
We apologize for a word error in our manuscript that appears to have led to a misunderstanding. In the explanation of the general structure of VAEs, the term "maximization" under Equation (1) should have been "minimization". As you correctly pointed out, the negative ELBO is indeed the subject to be minimized, and our treatment of ELBO optimization in VAEs in this paper is consistent with standard practice. This section pertains to the general formalization of VAEs and is not directly related to our proposed method. Given that the presentation scores from the other four reviewers were all highest, we believe that this single word error has not significantly impacted the overall manuscript. Moreover, the novelty of our proposed method is described in Sec. 3. 

**As we mentioned in the introduction, our novelty lies not in the minimization of ELBO but in proposing a framework that maintains the probabilistic model structure while incorporating domain knowledge into the optimization of ELBO, and we kindly request a review focused on the novel aspects.** Thank you for pointing out this important issue. We are grateful for your meticulous and constructive review.

Reviewer comment (Theory):
On page 4, in the last equation in (16), it seems that the paper's method seeks to maximize the reconstruction loss, given by a positive distance cf eqs (9), (18).

Author response:
**pages 2-3: "...with negative ELBO objective... objective is minimized. page 4, last equation ... eqs (9), (18).**
The error regarding minimization has been rectified.

Reviewer comment (Evaluation):
To demonstrate the effectiveness of the method, the standard deviations should be measured in the experiments for various random seeds.

Author response:
**Some suggestions:** 
Thank you for your feedback. Indeed, we agree that conducting experiments with different random seeds is important for enhancing the reliability of the paper. As for the validity of the evaluation metrics, as stated above, we have utilized metrics suited for verifying our theory and selected for evaluating various dissimilarity functions, which we believe are appropriate for testing the effectiveness of the theory. Regarding the datasets, the selection is crucial for demonstrating our theory; thus, we have already performed validations with five different datasets, which is more extensive than in previous literature. Consequently, while we concur with the importance of experiments to assess the impact of random seeds in the camera-ready version, we cannot agree that it directly leads to a lower evaluation without considering other validations.

Reviewer comment (Evaluation):
To demonstrate the effectiveness of the method, more than just two metrics should be measured.

Author response:
**Some suggestions:** 
Thank you for your feedback. Indeed, we agree that conducting experiments with different random seeds is important for enhancing the reliability of the paper. As for the validity of the evaluation metrics, as stated above, we have utilized metrics suited for verifying our theory and selected for evaluating various dissimilarity functions, which we believe are appropriate for testing the effectiveness of the theory. Regarding the datasets, the selection is crucial for demonstrating our theory; thus, we have already performed validations with five different datasets, which is more extensive than in previous literature. Consequently, while we concur with the importance of experiments to assess the impact of random seeds in the camera-ready version, we cannot agree that it directly leads to a lower evaluation without considering other validations.

Reviewer comment (Experiments):
The comparison with baselines performance in other domains and on more complex real-world datasets is necessary for evaluation of the method's practicality.

Author response:
**Some suggestions:** 
Thank you for your feedback. Indeed, we agree that conducting experiments with different random seeds is important for enhancing the reliability of the paper. As for the validity of the evaluation metrics, as stated above, we have utilized metrics suited for verifying our theory and selected for evaluating various dissimilarity functions, which we believe are appropriate for testing the effectiveness of the theory. Regarding the datasets, the selection is crucial for demonstrating our theory; thus, we have already performed validations with five different datasets, which is more extensive than in previous literature. Consequently, while we concur with the importance of experiments to assess the impact of random seeds in the camera-ready version, we cannot agree that it directly leads to a lower evaluation without considering other validations.