From: ~Matthias_Seeger1
title: Compare against bilevel optimization techniques?
comment: I feel that this paper misses a comparison and mentioning of bilevel optimization methods as pretty obvious baselines to compare against.
Since beta is just a single scalar parameter, this should be quite effective.

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. ICML 2017

Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration com- plexity of hypergradient computation. ICML 2020.

https://arxiv.org/abs/2212.14032 even has one of the authors here on the paper as well.

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: notable-top-5%
metareview:_summary,_strengths_and_weaknesses: This paper proposes a method to learn a hypernetwork for Beta VAEs - a network that can output the parameters of a VAE for all possible values of beta. The paper received very strong reviews which praised its very well-written, high-quality exposition, its theoretical justification and its strong experimental results. While there were a very small number of concerns by the reviewers, the authors have adequately addressed all of them, with the paper having no outstanding concerns from the reviewers. This paper is further of great interest to the community as Beta VAEs are widely used and the paper provides a principled approach to get rid of a fundamental difficulty of Beta VAEs, i.e. that one has to retrain the model for all different values of beta. I hence strongly recommend to accept this paper.
justification_for_why_not_higher_score: N/A
justification_for_why_not_lower_score: The paper had very strong reviews, with no concerns. At the same time, the paper is highly relevant to a large part of the community.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper2076/Reviewer_MZus
title: Response
comment: Thank you for the detailed clarifications and answers to my questions. I will raise my recommendation.

From: ICLR.cc/2023/Conference/Paper2076/Authors
title: Reply to Reviewer MZus
comment: Thank you for your feedback and help in improving the paper. We will address your concerns and questions below.

> Q. Theorem 1: It remains unclear to me, why it is necessary to represent the parameter matrices of the model as products of matrices? Why are then the affine transforms applied directly to the VAE weights for the non-linear models in (8,9)?

The product of two matrices in Theorem 1 is necessary to match the functional form of linear VAE response functions. As a motivating example, the derivation in Appendix A.2 highlights that the optimal parameters of the encoder can be written in a form of $\mathbf{A} \mathbf{D}(\beta) \mathbf{B}$, where $\mathbf{D}(\beta)$ formulates a diagonal matrix conditioned on $\beta$. This diagonal matrix can be seen as the scaling of activation after the first layer (or equivalently, the first matrix in the product). Hence, we use two MR-VAE layers to guarantee recovery of the optimal solution.

While the exact response function for deep VAEs is not straightforward to approximate, we use the analysis of linear VAE as a motivation. In Appendix D, we further show that a simple affine transformation obtains similar performance to more expressive architectures such as 2-layer MLP and is sufficient to learn the full rate-distortion curve. We thus use an affine transformation in all our experiments.

> Q. Step from (18) to (19,20): Taking the expectation over the data and maximizing it w.r.t. $\mu$ will give a $\mu_{\text{MLE}}$ that depends on the other model parameters. It remains unclear how to derive the gradients of the objective w.r.t. the latter.

Our linear VAE formulation is equivalent to that presented in [1]. As in [1], we utilize the fact that the optimal parameters for the encoder and decoder coincide with the optimal parameters of the probabilistic PCA (pPCA). Hence, $\mu_{\text{MLE}}$ is the mean of the data and we assume that $\mu$ is fixed throughout the derivation. We apologize for the confusion and will clarify this connection in our updated manuscript. 

> Q. It remains unclear to me, why applying the affine transform in all layers of the encoder/decoder of non-linear VAEs is sufficient for obtaining the required hypernetwork. Is this may be related to the known rescaling properties of ReLU networks? Would this work also for other non-linearities?

Previous works (e.g., [2]) highlight that multiplicative interactions (as done in MR-VAE) enrich the representable function classes of neural networks and are successful in many cases where multiple pieces of information need to be combined. We are not completely sure what you mean by the connection to the rescaling properties of ReLU networks. In Section 5, we show that MR-VAEs are compatible with various activation functions; we explored ReLU, ELU, Swish, and Tanh activation functions. Does our response address your question?

> Q. More generally, can this approach be used also for non-Gaussian VAEs with discrete latent variables?

While our analysis and experiments are limited to Gaussian VAEs, it is straightforward to apply MR-VAE methodology to various VAE architectures such as VQ-VAE [3], which we leave for future work. As an example, we can construct the hypernetwork formulation conditioned on $\lambda$ [4] to learn a tradeoff between reconstruction loss and codebook loss + commitment loss in a single training run. 

Please let us know if our response addresses your concerns; we are happy to make additional clarifications.

[1] Lucas, J., Tucker, G., Grosse, R. B., & Norouzi, M. (2019). Don't blame the elbo! a linear vae perspective on posterior collapse. Advances in Neural Information Processing Systems, 32.

[2] Jayakumar, S. M., Czarnecki, W. M., Menick, J., Schwarz, J., Rae, J., Osindero, S., ... & Pascanu, R. (2020). Multiplicative interactions and where to find them.

[3] Van Den Oord, A., & Vinyals, O. (2017). Neural discrete representation learning. Advances in neural information processing systems, 30.

[4] Wu, H., & Flierl, M. (2019, November). Learning product codebooks using vector-quantized autoencoders for image retrieval. In 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP) (pp. 1-5). IEEE.

From: ICLR.cc/2023/Conference/Paper2076/Authors
title: Reply to Reviewer 47TW
comment: We are pleased that the reviewer found our work to be very well-written with strong empirical results. Please let us know if you have any additional comments.

From: ICLR.cc/2023/Conference/Paper2076/Authors
title: Reply to Reviewer fEL1
comment: We appreciate the positive review. Please let us know if you have any additional comments.

From: ICLR.cc/2023/Conference/Paper2076/Authors
title: General Response to All Reviewers
comment: Dear Reviewers,

We thank all reviewers for their detailed reviews and comments! We are pleased that the reviewers identified our work to be well-written and easy to read (fEL1, 47TW, MZus), interesting and novel (fEL1, MZus), technically solid (fEL1, 47TW), and the experiments to be comprehensive (fEL1, 47TW, MZus).

We will address the remaining concerns and questions in an individual response. We thank the reviewers again for their time.

From: ICLR.cc/2023/Conference/Paper2076/Reviewer_fEL1
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper introduces Multi-Rate VAE, which is a hyper-network that outputs the parameters of a $\beta$-VAE network, when given a value of $\beta$. The rate-distortion curve created using the the networks computed from this hyper-network matches the one that is obtained by repeatedly training $\beta$-VAEs for different values of $\beta$. This hyper-network also has similar number of parameters as that of the $\beta$-VAE network it outputs, and has similar training time. The construction of the hyper-network is theoretically principled, because the paper proves that the hyper-network can output the optimal $\beta$-VAE when the VAE is linear.
strength_and_weaknesses: Strengths:
1. The proposed hyper-network construction seems to be theoretically principled. This is because for linear VAEs, the paper provides a theorem that proves that indeed the optimized hyper-network can output parameters for the optimal linear $\beta$-VAE.
2. The hyper-network is of practical importance since it bypasses the need to retrain $\beta$-VAEs for different values of $\beta$, while fine tuning for optimal $\beta$.
3. The hyper-network has similar size and training time as the $\beta$-VAE it outputs.
4. The results have been empirically verified on MNIST, OMNIGLOT, CIFAR10 and CelebA datasets, where it is shown that the rate-distortion curve created using the the networks computed from this hyper-network matches the one that is obtained by repeatedly training $\beta$-VAEs for different values of $\beta$.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and the idea is novel.
summary_of_the_review: I think the idea of hyper-networks outputting $\beta$-VAE for different values of $\beta$ without the need for retraining from scratch for each $\beta$, is of practical importance. Furthermore, the proposed framework is both theoretically principled and empirically verified to work.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper2076/Reviewer_47TW
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes MR-VAE, an algorithm to train a VAE that corresponds to an interval of $\beta$ values in a single training run.  The proposed method trains a hypernetwork that modifies the weights of a regular VAE at inference time for a given value of $\beta$, thus eliminating the need to retrain a VAE for each value of $\beta$.  The choice of hyperparameters for the hypernetwork is based on the analysis on linear VAEs, and the empirical results demonstrate that the method generalizes well to modern VAE architectures.
strength_and_weaknesses: **Strengths**
* Very well-written paper with a clear motivation.
* Strong empirical results on diverse datasets.  In particular, the paper addresses the important questions that naturally arise: (1) does MR-VAE introduce performance degradation?  (2) does MR-VAE generalize to larger, complicated architectures?  (3) how sensitive is MR-VAE to the choice of its (only) additional hyperparameter $(a, b)$?  The experiments convincingly and positively answer all these questions.

**Weaknesses**
* I did not find any particular weakness.
clarity,_quality,_novelty_and_reproducibility: **Clarify & Quality**
The paper is very well-written, and the quality of the exposition is high.

**Novelty**
While the question studied in the paper is not very novel, the paper provides a clean algorithm with good justification that also works well in practice.  The analysis of optimal response functions on linear VAEs was insightful.
summary_of_the_review: This paper provides an answer to a somewhat obvious but practically important question of whether it's possible to train a VAE that can operate on a continuum of rate-distortion points.  The presentation is clear, with clear experimental results to back up the usability (i.e. R-D performance, computational/memory cost, stability to hyperparameters).  I believe this is a valuable contribution to the field.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper2076/Reviewer_MZus
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper considers $\beta$-VAEs and treats the weight $\beta$ of the KL-divergence term as a hyper-parameter. The authors develop a hypernetwork model which learns the optimal VAE parameters (i.e. weights of the encoder and decoder) simultaneously for all $\beta$ values within a pre-specified range. The approach is experimentally tested on different datasets like MNIST, SVHN and others as well as for different VAE models like standard VAEs and hierarchical VAEs.
strength_and_weaknesses: Paper strengths:
The authors start from simple log-linear Gaussian VAEs and prove theoretically that a simple affine transform of the encoder/decoder weights parametrised by $\beta$ provides the dependence of the optimal wights on the hyper-parameter $\beta$. They conjecture that the hyper-network for complex, non-linear Gaussian VAEs is obtained by applying such affine transforms of the base weights in each layer of the encoder/decoder pair. The experiments compare achievable data likelihoods for VAEs (with  fixed architecture), when learned with fixed betas (aka response curve), with the one obtained from learning the hypernetwork. It is striking to see how good the corresponding rate curves are matched by the hypernetwork for different VAE variants.

Paper weaknesses:
There are a few conceptual choices and issues which remain unclear for me:
I was not able to fully follow the proofs given in the supplementary material for the log-linear Gaussian VAEs:
- Step from (18) to (19,20): Taking the expectation over the data and maximising it w.r.t. to $\mu$ will give a $\mu_{MLE}$ that depends on the other model parameters. It remains unclear how to derive the gradients of the objective w.r.t. the latter.
- Theorem 1: It remains unclear to me, why it is necessary to represent the parameter matrices of the model as products of 
matrices? Why are then the affine transforms applied directly to the VAE weights for the non-linear models in (8,9)?

It remains unclear to me, why applying the affine transform in all layers of the encoder/decoder of non-linear VAEs is sufficient for obtaining the required hypernetwork. Is this may be related to the known rescaling properties of ReLU networks? Would this work also for other non-linearities?. And more general, can this approach be used also for non-Gaussian VAEs with discrete latent variables?
clarity,_quality,_novelty_and_reproducibility: The paper is well organised and clearly written in most parts. However, the technical material given in the supplementary material is indispensable for understanding the approach. Even considering this, there are quite some open questions remaining (see above).
summary_of_the_review: The paper proposes an interesting hypernetwork approach for estimating the optimal weights of $\beta$-VAEs simultaneously for all $\beta$ values within a pre-specified range for given encoder/decoder architectures. The technical part (especially in  the supplementary material) is sometimes hard to follow and there are a few questions and issues remaining open. My current recommendation is a lower bound. The final recommendation will depend on the clarification of these questions in the discussion phase.

After rebuttal discussion: The clarifications provided by the authors answered almost all my questions and are convincing. I am raising my recommendation to 8.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper