Paper: Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps (ICLR-2024)

Reviewer comment (Writing):
The notation in some parts could be improved: E.g., eq. 27 takes the expectation over $\epsilon$ but refers to the noisy data variables $\tilde z^{(s)}$. On the next line, $\tilde z^{(s)}$ transforms to $\tilde x_k$. Small issues like this make the paper slightly difficult to read at times.

Author response:
**Eq. 27 -> Eq. 28.** The shift from $\mathbf{z}$ to $\mathbf{x}$ is correct and not a typographical error — this is due to Theorem 2 in Shirdhonkar. We provide a more in-depth treatment of this theorem for the reviewer, see below for details.

Reviewer comment (Theory):
Given that it is the central part of the proof, I would like more elaboration on how exactly the connection to Theorem 2 of Shirdhonkar & Jacobs is made. One way to do this would be to repeat the theorem and its assumptions in the paper and showing step-by-step how does it apply here. In particular, I wasn’t able to see how do the $2^{-j(s+n/2)}$ terms in their statement of the Theorem connect to the application of it in the paper.

Author response:
*More detailed treatment of Theorem 2 in Shirdhonkar et. al.* We have provided two additional statements in Appendix A. Namely, Lemma A.1, which restates Theorem 2 in Shirdhonkar et. al according to our notation, and Lemma A.2, which shows in detail how Lemma A.1 can be applied to Theorem 5.1. To answer the reviewer’s immediate question, observe that for all possible values of $j, s, n$, the term $2^{-j(s + n/2)}$ is upper bounded by 1. Therefore it can be safely ignored in the upper bound, which is all we require for our Theorem 5.1 (Shirdhonkar et. al, 2008 provides both upper and lower bounds on the EMD).

Reviewer comment (Theory):
What exactly does the paper claim by making the connection with score matching on wavelet representations and EMD? The last sentence of section 5.2 seems to claim that the method allows training diffusion models with the EMD measure, which does not quite seem to be the case here. While the connection between wavelets and EMD is definitely interesting, the connection to diffusion models doesn't seem particularly important as of now.

Author response:
*Significance of Theorem 5.1 --- it does not seem like this allows diffusion models to be trained with the EMD measure.* Actually, this theorem shows precisely that diffusion models can be trained with EMD. Namely, by training with the loss function Eq. 17, our model minimizes the EMD between the true and predicted scores of the data distribution $\mathcal{X}$, which can be understood as unnormalized histograms in Eq. 19. This is the surprising result of Theorem 5.1 – a model that minimizes Eq. 17 is simultaneously a hierarchical model on $\mathbf{z}^{(s)}$ and an EMD-based model on the original $\mathbf{x}$. We believe that this connection is significant, since the EMD measure is known to have desirable properties for inferring distances between spatially structured data.

Reviewer comment (Theory):
It seems that volume-preserving maps are not necessary to form an ELBO with cascaded diffusion models. The reasoning is as follows: We have the data $x$, and S sequences of latent variables $z_{1:T}^s$ with different resolutions and noise levels. The generative process, as defined in the paper, is $p_\theta(x,z)=p(x|z_1^S)\prod_{s=1}^S\prod_{t=2}^T p_\theta(z_{t-1}^s|z_{t}^s,z_1^{(<s)})p(z_T^s)$. We can then form the following inference process that is factorized for the different resolutions: $q(z|x) = \prod_{s=1}^S \prod_{t=2}^T q(z_t^s|z_{t-1}^s)q(z_1^s|x)$. Here $q(z_1^s|x)$ downsamples $x$ and adds the smallest level of noise in the diffusion process. Now if we form the ELBO, we get: $E_q[-\log p_\theta(x)]\leq E_q[-\log\frac{p_\theta(x,z)}{q(z|x)}]=E_q[-\log \frac{p_\theta(x|z_1^S)\prod_{s=1}^S\prod_{t=2}^Tp_\theta(z_{t-1}^s|z_t^s,z_1^{(<s)]})p(z_T^s)}{\prod_{s=1}^S \prod_{t=2}^T q(z_t^s|z_{t-1}^s)q(z_1^s|x)}]$, and further $=E_q[-\log p_\theta(x|z_1^S) - \sum_{s=1}^S[\sum_{t=2}^T\log\frac{p_\theta(z_{t-1}^s|z_t^s,z_t^{(<s)})}{q(z_t^s|z_{t-1}^s)} + \log p(z_T^s) - \log q(z_1^s|x) ]]$. From here, we can follow the standard derivation to get to the KL divergences (e.g., Appendix A in Ho et al., Eq.19., with the difference that their $x_{1:T}$ is redefined to $z_{1:T}^s$ and $x_0$ to $x$). This seems to result in the same ELBO as in the submission, except that instead of having a $p(x|z)$ term for each scale, this would only have it for the last scale. The authors are asked whether they agree with this point of view, or if there is a misunderstanding. In case this understanding is correct, it seems to be a major issue with the paper, since there is no comparison to cascading diffusion models without volume-preserving maps, making the significance of volume-preserving maps unclear.

Author response:
**On the ELBO and the necessity of volume-preserving maps.**  
Indeed, when the goal is to compute a lower bound on the likelihood (i.e., the ELBO), the reviewer is correct, and volume-preserving maps are not strictly necessary. However, this derivation is different (and we argue, inferior) to our framework (Eq. 15) in two ways:  

First, the resulting ELBO is looser than our ELBO (Eq. 15), since it does not utilize the volume preservation property to obtain the equality in Eq. 9. Empirically, this difference is significant: in Appendix C and Tables 4-6 we compare our model to a hierarchical model that does not utilize a volume-preserving $h$ (i.e., a standard cascading hierarchy) derived without Eq. 9. With all other hyperparameters (architecture, noise schedule, and code implementation) held constant, our model improves significantly over the standard hierarchical baseline.  

Second, its inability to produce exact likelihoods $p_\theta(x)$ (e.g. Eq. 6) greatly limits its applicability. For example, the derivation does not work with autoregressive models or normalizing flows, while our framework does. Furthermore, a key property of diffusion models is that they can be interpreted both as an ODE (Eq. 6) and a latent variable model (Eqs. 3-5). As such, this derivation would also be useful for diffusion models that use the exact likelihood formula under Eq. 6 (such as Song et. al, 2020b, Song et. al, 2021, Lipman et. al, 2022, or Zheng et. al, 2023). While we derive our hierarchical diffusion model in its latent variable form, we note that all reported likelihoods (i.e., Tables 1-3) can also be computed exactly via Eq. 6.

Reviewer comment (Theory):
While the paper is well-written, the method showcases improvements, and is well-evaluated, the benefit of volume-preserving maps is not clear. More elaboration is needed on what exactly is the role of volume-preserving maps in likelihood training of cascaded diffusion models. Additionally, the paper requires a clearer derivation for the EMD metric connection, as well as more elaboration on the significance of this connection.

Author response:
**On the ELBO and the necessity of volume-preserving maps.**  
Indeed, when the goal is to compute a lower bound on the likelihood (i.e., the ELBO), the reviewer is correct, and volume-preserving maps are not strictly necessary. However, this derivation is different (and we argue, inferior) to our framework (Eq. 15) in two ways:  

First, the resulting ELBO is looser than our ELBO (Eq. 15), since it does not utilize the volume preservation property to obtain the equality in Eq. 9. Empirically, this difference is significant: in Appendix C and Tables 4-6 we compare our model to a hierarchical model that does not utilize a volume-preserving $h$ (i.e., a standard cascading hierarchy) derived without Eq. 9. With all other hyperparameters (architecture, noise schedule, and code implementation) held constant, our model improves significantly over the standard hierarchical baseline.  

Second, its inability to produce exact likelihoods $p_\theta(x)$ (e.g. Eq. 6) greatly limits its applicability. For example, the derivation does not work with autoregressive models or normalizing flows, while our framework does. Furthermore, a key property of diffusion models is that they can be interpreted both as an ODE (Eq. 6) and a latent variable model (Eqs. 3-5). As such, this derivation would also be useful for diffusion models that use the exact likelihood formula under Eq. 6 (such as Song et. al, 2020b, Song et. al, 2021, Lipman et. al, 2022, or Zheng et. al, 2023). While we derive our hierarchical diffusion model in its latent variable form, we note that all reported likelihoods (i.e., Tables 1-3) can also be computed exactly via Eq. 6.  

*Significance of Theorem 5.1 --- it does not seem like this allows diffusion models to be trained with the EMD measure.* Actually, this theorem shows precisely that diffusion models can be trained with EMD. Namely, by training with the loss function Eq. 17, our model minimizes the EMD between the true and predicted scores of the data distribution $\mathcal{X}$, which can be understood as unnormalized histograms in Eq. 19. This is the surprising result of Theorem 5.1 – a model that minimizes Eq. 17 is simultaneously a hierarchical model on $\mathbf{z}^{(s)}$ and an EMD-based model on the original $\mathbf{x}$. We believe that this connection is significant, since the EMD measure is known to have desirable properties for inferring distances between spatially structured data.

Reviewer comment (Presentation):
The theoretical setting of the paper is general, with possibly non-linear volume-preserving maps $h$, but in practice only linear $h$ are used, which thus boil down to using an orthogonal transform (for wavelets) or a tight frame (for Laplacian pyramid). The linear setting already leads to important improvements, and the possibility of the extension to non-linear maps is interesting for future research, but it obfuscates slightly the results of the paper behind unnecessary complexity. In particular, using "Hierarchical volume-preserving maps" in the title instead of a more explicit reference to orthogonal wavelet transforms (which yield the best results) does not accurately describe the contents of the paper and makes it harder for the reader to connect the contributions of the paper to known concepts. Orthogonal transforms are not mentioned anywhere before page 6, yet are the core component of the method.

Author response:
**The framework proposed considers all homeomorphisms that satisfy Eq. 8, but only two linear examples of $h$ are empirically evaluated.** We believe a major contribution of our work is demonstrating general conditions under which hierarchical models also yield tractable and exact likelihood models, which can greatly inform future research on diffusion models, many of which are hierarchical. We leave further exploration of various $h$, including non-linear cases, to further work. We do recognize that the linear scope of our experiments should be clearly stated early on, and we have added sentences clarifying this point to the abstract, introduction, and Section 4 (introducing the hierarchical volume-preserving map). To quote our added statement in the list of contributions in Section 1:
“We show that two well-known multiscale decompositions, the Laplacian pyramid and wavelet transforms, are hierarchical volume preserving maps, where the linearity of the maps reduce the volume-preserving property into more familiar notions of orthogonality and tight frames (Unser et. al, 2011).”

Reviewer comment (Theory):
The connection to the earth's mover distance seems problematic to me. First, it is unclear what benefits it brings to the paper: I do not understand how to interpret Theorem 5.1, since here the transport is between the true and approximated scores (seen as distributions on $\mathbb R^2$), as opposed to transport between distributions of images as is usually the case. Second, the soundness of the results look questionable to me. For instance, the Wasserstein-p distance is defined between probability distributions, which thus requires that the scores have non-negative entries that sum to one. The authors do not mention this restriction anywhere in the paper. Second, the proof seems to assume that the wavelet coefficients of the image score $\nabla \log q(x)$ are the wavelet conditional scores $\nabla \log q(z^{(s)} | z^{(<s)})$, but this is not the case. Indeed, one has $\log q(x) = \log q(z^{(1)}) + \log q(z^{(2)} | z^{(1)})$ (with $S = 2$ for simplicity), and thus the $z^{(1)}$ component of the image score includes an additional term coming from the high-frequency score component (called "free energy" in [2]). For this reason, the equivalence between wavelet and Laplacian pyramid for the score matching losses also breaks down (which is a more plausible explanation for the difference in performance than the authors' footnote 1).

Author response:
**Connection to the EMD Metric.**

*Transport between scores, rather than images.* Indeed, Theorem 5.1 relates our training loss (Eq. 17) to the transport between scores of the images, rather than images themselves. However, this is not so out of the ordinary. The notion of applying Earth Mover’s Distance to features of the image histogram, rather than its pixels directly, is a relatively common strategy in the literature — for example, the seminal work Rubner et. al, 2000 (as cited in our text) finds that the EMD distance on image “signatures” such as texture, color, and joint color/position embeddings work better than the raw pixels themselves.

*The Wasserstein-p distance is defined between probability distributions, and the score is not a probability distribution.* In Eq. 18, we do in fact consider the setting where $x$ and $y$ are unnormalized and do not sum to one (see the sentence leading up to Eq. 18). This unnormalized case is known as partial matching, and is a setting considered in Rubner et. al, 2000 as well as Shirdhonkar et. al, 2008 which our theory builds off of (see respective references in our work). On the other hand, non-negative entries in the score can indeed be problematic, but this can be circumvented by subtracting the most negative entry from both $x$ and $y$. Namely, let $c$ be such that $c \leq x_{ij}$ and $c \leq y_{ij}$ for all $i, j$. Then $x - c$ and $y - c$ are surely non-negative. Moreover, inspecting Eq. 18, we can see that this transformation does not affect the unnormalized Wasserstein metric, since the $c$ from each unnormalized histogram cancels.

*Free energy term.* We do not quite follow the reviewer’s argument here. What is the reviewer referring to as the “additional free energy term” included in the $z^{(1)}$ component of the wavelet score? As it stands, the equation provided by the reviewer appears to agree entirely with our original formulation (Eq. 15).

Reviewer comment (Theory):
If $z$ is higher-dimensional than $x$, then $z = h(x)$ does not admit a probability density (it is not absolutely continuous with respect to the Lebesgue measure). How do the authors deal with this difficulty, and how is Lemma 4.1 not meaningless in this case?

Author response:
**Preserving likelihoods when $z$ and $x$ are not the same dimension.** We agree that likelihood computation is much trickier when $z$ and $x$ are not the same dimension. However, we can preserve a pushforward measure when we assume $z \in \mathcal{M}$ and $x \in \mathcal{N}$ are from differentiable manifolds $\mathcal{M}$ and $\mathcal{N}$, and $h$ is a diffeomorphism between the two spaces (we recommend referring to [1, 2]). This is satisfied because $h$ is linear and invertible in the cases we consider.
[1] Foundations of differential geometry. S Kobayashi, K Nomizu. 1996.
[2] Normalizing flows on riemannian manifolds. M Gemici, D Rezende, S Mohamed. 2015.

Reviewer comment (Evaluation):
The authors mention switching to a different architecture and weighting scheme for optimizing FID as opposed to likelihood. Does the improvement in FID come with an increase in BPD? If so, the tradeoff between the two should be acknowledged explicitly.

Author response:
**Acknowledging the trade-off of a likelihood weighting scheme optimizing FID.** Optimizing for FID, we obtain BPD 9.55 and 10.31 respectively for W-PCDM and LP-PCDM, respectively. We have added an acknowledgement of this trade-off.

Reviewer comment (Experiments):
Why not using $M=1$ in the OOD detection task? In high-dimensions, we expect from concentration of measure that $-\log p(x) \approx \mathbb H(p(x))$ for almost all $x$, so we should be able to detect changes in distribution from just one sample?

Author response:
**Why not use $M=1$ for OOD detection?** Thank you for this suggestion! We tried using $M=1$, and saw little appreciable change in performance (AUROC of out-of-detection classification of SVHN images was reduced by a hundredth of a point), so we have included this result in our work instead of the $M=2$ case, which we had chosen following the experiments in Nalisnick et. al, 2019. Anomaly detection with $M=1$ is clearly more difficult than $M=2$, so this is a better result. As the reviewer suggests, the superior performance even at $M=1$ may be closely related to the concentration of measure in high dimensions.

Reviewer comment (Writing):
Line before equation (5) should read $\log p_\theta(x_0)$ instead of $p_\theta(x_0)$.

Author response:
**Minor suggestions.** We thank the reviewer for bringing these errors to our attention, and have incorporated them into the text.

Reviewer comment (Theory):
The theoretical presentation is informal and imprecise. I couldn’t follow the theory or substantiate the claims made by the paper. The final model is also ultimately undefined and feels very adhoc despite the principled approach. It’s also confusing what is the contribution of this paper, since it seems that ultimately the paper just uses the Laplace/Wavelet transforms as-is, which are already well known.

Author response:
**Informal and imprecise theoretical presentation.**
We do take very seriously criticisms of the informality and incorrectness of our text. If the reviewer’s concerns with the theoretical presentation are related to the other points (and minor comments) further in the review, we hope we have addressed them in a satisfactory manner and the reviewer may reconsider this judgment. We are happy to answer any further questions.

**Contribution of the paper.**
The paper uses Laplacian pyramid / Wavelet transforms as is. To summarize our contributions, we have proposed a generalized framework for hierarchical modeling that allows for tractable likelihood computations, which was previously not considered or possible with related works. While two notable maps are the Laplacian pyramid / Wavelet transform, our framework is certainly not limited to these transforms. We are not aware of Laplacian pyramid / wavelet diffusion models that derive tractable likelihoods or even conduct density estimation experiments. Finally, we demonstrate significant improvements to the state-of-the-art with our hierarchical framework, and enable future work with more complex hierarchical volume-preserving maps.

Reviewer comment (Experiments):
The experiments do not compare to recent diffusion works [there is no comparisons to 2023 methods, and only one 2022 method (!)], and lack direct comparisons to diffusion models with super-resolution, wavelet or laplace transforms. The experiments are quite roundabout and describe just the overall performance, and don’t directly substantiate specific contributions made in this paper. It is then difficult to see why we improve, or if the theoretical contributions had anything to do with it. There are no ablations. There are very little result exposition or illustrations.

Author response:
**There are few / no comparisons to 2022 and 2023 methods.**
We did not include more recent results as we did not find significant improvement over existing results in Table 1. As per the reviewer’s request, we have incorporated several more 2022 and 2023 methods (Soft Truncation Kim et. al 2022, INDM Kim et. al 2022, i-DODE Zheng et. al 2023). Crucially, we still obtain significant improvements over these methods. We would be happy to include any other results that the reviewer suggests.

**There are no comparisons to diffusion models with super-resolution, wavelet, or laplace transforms.**
As discussed in our work, there did not previously exist super-resolution, wavelet, or Laplacian pyramid -based diffusion models that preserve likelihood computation. Therefore, we cannot form any comparisons on the experiments in Section 6. That being said, as pointed out by another reviewer, it is possible to fold the latent scales of standard cascading hierarchies (that do not satisfy Definition 1) into a looser variational lower bound. However, this comes at a significant cost to modeling performance, as seen in Table 4-6 (Appendix C).

**The experiments are roundabout, and don’t directly substantiate specific contributions made in this paper.** Can the reviewer elaborate on this point? If it is due to points in the minor comments below, we hope we have adequately addressed this criticism. The two central claims in this paper are that 1) we enable exact likelihood estimation, which is substantiated by Lemma 4.1 and 2) that hierarchical modeling with wavelets and Laplacian pyramids approximate an EMD loss on the scores, which improves modeling performance and is substantiated by significant improvements in performance (Tables 1-3).

**There are no ablations.** Our paper claims that hierarchical volume-preserving maps allow for likelihood modeling in cascaded models. Unfortunately, hierarchical maps that are not volume-preserving (such as a standard cascading model) do not allow for exact likelihood computation, therefore direct ablations are not possible. However, as previously mentioned, one can still obtain a loose loser bound on the likelihood. We compare our work against this lower bound in a series of ablations in Appendix C and Tables 4-6, and find that using a standard cascading hierarchy (end of Section 4.1) significantly reduces modeling performance.

Reviewer comment (Theory):
The eq 3 is looks like the simple forward process, but is instead the forward “posterior”. For instance, DDPM paper (Ho et al 2020) eq 2 LHS is the same equation as eq 3 LHS here, but has a conflicting RHS. Can you clarify which q interpretation you are using, and which q the eq 5 is taken over.

Author response:
**Eq. 3 differs from Ho et. al, 2020.** The RHS here is different from Ho et al, 2020 because they are different Markov processes. Eq. 2 in Ho et. al, 2020 describes the forward diffusion process, whereas Eq. 3 here describes the reverse diffusion process. The $q$ in Eq. 5 is also taken over the same reverse diffusion. For more information, see Appendix E.1 of Kingma et. al, 2021 (as cited in our work).

Reviewer comment (Theory):
I’m not sure I if I understand the premise of the paper. I agree that with invariant transformation we have p(x) = p(h(x)) property. However, this is useful only if the joint distribution p(z^1, …, z^S) decomposes independently into p(z^1) * … * p(z^S); and I don’t see why this would happen. Surely the different scales are highly dependent, and you would have to do a chain rule p(z^S | z^S-1) … p(z^2 | z^1) p(z^1) [or some other decomposition], and in this setting I don’t see the invariant property being useful. The paper should clarify the invariance wrt scale independence, preferably in the introduction and in beginning of sec 4. Right now the main claim of the paper does not convince. For instance, *“if h(x) is the scales of the hierarchical model, then we can directly use the joint likelihood over these scales as the desired model likelihood”* is generally false as far as I can see.

Author response:
**Premise of the paper.** We do not understand the reviewer’s concern here. Why must we assume independence of the scales $z^{(1)}, z^{(1)}, \dots, z^{(S)}$ for the invariance to hold? In fact, we consider solely the dependent case, as this is the necessary conditional structure for a hierarchical multiscale model.

Reviewer comment (Theory):
Eq 8 has square root, but why? It does not seem to do anything when det=1.

Author response:
**Square root in Eq. 8.** Indeed, the square root is not strictly necessary here. However, this entire term (including the square root) is mathematically meaningful in normalizing flows and differential geometry as the instantaneous change in volume between differentials (infinitesimal volumes) of the two Riemannian manifolds at $x$ and $h(x)$ (Berger and Gostiaux, 2012), so we have opted to keep the square root.

Reviewer comment (Writing):
What does z(1), z(2), . . . , z(S) = h(x) mean? I don’t understand the notation. Does h(x) return S different things, or does h return one thing as in z1=h; z2=h; etc? h seems to be a downscaling operator, is this true? The nature of “h” is a bit confusing at this stage of the paper. What does h output? What resolution?

Author response:
**What does $z^{(1)}, \dots, z^{(S)} = h(x)$** mean? Since $h$ defines a hierarchical map, the range of $h$ is the cartesian product of the $S$ scale spaces, i.e., $h: \mathcal{X} \rightarrow \mathcal{Z}^{(1)} \times \mathcal{Z}^{(2)} \times \dots \times \mathcal{Z}^{(S)}$, where $x \in \mathcal{X}$ and $z^{(s)} \in \mathcal{Z}^{(s)}$. We have swapped sides of the equality and added parentheses around the RHS in the equation $h(\mathbf{x}) = (z^{(1)}, \dots, z^{(S)})$ to emphasize that the functional output of $h$ is a cartesian product, and further clarified the functional definition of $h$ in Section 5.1.

Reviewer comment (Reproducibility):
What does “p_theta trained on z1..zS” mean? Trained how? Against what objective? What kind of p_theta? These statements are overly casual, and need to be made precise, explicit, rigorous and transparent.

Author response:
**How is $p_\theta$ “trained” in Lemma 4.1?** We have removed the word “trained” in this sentence. $p_\theta$ need not be trained in any manner — the theorem holds as long as $p_\theta$ is a valid likelihood function on $\mathbf{z}^{(1)}, \dots, \mathbf{z}^{(S)}$.

Reviewer comment (Writing):
“Then the model likelihood”. What does model likelihood mean? Can you make this rigorous? This is too informal. Or is this supposed to be informal and imprecise?

Author response:
**What is $p_\theta(x)$? What is the model likelihood?** The model likelihood is the joint likelihood of the model and data, rigorously defined as $p_\theta(x)$. $p_\theta$ simply refers to a likelihood function that takes a data point (or its latent representation) as input, and returns its modeled density. We have changed "model likelihood" to simply "likelihood" to reduce confusion.

Reviewer comment (Writing):
What does p_theta(x) mean in eq 9? Does it refer to eq 7 or something else? What does it mean to apply h(x) to eq 7: do we apply h() to all z’s together, or separately, or what..?

Author response:
**The statements after Eq. 9 are not true.** We state after Eq. 9 that Eq. 9 (i.e., $\log p_\theta(x) = \log p_\theta(h(x))$) demonstrates a form of probabilistic invariance. Formally, letting $g_x(h) = p_\theta(h(x))$, the set of hierarchical volume-preserving maps $\mathcal{H}$ form an equivalence class under the the equivalence relation $g_x(h) = g_x(h’)$ for $h, h’ \in \mathcal{H}$ and $x \in \mathcal{X}$. This equivalence relation can be clearly seen by applying Eq. 9 to $h$ and $h’$, and invoking the transitive property.

Reviewer comment (Theory):
I don’t think the statements after eq 9 are true; I see no convincing arguments for this. It’s not clear what the terms in eq 9 even mean or how they are defined, and the dependency issue of the joint still remains.

Author response:
*Understanding Eq. 15.* Note that we are considering the log probabilities in Eq. 15. The sum of log probabilities is equal to the product of the probabilities. Therefore we equivalently have $p(x) = p(z^{(1)}) p(z^{(2)}|x^{(1)}) \cdots p(z^{(S)} | z^{(1)}, \dots, z^{(S-1)})$. The RHS of this equality becomes the joint likelihood $p(z^{(1)}, \dots, z^{(S)})$ by the probabilistic chain rule. Therefore this is just an application of Eq. 9. Many likelihood models consider log probabilities rather than probabilities for numerical stability.

Reviewer comment (Writing):
In eq 10 the h is now some kind of linear operator, while earlier it was a function. Is the idea that h(x)=h*x? It would be good to clarify the distinction between function h and operator tensor h.

Author response:
**$h$ in Eq. 10 is a linear operator. Isn't it a function?**
Indeed, in Eq. 10, $h$ is a function that is happens to also a linear operator, since we are considering the standard cascading hierarchy as described in Ho et. al, 2020, which can be seen as a bilinear resampling operation. In general, $h$ can be an arbitrary function that satisfies Eq. 9.

Reviewer comment (Writing):
What does * mean in (j*2)? Is this a product or convolution? If this is a convolution, what does it mean to convolve j with 2?

Author response:
**The * in (j*2).** The asterisk is simply the product, not a convolution. We see how it can be confusing, and have replaced the asterisk with a dot (i.e., $j \cdot 2$) for clarity.

Reviewer comment (Presentation):
What does fig 2 show? I think it shows y^4 and z^3...z^1. Can you clarify?

Author response:
**What does Figure 2 show?** In this figure, we show the Laplacian Pyramid coefficients, i.e., $z^{(1)}, z^{(2)}, …, z^{(4)}$ from left to right.

Reviewer comment (Writing):
What does tight frame mean? Can you give a conceptual explanation? What does “h” mean in parseval equation? There is no “h” in the Laplace section of the paper at all: does some of this stuff (y,z,d,u) relate to h?

Author response:
**What is a tight frame?** Informally, a tight frame describes a representation (of a signal) that is norm-preserving. For a more formal treatment, please see Unser et. al, 2011 (as cited in our manuscript).

Reviewer comment (Writing):
Again, what is the “mapping” h in Wavelet case? Is it the z’s, or y’s, or them together, or something else?

Author response:
**What is $h$ in the Laplacian Pyramid section in 4.2? What is $h$ in the Wavelet Decomposition section in 4.2?** $h$ refers to the Laplacian pyramid or wavelet transform, respectively. Formally, $h(\mathbf{x}) = (z^{(1)}, z^{(2)}, …, z^{(S)})$ where where $(z^{(1)}, z^{(2)}, …, z^{(S)})$ are as defined in the respective subsection (Wavelets or Laplacian Pyramids) in Section 4.2. While the functional form is not explicit to maintain generality w.r.t. $S$, this definition is rigorous.

Reviewer comment (Writing):
The wavelet and Laplacian pyramids both retain the original resolution of the image. Wasn’t the goal of this paper to reduce resolution? I’m a bit confused. Can you clarify the notion of resolution vs scale, and clarify which one you want to reduce? The very first sentence of the paper in introduction talks about super-resolution, which leads me to believe that one should change resolution somewhere in the paper, but neither of the transform types seems to change resolution. Can you clarify these aspects?

Author response:
**Wasn’t the goal of this paper to reduce resolution? Resolution vs scale?**
The goal of the paper is not to reduce resolution. Hierarchical models are generally useful because they enable an explicit multi-scale model of the data. While super-resolution is used to obtain finer scales from coarser scales (e.g. $\mathbf{z}^{(s)}$ from $\mathbf{z}^{(s - 1)}$), all scales are retained in our model, and therefore information is not lost. In fact, in Ho et. al, 2022 (in the paper) the sequence of scales actually contains more parameters (i.e., pixels) than the original image. Moreover, when $h$ is the wavelet transform, the size of the largest scale is 25% the original image.

Reviewer comment (Writing):
What is the q in eq 16?

Author response:
**What is the q in eq 16?**
The $q$ in Eq. 16 is the same $q$ as in Eqs. 3 and 5.

Reviewer comment (Writing):
What is z_0^s in eq 16? How do you get them? How do you get the z_1’s? How do you get the z_k’s? What are the terms here: I can’t really follow any of them. Can you explain all three terms (the logp and two KLs). For instance, the regular logp term is the probability of observed image given z^1 while marginalising all intermediate states away. Here isntead we have p(z^s_0 | z_1^s, z_0^<s). So we are looking at one scale likelihood, but somehow conditioning with earlier scale “observations”? I can’t really follow at all what is happening, since very little of eq 16 or the underlying processes have been defined or characterised. Please include an algorithm boxes for training and sampling. It’s also confusing what is h here.

Author response:
**What are the $z_{0:T}^{(s)}$ in Eq. 16?** $z_{0:T}^{(s)}$ are the variables of the diffusion process corresponding to each scale $s$, where $z_0^{(s)}$ = $z^{(s)}$. This is much like $x_{0:T}$ are variables of standard diffusion process as described in Section 3 where $x_0 = x$, except that the diffusion described by $z_{0:T}^{(s)}$ are additionally conditioned on the prior scales $z^{(<s)}_0 = z^{(<s)} = (z^{(1)},z^{(2)}, \dots, z^{(s-1)})$.

Reviewer comment (Writing):
I’m having hard time understanding what the eq 18 is conceptually representing. Is the idea that criss-cross the pixels from one image to another by moving the pixel locations around?

Author response:
**Understanding Eq. 18.** This equation defines a distance between two distributions $\mathbf{x}$ and $\mathbf{y}$ based on the cost to transport mass from each pair of locations ($(i,j)$ in $\mathbf{x}$ and $(k, \ell)$ in $\mathbf{y}$) so that $\mathbf{x}$ looks like $\mathbf{y}$. The infimum ensures that this cost is the most reasonable (i.e., efficient) transport plan. Intuitively, we are trying to measure the effort required to shift pixel intensities from the “high mass” areas of $\mathbf{x}$ towards the “high mass” areas of $\mathbf{y}$, until $\mathbf{x}$ resembles $\mathbf{y}$. For a rigorous understanding, we encourage the reader to look into more detailed treatments on optimal transport [1] or Earth Mover’s Distance (Rubner et. al, 2000).

[1] Computational Optimal Transport. https://arxiv.org/abs/1803.00567

Reviewer comment (Writing):
At experiments I still don’t know what does the function “h” mean in this paper, or what it is. I guess it’s the Laplace pyramid, but not sure which part. I’m also not sure how did we now solve the problem of joint needing to be marginalised to only evaluate the final image. I can’t really connect the problem statement in introduction to the methods presented in the paper. I think the paper needs a method summary section that explains how all the pieces come together to solve the original problem, while including an algorithm box.

Author response:
**What is $h$ in Section 6 (experiments)?** We discuss $h$ at length in Section 4. In general, $h$ is a hierarchical volume-preserving map — any function that maps data (e.g., images) $\mathbf{x}$ to a sequence of latent variables $\mathbf{z}^{(1)}, \dots, \mathbf{z}^{(S)}$ and satisfies Eq. 8. In our experiments, $h$ refers to either the Laplacian pyramid transform or the wavelet transform (LP-PCDM and W-PCDM, respectively).

Reviewer comment (Novelty):
The paper should cite to simple diffusion, that also tackles multi-resolution.

Author response:
**Citing simple diffusion.** If the reviewer is referring to “simple diffusion: End-to-end diffusion for high resolution images” by Heek et. al, 2023, it appears that this method specifically does not involve any multiscale (i.e., hierarchical, cascading) architecture. (See e.g. Section 1 and Figure 2 caption of their paper.) Our understanding is that they tackle diffusion purely on the pixel space.

Reviewer comment (Experiments):
In experiment the key comparison is how does this method fare against diffusion models that utilise laplace or wavelet pyramids; or have an explicit super-resolution component. Looking at table 1 almost all comparison targets are irrelevant old methods, and I fail to see many if any super-resolution/laplace/wavelet methods. There is only a single comparison to a 2022 method, and none to 2023. Given the astronomical pace of the research in diffusion, this is not acceptable. The paper needs to compare both to (i) other super/wavelet/laplace diffusion models, and more comprehensively to 2022 and 2023 diffusion models. The claims of state-of-the-art performance is unsubstantiated.

Author response:
**There are few / no comparisons to 2022 and 2023 methods.**
We did not include more recent results as we did not find significant improvement over existing results in Table 1. As per the reviewer’s request, we have incorporated several more 2022 and 2023 methods (Soft Truncation Kim et. al 2022, INDM Kim et. al 2022, i-DODE Zheng et. al 2023). Crucially, we still obtain significant improvements over these methods. We would be happy to include any other results that the reviewer suggests.

**There are no comparisons to diffusion models with super-resolution, wavelet, or laplace transforms.**
As discussed in our work, there did not previously exist super-resolution, wavelet, or Laplacian pyramid -based diffusion models that preserve likelihood computation. Therefore, we cannot form any comparisons on the experiments in Section 6. That being said, as pointed out by another reviewer, it is possible to fold the latent scales of standard cascading hierarchies (that do not satisfy Definition 1) into a looser variational lower bound. However, this comes at a significant cost to modeling performance, as seen in Table 4-6 (Appendix C).

Reviewer comment (Experiments):
The OOD experiments only seem to compare to quite old generative models. Why not compare to other diffusion models? Wasn’t the point of the paper to improve the multi-scale handling of diffusion models, so surely one should then compare to other diffusion models with more adhoc multi-scalings, or to diffusion models with a single scale only. I’m also not sure what the table 2 rows mean.

Author response:
**Comparing OOD benchmark to modern diffusion models.** We have included a column in Table 2 against the state-of-the-art diffusion model in likelihood modeling, the variational diffusion model (VDM, Ho et. al, 2021).

**What do the rows in Table 2 mean?** Why is there no comparison against modern diffusion models? We include detailed information about the experiment in Table 2 in Appendix B.1. We compare against the current state-of-the-art diffusion model, variational diffusion models (VDM, Ho et. al, 2021).

Reviewer comment (Experiments):
In 6.3. there are comparisons to uni-scale diffusion models (which is great), but one should also compare to competing multi-scale diffusion models.

Author response:
**Why are there no comparisons to other multiscale diffusion models (Section 6.1, 6.2, 6.3)?** There are no other multiscale diffusion models that produce tractable likelihoods — in fact, this is the main contribution of our work. Tractable likelihoods are required for any experiments in Sections 6.1, 6.2, and 6.3. The only known method to obtain a lower bound on the likelihood is via a variational lower bound. We derive and compare against this setting in Appendix C and Tables 4-6, and show that our method performs significantly better. On the other hand, we have added more results for standard diffusion models 2022 and 2023. We note that we still demonstrate significant improvements over these results. We are happy to include any other recent works, or likelihood-based multiscale diffusion models the reviewer suggests.

Reviewer comment (Novelty):
The use of semi-autoregressive hierarchical diffusion models is a common approach, including the use of wavelets, latent variables, and null-space vectors for upsampling the images. The use of cascaded volume-preserving maps does not seem to give differences to these existing works, and the related works on multi-scale diffusion models should be more carefully added.

Author response:
**Differences from existing hierarchical diffusion models.** Indeed, hierarchical diffusion models exist in the literature, and we thank the reviewer for the references, which we have added to the text. However, as discussed, these models do not support tractable likelihood computation. The main contribution of our work is to show the conditions under which hierarchical models such as these are also feasible likelihood models (Eq. 8), which enable many downstream analyses (anomaly detection, data compression, data enrichment, etc.). To further support the importance of our hierarchical volume-preserving property and the **exact** likelihood computation it enables (Eqs. 8-9), we provide additional experiments in Appendix C and Tables 4-6 that compare our model against $h$ that does not satisfy Eq. 8.

Reviewer comment (Reproducibility):
According to the specific features of the hierarchical models, the architecture description in appendix B should be more precisely described. Now it is not easy to understand which architecture is used for training multiscale models (single model for all scales, or multiple models such that the (conditional) score for each scale is learned in each model?), including the number of parameters. (This might not be a problem since this paper used the same architecture as the VDM paper.)

Author response:
**Architecture description in Appendix B.** Indeed, there is a separate score model for each scale. We have added a sentence clarifying this point in Appendix B. We have also added the number of parameters for each model (also Appendix B).

Reviewer comment (Presentation):
Many parentheses are not closed; it is recommended to use ( \left[, \right] ) and ( \left(, \right) ) commands.

Author response:
**Closing parentheses.** Thank you for bringing this to our attention. Indeed, the \left( and \right) latex operations are very helpful, and we have incorporated them here and elsewhere in the text.

Reviewer comment (Writing):
In the first paragraph of Section 5.1, $z^{(1)},\cdots,z^{(S)} = h(x)$ is considered to be the abuse of denoting the hierarchical volume-preserving map. It is suggested to use clearer notation to represent this.

Author response:
**$h(x)$ notation.** We apologize for the confusion. The range of $h$ is the cartesian product of the $S$ scale spaces, i.e., $h: \mathcal{X} \rightarrow \mathcal{Z}^{(1)} \times \mathcal{Z}^{(2)} \times \dots \times \mathcal{Z}^{(S)}$. We have reversed the equality and added parentheses around the RHS in the equation $h(x) = (z^{(1)}, \dots, z^{(S)})$ to alleviate this confusion, and further clarified the functional definition of $h$ in Sections 3.1 and 5.1.