Paper: Sequential Data Generation with Groupwise Diffusion Process (ICLR-2024)

Reviewer comment (Writing):
Section 3.1: The first paragraph should be rewritten in a more detailed way. This paragraph is key to lead the reader to understand the way data division is performed. For instance, it is not clear "we divide data into $1 \leq k \leq d$ groups", does this mean we divide the data into any possible integer in the interval $1 \leq k \leq d$, then that integer value represents the number of groups? Or is it that you want to divide data into a number of $d$ groups, where each group has a length of $k$? Shouldn't we refer to the total number of data and then refer to the splits of that number as per your method proposes?

Author response:
> ...does this mean we divide the data into any possible integer in the interval $1 \leq k \leq d$, then that integer value represents the number of groups?  

This is right. We revised the paragraph as follows:  
> ...we divide data $\boldsymbol x \in \mathbb R^d$ into $k\in \{1,...,d\}$ disjoint groups and assign different noise schedules for each group.  
Specifically, we divide the indices $\{1,...,d\}$ into a partition $\{S_j\}_{j=1}^k$ where $\sum_j |S_j|=d$. $S_j$ is used for determining membership of $\boldsymbol x_i$ to $j$-th group by testing $i \in S_j$.  

For example, in the block-wise grouping, $j$-th group corresponds to the $j$-th block of an image (there are $k$ blocks in total). $S_j$ is the set of the indices of each block.

Reviewer comment (Writing):
Also Section 3.1: It suddenly appears "we divide the indices $\{1, ..., d\}$ into a partition $\{S j \}^k_{j=1}$". Do you divide the set $\{1, ..., d\}$ into another set $\{S_1,S_2...,S_k\}$? So, what is the meaning or definition of each $S_i$? That is not clear to me.

Author response:
> ...does this mean we divide the data into any possible integer in the interval $1 \leq k \leq d$, then that integer value represents the number of groups?  

This is right. We revised the paragraph as follows:  
> ...we divide data $\boldsymbol x \in \mathbb R^d$ into $k\in \{1,...,d\}$ disjoint groups and assign different noise schedules for each group.  
Specifically, we divide the indices $\{1,...,d\}$ into a partition $\{S_j\}_{j=1}^k$ where $\sum_j |S_j|=d$. $S_j$ is used for determining membership of $\boldsymbol x_i$ to $j$-th group by testing $i \in S_j$.  

For example, in the block-wise grouping, $j$-th group corresponds to the $j$-th block of an image (there are $k$ blocks in total). $S_j$ is the set of the indices of each block.

Reviewer comment (Writing):
At the end of Section 3.1: It reads "the elements of j-th latent group are diffused into noise", what is a latent group? When was the name latent group introduced? Or what variable corresponds to such a latent group?

Author response:
> What is a latent group? When was the name latent group introduced? Or what variable corresponds to such a latent group?  

Throughout the paper, "group of latent variables" and "latent group" were used interchangeably. In the revised version, we clarified this in the last paragraph of the introduction. Also, there was a typo in Sec. 3.1 which we replaced "latent group" with "group" during the revision. Thank you for pointing out.

Reviewer comment (Experiments):
In the Fig. 6 where traversing a single latent variable is associated to a particular feature changed in the images, what is the effect of traversing other elements like (2), (3), (10) or (20)?

Author response:
> In the Fig. 6 where traversing a single latent variable is associated to a particular featured changed in the images, what is the effect of traversing other elements like (2), (3), (10) or (20)?  

We have tried traversing other elements, but in some cases the factors captured are not interpretable by humans. We argue that this is not a limitation of our work in particular but due to the ill-posedness of the problem itself. For example, in the first column of Fig. 1 in $\beta$-VAE paper [1], one can see that changing the azimuth also affects e.g. hair color. Without additional assumptions or inductive biases, it is too ambitious to hope that a factor captured by neural networks is always interpretable by humans because the generative model that transports $p(\boldsymbol z)$ to $p(\boldsymbol x)$ is not unique [2].

Reviewer comment (Evaluation):
In practice, in which scenarios should we apply GDM or GDM-F?

Author response:
### In which scenarios should we apply GDM or GDM-F?  
Compared to standard diffusion models, GDM can be particularly useful when we have some apriori knowledge of the underlying data-generating process. For instance, in video data, temporal ordering can be a natural design choice, so sequentially generating each frame could be more effective. In this case, the pixels in each frame consist of a group, and $k$ is equal to the number of frames.  

One could further consider generating some frames in parallel rather than generating each frame sequentially because the information in neighboring frames would be highly redundant. One could even generate some key-frames first and then interpolate between them, or generate key pixels/frames first and then fill in the rest across the spatial/temporal axis, which has shown to be effective in practice [3]. GDM unifies all these grouping and ordering strategies into a single unified framework.  

One might consider applying GDM-F if the most salient features of the data are captured by low-frequency components as in images. If this is the case, GDM-F opens up many interesting applications such as hierarchical latent space, extracting independent generative factors that are semantically meaningful, editing, etc., as we have shown in the paper.

Reviewer comment (Novelty):
Overall, I find this method has limited novelty. It is a simple extension of the model such as rectified flow by using a matrix interpolation function.

Author response:
Our method uses a very specific interpolation function in Eq. 10, not just any matrix-valued function. The connection between autoregressive models and cascaded diffusion models is only valid because of this specific choice of the interpolation function. To our knowledge, extending diffusion models to support the sequential generation of each group has not been done previously. The other two reviewers also agreed that the connection with autoregressive and cascaded diffusion models is indeed novel.

#### Other contributions
Our unified model also accompanies useful properties including hierarchical latent space, finding independent generative factors, etc., which have not been explored in previous work.

Reviewer comment (Theory):
When applied to the real problem, this approach seems adhoc. It is not explained and supported that why we should partite the data by a certain order and why such partition gives better generation quality.

Author response:
The aim of this paper is not to propose the best grouping and ordering strategy but to show that grouping and ordering is important design choices in sequential generative models, and we are already very explicit about this in the paper. The reviewer is correct that the reason why certain grouping and ordering work better than others is often not human-interpretable. Indeed, the progressive growing strategy of CDM is outperformed by bottom->top ordering or even CDM-inverse, and it is hard to interpret the meaningful distinction between the best and worst performing orderings of the block-wise grouping in Fig. 5. This is **not a limitation of our work but is in fact a significant observation** because it implies that the commonly used groupings and orderings, even if they appear to be a good design choice from a perspective of a human, could be not the best choice.

Reviewer comment (Experiments):
It seems that the number of groups, k, is a hyperparameter where not much discussion is given.

Author response:
### The number of groups $k$
$k$ can be chosen based on the user's demand. For example, if a user wants autoregressive models, $k$ should be set to $d$. If a user wants cascaded diffusion with two stages, $k$ should be 2. What types of sequential generation a user needs depends on the data type and the user's prior knowledge of it.

Also, setting $k$ to be greater than 1 enables various use cases such as obtaining hierarchical representation, extracting independent generative factors, image editing, and so on.

Reviewer comment (Experiments):
There are no comparisons to other generative models like VAE or GAN or even guided diffusion.

Author response:
### Comparison with VAEs, GANs, and guided diffusion
We would like to emphasize that the primary focus of our paper is to propose a unified framework for diffusion models and other sequential generative models and dissect interesting properties arising from it. 

Since we do not claim state-of-the-art generation performance, we believe it is not necessary to compare with VAEs and GANs, which are distinct families from diffusion and autoregressive models. Also, guided diffusion [4] is a method that utilizes classifier gradients for making a diversity-fidelity trade-off, which is not directly aligned with the scope of our work.

When $k=1$, GDM reduces to standard diffusion models, so the sample quality is on par with other diffusion models, given that the same architecture and training configuration are used. For $k>1$ case, please refer to Appendix A.2.

[1] Cascaded Diffusion Models for High Fidelity Image Generation

[2] PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

[3] simple diffusion: End-to-end diffusion for high resolution images

[4] Diffusion Models Beat GANs on Image Synthesis

Reviewer comment (Experiments):
How do you choose the number of groups?

Author response:
### The number of groups $k$
$k$ can be chosen based on the user's demand. For example, if a user wants autoregressive models, $k$ should be set to $d$. If a user wants cascaded diffusion with two stages, $k$ should be 2. What types of sequential generation a user needs depends on the data type and the user's prior knowledge of it.

Also, setting $k$ to be greater than 1 enables various use cases such as obtaining hierarchical representation, extracting independent generative factors, image editing, and so on.

Reviewer comment (Experiments):
If we set the number of groups equal to 1, what should we expect? How does it compare with the generated data from DDPM or DDIM?

Author response:
### If we set the number of groups equal to 1, how does it compare with DDPM or DDIM?
GDM is equivalent to previous diffusion models (or rectified flow) when $k=1$. For example, if we use the interpolation coefficients in Eq. 1, GDM reduces to the variance-preserving (VP) diffusion models. We made this even more clear in the last paragraph of Sec. 3.1. DDPM is a discretization of VP diffusion, and DDIM is a discretization of the probability-flow ODE.