Paper: Relay Diffusion: Unifying diffusion process across resolutions for image synthesis (ICLR-2024)

Reviewer comment (Experiments):
The advantage of the proposed method is emphasized with training efficiency, however, it only considers the setting without CFG where some competitors do not yield the optimal output. I believe that it is better also more meaningful to compare all methods in the setting with CFG where the existing methods achieve the best performance (optimal setting). Furthermore, when claiming the efficiency, I would also expect the comparison of the inference time/steps of the proposed method with the existing ones in their best optimal regime (including sampling steps in Table 3.).

Author response:
We actually want to make comparison of the intermediate results in the best regime but baselines including DiT[1], MDT[2], etc. only provide intermediate results without CFG (only report final results with CFG). When CFG is implemented, the performance of RDM in Figure 1 and other simlar comparisons will gain significant advancements as well.

Also, we have added a column of sampling NFE for a clearer comparison in Table 3. The new table is presented as follows:

https://i.postimg.cc/bwVyt8rh/new-table.jpg

Reviewer comment (Evaluation):
While the proposed method achieves slightly better sFID compared to MDT-XL/2-G on class-conditional generation ImageNet 256x256 (Table 3) (3.97 vs. 4.57, not really "a large margin" as stated in the abstract), however, most other metrics lag behind MDT-XL/2-G (such as FID, IS, and Recall). This indicates that to show its advantages over the existing approaches, it may need to present more evidence.

Author response:
sFID is an improved version of FID, using spatial features instead of the standard pool features. It better captures spatial relationships and gives image distributions a coherent high-level structure. MDT[2] only drops sFID from previous SoTA 4.60 to 4.57, which indicates an improvement of 4.57 to 3.97 to be a large margin.

From the modeling perspective, MDT has a complex structure 'asymmetric masking', making it tough to scale to higher resolutions and only perform well on 256x256 generation. In comparison, **RDM has better extendability and can easily scale to high-resolution image generation** without carefully adjusting the hyperparameters of noise schedule and architecture.

To prove such a claim, we have conducted further experiments at higher resolutions (512 and 1024), training new RDM models on CelebA-HQ for 256-to-512 and 256-to-1024 super-resolution tasks. We tested their performance through a 3-stage generation process, reusing the previously released models for 64 and 64-to-256 resolutions. The RDM models achieved **SoTA performance on CelebA-HQ 512 and 1024 generation**, successfully demonstrating RDM's capability in high-resolution image generation. The results are presented as follows:

https://i.postimg.cc/X71hwLST/celebahq-hres-results.jpg

Additionally, we have included these results in Appendix C.1.

Reviewer comment (Experiments):
Section 4.2 talking about CeleA-HQ, mentioned fewer training iterations while it is given with 50M and 820M trained images, making it a bit confusing that the dataset contained a hundred million images or that is just the total training steps. I recommend revising it more clearly and putting a column on the side of that table about the total training iterations or total training images.

Author response:
For 'Number of Training Images', this term refers to **the total number of image samples that the model processes during its training phase, which is equal to (iters x batch_size)**. 

We appreciate your advice and have added a column of training cost, in the form of (iters x batch_size), in Table1 and Table3 for greater clarity. For example, the new Table 3 is presented as:

https://i.postimg.cc/bwVyt8rh/new-table.jpg

Reviewer comment (Presentation):
What is the specific of each group in Table 3? It shows that the first column (Model) presents 5 groups (separated by \midrule) but unclear why is it divided like that.

Author response:
In the 'Model' column of the table, we categorize the results based on different settings of model training and sampling strategies. The categories, listed from top to bottom, are:

1. GANs
2. Diffusion Models without CFG(Classifier-Free Guidance) used during sampling.
3. Cascaded Diffusion Models.
4. Diffusion Models with dynamic-scale CFG.
5. Diffusion Models with fixed-scale CFG.

We hope this explanation clarifies how the models are organized in the table. To address any potential confusion regarding the table's presentation, we have restructured the divisions as follows:

1. GANs.
2. Diffusion Models without CFG.
3. Diffusion Models with CFG.

The new table is presented as:

https://i.postimg.cc/bwVyt8rh/new-table.jpg

This revised table arrangement is now updated in Section 4.2 of our paper.

Reviewer comment (Experiments):
I wondering if the number of data used in this paper is wrong, for example, Table 4 stated that 2500M training images were used for ImageNet64, is that 2.5 billion images? To my knowledge, ImageNet contains more than one million images, am I wrong? The same for all other datasets., does CelebA-HQ have 70 million images for training? Also, in Figure 1, is the horizontal axis the real number of images for training or it is just the iterations?

Author response:
For 'Number of Training Images', this term refers to **the total number of image samples that the model processes during its training phase, which is equal to (iters x batch_size)**. This aligns with commonly reported metrics for training costs used by baselines such as DiT[1] and MDT[2].

For our experiments, we used the original datasets from ImageNet and CelebA-HQ, which contain 1,281,167 samples and 30,000 samples, as elaborated in Section 4.1 of our paper. Similarly, in Figure 1, the horizontal axis illustrates the number of image samples that the model processes during training (iters x batch_size).



In light of potential misunderstandings regarding the content of our paper, we welcome any further questions or clarifications you may need. Please feel free to reach out with any inquiries.

Also, we believe that the demonstrated capability of RDM in high-resolution generation, **the achievement of SoTA on CelebA-HQ at 512 and 1024 resolutions**, marks a significant improvement.

If the responses provided above have satisfactorily addressed your concerns, would you please consider increasing your rating accordingly?



[1] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.

[2] Gao, Shanghua, et al. "Masked diffusion transformer is a strong image synthesizer." *arXiv preprint arXiv:2303.14389* (2023).

Reviewer comment (Experiments):
This paper focuses on high-resolution image generation. However, the biggest image resolution used in this paper is only 256x256, which is much smaller than the existing definition of ‘high-resolution‘. I would expect an experiment result that has a resolution at least 512 or 1024 to see whether RDM still works.

Author response:
**Re:** We have conducted further experiments at higher resolutions (512 and 1024), training new RDM models on CelebA-HQ for 256-to-512 and 256-to-1024 super-resolution tasks. We tested their performance through a 3-stage generation process, reusing the previously released models for 64 and 64-to-256 resolutions. The RDM models achieved **SoTA performance on CelebA-HQ 512 and 1024 generation**, successfully demonstrating RDM's capability in high-resolution image generation. The results are presented as follows:

https://i.postimg.cc/X71hwLST/celebahq-hres-results.jpg

Additionally, we have included these results in Appendix C.1.

Reviewer comment (Experiments):
The authors claim that any artifacts in the low-res images can be corrected in the high-res stage, we expect some qualitative cases in experiments to verify such a claim.

Author response:
**Re:** To demonstrate the capability of RDM to correct artifacts from the low-resolution stage, we conducted additional experiments comparing super-resolution generation between RDM and ADM-U. We used 64x64 samples from ImageNet as low-resolution inputs, and introduced  artifacts by adding 0.1 scale of Gaussian Noise. RDM successfully managed the noise, producing clean 256x256 samples, while ADM-U transferred the noise to its 256x256 samples. The comparison is presented as follows:

https://i.postimg.cc/Nj1pyVh5/artifacts.jpg

We have also included this comparison in our paper under Appendix C.3. 

Additionally, recognizing the potential for overstatement, we have amended our initial claim from 'any artifacts can be corrected' to 'artifacts could be corrected' for a more accurate representation.

Reviewer comment (Experiments):
My major concerns are about the experiment on really high-resolution images and some experiments to verify the superiority of the RDM further.

Author response:
**Re:** Thank you very much for your recognition! Please refer to our reply for **Weakness 1** and **Weakness 2** to check our additional experiments on further verification of RDM's capability.



From our perspective, the demonstration of RDM's capability in high-resolution generation, achieving **SoTA results on CelebA-HQ at 512 and 1024 resolutions**, represents a significant improvement. 

If the replies provided above have addressed your concerns, would you please consider revising your rating upwards?

Reviewer comment (Experiments):
Figure 1 (and the accompanying claims about faster convergence in the text) seems misleading: the model is not trained from scratch, but compares the convergence with from-scratch trained models.

Author response:
**Re:** This is a misunderstanding. The models can train well from scratch.

We guess the misunderstanding is from 'we directly use the released checkpoint from EDM in ImageNet in the 64 × 64 stage'. However, **the 64-to-256 model is a separate model from the first stage model** (64x64 model) and **both stages are trained from scratch**.

EDM[4] released its code, checkpoints as well as all the training hyperparameters for 64x64 models. This makes training the first stage of RDM on ImageNet for 64x64 generation a reproducible and repetitive work from EDM. We have verified the reproducibility of EDM by repeat the early training of EDM's 64x64 ImageNet model. Yet eventually, to avoid wasting computing resources, we directly borrow the pretrained EDM 64x64 ImageNet model as the first stage of RDM on ImageNet. In our calculation of total training consumption, the training cost of the pretrained model is also counted.

While in the experiment of CelebA-HQ, both stages of models are trained from scratch by ourselves.

Reviewer comment (Novelty):
f-DM (Gu et ICLR'22) can also be seen as a "relay" diffusion, but it is not even mentioned in the paper.

Author response:
**Re:** There are distinguishable aspects of difference between them as follows:

- Compare to f-DM using linear interpolation, we model the relaying resolutions with **Heat Dissipation (Blurring process)**.
- We introduce **Block Noise** to further bridge the gap between stages and corrupt low-frequency information, which is an essential part of RDM and brings significant improvement. 
- While f-DM uses one model for unified multiple stages’ generation, RDM uses **separate models to handle different stages**. Encoding multiple resolutions of generation into one model could be difficult for a UNet model to learn. In comparison, RDM breaks down the generation task into multiple, simpler tasks, which largely ease the training.
- On ImageNet 256x256, **the performance of RDM is far improved** compared to f-DM.
- RDM shows **greater extendability over f-DM**. For example, if the generation on higher resolution is desired, by RDM we can simply train a new super-resolution model on relaying and reuse the previous stages (exactly what we do to extend to CelebA-HQ 512 and 1024 generation). While for f-DM, the arrangement of the schedule and the training of the model, etc., are required to be conducted all over again.

Thank you very much for pointing out. f-DM and RDM share the idea to unify multiple resolutions of image generation in one diffusion schedule, which is a work we didn't previously draw attention to. We will definitely add the mention of f-DM and discussion of the difference to our paper later.

Reviewer comment (Experiments):
Would the method work on $512^2$, $1024^2$ and/or higher resolutions?

Author response:
**Re:** Please refer to our reply for **Weakness 1**. We have further verify the capability of RDM on high-resolution generation, achieving **SoTA on CelebA-HQ 512 and 1024**.

Reviewer comment (Experiments):
How does the method perform with from-scratch training? If the model cannot train well from scratch, then the claim about its simplicity compared to CDMs does not hold and should be removed. Also, it's not fair to claim the reduced amount of training steps for a non-from-scratch trained model.

Author response:
**Re:** This is a misunderstanding. The models can train well from scratch. 

In RDM, **the 64-to-256 model is a separate model from the first stage model** (64x64 model) and **both stages are trained from scratch**.

we have calculate the training cost based on the number of training samples (iters x batch_size) on 256 resolution and compared with other baselines. This method aligns with commonly reported metrics for training costs used by baselines such as DiT[1] and MDT[2]. To make a fair comparison, **we have included two stages of RDM in our calculation of the training cost**. As explained in Appendix B, the FLOPS of the 64-resolution model are less than 1/10th of the 256-resolution model. So we have incorporated the training cost of the low-resolution stage as 1/10th into the total cost. For greater clarity, we have now included the training cost calculation process in the 'Architecture and Training' section of Section 4.1.

Reviewer comment (Evaluation):
Can it be the case that the reduced amount of sampling steps is due to the 2-nd order sampler? For example, MDT uses DDPM sampler, which was shown by EDM to require more sampling steps.

Author response:
We supplement the experiments on the first-order sampler. All models use first-order sampler for generation on ImageNet-256 with 80 sampling steps. RDM still outperforms MDT and DiT. The result is as follow:

https://i.postimg.cc/hP6cMYCV/ddim-comparison.png

Reviewer comment (Evaluation):
For a fair comparison, please report the training/inference costs of the developed model vs the baselines (where possible).

Author response:
**Re:** We have added training cost in Table 1 and Table 3. The new Table 3 is presented as follows, for example:

https://i.postimg.cc/bwVyt8rh/new-table.jpg

Please refer to the reply for **Weakness 2** for a detailed interpretation of the calculation of the training cost.

Reviewer comment (Evaluation):
Please, update the claims about SotA FID based on the unbalanced FID instead of the class-balanced FID.

Author response:
**Re:** We only claim the best sFID but not FID on ImageNet 256. Also, without class-balancing, RDM still achieves SoTA sFID on ImageNet 256x256, and the second place ranking on FID (inferior to MDT). sFID is an improved version of FID, using spatial features instead of the standard pool features. It better captures spatial relationships and gives image distributions a coherent high-level structure.

Reviewer comment (Experiments):
Why not try dynamic CFG as well to further improve FIDs? Or it was tried and didn't work?

Author response:
**Re:** In the experiment of ImageNet 256x256, the low resolution stage of RDM (64x64) is trained without condition dropout,. Therefore the low-resolution stage of RDM is not able to implement dynamic CFG (or fixed CFG), while the improvement of only implementing CFG on the second stage is much restricted.

Reviewer comment (Novelty):
It would be good to describe the differences with f-DM.

Author response:
**Re:** There are distinguishable aspects of difference between them. Please refer to our reply for **Weakness 5** for a detailed description of the differences.

Thank you very much for pointing out. f-DM and RDM share the idea to unify multiple resolutions of image generation in one diffusion schedule, which is a work we didn't previously draw attention to. We will definitely add the mention of f-DM and discussion of the difference to our paper later.

Reviewer comment (Writing):
The transition from section 3.1 to section 3.2 is a little incomprehensible; why combining block noise and blurring diffusion model in the second stage needs further explanation. Besides, from the equation and algorithm, I can’t find an explicit connection between low-resolution diffusion and high-resolution diffusion; such an explicit statement may be necessary for reader to understand.

Author response:
Please refer to the replies for **Question 3** and **Question 6**, which explained the connection of low-res and high-res stages. The incomprehensibility may be largely from the error in Algorithm 1, where the starting point of sampler is supposed to be '$X_N$ ~ low-resolution images' and we have corrected it in Algorithm 1. From the perspective of forward process, the final state of a patch-wise blurring process is a nearest-upsampled version of a low-resolution image. This guarantees a seamless integration between different resolutions. 

We have explained the combination of block noise and blurring diffusion in the reply for **Question 5**. The blurring process interpolates between an upsampled low-resolution start and a high-resolution end, while the noise corruption meantime performs a standard diffusion process.

Reviewer comment (Experiments):
Missing experiments: How does an end-to-end model for high-resolution images by introducing block noise in early diffusion steps perform? The author mentions it on page 5 but does not explain it.

Author response:
We have conducted further ablation study of block noise on an end-to-end model and added it in Appendix C.2. We trained ADM on CelebA-HQ 256x256 w/ and w/o block noise and tested them on FID-5K. The result is as follows:

https://i.postimg.cc/GhH5mbjQ/block-noise-end2end.jpg

We found that ADM with block noise achieves better and more stable FID performance, which verifies the effectiveness of the block noise on end-to-end models.

Reviewer comment (Theory):
On page 2, “Training Efficiency”, the author takes the cascaded method as a solution to mitigate memory and computation costs, but the cascaded method still needs to train and inference in the highest resolution. How does such a setting mitigate memory and computation cost? For RDM, the problem still exists.

Author response:
Compared to end-to-end high-resolution training, a cascaded pipeline breaks down the generation task into multiple, simpler tasks across increasing resolutions. This approach significantly reduces the burden of training diffusion models at high resolutions. Although training at lower resolutions is still necessary, it requires far less computational resources, as evidenced by the reduced FLOPS. For example, In RDM, training for 64x64 resolutions needs only 1/10 FLOPS compared to 256x256 models.

This efficiency is a key reason why cascaded configurations have become the standard in current SoTA text-to-image models, such as DALL-E[1] and Imagen[2], as they help to 'mitigate memory and computation costs'.

Furthermore, RDM is even more efficient than typical cascaded pipelines. In RDM, the high-resolution stage begins diffusion from a midpoint in the schedule, cutting more than half of the sampling steps compared to conventional cascaded models. In contrast, these models typically generate through an entire diffusion schedule at the high-resolution stage.

Reviewer comment (Presentation):
I can’t understand why, for Fig.2, the high-frequency period is meaningless, and the difference can be neglected. More explanation may be required.

Author response:
In Figure 2(a), the high-frequency component is meaningless because, to make a meaningful comparison in the same frequency space, we first upsample the 64x64 image to a 256x256 image. Therefore, in the Signal-to-Noise Ratio (SNR) curve, the information in the high-frequency part is a product of the upsampling-and-interpolation process and holds little significance. However, the high-frequency part in Figure 2(c) is meaningful. The block noise in this figure does not significantly corrupt the high-frequency information. Consequently, we opted for a mixture of Gaussian noise and block noise, which more effectively corrupts both the low-frequency and high-frequency information in the image.

Reviewer comment (Theory):
How to choose $D_{T}^{p}$ to guarantee $VD_{T}^{P}V^{T}x_{0}$ in the same distribution as $x^{H}$; this is the key to connect low-resolution diffusion and high-resolution diffusion.

Author response:
We provide the detailed derivation of $D^P_T$ in Appendix A.1. $D^P_T$ is defined as a patch-wise blurring transformation matrix. In a patch-wise blurring process, the final state of the image results in each patch having a uniform pixel value, which is the average value of the patch at the beginning. For a 4x4 patch in the RDM setting, this approach ensures that the final state of the patch-wise blurring will be a nearest-upsampled version of a low-resolution image. This guarantees a seamless integration between different resolutions.

Reviewer comment (Experiments):
Adding the training iterations of each method in Tables 1, and 3 may make it more readable. The comparison may be a little unfair because RDM’s low-resolution diffusion model is pre-trained.

Author response:
The claim of the unfair comparison may be a misunderstanding. We have factored in two stages when calculating the training cost. We calculate the training cost as the number of training samples (iters x batch_size) on 256 resolution. As mentioned in Appendix B, the FLOPS of the 64-resolution model are less than 1/10 of the 256-resolution model, so we have incorporated the training cost of the 64-resolution stage as 1/10 to the total cost. Same incorporation is also conducted in the ablated comparison of sampling steps in Section 4.3. For greater clarity, we have also detailed the process of calculating the training cost in the 'Architecture and Training' section, found in Section 4.1.

Thank you for the suggestion. In response to considerations about the different training batch sizes used in previous works, we have revised our paper to add a column titled 'Number of Training Samples (iter x batch_size)' in the table as follows.

https://i.postimg.cc/L6BLG8hX/table3-new.jpg

We hope this will make a more clear comparison.

Reviewer comment (Theory):
For the text description below equation (11), why blurring corruption and block noise corruption can be considered independently, i.e., why can we just replace $\epsilon$ with $\tilde{\epsilon}$? Is there any formal proof?

Author response:
We replace $\epsilon$ with $\tilde{\epsilon}$ corresponding to the forward process in Eq.8: $x_t={{VD^p_tV^\mathrm{T}}}x+\frac{\sigma}{\sqrt{1+\alpha^2}}(\mathbf{\epsilon}+\alpha\cdot$$\text{Block}$[s]$(\mathbf{\epsilon'}))$. Since the mixing ratio is fixed, it retains linear properties and can directly replace Gaussian noise in the sampler. This adjustment better bridges the gap between multiple resolutions, achieving improved results.

The blurring corruption process and the block noise corruption process evolve separately depending on the time $t$. The blurring process interpolates between an upsampled low-resolution start and a high-resolution end. The noise corruption otherwise performs a standard diffusion process. These two processes are orthogonal. We have also provided a detailed proof of the sampler deduction for further verification of the sampler in Appendix A.3.

Reviewer comment (Reproducibility):
For Algorithm 1 in A.4, it seems that the sampling algorithm does not contain the process of the low-frequency diffusion model. Please clarify.

Author response:
Thanks for your attentiveness. This is a writing error and actually, the original first line in alg1 does not match the actual practice, which is supposed to be “sample $X_N$ ~ low-resolution images”, we have corrected it in our paper. The Algorithm Section is now presented as follows:

https://i.postimg.cc/KcBwRw0T/rdm-sampler.jpg

For the first stage of RDM, we use a standard diffusion sampler borrowed from EDM[3] as follows:

https://i.postimg.cc/4yjLMN3w/edm-sampler.jpg