Paper: SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (ICLR-2024)

Reviewer comment (Presentation):
The rightmost examples in Fig. 5 look confusing to me: does (512, 512) mean the whole images are cropped out and there should be nothing inside (the caption says images are from a 512^2 model)? In this case, why does the top example still show basically the same image as the leftmost one while the bottom example shows mainly the background?

Author response:
Dear Reviewer rcq7,

Thank you for your careful review of our manuscript and your appreciation for our efforts. To address your questions:

In general, crop-conditioning can be applied as a finetuning technique and address issues observed in previous versions of SD when random cropping expansion from training carried over into the synthesized samples. However, as described in Section 2.3, crop-conditioning is particularly beneficial when we train at a fixed resolution, e.g. 512x512 pixels. For the final multi-aspect model, crop-conditioning does not need to be applied. The effectiveness of size conditioning is shown in Tab. 2. In Fig 5 the model is sampled with different inputs for the crop conditoning, but no actual cropping is happening. The rightmost examples in Fig. 5 are OOD, and the behavior here is therefore less predictable.

Reviewer comment (Experiments):
Also, SDXL still relied on two-stage training where a high-quality encoder is required. I wonder whether the two stage can be combined and lead to further performance boost?

Author response:
Single-Stage Training: We agree that single-stage training is an exciting avenue for future research, and both [1] and [2] present promising results in this direction. Currently, two-stage, latent diffusion approaches give the best results [5] for text-to-image synthesis and we follow this paradigm.

Reviewer comment (Experiments):
As discussed in your future work section, can SD be combined into single-stage training and further boost the performance?

Author response:
Single-Stage Training: We agree that single-stage training is an exciting avenue for future research, and both [1] and [2] present promising results in this direction. Currently, two-stage, latent diffusion approaches give the best results [5] for text-to-image synthesis and we follow this paradigm.

Reviewer comment (Experiments):
Can similar training pipeline introduced in this paper be applied to 3D diffusion training, and what are the challenges?

Author response:
For 3D, the main issue with directly scaling diffusion-based approaches is data scarcity [3]. However, we hypothesize that multimodal approaches that combine explicit 3D training with 2D and video training can eventually help to overcome this issue. The micro-conditioning techniques that we introduced in our paper can be adapted and applied to other modalities, see e.g. [4]

Reviewer comment (Novelty):
The manuscript appears to lack detailed explanations on how the autoencoder (AE) has been improved or advanced.

Author response:
As we describe in Sec 2.4, the autoencoder has been improved by adding EMA-based weight tracking and increasing the batch size used during training. Since reviewer pUdr also asked about the autoencoder, we answer that in our general response. TL;DR: EMA consistently gives better reconstruction metrics. Larger batch sizes provide better gradient updates.

Reviewer comment (Experiments):
Could you give me some intuition and the effect of using a combination of language models (CLIP ViT-L & OpenCLIP ViT-bigG)? Is it crucial to use all of them?

Author response:
We empirically found that these CLIP text encoders contain complementary semantic knowledge, for example about visual styles.

Reviewer comment (Experiments):
I wonder the author's opinion about using TransFormer architecture to train the DMs. Could you share more information on "no immediate benefit"?

Author response:
We briefly experimented with the DiT[1] and UViT[2] architectures, but found that the images produced were not of higher quality. However, this could be due to the suboptimal choice of hyperparameters during the short experimentation phase. we remain optimistic that these architectures are good candidates for scaling diffusion models to much larger sizes, and that the simplified architectures may ease engineering efforts.

Reviewer comment (Experiments):
There could be more explanations on why your current autoencoder provides better reconstruction performance: which factor contributes more, the EMA choice, or larger batch size?

Author response:
Since reviewer n7kc also asked about the autoencoder, we answer that in our general response. TL;DR: EMA consistently gives better reconstruction metrics. Larger batch sizes provide better gradient updates.

Reviewer comment (Experiments):
How well does the refinement model work when refining other real images, or images generated by previous SD versions? That is, instead of directly refining latents in VAE latent space, but refining the VAE encoded latents from other real / generated images.

Author response:
The refiner can be applied both on real images (in the SDEdit [1] fashion), or to “finish” the denoising process from the base model. Both work equally well, note that the latter option is cheaper when synthesizing from a text prompt. We find that the refiner is also able to add high frequency details when utilizing the SDEdit approach.