Paper: Towards Enhanced Controllability of Diffusion Models (ICLR-2024)

Reviewer comment (Theory):
So called style condition looks more like another object feature condition. The style image thus is quite close to the content image on the manifold.

Author response:
As shown in Fig. 7 in the main paper and Fig. 18-19 in the supplementary, our style encoder encodes non-structural information, e.g., textual information (in LSUN-church and AFHQ) and facial attribute information (in FFHQ) while structural information is encoded in the content encoder. (Please see Table 4 if the reviewer wants to see the quantitative verification.) Similarly, we strongly believe that the low-frequency texture information would be encoded in our style encoder if we train our framework with (only) Art dataset. This is because the non-structural information in Art dataset mainly includes the low-frequency texture of the arts, and the style encoder is designed to learn the non-structural information as verified by the aforementioned experiments.

Reviewer comment (Experiments):
All examples of content and style pairs are quite similar, i.e. both faces, cat and dog, cat and tiger. This undermines the effectiveness of this method on more general cases.

Author response:
FFHQ, AFHQ, and LSUN-church datasets are widely-used benchmark datasets to evaluate the performance of generative models in Image Translation tasks [1-4].

[1] Multimodal Unsupervised Image-to-Image Translation, Huang et al., ECCV'18

[2] StarGAN v2: Diverse Image Synthesis for Multiple Domains, Choi et al., CVPR'20

[3] Swapping Autoencoder for Deep Image Manipulation, Park et al., NeurIPS'20

[4] Diffusion Autoencoders: Toward a Meaningful and Decodable Representation, Preechakul et al., CVPR'22

Reviewer comment (Reproducibility):
Can you explain how do you apply your method to SD text-to-image generation in detail?

Author response:
given a caption of "photo of a red bird and a yellow flower", we need to compute three terms for obtaining $\tilde{p}$, the GCDM formulation (Eq. 3 in the main paper). In specific, $\epsilon(x_t, t, c_1)$ is obtained by forwarding "photo of a red bird" and $\epsilon(x_t, t, c_2)$ is obtained by forwarding "photo of a yellow flower". Lastly, the joint term $\epsilon(x_t, t, c_1, c_2)$ is obtained by forwarding "photo of a red bird and a yellow flower".

Reviewer comment (Experiments):
Have you tried using another style image that is more different in style instead of just content? It seems more like a object feature transfer case in your experiments. Can you try an obviously different style image, such as a painting etc?

Author response:
As shown in Fig. 7 in the main paper and Fig. 18-19 in the supplementary, our style encoder encodes non-structural information, e.g., textual information (in LSUN-church and AFHQ) and facial attribute information (in FFHQ) while structural information is encoded in the content encoder. (Please see Table 4 if the reviewer wants to see the quantitative verification.) Similarly, we strongly believe that the low-frequency texture information would be encoded in our style encoder if we train our framework with (only) Art dataset. This is because the non-structural information in Art dataset mainly includes the low-frequency texture of the arts, and the style encoder is designed to learn the non-structural information as verified by the aforementioned experiments.

Reviewer comment (Experiments):
The method adds two additional encoders and needs to be trained from scratch which makes it hard for larger text-to-image diffusion models which require a large amount of time training.

Author response:
We believe the two additional encoders can be trained by fine-tuning the pre-trained Diffusion Models (not necessarily training from scratch). FastComposer [1] and ControlNet [2] have shown that it is possible to train their additional encoders while fine-tuning or fixing the off-the-shelf pretrained Diffusion Models. Given the recent research reports in this field, it should be possible to train the additional encoders in a fine-tuning manner.

[1] FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention, Xiao et al., ArXiv'23

[2] Adding Conditional Control to Text-to-Image Diffusion Models, Zhang et al., ICCV'23

Reviewer comment (Evaluation):
From Figure 15 on the ablation of different weighting schedules, the difference seems little between the sigmoid and the linear ones.

Author response:
thank you for pointing out a valid point. Even though the performance difference is not huge, we can have additional extended control if we use the sigmoid scheduling. Please see Fig. 1 in the supplementary. The results on the fifth and sixth columns from the right (compared to the baseline on the third column from the left) show that we can have additional control, which is not achievable in the linear scheduling. Please note that both the sigmoid and the linear scheduling are proposed by us.

Reviewer comment (Experiments):
For the comparison with DiffAE+MagicMix, it seems that the image quality is really bad, does the author try different parameters over this? The paper states that the model takes $x_600$ as input but no other results are presented here.

Author response:
For this experiment, we used $K_{max}=0.6T$ and $K_{min}=0.3T$ with $v=0.5$, which are suggested by MagicMix [3] as the best hyperparameters. Please see "Preserving more layout details" paragraph on page 8 of their paper.

[3] MagicMix: Semantic Mixing with Diffusion Models, Liew et al., ArXiv'22

Reviewer comment (Evaluation):
Overlapping latent spaces: Are the latent spaces independent of each other? How can this be verified?

Author response:
Thank you for the good question. Since our encoders are trained fully unsupervised manner (i.e., no attribute label is used specifically for disentangling the encoders), they are not strictly independent. However, still they are trained to mainly encode different features of the image. It can be supported by three experiment results. First, we can see that the principal components of the style space are high-level attributes (Fig. 5 in the supplementary) while the principal components of the content space are more related to the spatial information, e.g., pose, and face size (Fig. 6 in the supplementary). Second, Table 4 in the main paper shows that the more style information is given ($\lambda=0.25$) the more attribute from the style image is applied (and thus less attribute from the content image is maintained). In other words, lining up with the PCA experiments, the classifier-based experiment shows that the style space mainly encodes high-level semantics. Third, by seeing Fig. 8 in the main paper and Fig. 19 in the supplementary, we can see that the structure of the building comes from the content image while the color and the texture of the building come from the style image.

Reviewer comment (Experiments):
Is there a possibility to extend the framework to more than two latent spaces, and if so, how would this affect the model's complexity and performance?

Author response:
Similar to the previous studies [1,2], our content encoder mainly encodes spatial information and the style encoder mainly encodes non-spatial information. And it is done by leveraging the inductive bias of Diffusion Models under the sophisticated design of our proposed framework. Thus, it is not possible to add the third encoder under the current framework. However, with our proposed two spaces, we can have extended controllability such as finding semantically meaningful direction vectors (Fig. 5-7 in the supplementary) and conducting interpolation in the latent spaces (Fig. 8-10 in the supplementary)

[1] Multimodal Unsupervised Image-to-Image Translation, Huang et al., ECCV'18

[2] Swapping Autoencoder for Deep Image Manipulation, Park et al., NeurIPS'20

Reviewer comment (Theory):
The formulation of the style coder is based on human heuristic, instead of data driven. How can you tell what styles can be controlled?

Author response:
Thank you for the good question. Since our encoders are trained fully unsupervised manner (i.e., no attribute label is used specifically for disentangling the encoders), they are not strictly independent. However, still they are trained to mainly encode different features of the image. It can be supported by three experiment results. First, we can see that the principal components of the style space are high-level attributes (Fig. 5 in the supplementary) while the principal components of the content space are more related to the spatial information, e.g., pose, and face size (Fig. 6 in the supplementary). Second, Table 4 in the main paper shows that the more style information is given ($\lambda=0.25$) the more attribute from the style image is applied (and thus less attribute from the content image is maintained). In other words, lining up with the PCA experiments, the classifier-based experiment shows that the style space mainly encodes high-level semantics. Third, by seeing Fig. 8 in the main paper and Fig. 19 in the supplementary, we can see that the structure of the building comes from the content image while the color and the texture of the building come from the style image.

Reviewer comment (Presentation):
Poor formatting. Why are some equations not numbered in the main text? Some variables are never defined in the un-numbered equations. (See questions.)

Author response:
We will make a revision by adding a number in the first equation and defining $\epsilon$ in the first equation. $\epsilon$ is a denoising UNet.

Reviewer comment (Writing):
Unclear exposition. The concepts and notation in Sections 3.1 and 3.3 are difficult to reconcile. What is $\epsilon(x_t, t, c_1, c_2)$? Are $z_s$ and $z_c$ just the conditions $c_1$ and $c_2$, respectively? How is this a generalization of CDM?

Author response:
$\epsilon(x_t, t, c_1, c_2)$ is a denoising UNet taking two conditions. As mentioned right above Definition 3.1 in the main paper, $c_1$ and $c_2$ are $z_s$ and $z_c$, respectively. As for the question of how GCDM can generalize CDM, as shown in Eq. 5, given $\lambda=0$, GCDM formulation simplifies to CDM.

Reviewer comment (Novelty):
Limited novelty. It appears to me that the main contribution involves the addition of joint conditions, rather than simple independent conditions (CDM) in Eq. 3. This is overall a rather simple addition from CDM. Proposition 3.2 is not very interesting, as it appears to simply state that GCDM generalized CDM (which is known) and CDM generalized CFG (which is known).

Author response:
As mentioned in the introduction of the main paper and overview of the supplementary material (Section A), our technical novelties are not limited to extending CDM to GCDM. We are proposing three methods for enhancing controllability of Diffusion Models; 1. a framework for having content and style latent space, 2. timestep scheduling during the sampling, 3. GCDM. Moreover, regarding a justification of GCDM, even though its form can be seen as a simple extension of CDM, we argue that the simplicity cannot be a lack of novelty if the extension makes something better that is meaningful and was not achievable beforehand. For example, GCDM gives more controllability to the Diffusion sampling algorithm by merging CDM and joint guidance in the form of a geometric average (which helps us to sample from both the joint and the non-joint terms while maintaining the amount of the conditional signal compared to the unconditional signal.) By doing so, we have shown that we can have one more dimension to control (i.e., realism) compared to CDM. The low FID score of the high $\lambda$ from Table 1 and Table 2 are quantitatively supporting our argument. Also, Fig. 5 is qualitatively verifying our argument.

Reviewer comment (Theory):
Lack of scalability. In the current formulation, it is not clear to me how the proposed method will scale well with increasing conditions. The number of joint conditions required in Eq. 3 seems to grow combinatorially with the number of conditions.

Author response:
Thank you for pointing out a valid point. As pointed out, naively extending GCDM to $N$ conditions requires computing a lot of terms, e.g., $\epsilon(x_t, t, c_1, c_2, ..., c_N), \epsilon(x_t, t, c_1, c_2, ..., c_{N-1}), \epsilon(x_t, t, c_1, c_2, ..., c_{N-2}), ...$. However, the scalability issue can be resolved by only computing the full joint and the single conditionals, i.e., $\epsilon(x_t, t, c_1, c_2, ..., c_N), \epsilon(x_t, t, c_1), \epsilon(x_t, t, c_2), ..., \epsilon(x_t, t, c_N)$. Thus, given $N$ conditions (where $N \geq 2$), Eq. (3) in the main paper turns to:

$$
\nabla_{x_t} \log \tilde{p}_{\alpha, \lambda, \beta_1, \beta_2, ..., \beta_N}(x_t|c_1,c_2, ... ,c_N) = \epsilon(x_t,t) + \alpha \left[ \lambda \Bigl( \epsilon(x_t, t, c_1,c_2, ..., c_N) - \epsilon(x_t, t) \Bigr) + (1-\lambda) \sum _{i=1}^N \beta_i \Bigl( \epsilon(x_t,t,c_i)- \epsilon(x_t,t) \Bigr) \right]
$$

Additionally, the computational cost of the changed form can be even reduced if we can have a prior, e.g., which condition would be more emphasized so that only a few single conditionals (out of $N$-single conditionals) and the full joint will be computed during the diffusion sampling process.

Reviewer comment (Experiments):
Lack of ablation study. The authors propose several orthogonal improvements (e.g. timestep scheduling and an adaptive group normalization, a.k.a. AdaGN, layer). How much do these aspects contribute to the performance of the model?

Author response:
our technical novelties are 1. having two latent spaces, 2. timestep scheduling, 3. GCDM. The performance with/without timestep scheduling and the performance with/without GCDM are reported in Table 3 and Fig. 6 in the main paper. Details are briefly but clearly mentioned in "Effect of Timestep Scheduling" paragraph on page 8. 

Regarding the first novelty (having two latent spaces), its performance improvement can be seen by comparing DiffAE+SDEdit in Table 1 (single latent space) and CDM, w/o schedule in Table 3 (two latent spaces). As can be seen from the comparison, by setting two latent spaces, we can get better improvement in FID. Also, having two spaces gives us richer controllability as shown in Section F in the supplementary (please skim through the figures). Please note that we did not argue that the variant of AdaGN is our technical novelty.

Reviewer comment (Reproducibility):
Significant increase in hyperparameters. It appears that at least 9 new hyperparameters are introduced in this conditioning method ($a$, $b$, $\alpha$, $\lambda$, $\beta_i$, $t_1$, $t_2$, $t_3$). It is not clear to me how to choose these hyperparameters, and to what extent the observed empirical improvements can be attributed to hand-tuning of these hyperparameters.

Author response:
There are five hyperparameters in our proposed sampling methods ($a, b$ from timestep scheduling and $\alpha, \lambda, \beta$ from GCDM). Please note that $t'$s are timestep embeddings. Since we agreed with the complexity of the hyperparameter tuning, we reported specific visual guidance on how to control each hyperparameter (in Fig. 1 in the supplementary). Please see the figure.
Even though the number of hyperparameters can be complex in the first place, we believe it enables more subtle control for users to find the best output, as shown in the wide range of controllability in Fig. 1 in the supplementary.

Reviewer comment (Writing):
What is $\epsilon_t$ in the un-numbered (first) equation in Section 3.1?

Author response:
We will make a revision by adding a number in the first equation and defining $\epsilon$ in the first equation. $\epsilon$ is a denoising UNet.

Reviewer comment (Experiments):
How are the various new hyperparameters $a$, $b$, $\alpha$, $\lambda$, $\beta_i$, $t_1$, $t_2$, $t_3$ chosen? How robust are the results to perturbations in these hyperparameters?

Author response:
There are five hyperparameters in our proposed sampling methods ($a, b$ from timestep scheduling and $\alpha, \lambda, \beta$ from GCDM). Please note that $t'$s are timestep embeddings. Since we agreed with the complexity of the hyperparameter tuning, we reported specific visual guidance on how to control each hyperparameter (in Fig. 1 in the supplementary). Please see the figure.
Even though the number of hyperparameters can be complex in the first place, we believe it enables more subtle control for users to find the best output, as shown in the wide range of controllability in Fig. 1 in the supplementary.