Paper: Experts on Demand: Dynamic Routing for Personalized Diffusion Models (ICLR-2024)

Reviewer comment (Novelty):
Novelty is limited. The MoE structure is widely applied for the AIGC models, especially large foundation models. As for the diffusion models, MoE structure is not firstly introduced for the text2image diffusion models, e.g. [RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths]. The sparse parameter pruning with MoE is also widely used for the large dense model, e.g. [Task-Specific Expert Pruning for Sparse Mixture-of-Experts]. Although this paper introduces a sparse MoE structure to prune the large scale of parameters of dense diffusion models, the novelty of the proposed methods is limited.

Author response:
The novelty we wish to emphasize that while pursuing general, large-scale models, it's imperative to consider the specific needs of users. The long-term use of models for specialized purposes is a common occurrence, yet it is often overlooked by the community. Employing methods like layer removal and MoE are merely tools for achieving this objective.

Reviewer comment (Evaluation):
Experiments are probably not sufficient. It is not easy to evaluate the performance of the image generation method. The evaluation metrics, KID and FID are actually insensitive to the generated image quality and these metrics cannot evaluate the image quality well.

Author response:
Thank you for pointing out the issues with the evaluation metrics. We have provided numerous visualizations of generated images, along with comparisons to baseline models, in the appendix of our paper. We will place greater emphasis on this aspect in the updated version of our paper.

Reviewer comment (Reproducibility):
Please clarify more details, e.g. the expert balance operation, of the proposed MoEDM.

Author response:
In Section 3.2 of our paper, we have clearly stated: "Fortunately, in diffusion models, the time step t is always known, allowing for targeted activation based on t. This makes the gated mechanism of MoEDM a training-free approach." This implies that we can directly select experts with smaller id based on a smaller value of t. And it is crucial to emphasize that the speed improvement of MoEDM is not related to the number of experts. This is because the number of experts does not affect the volume of parameters involved in the computation at each step of the diffusion sampling process.

Reviewer comment (Experiments):
Please add more image quality comparison between the proposed MoEDM and baseline.

Author response:
We have provided numerous visualizations of generated images, along with comparisons to baseline models, in the appendix of our paper. We will place greater emphasis on this aspect in the updated version of our paper.

Reviewer comment (Reproducibility):
The paper could benefit from providing an explicit pruning algorithm to enhance understanding.

Author response:
- As mentioned in Equation 1, after zeroing out a specific channel, we conduct a diffusion sampling and measure the difference between the generated image and the standard image. This process is indeed time-consuming, but we have observed a consistent pattern across different categories and models: the layers at both ends are absolutely crucial, while the importance of the middle layers is much lower. Consequently, in our MoEDM approach, we adopt the strategy of directly removing the middle layers, eliminating the need for further assessment of parameter importance.
  -  In Section 3.2 of our paper, we have clearly stated: "Fortunately, in diffusion models, the time step t is always known, allowing for targeted activation based on t. This makes the gated mechanism of MoEDM a training-free approach." This implies that we can directly select experts with smaller id based on a smaller value of t.
  - As mentioned above, our specific pruning algorithm involves removing the layers located in the middle of the model.

Reviewer comment (Evaluation):
The performance for high-resolution images appears to be only marginally improved, which could be discussed in more detail.

Author response:
It is important to emphasize that our study focuses not on improving the model's performance, but on enhancing efficiency while maintaining existing performance levels. As shown in Table 3 of our paper, we have significantly increased the model's operating speed while maintaining the performance of high-resolution models on specific tasks.

Reviewer comment (Experiments):
Ablation study regarding the number of experts should be included, as most experimental results consistently show a 50% boost in inference speed.

Author response:
Thank you for pointing out the need for more ablation studies. We will supplement our studies with these additional ablation experiments. However, it is crucial to emphasize that the speed improvement of MoEDM is not related to the number of experts. This is because the number of experts does not affect the volume of parameters involved in the computation at each step of the diffusion sampling process.

Reviewer comment (Experiments):
The experiments are limited to U-Net-based models, and it's uncertain whether the proposed method is applicable to various architectures such as DiT [1].

Author response:
Thank you for pointing out the need for experiments on DiT, and we will supplement our studies with these additional experiments on DiT.

Reviewer comment (Experiments):
The paper lacks baselines. For example, [2] can be served as a pruning baseline.

Author response:
We emphasize that this paper (Structural Pruning for Diffusion Models) was made public after the submission deadline for ICLR 2024, and we will include this work as a baseline in our supplementary materials.

Reviewer comment (Reproducibility):
How many random seeds are used throughout the experiments?

Author response:
Regarding randomness, we did not deliberately control it in our study. Random seeds can be randomly selected during the sampling process.

Reviewer comment (Writing):
The term "an all-encompassing arsenal" is introduced in the second paragraph of the Introduction but is not explained or mentioned later, leading to confusion regarding this concept.

Author response:
"All-encompassing arsenal" means that the model is capable of accommodating a wide variety of user requirements.

Reviewer comment (Presentation):
In the Introduction, the paper mentions “…, deploying a general-purpose diffusion model is not just inefficient but egregiously wasteful”. However, the paper does not provide a comprehensive explanation of why it is inefficient and wasteful, and Figure 1 also fails to illustrate this point.

Author response:
As we mentioned at the beginning of the Introduction, large-scale, general-purpose diffusion models contain 3.5 billion or even more parameters. Such a vast number of parameters inevitably leads to inefficiency, a consensus in the academic community. Additionally, many users are likely to only require specific functionalities of diffusion models for extended periods. For example, a pet store may only attempt to create images of pets, thus deploying an ultra-large-scale model would be wasteful.

Reviewer comment (Writing):
The term 'minimal computational cost' is used in the paper, but it lacks a clear definition, making it a concept that may lead to confusion.

Author response:
The term "minimal computational cost" refers to our aim to significantly reduce the runtime of diffusion models by decreasing the number of parameters, particularly for user-specific tasks. We will also provide detailed and clear explanations in the updated version of our paper.

Reviewer comment (Theory):
The reason why setting diffusion models to zero works is not explained.

Author response:
This is because, in our study, the task of image synthesis is specific, rather than for a general purpose.  Simultaneously, due to the integration of the Mixture of Experts (MoE) algorithm, our approach can achieve performance equivalent to that of the original model.

Reviewer comment (Experiments):
The paper only provides 64x64 and 256x256 resolution outputs, while diffusion-based models can generate high-resolution results of 512x512 (e.g., Stable Diffusion).

Author response:
Experiment 1. Thank you for pointing out the need for experiments targeting higher resolutions. We will supplement our studies with these additional experiments.

Reviewer comment (Experiments):
It would be more valuable to see if this method can be built with more diffusion models, like Stable Diffusion and DeepFloyd-IF.

Author response:
Experiment 3. Thank you for pointing out the need for experiments on other personalized diffusion models. We will supplement our studies with these additional experiments.

Reviewer comment (Writing):
The use of distillation to enhance data efficiency is an interesting approach, though its integration with the other proposed methods (aiming for efficiency improvements) appears tangential and warrants a clearer rationale.

Author response:
Indeed, the removal of intermediate layers significantly enhances efficiency, but the resultant performance loss is not negligible. To address this, we introduce the MoE mechanism to compensate for these performance deficits, as emphasized in our ablation experiments. Furthermore, we will conduct additional experiments on various datasets and present the results in the updated version of our paper.

Reviewer comment (Writing):
The clarity and structure of the paper would greatly benefit from revision. For instance, the processes following the removal of intermediate layers, including whether the model undergoes fine-tuning with the original dataset, are not clearly outlined.

Author response:
Thank you for pointing out the clarity and structure of our paper. In fact, in our ablation experiments, we have demonstrated the results of fine-tuning the original model. We will train the MoEDM using the same loss function. We will enhance the clarity of our paper's presentation in the updated version.