Paper: AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (ICLR-2024)

Reviewer comment (Novelty):
Lack of technical novelty, the proposed domain adapter is based on LoRA and the motion module is following standard feature inflation and the architecture is following timesformer.

Author response:
**Q1: Lack of technical novelty, the proposed domain adapter is based on LoRA and the motion module is following standard feature inflation and the architecture is following timesformer.**

As motivated in Sec. 1, our primary goal is to develop an effective pipeline to animate personalized T2I models. The technical novelty of our work is not in the modules themselves, but in the unique way they are combined and utilized, leading to an innovative solution that is greater than simply putting them together.

Since our goal is to turn a T2I model into an animation generator, feature inflation is the most efficient practice. Extensive experiments in Sec. 5 and high-quality visual results in Fig. 1,4,9,10 also validate the effectiveness of our architecture choices.

Additionally, it is not ad-hoc to use these two modules. In Sec. 5.3 and Sec. 4.2, we investigate different options for the motion module, e.g., a convolution version, and it turns out the current simple design is strong and efficient. In terms of the domain adapter, as discussed in Sec. 4.1 and Sec. 5.3, it’s worth noting that we didn’t use it for domain personalization as in previous work. Instead, we only use it during training to reduce the gap between video training data and high-quality training images of T2I. This not only results in high-quality video generation but also offers a new usage of domain adapters.

Reviewer comment (Experiments):
I do have some concerns about the generalization of the motion module.

Author response:
**Q2: I do have some concerns about the generalization of the motion module. It is not clear to me how reliable to use LoRA for domain adaption, and if the motion module does not generalize well, should we further fine-tune the motion module on the task domain data?**

To be clear, the domain adapter (implemented as an image LoRA) aims to bridge the gap between the training videos and the images (for base T2I models), which would be discarded after training. Sec.5.3 and Fig.6 demonstrate that this design indeed helps alleviate the negative effects of the video dataset. 

The motion module is proposed to learn the motion pattern regarding the video dataset. For instance, videos in the training set usually involve how humans should act and how cars should move. In this case, the generalization ability largely depends on the motion that the training set covers. After this learning, personalized video generation is enabled with a user-specific domain by inserting a personalized image LoRA.

One straightforward solution for some specific complex movements that might not appear in the training set is to collect the larger-scale corresponding videos into the training set and tune the motion module. However, this is costly and impractical. We, therefore, propose MotionLoRA as a complementary method that could learn a specific motion pattern based on the general motion prior (learned by our motion module). As mentioned in Sec. 4.3, training such a MotionLoRA usually requires 20-50 reference videos and takes around 1 hour, which provides a more practical solution.

Combining them all, our AnimateDiff could not only alleviate the effect of training videos but also capture their motion prior. More importantly, it could quickly and efficiently adapt to other new/complex motion patterns that the aforementioned training set cannot cover.

Reviewer comment (Experiments):
It is not clear to me how reliable to use LoRA for domain adaption.

Author response:
**Q2: I do have some concerns about the generalization of the motion module. It is not clear to me how reliable to use LoRA for domain adaption, and if the motion module does not generalize well, should we further fine-tune the motion module on the task domain data?**

To be clear, the domain adapter (implemented as an image LoRA) aims to bridge the gap between the training videos and the images (for base T2I models), which would be discarded after training. Sec.5.3 and Fig.6 demonstrate that this design indeed helps alleviate the negative effects of the video dataset. 

The motion module is proposed to learn the motion pattern regarding the video dataset. For instance, videos in the training set usually involve how humans should act and how cars should move. In this case, the generalization ability largely depends on the motion that the training set covers. After this learning, personalized video generation is enabled with a user-specific domain by inserting a personalized image LoRA.

One straightforward solution for some specific complex movements that might not appear in the training set is to collect the larger-scale corresponding videos into the training set and tune the motion module. However, this is costly and impractical. We, therefore, propose MotionLoRA as a complementary method that could learn a specific motion pattern based on the general motion prior (learned by our motion module). As mentioned in Sec. 4.3, training such a MotionLoRA usually requires 20-50 reference videos and takes around 1 hour, which provides a more practical solution.

Combining them all, our AnimateDiff could not only alleviate the effect of training videos but also capture their motion prior. More importantly, it could quickly and efficiently adapt to other new/complex motion patterns that the aforementioned training set cannot cover.

Reviewer comment (Experiments):
If the motion module does not generalize well, should we further fine-tune the motion module on the task domain data?

Author response:
**Q2: I do have some concerns about the generalization of the motion module. It is not clear to me how reliable to use LoRA for domain adaption, and if the motion module does not generalize well, should we further fine-tune the motion module on the task domain data?**

To be clear, the domain adapter (implemented as an image LoRA) aims to bridge the gap between the training videos and the images (for base T2I models), which would be discarded after training. Sec.5.3 and Fig.6 demonstrate that this design indeed helps alleviate the negative effects of the video dataset. 

The motion module is proposed to learn the motion pattern regarding the video dataset. For instance, videos in the training set usually involve how humans should act and how cars should move. In this case, the generalization ability largely depends on the motion that the training set covers. After this learning, personalized video generation is enabled with a user-specific domain by inserting a personalized image LoRA.

One straightforward solution for some specific complex movements that might not appear in the training set is to collect the larger-scale corresponding videos into the training set and tune the motion module. However, this is costly and impractical. We, therefore, propose MotionLoRA as a complementary method that could learn a specific motion pattern based on the general motion prior (learned by our motion module). As mentioned in Sec. 4.3, training such a MotionLoRA usually requires 20-50 reference videos and takes around 1 hour, which provides a more practical solution.

Combining them all, our AnimateDiff could not only alleviate the effect of training videos but also capture their motion prior. More importantly, it could quickly and efficiently adapt to other new/complex motion patterns that the aforementioned training set cannot cover.

Reviewer comment (Theory):
One aspect that could be further refined or discussed is the dependency of AnimateDiff on the underlying T2I models. It appears that the success of AnimateDiff is inherently tied to the performance and reliability of the existing T2I models it seeks to enhance. Specifically, the methodology might face challenges if the base T2I models struggle or fail to accurately generate images based on user-provided prompts. In such cases, AnimateDiff might encounter difficulties in achieving its animation objectives, potentially limiting the overall effectiveness and applicability of the proposed solution.

Author response:
**Q1: About the dependency of AnimateDiff on the underlying T2I models.**

Current AnimateDiff indeed relies on the underlying T2I models. Due to such dependency, our method could inherit their appealing generated appearance and might hardly avoid some failure cases caused by T2I models. On the other hand, AnimateDiff could be further improved with the advancement of T2I models. We implement AnimateDiff on Stable Diffusion XL (SDXL), which enhances the visual quality and text conditioning on T2I. The comparison of two different T2I models is presented in Appendix D.2 and Fig. 20,24, showing that our AnimateDiff could also be compatible with other advanced T2I models, namely, avoid struggling with the limitations caused by a certain T2I model. We have already updated the related discussion in Appendix D.

Reviewer comment (Experiments):
Could the authors provide visualization results that specifically exclude the domain adapter training state? To clarify, I am interested in viewing the outcomes when the domain adapter training (stage 1) is entirely omitted from the process, not just when the parameter $\alpha$ is set to zero at the inference state.

Author response:
**Q2: Domain Adapter Visualization.**

We retrain the motion module with the adapter entirely removed from the pipeline and present the visualizations in Appendix C.2 and Fig.18,22. Obviously, when the domain adapter is entirely removed from the training pipeline, the visual appearance in the training dataset, i.e., the watermarks, appears on the synthetic animations (1st row). Similarly, the watermarks appear when the adapter has its full impact (2nd row). In contrast, by fitting the visual distribution to a separate domain adapter and dropping it at inference, our full pipeline (3rd row) achieves favorable quality with no watermarks.

Reviewer comment (Experiments):
Regarding the visual quality, how does AnimateDiff perform when applied directly to the base T2I model that you used to train the motion module on WebViv? I am particularly interested in understanding the performance and visual outcomes of AnimateDiff when integrated with the foundational T2I models, excluding any enhancements or personalizations from external models such as Civitai.

Author response:
**Q3: Visual Quality of AnimateDiff in Base T2I Models.**

Although animating the original base T2I model is not our focus, we present the generated videos upon the original T2I in Appendix C.1 and Fig.17,21. It is obvious that the synthesized frames *remain smooth over time*. However, due to the lack of enhancement from personalized T2I models, the produced image corresponds closely with the video dataset (containing watermarks sometimes). Appendix D.2 and Fig.20,24 showcase the results from enhanced base T2I models (like SDXL). Since it is challenging to collect a larger-scale and higher-quality dataset than WebVid, we recommend users leverage the power of personalized image models since it is obviously more flexible and practical.

Reviewer comment (Evaluation):
In certain animation results, the motion amplitudes are minimal and flickering between frames is noticeable. This suggests room for improvement in the motion prior. These issues may be related to the size of the video dataset and model design, and a thorough analysis of these factors should be included in the limitations section.

Author response:
Thanks for pointing it out. You are right about the future improvement of motion prior. Our experimental results in Appendix C.3 and Fig. 19,23 demonstrate that the generated motions could benefit from *scale-up training*, producing videos with better motion amplitude, diversity, and temporal consistency. This also implies that the current model design is *capable enough* of capturing larger motion. In addition, enlarging the video-text paired dataset is indeed a potential avenue for further advancing our approach. However, it is important to acknowledge that this task can be challenging and costly at this moment. We have already updated the related discussion in Appendix D.

Reviewer comment (Experiments):
Qualitative comparisons are not enough. Please provide visual results for more cases like in Figure 5.

Author response:
We have updated more qualitative results in Fig.9,10,11 on pages 16,17. Importantly, 60 samples are involved in our user study in Tab.1 for quantitative comparison, demonstrating the effectiveness of our approach.

Reviewer comment (Evaluation):
For image animation, the identity preservation is poor, in other words, the characters in the animation results are not very similar to the one in the input image.

Author response:
There might be a need for clarification since our method aims to generate personalized videos given a text prompt. Although animating a given image might be achieved in a certain way, it is *not very related* to this submission. Overall, the temporal consistency could also be improved through scale-up training, as shown in  Appendix C.3 and Fig.19,23.

Reviewer comment (Presentation):
Why does the scale of the domain adaptor in Figure 6 appear to be visually linked with camera movement?

Author response:
There might be a misunderstanding. In terms of the domain adapter, it’s worth noting that we didn’t use it for domain transfer during inference as in previous work. Instead, we only use it during training to reduce the gap between video training data and high-quality training images of T2I. This not only results in high-quality video generation but also offers a new usage of domain adapters.

The pose change in Fig. 6 is not due to the camera movement but because of the varying image distribution. When adjusting the scale of the domain adaptor from 1.0 to 0.0, the generated appearance goes to the personalized image domain from the WebVid domain.

Reviewer comment (Novelty):
Some modules are not that novel from a technical perspective. E.g. the transformer block is also explored in GEN-2 [1]. LORA and adapter are also commonly used in various personalization papers.

Author response:
**Q1: Some modules are not that novel from a technical perspective. E.g., the transformer block is also explored in GEN-2. LORA and adapter are also commonly used in various personalization papers.**

As motivated in Sec. 1, our primary goal is to develop an effective pipeline to animate personalized T2I models. The technical novelty of our work is not in the modules themselves, but in the unique way they are combined and utilized, leading to an innovative solution that is greater than simply putting them together.

And it is not ad-hoc to use these two modules. In Sec. 5.3 and Sec. 4.2, we investigate different options for the motion module, e.g., a convolution version, and it turns out the current simple design is strong and efficient. In terms of the domain adapter, as discussed in Sec. 4.1 and Sec. 5.3, it's worth noting that we didn't use it for domain personalization as in previous work. Instead, we only use it during training to reduce the gap between video training data and high-quality training images of T2I. This not only results in high-quality video generation but also offers a new usage of domain adapters.

Reviewer comment (Experiments):
The paper somewhat overlooks a thorough discussion of the method's limitations. Specifically, it would be beneficial for the authors to clarify the learning dynamics of the motion module when pre-trained on the WebVid dataset—whether motions like zoom-ins and rolls require explicit training (using LORA with curated dataset) or can be inferred from textual cues alone. Additionally, a systematic categorization of which motions are effectively learned and which are not could significantly enhance the reader's understanding of the model’s practical applications and boundaries.

Author response:
**Q2: The paper somewhat overlooks a thorough discussion of the method's limitations.**

We have added a section in the revised submission to discuss the limitations of our method (see pages 22-23, Appendix D). We address the concerns in [W2] in the following Q3-Q5. 

**Q3: About the learning dynamics of the motion module when pre-trained on the WebVid dataset.**

AnimateDiff utilizes data priors to learn motions in the training videos dataset WebVid-10M, which contains mainly real-world footage such as outdoor scenes and human activities. By training upon it, the motion module could learn common real-world motion patterns, e.g., ocean waves, the movement of cars and boats, and human body/hand motions, as shown in Fig. 1. We have already updated the related discussion in Appendix D.1.

**Q4: About the learning of MotionLoRA and whether the motion patterns can be inferred from text clues.**

In AnimateDiff, whether a kind of motion can be triggered via text prompt depends on whether the training dataset accurately labeled such motion. We investigate the WebVid-10M dataset and find its motion labels very coarse. For instance, the zoom-in and zoom-out videos in WebVid are mostly labeled with a single tag, “zooming.” As a result, using text hints such as “zoom in” results in both zoom-in and zoom-out generation. On the other hand, more complex motion patterns such as “Rolling” do not have a specific text label, and thus, these motion patterns cannot be triggered via simple textual clues.

This observation suggests that MotionLoRA does not learn new motion patterns entirely from scratch but refines and enhances the pre-existing motion priors (regardless of whether they can be triggered by text) obtained during pre-training. It thus enables the motion module to express the desired motion priors at inference. We have already updated the related discussion in Appendix D.1.

**Q5: About the performance on different motion categories.**

In practice, we find the pipeline works well on non-violent motions such as fluid (e.g., ocean waves, fog, etc.), rigid objects (e.g., cars, boats), and simple human movements (e.g., walking, facial expressions), as shown in Fig. 1,4,9. However, if the pipeline is instructed to generate complex or rare motions rarely seen in the training dataset, e.g., dance or camera movements with drastic scene change, the model will likely generate static animation or unnatural deformations. We have already updated the related discussion in Appendix D.1.