Paper: LLM-grounded Video Diffusion Models (ICLR-2024)

Reviewer comment (Novelty):
The idea of using LLM to generate layout is already explored in LayoutGPT (Feng et al., 2023) and LMD (Lian et al., 2023). LMD also adapts in a training-free manner. It is beneficial for the authors to include a more detailed comparison.

Author response:
Thank you for your encouraging review! Here we address your comments individually:

> The idea of using LLM to generate layout is already explored in LayoutGPT (Feng et al., 2023) and LMD (Lian et al., 2023).

> The first layout generation part is similar to LMD, and the second part is a straightforward application of other training-free layout-to-image methods in the video domain.

Thanks for the comment. However, we would like to point out a few differences in terms of findings and the proposed methods between our work and LMD.

As for the LLM-grounded dynamic scene layout generation stage, **there are two noteworthy distinctions between our work and previous works, as explained in detail in the General Rebuttal \[Q2\]**.
1. Most of text-to-image-layout generation tasks introduced in previous works (LayoutGPT and LMD) can be completed without in-depth understanding of the properties of the objects in the prompt. In contrast, even a simple generation of dynamic scene layouts of "a falling object" requires understanding several world properties, object properties, and camera properties, which is **highly non-trivial for an LLM and not tested by previous works**.
2. More importantly, previous works either only evaluate properties that are **already seen in in-context examples** (e.g., the 4 main tasks that LMD showcases are already demonstrated in the in-context examples in Table 10 of the LMD paper), or **retrieve related layouts from a layout database** as in-context examples (LayoutGPT). **Neither work demonstrates whether LLMs have the knowledge for generalization without relevant in-context examples.** In contrast, in Section 3 (Discoveries), we show that LLMs can generalize to physical properties or object properties that are **unseen in the in-context examples** (e.g., inferring the property of a paper airplane, while similar concepts are never mentioned in the in-context examples). This implies LLMs have an intrinsic understanding of the physical world instead of just emulating from in-context examples.

As for the dynamic scene layout to video generation stage, **as explained in the General Rebuttal \[Q1\],** we propose a temporal energy term that provides guidance on the positions and velocities of center-of-mass of each object, which significantly improves our generation accuracy, especially on the temporal tasks. **Since previous layout-to-image methods only focus on generating one image (i.e., do not take temporal consistency into accounts), we believe our introduction of temporal center-of-mass energy term sets our method further apart from previous image layout conditioning methods that you mentioned.** We have updated our paper PDF to include the descriptions to this energy term.

**We hope the reviewer can agree with us on these significant distinctions that are either not applicable to previous LLM-grounded text-to-image works or are not demonstrated by them.** We will update the manuscript accordingly to express these distinctions more clearly.

Reviewer comment (Novelty):
The technical contribution is limited. The first layout generation part is similar to LMD, and the second part is a straightforward application of other training-free layout-to-image methods in the video domain.

Author response:
Thank you for your encouraging review! Here we address your comments individually:

> The idea of using LLM to generate layout is already explored in LayoutGPT (Feng et al., 2023) and LMD (Lian et al., 2023).

> The first layout generation part is similar to LMD, and the second part is a straightforward application of other training-free layout-to-image methods in the video domain.

Thanks for the comment. However, we would like to point out a few differences in terms of findings and the proposed methods between our work and LMD.

As for the LLM-grounded dynamic scene layout generation stage, **there are two noteworthy distinctions between our work and previous works, as explained in detail in the General Rebuttal \[Q2\]**.
1. Most of text-to-image-layout generation tasks introduced in previous works (LayoutGPT and LMD) can be completed without in-depth understanding of the properties of the objects in the prompt. In contrast, even a simple generation of dynamic scene layouts of "a falling object" requires understanding several world properties, object properties, and camera properties, which is **highly non-trivial for an LLM and not tested by previous works**.
2. More importantly, previous works either only evaluate properties that are **already seen in in-context examples** (e.g., the 4 main tasks that LMD showcases are already demonstrated in the in-context examples in Table 10 of the LMD paper), or **retrieve related layouts from a layout database** as in-context examples (LayoutGPT). **Neither work demonstrates whether LLMs have the knowledge for generalization without relevant in-context examples.** In contrast, in Section 3 (Discoveries), we show that LLMs can generalize to physical properties or object properties that are **unseen in the in-context examples** (e.g., inferring the property of a paper airplane, while similar concepts are never mentioned in the in-context examples). This implies LLMs have an intrinsic understanding of the physical world instead of just emulating from in-context examples.

As for the dynamic scene layout to video generation stage, **as explained in the General Rebuttal \[Q1\],** we propose a temporal energy term that provides guidance on the positions and velocities of center-of-mass of each object, which significantly improves our generation accuracy, especially on the temporal tasks. **Since previous layout-to-image methods only focus on generating one image (i.e., do not take temporal consistency into accounts), we believe our introduction of temporal center-of-mass energy term sets our method further apart from previous image layout conditioning methods that you mentioned.** We have updated our paper PDF to include the descriptions to this energy term.

**We hope the reviewer can agree with us on these significant distinctions that are either not applicable to previous LLM-grounded text-to-image works or are not demonstrated by them.** We will update the manuscript accordingly to express these distinctions more clearly.

Reviewer comment (Evaluation):
From Table 3, we can see that LVD improves the video quality. What causes the improvement?

Author response:
> From Table 3, we can see that LVD improves the video quality. What causes the improvement?

Our method indeed improves on the FVD scores on both UCF-101 and MSR-VTT datasets over the base video diffusion models **without additional training**, as shown in Table 3.

FVD measures the alignment between the generated videos and reference videos, which in turn reflects both video-text alignment and video quality. Two factors together contributed to our improvements in the FVD score:

1. Our method, with an LLM processing input text prompts to generate dynamic scene layouts, has improved ability to understand the input text prompt. This is reflected in the improved video-text alignment.
2. Although improving video quality is not our main goal, our method also improves the quality of generated videos by using the intermediate dynamic scene layouts for planning, which in turn leads to better attribute binding and spatial layouts that are more reasonable.

**For example,** in the first row of Fig. 1 in our paper, baseline ModelScope confuses the semantics of the bear and the pikachu mentioned in the prompt and generates an object that looks neither like a bear or like a pikachu. By giving spatial-temporal guidance to the video diffusion model, our method can successfully generate a bear and a pikachu with high visual quality, as required by the prompt. The ability for high-quality video generation in cases like this leads to our improved FVD score.

Reviewer comment (Experiments):
Is LLM able to generate reliable layouts using text without direction information, such as “a walking dog”.

Author response:
> Is LLM able to generate reliable layouts using text without direction information, such as “a walking dog”.

That is a great point! **As your requested, we ask the LLM to generate a walking dog without specifying the direction information, as shown in Figure 8(c\) in the appendix of the updated paper PDF.** We observe that the LLM is not only able to generate dynamic scene layouts for a walking dog but also **provide assumptions that it makes about the walking directions in the reasoning statement**. This shows that our LLM is indeed able to make reasonable guesses for ambiguous user prompts.

```
Reasoning: A dog walking would have a smooth, continuous motion across the frame. **Assuming it is walking from left to right**, its x-coordinate should gradually increase. The dog's y-coordinate should remain relatively consistent, with minor fluctuations to represent the natural motion of walking. The size of the bounding box should remain constant since the dog's size doesn't change.
```

If the assumption does not meet the user's expectation (e.g., the user wants a dog walking from the right instead), the user can reply to the LLM, and the LLM can incorporate additional information to satisfy the user's needs.

Given the successful generation of the dynamic scene layouts, our layout-conditioned video generation method also successfully generates a video of a walking dog. We have added this example in the appendix of our updated manuscript. **Please see Sec A.1 and Fig. 8 (a) in our updated PDF for the video generation.** Thanks again for the suggestion!

In light of this, we would like to kindly ask if there is anything to explain to make you consider raising your score. Additionally, we are open to addressing any additional concerns or answering any questions you may have!

Reviewer comment (Evaluation):
Even though the proposed method is training free, it takes longer time during inference to generate videos due to the optimization steps needed for the energy function.

Author response:
Thanks for the great point! All previous methods based on backward guidance for training-free guidance (e.g., LMD, BoxDiff, Layout Guidance, Self-Guidance, and our method) take more inference time. The reason is that such methods need to find a way to leverage the knowledge inside the pretrained model during inference, without supervision from any external box-video pairs. 

However, training-free methods not only are free of training costs (time and compute requirements) but also have **no requirements on external data or human annotations (e.g., box annotations on frames)**. For example, our method can be applied off-the-shelf on video diffusion models such as ModelScope without external annotations. Moreover, training-free allows exploring a wide range of formulations without training them each time.

Furthermore, even though inference time is usually not the main focus of this line of work (e.g., LMD, LayoutGPT, BoxDiff, etc.), **if low inference time is needed for the production environment, our method can be distilled into a conditioned generation model** so that the inference-time optimization of energy function is no longer required. Specifically, the generation of our method can potentially be distilled into a model with additional input tokens (e.g., ReCo) or attention adapters (e.g., GLIGEN), thus achieving fast generation without external datasets or additional human annotations. **This way, we get the advantages of being both free of external annotated data and having fast inference, enjoying the benefits of existing training-free and training-based methods.** We leave the distillation to future work as this is a general enhancement applicable to many training-free guidance methods.

Reviewer comment (Novelty):
Training-free layout control already exists in previous literatures [1, 2]. Therefore, the design of the energy function and backward guidance is not that novel.

Author response:
The energy function proposed in our pre-rebuttal manuscript, which applies spatial control per frame, is indeed a little similar to the energy function used in previous works. However, **as explained in the General Rebuttal \[Q1\],** we proposed a temporal energy term that provides guidance on the positions and velocities of center-of-mass of each object, which significantly improves our generation accuracy, especially on the temporal tasks. Since previous layout-to-image methods only focus on generating one image (i.e., do not take temporal consistency into accounts), **we believe our introduction of temporal center-of-mass energy term sets our method further apart from previous image layout conditioning methods mentioned by the reviewer.** We will update the manuscript accordingly. Thanks for the good point!

Reviewer comment (Experiments):
Ablation study of the model design is not given (e.g., number of DSL guidance steps, energy function design).

Author response:
Thank you for the suggestion. We have conducted ablation studies on the CoM energy function design as well as the number of DSL guidance steps in LVD. **Please refer to the General Rebuttal [Question 4] for more details.**

Reviewer comment (Evaluation):
Could the authors provide some reasoning why they report video-text similarity metric in Bain et al., 2021? It would be nice to also report CLIPScore, since its widely reported in other text-to-video generation baseilnes.

Author response:
Good question! CLIPSIM, the video-text alignment score obtained from CLIP, is computed per-frame and then averaged across the feames ([reference](https://arxiv.org/abs/2104.14806)). This leads to its temporal ambiguity. **For example,** it could not distinguish a barrel floating from the left to the right vs from the right to the left. Instead, we use Frozen-In-Time, a CLIP-like model that takes in the whole video sequence and outputs a video-text alignment, which does not treat a video as a bag of frames and can potentially model object motion in the alignment score.

**Per your request, we also evaluate our model and report the CLIPSIM metric compared to the baseline**, following a similar setting as Tab. 4 (Tab. 5 in the updated PDF) except we change the alignment model to a per-frame CLIP.

|   | ModelScope | LVD on ModelScope |
|:---:|:---:|:---:|
| CLIPSIM | 0.2947 | **0.3001** |

Our method gives better CLIPSIM compared to baseline ModelScope, indicating better text-to-video alignment.

However, we would like to mention that previous benchmarks, such as FVD or similarity-based metrics (e.g., metrics based on CLIP or Bain et al., 2021), do not perform detailed checks for correctness in aspects such as spatial dynamics and generative numeracy. While these methods mainly focus on the semantic alignment and generation quality, **our proposed benchmark in Tab. 2 pioneers in quantitatively assessing the correctness in prompt understanding in video generation, opening up a direction to explore future video generation research.**

Reviewer comment (Theory):
If there are multiple objects, is the final energy function summing over the energy function corresponding to each object?

Author response:
You are right! The final energy function is the sum over the one for each object. We will make it clear in the manuscript.

Reviewer comment (Evaluation):
It seems that the background areas of the generated images with proposed method are quite static (Fig1, 7, 8, 9). Is this because the model encourages static background, or becuase the visualized images happens to have relatively static background?

Author response:
Good point! Empirically we found that our base model, ModelScope, often tends to generate videos with small background variations, potentially because it uses a lot of static images in training. **To ensure that our model retains the ability to generate non-static background, we let it generate `a bear taking a shower in a lake, lightning strikes` as a simple example, which we show in Figure 8(c\) in the paper appendix (Sec A.1 in the updated PDF).** Although the background motion is not extreme, the model indeed generates rain dropping on the water and the lightning, showing that our method does not impede the ability to generate non-static background.

Reviewer comment (Novelty):
Based on my understanding, another concurrent work, VideoDirectorGPT [1], is also for text-to-video generation with layout guidance. Even though the technical routes are different from this paper, it would be nice to have some discussions and comparison in the related work section.

Author response:
Thanks for the suggestion! **We have included a reference to [1] and updated a comparison in the related work section in the paper.** Since this work was released on arXiv only a few days before the submission deadline, **[1] is a concurrent work that should not be used to devalue our work**.

Our work has a different focus compared to [1]:
1. Our work is training-free and focuses on inference-time algorithms, whereas [1] trains adapters with human annotated images.
2. Our work focuses more on in-depth analysis on the capabilities of LLMs to generalize to unseen properties and to use knowledge in the weights rather than mimick in-context examples. 

**In light of the completion of several requested experiments and comparisons, we would like to kindly ask if there are any other concerns or additional questions that we can respond to or if you are willing to consider increasing your score!**

Reviewer comment (Novelty):
The idea of guidance via energy functions and cross-attention maps seems to be basically derived from BoxDiff (Xie et al., 2023;) and Chen et al. 2023a;. It is unclear how much of this work is based on previous research and how much is new. Since they are dealing with video generation using layouts, it would have been nice to see the authors' contribution in extending to the temporal axis, but this is not evident, which is disappointing.

Author response:
Thanks for the suggestion! Indeed, although our method already greatly outperformed the baseline with the current energy function, innovation on temporal axis should further boost the temporal consistency of the generated objects. Therefore, **as introduced in the General Rebuttal \[Q1\]**, we added a temporal center-of-mass term, which significantly improves our generation accuracy by 10%, especially on the temporal dynamics and sequential movements, while reducing the mismatches between the generated objects and the dynamic scene layout input. **Our introduction of center-of-mass energy term sets our method further apart from previous image layout conditioning methods that do not deal with the temporal axis. We hope this novel energy term meets your expectation for novelty related to the temporal axis! Thanks for your suggestion again!**

Reviewer comment (Experiments):
I am concerned that the scale of the sample size for the proposed DSL benchmark may be too small to conduct a sufficiently robust evaluation.

Author response:
This point is totally valid! Taking your suggestions, we propose an enlarged version of our benchmark that is 10x larger compared to the previous version. Specifically, each task now has 5x more prompts in a similar structure as the previous version, with 500 prompts in total. For each prompt, we generate two videos from dynamic scene layout, with **1000 videos per benchmark run in total**.

We run both the baseline and our method twice with the enlarged benchmark and **we observe evaluation results with small overall variation across the runs**.

|   | Numeracy | Attribution | Visibility | Dynamics | Sequential | Average | **Variation** | Previous smaller benchmark |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|---|---|
| ModelScope Baseline (run 1/run 2) | 7.5%/5% | 65.5%/66% | 1.5%/1% | 18.5%/23.5% | 0%/0% | 18.6%/19.1% | **0.5%** | 16% |
| Ours (run 1/run2) | 59.5%/55.5% | 94%/88.5% | 40%/50.5% | 65.5%/67.5% | 34%/32% | 58.6%/58.8% | **0.2%** | 58% |

**Furthermore, our enlarged benchmark still results in very similar results compared to the previous smaller benchmark, which also validates the findings obtained with the previous benchmark. We hope the robustness in our benchmark results can address your concerns!**

Finally, we would like to note that while previous benchmarks (e.g., CLIPSim and FVD) focus on generation quality and semantics, **our benchmark is a pioneer for correctness in spatial-temporal alignment in the literature,** which is of great significance for future research, especially given that the results justify the robustness of our benchmark.

For the final version of the paper, we plan to run all our ablation studies on this enlarged benchmark and include the results in our work.

Reviewer comment (Novelty):
The paper's contribution appears to lack novelty. There is existing work in text-to-image generation that has already established the capability of LLMs to create layouts, and this research seems to merely extend that to assess whether the same capability applies to temporal understanding. I didn't perceive any novel ideas stemming from the temporal aspect of the problem that would distinguish this work significantly from its predecessors.

Author response:
Thanks for the comment. There may be some misconceptions on the capabilities justified by previous works that use LLMs for image layout generation. **There are two noteworthy distinctions between our work and previous works, which we briefly explain here, and we provide more detailed clarifications in the General Rebuttal \[Q2\]**.
1. Most of text-to-image-layout generation tasks introduced in previous works (LayoutGPT and LMD) can be completed without understanding the properties of the objects in the prompt in depth. In contrast, even a simple generation of dynamic scene layouts of "a falling ball" requires understanding of complex properties of the physical world, such as physical property (e.g., gravity which decides the trajectory of the ball), object interaction (e.g., ball hitting the ground), object properties (e.g., elasticity of ball which decides how much the ball will bounce), and camera properties (e.g., perspective geometry which decides if the size of the ball will change based on its distance to the camera), which are all **highly non-trivial for an LLM and have not been assessed, demonstrated, or evaluated by previous works**. **In contrast, we show that LLMs demonstrate the understanding of these properties, as illustrated in Figure 4 in the paper.**
2. Previous works either only evaluate properties that are **already seen in in-context examples** (e.g., the 4 main tasks that LMD showcases are already demonstrated in the in-context examples in Table 6 of [the LMD paper](https://arxiv.org/pdf/2305.13655.pdf)), or **retrieve related layouts from a layout database** as in-context examples (LayoutGPT). **Neither work demonstrates whether LLMs have the knowledge for generalization without relevant in-context examples.** In contrast, in Section 3 (Discoveries part), we show that LLMs can generalize to physical properties or object properties that are **unseen in the in-context examples** (e.g., as shown in Figure 4 (c\) in the paper, when generating layouts of throwing a paper airplane, it can figure out the airplane will slide in the air due to its weight and aerodynamics, while similar concepts are never mentioned in the in-context examples). This implies LLMs have an intrinsic understanding of the physical world instead of just emulating from in-context examples.

We will clarify these two points further in our manuscript. Thanks again for the valuable feedback!

Reviewer comment (Experiments):
The paper seems to lack a detailed analysis or ablation studies concerning the prompts given to the LLM for generating Dynamic Scene Layouts (DSLs). Such investigations are crucial to understand how different prompts affect the LLM's output and the subsequent quality of the video generation. Further exploration in this area could significantly bolster the robustness of the presented approach.

Author response:
Thank you for the suggestion! We have added ablation studies on how the number, content, and format of prompted in-context examples will affect the generation quality. **Please refer to the General Rebuttal \[Q3\],** in which we show that 3 in-context examples already result in high-quality dynamic scene layout generation and LLM is robust to different in-context examples in both content and format.

Reviewer comment (Experiments):
The paper's current framework could indeed benefit from additional ablation studies or analytical experiments to demonstrate the effectiveness of using DSLs for training-free guidance of text-to-video diffusion models. Moreover, a theoretical explanation of why this particular approach is effective would be valuable. It's important for the research to not only present the method but also to thoroughly validate and explain why certain choices were made and how they contribute to the overall performance and reliability of the model.

Author response:
Thank you for the comment. To analyze different design choices of the DSL-to-video stage in LVD, we conduct ablation studies on the CoM energy function design as well as the number of guidance steps in the DSL-grounded video generation pipieline. **Please refer to the General Rebuttal \[Q4\] for more details.**

Reviewer comment (Evaluation):
Can the authors elaborate on how the model performs with ambiguous or complex text prompts that might yield multiple valid interpretations in terms of spatial and temporal dynamics?

Author response:
That is a good point! The potential ambiguity in the user-provided prompts is indeed worth exploring. We find that LLMs can oftentimes perform reasonable guesses if the spatial-temporal information is missing in the user-given prompt.

For example, with a simple prompt as `a walking dog`, the LLM gives the following reasoning statement, with the assumption clearly written in it:
```
Reasoning: A dog walking would have a smooth, continuous motion across the frame. **Assuming it is walking from left to right**, its x-coordinate should gradually increase. The dog's y-coordinate should remain relatively consistent, with minor fluctuations to represent the natural motion of walking. The size of the bounding box should remain constant since the dog's size doesn't change.
```

This allows the subsequent layout generation to be consistent with respect to the assumption of walking from the left to the right. **We show the generated video of this example in Figure 8(a) in appendix of the updated paper PDF.**


Additionally, if the assumption does not meet the user's expectation (e.g., the user wants a dog walking from the right instead), the user can reply to the LLM, and the LLM will incorporate additional information to satisfy the user's needs.

Reviewer comment (Evaluation):
Could the authors discuss any observed limitations or failure modes of the LVD approach, particularly in cases where the LLM might generate less accurate DSLs?

Author response:
Thank you for the suggestion. We show two failure cases in the DSL-to-video stage in Figure 10 and Figure 11 of the updated paper PDF. For the text-to-DSL stage, although the LLM-generated DSLs have a high accuracy in following the prompt (98% accuracy in Table 1 in the paper), it still has some failure cases. The failures mainly come from generating complex dynamics such as sequential movements. One failure case, which we added in the updated paper PDF, is shown as below and also demonstrated in Figure 9 in Section A.6 of the paper appendix:

*prompt*: A realistic lively video of a top-down viewed scene in which a ball initially on the lower left of the scene. It first moves to the upper left of the scene and then moves to the upper right of the scene, outdoor background.

*generated boxes* (visualized in Figure 9 of the updated paper PDF):
```
Frame 0: (0.097, 0.878, 0.195, 0.976) 
Frame 1: (0.097, 0.683, 0.195, 0.781) 
Frame 2: (0.097, 0.488, 0.195, 0.585)
Frame 3: (0.292, 0.488, 0.390, 0.585) 
Frame 4: (0.488, 0.488, 0.585, 0.585) 
Frame 5: (0.683, 0.488, 0.781, 0.585)
```

Note that for each frame, the box is represented by 4 float numbers, where the first two are the x and y coordinates of the upper left corner of the box, and the last two are the x and y coordinates of the lower right corner of the box.

In this case, although the prompt indicates the ball moves to the upper left corner of the scene, the LLM does not generate the box in the upper left corner, but instead a box slightly above the middle of the whole scene. The failure is probably because the sequential movements contain multiple stages and LLM has a higher probability to oversee or misunderstand one of the stages. We will add these discussions on the failure cases to the paper.

Reviewer comment (Writing):
(Minor point) Typographical error in Section 4, second paragraph. The sentence in question should indeed conclude with "feature map" instead of "feature ma." A revision is recommended for accuracy and clarity.

Author response:
Thanks for the good point! We have fixed this in our manuscript.