Paper: Seer: Language Instructed Video Prediction with Latent Diffusion Models (ICLR-2024)

Reviewer comment (Presentation):
The writing could be improved for better clarity. Additionally, the positioning of figures could be optimized for easier readability; consider rearranging Figures 3 and 4.

Author response:
Thank you for your valuable feedback. In response to your concerns, which were also raised by Reviewer shhp and Reviewer xJcW regarding the clarity of our paper, we have undertaken comprehensive revisions to enhance readability on four folds:

1. We have streamlined our motivation section, minimizing detailed discussions on robot learning, and have refocused the introduction to underscore the main contributions of Seer.
2. The mistaken claim that text-conditioned video prediction is more challenging than text-to-video generation has been clarified in the related work section.
3. To improve the understanding of our method, we have highlighted key components of Seer in the method figure and reorganized the flowchart for better clarity.
4. Figures have been strategically repositioned to enhance overall paper readability.

These revisions have been diligently incorporated into our paper to address the concerns raised, with the aim of significantly improving the overall readability of our work.

Reviewer comment (Evaluation):
It would better to enhance the paper to include analysis of failure cases and generated video examples for a more comprehensive evaluation.

Author response:
Thank you for your constructive suggestion. We have expanded our qualitative analysis of failure cases, incorporating a dedicated section in Appendix D.7 of our revised paper. Figure 18 in the revised paper showcases instances where Seer struggles to capture appearance consistencies, such as the color of the card in the previous frame and the cover of the book in the reference environment. These challenges are attributed to the difficulty Seer faces in forecasting newly occurring concepts and maintaining temporal consistency within complex backgrounds. In the context of the Epic-Kitchens-100 dataset, Seer tends to predict camera pose movements and generate novel views of the environment, rather than accurately learning human motion. This difficulty arises due to the prevalent scene transitions in Epic-Kitchens. Further insights into these failure cases are discussed in detail in the new Appendix D.7 section.

Reviewer comment (Theory):
The authors generate subinstructions by 1) initializing subinstructions using the global embeddings and 2) performing cross attention with the global embeddings. This results in subsets of the same information within a global embedding being selected for each subinstruction embedding, similar to performing attention on the global embedding. The reviewer questions whether it is possible for a model to learn proper subgoals/subinstructions in such a way when it is constrained to manipulation of global information. For example, for a global instruction like "cook a pasta dinner", subinstructions might be "boiling water" and "heating up sauce", but such details may not inherently be extractable from the global embedding of "cook a pasta dinner". The reviewer would be more convinced if the global text was expanded to include intermediate information, e.g., by generating "step by step" sequences via a pretrained LLM, which can then be converted into subinstructions per timestep. The current formulation does not seem conceptually convincing beyond being a slightly more advanced attention head, which may be inherently limiting. Additional investigations into what the subinstructions are encoding in terms of textual information would be helpful.

Author response:
We appreciate the reviewer's insightful feedback and appreciate the thorough suggestions provided. Overall, language instructions can be categorized into macroscopic-level and microscopic-level instructions. Macroscopic-level instructions involve abstract narrations of multi-step tasks, such as "cook a pasta dinner." On the other hand, microscopic-level instructions focus on one-step actions, like "boiling water" within the broader context of "cook a pasta dinner." In response to these valuable suggestions, we emphasize that our primary focus is on microscopic-level action generation. Specifically, the FSText decomposer is designed to transform one-step instructions into sequential fine-grained subinstructions. Through the alignment of visual tokens at each time step, these subinstructions effectively plan a continuous motion trajectory. Consequently, these subinstructions can be considered equivalent to low-level robotic instructions, providing specific values such as "rotate hand by 30 degrees then move forward 1/32 of the distance between the hand and the target object."  
Furthermore, we agree with the reviewer's suggestion regarding macroscopic-level global instructions. We acknowledge that these multi-step global instructions can be expanded into intermediate information through a pre-trained Large Language Model (LLM). This intermediate information can then be systematically converted into one-step instructions for each state, ultimately serving as the foundation for generating low-level subinstructions for video prediction. And recent approaches such as SayCan\[1\], have attempted grounding language in robotic tasks by translating high-level abstract knowledge of macroscopic tasks into procedures for performing complex and temporally extended instructions. From the above discussion, we see the value in exploring this direction further in our future work.

*References:*

[1\] Ahn, Michael, et al. "Do as i can, not as i say: Grounding language in robotic affordances." *arXiv preprint arXiv:2204.01691* (2022).

Reviewer comment (Experiments):
While Text-Video Prediction (TVP) is motivated as a way to help robots follow human instructions by mapping abstract language instructions to policy actions, the reviewer notes that an additional step is required to ground generated text-conditioned videos to actual policies. Given the motivations, it would strengthen the work to demonstrate an example (even synthetic) where videos generated by Seer help policy learning in a robot, simulated environment, or game.

Author response:
Thank you for your suggestion. We have incorporated additional visualizations of Seer's performance on robot benchmarks, including MetaWorld, RLBench, and Cliport, in Appendix D.5 of our revised manuscript. In this setup, commencing with the Seer model fine-tuned on SSv2, we have frozen the FStext decomposer and conducted further fine-tuning on a diverse dataset comprising SSv2, Bridgedata, 236 MetaWorld video clips, 785 CliPort video clips, and over 2000 video clips from RLBench. This fine-tuning process spans 80k steps with a reduced learning rate of 4e-5.

The results of these experiments are presented in Appendix Figure 18 of our revised paper, illustrating Seer's adaptability to a multi-task environment that includes human video, robot manipulation, and simulation scenarios. Importantly, Seer maintains robust temporal alignment with task descriptors across these diverse video domains.

Furthermore, to investigate whether Seer can help policy learning, we choose the UniPi\[1\] as our baseline in the simulated robotics environment Meta-World \[2\], which generates videos from the initial state and infers actions from the adjacent frames via a pre-trained inverse-dynamics model. Specifically, we distill a policy model from the videos generated by the Seer and the labeled actions from the pretrained inverse-dynamics model. We choose 3 tasks and use 10 in-domain videos for each task to fine-tune the Seer. And we make comparisons between:

a\. Distilled policy from the dataset generated by the Fine-tuned Seer model (1000 generated videos for each task).

b\. Distilled policy from the 10 in-domain videos for each task.

The results are as follows:

|                         |                                       |                           |
| ----------------------- | ------------------------------------- | ------------------------- |
| **Success rates**       | **a\. 1k videos generated from Seer** | b\. 10 groud-truth videos |
| button-press-topdown-v2 | **0.45**                              | 0.4                       |
| drawer-close-v2         | **0.1**                               | 0                         |
| drawer-open-v2          | **0.05**                              | 0                         |

We find that compared to utlize the given 10 in-domain videos to generate policy, fine-tuning the Seer with them and generating more videos can be better, because it can acquire more scalable data, which is of comparable quality. Therefore, the videos generated by Seer can help policy learning in simulated robotics tasks in a way.

\[1\] Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J. B., ... & Abbeel, P. (2023, November). Learning universal policies via text-guided video generation. In *Thirty-seventh Conference on Neural Information Processing Systems*.

\[2\] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., & Levine, S. (2020, May). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In *Conference on robot learning* (pp. 1094-1100). PMLR.

Reviewer comment (Experiments):
Why were 2 reference frames used for Something Something-v2, but not for BridgeData or EpicKitchens100? Is there something special about SSv2, or were the generated results simply empirically worse for 1-reference frame SSv2, and why might this be the case for this particular dataset?

Author response:
This comment is regarding the choice of reference frames in training datasets. The Something Something-v2 (SSv2) dataset, which is concerning real-world human behaviors, contains various unexpected cases in real-world video such as object occlusions, scene changes, and camera view movements. Initially, we were concerned that Seer might face challenges in learning from such intricate scenes when provided with only one reference frame and lacking additional background information. Hence, SSv2 was chosen as our primary training dataset, and we employed the initial setting of using 2 reference frames for generating 12-frame-long videos on SSv2.

To our surprise, Seer demonstrated robust performance even with only one reference frame in the SSv2 dataset in our follow-up test in Appendix Figure 14 of the revised paper. This result prompted us to extend our experiments to the Egocentric human kitchen video dataset EpicKitchens100, adopting the 1 reference frame condition. Remarkably, Seer exhibited competitive performance compared to other baseline methods in this setup.

Reviewer comment (Experiments):
Seer was demonstrated for 12 and 16 frame generations, but how does the model perform for long-form generation? Can the final frame(s) of an output sequence be used as a reference to generate subsequent frames using the same text prompt to create a longer video? Or would the downsampling assumptions used prevent that from being meaningful, since it might generate 12 frames spanning an entire video all at once? Demonstrating and evaluating longer-form video generation would strengthen the work.

Author response:
Thank you for your valuable suggestion. In this paper, our primary goal is task-level video prediction with language guidance, and most downsampling video clips inherently have sufficient length for task-level behavior performance. Intuitively, appending an additional temporal upsampler network is compatible with most video diffusion models including Seer further aid Seer in expanding video frames.  
Moreover, considering the suggestion about using the final frame of the output sequence as a reference for generating the next sequence, this approach could offer the advantage of sampling longer video sequences with the alignment of a long sequence of subinstructions. Therefore, we realize the potential benefits of exploring the expansion of macroscopic-level global instruction to multi-step intermediate guidance via a Large Language Model (LLM) for long-sequence video prediction. This direction is certainly worth further exploration in our future work.

Regarding your suggestion on using the same text for long video generation, we have conducted an additional experiment on long video prediction using a 12-frame SSv2 fine-tuned Seer model. The results of this experiment are presented in Appendix D.6 Figure 19 of the revised paper. These findings demonstrate that using the same sub-instruction to generate the next sequence of output can maintain coherent motion, persisting until the target object is no longer present or the current motion is in a terminated state. While this strategy enables the generation of coherent movements in the output sequence, it is distinguished from the upsampling of video clips along the frame axis. Based on these outcomes, we believe that expanding the generated subinstruction embeddings along the frame axis during the training stage represents a promising direction for enabling Seer to generate longer video frames in multi-steps.

Reviewer comment (Evaluation):
Are there ways to probe, textually, the subinstructions learned to understand what they are encoding?

Author response:
We appreciate the reviewer's observation and understand the challenges posed by the limitations of CLIP in generating readable textual content from encoded embeddings. In the learning process of FSText decomposer, where subinstructions are acquired from transition distributions along the temporal dimension, we have noted that attempting to generate text-to-image content solely based on subinstructions without temporal awareness results in content that is not easily interpretable by humans.

Consequently, to indirectly explore the semantic expression of subinstructions, we have included visualizations of videos generated with the guidance of constant clones of subinstructions from the third or twelfth frame along the video axis. These visual samples are presented in Figure 9 of Section 5.6 and Figure 16 in Appendix Section D.3. Through these visual representations, we aim to illustrate that subinstructions indeed provide proximate semantic guidance for the motion trajectory at each time step.

Reviewer comment (Reproducibility):
The paper repeatedly refers to "initializing" the subinstruction embeddings to be CLIP global embeddings. However, at the end of Section 4 on Page 6, a loss equation is provided. Is this additional objective utilized during optimization of the FSText decomposer during fine-tuning, or are the subinstructions explicitly initialized to the global CLIP embeddings before performing self-attention, cross-attention, and temporal attention? This distinction is important.

Author response:
Thank you for your comment, we clarify that the additional objective at the end of Section 4 is utilized during the initialization stage of the FSText decomposer in Figure 4 of the paper. This stage is performed prior to the diffusion process.

Reviewer comment (Experiments):
Given that the global CLIP embedding is used for initialization as well as for cross-attention, there seems to be no additional textual or contextual data available for generating subinstructions. In this case, subinstructions appear to be purely derived from the information captured in the specific words of the global instruction; to the reviewer, the FSText decomposer thus acts as a form of attention network over the existing global embedding. The reviewer asks how this is conceptually different, given that each subinstruction’s information is simply a subset of the global instruction, and notes that this approach may be highly dependent on the initial global instruction. Are there alternative choices for the text embedding that could be tested in ablation studies?

Author response:
Thank you for your comments. We would like to clarify that the generated information in subinstructions is not simply an expanding subset of the global instruction but reflects the fine-grained guidance of the transition of the states at the microscopic level. For instance, consider the global instruction "moving object from left to right" as shown in Appendix Figure 15. With the temporal alignment of motion in training video samples, the generated subinstruction embeddings can decompose the high-level abstract information of language into low-level instructions at continuous time steps, containing low-level control information such as relative position or angle, In this case, the subinstruction at 1/32 time step of video can be equivalent to the microscopic level instruction of "moving right in 1/32 of image width". This is where the FSText Decomposer assumes a crucial role, facilitating the transformation of global language instructions into coherent subinstructions guidance along the temporal sequence, leading to the prediction of continuous motion.

Reviewer comment (Presentation):
Improve the motivation for the paper: The first few paragraphs of the introduction focus on robot learning, but the paper completely eschews this domain. It would be useful to motivate the paper from a different perspective that does not involve robotics. I appreciate that there are experiments conducted on the bridge dataset but I still believe this is primarily not a robotics paper as there are no resultant policies.

Author response:
Thank you for your thoughtful suggestion. We acknowledge that we overemphasize robot learning in our introduction section. The primary goal of this paper is to introduce a text-conditioned video prediction (TVP) model to predict task-level videos based on a specific task descriptor that describes behavioral movements in both the real world and robot environment. We have duly noted your feedback and have revised the motivation in the introduction section to accurately reflect the primary focus of our research.

Reviewer comment (Theory):
Difficulty in comparison to text-to-video generation: Playing devil’s advocate here, one could argue that performing text-to-video generation involves learning both the initial state distribution along with the transition distribution, while prediction involves only learning the transition distribution. Therefore it is unclear which is harder and it would be useful to avoid making such claims.

Author response:
Thank you for bringing this to our attention. We acknowledge the concern that claiming text-conditioned video prediction is inherently more challenging than text-to-video generation tasks may not be appropriate. In response to this feedback, we have revised the description in the introduction section of the paper to accurately represent the nature of our task.

Reviewer comment (Experiments):
How does sequence length of the video along with number of steps of prediction affect performance?

Author response:
Thank you for your comments. We have included additional analysis regarding the correlation between sampling steps and the quality of video predictions, particularly with two sequence lengths (12 and 16) in Appendix section B.3 of the revision. Specifically, we conducted an assessment for both 12-frame and 16-frame video predictions using various DDIM sampling steps (10, 20, 30, 40, 50, 60 DDIM steps). All generated outputs were sampled using a 12-frame SSv2 fine-tuned model. The comparative results are illustrated in Appendix Figure 10 of the revision and the following table. Notably, the 16-frame curve experiences a steeper decline from DDIM steps 10 to 20 compared to the 12-frame curve. As both curves progress beyond DDIM step 30, they tend to stabilize and show marginal gains. These results collectively demonstrate that increasing DDIM sampling steps notably improves video quality with longer video sequences when the sampling steps are in the lower range (10 to 30 DDIM steps). However, the improvement of generated video is limited as the DDIM steps are increased to larger than 30.

Additionally, the qualitative results of the 16-frame video are illustrated in Appendix Figure 12 of the revision. Although the generated 16-frame video shows a more downgraded quality compared to the 12-frame video, both 16-frame and 12-frame videos can exhibit coherent motion aligned with language instruction. And since 16-frame videos are not the expected sequence length for a 12-frame model, the generation quality of the 16-frame video can be further improved by training a dedicated 16-frame Seer model.

**Appendix Figure 10 results**

|         |            |            |
| ------- | ---------- | ---------- |
| **FVD** | seq len=12 | seq len=16 |
| DDIM=10 | 280.7      | 772.8      |
| DDIM=20 | 129.8      | 448.2      |
| DDIM=30 | 112.9      | 355.4      |
| DDIM=40 | 101.3      | 359.3      |
| DDIM=50 | 101.8      | 348.7      |
| DDIM=60 | 105.2      | 346.2      |

|         |            |            |
| ------- | ---------- | ---------- |
| **KVD** | seq len=12 | seq len=16 |
| DDIM=10 | 0.73       | 2.00       |
| DDIM=20 | 0.16       | 1.11       |
| DDIM=30 | 0.12       | 0.56       |
| DDIM=40 | 0.08       | 0.63       |
| DDIM=50 | 0.08       | 0.47       |
| DDIM=60 | 0.06       | 0.48       |

Reviewer comment (Theory):
What is the inductive bias that forces the Text Decomposer to produce coherent sub-instructions? Its unclear why it would segment the instructions into substeps on its own.

Author response:
In our work on generating subinstruction embeddings, the identity mapping of global instruction, the distillation of frozen 2D text-to-image guidance, and visual temporal alignment in 3D temporal layers collectively contribute to the generation of coherent subinstructions. Specifically, based on the above three constraints, the optimization of the FSText decomposer follows a two-stage process. Initially, this training process forces the sub-instructions to have a context distribution close to the global instruction, which is interpretable for the pre-trained text-image conditional U-Net during the first stage.  
In the second stage, the learnable initial tokens are transformed into optimized conditional embeddings via the FSText decomposer. Through the end-to-end training of the diffusion model, these optimized conditional embeddings are constrained by the frozen pre-trained text-to-image conditioning module of the U-Net to generate related content within the image space. The training video samples provide supervision of temporal alignment within the trainable 3D layers during the fine-tuning period. The combination of the supervised visual-temporal signal from the temporally aligned 3D layer and the guidance distillation from the frozen text-to-image conditioning ensures the optimization of these subinstructions during the diffusion process.

Reviewer comment (Experiments):
The design choices behind FSText are explained by the authors in Sec 4.3, but the overly complicated design choices could limit the utility of SEER outside of highly specific stepwise instruction videos. Did the authors perform any ablation studies that justified the various design choices in FSText (e.g. using cross-attention, using directed temporal attention, starting with CLIP text embedding etc.)

Author response:
Thank you for your suggestion. we have included additional ablation results for the FSText network design in Appendix Table 11 of the revision. To ensure a fair comparison within the same network depth, we opted to replace the ablated component with a Cross-Attention layer. Notably, the Cross-Attention layer plays a pivotal role in the FSText decomposer network for global instruction projection, and our experimentation revealed that removing this layer induces a vanishing gradient problem during training. Consequently, we did not perform an ablation study on the Cross-Attention layer in the FSText decomposer. As presented in Appendix Table 10, our proposed configuration, featuring appended Text-Sequential-Attention layers and Temporal Attention layers, outperforms the other two settings. This outcome underscores the critical functionality of Text-Sequential-Attention layers and Temporal Attention layers in capturing text contextual information and modeling temporal dependencies, which all contribute to the performance improvement effectively.

**Appendix Table 11**. Components in FSText Decomposer (SSv2)

|                                       |         |         |
| ------------------------------------- | ------- | ------- |
| **Components in FSText.**             | **FVD** | **KVD** |
| Temporal-Atten.                       | 125.8   | 0.13    |
| Text-Seq-Atten.                       | 127.7   | 0.14    |
| Text-Seq-Atten.+Temporal-Atten.(Ours) | 112.9   | 0.12    |

Reviewer comment (Experiments):
For the ablations, could these be reported on more than one dataset? Results on one particular task in the SSv2 are quite convincing, but I am wondering if there could be dataset biases that seem to justify these design choices. For example, the BridgeData is a robotic task dataset versus human, so maybe that could be a good candidate to ensure results are not cherry picked.

Author response:
Thank you for your constructive advice. We have incorporated additional ablation results in Appendix Table 12 and Table 13, specifically including the BridgeData dataset, in our revision. In these results, we consistently observe performance improvements with Seer across various settings. Notably, similar to the ablation results on SSv2, the additional fine-tuning "cross+temp-attn." leads to significant degradation of Seer performance, as observed in the results presented in Appendix Table 13. These findings underscore the robustness of the Seer design, reaffirming its consistent performance across different datasets.

**Appendix Table 12**. Ablation results of temporal attention in Inflated 3D U-Net (BridgeData)

|                        |         |         |
| ---------------------- | ------- | ------- |
| **Temporal Attention** | **FVD** | **KVD** |
| bi-direct.             | 284.5   | 0.71    |
| directed.              | 258.0   | 0.64    |
| autoreg.               | 261.5   | 0.83    |
| win-auto.(Ours)        | 246.3   | 0.55    |

**Appendix Table 13**. Ablation results of fine-tune setting in Inflated 3D U-Net (BridgeData)

|                       |                     |         |         |
| --------------------- | ------------------- | ------- | ------- |
| **Fine-tune setting** | **Use** **FSText.** | **FVD** | **KVD** |
| temp-attn.            | No                  | 410.7   | 0.97    |
| cross+temp-attn.      | No                  | 319.9   | 1.01    |
| cross+temp-attn.      | Yes                 | 2058.4  | 9.43    |
| temp-attn.            | Yes                 | 246.3   | 0.55    |

Reviewer comment (Evaluation):
In Sec 1. Introduction *'existing text-conditioned video generation task (use) text specifying the general content, such as “a person is skiing”, while our aim in the TVP task is to use the text as a “task descriptor” in robotics, such as “tipping a beer can”'*. Could the authors please explain how does using text as a task descriptor differ from text specifying general content? Does the VLM model encode the intent of text in the former, and if so are there significant differences across the two representations (e.g. CKA value)? How does it relate to the frame aligned sub-instructions in the FSText decomposer?

Author response:
Thank you for your feedback. Our work places a primary emphasis on temporal movement. Unlike text that specifies general content and tends to guide the generation of similar scenes with inertial motion along the frame axis, our approach centers around a task descriptor. This descriptor guides the generation of a sequence of behaviors, necessitating the decomposition of one-step global behavioral instructions into multi-step, fine-grained motion guidance. Compared to our method with task descriptor guidance, most text-to-video and text-video-editing methods, such as Make-A-Video \[1\], MagicVideo \[2\], Align your Latents \[3\], and Tune-A-Video \[4\] more focus on the alignment between text-specifying content and the overall scene generation of frames with temporal consistency, but neglect the fine-grained motion guidance at the task level.

*References:*

*\[1\] Uriel Singer, et al. "Make-a-video: Text-to-video generation without text-video. data." arXiv preprint arXiv:2209.14792, 2022*

*\[2\] Daquan Zhou, et al. "Magicvideo: Efficient video generation with latent diffusion models." arXiv preprint arXiv:2211.11018, 2022*

*\[3\] Andreas Blattmann, et al. "Align your latents: High-resolution video synthesis with latent diffusion models." In Proceedings of the *IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 22563-22575, 2023*

*\[4\] Jay Zhangjie Wu, et al. "Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation." In Proceedings of the *IEEE/CVF International Conference on Computer Vision*, pp. 7623--7633, 2023*