Paper: Probabilistic Adaptation of Black-Box Text-to-Video Models (ICLR-2024)

Reviewer comment (Experiments):
The authors assume pretrained model weights are not accessible. Table 1 and figure 9 verifies the authors’ claim. However, it is not clear how does the model perform compared to other methods, such as LoRA and prefix-tuning if pretrained model weights are accessible.

Author response:
> Comparison to parameter efficient finetuning.

The problem setting of VideoAdapter is that pretrained model weights are **NOT** accessible. This scenario is common in large language models (e.g., GPT-4, Bard), and large text-to-video models are heading in the same direction. Parameter efficient finetuning (e.g., LoRA, null-text inversion, prefix-tuning) requires access to pretrained model weights, whereas VideoAdaptor does not. Therefore VideoAdapter is not comparable to parameter efficient finetuning.

Nevertheless, we conducted comparisons to LoRA and null-text inversion out of curiosity (prefix-tuning is omitted since it has only been applied to language models). For LoRA, we use rank 1 and rank 64 to compare to the smaller and larger task-specific VideoAdapter model. For null-text inversion, we use an unconditional null embedding of size [64, 4096] (the same dimension as the original text embeddings). We report the video modeling metrics in the following table: 


| Method | Bridge FVD &darr; | Bridge FID &darr; | Ego4D FVD &darr; | Ego4d IS &uarr; |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| VideoAdapter Small | 177.4 | 37.6 | 156.3 | 2.8 |
| LoRA-Rank1 | 170.2 | 32.2 | 74.5 | 3.4 |
| Small (no adaption)  | 165.5 | 30.1 | 65.1 | 3.3 |
| Video Adapter Large | 148.1 | 29.5 | 52.5 | 3.5 |
| LoRA-Rank64 | 165.5 | 31.6 | 50.3 | 3.5 |
| Null-text inversion | 288.8 | 40.2 | 90.2 | 3.1 |

We observe that LoRA-Rank1 performs slightly better than VideoAapter (small). However, In the LoRA-Rank1 case, LoRA still performs worse than training a small domain specific model. In this case, VideoAdapter can simply use the small model without the pretrained prior. In comparison, we found LoRA-Rank64 leads to mixed results when compared to VideoAdaptor (large), i.e., LoRA outperforms VideoAdapter on Ego4D but not on Bridge data. We found that null-text inversion performs the worst, potentially due to limited flexibility of null-embeddings during finetuning.

Our results illustrate that VideoAdapter, despite requiring only black-bo adaptation without access to pretrained model weights performs better than Null-text inversion and very comparably to LoRA finetuning (with pretrained model weights).

Reviewer comment (Experiments):
Method in Sec 3 is not limit to text-to-video generation and should also be applicable to text-to-image generation. Have you considered other use cases?

Author response:
> Use case in text-to-image adaptation.

VideoAdapter is indeed applicable to the text-to-image setting, as text-to-image is a special case of text-to-video. We focused on the text-to-video setting because the pretrained text-to-video models are generally much bigger than pretrained text-to-image models, which requires even more efficient adaptation.

Reviewer comment (Theory):
Only two models (a pretrained model and a domain-specific model) are involved in derivation in Sec. 3. If we have more than 1 domain-specific model, can we easily extend this method to multiple domain-specific models?

Author response:
> Composing multiple models.

This is not only possible but in favor of VideoAdapter compared to other adaptation methods (e.g., efficient finetuning), since VideoAdapter can compose arbitrary black-box models at inference time. We provide an additional analysis on adaptation in Appendix D.

Reviewer comment (Experiments):
I know the assumption here is that pretrained models are not accessible during training. But what if we can access the pre-trained models during training, will it outperform LoRA and prefix-finetuning?

Author response:
> Comparison to parameter efficient finetuning.

The problem setting of VideoAdapter is that pretrained model weights are **NOT** accessible. This scenario is common in large language models (e.g., GPT-4, Bard), and large text-to-video models are heading in the same direction. Parameter efficient finetuning (e.g., LoRA, null-text inversion, prefix-tuning) requires access to pretrained model weights, whereas VideoAdaptor does not. Therefore VideoAdapter is not comparable to parameter efficient finetuning.

Nevertheless, we conducted comparisons to LoRA and null-text inversion out of curiosity (prefix-tuning is omitted since it has only been applied to language models). For LoRA, we use rank 1 and rank 64 to compare to the smaller and larger task-specific VideoAdapter model. For null-text inversion, we use an unconditional null embedding of size [64, 4096] (the same dimension as the original text embeddings). We report the video modeling metrics in the following table: 


| Method | Bridge FVD &darr; | Bridge FID &darr; | Ego4D FVD &darr; | Ego4d IS &uarr; |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| VideoAdapter Small | 177.4 | 37.6 | 156.3 | 2.8 |
| LoRA-Rank1 | 170.2 | 32.2 | 74.5 | 3.4 |
| Small (no adaption)  | 165.5 | 30.1 | 65.1 | 3.3 |
| Video Adapter Large | 148.1 | 29.5 | 52.5 | 3.5 |
| LoRA-Rank64 | 165.5 | 31.6 | 50.3 | 3.5 |
| Null-text inversion | 288.8 | 40.2 | 90.2 | 3.1 |

We observe that LoRA-Rank1 performs slightly better than VideoAapter (small). However, In the LoRA-Rank1 case, LoRA still performs worse than training a small domain specific model. In this case, VideoAdapter can simply use the small model without the pretrained prior. In comparison, we found LoRA-Rank64 leads to mixed results when compared to VideoAdaptor (large), i.e., LoRA outperforms VideoAdapter on Ego4D but not on Bridge data. We found that null-text inversion performs the worst, potentially due to limited flexibility of null-embeddings during finetuning.

Our results illustrate that VideoAdapter, despite requiring only black-bo adaptation without access to pretrained model weights performs better than Null-text inversion and very comparably to LoRA finetuning (with pretrained model weights).

Reviewer comment (Experiments):
My major concern is the lack of comparison with other baseline models like Lora, null-text inversion and prefix-tuning. Surely, these methods I mentioned above were designed for image adaptation but, correct me if I am wrong, it seems that they can be easily transplanted to the video diffusion model to serve as baselines models. Authors should explain more about why such comparison is not necessary.

Author response:
The problem setting of VideoAdapter is that pretrained model weights are **NOT** accessible. This scenario is common in large language models (e.g., GPT-4, Bard), and large text-to-video models are heading in the same direction. Parameter efficient finetuning (e.g., LoRA, null-text inversion, prefix-tuning) requires access to pretrained model weights, whereas VideoAdaptor does not. Therefore VideoAdapter is not comparable to parameter efficient finetuning.

Nevertheless, we conducted comparisons to LoRA and null-text inversion out of curiosity (prefix-tuning is omitted since it has only been applied to language models). For LoRA, we use rank 1 and rank 64 to compare to the smaller and larger task-specific VideoAdapter model. For null-text inversion, we use an unconditional null embedding of size [64, 4096] (the same dimension as the original text embeddings). We report the video modeling metrics in the following table: 


| Method | Bridge FVD &darr; | Bridge FID &darr; | Ego4D FVD &darr; | Ego4d IS &uarr; |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| VideoAdapter Small | 177.4 | 37.6 | 156.3 | 2.8 |
| LoRA-Rank1 | 170.2 | 32.2 | 74.5 | 3.4 |
| Small (no adaption)  | 165.5 | 30.1 | 65.1 | 3.3 |
| Video Adapter Large | 148.1 | 29.5 | 52.5 | 3.5 |
| LoRA-Rank64 | 165.5 | 31.6 | 50.3 | 3.5 |
| Null-text inversion | 288.8 | 40.2 | 90.2 | 3.1 |

We observe that LoRA-Rank1 performs slightly better than VideoAapter (small). However, In the LoRA-Rank1 case, LoRA still performs worse than training a small domain specific model. In this case, VideoAdapter can simply use the small model without the pretrained prior. In comparison, we found LoRA-Rank64 leads to mixed results when compared to VideoAdaptor (large), i.e., LoRA outperforms VideoAdapter on Ego4D but not on Bridge data. We found that null-text inversion performs the worst, potentially due to limited flexibility of null-embeddings during finetuning.

Our results illustrate that VideoAdapter, despite requiring only black-bo adaptation without access to pretrained model weights performs better than Null-text inversion and very comparably to LoRA finetuning (with pretrained model weights).

Reviewer comment (Evaluation):
In table 1, small+pretrained demonstrate the best quantitative performance on both Bridge and Ego4D. However, these numbers are very close to only small variant. Considering the poor generative quality of small model in Figure 7 and 8, it is questionable whether small+pretrained can generate high-quality videos consistently.

Author response:
We used the larger version of the small model, i.e., Small (L), in Figure 7 and Figure 8. We found that for harder dataset (e.g., Ego4D), too small of an adapter model fails to adapt to the target domain. The size of the adapter should in fact be determined by the complexity of the video domain one wishes to adapt to, and VideoAdapter enables such flexibility during adaptation. We also found that video modeling metrics sometimes do not faithfully reflect sample quality, especially in failing to reflect whether the generated videos corresponds to the given text instruction.

Reviewer comment (Experiments):
The prior strength weight is constant during the sampling. It would be interesting to see if that the prior strength weight is changing with t could be helpful.

Author response:
We have indeed found tunable prior strength to be helpful. For instance, we set the prior strength to 0 in the last 100 timesteps of diffusion (last paragraph in Appendix B.1), which we found to result in better generation quality than a fixed prior strength. In general, we found that  higher strengths at higher noise levels were helpful – perhaps due to more accurate score functions at higher timesteps. We have made tunable prior strength more explicit in Section 3.2 (Page 4).

Reviewer comment (Writing):
The description of “Controllable video synthesis” is not very clear in terms of inputs. The text description as inputs is not provided very clearly hence someone without knowing the problem statement of this particular problem has difficulty in understanding what are the inputs.

Author response:
We have updated “Controllable video synthesis” to “Adapting to Specific Video Domains” for clarity.

Reviewer comment (Theory):
The paper highlights the ability of adapting black-box pretrained large text-video model. However, it relaxes the assumption of obtaining intermediate generation results of the black-box model, which is not realistic, and even contradicts with the term "black-box". It is inappropriate to use an existing term in a completely different setting.

Author response:
We believe that our approach does act in a “black box” manner since it does not require access or knowledge about any of the weights of a neural network (treating the neural network itself as a “black box”), but rather only access to predictions from the neural network (albeit at each point of sample generation). While this uses more information than having only the final generated sample, there is no free lunch, and the greater the information given, the more effectively models are able to be adapted. There is precedence for giving access to such intermediate predictions in black-box models, for example the chatGPT API gives intermediate prediction logits.

However, if the reviewer feels strongly about this, we are happy to adopt another word to describe our approach.

Reviewer comment (Experiments):
The manuscript highlights parameter efficiency, but the current evidence is not enough to support this claim. There have been many parameter efficient fine-tuning techniques, and comparison with such existing methods is needed to claim superiority. Specifically, comparisons with LoRA and other SOTA PEFT methods should be provided to let the audience understand the performance of existing methods.

Author response:
The problem setting of VideoAdapter is that pretrained model weights are NOT accessible. This scenario is common in large language models (e.g., GPT-4, Bard), and large text-to-video models are heading in the same direction. Parameter efficient finetuning (e.g., LoRA, null-text inversion, prefix-tuning) requires access to pretrained model weights, whereas VideoAdaptor does not. Therefore VideoAdapter is not comparable to parameter efficient finetuning.

Nevertheless, we conducted comparisons to LoRA and null-text inversion out of curiosity (prefix-tuning is omitted since it has only been applied to language models). For LoRA, we use rank 1 and rank 64 to compare to the smaller and larger task-specific VideoAdapter model. For null-text inversion, we use an unconditional null embedding of size [64, 4096] (the same dimension as the original text embeddings). We report the video modeling metrics in the following table: 


| Method | Bridge FVD &darr; | Bridge FID &darr; | Ego4D FVD &darr; | Ego4d IS &uarr; |
| ----------- | ----------- | ----------- | ----------- | ----------- |
| VideoAdapter Small | 177.4 | 37.6 | 156.3 | 2.8 |
| LoRA-Rank1 | 170.2 | 32.2 | 74.5 | 3.4 |
| Small (no adaption)  | 165.5 | 30.1 | 65.1 | 3.3 |
| Video Adapter Large | 148.1 | 29.5 | 52.5 | 3.5 |
| LoRA-Rank64 | 165.5 | 31.6 | 50.3 | 3.5 |
| Null-text inversion | 288.8 | 40.2 | 90.2 | 3.1 |

We observe that LoRA-Rank1 performs slightly better than VideoAapter (small). However, In the LoRA-Rank1 case, LoRA still performs worse than training a small domain specific model. In this case, VideoAdapter can simply use the small model without the pretrained prior. In comparison, we found LoRA-Rank64 leads to mixed results when compared to VideoAdaptor (large), i.e., LoRA outperforms VideoAdapter on Ego4D but not on Bridge data. We found that null-text inversion performs the worst, potentially due to limited flexibility of null-embeddings during finetuning.

Our results illustrate that VideoAdapter, despite requiring only black-bo adaptation without access to pretrained model weights performs better than Null-text inversion and very comparably to LoRA finetuning (with pretrained model weights).

Reviewer comment (Experiments):
The current setup is not strongly emphasizing efficiency. The largest small model can take half of the computation of the large model. It would be more rigorous if the authors set a computation budget and compare methods under a fixed budget.

Author response:
In the Bridge setting, our small model is substantially smaller in size than our original pretrained model. However, in more complex domains, it is necessary to use larger models to capture the characteristics of that domain, and thus our larger model is similar in size to our original pretrained model.  In the table above, however, we have equated the computational budget across both our approach and parameter-efficient adaptation methods and find that our approach performs comparably to such approaches.

Reviewer comment (Evaluation):
Evaluation of the video generation is insufficient. From qualitative inspection, Video Adapter in Figure 8 does not seem clearly better. There should be further case studies examining human preference and the scores obtained from automatic evaluation, especially on datasets like Ego4D.

Author response:
We have added a video comparison in https://drive.google.com/file/d/16O_RsGhwDhUmUsHTf8lSW8LvvCbEQ01r/view?usp=sharing. From these video results, it’s clear that our approach substantially better than the pretrained model without adaptation.