Creating Videos with Stable Video Diffusion
Tags: showdev, tutorial, ai, generativeai
Published: 2024-02-12 18:30:44

---
title: Creating Videos with Stable Video Diffusion
published: true
description: "Want to generate AI videos from your machine? Stability AI just released Stable Video Diffusion, which you can download and run on your own machine. Here's the full instructions to set it up."
tags: showdev,tutorial,ai,generativeai
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jfv97ih8hmqwek5nb6f1.png
canonical_url: https://www.jeremymorgan.com/blog/generative-ai/how-to-make-ai-generated-videos/
---

If you're interested in generative AI, you've likely seen tools like [Runway](https://www.runwayml.com) that generate video from an image. They're cool tools. Some of you, like me, want to run these tools for ourselves and have them on our personal machines. 

If that's you, you've come to the right place. We're going to install a tool to generate AI videos today. [Stable Video Diffusion](https://stability.ai/news/stable-video-diffusion-open-ai-video-model) was just released, and we're going to try it out.  

What we'll do:

- Install some pre-requisites
- Configure our system for Stable Diffusion Video
- Install the Stable Diffusion tools and checkpoints, and run it all with [Streamlit](https://streamlit.io/). 
- Generate [this video](https://twitter.com/JeremyCMorgan/status/1727178146989088841)

In this tutorial, I'm using Linux and [Anaconda](https://www.anaconda.com) to run this. The directions should be the same for Windows or Mac.

If this is your first time doing this kind of thing, don't worry. We'll start by setting up your environment with the right tools and dependencies and installing a few things. We aren't going to dive too deep into how this product works. We'll just install it and try it out. 

## Step 1: Install Dependencies

We're going to grab this repo from Stability AI.

```bash
git clone https://github.com/Stability-AI/generative-models.git && cd generative-models
```

Next, we'll create a virtual environment. It's essential to use Python 3.10. For some reason, it was fussy with the newer versions. 

```bash
conda create --name stablevideodiff python=3.10.0
conda activate stablevideodiff
```

And then install all the requirements:

```bash
pip3 install -r requirements/pt2.txt
```

And you'll see all the dependencies get installed:

!["How to make AI generated Videos"][0]


But you may see an error like this: 

```bash
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
```

!["How to make AI generated Videos"][1]

This is sometimes due to the lack of a RUST compiler. You can install it with this command (in Linux):

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

Once it's done installing you should see no errors. 

!["How to make AI generated Videos"][2]

## Step 2: A couple of finishing touches

First, in your application folder (generative-models) type in the following. 

```bash
pip3 install .
```
Then we want to install sdata for training: 

```bash
pip3 install -e git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata
```

And now we're just about ready to let this thing cook.

## Step 3: Running the Demo

Ok, now you want a demo. 

Log in to [Hugging Face](https://huggingface.co/). You can create an account for free if you don't have one. 

You will need to download [svd.safetensors](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid/blob/main/svd.safetensors) to the `/checkpoints` directory. If it's not there, create a directory named checkpoints at the base of the app. 


Note: I am running an RTX 4090 with 24G VRAM and have 64GB of RAM on the PC. I still had to do this:

```bash
export 'PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512'
```

This helps with memory allocation as you run it, so you aren't as likely to run out of memory. I did a few times before playing with it. 

Now we can fire up the web app:

```bash
PYTHONPATH=. streamlit run scripts/demo/video_sampling.py --server.port 8080
```

In your browser, you should see this: 

!["How to make AI generated Videos"][5]

So let's try it out. Click "load model".

It will start running.

Then you'll see this message:

!["How to make AI generated Videos"][6]

Awesome, 

Now I'll load up an image sample using the "Browse files" button you see above. 

And I want to set this to a lower value of 2, so I don't run out of VRAM. Again, I have 24G to work with, but I must be careful. Pytorch is happy to use all the memory for this.

!["How to make AI generated Videos"][7]

and then click sample. You will see it start to work:

!["How to make AI generated Videos"][8]

And there we have it!! We produced a video.

!["How to make AI generated Videos"][9]

It's only 2 seconds long but pretty awesome. You can [download it here](/hosted/demo.mp4)

I love it and can't wait to do more of these. 

## What did we make?

So here, I generated exactly 14 frames from an image. It's 2 seconds, 7 frames per second. 

You can change that setting here: 

!["How to make AI generated Videos"][10]

You can set how many frames total you can generate. I've managed to get up to 3 seconds but ran out of VRAM.

Your results may vary. 

## Conclusion

Congratulations, you did it! This tutorial taught us how to set up an environment for Stable Video Diffusion, install it, and run it. This is an excellent way to get familiar with generative AI models and how to tune Stable Diffusion. You can produce some awesome stuff by getting to know your way around. Experiment with different settings, frames, and images to see how you can push the boundaries of this thing. Your results may vary based on hardware limitations. I'm running a 4090, as I mentioned, and it's still getting taxed. But like with all these things, it's about the learning experience. 


What are you doing with Stable Diffusion or Generative AI today? [Let me know! Let's talk](https://www.twitter.com/JeremyCMorgan). 

Also, if you have any questions or comments, feel free to reach out.

Happy hacking! 


[0]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/szbazckff2ec3m52k0le.png
[1]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fosb50me2rcgk0qg6mhf.png
[2]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e49o9lteqwkd9qhjpfu5.png
[3]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rjxt7kjj24byivvgef80.png
[4]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mvv594ukgm69zf2lzhf1.png
[5]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/75udhicxetk8sv70n0za.png
[6]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8o65vbuo9gljv3965c9k.png
[7]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w5r87jvpqaddc32emaol.png
[8]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t71mskxild7chrugvtuh.png
[9]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qd7hfu8fpnn6hhq53pyx.png
[10]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jrwgpk29p2mjn8f81tyc.png
[11]: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ba5tpa4f8wv7fa47cj6a.png