Generating Videos from Images with Stable Video Diffusion and FiftyOne
Tags: ai, computervision, machinelearning, datascience
Published: 2024-02-07 21:16:55

_Author: [Dan Gural](https://www.linkedin.com/in/daniel-gural/) (Machine Learning Engineer at [Voxel51](https://voxel51.com/))_

## Bring images to life in seconds

Stability AI just released [Stable Video Diffusion](https://stability.ai/news/stable-video-diffusion-open-ai-video-model), allowing a whole new way to experience GenAI! By passing in an image, you can now breathe life into it, allowing for a short clip to be generated. Now, you can use Stable Video Diffusion with your FiftyOne datasets with my new [image to video plugin](https://github.com/danielgural/img_to_video_plugin)!



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ae0inzl94ht04z0rvji.gif)


## Installation

To get started, you will need two things, FiftyOne installed and a [replicate](https://replicate.com/) account to run the model through its API. After you have an account set up and FiftyOne installed, in you command terminal run the following:

```
fiftyone plugins download https://github.com/danielgural/img_to_video_plugin
```
This will install the plugin into your FiftyOne app. Afterwards, we set our replicate token environment variable as well.

```
export REPLICATE_API_TOKEN=<your token>
```

## Execution

From there, load up any FiftyOne dataset, [quickstart](https://docs.voxel51.com/user_guide/dataset_zoo/datasets.html#dataset-zoo-quickstart) for example, and select the sample or samples that you want to turn into videos using Stable Video Diffusion. We will be using [this replicate endpoint](https://replicate.com/stability-ai/stable-video-diffusion/api?tab=python) for Stable Video DIffusion.

```
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

session = fo.launch_app(dataset)
```
Use the “ ` “ or backtick on your keyboard when in the app to open the operator menu, type in the `img2video` operator. From there you will get several options to change settings for the model run. The different parameters can be found [here](https://replicate.com/stability-ai/stable-video-diffusion/api?tab=python).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sxkjbuf3uvm5ilbg8ygk.png)

After executing, the results will be saved in a new dataset called image2video. The mp4 will be saved alongside wherever the original data is. Explore different inputs and play with different images. You never know how it might turn out! Here are some of my results:

**In**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/re40k2u88buzk16t1ouf.png)

**Out**


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9itilzgwq1gamooujw0l.gif)

**In**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lsk6k2uhgw9xzdxoou7f.jpeg)

**Out**


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u49alovawi5m1szlz7kf.gif)

**In**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aqd0g7vhhon6rayi0g2i.jpeg)

**Out**


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l19qx5rsaxt0065oge55.gif)

## Conclusion

No matter what image you put in, you should be set for a good laugh. I found that using the large 24 frame model typically leads to falling off the rails as seen in the last output. I do have to say that playing with this plugin is addicting and it is so much fun to plug more and more pictures in. I can't wait to see what cool ideas and works come from Stable Video Diffusion!

As always, be sure to check out all of our [FiftyOne Plugins ](https://github.com/voxel51/fiftyone-plugins)for more GenAI plugins for computer vision plus much, much more! If you are interested in building your own plugins, hop into the [community slack](https://slack.voxel51.com/) to join other developers and get access to tons of resources!

{% embed https://player.vimeo.com/video/826021982 %}