Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

InstructPix2Pix for AnimateDiff

This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model.

Inference

You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is here

Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully.

Init Image

Init Image

Make her smile

Smiling

Make her on fire

On Fire

Training

The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1.

LICENSE

Creative OpenRAIL License

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Unable to determine this model's library. Check the docs .