YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
InstructPix2Pix for AnimateDiff
This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model.
Inference
You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is here
Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully.
Init Image
Make her smile
Make her on fire
Training
The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1.
LICENSE
Creative OpenRAIL License