| # InstructPix2Pix for AnimateDiff | |
| This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model. | |
| ## Inference | |
| You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is [here](https://github.com/viddle-app/viddle-pix2pix-animatediff) | |
| Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully. | |
| ### Init Image | |
|  | |
| _Make her smile_ | |
|  | |
| _Make her on fire_ | |
|  | |
| ## Training | |
| The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1. | |
| ## LICENSE | |
| Creative OpenRAIL License |