Spaces:
Running
Running
| # Outpainting Pipeline | |
| - Intial Version of the project used a combination of Yolov8s Segmentation model to provide a rough mask , which was then inverted for Outpainting to use with Models like stable diffusion inpainting model from Runway along with ControlNet to control the outpainted generated output , | |
| - There were blockers in that approach as first the mask , was of a poor quality with rough edges which were messing with the outpainting output , even with techniques like blurring the mask the output quality was poor initailly with stable diffusion models | |
| - To address this , changes like detecting the ROI of the object in focus in addition to extending and resizing the image was done , the model for segmentation was upgraded to Segment Anything VIT Huge with yolov8l model , providing the bounding boxes for the Box prompt which was then inverted for outpainting | |
| - The model was changed kandinsky-v2.2-decoder-inpaint with 800 inference steps , a guidence scale of 5.0 to 7.5 and then the following results were achieved | |
| - GPU used Nvidia A100 40GB | |
| ## ARCHITECTURE | |
|  | |
| ## Installation | |
| To install the necessary requirements, you can use pip: | |
| ```bash | |
| pip install -r requirements.txt | |
| wandb login | |
| huggingface-cli login | |
| cd scripts | |
| ``` | |
| This will install all necessary libraries for this project, including PIL , Diffusers , Segment Anything, wandb , | |
| ```bash | |
| python run.py --image_path /path/to/image.jpg --prompt 'prompt' --negative_prompt 'negative prompt' --output_dir /path/to/output --mask_dir /path/to/mask --uid unique_id | |
| ``` | |
| ### MODELS USED | |
| EXPERIMENTATION WITH THE FOLLOWING models | |
| - https://huggingface.co/runwayml/stable-diffusion-inpainting | |
| - https://huggingface.co/lllyasviel/sd-controlnet-seg | |
| - https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder-inpaint | |
| ### WEIGHTS AND BIASES EXPERIMENTATION REPORT | |
| [WANDB REPORT](https://wandb.ai/vikramxd/product_placement_api/reports/Experimentation-Report---Vmlldzo3Mjg1MjQw) | |
|  | |
|  | |
|  | |
|  | |
|  | |
| ## Some Improvements | |
| - Working on API to deploy this model in batch mode adding loggers from prompt and generated output | |
| - Implementation of UI in Gradio / Streamlit for checking the model out in visual way | |
| - <s>Experimenting with image to video model pipeline to generate a video output thinking of using (https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt) model for this</s> | |
| ## Short Video | |
| https://github.com/VikramxD/product_diffusion_api/assets/72499426/c935ec2d-cb76-49dd-adae-8aa4feac211e | |