h1t/oms_b_openclip_15_21

One More Step

One More Step (OMS) module was proposed in One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls by Minghui Hu, Jianbin Zheng, Chuanxia Zheng, Tat-Jen Cham et al.

By adding one small step on the top the sampling process, we can address the issues caused by the current schedule flaws of diffusion models without changing the original model parameters. This also allows for some control over low-frequency information, such as color.

Our model is versatile and can be integrated into almost all widely-used Stable Diffusion frameworks. It's compatible with community favorites such as LoRA, ControlNet, Adapter, and foundational models.

Usage

OMS now is supported 🤗 diffusers with a customized pipeline github. To run the model (especially with LCM variant), first install the latest version of diffusers library as well as accelerate and transformers.

pip install --upgrade pip
pip install --upgrade diffusers transformers accelerate

And then we clone the repo

git clone https://github.com/mhh0318/OneMoreStep.git
cd OneMoreStep

SD15 and SD21

Due to differences in the VAE latent space between SD1.5/SD2.1 and SDXL, the OMS module for SD1.5/SD2.1 cannot be shared with SDXL, however, SD1.5/SD2.1 can share the same OMS module as well as with models like LCM that are based on SD1.5 or SD2.1. For more details, please refer to our paper.

We have uploaded one OMS module for SD15/21 series at h1t/oms_b_openclip_15_21, which has a base architecture, an OpenCLIP text encoder.

We simply put a demo here:

import torch
from diffusers import StableDiffusionPipeline, LCMScheduler

sd_pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16, variant="fp16", safety_checker=None).to('cuda')

pipe = OMSPipeline.from_pretrained('h1t/oms_b_openclip_15_21', sd_pipeline = sd_pipe, torch_dtype=torch.float16, variant="fp16", trust_remote_code=True)
pipe.to('cuda')

generator = torch.Generator(device=pipe.device).manual_seed(100)

prompt = "a starry night"

image = pipe(prompt, guidance_scale=7.5, num_inference_steps=20, oms_guidance_scale=2., generator=generator)

image['images'][0]

and without OMS:

image = pipe(prompt, guidance_scale=7.5, num_inference_steps=20, oms_guidance_scale=2., generator=generator, oms_flag=False)

image['images'][0]

We found that the quality of the generative model has been greatly improved. For more models and more functions like diverse prompt, please refer to OMS Repo.