Edit model card

KerasCV Stable Diffusion in Diffusers πŸ§¨πŸ€—

DreamBooth model for the puggieace concept trained by nielsgl on the nielsgl/dreambooth-ace dataset. It can be used by modifying the instance_prompt: a photo of puggieace.

The examples are from 2 different Keras CV models (StableDiffusion and StableDiffusionV2, corresponding to Stable Diffusion V1.4 and V2.1, respectively) trained on the same dataset (nielsgl/dreambooth-ace).

Description

The Stable Diffusion V2 pipeline contained in the corresponding repository (nielsgl/dreambooth-keras-pug-ace-sd2.1) was created using a modified version of this Space for StableDiffusionV2 from KerasCV. The purpose is to convert the KerasCV Stable Diffusion weights in a way that is compatible with Diffusers. This allows users to fine-tune using KerasCV and use the fine-tuned weights in Diffusers taking advantage of its nifty features (like schedulers, fast attention, etc.). This model was created as part of the Keras DreamBooth Sprint πŸ”₯. Visit the organisation page for instructions on how to take part!

Examples

Stable Diffusion V1.4

Portrait of puggieace dog as a Roman Emperor, city in background

Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha

Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour

Photo of puggieace dog wearing sunglasses on the beach, sunset in background, golden hour

Photo of cute puggieace dog as an astronaut, planet and spaceship in background

Photo of cute puggieace dog as an astronaut, planet and spaceship in background, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. trending on artstation

Stable Diffusion V2.1

Portrait painting of a cute puggieace dog as a samurai

Portrait painting of a cute puggieace dog as a samurai, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha

Photo of cute puggieace dog as an astronaut, space and planet in background

Photo of cute puggieace dog as an astronaut, space and planet in background, ultra realistic, concept art, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater, trending on artstation

A photo of a cute puggieace dog getting a haircut in a barbershop

A photo of a cute puggieace dog getting a haircut in a barbershop, ultra realistic, intricate details, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha

Portrait photo of puggieace dog in New York

Portrait photo of puggieace dog in New York, city and skyscrapers in background, highly detailed, photorealistic, hdr, 4k

Portrait of puggieace dog as a Roman Emperor, city in background

Portrait of puggieace dog as a Roman Emperor, city in background, ultra realistic, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie bowater and magali villeneuve and alphonse mucha

Usage with Stable Diffusion V1.4

from huggingface_hub import from_pretrained_keras
import keras_cv
import matplotlib.pyplot as plt


model = keras_cv.models.StableDiffusion(img_width=512, img_height=512, jit_compile=True)
model._diffusion_model = from_pretrained_keras("nielsgl/dreambooth-pug-ace")
model._text_encoder = from_pretrained_keras("nielsgl/dreambooth-pug-ace-text-encoder")

images = model.text_to_image("a photo of puggieace dog on the beach", batch_size=3)
plt.imshow(image[0])

Usage with Stable Diffusion V2.1

from diffusers import StableDiffusionPipeline

pipeline = StableDiffusionPipeline.from_pretrained('nielsgl/dreambooth-keras-pug-ace-sd2.1')
image = pipeline().images[0]
image

Training hyperparameters

The following hyperparameters were used during training for Stable Diffusion v1.4:

Hyperparameters Value
name RMSprop
weight_decay None
clipnorm None
global_clipnorm None
clipvalue None
use_ema False
ema_momentum 0.99
ema_overwrite_frequency 100
jit_compile True
is_legacy_optimizer False
learning_rate 0.0010000000474974513
rho 0.9
momentum 0.0
epsilon 1e-07
centered False
training_precision float32
Downloads last month
1
Hosted inference API

Inference API does not yet support keras models for this pipeline type.

Dataset used to train keras-dreambooth/dreambooth-pug-ace