zeromodels/stable-diffusion-xl-refiner-1.0

See our collection for all Stable Diffusion XL checkpoints.

Run Stable Diffusion XL with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/stable-diffusion-xl-refiner-1.0

Paper: SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (arXiv:2307.01952) | HF Papers

Pure-Keras 3 conversion of stabilityai/stable-diffusion-xl-refiner-1.0 for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX. The whole text-to-image model ships as one container: the UNet denoiser, the VAE and the CLIP ViT-L/14 and OpenCLIP ViT-bigG/14 text encoders (penultimate layers) in model.weights.json shards (3.04B parameters, 5.81 GB), plus zm_config.json (the four component configs, the checkpoint's EulerDiscreteScheduler schedule with its epsilon objective and the default generation settings) and the tokenizer as tokenizer.json. Weights are stored in float16, the checkpoint's native precision (the VAE in float32: it overflows in float16), and load in float16 by default; pass load_dtype="float32" to from_weights for a float32 model. This checkpoint generates 1024x1024 images (a 128x128 latent).

For model details, intended use and limitations, see the upstream model card.

Architecture

Component	zeromodels class	Details
Denoiser	UNet2DConditionModel	(384, 768, 1536, 1536) channels, 2 ResNet blocks per level, (6, 12, 24, 24) attention heads on the 1280-d text context, linear token projection, 4 transformer blocks per level, text_time micro-conditioning (pooled text embedding + size / crop ids), 128x128x4 latent
Autoencoder	AutoencoderKL	(128, 256, 512, 512) channels, x8 spatial compression to 4 latent channels, scaling_factor 0.13025, float32 (force_upcast)
Text encoder	functional CLIP text tower	OpenCLIP ViT-bigG/14: 1280-d, 32 layers, 20 heads, gelu, 1280-d projection; penultimate hidden state + projected pooled state
Scheduler	EulerDiscreteScheduler	scaled_linear betas 0.00085 to 0.012 over 1000 steps, epsilon, leading timestep spacing; DDIM / PNDM / Euler / Euler-ancestral are drop-in

Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.models.stable_diffusion_xl import StableDiffusionXLRefinerImageToImage, StableDiffusionXLTokenizer

model = StableDiffusionXLRefinerImageToImage.from_weights("zeromodels/stable-diffusion-xl-refiner-1.0")
tokenizer = StableDiffusionXLTokenizer.from_weights("zeromodels/stable-diffusion-xl-refiner-1.0")

inputs = tokenizer("a photograph of an astronaut riding a horse")
images = model.generate(**inputs, num_inference_steps=50, guidance_scale=5.0, seed=0)
Image.fromarray(images[0]).save("astronaut.png")  # (1024, 1024, 3) uint8

This is the refiner, an image-to-image model: the second half of the SDXL ensemble of experts (the base model denoises the first 80% of the schedule, the refiner the rest), or a refiner of any image with strength. The plain text-to-image call above works but is not what it was trained for. Ensemble:

from zeromodels.models.stable_diffusion_xl import StableDiffusionXLTextToImage

base = StableDiffusionXLTextToImage.from_weights("zeromodels/stable-diffusion-xl-base-1.0")
latent = base.generate(**inputs, num_inference_steps=50, denoising_end=0.8, output_type="latent")
images = model.generate(**inputs, latents=latent, num_inference_steps=50, denoising_start=0.8)

Image-to-image (image is (batch, H, W, 3) uint8 or [0, 1] float; strength 0.3 by default, aesthetic_score 6.0 / negative_aesthetic_score 2.5):

images = model.generate(**inputs, image=images, strength=0.3, seed=0)

generate takes the tokenizer's input_ids (batch them for several prompts), an optional negative_input_ids (tokenize the negative prompt), num_inference_steps, guidance_scale, a seed, or explicit latents of shape (batch, 128, 128, 4) for results that are identical across backends.

Load any Stable Diffusion XL checkpoint the same way with from_weights("zeromodels/<variant>"):

Variant	Hub	Training
stable-diffusion-xl-base-1.0	zeromodels/stable-diffusion-xl-base-1.0	1024px, epsilon, Euler (leading spacing): the SDXL 1.0 base model, multi-aspect training with size / crop micro-conditioning
stable-diffusion-xl-refiner-1.0	zeromodels/stable-diffusion-xl-refiner-1.0	image-to-image refiner of SDXL 1.0: refines the base's latents (denoising_start 0.8) or an image (strength 0.3); one text tower, aesthetic-score conditioning
sdxl-turbo	zeromodels/sdxl-turbo	512px, epsilon, ancestral Euler (trailing spacing), 1 to 4 steps, no guidance: SDXL 1.0 distilled with Adversarial Diffusion Distillation (Stability AI Community License)

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• The graphs are built for 1024px. Pass unet_sample_size=<px / 8>, vae_sample_size=<px> to from_weights to build for another multiple of 64px (the weights are resolution-independent).
• Swap the sampler any time: model.scheduler = EulerDiscreteScheduler.from_config(model.config.scheduler_config) (zeromodels.base.base_scheduler).
• StableDiffusionXLRefinerModel.from_weights(...) loads the same repo as the bare container (UNet / VAE / text encoders as .unet / .vae / .text_encoder / .text_encoder_2) without the generation loop.
• SDXL micro-conditioning: generate(..., original_size=(h, w), crops_coords_top_left=(top, left), target_size=(h, w)), plus negative_* variants; the sizes default to the image size. Without a negative prompt the unconditional branch is zero embeddings (force_zeros_for_empty_prompt), as in diffusers.
• Both channels_last and channels_first are supported (keras.config.set_image_data_format before loading); generate always returns (batch, H, W, 3) uint8.
• On-the-fly hf: conversion is not supported for diffusion models; the checkpoints are hosted here, converted once.
• See the Stable Diffusion XL docs.

License

The weights are redistributed under the CreativeML Open RAIL++-M License of the upstream checkpoint, including its use-based restrictions. By using them you agree to those terms.

Notice

Modifications by zeromodels (https://github.com/IMvision12/ZeroModels): the checkpoint released at https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0 was converted to the Keras 3 weights layout of zeromodels (model.weights.json, model_00000.weights.h5, model_00001.weights.h5, zm_config.json, tokenizer.json), stored in float16, the upstream fp16 files, with the VAE in float32. The model architecture and the parameter values are unchanged; the weight names and the file format differ from the release.

Special Thanks

Thank you to Stability AI and the LAION / OpenCLIP teams for training and releasing Stable Diffusion, and to the Hugging Face diffusers team, whose implementation this port was verified against.