zeromodels/t5-v1_1-xxl-encoder

Paper: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (arXiv:1910.10683)

Pure-Keras 3 conversion of the T5 v1.1 XXL encoder (Google's t5-v1_1-xxl, as shipped in stabilityai/stable-diffusion-3-medium-diffusers as text_encoder_3) for zeromodels, as the Stable Diffusion 3 family's SD3T5EncoderModel: 24 layers, 4096-d, 64 heads, gated-GELU feed-forward (10240 wide), 4.76B parameters in model.weights.json shards (8.87 GB, float16), plus zm_config.json. It is the third text encoder of Stable Diffusion 3 and 3.5: every SD 3 / 3.5 repo under zeromodels/ attaches this one repo with text_encoder_3=, and the SD 3 tokenizer's input_ids_3 are its input.

Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from zeromodels.models.stable_diffusion_3 import StableDiffusion3TextToImage

model = StableDiffusion3TextToImage.from_weights("zeromodels/stable-diffusion-3-medium", text_encoder_3="zeromodels/t5-v1_1-xxl-encoder")

On its own (the prompt ids come from the SD 3 tokenizer):

from zeromodels.models.stable_diffusion_3 import SD3T5EncoderModel, StableDiffusion3Tokenizer

encoder = SD3T5EncoderModel.from_weights("zeromodels/t5-v1_1-xxl-encoder")  # float16; load_dtype="float32" / quantization="int8"
tokenizer = StableDiffusion3Tokenizer.from_weights("zeromodels/stable-diffusion-3-medium")
ids = tokenizer("a photograph of an astronaut riding a horse")["input_ids_3"]  # (1, 256)
features = encoder({"input_ids": ids, "attention_mask": ids * 0 + 1})["last_hidden_state"]  # (1, 256, 4096)

License

The weights are redistributed under the Apache License 2.0 of the upstream checkpoint. By using them you agree to those terms.

Notice

Modifications by zeromodels (https://github.com/IMvision12/ZeroModels): the checkpoint released at https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers was converted to the Keras 3 weights layout of zeromodels (model.weights.json, model_00000.weights.h5, model_00001.weights.h5, zm_config.json), only its text_encoder_3 folder (Google's T5 v1.1 XXL encoder, google/t5-v1_1-xxl), stored in float16 as shipped there. The model architecture and the parameter values are unchanged; the weight names and the file format differ from the release.