oakoliver/eg2-iris-3b-adapter

EG2 -> Iris-3B conditioning adapter (permissive data, Apache-2.0)

A 105M-parameter adapter that lets EmbeddingGemma 2 (EG2, 0.74B, Apache-2.0) stand in for the Qwen3-VL-4B text encoder of Iris-3B (Apache-2.0, Speridlabs). EG2 per-token states (768-d) go through eight pre-LN transformer layers (1024 wide, 16 heads, FFN 4096) and come out as the 300 x 2560 lap_pooled conditioning rows Iris's frozen refiner expects. Five learned tail rows, no head rows. With it, Iris never loads Qwen3-VL: text stage 10 ms warm instead of 80 ms, and the same EG2 front end serves our Qwen-Image adapter.

Everything in the chain is Apache-2.0 or CC0 / CC BY, so these weights are released under Apache-2.0. Attribution for the training photographs is in iris_images_attribution.md.

Try it in the browser (free ZeroGPU): https://huggingface.co/spaces/oakoliver/eg2-iris-3b (Turbo, or Base with your own steps and CFG).

Files

file	what
adapter.safetensors	the adapter (sha256 6da1b9aa…a660ef), export of stage-2 step 4000
config.json	architecture, EG2 prefix, target (iris3b lap_pooled), text length 300
iris_images_attribution.md	per-image author and license of the 3,000 Wikimedia Commons photographs fetched for stage 2 (2,115 passed the caption filter and were used)
bench/report.md, bench/summary.json, bench/grid.png	the benchmark below with side-by-side images
examples/	two rows of the benchmark grid

Training data (all permissive)

• Stage 1 (text distillation onto Iris's own conditioning): COCO captions (CC BY 4.0), Localized Narratives (CC BY 4.0, Pont-Tuset et al., ECCV 2020), DOCCI (CC BY 4.0, Onoe et al., ECCV 2024), DiffusionDB prompts (CC0), 1.95M rows.
• Stage 2 (velocity distillation through the frozen Iris base DiT, 4,000 steps, resolution buckets up to 1216 x 832): 2,115 real photographs from Wikimedia Commons declared CC0 / CC BY 3.0 / CC BY 4.0 by their uploaders, captioned by Qwen3-VL-4B-Instruct (Apache-2.0) with titles and authors hidden from the captioner; 1,002 images generated by Iris Turbo itself; 1,173 OCR-verified synthetic text renders (open fonts, no benchmark words).
• Not used: ShareGPT4V, ESC-50, MSR-VTT, COCO images, Pexels, YouTube.

Results

Iris Turbo 1024², 30 prompts x 3 seeds, adapter versus Iris's native Qwen3-VL conditioning, paired per prompt (bootstrap 95 % CI):

metric	native	this adapter	previous (non-permissive) adapter
SigLIP2 x100, paired gap	0	-0.29 [-1.09, 0.41]	-0.12
CLIP-L, paired gap	0	-0.83	-0.25
GenEval-style pass (OWLv2, 24 prompts)	86.1 %	75.0 %	79.2 %
OCR word accuracy (6 rendered-text prompts)	100 %	22.2 %	27.8 %

General alignment is statistically indistinguishable from native. Rendered text and spatial relations ("a cat to the left of a dog") are the losses. Before the synthetic text renders were added, OCR word accuracy was 2.8 %.

Speed (Rust/Metal engine iris-rs, Apple M5 Max, exclusive GPU)

Turbo 1024², 4 steps: 2.2 to 2.4 s wall per image warm (GPU 1.9 to 2.0 s), 7.3 GB on the device, EG2 text stage 88 ms cold / 10 ms warm, refiner 0.98 s. Base model 100 steps: 112 s. Upscaler 4x: 50 dB against the reference.

How to run

With the iris-rs engine (Rust, Metal; the EG2 snapshot is read from the Hugging Face cache):

iris generate --weights turbo/ --text-backend eg2 --eg2-adapter ./ \
  --prompt "a brown dog running on the beach at sunset" --size 1024 --seed 7 --out dog.png

iris generate --weights base_bf16/ --mode base --steps 100 --cfg 3 --text-backend eg2 --eg2-adapter ./ --prompt "..."
iris serve --weights turbo/ --eg2-adapter ./ --listen 127.0.0.1:8793      # resident, POST /generate

The adapter also ships inside OakOliver Studio as the iris-turbo-metal-eg2 and iris-base-metal-eg2 catalog rows.

Known limits

Rendered text is far below native Iris (22 % vs 100 % word accuracy). Spatial relations between two objects are the weakest category. The adapter is trained for English prompts up to 295 EG2 tokens.

Citation and credits

Iris-3B: Speridlabs, speridlabs/iris-3b (Apache-2.0). EmbeddingGemma 2: Google, google/embeddinggemma-2 (Apache-2.0). Wikimedia Commons photographers listed in iris_images_attribution.md. Training code, benchmarks and the engineering write-up: https://engineering.oakoliver.com/articles/sounds-and-videos-as-prompts-embeddinggemma-2-adapter-qwen-image-iris