ntc-ai/model-glue-sd15-sana-text

SD1.5 CLIP → SANA text bridges

Three trained bridges translate 77 × 768 SD1.5 CLIP text states into 77 × 2304 SANA conditioning states and a mask. The bridge runs as an ordinary deterministic network forward, followed by normal frozen SANA sampling. There is no fitting, retrieval, particle sampling, or candidate selection during inference.

These are bridge weights for the model-glue implementation, not a complete Diffusers pipeline. The donor text encoder, SANA transformer, and DC-AE decoder are loaded separately at pinned revisions. SANA's native empty conditioning is stored in each checkpoint, so deployment needs no Gemma encoder.

Choose a checkpoint

File	Role	Selected training update
model.safetensors	Default: validation-selected linear continuation with fixed mask calibration	1250
alternatives/linear-uncalibrated.safetensors	Same trained head before calibration; retained for its mask tradeoff	1250
alternatives/expanded-source.safetensors	Expanded output head with full-source access and fixed mask calibration	1000

The expanded-source architecture repairs a verified blind space in the old source projection, but did not win the combined validation ranking. The default therefore retains the linear head. All three checkpoints include the complete bridge, cloud, calibration buffers, architecture metadata, and pinned source/recipient identities. They load directly without the parent checkpoint or training caches.

Architecture and winning objective

The default contains 7,182,917 parameters. SD1.5 CLIP provides 77 × 768 contextual states. LayerNorm and a linear 768 → 384 layer produce source features. A mean-pooled source passes through a 768 → 64 → 4 SiLU router; softmax attention over the saved 256 × 4 particle cloud gives a deterministic four-dimensional mixture. Its projection offsets 77 learned queries, which also receive the projected source states.

Four width-384 blocks each apply six-head cross-attention to the source followed by a 384 → 1152 → 384 GELU feed-forward network, with residual connections and LayerNorm. A final linear 384 → 2304 head produces normalized conditioning, rescaled with saved target statistics. A separate linear 384 → 1 head predicts mask logits. The first BOS slot is always kept.

The donor diffusion U-Net is outside this path. This checkpoint transfers CLIP text states; it does not establish transfer of SD's generated style or composition.

The winning continuation updates only the 887,040 output-projection parameters that receive nonzero gradients. The parent stem, router, cloud, and mask values remain fixed during training. Afterward, validation calibration subtracts 2.0417721271514893 from the old mask-head bias. The release stores that fixed adjustment and thresholds its logits at zero.

The active training objective is:

$$L=0.1L_{embed}+\frac{\mathrm{MSE}(s_{+},t_{+})+\mathrm{MSE}(s_0,t_0)+0.05\mathrm{MSE}(g_s,g_t)}{\max ({10}^{-5},\mathrm{mean}((t_{+}-t_0{)}^2))},L=0.1L\_\{embed\}+\; \backslash frac\{\backslash operatorname\{MSE\}(s\_+,t\_+)+\backslash operatorname\{MSE\}(s\_0,t\_0)\; +0.05\backslash operatorname\{MSE\}(g\_s,g\_t)\}\; \{\backslash max(10^\{-5\},\backslash operatorname\{mean\}((t\_+-t\_0)^2))\},$$

where $L_{embed}$ is embedding squared error on teacher-valid tokens, divided by the saved per-channel variance. $s_+,s_0$ and $t_+,t_0$ are frozen-SANA predictions with student and native positive/empty conditioning at the same latent and timestep. The guided predictions are $g_s=s_0+4.5(s_+-s_0)$ and $g_t=t_0+4.5(t_+-t_0)$. The denominator is detached native prompt-effect power. Student calls backpropagate into the bridge; teacher calls and SANA parameters remain frozen. Both empty branches use stored native conditioning.

The run uses Adam at initial LR 1e-5, betas (0.0, 0.999), delayed cosine decay, gradient clipping at 1.0, and EMA 0.98. Each update uses 32 embedding rows and four functional examples; it chooses cached native states with probability 0.5 and fresh initial noise otherwise. Selection uses rendered validation LPIPS.

GAN, b_cap, critic-feature, particle, mask-BCE, soft-mask, and full-rollout losses are inactive in this winning continuation. Its parent has ParticleGAN provenance. LPIPS and CLIP are evaluation signals here, not differentiable image losses. The bridge runs once per prompt; ordinary SANA sampling reuses its conditioning.

The expanded-source alternative adds $W_2\operatorname{SiLU}(W_1h+W_sx+b_1)+b_2$ alongside the original output head. Its hidden branch is 2304-wide and sees the full 768-channel source directly. It has 15,150,149 parameters, with 7,967,232 new branch parameters trained under the same active loss. It removes the old source-projection blind space and affine-output restriction but did not win combined validation.

The complete formulation includes equations, an architecture diagram, parameter counts, the mask calibration sign, training controls, and deployment details.

Run

Use the implementation revision that was tested for this release:

git clone https://github.com/255BITS/model-glue.git
cd model-glue
git checkout f67054d329fe5246510a094dd791d485526e3dd6
python3 -m venv .venv
.venv/bin/python -m pip install -e .
.venv/bin/python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="ntc-ai/model-glue-sd15-sana-text",
    local_dir="artifacts/huggingface/sd15-sana-text",
    allow_patterns=["model.safetensors", "manifest.json", "SHA256SUMS"],
)
PY
CUDA_VISIBLE_DEVICES=1 .venv/bin/python -m model_glue.sana_infer \
  --checkpoint artifacts/huggingface/sd15-sana-text/model.safetensors \
  --prompt 'A park bench on a beach of white sand looking out to the ocean.' \
  --seed 87654 --steps 20 --height 512 --width 512 --guidance 4.5 \
  --out artifacts/sana-text-example

Choose an available CUDA device for your machine. The output directory must be new. The command saves image.png and render.json, including the checkpoint hash and deterministic-forward verification. To use an alternative, download its path in allow_patterns and pass that checkpoint to the same CLI. For a reproducible download, set revision to the desired Hugging Face commit hash.

The tested deployment uses CLIP FP16 with a fixed batch padded to 16, a FP32 bridge, SANA BF16, and the pinned 512-pixel sampler recipe. Input is limited to 77 tokens. The source skip uses matching sequence lengths; it does not establish word-level alignment between tokenizers. config.json describes the default bridge, while the loader treats checkpoint metadata as authoritative.

Measured quality

Selection used 58 validation prompts, averaging seeds within each prompt. The final test was held out until selection was frozen: 16 prompts × 3 seeds, 20 sampling steps, 512 × 512, guidance 4.5. Every declared output was scored.

Method	Native-image LPIPS ↓	Pixel MAE ↓	CLIP cosine ↑
Original parent, comparison baseline	0.38347	0.14984	0.30620
Uncalibrated linear continuation	0.37711	0.14652	0.30744
Validation-selected default	0.37802	0.14609	0.30757
Expanded-source alternative	0.37778	0.14716	0.30815
Native SANA reference	0.00000	0.00000	0.30820

The default improves LPIPS by 1.42% and pixel MAE by 2.50% against the original parent. LPIPS improves on 11/16 prompt averages, but the paired difference of −0.00545 has a 95% prompt-bootstrap interval of [−0.01769, +0.00817]. The interval includes no improvement. These are modest observed fidelity gains, not established gains in general prompt accuracy or aesthetics. The test ranking did not replace the frozen validation choice.

Mask calibration reduces validation errors from 6 to 4, but increases missed test positions from 1 to 3; exact test masks decline from 15/16 to 14/16 prompts. The uncalibrated control has slightly better test LPIPS, while the calibrated default has better pixel MAE and CLIP. Composition and attribute errors remain.

In this illustrative beach-bench case, the default's orientation is closer to native, while the expanded-source alternative changes the bench color. Columns are native SANA, original parent, selected default, and expanded-source alternative.

The first four test prompts include failures: the small car between trucks is not recovered, and the swan remains goose-like. These panels show seed 87654; all three seeds contribute to the table. Both displayed panels were visually reviewed; neither contains people.

Training and provenance

Seven continuations completed 8,500 updates on GPU 1, starting from the same trained parent. The original stem, cloud, and masks stayed fixed during training; linear runs trained the output projection, and source-head runs trained the added branch. Mask calibration later folded one validation-fitted scalar into the mask bias. All new cloud controls are fixed; this study establishes no particle-movement advantage. SANA and its decoder were frozen throughout.

Training used 8,192 paired captions, paired embedding supervision, and frozen-SANA functional distillation. Full-trajectory and repaired b_cap arms were also tested but did not win selection. These released winners were trained with the supervised distillation recipe. There is no new Anima result in this release.

• Complete study and reproduction details
• Release manifest and per-file hashes
• Checksums
• Training configurations
• Selection and measured results

The implementation passed 82 tests, including checkpoint migration, resume, source reachability, mask geometry, and differentiable rollout checks. The Gaussian reference audit has zero cap-parameter-gradient discrepancy. Exact upstream revisions and checkpoint SHA256 values are in the manifest. The release contains learned bridge weights under the repository's MIT license; the separate upstream models retain their own licenses. No donor or recipient model weights, training datasets, optimizer states, or credentials are included.