diffusiongemma-26B-A4B-it-ALWAYS-THINK

Weights are an unmodified copy of google/diffusiongemma-26B-A4B-it. The only intended difference is chat_template.jinja: thinking should be on by default, without the caller having to ask for it in the system prompt.

Thinking

The stock template gates reasoning behind an enable_thinking kwarg that injects a <|think|> marker into the system turn. Probing the base model, that marker is the switch — the model emits <|channel>thought on its own either way, but without the marker it closes the channel immediately and answers directly.

Pre-filling <|channel>thought into the generation prompt instead does not work: the model emits its own opener on top of the prefilled one, duplicating the token.

Caveat: the marker is necessary but not sufficient. On 14 held-out prompts the base model still produced an empty thought block every time, so "always think" is not yet reliably achieved by the template alone.

Finetuning attempt (abandoned)

Tried LoRA SFT on sriq-ai/sriq-sft-v1.5 to move reasoning into compressed Chinese. It did not work and no adapter is published.

DiffusionGemmaForBlockDiffusion.forward takes no labels and returns no loss, so the block-diffusion objective has to be written by hand: corrupt the 256-token canvas with uniform-random tokens at rate t, cross-entropy on the content tokens, padding tail ignored. r=64, alpha=128 on attention + dense MLP across the text backbone (the fused 3D MoE experts cannot be targeted by PEFT).

Measured on 14 held-out prompts, adapter loaded unmerged:

	base	30 steps	100 steps
mean CJK of generation	7.8%	5.1%	3.3%
non-empty thought blocks	0/14	6/14	4/14

The training objective improved cleanly every run (hardest corruption bucket 3.89 → 2.82), but the target behaviour did not transfer — Chinese reasoning moved down. The CJK figures rest on the 2 of 14 rows that produce any Chinese at all, so treat them as "no transfer" rather than as a measured decline.

Best guess at why: the architecture refines a 256-token canvas in parallel, which suits global-constraint tasks — Unsloth's Sudoku notebook reaches 89.5%, but needed 4000 steps on targets that fit a single canvas. Thousand-token sequential chain-of-thought is a poor fit, and LoRA cannot reach the MoE experts where most of the capacity lives.

Notes

• Do not merge_and_unload() a LoRA adapter here. Gemma4ClippableLinear wraps the nn.Linear; merging corrupts it. Load unmerged and generate through the PeftModel.
• Weight loading needs a transformers build that ships the DiffusionGemma classes.