kennethgrace/smollm2-1.7b-collapsed-gdn

SmolLM2-1.7B collapsed into a Gated DeltaNet

This is SmolLM2-1.7B with its tensors converted and stitched into a Gated DeltaNet configuration: 24 layers, d_model=2048, 32 states of width d_k=64 / d_v=256, one mixer per layer, no attention and no separate feed-forward anywhere in the model.

It is completely untrained in this form. The weights are the donor's, rearranged. Nothing here has taken a gradient step since the surgery.

A 135M sibling, cut the same way from SmolLM2-135M, is at kennethgrace/smollm2-135m-collapsed-gdn.

What was done

Each donor decoder layer is two sub-layers — attention, then a gated feed-forward. Both fold into a single Gated DeltaNet. Five of the donor's seven matrices transfer with no slicing and no padding:

donor tensor	shape	→	GDN	role
self_attn.q_proj	(2048, 2048)	→	q_proj	the memory read
self_attn.k_proj	(2048, 2048)	→	k_proj	the address
mlp.up_proj	(8192, 2048)	→	v_proj	the payload
mlp.gate_proj	(8192, 2048)	→	g_proj	the output gate
mlp.down_proj	(2048, 8192)	→	o_proj	the residual write

A Gated DeltaNet's output path already is an up/gate/down arrangement — the same shape as a SwiGLU — so the feed-forward has somewhere to land, while the query and key projections are already shaped for a key-addressed memory.

All 120 transplants are bit-exact. Embeddings, the tie, and every RMSNorm come across untouched. Attention's v_proj and o_proj are homeless — the feed-forward claimed the value and output side first — and are left out. The short convolutions are initialised to a causal identity (all taps zero but the last), so a freshly collapsed layer computes its transplanted projections untouched; the default convolution init would scramble them before a single gradient step.

The head partition survives here

n_heads must divide the query rows and the feed-forward width. This donor is plain multi-head attention — 32 query heads over 32 key/value heads, so the query/key ratio is 1 — which leaves every power of two up to 2048 admissible.

32 was chosen because d_k=64 is exactly the donor's head_dim. The head partition therefore transfers intact: 32 attention heads of width 64 become 32 memory states of width 64, same numbers read against the same grouping. Each state gets its own key group, so the layout is diagonal.

This is the one thing that does not survive in the 135M sibling, where nine query heads of width 64 had to become three states of width 192 — nine does not divide a 1536-wide feed-forward. Smaller head counts here would buy wider keys at the cost of a larger recurrent state and a re-cut partition; larger ones shrink both.

Contents

• model-0000{1,2}-of-00002.safetensors — 1,514,333,696 parameters, float32
• model.safetensors.index.json — the shard map
• collapse.json — the manifest the loader reads (donor revision, layer count, plan)
• modeling_collapsed.py — standalone loader, depends only on torch + lumen
• tokenizer files — the donor's, unmodified

Use

pip install torch safetensors
pip install git+https://github.com/latticedynamics/lumen.git

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("kennethgrace/smollm2-1.7b-collapsed-gdn")
sys.path.insert(0, path)

from modeling_collapsed import load_collapsed

model, manifest = load_collapsed(path, device="cuda")
logits = model(ids)          # (B, T) token ids -> (B, T, vocab)

This is not a transformers architecture and will not load with AutoModelForCausalLM. Generation runs in constant memory through model.step(ids, states) rather than re-reading a growing prefix.

Note that float32 weights alone are ~6 GB, so a full fine-tune with Adam wants considerably more device memory than that.

Provenance and license

Donor: HuggingFaceTB/SmolLM2-1.7B at revision effd688a12921b4cc83e3312b6feb579f70f9c71, Apache-2.0. This derivative is released under the same license. The Gated DeltaNet implementation is lumen, MIT.