LemonSeed 1.5B — Base

A from-scratch 1.5B-parameter hybrid Mixture-of-Experts language model, pre-trained end-to-end on a single AMD MI300X (ROCm) using an MLX training stack. This is a base model — pre-training only, no instruction/chat fine-tuning yet.

Author: Geramy L. Loveless · Trained Aug 2026 · Framework: MLX (ROCm backend)

▶ Primary engine: LSE — Lemon Seed Engine (C++ · HRX/ROCm) · 📖 Docs: lemonseed-docs

TL;DR


Params (nominal)	1.51 B
Params (effective)	~0.90 B (68/160 routed experts trained — see “Honest capacity note”)
Active params / token	~0.46 B (top-2 of 8 experts + shared + backbone)
Training	1,733,000 steps · ~28.4 B tokens · ~3 epochs
Final validation loss	1.78
Context length (trained)	2048
Precision	bfloat16
Tokenizer	Qwen 3.6 BPE · 248,320 vocab · digit-split
Hardware	1× AMD Instinct MI300X (gfx942, ROCm)

Architecture

LemonSeed is a hybrid — it interleaves linear-attention, softmax-attention, and sparse-MoE feed-forward, with a Mixture-of-Depths gate on top:

Sequence mixing (per layer):
- Gated DeltaNet (linear attention) on most layers — O(N) compute, O(1) recurrent state → cheap long context.
- Full softmax attention every 4th layer (global attention layers 3, 7, 11, 15, 19), sliding window 256, RoPE θ=500000.
Feed-forward: Mixture-of-Experts — 8 routed experts, top-2 routing, 1 always-on shared expert, expert intermediate 2176, DeepSeek aux-loss-free load-balancing bias, score band 0.15 (see note).
Mixture-of-Depths: each layer processes only ~top-15% of tokens through the expert path (mod_top_k=3, threshold 0.15).
Shape: 20 layers · hidden 1024 · 16 attn q-heads / 2 kv-heads (GQA) · 8 GDN qk-heads / 4 v-heads · RMSNorm.

Full config is in config.json.

Prefill & decode — the algorithm

LemonSeed's efficiency comes from the hybrid: most layers are linear-attention (Gated-DeltaNet) with an O(1) recurrent state, and only every 4th layer is softmax attention with a growing KV cache. So decode is cheap and mostly constant-memory.

Per-layer state carried between steps:

GDN layers → a fixed-size recurrent state S (a [v_heads, head_dim, head_dim] matrix). Does not grow with sequence length.
Attention layers (3, 7, 11, 15, 19) → a standard KV cache (grows with context; sliding-window 256).

Prefill — process a prompt of length `T` (one pass)

h = embed(tokens)                         # [T, D]
for each layer L:
    x = rmsnorm(h)
    if L is attention:
        q,k,v = proj(x)                   # GQA: 16 q-heads, 2 kv-heads
        y = causal_softmax_attn(q,k,v)    # RoPE, sliding window 256
        cache[L].kv = (k, v)              # save for decode
    else:  # Gated DeltaNet (linear attention)
        # chunked delta-rule scan over the T tokens:
        #   S_t = S_{t-1} (I - beta_t k_t k_tᵀ) + beta_t v_t k_tᵀ   (delta rule)
        #   y_t = q_t · S_t
        y, S = gdn_scan(x)                # O(T); keep final S
        cache[L].state = S               # save for decode
    h = h + out_proj(y)                  # residual

    x = rmsnorm(h)
    keep = mod_gate(x)                    # Mixture-of-Depths: ~top-15% of tokens
    r = moe(x[keep])                      # routed: top-2 of 8 experts (+ shared always-on)
    h = h + shared(x) ; h[keep] += r      # scatter routed output back
logits = tie_embed(rmsnorm(h))[-1]        # distribution for the FIRST new token

Decode — generate token `t+1` from the saved state (one token in, one out)

h = embed(new_token)                      # [1, D]
for each layer L:
    x = rmsnorm(h)
    if L is attention:
        q,k,v = proj(x)
        cache[L].kv.append(k, v)          # KV grows by 1 (window-capped at 256)
        y = attend(q, cache[L].kv)        # O(context) — but only ~5 layers
    else:  # GDN — O(1), constant memory
        S = cache[L].state
        S = S*(I - beta·kkᵀ) + beta·v·kᵀ  # in-place delta-rule update
        y = q · S                         # read
        cache[L].state = S
    h = h + out_proj(y)

    x = rmsnorm(h)
    if mod_gate(x):                       # is this token routed this layer?
        h = h + shared(x) + moe_top2(x)
    else:
        h = h + shared(x)
logits = tie_embed(rmsnorm(h))            # sample -> next token, repeat

Why it's fast: 15 of 20 layers are GDN → O(1) per decode step, no KV growth. Only the 5 attention layers pay O(context). And MoE runs just top-2 of 8 experts per token, so ~0.46 B active params/token despite 1.5 B nominal. Net: a small KV footprint and near-constant-time decode.

Training

From scratch (random init) — no distillation, no continued-pretrain from another model.
Objective: next-token cross-entropy + load-balance aux loss + router z-loss.
Optimizer: Fused AdamW (bf16 params + fp32 master), cosine LR (peak 3e-4 → ~3e-5), weight decay 0.1.
Batch: 4 × 2048 × grad-accum 2 = 16,384 tokens/step.
Corpus (v7, ~9.6 B unique tokens, ~3 epochs):

domain share

code 45.1%

math 15.2%

web 10.7%

logic 10.4%

docs 9.7%

conversational 7.6%

agent 1.3%
Curriculum: domain-ordered (per-block loss swings 2.5–3.7 are the curriculum, not instability).

domain	share
code	45.1%
math	15.2%
web	10.7%
logic	10.4%
docs	9.7%
conversational	7.6%
agent	1.3%

Honest capacity note ⚠️

By weight, only 68 of 160 routed experts actually trained (the rest sit near zero-init), so the effective size is ~0.90 B, not the full 1.51 B. The router dispatched tokens across all experts (balanced), but the tight 0.15 score band ran the model effectively top-1 for most of training, starving non-top experts of gradient — a classic winner-take-all collapse, worst in the early layers, full in the deep layers.

This is lost parameter budget, not lost quality — the model works well on what it trained, and MoE only runs the top-2 per token, so dead experts cost memory but no compute. It's documented transparently here rather than hidden. The next model uses a wider score band from step 0 and a per-layer expert schedule to fix it.

Using the untrained experts — expansion slots for post-training (RL / tools / agents)

Here's the useful part: those 92 untrained experts (~0.6 B of near-zero capacity) are empty expansion slots. You can deliberately fill them during post-training with new skills — tool-calling, agentic planning, chain-of-thought reasoning — without growing the model and with catastrophic-forgetting protection built in (freeze the trained base, train only the fresh experts). The inference cost stays the same (still top-2 per token).

Why this works in post-training when mid-pretrain revival did not: a converged base has no unmet demand, so any expert you re-seed is redundant and gets pruned within ~1k steps (we tried; it collapses). But a new task the model can't yet do manufactures demand — the task gradient has nowhere good to go in the existing experts, so the fresh experts find a real niche and stick. You're not reviving generic capacity; you're colonizing empty slots with a skill that needs them.

Which experts are dead (target map, from the final checkpoint)

tier	layers	dead experts (indices)	best for
fully dead (0/8 live)	0, 1, 9, 10, 13	all 8	hardest — no in-layer sibling to clone; use the shared expert as the seed
sparse (1–2 live)	2, 5, 6, 7, 8, 11	most of `0–7`	best targets — has a live sibling to clone and room; a re-seeded expert finds a niche
mid (3–6 live)	3, 4	`{0,1,2,4,6}`, `{1,2,4,5,7}`	good targets — plenty of dead slots + diversity
full (7–8 live)	12, 14–19	only L12→`{7}`, L14→`{0}`	leave alone (deep layers are already fully used)

Note the dead capacity sits in the early/mid layers; the deep layers (14–19) are full. So mid-level skills (structural patterns, composition, formatting like tool-call syntax) map cleanly onto the available slots; the deep layers will consume the new mid-layer features downstream.

How to target them (recipe)

Re-seed the dead experts to a usable scale — clone a live sibling in the same layer (or the shared expert for fully-dead layers) + ~10% noise, so they start with useful features (random init just gets pruned). Also clone the router row so they get competitive routing weight.
```
# in the lemonseed repo — targeted, incremental, or whole-model:
python scripts/revive_experts.py IN.safetensors OUT.safetensors \
    --clone --noise=0.1 --layers=2,5,6,7,8,11 --max-per-layer=2
```
Widen the score band so non-top experts actually receive combine-weight + gradient (the tight 0.15 band is what starved them):
```
export LEMONSEED_MOE_SCORE_BAND=0.5     # true top-2
```
Freeze the base, train only the fresh experts — this forces the new skill into the empty slots and prevents forgetting (the trained backbone + live experts can't drift). Unfreeze everything for a short final anneal if you want.
Drive it with task demand — SFT on tool-call / agentic / reasoning traces, then RL (GRPO/PPO). The strong, specific reward signal is what makes the re-seeded experts stick and specialize.
Monitor with scripts/expert_health.py each ~1k steps — a colonized expert's w_up std should climb (staying above 1e-4) instead of decaying back. If it decays, the layer had no demand for it; move to a different layer.

Start with the sparse layers (2, 5, 6, 7, 8, 11) — they have both a live sibling to clone from and open slots, so they take most reliably. The fully-dead layers (0, 1, 9, 10, 13) are the stretch goal (seed from the shared expert) and need the strongest task signal.

Tooling for all of this — revive_experts.py (re-seed/clone), expert_health.py (measure), and the LEMONSEED_MOE_SCORE_BAND knob — ships in the lemonseed-docs repo.

Capabilities & intended use

Base model for research / continued fine-tuning. It is not instruction-tuned — it completes text, it doesn't follow chat instructions out of the box.
Arithmetic: solid with chain-of-thought / scratchpad prompting (the Qwen3.6 digit-split tokenizer helps); one-shot answers are noisier.
Good starting point for: SFT, RLHF/GRPO, long-context extension, and MoE research.

How to run

LemonSeed uses a custom architecture (hybrid GDN + MoE + MoD), so it is not loadable via transformers. Files here: model.safetensors, config.json, tokenizer.json.

▶ Primary engine — LSE (Lemon Seed Engine)

The reference runtime is LSE — Lemon Seed Engine: a modular C++ training and inference engine built for this architecture, running on the HRX native ROCm runtime. Ops record into a lazy DAG, fuse, JIT to AMDGPU code objects via amd_comgr, and dispatch through the HRX ABI (not HIP). It's the performant, first-class way to load and serve LemonSeed.

git clone https://github.com/Geramy/LSE && cd LSE
cmake -S . -B build -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo && cmake --build build
# point it at model.safetensors + config.json + tokenizer.json from this repo

Reference path (Python / MLX)

For a simple/reference implementation, the MLX loader:

from lemonseed.model import load_model
from lemonseed.tokenizer import load_tokenizer
from lemonseed.generate import generate_text

model = load_model("model.safetensors")          # + config.json alongside
tok   = load_tokenizer("qwen3.6")                # or point at the bundled tokenizer.json
print(generate_text(model, tok,
    "Question: What is 8 × 6?\nLet's think step by step.\nAnswer:",
    max_new_tokens=64, greedy=True))

See example_generate.py for a template, and github.com/Geramy/lemonseed-docs for the full architecture + prefill/decode reference.

License & attribution

Model weights: released under a research / non-commercial license (license_name: lemonseed-research) — see the caveat below.
Tokenizer: the bundled tokenizer.json is the Qwen 3.6 tokenizer, © Alibaba Cloud / Qwen, under its own license.
Training data attribution: the corpus includes openly-licensed educational material, notably MIT OpenCourseWare (CC BY-NC-SA). Because that source is NonCommercial, this model is intended for research / non-commercial use. If you plan commercial use, review the data licensing first. Credit to MIT and the open-education community — solo foundation-model work runs on open resources.

Limitations

Small (~0.9 B effective) base model — expect hallucination, limited world knowledge, no safety tuning.
Not instruction-tuned; no chat template applied.
~half the routed experts are untrained (see capacity note) — a fine-tune (esp. one that colonizes the dead experts) can recover capacity.

Built entirely on MLX (ROCm backend) — hat tip to the MLX maintainers. Trained through a machine migration and a full-disk crash, finished clean at val loss 1.78.

Downloads last month: 810

Safetensors

Model size

2B params

Tensor type

BF16

F32

MLX

Hardware compatibility

Quantized