muonai/photon-5

Photon 5

Photon 5 is a 5.03M-parameter language model trained from scratch — no base model, no pretrained weights. Every component (architecture, tokenizer, training loop) was built from the ground up.

The design goal is not to compete with larger models, but to explore a research question: can a very small language model benefit from explicit per-domain parameter specialization?

Instead of training one generic model on a mixed corpus, Photon 5 uses seven domain-specific LoRA adapters (rank 8) applied to the feed-forward down-projection of every transformer block. During training, the domain of each batch is passed explicitly (via a domain_ids argument), and the model learns seven independent low-rank specializations of the shared FFN backbone.

This model card is intentionally honest about what 5M parameters can and cannot do. See the Limitations section.

---

Architecture

Component	Value
Parameters	5,028,928 (~5.03M)
Hidden size	192
Layers	7
Attention heads	6 (query), 2 (key/value)
Head dim	32
Grouped Query Attention	yes (GQA, ratio 3:1)
Intermediate size (FFN)	608 (SwiGLU)
Activation	SiLU (SwiGLU gate)
Normalization	RMSNorm, pre-norm
Positional encoding	RoPE (theta = 10000 base)
Max sequence length	1024
Embeddings	Tied (input = output)
Vocab size	8192
Domains	7
LoRA rank	8
Precision	fp32 weights, bf16 during training

The architecture is a faithful implementation of the Llama family (RMSNorm, RoPE, GQA, SwiGLU, tied embeddings, depth-scaled init), but written entirely in plain PyTorch. There is no dependency on the Hugging Face Llama classes or any pretrained checkpoint.

Domain specialization

Every FFN layer contains:

• A shared SwiGLU FFN: gate projection, up projection, down projection
• Seven LoRA pairs of rank 8, one per domain, applied to the down projection
• No router — the domain is passed explicitly as domain_ids at each forward pass

This is a hard-routed mixture of low-rank experts. During inference, a caller specifies the domain and only that domain's LoRA contributes to the output.

Domains (index order):

ID	Domain	Purpose
0	chat	General conversation, instruction following
1	code	Python, algorithms, syntax
2	agent	Tool calling, function calling
3	terminal	Shell, SWE-agent traces, CLI
4	finance	Financial reasoning, quantitative text
5	law	Legal text, statutory language
6	benchmark	Math word problems, multiple-choice, QA

---

Tokenizer

A byte-level BPE tokenizer trained from scratch on the training corpus (no Llama or GPT tokenizer dependency).

• Vocab size: 8192
• Special tokens: <unk>, <bos>, <eos>, <pad>
• Training time: 11.7 s on the full 33k-example corpus
• Round-trip decode verified: encode then decode is identity on all sampled inputs

The compact vocabulary (8192 vs 32k for Llama) is a deliberate design choice. It frees ~1.6M parameters that would otherwise sit idle in embeddings and redirects them into the transformer stack itself — a significant trade at this scale.

---

Training

Setting	Value
Training tokens	~40M (with padding), ~17M unique
Epochs (equivalent)	~2.4
Batch size	32 sequences x 512 tokens = 16,384 tok/step
Steps	2000
Optimizer	AdamW (fused), beta = (0.9, 0.95)
Learning rate	6e-4, cosine schedule, 100 warmup steps
Weight decay	0.01
Max grad norm	1.0
Precision	bf16 autocast
Hardware	2 x NVIDIA Tesla T4
Wall-clock time	~14 minutes

Training data

The corpus is a curated mixture of nine public datasets covering the seven domains. Datasets reused across multiple domains (e.g. legal-finance mix, function-calling corpora) are shuffled with different seeds so each domain sees a distinct slice.

Domain	Source datasets	Samples
chat	HuggingFaceH4/ultrachat_200k	8,000
code	sahil2801/CodeAlpaca-20k, iamtarun/python_code_instructions_18k	7,000
agent	Salesforce/xlam-function-calling-60k, glaiveai/glaive-function-...	7,000
terminal	glaiveai/glaive-function-calling-v2, DCAgent2/financeagent_...	3,148
finance	IlyasFardaouixx/legalfinance-500k-mixed	3,000
law	IlyasFardaouixx/legalfinance-500k-mixed	3,000
benchmark	openai/gsm8k, allenai/ai2_arc, TIGER-Lab/MMLU-Pro	7,000
Total		38,148

Training loss

Final training loss: 3.325 (cross-entropy, causal, bf16, 2000 steps).

Loss trajectory (excerpt):

Step	Loss
25	8.73
50	7.98
100	~6.9
500	~5.2
1000	~4.4
2000	3.33

---

Evaluation

Only perplexity was measured. No downstream benchmark is reported — at this parameter scale, binary multiple-choice accuracy sits at chance level and does not carry information beyond what PPL already provides.

Perplexity is computed on a held-out tail of the training corpus, with max length 256, causal shift, and padding masked out of the cross-entropy.

Held-out perplexity

Metric	Value
Perplexity	17.93
Cross-entropy (NLL)	2.887
Uniform-random baseline (8192 vocab)	8192
Improvement over random	x 457

Reference points

• Uniform random over the 8192-token vocabulary: PPL = 8192
• GPT-2 small (124M parameters, 24x larger) on WebText: PPL ~ 20-30
• Photon 5 operates at PPL ~ 18 with 24x fewer parameters

Important caveat: the training corpus is dominated by highly templated content (JSON tool calls, Q/A pairs, Python code, GSM8K chain-of-thought markers). Perplexity on such data is structurally lower than on natural web text and is not directly comparable to PPL figures reported on WebText, WikiText-103, or the Pile.

---

Limitations

Read this section before using the model.

Photon 5 is a research artifact, not a usable assistant. At 5M parameters, the model is three orders of magnitude smaller than the smallest general-purpose LLMs in production use. What this means in practice:

1. Hallucination is the default behavior. Every generated token is a statistical guess. Any factual claim the model produces is unreliable.

2. No reasoning. The model does not perform arithmetic, does not follow multi-step logic, and does not understand instructions. It reproduces surface patterns of reasoning-shaped text (chain-of-thought tags from GSM8K, answer markers, etc.) without the underlying computation.

3. Memorization of local patterns. The model frequently emits verbatim fragments from the training data — hex identifiers, URL slugs, dataset boilerplate. This is expected at this scale and with ~2.4 epochs over a small corpus. It is not evidence of understanding.

4. No native instruction following. Prompts must be formatted in the same shape as the training data (role prefixes, Q/A templates). The model will not generalize to novel prompt formats.

5. Language coverage. Predominantly English, with some French from the corpus. Other languages will produce degraded output.

6. Tokenizer edge cases. The from-scratch BPE tokenizer handles code well but splits rare proper nouns and long identifiers into subword pieces, which compounds hallucination on those tokens.

7. No safety alignment. There is no RLHF, no DPO, no red-teaming. The model will produce whatever pattern most resembles its training distribution, including problematic content if the pattern matches.

---

Intended use

Photon 5 is intended for:

• Research on per-domain parameter specialization at very small scale. Does a rank-8 LoRA per domain meaningfully diverge from a shared backbone at 5M parameters? This model is a data point on that question.
• Education and pedagogy. A complete, readable implementation of a modern LLM (RoPE, GQA, RMSNorm, SwiGLU, tied embeddings, LoRA) that runs end-to-end in 15 minutes on a free GPU.
• Tokenizer and architecture ablations. The from-scratch BPE tokenizer and depth-scaled initialization are directly reusable.
• Speculative decoding drafts if paired with a larger model using the same tokenizer family (note: tokenizer is custom, so this requires retraining or vocabulary alignment).

Photon 5 is not intended for:

• Production use of any kind
• Factual question answering
• Code generation in any serious context
• Any application where output reliability matters

---

Files

File	Size	Description
model.safetensors	19.2 MB	Model weights (fp32, 5.03M params)
config.json	0.6 KB	Architecture and training metadata
tokenizer.json	540 KB	Byte-level BPE vocabulary
tokenizer_config.json	0.2 KB	Tokenizer wrapper configuration
banner.png	—	Repository banner
README.md	this file	Model card

Weights are stored as float32. The bf16 flag during training only affected the autocast context, not the storage precision — reloading the model gives the exact trained weights without loss.

---

How to load

Photon 5 uses a custom architecture. To reload the weights, the TinyLlamaFromScratch class defined in the training notebook is required. The class implements the full forward pass (RMSNorm, RoPE, GQA, SwiGLU, per-domain LoRA) and exposes a forward method taking input token IDs, a domain ID tensor, an optional attention mask, and optional labels.

Loading requires four steps:

1. Load the tokenizer from tokenizer.json (byte-level BPE, vocab 8192).
2. Load config.json and instantiate a TinyLlamaConfig from its fields.
3. Instantiate TinyLlamaFromScratch with that config and load the safetensors state dict with strict=False (the tied lm_head weight is intentionally omitted from the file).
4. Call the model with an explicit domain ID (0 through 6, see domain table above) alongside the input token IDs.

A complete reference implementation of the class and the loading procedure is provided in the accompanying training notebook. It is approximately 200 lines of plain PyTorch and has no dependency on the Hugging Face Llama implementation.

---

Design notes

Why a custom tokenizer?

Using Llama's 32k vocabulary would have consumed 6.1M parameters in the embedding matrix alone — more than the entire model. A 8192-token BPE tokenizer, trained on the actual corpus, keeps the embedding at 1.57M parameters (tied, so counted once) and leaves 3.45M for the transformer stack. At 5M total parameters, that trade is decisive.

Why per-domain LoRA instead of a mixture-of-experts router?

A learned router requires training signal for both the routing decision and the expert weights. At 5M parameters and 40M tokens, there is not enough signal to train a router reliably. Passing the domain explicitly (hard routing) guarantees that each LoRA receives gradient only from its own domain, which is the cleanest possible specialization signal at this scale.

Why 7 layers rather than wider or deeper?

Depth helps compositional generalization more than width at small scale. Seven layers with hidden 192 keeps the parameter budget in a range where RoPE, GQA and per-layer LoRA can all coexist without compromising any one of them. Deeper (12+) would force narrower hidden, hurting token embeddings; wider (256+) would force fewer layers, hurting composition.

---

Citation

If you use Photon 5 in academic work, please cite it as:

muon AI. Photon 5: A 5M-Parameter Language Model with Per-Domain LoRA Specialization. 2026. Available at https://huggingface.co/muonai/photon-5

---

Acknowledgments

Dataset sources are listed above and remain under their respective licenses. The architecture draws on the Llama family (Touvron et al., 2023) and the LoRA method (Hu et al., 2021).