Photon-P1

Photon-P1 is a 1.003M-parameter language model trained from scratch — no base model, no pretrained weights, no gated dependencies. It is a follow-up to Photon 5 (5.03M parameters, seven per-domain LoRA adapters) exploring a more aggressive form of parameter specialization at a much smaller scale.

The core idea: instead of seven fine-grained LoRA adapters, use two fully independent down-projection matrices per layer — one for chat, one for everything else. No router, no soft mixing. A caller specifies the group and only the corresponding matrix is used.

This is an intentionally extreme experiment. Photon-P1 is overfit by design, evaluated only on training data, and intended as a research probe rather than a usable assistant.

---

Headline result

On training data (16 epochs, overfit regime), forcing the wrong down-projection matrix produces:

Data group	Correct down_proj	Wrong down_proj	Ratio
chat	PPL 30.19	PPL 412.09	x 13.6
rest	PPL 2.56	PPL 103.67	x 40.5

Per-layer cosine similarity between the two down projections ranges from 0.19 to 0.27 — far from 1.0 (identical) and far from 0 (orthogonal). This is the signature of genuine specialization: shared initialization base, functionally disjoint roles.

Per-layer Frobenius norms of the down projections grow with depth, and the "rest" matrix consistently exceeds the "chat" matrix by 7% (layer 0) to 26% (layer 5) — consistent with the "rest" group being dominated by highly templated content (JSON, Python, GSM8K) requiring larger weight magnitudes.

---

Architecture

Component	Value
Parameters	1,002,720 (~1.003M)
Hidden size	96
Layers	6
Attention heads	4 (query), 2 (key/value)
Head dim	24
Grouped Query Attention	yes (GQA, ratio 2:1)
Intermediate size (FFN)	192 (SwiGLU)
Normalization	RMSNorm, pre-norm
Positional encoding	RoPE (theta = 10000 base)
Max sequence length	1024
Embeddings	Tied (input = output)
Vocab size	4096
Groups	2 (chat, rest)
Precision	fp32 weights, bf16 during training

The architecture is a faithful implementation of the Llama family, written entirely in plain PyTorch.

Dual-group specialization

Every FFN layer contains:

• A shared SwiGLU: gate projection, up projection
• Two independent down projections of shape (192, 96), one per group
• No router — the group is passed explicitly at each forward pass

During training, a batch from "chat" only ever updates the chat down projection. A batch from "rest" only ever updates the rest down projection. The gate, up, attention, and embeddings are shared and updated by both.

This is a hard-routed dual-expert FFN. Each group's down projection receives gradient from exactly one source, which is the cleanest possible specialization signal.

Groups:

ID	Group	Content
0	chat	Conversational data (UltraChat)
1	rest	Code, function calling, terminal, finance, law, math

---

Tokenizer

Byte-level BPE tokenizer trained from scratch on the training corpus.

• Vocab size: 4096
• Training time: 27.1 s
• Round-trip decode verified

The compact vocabulary (4096) is deliberate: at 1M parameters, a larger vocab would consume a disproportionate share of the budget. 4096 tokens keeps the (tied) embedding at ~393k parameters and leaves ~610k for the transformer stack.

---

Training

Setting	Value
Training tokens	~7.5M unique, ~120M seen (16 epochs)
Batch size	32 sequences x 512 tokens = 16,384 tok/step
Steps	4000
Optimizer	AdamW (fused), beta = (0.9, 0.95)
Learning rate	8e-4, cosine schedule, 100 warmup steps
Weight decay	0.005
Precision	bf16 autocast
Hardware	1 x NVIDIA T4 (Kaggle)
Wall-clock time	9.46 minutes

Final training loss: 2.94 (cross-entropy, causal, bf16).

Loss trajectory:

Step	Loss
50	8.01
500	3.74
1000	3.45
2000	3.17
3000	3.07
4000	2.94

The loss curve flattens and begins oscillating between steps 3000 and 4000 — the model has memorized its training corpus and is no longer generalizing. This is the intended behavior for this experiment.

Training data

Group	Sources	Samples
chat	HuggingFaceH4/ultrachat_200k	8,000
rest	CodeAlpaca, python_instructions, tool-calling-english-100k, glaiveai, legalfinance, gsm8k, ARC-Challenge	~8,000

Datasets reused across slices are shuffled with different seeds.

---

Evaluation

Warning: all reported perplexities are computed on training data. No held-out split was used. The training regime is intentionally overfit (16 epochs on ~8k examples per group). The numbers below measure how well the model fits data it has already seen — they are not a measure of generalization.

Perplexity is computed with max length 256, causal shift, padding masked out.

Cross-group perplexity (headline result)

Data group	down_proj used	PPL	NLL	Gap vs correct
chat	chat (correct)	30.19	3.407	—
chat	rest (wrong)	412.09	6.021	+1265.2%
rest	rest (correct)	2.56	0.940	—
rest	chat (wrong)	103.67	4.641	+3951.4%

The wrong projection matrix produces perplexities that are 13.6x (chat) and 40.5x (rest) worse than the correct one. The specialization is not marginal — the two matrices encode fundamentally different functions.

Per-layer divergence

Layer	chat norm	rest norm	cos(chat, rest)
0	4.940	5.285	+0.2719
1	5.087	5.601	+0.2565
2	5.145	5.933	+0.2104
3	5.640	6.147	+0.1890
4	5.867	6.835	+0.1879
5	6.075	7.636	+0.2166

Cosine similarity between the two matrices stays in the 0.19-0.27 range at every layer. They share about 20% of their direction (from shared initialization and shared upstream gradients) but diverge sharply beyond that. The divergence increases slightly with depth (0.27 at layer 0, down to 0.19 at layer 3-4, mild recovery at layer 5) — consistent with the deeper layers performing more group-specific transformation.

The norm of the "rest" matrix grows faster with depth, ending at 26% larger than the "chat" matrix at layer 5. This is coherent with the "rest" group containing highly templated content (JSON, Python syntax, GSM8K reasoning markers) that requires larger weight magnitudes to represent.

---

Interpretation

The result is striking given the parameter count. Each down projection holds 18,432 parameters (192 x 96), or 11% of the model's total parameter budget per group per layer. With only 11% dedicated to a group, the model still produces a 13-40x perplexity degradation when forced to use the wrong projection.

Three conclusions:

1. Hard-routed specialization works exceptionally well at 1M parameters. The theoretical concern — that 1M is too small for meaningful specialization — is empirically false.

2. The bottleneck is not the shared trunk. Attention, embeddings, gate, and up projections are shared and perform well on both groups. The specialization lives entirely in the down projections, which are the "final projection to vocabulary space."

3. Cosine similarity in the 0.2 range is the sweet spot. The two matrices need to share enough of the base to remain compatible with the shared trunk, but diverge enough to encode group-specific output distributions. Photon-P1 sits exactly in that regime.

---

Comparison with Photon 5

Property	Photon 5	Photon-P1
Parameters	5.03M	1.003M
Domains	7 (chat, code, agent, ...)	2 (chat, rest)
Specialization	LoRA rank-8 on down_proj	Full down_proj per group
Routing	Hard (domain_ids)	Hard (group_ids)
Training epochs	~2.4	~16
Training loss	3.33	2.94
Cross-group test	Not measured	13.6x / 40.5x gap

Photon 5 asked whether fine-grained specialization is possible. Photon-P1 answers a sharper question: what happens when the specialization is coarse but total?

---

Limitations

Read this before using the model.

Photon-P1 is a research artifact, not a usable assistant. At 1M parameters, the model is four orders of magnitude smaller than production LLMs.

1. Extreme overfitting. 16 epochs on 8k examples per group. The model has memorized its corpus. Any novel input will produce unpredictable output.

2. No held-out evaluation. All perplexities reported are on training data. They measure fitting, not generalization.

3. Hallucination is the default behavior. Every generated token is a statistical guess.

4. No reasoning. The model does not perform arithmetic, does not follow multi-step logic, and does not understand instructions.

5. Group routing is manual. The caller must specify whether the input is "chat" or "rest". The model has no built-in classifier.

6. Language coverage. Predominantly English, some French.

7. No safety alignment. No RLHF, no DPO, no red-teaming.

---

Intended use

Photon-P1 is intended for:

• Research on hard-routed per-group specialization at very small scale.
• Ablation baseline against Photon 5 (7 LoRA soft-routed) and other specialization strategies.
• Educational demonstrations of how specialization emerges in tiny models.
• Study of cosine similarity as a specialization metric — the 0.19-0.27 range observed here is a useful empirical data point.

Photon-P1 is not intended for any production use, factual question answering, or code generation.

---

Files

File	Size	Description
model.safetensors	3.8 MB	Model weights (fp32, 1.003M params)
config.json	0.6 KB	Architecture and training metadata
tokenizer.json	280 KB	Byte-level BPE vocabulary (4096)
tokenizer_config.json	0.2 KB	Tokenizer wrapper configuration
README.md	this file	Model card

---

Design notes

Why 1M parameters?

Photon 5 (5M) was too large to truly test the specialization-vs-capacity tradeoff — the model had enough room to encode multiple domains in the shared trunk even without specialization. At 1M, the capacity constraint is severe enough that specialization becomes structurally necessary, not just beneficial.

Why a full down_proj per group, not LoRA?

LoRA gives low-rank specialization (rank 8 out of 96 in Photon 5, so ~8% of the matrix capacity). Photon-P1 gives full-rank specialization by duplicating the down projection entirely. This is the maximal specialization the architecture can support without doubling the parameter count.

Why only two groups?

Seven groups at 1M parameters would give each group only 3% of the parameter budget. Two groups give each 11%, which is enough to encode a full output distribution. The choice of "chat" vs "rest" is arbitrary but convenient: chat is open-ended, rest is templated, so the specialization is easy to measure.

Why so many epochs?

16 epochs forces memorization. Without overfitting, the cross-group perplexity gap would be diluted by generalization error on both sides. By overfitting, we isolate the specialization signal: the model has memorized each group's distribution separately, and we can measure how much the wrong matrix breaks that memorization.

---

Citation

If you use Photon-P1 in academic work, please cite it as:

muon AI. Photon-P1: Hard-Routed Dual-Expert Specialization at 1M Parameters. 2025. Available at https://huggingface.co/muonai/photon-p1

---

Acknowledgments

Dataset sources remain under their respective licenses. Architecture draws on the Llama family (Touvron et al., 2023) and the MoE literature (Shazeer et al., 2017; Fedus et al., 2021).