muonai/photon-x-preview

Photon-X (Preview)

Photon-X is a 10.1M-parameter language model trained from scratch, with seven hard-routed domain-specific feed-forward down-projections. It is the third model in the Photon family, following Photon 5 (5M params, seven LoRA adapters) and Photon-P1 (1M params, two full down-projections).

The design goal: push per-domain specialization to a scale where each domain gets roughly 1.4M effective parameters, and observe what happens empirically. The answer turns out to be more interesting than a single number — Photon-X exhibits three distinct specialization regimes within the same model.

This model card is intentionally honest about what the model is, what it is not, and what its evaluation numbers actually measure.

---

Headline results

Held-out perplexity per domain, and cross-domain specialization gaps (ratio of wrong-projection PPL to correct-projection PPL on the same data):

Domain	Held-out PPL	Diag PPL (train-tail)	Gap vs wrong projections
chat	72.14	43.7	x 11 889
code	24.89	10.0	x 27 563
agent	1.02	1.0	x 199 501
terminal	1.20	1.1	x 281 002
finance	23.72	20.9	x 18.9
law	23.87	21.5	x 18.4
benchmark	54.65	27.3	x 10 209

The three regimes are discussed below in the Interpretation section.

Mean cosine similarity between the seven down-projection matrices, per layer, ranges from 0.048 to 0.080 — far lower than Photon-P1 (0.19-0.27), indicating that at this parameter scale the domain-specific matrices become nearly orthogonal.

---

Architecture

Component	Value
Parameters	10,104,000 (~10.1M)
Hidden size	192
Layers	8
Attention heads	6 (query), 2 (key/value)
Head dim	32
Grouped Query Attention	yes (GQA, ratio 3:1)
Intermediate size (FFN)	560 (SwiGLU)
Normalization	RMSNorm, pre-norm
Positional encoding	RoPE (theta = 10000 base)
Max sequence length	1024
Embeddings	Tied (input = output)
Vocab size	8192
Domains	7
Specialization	Full down-proj per domain (hard-routed)
Precision	fp32 weights, bf16 during training

Architecture is a faithful Llama-style implementation (RMSNorm, RoPE, GQA, SwiGLU, tied embeddings, depth-scaled init), written entirely in plain PyTorch.

Domain specialization

Every FFN layer contains:

• A shared SwiGLU gate and up projection (input side)
• Seven independent down-projections (output side), each of shape (560, 192)
• No router — the domain is passed explicitly as domain_ids at each forward pass

During training, a batch from domain d only ever updates down_proj[d]. The gate, up, attention, and embeddings are shared across all seven domains.

Domains (index order):

ID	Domain	Content
0	chat	General conversation (UltraChat)
1	code	Python code, algorithms
2	agent	Function / tool calling
3	terminal	Shell, SWE-agent traces
4	finance	Financial reasoning
5	law	Legal text
6	benchmark	Math word problems, multiple choice, knowledge QA

---

Tokenizer

Byte-level BPE tokenizer trained from scratch on the training corpus.

• Vocab size: 8192
• Special tokens: unk, bos, eos, pad
• Round-trip decode verified

---

Training

Setting	Value
Training samples	17,648 sequences (500 held out)
Batch size	16 sequences x 512 tokens = 8,192 tok/step
Steps	2,500
Total tokens seen	~20.5M
Epochs (equivalent)	~2.3
Optimizer	AdamW (fused), beta = (0.9, 0.95)
Learning rate	6e-4, cosine schedule, 100 warmup steps
Weight decay	0.005
Precision	bf16 autocast
Hardware	1 x NVIDIA T4 (Kaggle)
Wall-clock time	~12 minutes

Training loss trajectory (excerpt):

Step	Loss
50	8.38
100	6.59
200	5.16
400	4.22

The training run was stopped at step 2,500 (scheduled end). Loss continued to decrease monotonically without oscillation through the end of the run.

Training data

Domain	Sources	Samples
chat	HuggingFaceH4/ultrachat_200k	4,500
code	CodeAlpaca-20k, python_code_instructions_18k	3,500
agent	tool-calling-english-100k	2,500
terminal	glaiveai/glaive-function-calling-v2, DCAgent2/financeagent_...	1,648
finance	IlyasFardaouixx/legalfinance-500k-mixed	2,000
law	IlyasFardaouixx/legalfinance-500k-mixed	2,000
benchmark	openai/gsm8k, allenai/ai2_arc, TIGER-Lab/MMLU-Pro	2,000

Datasets reused across multiple domains (legal-finance mix, function-calling corpora) are shuffled with different seeds so each domain sees a distinct slice.

---

Evaluation

Method. Two evaluations were performed:

1. Held-out perplexity. 500 examples were reserved before tokenization (70 per domain, 35 for terminal, 55 for finance, 65 for benchmark). PPL is computed on these examples with the correct down-projection, max length 256, padding masked out of the cross-entropy.

2. Cross-domain matrix. For each pair (used down-projection, data domain), PPL is measured on 25 training examples from the target domain. The diagonal represents the correct pairing. The off-diagonal represents forcing the wrong matrix.

Held-out perplexity

Domain	PPL
chat	72.14
code	24.89
agent	1.02
terminal	1.20
finance	23.72
law	23.87
benchmark	54.65

Cross-domain matrix

Rows = down-projection used, columns = data domain.

chat code agent terminal finance law benchmark
chat 43.7 79.6 293.9 80.6 294.1 293.2 210.5
code 343.8 10.0 320.7 37.6 510.2 520.9 658.4
agent 3444.4 2438.1 1.0 1311.5 599.5 586.0 4943.7
terminal 2235.7 1409.8 1553.1 1.1 376.7 379.9 1981.2
finance 1402967.8 759740.6 619785.4 1049052.1 20.9 22.5 739702.8
law 1704576.7 883152.9 598000.4 813932.3 21.6 21.5 924587.1
benchmark 263.2 633.8 789.9 612.8 569.9 576.4 27.3

Per-layer cosine similarity

Average pairwise cosine similarity between the 7 down-projection matrices, at each layer:

Layer	cos_mean
0	+0.059
1	+0.054
2	+0.050
3	+0.048
4	+0.056
5	+0.061
6	+0.067
7	+0.080

For comparison:

• Photon-P1 (1M params, 2 groups): cos_mean 0.19-0.27
• Photon-X (10M params, 7 groups): cos_mean 0.05-0.08

At larger per-expert capacity, the specialization becomes nearly orthogonal.

---

Interpretation

The cross-domain matrix reveals three distinct specialization regimes coexisting in the same model:

Regime 1 — Moderate specialization (chat, code, benchmark)

• Held-out PPL: 24-72
• Off-diagonal damage: 5x to 40x
• The domain-specific down-projections generalize gracefully to neighboring distributions.

These are the domains where the training data is heterogeneous (UltraChat, multi-source code, mixed benchmark QA). The model learns a robust general representation that transfers partially outside its own domain.

Regime 2 — Total specialization (agent, terminal)

• Held-out PPL: ~1.0 (essentially perfect token-level prediction)
• Off-diagonal damage: 10^5 to 10^6 (PPL of millions when the wrong projection is used)
• The domain-specific down-projections have memorized their training distribution and produce numerically degenerate outputs when fed off-distribution hidden states.

These are the domains where the training data is extremely templated (function-calling JSON with a fixed structure repeated thousands of times). The model has memorized the pattern exactly, and the wrong projection produces garbage because it has never seen anything else.

Regime 3 — Merged specialization (finance, law)

• Held-out PPL: ~24 for both
• Cross-perplexity between finance and law: 20.9 / 22.5 and 21.6 / 21.5 (statistically indistinguishable from their own diagonal)
• Both projections do essentially the same work.

This is the most revealing observation. Finance and law draw from the same source dataset (legalfinance-500k-mixed), shuffled differently. The model has detected that the two domains do not constitute a real boundary and converged to a single effective specialization. This is the correct behavior given the data, but it means the "7 domains" abstraction is nominal rather than functional for this pair.

---

Limitations

Read this section before using the model.

Photon-X is a research artifact, not a usable assistant. At 10.1M parameters, it is approximately 300x smaller than production LLMs.

1. Hallucination is the default behavior. Every generated token is a statistical guess. Any factual claim the model produces is unreliable.

2. No reasoning. The model does not perform arithmetic, does not follow multi-step logic, and does not understand instructions.

3. Group routing is manual. The caller must specify the domain ID at each forward pass. The model has no built-in classifier. Passing the wrong domain ID will produce output with perplexity orders of magnitude higher than the correct domain (see matrix above).

4. Domain leakage risk. Passing an out-of-distribution input to a specialized down-projection can produce numerically degenerate output. This is especially true for agent, terminal, finance, and law (see matrix).

5. Heavy memorization on templated corpora. On the agent and terminal domains, held-out PPL of ~1.0 indicates near-perfect memorization. The model will reproduce training corpus patterns rather than produce novel content.

6. Nominal finance/law separation. The two projections are interchangeable in practice (see Regime 3 above).

7. Language coverage. Predominantly English, some French.

8. No safety alignment. No RLHF, no DPO, no red-teaming.

---

Intended use

Photon-X is intended for:

• Research on hard-routed per-domain specialization at the ~10M scale.
• Ablation baseline against Photon 5 (soft-routed LoRA at 5M) and Photon-P1 (hard-routed, 2 groups, at 1M).
• Study of specialization regimes — the three-regime finding (moderate / total / merged) is directly measurable and can be extended to other corpora.
• Educational demonstration of how specialization and generalization trade off against per-expert capacity.

Photon-X is not intended for any production use, factual question answering, code generation, or any application requiring reliable output.

---

Comparison with Photon 5 and Photon-P1

Property	Photon 5	Photon-P1	Photon-X
Parameters	5.03M	1.003M	10.1M
Domains	7 (LoRA r=8)	2 (full down)	7 (full down)
Params / domain	~700k (r=8 only)	~500k	~1.44M
Routing	Hard (domain_id)	Hard (group_id)	Hard (domain_id)
Epochs	~2.4	~16	~2.3
Held-out evaluation	No	No	Yes
Cos similarity (mean)	n/a (LoRA)	0.19-0.27	0.05-0.08

Photon-X is the first model in the family with a genuine held-out split and the first to empirically observe the three-regime behavior.

---

Files

File	Size	Description
model.safetensors	~40.4 MB	Model weights (fp32, 10.1M params)
config.json	0.7 KB	Architecture and training metadata
tokenizer.json	540 KB	Byte-level BPE vocabulary
tokenizer_config.json	0.2 KB	Tokenizer wrapper configuration
banner.png	—	Repository banner
README.md	this file	Model card

Weights are stored as float32. The bf16 autocast during training did not affect storage precision.

---

How to load

Photon-X uses a custom architecture. The PhotonX class (defined in the training notebook) is required to reload the weights.

Loading requires:

1. Load the tokenizer from tokenizer.json (byte-level BPE, vocab 8192).
2. Load config.json and construct a PhotonXConfig from its fields.
3. Instantiate PhotonX(config) and load the safetensors state dict with strict=False (the tied lm_head weight is intentionally omitted from the file).
4. Call the model with input token IDs, a domain ID tensor (0 through 6), and optional attention mask and labels.

A complete reference implementation is provided in the accompanying training notebook.

---

Design notes

Why full down-projections rather than LoRA?

Photon 5 used rank-8 LoRA adapters on the down projection (~8% of matrix capacity per domain). Photon-P1 showed that full down projections enable much stronger specialization. Photon-X extends Photon-P1's approach from 2 groups to 7 groups at 10M total parameters.

Why 7 domains?

Matches Photon 5's granularity so the two models are directly comparable in domain structure. Only the specialization mechanism differs (LoRA vs. full down), isolating that single variable.

Why 2,500 steps (~2.3 epochs)?

Matches Photon 5's training regime. This keeps Photon-X in the "well-trained but not overfit" zone, in contrast to Photon-P1 (16 epochs, deliberate overfit). The three-regime finding emerges precisely because Photon-X is not overfit — the agent/terminal regime appears despite only 2.3 epochs.

Why per-layer cosine similarity in the 0.05-0.08 range?

At ~1.44M parameters per domain, the seven down projections have enough capacity to become nearly orthogonal. This is both a strength (maximal specialization) and a weakness (no graceful degradation when the wrong projection is used). The three regimes reflect different trade-offs along this axis.

---

Citation

If you use Photon-X in academic work, please cite it as:

muon AI. Photon-X: Hard-Routed Seven-Domain Specialization at 10M Parameters. 2025. Available at https://huggingface.co/muonai/photon-x-preview

---

Acknowledgments

Dataset sources remain under their respective licenses. Architecture draws on the Llama family (Touvron et al., 2023) and the MoE literature (Shazeer et al., 2017; Fedus et al., 2021).