SLM-10M

A 9.97M parameter causal language model trained from scratch, targeting the Open SLM Leaderboard <10M tier.

Intended Use

This is a research model optimised for NLU benchmarking tasks, not open-ended generation. It is best suited for:

Task	Examples
Multiple-choice QA	ARC, HellaSwag, PIQA, ArithMark — score each candidate and pick the highest
Log-likelihood ranking	Rank candidate continuations or document relevance by perplexity
SLM research	Ablations, architecture studies, efficiency benchmarks at the <10M scale
Perplexity evaluation	Measuring language model fit on held-out text corpora

It is not suited for open-ended text generation, chat, or instruction following — at 10M parameters the vocabulary (8,192 tokens) and capacity are too limited for fluent free-form output.

Model Details

Property	Value
Parameters	9,968,640 (~10M)
Architecture	Causal Transformer
Vocabulary	8,192 tokens
Context length	1,024 tokens
Training tokens	25B
Precision	bfloat16

Architecture

Component	Config
Hidden size	256
Layers	12
Q heads / KV heads	8 / 2 (GQA)
Head dim	32
FFN intermediate	640
Positional encoding	RoPE (θ=100k)
Normalization	RMSNorm (fp32 upcast)
Activation	SwiGLU
Attention	GQA + QK-Norm
Weight tying	Embed ↔ LM head

Design follows SotA SLM recipes (GPT-X2, Qwen3, Gemma2): QK-Norm prevents attention logit explosion, Z-loss stabilises early training (disabled after 31B tokens), scaled residual init keeps residual stream variance bounded.

Training

Data mix (25B tokens total):

Source	Weight
FineWeb-Edu	55%
Cosmopedia-v2	25%
FineWeb-HQ	10%
FineMath	10%

Optimizer: AdamW (fused) — lr=3e-3, min_lr=3e-4, β=(0.9, 0.95), wd=0.1, grad_clip=1.0

LR schedule: Warmup (1k steps) → stable → cosine decay tail (last 15% of steps)

Batch: 512K tokens/step (micro-batch 32 × grad_accum 16 × seq_len 1024)

Hardware: NVIDIA GB10, bfloat16, torch.compile

Evaluation

Zero-shot evaluation on the Open SLM Leaderboard benchmarks:

Benchmark	Score
HellaSwag (acc_norm)	26.53%
ARC-Easy (acc_norm)	30.47%
ARC-Challenge (acc_norm)	25.00%
PIQA (acc_norm)	50.92%
ArithMark-2.0	24.32%
Avg	32.38%

Avg = (HellaSwag + (ARC-Easy + ARC-Challenge) / 2 + PIQA + ArithMark) / 4

Evaluated using lm-evaluation-harness and the ArithMark-2.0 custom benchmark script.

Usage

This model is a research artifact for benchmarking, not a chat or generation model. At 10M parameters it excels at log-likelihood ranking tasks (multiple-choice benchmarks) rather than free-text generation.

Scoring / ranking (recommended)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import torch.nn.functional as F

model = AutoModelForCausalLM.from_pretrained(
    "liodon-ai/slm-10m",
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("liodon-ai/slm-10m", trust_remote_code=True)

def score(context, completion):
    full = tokenizer.encode(context + completion, return_tensors="pt").to("cuda")
    ctx_len = len(tokenizer.encode(context, add_special_tokens=False))
    with torch.no_grad():
        logits = model(full).logits[0]
    return -F.cross_entropy(logits[ctx_len - 1:-1], full[0, ctx_len:]).item()

context = "Which is an example of a renewable energy resource? Answer:"
choices = [" biomass", " coal", " gas", " oil"]
scores  = [score(context, c) for c in choices]
best    = choices[scores.index(max(scores))]
print(f"Best answer: {best.strip()}")
# → Best answer: biomass

Citation

@software{liodonai2026slm10m,
  author = {{Liodon AI}},
  title = {SLM-10M},
  year = {2026},
  url = {https://huggingface.co/liodon-ai/slm-10m}
}

License

Apache 2.0