Sane-118M

A 118.5M-parameter English language model trained from scratch — architecture, tokenizer, training code and behaviour data are all original. No existing weights were fine-tuned, distilled or merged.

Same design goal as Sane-47M, 2.5× the size: a small honest chat model whose knowledge is meant to come from a supplied context rather than from its own weights.


Parameters	118.48 M (tied embeddings)
Architecture	RoPE · RMSNorm · SwiGLU · pre-norm, 15 layers, d_model 768, 12 heads, FFN 2048
Context	1024 tokens
Tokenizer	own byte-level BPE, vocabulary 16 000 (shared with Sane-47M)
Training tokens	≈ 3.6 B (≈ 2.9 passes over a 1.25 B-token corpus)
Hardware	Kaggle TPU v5e-8, ~6 hours
Schedule	WSD (warmup → stable 5e-4 → cosine decay to 5e-5 over the last 4 000 steps)
Final step	22 000
Language	English only
Precision	fp16 weights (~249 MiB)

Quickstart

Everything needed is in this repository.

pip install torch tokenizers huggingface_hub
hf download sekund0chka/sane-118m --local-dir sane-118m
cd sane-118m
python sane_chat.py --cpu --ckpt sane_final_fp16.pt --tok tokenizer.json

Runs on CPU. Commands inside the chat: /ctx <text> supplies a retrieval context, /reset clears history, /q quits.

Prompt format

Four control tokens, no system prompt — the persona lives in the weights.

<|user|>
{question}
<|sane|>
{answer}<|end|>

With retrieval, the context goes before the question:

<|context|>
Search results: The Ridgeloom V4 weighs 34 kg and sells for 690 euros.
<|user|>
How much does the Ridgeloom V4 weigh?
<|sane|>

Sampling defaults: temperature 0.7, top_k 50, top_p 0.92, repetition_penalty 1.12. For factual questions over a context, drop to temperature 0.3 or decode greedily.

Evaluation

Zero-shot, log-likelihood scoring, 200 examples per task, measured with the project's own sane_eval.py:

Task	Metric	Sane-118M	Random	GPT-2 small (124M)
HellaSwag	acc_norm	36.0	25.0	~31
ARC-Easy	acc	47.5	25.0	~44
ARC-Easy	acc_norm	28.5	25.0	—
LAMBADA	acc	22.0	~0	~33
FineWeb held-out	perplexity	19.6	—	—

Read this as: roughly GPT-2-small level, on ~40× less training data than GPT-2 saw relative to its size — clearly above chance, clearly not a parrot, and clearly a small model. LAMBADA is the weak spot: it rewards long-range context, which 1024 tokens and 118M parameters do not give much of. PIQA was not measured.

What it does and does not do

Works:

stable identity — knows it is a model, will not roleplay as a human;
reads <|context|> and answers from it, including messy search-result text with distractors, and says when the answer is not there;
short multi-turn conversation inside the window;
simple code and string tasks (s[::-1] and friends).

Does not work:

facts without a context are unreliable — a capacity wall, not a tuning problem. Additional factual fine-tuning was tried and rejected: it produced more pattern-matched leakage, not more knowledge;
arithmetic;
long free-form generation stays grammatical but loses the thread;
anything outside English.

Training data

Source	License
HuggingFaceFW/fineweb-edu (`sample-10BT`)	ODC-By 1.0
HuggingFaceTB/smol-smoltalk	Apache-2.0

The behaviour pool — identity, retrieval-reading, refusal-of-roleplay — is procedurally generated by the author and contains no third-party text.

ODC-By requires attribution; ATTRIBUTION.md in this repository is that attribution and must travel with the model if you redistribute it.

Bias, risks and limitations

Inherits the biases of web text filtered for educational quality. No safety tuning, no harmful-request refusals, no guardrails — treat output as untrusted text. No memory between sessions, no internet access: anything time-sensitive must be supplied in <|context|>.

License

Released under the Sane Community License 1.0 (see LICENSE). In short: free to use, modify and redistribute, including commercially, until the product built on it earns more than 1 000 USD per month — beyond that, contact the licensor. Attribution notice required: "Powered by Sane-118M."

Citation

@misc{sane118m,
  title  = {Sane-118M: a small language model trained from scratch},
  author = {S.C.},
  year   = {2026},
  url    = {https://huggingface.co/sekund0chka/sane-118m}
}

Downloads last month: -; Downloads are not tracked for this model. How to track

sekund0chka
/

sane-118m