OtakMini

OtakMini is a 135M-parameter English language model pretrained from scratch on a single consumer GPU (RTX 5050, 8 GB) in about 12 days, on 18B tokens. This is a base model: no instruction tuning, RLHF or safety tuning. It uses the Qwen3 architecture (grouped-query attention, QK-norm, SwiGLU, tied embeddings) so it loads with stock transformers.

Model details

Parameters	135.03M (18.9M of them in the tied embedding)
Layers / width	30 layers, hidden size 576, FFN 1728
Attention	9 heads, 3 KV heads, head dim 64, RoPE (theta 1e5)
Context length	2048 tokens
Vocabulary	32,768 (custom byte-level BPE, digits split)
Special tokens	<|endoftext|> (id 0: BOS/EOS/PAD), <|im_start|> (1), <|im_end|> (2, reserved for chat tuning, untrained here)
Precision	bf16 weights (model.safetensors, 270 MB)

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "syaffers/OtakMini"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16)

ids = tok("The capital of France is", return_tensors="pt")
out = model.generate(**ids)  # uses the sampling defaults in generation_config.json
print(tok.decode(out[0]))

generation_config.json ships with temperature=0.7, top_p=0.9, repetition_penalty=1.1.

Evaluation

Zero-shot, full test sets, bf16, lm-evaluation-harness 0.4.13. acc_norm where available (HellaSwag, ARC, PIQA), otherwise acc.

Task	Score	±stderr	Chance
HellaSwag (acc_norm)	39.9	0.5	25.0
ARC-Easy (acc_norm)	52.2	1.0	25.0
ARC-Challenge (acc_norm)	29.4	1.3	25.0
PIQA (acc_norm)	68.0	1.1	50.0
Winogrande (acc)	51.8	1.4	50.0
LAMBADA (acc)	33.9	0.7
LAMBADA (perplexity)	38.2	1.5

Validation loss on held-out web text is 2.606 (perplexity about 13.5).

Training

• Data: about 19B tokens of English text, 11 sources, one pass over the pool (about 0.95 epochs). Mix for most of training, with the mix for the last 10% in brackets:

  Source	Share
  ClimbMix	30% (20%)
  Ultra-FineWeb	20% (15%)
  Nemotron-CC-Math-v1 (4plus)	10% (15%)
  Cosmopedia v2	10% (12%)
  FineWeb-Edu	7% (10%)
  FinePhrase (FAQ / tutorial / math / table rewrites)	15% (20%)
  Ultra-FineWeb-L3 (multi-style)	5% (5%)
  StarCoderData (Python)	3% (3%)

• Decontamination: documents with a 13-gram (word-level) overlap with HellaSwag, ARC, PIQA, Winogrande or LAMBADA were removed: 10,937 of 28.35M documents (0.04%).

• Optimiser: Muon (lr 0.01, weight decay 0.01) on 2D weight matrices, AdamW (lr 1.5e-3) on embeddings and norms; batch 524,288 tokens; 34,332 steps.

• Schedule: 2,000 warmup steps, constant lr, then a 1-sqrt cooldown over the last 10% of steps with the anneal data mix.

• Hardware: one RTX 5050 (8 GB) at about 17k tokens/s, 2026-10-16 to 2026-10-28.

Loss curves

Train loss (3-hour means) and validation loss over the run. The drop at the right is the cooldown with the anneal data mix. The hours axis is approximate.

Intended use and limitations

Meant for research, experiments with small models, and as a starting point for fine-tuning. It is not meant for anything where correctness matters.

• It writes fluent text but gets facts wrong (for example, it will say Paris is the largest city in the world, or invent populations and dates), and it cannot do arithmetic reliably.
• Without a repetition penalty it loops. Text drifts off-topic after a sentence or two.
• English only, 2048-token context, no instruction following and no refusal behaviour. It can reproduce biases and unsafe content from web data.
• The ChatML tokens exist in the tokenizer but the model has never seen them in use. A chat template is not included.

License and data

Released under CC BY-NC 4.0 (non-commercial), because the training data includes non-commercial and agreement-restricted sources (ClimbMix is CC BY-NC 4.0; NVIDIA's Nemotron-CC-Math data agreement and the StarCoderData terms of use also apply). Check each dataset's own license before reusing the weights beyond research.

Citation

@misc{otakmini2026,
  title  = {OtakMini: a 135M-parameter language model pretrained from scratch on a single consumer GPU},
  author = {syaffers},
  year   = {2026},
  url    = {https://huggingface.co/syaffers/OtakMini}
}