Dot-125M

Dot-125M is a 125M-parameter (133.7M actual), Llama-3-style decoder-only transformer, pretrained entirely from scratch — no fine-tuning or continued pretraining from an existing checkpoint — by Perletter, part of Chirping Waves Limited (Ireland).

It was trained on 2.0B tokens (4 epochs over a 500M-token filtered, deduplicated sample of FineWeb-Edu) on a single consumer laptop GPU.

This is a base (pretrained) language model — it has not been instruction-tuned, RLHF'd, or chat-templated. It completes text; it does not reliably follow instructions or hold a conversation.

Model details

Parameters	133.7M
Architecture	Llama-3-style decoder-only, GQA, RoPE, RMSNorm, SwiGLU, tied embeddings
Layers / heads / KV heads	12 / 12 / 4
Hidden size	960
Context length	512
Vocab size	16,384 (byte-level BPE, trained from scratch on the training split)
Training tokens	2.0B (4 epochs × 500M-token corpus)
Training data	FineWeb-Edu (sample-10BT), quality-filtered + exact/near-deduplicated, English only
License	Apache 2.0

Benchmarks

Compared against GPT-2-small (124M, ~10B training tokens) — see full write-up for methodology.

Bits-per-byte (primary metric, tokenizer-fair comparison), on a held-out test split:

	bpb	ppl
Dot-125M	1.0142	20.64
GPT-2-small	1.0281	27.19

lm-evaluation-harness:

task	Dot-125M	GPT-2-small
arc_easy (acc)	48.23%	43.81%
hellaswag (acc_norm)	30.86%	31.14%
piqa (acc)	61.43%	62.89%
winogrande (acc)	49.57%	51.62%
lambada_openai (acc)	23.02%	32.56%

Mixed on the individual benchmark tasks (stronger on arc_easy, weaker on lambada_openai's long-range prediction — expected given the token/context budget: Dot-125M saw 500M unique tokens across 4 epochs at 512 context vs. GPT-2's ~10B tokens single-pass at 1024 context), but wins on the primary bits-per-byte metric.

Quantization (GGUF, via llama.cpp): Q8_0 stays within 0.01% bpb of full-precision f16. Q4_K_M is available but falls back to a different quant scheme for most tensors (this model's hidden size isn't a multiple of 256, the k-quant block size) — still only ~0.17% bpb degradation vs. f16, but not "true" Q4_K_M. Q8_0/Q4_0/Q5_0/Q5_1 are the quant types this model size supports natively.

Intended use

Research, experimentation, and demonstration of from-scratch small-LM pretraining. Not instruction-tuned — do not expect chat-assistant behavior out of the box. Not suitable for production use requiring factual reliability, safety filtering, or instruction following without further fine-tuning.

How to use

Transformers (safetensors):

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("perletter/Dot-125M")
model = AutoModelForCausalLM.from_pretrained("perletter/Dot-125M")

inputs = tok("The history of the internet", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50)
print(tok.decode(out[0], skip_special_tokens=True))

llama.cpp (GGUF):

llama-cli -m model-Q8_0.gguf -p "The history of the internet" -n 50

Limitations

• Small model, small training budget — general knowledge and reasoning are limited compared to larger contemporary models.
• English only.
• Base model only — no safety fine-tuning, no RLHF, no instruction-tuning. It will complete harmful, biased, or false text if prompted toward it, the same as any unaligned base LM.
• 512-token context window.

License

Apache 2.0 — the model weights are freely available for any use, including commercial, with no attribution requirement beyond the license notice. See LICENSE.

The training code/pipeline used to produce this model is not included in this release.

Citation

@misc{dot125m2026,
  title  = {Dot-125M},
  author = {Perletter, part of Chirping Waves Limited},
  year   = {2026},
  url    = {https://perletter.com}
}