mtlm-7m-base — a language model trained end-to-end in machin

MTLM (machin tiny language model) is an experiment in building an LLM with no ML framework at all: the tokenizer training, the pretraining loop (AdamW, RoPE, RMSNorm, SwiGLU, tied embeddings), the int8 export and the inference server are all written in machin (MFL, a machine-first language that compiles to C), and the whole thing was trained on a 6-core CPU shared with other services.

This repo is the base model: 7.2M parameters, fluent TinyStories-class English. The tool-calling fine-tune lives in javimosch/mtlm-7m-tools.

Honest numbers

architecture	llama-2 style: dim 288, 6 layers, 6 heads, hidden 768, vocab 4096 (custom byte-level BPE), context 256
parameters	7,155,360
data	TinyStoriesV2-GPT4 train split, 98.3M tokens seen (12,000 steps × 32 × 256)
optimizer	AdamW β=(0.9, 0.95), wd 0.1, lr 5e-4, 500 warmup, cosine to 5e-5, grad clip 1.0
hardware	Intel i5-13400T LXC, 6 cores, no GPU, shared box (avg 741 tok/s)
wall time	36.8 h
validation loss	3.63 (step 500) → 1.916 (step 12,000), monotone; full curve in train_log.jsonl

Sample (temperature 0.7, top-p 0.9):

Tom and Lily went to the park. They saw a little bird on a branch. The bird looked very helpless. It had many feathers and claws. "Hello, bird," Tom said. "Why are you in the wind?" "I'm sad," Lily said. "I want to be independent and find ways to fly." "Me too, me too!" Tom said. "But I can't fly."

It is a story model: grammar, dialogue and paragraph structure are solid, world-sense is not. It knows nothing outside children's stories.

Files

• m7.mtlm — fp32 checkpoint in the mtlm1 layout (256-byte header + tensors), the training-side format.
• m7.bin — llama2.c "version 2" int8 export (ak42, group size 32), 8.0 MB. Quantization max abs error 1.7e-3.
• tokenizer.bin — llama2.c tokenizer format (4096 pieces with merge scores; ids 0..2 = unk/bos/eos, 3..258 = raw bytes).
• train_log.jsonl — one JSON line per step (loss, lr, grad-norm, tok/s) plus the evals.

Run it

Serving is done by anvil, the pure-MFL runtime (OpenAI-compatible server, int8 kernels):

anvil models/m7.bin 80 "Once upon a time"        # CLI, greedy
anvil-serve models/m7.bin 8090                     # /v1/chat/completions

tokenizer.bin must sit at models/tokenizer.bin relative to the working directory. The files are not transformers checkpoints; a safetensors conversion is straightforward, a compatible HF tokenizer is not (the tokenizer stores merge scores, not merge pairs), so none is provided yet.

What is and is not machin

Pure MFL: BPE tokenizer training and encoding, dataset tokenization, the transformer forward/backward (gradient-checked at 1e-5 against a float64 numpy oracle), AdamW, checkpointing, the int8 exporter, the CPU kernels (a cache-blocked GEMM and fused attention/RMSNorm/SiLU/softmax builtins were added to machin for this), and inference. Python was used only as a test oracle. The trainer found and fixed four real bugs in itself along the way (a window sampler, an optimizer-state save, a stderr logging footgun, an arena escape), all documented in the source.

Why

MTLM's thesis is that the language part of an assistant can be a tiny, cheap, verifiable artifact trained once on a CPU, and that everything else (knowledge, actions) belongs in tools. This base is step one; the tools model is step two.