ohara-base-d12

The pretrained base model from a nanochat-style run with ohara. 124M effective parameters, trained from scratch on 1.48B tokens of ClimbMix on 2xA100-80GB in about 3.3 hours.

This is a base language model, not a chat model. It continues text; it does not answer questions and it never stops on its own. Given "Why is the sky blue?" it will happily continue into an unrelated encyclopedia article. If you want something that responds to you, use ohara-chat-d12, which is this checkpoint after supervised finetuning.

Use this one if you want to run your own finetuning from a base.

Architecture

Llama-style decoder: RoPE, SwiGLU, RMSNorm, untied embeddings, nanochat-style initialization, trained with Muon (matrices) + AdamW (embeddings and scalars).

Effective / total params	123.5M / 162.2M
Layers / hidden / heads	12 / 768 / 6
Context length	2048
Vocabulary	50,265 (gpt-neo-125m + 8 reserved chat tokens)

The 8 conversation special tokens are reserved in the vocabulary but never seen during pretraining, so a finetune can use them without resizing the embedding.

Training

2,827 steps over 1.48B tokens, batch 524,288 tokens, Muon at lr 0.02 under a warmup-stable-decay schedule, bf16 mixed precision.

step	val bits/byte
750	1.0625
1250	1.0057
2000	0.9458
2827	0.9062

Final val loss 2.9153, next-token accuracy 43.6%.

For scale: nanochat reaches GPT-2 grade at 0.718 bpb using roughly 28x more compute than the 1.4e18 FLOPs spent here. This is an honestly small model.

Usage

import torch
from ohara.chat_engine import config_from_state_dict
from ohara.models.llama import Llama

state = torch.load("base_d12.pt", map_location="cpu")["model"]
model = Llama(config_from_state_dict(state))
model.load_state_dict(state)

To finetune it into a chat model:

python examples/train_sft.py --pretrained-checkpoint base_d12.pt

Reproducing

DEPTH=12 bash runs/speedrun.sh