kaizen-72M-lit-base

A 72M parameter decoder-only Transformer base language model trained from scratch on the TinyStories V2 (GPT-4) dataset (roneneldan/TinyStories).

Model Specs

Attribute	Value
Parameters	71.99M
Architecture	8L / 12H / 768D / 2048FF (SwiGLU, RoPE, RMSNorm)
Dataset	TinyStories V2 GPT-4 (roneneldan/TinyStories)
Context Length	256 tokens
Vocab Size	10,000 (BPE trained with gigatoken)
Final Loss	Train: 1.2496 | Val: 1.2521
Perplexity	~3.49

Training Curve

Usage

import torch
from safetensors.torch import load_file
from model import TransformerLM
from tokenizer import Tokenizer

# Load tokenizer & model
tok_data = torch.load("tokenizer.pt", map_location="cpu")
tokenizer = Tokenizer(vocab=tok_data["vocab"], merges=tok_data["merges"], special_tokens=["<|endoftext|>"])

model = TransformerLM(10000, 256, 768, 8, 12, 2048, 10000.0)
state_dict = load_file("model.safetensors", device="cpu")
model.load_state_dict(state_dict)