mixailplastinin/llm-from-scratch-1b

LLM from Scratch — 1B Model

GPT-style decoder-only transformer trained from scratch with custom framework.

Architecture

• Parameters: ~1.07B
• Layers: 20
• d_model: 2048
• Heads: 32
• Context: 1024 tokens
• Vocab: 32K BPE
• Norm: RMSNorm
• Position: RoPE

Training

• Warm-start from 113M base (v0)
• 5000 steps on FineWeb-Edu (30B tokens)
• AdamW, lr=1.5e-4, cosine schedule
• CPU-only training

Framework

Custom multi-agent framework in agents/:

• Architect (model design)
• Data Engineer (corpus preparation)
• Trainer (training loop)
• Curator (checkpoint management)
• Orchestrator (coordination)

Code

Full source code: https://github.com/plastininmixail-ai/llm-from-scratch-1b