MaxGPT-3 (235M)
The largest of the four Classic MaxGPT models, pretrained from scratch on about 5 billion unique tokens.
What it is
- 235M parameters: 16 transformer blocks, hidden size 1024, 16 heads, a 1,024-token context.
- The same 16,000-token byte-pair tokenizer, written from scratch.
- Pretrained for about 6 to 7 days on one RTX 5070 (12 GB): 2.44M steps, about 20B token-passes over a seven-dataset mix of roughly 5B unique tokens (past Chinchilla-optimal, the LLaMA "train past optimal" recipe), finishing at a loss around 2.0.
What to expect
Real-looking, structured answers that drift off topic, as a 235M model does. MaxGPT-3.5 is this model after chat fine-tuning. MaxGPT-3 also could not remember a name told to it a few turns earlier, which is the question Max's MaxGPT-Planck research now studies.
Run it
pip install torch safetensors
python run.py "USER: How do I make brownies?\nASSISTANT:"
run.py loads model.safetensors with the model's own code in this folder (config.py, model.py, tokenizer.py, sample.py) and samples 120 tokens at temperature 0.8, top-k 50, on CUDA, Apple MPS or CPU.
The MaxGPT family
Built from scratch by Maxwell Brohm, a high school senior, in PyTorch with no pretrained weights and no transformers library. Code and full write-up: https://github.com/MaxwellBrohm/MaxGPT.
| Model | Parameters | Layers x width | Context | What it shows |
|---|---|---|---|---|
| MaxGPT-1 | 23M | 6 x 384 | 256 | coherent English emerges |
| MaxGPT-2 | 110M | 12 x 768 | 512 | learns the USER/ASSISTANT chat format |
| MaxGPT-3 | 235M | 16 x 1024 | 1024 | a real-looking chatbot |
| MaxGPT-3.5 | 235M | MaxGPT-3 plus chat fine-tuning | 1024 | replies like an assistant |
| MaxGPT-Mini (Neo series) | 113M | 12 x 768, grouped-query attention, RoPE, SwiGLU | 1024 | the modern design, chat-tuned with SFT and DPO |
MaxGPT-Ultra, a 1.09B model on a modern decoder design, is in progress.