zuu007/gpt2-from-scratch-chat

GPT-2 (124M), trained from scratch: chat model

zuu007/gpt2-from-scratch after supervised fine-tuning on databricks/databricks-dolly-15k (loss on the answers only). Part of github.com/zuurashad/gpt2-from-scratch, a PyTorch reproduction of GPT-2 small trained from scratch on a single 6GB laptop GPU. Try it in your browser.

The architecture is exactly GPT-2 small: 12 layers, 12 heads, width 768, a 1024-token context and the GPT-2 BPE tokenizer. The weights load into Hugging Face's GPT2LMHeadModel unchanged.

Files

file	what it is
model.safetensors	full-precision (fp32) weights
onnx/model_quantized.onnx	per-channel int8 ONNX for the browser demo. Validation loss vs fp32: +0.0013 generating token by token (8,192 FineWeb-Edu tokens), +0.0058 over one forward pass (51,200 tokens)
training.json	training step, validation loss and the training arguments

Use it

from transformers import pipeline

generate = pipeline("text-generation", model="zuu007/gpt2-from-scratch-chat")
print(generate("<|system|>\nYou are a helpful assistant.<|endoftext|><|user|>\nWhat is photosynthesis?<|endoftext|><|assistant|>\n", max_new_tokens=60)[0]["generated_text"])

The model expects the chat template it was fine-tuned on (as above): each turn is <|system|>, <|user|> or <|assistant|> plus a newline and the text, ended by the single token <|endoftext|> (id 50256), where it also stops. The role markers are plain text, not special tokens.

In the browser, with transformers.js: await AutoModelForCausalLM.from_pretrained("zuu007/gpt2-from-scratch-chat", { dtype: "q8" }).

Limitations

This is a 124M-parameter model trained on 2.5B tokens. It writes fluent English, but it is frequently wrong, especially about facts, arithmetic and recent events. Its only alignment is supervised fine-tuning on a small instruction dataset, so it can produce incorrect or inappropriate text.

Known limitation: follow-up questions. The chat model was fine-tuned only on single-turn examples (Dolly-15k), so it never learned to prioritise the newest message in a conversation. With 124M parameters it tends to continue the topic that fills most of its context, so an unrelated follow-up often gets an answer about the earlier question. Press Clear before changing topic. Multi-turn fine-tuning data (e.g. OpenAssistant) would be the fix.

Data and licences

• Pretraining: FineWeb-Edu (ODC-By 1.0)
• Fine-tuning (chat model only): databricks-dolly-15k, licensed CC BY-SA 3.0. Treat the chat weights as share-alike.
• Code: MIT, derived in part from Andrej Karpathy's build-nanogpt (MIT)