rahuliyer/gpt2-small-smoltalk

GPT-2 small, instruction-tuned on SmolTalk

The released GPT-2 small weights (124M parameters) fine-tuned into a chat model on smol-smoltalk, with the loss on assistant replies only. Small, medium and large versions are compared in the instruction-tuning write-up.

Results

Validation loss (assistant tokens, 1,000 test conversations)	1.446
Training	one epoch, 460,341 conversations, 14,386 steps of 32
Batch × accumulation, peak learning rate	2 × 16, 5e-5
Hardware and time	one RTX 2080 in fp32, 14.2 hours

What is the capital of France?

The capital of France is Saint-Ouen-sur-Mer in Belgium.

Files

• best.safetensors: the weights at the lowest validation loss (the final step).
• config.json: the training config.

Loading

These weights use the GPT2 class from gpt2-repro, not Hugging Face Transformers, and a vocabulary of 50,263 tokens: GPT-2's plus six chat tokens. Clone gpt2-repro and install its dependencies with uv, then run the code below from the repository root:

git clone https://github.com/rahuliyer/gpt2-repro
cd gpt2-repro
uv sync

from huggingface_hub import hf_hub_download

from inference import generate_text
from instruct import get_tokenizer
from instruct.chat import encode_prompt
from instruct.tokenizer import ASSISTANT_END_TOKEN
from model import GPT2, GPT2SmallConfig

tokenizer = get_tokenizer()
path = hf_hub_download("rahuliyer/gpt2-small-smoltalk", "best.safetensors")
model = GPT2.load_from_checkpoint(GPT2SmallConfig(vocab_size=tokenizer.n_vocab), path)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]
prompt = encode_prompt(tokenizer, messages)
text = generate_text(
    model, tokenizer, prompt, max_length=256,
    stop_token_id=tokenizer.encode_single_token(ASSISTANT_END_TOKEN),
)
print(text[len(tokenizer.decode(prompt)):])

generate_text samples with temperature 1 and top-5 by default.

Chat format

<|system-start|>...<|system-end|><|user-start|>...<|user-end|><|assistant-start|>

The system message is optional. The model ends its reply with <|assistant-end|>. Conversations were truncated to 1,024 tokens in training.

Limitations

It learned the chat format but often gets facts wrong, as above. English only. Not suitable for anything that relies on accurate or safe output.