rahuliyer/gpt2-medium-smoltalk

GPT-2 medium, instruction-tuned on SmolTalk

The released GPT-2 medium weights (355M parameters) fine-tuned into a chat model on smol-smoltalk, with the loss on assistant replies only. Small, medium and large versions are compared in the instruction-tuning write-up.

Results

Validation loss (assistant tokens, 1,000 test conversations)	1.216
Training	one epoch, 460,341 conversations, 14,386 steps of 32
Batch × accumulation, peak learning rate	16 × 2, 3e-5
Hardware and time	one H100 in bf16, 54 minutes

What is the capital of France?

The capital of France is Paris. It is situated in the northern part of the country, and it is a major cultural, historical, and commercial center of the French Republic.

Files

• best.safetensors: the weights at the lowest validation loss (the final step).
• config.json: the training config.

Loading

These weights use the GPT2 class from gpt2-repro, not Hugging Face Transformers, and a vocabulary of 50,263 tokens: GPT-2's plus six chat tokens. Clone gpt2-repro and install its dependencies with uv, then run the code below from the repository root:

git clone https://github.com/rahuliyer/gpt2-repro
cd gpt2-repro
uv sync

from huggingface_hub import hf_hub_download

from inference import generate_text
from instruct import get_tokenizer
from instruct.chat import encode_prompt
from instruct.tokenizer import ASSISTANT_END_TOKEN
from model import GPT2, GPT2MediumConfig

tokenizer = get_tokenizer()
path = hf_hub_download("rahuliyer/gpt2-medium-smoltalk", "best.safetensors")
model = GPT2.load_from_checkpoint(GPT2MediumConfig(vocab_size=tokenizer.n_vocab), path)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]
prompt = encode_prompt(tokenizer, messages)
text = generate_text(
    model, tokenizer, prompt, max_length=256,
    stop_token_id=tokenizer.encode_single_token(ASSISTANT_END_TOKEN),
)
print(text[len(tokenizer.decode(prompt)):])

generate_text samples with temperature 1 and top-5 by default.

Chat format

<|system-start|>...<|system-end|><|user-start|>...<|user-end|><|assistant-start|>

The system message is optional. The model ends its reply with <|assistant-end|>. Conversations were truncated to 1,024 tokens in training.

Limitations

It answers simple factual questions well but drifts or contradicts itself in longer replies. English only. Not suitable for anything that relies on accurate or safe output.