rahuliyer/gpt2-large-smoltalk

GPT-2 large, instruction-tuned on SmolTalk

The released GPT-2 large weights (774M parameters) fine-tuned into a chat model on smol-smoltalk, with the loss on assistant replies only. Small, medium and large versions are compared in the instruction-tuning write-up.

Results

Validation loss (assistant tokens, 1,000 test conversations)	1.074
Training	one epoch, 460,341 conversations, 14,386 steps of 32
Batch × accumulation, peak learning rate	16 × 2, 2e-5
Hardware and time	one H100 in bf16, 1.7 hours

What is the capital of France?

The capital of France is Paris. It is a city known for its rich history and cultural significance. Paris is located in the northern part of France, and it is a major tourist destination.

Files

• best.safetensors: the weights at the lowest validation loss (the final step).
• config.json: the training config.

Loading

These weights use the GPT2 class from gpt2-repro, not Hugging Face Transformers, and a vocabulary of 50,263 tokens: GPT-2's plus six chat tokens. Clone gpt2-repro and install its dependencies with uv, then run the code below from the repository root:

git clone https://github.com/rahuliyer/gpt2-repro
cd gpt2-repro
uv sync

from huggingface_hub import hf_hub_download

from inference import generate_text
from instruct import get_tokenizer
from instruct.chat import encode_prompt
from instruct.tokenizer import ASSISTANT_END_TOKEN
from model import GPT2, GPT2LargeConfig

tokenizer = get_tokenizer()
path = hf_hub_download("rahuliyer/gpt2-large-smoltalk", "best.safetensors")
model = GPT2.load_from_checkpoint(GPT2LargeConfig(vocab_size=tokenizer.n_vocab), path)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]
prompt = encode_prompt(tokenizer, messages)
text = generate_text(
    model, tokenizer, prompt, max_length=256,
    stop_token_id=tokenizer.encode_single_token(ASSISTANT_END_TOKEN),
)
print(text[len(tokenizer.decode(prompt)):])

generate_text samples with temperature 1 and top-5 by default.

Chat format

<|system-start|>...<|system-end|><|user-start|>...<|user-end|><|assistant-start|>

The system message is optional. The model ends its reply with <|assistant-end|>. Conversations were truncated to 1,024 tokens in training.

Limitations

It is the strongest of the three sizes but still drifts or contradicts itself in longer replies. English only. Not suitable for anything that relies on accurate or safe output.