tinystories-50m-instruct

A ~50M-parameter GPT-style transformer (6 layers, 12 heads, 768 embd, 1024 context) trained from scratch on TinyStories and instruction-fine-tuned to chat in a User: ... / Assistant: ... format.

Quick start

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("hayder86al/tinystories-50m-instruct")
model = AutoModelForCausalLM.from_pretrained("hayder86al/tinystories-50m-instruct", trust_remote_code=True)

def chat(prompt):
    text = f"User: {prompt}\nAssistant:"
    inputs = tokenizer(text, return_tensors="pt")
    outputs = model.generate(
        **inputs, max_new_tokens=120, do_sample=True, temperature=0.5,
        top_k=40, use_cache=False,
        pad_token_id=tokenizer.pad_token_id, eos_token_id=tokenizer.eos_token_id,
    )
    text_out = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return text_out.split("Assistant:")[-1].strip()

print(chat("What is 2+2?"))

trust_remote_code=True is required because this repo defines a custom architecture (modeling_minigpt.py / configuration_minigpt.py) — review that code before running it.

Limitations

Single-turn only (no conversation memory), 1024-token context, small model — expect simple, sometimes ungrammatical answers, not factual accuracy.