gladios-tiny.story-0.1B

🧠 Overview

gladios-tiny.story-0.1B is a decoder-only Transformer (GPT-2-style architecture, 124M / 0.1B parameters) trained from scratch on the full TinyStories dataset (2.1 million children's stories, ~470M tokens).

The model uses the GPT-2 tokenizer (50,257-token vocabulary) and generates short, coherent stories in English, in the style of the training data.

👉 Try it without installing anything: Gradio demo (ZeroGPU) 👉 Full training code: github.com/plimb-ai/gladios-tiny-story

🏗️ Architecture

Type	Decoder-only Transformer (GPT-2 architecture)
Parameters	~124M (0.1B)
Layers	12
Attention heads	12
Embedding dim	768
Context length	512 tokens
Tokenizer	GPT-2 (tiktoken / 50,257-token vocab)
Weights	safetensors

🚀 Usage

from transformers import pipeline

gen = pipeline("text-generation", model="plimb/gladios-tiny.story-0.1B")
print(gen(
    "Once upon a time",
    max_new_tokens=200,
    do_sample=True,
    temperature=0.8,
    top_k=50,
)[0]["generated_text"])

Or directly with AutoModelForCausalLM:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained("plimb/gladios-tiny.story-0.1B")
tok = AutoTokenizer.from_pretrained("plimb/gladios-tiny.story-0.1B")

ids = tok("Once upon a time", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=200, do_sample=True, temperature=0.8, top_k=50,
                      pad_token_id=tok.eos_token_id)
print(tok.decode(out[0], skip_special_tokens=True))

📚 Training data

• Dataset: roneneldan/TinyStories (full train split)
• Tokenizer: GPT-2, with each story separated by the special <|endoftext|> token
• Trained on the entire dataset (no subsampling)

⚠️ Limitations

• Only writes short children's stories, in English
• No general world knowledge, no instruction-following, no conversational ability
• May hallucinate or lose coherence on prompts far outside the TinyStories style

📄 License

MIT