mvGPT

Minimum Viable GPT is a small decoder-only transformer, written from scratch in PyTorch for the YouTube series Minimum Viable GPT. It writes short children's stories, in the style of TinyStories. The code that trained it is at https://github.com/nenuadrian/mvGPT, with a tag for each step of each episode.

This revision, ep012-pretrain: 91 million parameters, pretrained for 30 minutes on four RTX 4090s, on all of TinyStories, with rotary embeddings and a byte-level BPE tokenizer.

It has 91.3 million parameters, saved as a GPTNeoXForCausalLM: 12 blocks of 12 heads, an embedding size of 768, rotary position embeddings, GELU, and a head tied to the token embeddings. Its vocabulary is 8,192 tokens: the 256 bytes, 7,934 merges learned by its own byte-level BPE tokenizer, written in C++, then padding and end of text, saved here as a tokenizer.json.

Use it

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("nenuadrian/mvGPT", revision="ep012-pretrain")
model = AutoModelForCausalLM.from_pretrained("nenuadrian/mvGPT", revision="ep012-pretrain")
ids = tokenizer("Once upon a time", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=200, do_sample=True, temperature=0.8, top_k=50)
print(tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

In a browser, with Transformers.js, from the 8-bit ONNX model in onnx/:

import { pipeline } from "https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1";

const writer = await pipeline("text-generation", "nenuadrian/mvGPT", { dtype: "q8", revision: "ep012-pretrain" });
const [out] = await writer("Once upon a time", { max_new_tokens: 100, do_sample: true, top_k: 50 });

Revisions

Each episode that changes the model pushes a new commit here, tagged after the episode, such as ep002-pretrain for the pretrained model and ep003-sft for the fine-tuned one, and ep012-pretrain for a model of 91 million parameters, pretrained on four GPUs. revision= loads a tag, and main is the latest.

Limitations

• It writes children's stories, and nothing else: no facts, arithmetic or chat.
• Its context window is 512 tokens.
• Any text can be encoded, and decoding gives it back as it was.