ShitpostGPT-10M

A 10M-parameter language model trained from scratch on one consumer GPU (RTX 3060 12GB) in about 33 minutes, to write Twitch chat, imageboard posts and Reddit shitposts. It is a toy: tiny, unfiltered and chaotic on purpose. Inspired by Codeically's "ChatGPT-2 on a potato" video.

This is the first, smaller model. Its bigger sibling, ShitpostGPT-20M, keeps writing for several lines and is the better one to use. This 10M version is mostly interesting as a "how small can it go" experiment.

It is a text-continuation model, not an assistant. It does not follow instructions or answer questions. It continues whatever you start, and it tends to stop early: most of its training text was short single lines, so it learned to end after a few words.

Quick start

from transformers import LlamaForCausalLM
import sentencepiece as spm, torch

m = LlamaForCausalLM.from_pretrained(".").eval()          # folder with the files of this repo
sp = spm.SentencePieceProcessor(model_file="tokenizer.model")

ids = [1] + sp.encode("PepeLaugh")                         # 1 = BOS, always prepend it
out = m.generate(torch.tensor([ids]), max_new_tokens=60, do_sample=True, temperature=0.9,
                 top_k=50, repetition_penalty=1.1, eos_token_id=2, pad_token_id=0)
print(sp.decode(out[0].tolist()[1:]))

Tips:

• Temperature 0.9 to 1.1 with top-k 50 to 100 works best.
• Add min_new_tokens=15 to generate to block the early stop and get longer output.
• Start with an emote or a short chat line (PepeLaugh, KEKW) or >be me to pick the style.

Example outputs from the training run (prompt, then what the model added):

hey            ->  hey mom 🙏 daddy
wtf            ->  wtf the game
how are you    ->  how are you the best

Short and silly, which is about what a 10M model can do.

Model details

Architecture	Llama (RoPE, RMSNorm, SwiGLU, tied embeddings)
Parameters	10.06M
Layers / hidden size / heads / FFN	6 / 320 / 5 / 864
Context length	1024
Tokenizer	SentencePiece BPE, 8,192 vocab, byte fallback, trained on the corpus; BOS = 1, EOS = 2
Training tokens	198M (one pass over the corpus)
Final validation loss	3.353 (perplexity 28.6)
Hardware / time	1x RTX 3060 12GB, about 33 minutes, about 100k tokens/s
Optimizer	AdamW, lr 1.5e-3 with cosine decay to 1.5e-4, bf16, 32k tokens per step, 6,104 steps

The validation loss is not comparable with the 20M model's, because the two use different vocab sizes and different training mixes.

Training data

About 620 MB of text (about 187M tokens) from public sources:

• Clean English web text: HuggingFaceFW/fineweb-edu (sample-10BT), about 200 MB
• Twitch chat: lparkourer10/twitch_chat, about 82 MB
• Imageboard posts: simonko912/chan-shitpost-2.5, about 200 MB
• Reddit shitposting comments: innovation-hacking2/shitposting_dataset0 to 5
• Greentext: DarwinAnim8or/greentext
• A small personal scrape of Twitch chat and Reddit comments (about 17 MB)

Cleaning: HTML unescaped, URLs and >> reply references removed, whitespace normalised, length filters, exact-duplicate removal. No toxicity filtering was applied.

Limitations

• It is tiny. Expect fluent-looking fragments with no real meaning, facts or reasoning.
• Replies are very short and often just echo your prompt, because the training lines were short and each ended with an end-of-text token.
• Output drifts and repeats.
• English only.

Content warning and intended use

The training data includes raw imageboard and Reddit text, so the model will produce slurs, sexual content, harassment and other offensive text, sometimes without being prompted. It is meant for entertainment, experimentation and learning how small language models are trained. Do not deploy it to users, use it for moderation or decisions, or treat anything it says as true.

License

The weights come with no warranty. The training data comes from several third-party datasets with their own licenses (and in places unclear ones), so check those terms before redistributing or using the model commercially.