nkthebass/TinyBrainBot-demo-216.5m

TinyBrainBot — demo (216.5M)

A 216.5M-parameter conversational language model trained from scratch on a custom decoder-only architecture. This is an early proof-of-concept / demo checkpoint: the goal is to show that a small model with a proper subword tokenizer can hold coherent, on-topic chat — not to compete with production models.

Trained end-to-end (tokenizer + architecture + pretraining + SFT) as a solo project on consumer hardware.

Model details

Parameters	216.5M
Architecture	Decoder-only transformer: RoPE + RMSNorm (pre-norm) + SwiGLU, tied embeddings
Layers	10
Hidden size	1032
Attention heads	12 (head_dim 86)
FFN size	4416
Context length	768
Vocabulary	36,000 (SentencePiece unigram, case-preserving, with chat/memory special tokens)
RoPE theta	10,000
Training tokens	~551M (this checkpoint)
Precision	this file is F16 GGUF

Note: ~551M training tokens is well under compute-optimal for this size, so this is an early, undertrained checkpoint. Expect coherent short replies but factual drift and looping on long generations. More training is planned.

How to run

LM Studio

Load the GGUF, then set sampling to:

Setting	Value
Temperature	0.70
Top P	0.90
Repeat penalty	1.15
Frequency penalty	0.5
Presence penalty	0.3

The chat template is embedded in the GGUF, so no manual prompt-format setup is needed.

llama.cpp

llama-cli -m tinybrainbot-216.5m-F16.gguf --jinja \
  --temp 0.70 --top-p 0.90 --repeat-penalty 1.15 \
  --frequency-penalty 0.5 --presence-penalty 0.3

Chat format

The model uses these special tokens (single IDs in the tokenizer):

<|user|>
{your message}
<|end|>
<|assistant|>

Generation stops at <|end|>.

The GGUF tokenizer is exported as UGM (unigram) so llama.cpp reproduces the training-time SentencePiece tokenization exactly — an SPM export would re-segment words and degrade output.

Intended use

• Demonstrations and education about small-model training/inference
• Research into tokenizers, architectures, and from-scratch training on modest hardware
• A base for further training / fine-tuning experiments

Not intended for factual question answering, production use, or any high-stakes application.

Limitations

• Small + undertrained: answers are often factually wrong and can wander off-topic on long outputs.
• Short context (768 tokens).
• English-only.
• Greets and opens coherently; degrades the longer it generates (mitigate with the frequency/presence penalties above).

Training data

Pretraining on public English text (e.g. Wikipedia, TinyStories, OpenWebText2) and SFT on public chat/instruction datasets. Respect the upstream licenses of those datasets for your own use.