TinyBrainBot 350M V3 — Instruct
A 348M-parameter decoder-only model, pretrained from scratch on 22.7B tokens and then instruction-tuned. Answers direct questions in a chat format. Successor to the TinyBrainBot 100M V3 Instruct.
TL;DR: Beats the 100M V3 Instruct on 4 of 7 benchmarks and on average (41.8 vs 41.3), with its biggest gains on HellaSwag (+3.1) and OpenBookQA (+2.4). It answers short factual questions well and explains concepts at a reasonable level for its size. For anything numeric, use the math variant.
Model details
| Parameters | 348,342,912 (~348M) |
| Architecture | Decoder-only, pre-norm, RMSNorm, SwiGLU, RoPE, GQA (18 Q / 6 KV heads) |
| Hidden size / layers | 1152 / 22 |
| Context length | 2048 (SFT ran at 1024) |
| Vocabulary | 32,000 (tbb-32k-v2, tied embeddings) |
| Base model | TinyBrainBot 350M V3 Base |
| SFT | 2,000 steps, lr 2e-5 → 0, WSD, 131M tokens, assistant chat format |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
m = "nkthebass/tinybrainbot-350mV3-instruct"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m)
msgs = [{"role": "user", "content": "What is the capital of France?"}]
p = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(p, return_tensors="pt"), max_new_tokens=150,
do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))
Use apply_chat_template — do not hand-build the prompt. This family's template is
space-separated, not newline-separated:
<|user|> What is the capital of France? <|end|> <|assistant|>
(note the leading space). Building it with newlines instead degrades output badly — the model
emits fragments or repeated tokens. The same applies to the GGUF, which carries this template
along with add_space_prefix=false; both halves are required.
Evaluation
EleutherAI lm-eval-harness, 0-shot, complete test sets — same harness as the published
100M V3 numbers. Compared instruct-to-instruct, not against a base model.
| Benchmark | 350M V3 Instruct | 100M V3 Instruct | Supra2-100M-Instruct |
|---|---|---|---|
| ARC-Easy | 50.9 | 53.7 | 44.4 |
| ARC-Challenge | 29.7 | 29.1 | 24.7 |
| HellaSwag | 35.9 | 32.8 | 35.9 |
| OpenBookQA | 33.8 | 31.4 | 30.4 |
| PIQA | 66.6 | 65.1 | 64.4 |
| WinoGrande | 51.2 | 51.3 | 50.5 |
| MMLU | 24.4 | 25.8 | 25.8 |
| Average | 41.8 | 41.3 | 40.8 |
→ 4/7 vs the 100M V3 Instruct.
Note on instruct vs base scores. Instruction tuning lowers multiple-choice benchmark scores for this family — the 350M V3 base averages 43.2 and scores 56.6 on ARC-Easy, versus 41.8 and 50.9 here. That is the normal cost of making a model usable, and it is why the base and instruct cards report separate tables. Do not compare this model against a base model's numbers.
SFT data
12 sources, 131M tokens: smoltalk 24%, magpie-pro-300k 20%, openhermes-clean 16%,
qa-distill 12%, reasoning-distill-chat 10%, longdef-sft 6%, plus small amounts of
math, reasoning and conversational-recall data.
Sample exchanges
All generated from this repo with temperature=0.7, top_p=0.9, unedited.
USER: What is the capital of France?
BOT : Paris is the capital of France.
USER: What causes tides?
BOT : Tides are caused by the gravitational attraction between the Earth and the Moon.
The Moon orbits the Earth, and when it is closest to Earth, its gravitational pull
is strongest, resulting in a higher high tide, while when it is farthest away, its
pull is weaker, resulting in a lower high tide...
USER: Give me three tips for studying effectively.
BOT : Here are three effective study tips:
1. **Set clear goals**: Identify what you want to achieve, such as improving your
grades or mastering a particular subject.
2. **Take regular breaks**: Regularly take breaks from your studying schedule to
avoid burnout and maintain your focus.
3. **Use active learning techniques**: ...
USER: Can you explain photosynthesis simply?
BOT : Photosynthesis is a process that converts light energy into chemical energy, storing
energy in the form of ATP and NADPH. In photosynthesis, light energy is used to
combine carbon dioxide and water into glucose and oxygen. This process occurs in
three stages: light absorption, energy conversion, and carbon fixation...
And where it falls down — the same run, same settings:
USER: Who was Marie Curie?
BOT : Marie Curie was a French mathematician and physicist who made significant
contributions to physics, mathematics, and medicine.
(Polish-French, and a chemist rather than a mathematician)
USER: What's the difference between a virus and a bacterium?
BOT : ...Viruses are composed of genetic material, such as DNA, and contain genetic
material, called genetic material, which can be replicated...
(correct direction, then degenerates into tautology)
USER: What is the Great Barrier Reef?
BOT : A remarkable underwater landmark that spans over 4,000 kilometers in length and
2,000 kilometers in width.
(roughly 2,300 km long; the width is wildly wrong)
The pattern is consistent: the shape of the answer is usually right and the specific details often are not. It reaches for the correct concept — gravity for tides, chlorophyll for autumn leaves, ATP/NADPH for photosynthesis — and then invents numbers, nationalities and qualifiers. Treat it as a model that knows the outline of things at 348M, not a reference.
Limitations
- Not a calculator. It will attempt arithmetic and often gets small sums right, but it does not reliably stop once it has the answer — it keeps generating and can talk itself into a wrong one. Use the math variant for anything numeric.
- Weak on open-ended conversation. It answers direct questions; it does not sustain multi-turn chat well, tends to restate the user's statement, and loses the thread across topic shifts.
- Hallucinates confidently on facts outside its training distribution.
- MMLU and ARC-Easy lag the 100M V3 Instruct.
- Trained almost entirely on English.
Related
- TinyBrainBot 350M V3 Base — the pretrained model, and the frozen-RMSNorm story behind it.
- TinyBrainBot 350M V3 Math — for arithmetic and worked solutions.
- Downloads last month
- 463