nkthebass/tinybrainbot-350mV3-math

TinyBrainBot 350M V3 — Math

A 348M-parameter decoder-only model fine-tuned to do column arithmetic with a written-out scratchpad. Successor to the TinyBrainBot 320M V2 Math.

TL;DR: 99.97% average across the nine GPT-3 arithmetic sub-tasks, beating the 320M V2 Math on all 9 of 9 (91.5% average). It adds and subtracts up to 5 digits essentially perfectly and does 2-digit multiplication. It is a calculator with a scratchpad, not a word-problem solver — see the honest limits below.

---

Model details

Parameters	348,342,912 (~348M)
Architecture	Decoder-only, pre-norm, RMSNorm, SwiGLU, RoPE, GQA (18 Q / 6 KV heads)
Hidden size / layers	1152 / 22
Context length	2048
Vocabulary	32,000 (tbb-32k-v2, tied embeddings)
Lineage	350M V3 Base → Instruct → 10 rounds of math SFT → 2 merged LoRA adapters

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

m = "nkthebass/tinybrainbot-350mV3-math"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m)

msgs = [{"role": "user", "content": "What is 4827 plus 3956?"}]
p = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(p, return_tensors="pt"), max_new_tokens=200, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Use apply_chat_template rather than hand-building the prompt — this family's template is space-separated, not newline-separated, and getting it wrong degrades output badly.

Greedy decoding (do_sample=False) is strongly recommended. The scratchpad is a deterministic routine; sampling introduces digit errors mid-column.

Evaluation — arithmetic

GPT-3 arithmetic protocol, zero-shot, n=300 per sub-task, greedy, exact match on the final number. Identical problems and seed for both models.

Sub-task	350M V3 Math	320M V2 Math
2-digit addition	100.0	83.0
3-digit addition	100.0	97.7
4-digit addition	100.0	98.7
5-digit addition	100.0	91.7
2-digit subtraction	100.0	95.3
3-digit subtraction	100.0	91.0
4-digit subtraction	100.0	94.7
5-digit subtraction	99.7	92.0
2-digit multiplication	100.0	79.3
Average	99.97	91.5

→ 9/9 sub-tasks improved.

Important: these operands are ordered, so no answer is negative

Both models are measured with the same harness, and that harness orders subtraction operands (if b > a: a, b = b, a) so every answer is non-negative. The GPT-3 protocol as originally defined samples operands independently and permits negative results, so this table is not directly comparable to published GPT-3 numbers. The figure above is 99.97% on non-negative subtraction.

Asked for a negative result the model does not refuse or error - it runs the column routine anyway and returns a confident, well-formatted, wrong positive number:

USER: What is 32 minus 83?
BOT : Subtracting. ones: 12 - 3 = 9, borrow 1. tens: 2 - 8 = 2, borrow 1. The answer is 29.

USER: What is 100 minus 250?
BOT : Subtracting. ones: 0 - 0 = 0. tens: 10 - 5 = 5, borrow 1. hundreds: 8 - 2 = 6.
      The answer is 650.          <- the "8" appears nowhere in the problem

2 - 8 = 2 is the routine running off the end of its own definition. Independent testing puts accuracy at 5-16% when b > a, varying by width. This is a data-coverage gap, not a capacity limit: every training generator in the project ordered its operands, so the model has never seen a negative result and the scratchpad has no sign-decision step to represent one. Order your operands, or check the sign yourself.

These are in-distribution numbers: the training data was generated to this exact question form. The section below is the part that matters for judging whether anything general was learned.

Does it generalize, or did it memorize?

The 99.97% above is in-distribution by construction — the training data was generated to that exact question form. These three probes are the honest test. All are outside the training distribution, and the phrasings below were deliberately excluded from the 30 question templates the model was trained on, so they cannot be re-fitting.

Probe	Before phrasing LoRA	Final	Reading
Held-out phrasings "How much is 412 plus 87?", "Sum 63 and 29.", "Deduct 18 from 62.", "What remains when 4794 is taken from 7572?"	33.3%	80.0%	Question parsing was the bottleneck, not arithmetic. A 60k-example phrasing adapter moved it 47 points without touching the routine.
7-digit addition one column wider than anything in training (capped at 6)	100% (n=24)	~98% (n=60)	The column routine is a real algorithm and extends past the longest operand it ever saw. Independent testing at n=60 found one miss - a dropped carry deep in the chain - so this is ~98%, not perfect. No length wall, but not flawless either.
3×3 multiplication multiplier never wider than 2 digits	8.3% (n=24)	~5% (n=60)	The failures are all near misses (375 × 903 → 338725 vs 338625). The training data contains zero examples with a 3-digit multiplier, so this is extrapolation to an unseen width, not memorisation. With two partial products the model works their sum column by column and is correct; with three it asserts the sum instead.

The honest summary: addition and subtraction learned a genuine, length-generalizing column algorithm. Multiplication learned one that degrades as products widen. And a large part of what looked like "bad at math" was actually "did not recognise the question."

Evaluation — word problems

Straight arithmetic is what this model was built for; word problems are the harder, independent test. None of these datasets appear in any TinyBrainBot training set.

Five independent, human-written sets. Zero-shot, greedy, exact match on the final number.

Benchmark	350M V3 Math	n	Structure
ASDiv	13.5	200	mostly single-step
MultiArith	8.9	180	2–3 operations
SVAMP	6.0	200	one-step, adversarially perturbed
GSM8K	4.0	200	multi-step, free-form
MAWPS	4.0	200	one/two-step

The spread tracks problem structure, and that is the most honest thing on this card: the model converts one sentence into one operation reasonably often (ASDiv 13.5), and essentially cannot chain operations (GSM8K/MAWPS 4.0). GSM8K improved 1.0 → 3.0 → 4.0 across the two adapters, so the trend is real, but single digits at 348M is the honest expectation — the 320M V2 Math card reports ~3–4%.

The 99.97% and the 4% are not a contradiction

They measure two different mechanisms. The column routine fires on the arithmetic question form; inside a word problem the model falls back to mental arithmetic, which is not reliable. An actual generation from the GSM8K run:

<think> The total ratio representing their ages was 7+11=181 ... </think>

7 + 11 = 181 — concatenation instead of addition, from a model that does 5-digit column subtraction at 99.7%. Treat the arithmetic score as a measure of a routine, not of numeracy. If you need a correct sum, ask for it directly.

Sample output

Verbatim from the F16 GGUF in this repo, greedy (--temp 0), unedited:

USER: What is 4827 plus 3956?
BOT : Adding. ones: 7 + 6 = 13, write 3 carry 1. tens: 2 + 5 + 1 (carry) = 8, write 8.
      hundreds: 8 + 9 = 17, write 7 carry 1. thousands: 4 + 3 + 1 (carry) = 8, write 8.
      The answer is 8783.

USER: What is 90210 minus 4877?
BOT : Subtracting. Pad 4877 to 04877. ones: 10 - 7 = 3, borrow 1. tens: 10 - 7 = 3, borrow 1.
      hundreds: 11 - 8 = 3, borrow 1. thousands: 9 - 4 = 5. ten-thousands: 8 - 0 = 8.
      The answer is 85333.

USER: What is 47 times 38?
BOT : Multiplying. Break 38 into digits. 47 x 8 = 376. 47 x 3 x 10^1 = 1410.
      Add 376 + 1410. Adding. Pad 376 to 0376. ones: 6 + 0 = 6, write 6.
      tens: 7 + 1 = 8, write 8. hundreds: 3 + 4 = 7, write 7. thousands: 0 + 1 = 1, write 1.
      The answer is 1786.

Note the third one: rather than asserting the sum of the partial products, it runs the column addition routine on them. That nesting is deliberate - asserting the sum was a measured failure mode in earlier rounds.

Reasoning tags on word problems

For word problems (not straight arithmetic) the model emits a <think> ... </think> block before its answer. Measured over 40 GSM8K generations: 100% of blocks were opened and closed correctly, with the answer outside the block.

<think> 3 boxes with 24 pencils each: multiply 3 * 24. 3 * 24 = 72.</think> The answer is 72.

These are ordinary text, not special tokens. <think> is not in added_tokens.json, so skip_special_tokens=True will not remove it and it will appear in your output.

llama.cpp's reasoning extraction does not work with this model — tested on b10034, both --reasoning-format deepseek and -rea on leave reasoning_content empty and the tags in content. llama.cpp only engages that parser for chat-template formats it recognises, and this model's template falls back to generic. Strip the block yourself:

import re
answer = re.sub(r"<think>.*?</think>", "", raw, flags=re.S).strip()

Because closure is reliable, that one line is sufficient. Do not add <think>/</think> to the tokenizer as special tokens to try to fix this — the model emits them as a multi-token character sequence and has no probability mass on any new single token, so registering them would break generation rather than improve it.

Worked examples

All verbatim from the F16 GGUF in this repo, greedy (--temp 0), unedited. These are selected: 7 of 12 attempts were correct. See the failure mode below - the successes and failures split on a single, visible criterion.

Borrowing across a run of zeros:

USER: There were 15000 votes and 6842 were rejected. How many counted?
BOT : <think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:
      ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.
      hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,
      borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>
      The answer is 8158.

Multiplication, with the column-addition routine run on its own partial products:

USER: Each crate holds 24 bottles. How many bottles are in 15 crates?
BOT : <think> 24 bottles per crate: multiply 15 * 24. Multiply 15 * 24 using partial
      products. Split 24 into 20 and 4. 15 * 4 = 60. 15 * 20 = 300. Add 60 + 300:
      ones: 0 + 0 = 0. tens: 6 + 0 = 6. hundreds: 0 + 3 = 3. So 15 * 24 = 360.</think>
      The answer is 360.

Recognising "how many more" as subtraction:

USER: A shop sold 3672 items in May and 2145 in June. How many more in May?
BOT : <think> ... ones: 12 - 5 = 7, borrow 1. tens: 6 (after borrow) - 4 = 2.
      hundreds: 6 - 1 = 5. thousands: 3 - 2 = 1. So 3672 - 2145 = 1527.</think>
      The answer is 1527.

Addition with carry, and a non-math question showing the format generalises:

USER: A library had 8340 books and received 1275 more. How many books now?
BOT : <think> ... tens: 4 + 7 = 11, write 1 carry 1. hundreds: 3 + 2 + 1 (carry) = 6.
      thousands: 8 + 1 = 9. So 8340 + 1275 = 9615.</think> The answer is 9615.

USER: What is the capital of Japan?
BOT : <think> The capital of Japan is Tokyo.</think> The answer is Tokyo.

The failure mode is visible in the first few words

USER: A field is 340 metres long and 27 metres wide. What is its area?
BOT : <think> First, calculate the area of the field by multiplying its length by its width:
      340 m * 27 m = 1080 m2.</think> The answer is 1080.        WRONG (9180)

Every correct answer above says "multiply X * Y" or "subtract X - Y column by column" and then runs the routine. Every wrong one opens with "First, calculate..." and asserts a number in prose. Same model, same class of question - the difference is entirely whether the trace routes into the column algorithm or into free-form reasoning. If the trace does not show columns, do not trust the number.

An edge case worth knowing: addition stops at 8 digits

The training data only ever names six place values (ones ... hundred-thousands). The model learned the sequence and invented two more on its own - millions and ten-millions appear in no training example - so it adds 7- and 8-digit numbers correctly. At 9 digits it has no name for the ninth column, silently skips it, and jumps to the final carry:

USER: What is 483729164 plus 519248637?
BOT : ... hundred-thousands: 7 + 2 = 9, write 9. millions: 3 + 9 = 12, write 2 carry 1.
      ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1.
      The answer is 102977801.                    WRONG (1002977801)

Count the columns: eight, for a nine-digit problem. Every column it does compute is arithmetically perfect - the answer is wrong only because one column was never enumerated, so it comes out exactly one digit short. The limit is the place-name vocabulary, not the algorithm. Keep addition to 8 digits or fewer.

Training

Ten rounds of arithmetic SFT, each diagnosing a specific failure in the last. The two findings that did the work, both of which are about data, not scale:

1. Control the joint (operation × operand-length) distribution, not the marginals. Round 7 was balanced per-operation but chose lengths independently, so 2-digit problems were 2.1× more likely to be multiplication than addition. The model learned that "2 digits" means use the partial-product template and fired it on addition — 2-digit addition collapsed to 46.3% while 3-digit held 87.7%. Specifying every (op, length) cell explicitly fixed it.

2. Write the padding step down. Subtraction with ragged operands (90210 − 4877) failed until the scratchpad said Pad 4877 to 04877. out loud. Making the alignment an explicit token the model emits, rather than something it had to do silently, took 5-digit subtraction from 70% to 96%.

Two LoRA adapters (r=16, α=32, 2.02% trainable) were then merged in: one adding word-problem and reasoning data, one adding phrasing variety. LoRA was used specifically so the arithmetic routine could not be traded away for the new capability — and it worked better than intended: the word-problem adapter was expected to leave arithmetic flat, but it lifted the average from 94.0 to 99.9, repairing the exact four cells that ten full rounds kept trading against each other. The plausible reason is the constraint itself — at 2% trainable parameters it cannot reallocate capacity between operations, so it can only sharpen the routine already there.

Full recipe

Ten full SFT rounds, a continuous chain from the instruct checkpoint (step 31,196) to step 71,196 — 40,000 steps. Weights are percentages of the sampled mix.

Round	Steps	LR	Data mix
math2	+1,000	2e-5	reasoning-v2 40 · math-v3 26.7 · qa-distill 12.5 · smoltalk 12.5 · longdef-sft 7.5
math3	+4,000	2e-5	same mix, extended
math4	+2,000	1e-5	math-v3 30 · math-v3fix 25 · math-v2 15 · reasoning-v2 15 · smoltalk 10 · qa-distill 5
math5	+2,000	1e-5	same as math4
math6	+3,000	1.5e-5	math-short 60 · math-v3 12 · math-v3fix 10 · smoltalk 10 · qa-distill 5 · reasoning-v2 3
math7	+5,000	1.5e-5	math-short2 55 · math-short 20 · math-v3 8 · math-v3fix 7 · smoltalk 7 · qa-distill 3
math8	+5,000	1.5e-5	math-balanced 65 · math-short2 10 · math-v3 8 · math-v3fix 7 · smoltalk 7 · qa-distill 3
math9	+5,000	1.5e-5	math-balanced2 70 · math-balanced 8 · math-v3 7 · math-v3fix 6 · smoltalk 6 · qa-distill 3
math10	+5,000	1.5e-5	math-balanced3 72 · math-balanced2 8 · math-v3 6 · math-v3fix 5 · smoltalk 6 · qa-distill 3
math11	+8,000	1.5e-5	math-balanced4 80 · smoltalk 8 · math-v3 5 · math-v3fix 4 · qa-distill 3

Then two merged adapters, 1 epoch each:

Adapter	Rows	Composition	LR
word-problem	71,000	80.3% <think> synthetic word problems · 15.3% column scratchpad · 4.4% reasoning prose	1e-4
phrasing	84,000	55.0% column scratchpad (30 question templates) · 28.6% <think> word problems · 16.4% reasoning prose	8e-5

The math-balanced* generators are the ones that mattered; each fixed a specific measured defect in its predecessor.

Three notes for anyone reproducing this

• The predecessor's recipe did not transfer. Rounds math2/math3 ran the exact data mix that produced the 320M V2 Math (91.5% average). On this model it scored 58.5%. The 350M base scores 0–3% on arithmetic before any math SFT, where the 320M's pre-math model was already at 55–69% — a pretraining difference, not an SFT one. Everything from math6 onward exists because a proven recipe was reused first and measured to fail.
• Straight arithmetic was bought with word problems, then bought back. reasoning-v2 went 40% → 15% → 3% → absent while chasing arithmetic defects, and word-problem accuracy fell to roughly 1 in 4. The first LoRA exists to refill that hole without re-opening the arithmetic trade.
• General-purpose ballast never left. smoltalk + qa-distill held 9–11% in every round. Without it the model degenerates into a calculator that cannot hold a sentence.

Full write-up: [PROJECT_BRIEF.md in the training repo].

Limitations

• Negative results are broken. If the second operand is larger, the model returns a confident wrong positive number rather than a negative one. See the section above.
• Multiplication degrades as products widen. Reliable to 2-digit × 2-digit; 3×3 is ~5%. The failures are near misses (375 × 903 → 339625 vs 338625), so it is running the partial-product routine and losing digits summing it, not failing to attempt. No division.
• Word problems are weak (GSM8K 4.0%, best word-problem set ASDiv 13.5%). It converts one sentence into one operation sometimes; it does not chain operations.
• Greedy decoding matters. Sampling corrupts the column routine.
• Phrasing-sensitive. It is strongest on the direct "What is X plus Y?" form.
• Not a general assistant — for conversation and facts, use the Instruct variant.
• Trained almost entirely on English.

Related

• TinyBrainBot 350M V3 Base — the pretrained model.
• TinyBrainBot 350M V3 Instruct — general question answering.