tetracta/tetracta-van-1b

Tetracta VAN-1B

A 0.94B-parameter decoder-only transformer trained from scratch on 10.32B tokens — one epoch, no data repetition — then instruction-tuned on 11,419 examples.

This model is a data-efficiency demonstration and a measurement instrument, not a production assistant. Its value is that we kept the raw pre-trained state, the SFT data, and every intermediate checkpoint — which let us measure what instruction tuning actually does to a model. Those measurements are in the paper below.

• 🌐 tetracta.ai — Tetracta, independent AI research lab
• 📄 Paper: "What does SFT do? Measuring a 0.94B model from the inside and the outside" → tetracta.ai
• 🐦 @tetractai · 💼 LinkedIn
• 🧪 Sibling: Tetracta-zkas-1B — identical data, identical SFT, a different (non-transformer) architecture. Results published, weights not.

---

What is honest about this model

Read this first. We publish our losses as loudly as our wins.

Blind judge panel — 64 prompts, 8 lanes, 24 independent judges, 192 decisions:

Opponent	Our share	Verdict
Qwen2.5-1.5B-BASE	48.4 %	parity — with ~1,750× less training data
Qwen2.5-1.5B-Instruct	22.6 %	we lose clearly
Qwen2.5-7B-Instruct	15.3 %	we lose clearly

Where we win, and where we don't:

We win (vs same-size BASE)	We lose (vs everyone)
identity 8-0 · honesty 7-1 · chat 7-1 · safety 5-2	knowledge · reasoning · long-form writing

That split is the paper's thesis: behaviour alignment can be taught with very little data; knowledge and reasoning cannot — those need scale. We won exactly the lanes SFT teaches, and lost exactly the lanes pre-training scale gives.

---

Model details

Architecture	Decoder-only transformer — deliberately vanilla, nothing proprietary here
Parameters	936,609,792 · d_model 2048 · 16 layers · 16 heads · head_dim 128
Components	RMSNorm (pre-norm) · RoPE · causal SDPA · GELU-MLP (4×) · tied embeddings
Context	8192 (pre-training) · 1024 (SFT, and the shipped RoPE cache)
Vocabulary	64,000 SentencePiece — chat/tool tokens are built in (no vocab resize needed for SFT)
Pre-training	10,321,920,000 tokens = 10.32B · ONE FULL EPOCH, no repetition · corpus 100 % English
Pre-training recipe	315,000 steps · 32,768 tokens/step · lr 3e-4 · 2000 warmup · cosine · seed 1 · fp32 master / bf16 autocast
Hardware	1×H200 · 678.5 ms/step · ~64 h · ~$270–290 total
Final pre-training	bpc 3.7610 → char-PPL 13.56
Instruction tuning	11,419 examples · 3 epochs · lr 1e-5 · batch 16 · T=1024 · 236 s on 1×H200

bpc = bits per character on a held-out slice never seen during training; PPL = 2^bpc.

Files

File	What it is
model.safetensors	Instruction-tuned model — use this one
van1b_base.safetensors	Raw pre-trained base (bpc 3.7610) — for running your own SFT
tokenizer.model	SentencePiece, 64k
modeling_van.py	Reference implementation + chat template + sampler
tools.py	Tool runtime — actually executes the model's tool calls
config.json	Full config and training metadata

Embeddings are tied: head.weight is intentionally absent from the safetensors files and re-tied on load.

---

Usage

pip install -r requirements.txt
python modeling_van.py --prompt "Write two sentences about the ocean."

from modeling_van import load, load_tokenizer, chat

sp = load_tokenizer()
model = load(device="cuda")                      # or "cpu"
print(chat(model, sp, "What is photosynthesis?"))

Chat template

The special tokens are native to the tokenizer, so the template is just token ids:

<s> [<|system|>…<|end|>] (<|user|>…<|end|> <|assistant|>…<|end|>)* <|user|>…<|end|> <|assistant|>

token	id	token	id
<s> (BOS)	1	<|end|>	7
</s> (EOS)	2	<|tool_call|>	13
<|system|>	4	<|tool_result|>	14
<|user|>	5	<|thought|>	17
<|assistant|>	6

build_prompt() in modeling_van.py builds this for you, multi-turn history included.

Tool use

The model emits <|tool_call|>{"name": "...", "arguments": {...}}. tools.py executes it and feeds the result back as <|tool_result|>:

python tools.py --ask "How many feet is 250 meters?"
#   [tool] unit_convert({"value": "250", "from": "meters", "to": "feet"}) -> {"result": 820.21, "unit": "ft"}

Built-in tools: calculate (safe AST arithmetic — no eval), web_search (DuckDuckGo), unit_convert, get_time.

Measured: 4/6 correct tool calls. The mechanism works — but read limitation 3 before relying on it.

---

Limitations — measured, not guessed

1. Reasoning and arithmetic are not reliable. Multi-step problems collapse.
2. Factual accuracy is weak. It fabricates on rare or detailed facts.
3. It corrupts digits when copying a tool result into its answer (tool returns 2726784 → model writes 2762784; it sometimes leaks raw JSON into the sentence). Render tool results programmatically instead of trusting the model to repeat them.
4. Without the tool runtime attached it may invent live data ("the weather in Paris is…").
5. It is inconsistent about its own abilities — it sometimes claims internet access. For calibration: Qwen2.5-1.5B-Instruct made the same error in our panel, answering "Yes, I can browse the internet." 1,750× more training data did not fix self-knowledge.
6. English only. There is no Turkish in the pre-training corpus.
7. Long-form writing is clearly behind (writing lane 0-8 — not one judge picked us).
8. Our judges are models, not humans.

What SFT cost, measured

Instruction tuning moved only 0.68 % of the weights — and overwhelmingly in the embedding layer (5.35 %), leaving normalisation layers essentially untouched (0.04 %). What it actually did was collapse output entropy from 5.951 to 1.795 (peak probability 0.279 → 0.730). The price: held-out plain-text loss rose 11.5 % (bpc 4.340 → 4.839). Almost the entire tax was paid in the first SFT round; a second, targeted round added only ~1 % more while fixing safety, identity and pronoun resolution.

Method and full numbers in the paper.

---

Intended use

Reasonable: research on data efficiency and instruction tuning · a clean, fully documented base for your own SFT experiments · tool routing and intent classification inside a program that validates the output · teaching, since every artifact is published.

Not reasonable: anything where a wrong fact or a wrong number matters. Do not run it unsupervised.

Citation

@misc{tetracta_van1b_2026,
  title  = {Tetracta VAN-1B: What does SFT do? Measuring a 0.94B model from the inside and the outside},
  author = {Tetracta},
  year   = {2026},
  url    = {https://tetracta.ai}
}

Training data provenance

Fully documented, and deliberately boring — we chose reproducible public corpora so the comparison would be checkable, and excluded synthetic data on purpose (it can carry hallucinations of its own, which would have contaminated exactly what we set out to measure).

Source	Share	Dataset	License
FineWeb-Edu	~66 %	HuggingFaceFW/fineweb-edu (sample-100BT)	ODC-By 1.0
Wikipedia (English)	~34 %	wikimedia/wikipedia (20231101.en)	CC-BY-SA 4.0
Synthetic data	0 %	—	excluded by design

~12.5B raw tokens → filtering and MinHash dedup → 10.32B trained tokens, one epoch. Attribution to both sources is required by their licenses and gladly given.

Instruction-tuning data is not published with this model.

License

Apache-2.0 for the weights, code and model card in this repository.

Chosen deliberately: this model exists to be examined and built on, not to be gated. Apache-2.0 also carries an explicit "AS IS", no-warranty and limitation-of-liability clause — which matters for a model whose measured failure modes (unreliable arithmetic, digit corruption when copying tool results, weak factual recall) are listed above in full. Use it accordingly.

If you build on it, a link back to tetracta.ai is appreciated but not required.