Chimera-1 — two skills, one model, training-free merge (method demo)
Chimera-1 (chimera: in biology, one organism carrying two genetic lineages — which is exactly what this model is) is a proof-of-concept for a simple question: can you take two specialists and stitch their skills into ONE model — with no training at the merge step, and no loss of general ability?
Answer, measured: yes, under conditions we state below.
| model | skill A (Roman numerals) | skill B (letter ops) | Belebele-EN | perplexity | chat |
|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct (base) | 76% | 45% | 50% | 68.1 | ✓ |
| specialist A | 98% | 45% | — | — | — |
| specialist B | 80% | 75% | — | — | — |
| Chimera-1 (this model) | 89% | 71% | 50% | 68.1 | ✓ identical |
The headline: +13 and +26 points on the two target skills, and zero measured regression on general ability — Belebele-English, fixed-text perplexity and chat outputs are bit-for-bit indistinguishable from the base. The merge itself is training-free (TIES: task-vector trim + sign election + disjoint mean), runs on CPU in minutes, and the result is a single 3B model with no extra inference cost.
What this is — and is not
- It is a method demo. The two skills (Roman-numeral conversion, character-level word operations) are deliberately simple, verifiable testbeds — chosen because the base model is measurably weak at them (headroom). This is not a production assistant; it is evidence about merging.
- The specialists were made by us with light fine-tuning (top-layers only) — the merge is the training-free part.
- The law we validated across 10 experiments: merge synergy = headroom × complementarity × proximity. If the base already solves the task, there is nothing to gain. If the parents aren't genuinely complementary, merging dilutes. If a parent has drifted far from the shared base (heavy continued-pretraining), aggressive merging produces mush — we show a negative control where a merged coder model drops to 0% on code.
- What we did not manage (stated honestly): three attempts at X-Ray-guided merge surgery (per-layer and per-module weighting/refereeing from internal divergence maps) did not beat uniform TIES on average — one variant preserved the concentrated skill best (95%) at the cost of the other. Blind TIES is a strong baseline. What internal measurement did reliably do is predict which pairs merge profitably before merging (3/3 in our runs) and expose the trade-off dial.
Full head-to-head vs the base — 1,210 questions (22 Jul)
Same LL-based harness for both models. Sanity check: the base scores MMLU 67.0%, matching its publicly reported numbers — the harness measures correctly.
| benchmark | base | Chimera-1 | Δ |
|---|---|---|---|
| MMLU (200) | 67.0% | 66.5% | −0.5 |
| ARC-Challenge (200) | 81.5% | 81.5% | 0.0 |
| HellaSwag (150) | 70.0% | 71.3% | +1.3 |
| Belebele-EN (200) | 51.0% | 50.5% | −0.5 |
| Belebele-TR (75) | 41.3% | 41.3% | 0.0 |
| Belebele-AR (75) | 50.7% | 49.3% | −1.3 |
| Belebele-ZH (75) | 50.7% | 50.7% | 0.0 |
| Belebele-RU (75) | 49.3% | 49.3% | 0.0 |
| Roman numerals (80) | 82.5% | 91.2% | +8.8 |
| Letter ops (80) | 37.5% | 62.5% | +25.0 |
Every standard benchmark sits inside the ±2pp bf16 run-noise band (max |Δ| = 1.3pp); the two implanted skills moved +8.8 and +25.0. A capability edit with a measured blast radius of zero — verified on 1,050 standard questions, not assumed from a spot check.
Why we built it
We build an interpretability instrument (an "X-Ray" for LLMs) at tetracta.ai. Model merging is the perfect stress-test for it: everyone merges blind and evals after; we wanted to know how much of the outcome is predictable from the weights, before the merge. Full write-up, per-layer divergence films and the honest failure catalogue: tetracta.ai/research.html.
Use
Qwen2.5 ChatML format, identical to the base model. License inherited from Qwen2.5-3B-Instruct (Qwen Research License — non-commercial, research use).
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("tetracta/Chimera-1-Qwen2.5-3B", dtype="bfloat16")
t = AutoTokenizer.from_pretrained("tetracta/Chimera-1-Qwen2.5-3B")
# try: "Convert the Roman numeral CDXLIV to a regular number." / "Spell the word 'measure' backwards."
— Tetracta AI Teams · for humans, like humans. We measure; we publish the failures too.
- Downloads last month
- 52