jasonjimnz/tyness-boe-2.6b-pre-alpha-2-instruct

Tyness BOE 2.6B Instruct — Pre-Alpha 2 (teacher–student + adversarial replay)

Fifth pre-alpha from Tyness Train — and our best model: 0.704 focused / 0.718 full-precept / 0.181 closed-book on the EVAL-SPEC v1.0.0 bank, up from 0.640/0.724/0.179 (pre-alpha-1). Built by combining two data signals that had only been tested separately: teacher–student multi-turn dialogues (which improved ficha, content, synthesis and closed-book but eroded defenses in isolation) and fresh adversarial replay (which restores them). Both together win.

⚠️ Status: PRE-ALPHA. Grounded reading is strong (0.70–0.72 with context; deepseek-v4-flash scores 0.99 on the same items), but parametric knowledge is weak by design (0.181 closed-book) and sycophancy resistance, while improved, is not solved. Always verify against boe.es.

What improved vs pre-alpha-1 (run24, same 102-item comparison)

Metric	pre-alpha-1	pre-alpha-2 (this)
focused (mean)	0.640	0.704 (+6.4)
content (focused)	0.64	0.70
multi-precept synthesis (focused)	0.00	0.50
identifier-trap rejection (focused)	0.86 → 0.54 with TS-only	0.67 (replay restores)
negative "no-consta" (focused)	0.63 → 0.47 with TS-only	0.72 (restored above baseline)
temporal (n=10 items)	0.70	0.81
full-precept mode	0.724	0.718 (flat, within noise)
closed-book	0.179	0.181 (flat)

Zero empty answers (a generation bug in pre-alpha-1 — answers trapped inside the <think> block on hard items — was diagnosed and fixed in the evaluation harness).

Model description

Base model	LiquidAI/LFM2.5-2.6B (2.69B params, hybrid: 22 conv + 8 GQA layers)
Task	Context-grounded Spanish law Q&A with exact official citations (BOE-A-AAAA-NNNNN)
Method	QLoRA SFT (TRL, NF4, fp32 compute) — focused relevance-ORDERED contexts + METADATOS system-prompt block + adversarial correction pairs (original + fresh quote-to-cite) + teacher–student multi-turn dialogues + empty-<think> targets
Hardware	NVIDIA GTX 1060 6GB (Pascal, sm_61) — 2016 consumer GPU
Training	5 epochs · seq 2048 · 440 steps · ~14.5h · final loss 0.752 · token accuracy 84.9%
Author	jasonjimnz

Training data and methodology

Dataset boe-focused-adv-v4 (1,397 entries) — three signals, one mix:

1. Base recipe (1,297 entries, byte-identical to our best 1.2B release): grounded entries over 10 BOE articles with relevance-ORDERED contexts (never relevance-filtered — measured at n=200), a METADATOS block in the system prompt (identifier, official number, rank, department, dates, status, article-title index), and 149 adversarial correction pairs (~11.5%): the user asserts a false premise, the target answer corrects it with the exact citation.
2. Teacher–student multi-turn dialogues (30 dialogues, 201 turns): progressive comprehension→application conversations per norm, including 11 temporal dialogues with step-by-step date computation (entry into force, transitional provisions). Every teacher turn gate-checked for literal grounding in the norm.
3. Fresh adversarial replay (70 quote-to-cite pairs): new adversarial entries whose targets must include a literal «quote» of the correct precept, each quote verified against the norm text. This replay is what restores identifier-trap and refusal behavior when teacher–student data is added — the two signals were only tested separately before (pre-alpha-1 without TS; an intermediate run with TS-only lost 0.32 on traps and 0.16 on refusals).

Empty-<think> targets: the LFM2.5-2.6B chat template forces a <think> tag at generation start; targets were prefixed with an empty <think>\n\n</think>\n\n block so the trained token stream matches inference (without this, training/inference mismatch silently corrupts the model).

Loss on assistant tokens only; evaluation items are never in any training split. Sources: consolidated Spanish legislation (BOE open data) via legalize-es, teacher-generated ChatML entries.

Training configuration

Parameter	Value
LoRA r / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj
Learning rate / scheduler	2e-4 / cosine
Epochs	5 (440 optimizer steps)
Effective batch	1 × 16
Sequence length	2048
Quantization	NF4 4-bit, fp32 compute (Pascal: never bf16/fp16)
Optimizer	paged_adamw_32bit

Evaluation results

Bank: es-eval-v1.1-pilot10 (EVAL-SPEC v1.0.0 + expanded temporal archetype, 107 items × 3 context modes). Questions generated by a teacher LLM from the norm text only, deterministic quality gates, same items for every system. Judge: glm5.3-flash.

Mode	pre-alpha-1 (run24)	pre-alpha-2 (this)	deepseek-v4-flash (ceiling ref.)
closed-book (no context)	0.179	0.181	0.466
focused (production recipe)	0.640	0.704	0.989
full precept	0.724	0.718	0.986

Per-archetype (focused): ficha 0.77 · content 0.70 · applied case 0.53 · multi-precept 0.50 · negative 0.72 · identifier-trap 0.67 · temporal 0.81.

Closed-book stays at 0.18 — parametric knowledge is weak by design: on this rig (1.2B/2.6B + QLoRA) knowledge injection empirically fails and erodes other skills, so the architecture is hybrid by decision: this model is the reader; the corpus lives in retrieval.

Score progression across the project (definitive FINAL): 0.005 → 0.035 → 0.267 → 0.3475 → 0.415 → 0.446 (1.2B best) → 0.480 (2.6B) → focused 0.704 (this, new bank).

Full details: eval-report.md.

What works: grounded reading of provided precepts, applied-case reasoning, multi-precept synthesis (0.50), identifier-trap rejection with quote-based correction, refusal calibration (restored), legal register. What fails: sycophancy under strong adversarial pressure (0.67, ceiling 1.00), parametric knowledge (closed-book 0.181), temporal/date arithmetic partially (0.81 with context).

How to run

llama.cpp (all quantizations included: F16 / Q8_0 / Q4_K_M)

llama-server -m tyness-boe-2.6b-pre-alpha-2-instruct-q4_k_m.gguf -ngl 99 -c 16384 --port 8080 --jinja
curl http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [
    {"role":"system","content":"METADATOS: identificador BOE-A-2019-18746, Orden HAC/1270/2019, ...\\n\\nEres un jurista experto en el BOE. Responde ÚNICAMENTE con el CONTEXTO. Si un dato no consta: No consta en el contexto proporcionado. Si el usuario afirma algo incorrecto, corrígelo citando el precepto exacto."},
    {"role":"user","content":"CONTEXTO:\n<paste relevant norm paragraphs here>\n\nPREGUNTA: ¿…?"}
  ],
  "temperature": 0
}'

Transformers (note: thinking-capable base)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("jasonjimnz/tyness-boe-2.6b-pre-alpha-2-instruct", dtype="auto")
tok = AutoTokenizer.from_pretrained("jasonjimnz/tyness-boe-2.6b-pre-alpha-2-instruct")
# This model was SFT'd with empty <think> blocks: it opens and immediately closes the
# thinking section, then answers directly from the provided context.
messages = [{"role": "user", "content": "CONTEXTO:\n...\n\nPREGUNTA: ¿…?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

Limitations

• Specialized in the 10-norm trained scope; not a general legal advisor.
• Sycophancy: improved (0.67) but not solved — verify corrections against boe.es.
• Parametric knowledge is weak by design (closed-book 0.181): always ground it with CONTEXTO + METADATOS.
• Temporal/date questions improved (0.81) but date arithmetic can still err.

Generated by Tyness Train on 2026-10-20. Base model © LiquidAI, LFM Open License v1.0.