srock44/cipher-nano

Cipher Nano

Cipher Nano is a full-parameter fine-tune of h2oai/h2o-danube3-500m-chat, trained on every LLM-backed feature of a local-first email assistant: email triage (importance/summary/category JSON), chat, daily-summary synthesis, draft reply, compose assist, and voice-command intent parsing — not just prompted for these tasks, actually trained on them.

Update: added a 6th task, voice-command intent parsing. Nano's smaller full-fine-tune capacity means it still sometimes substitutes a plausible-looking wrong address instead of copying the real one verbatim from the candidate list — 3 rounds of targeted oversampling narrowed but didn't eliminate this. The actual fix ended up being architectural, not a bigger model: the calling app never trusts an LLM-supplied email address at all anymore, re-deriving it deterministically in code from the model's extracted name against the real candidate list instead. That makes this habit irrelevant regardless of which Cipher tier is active — see generate_voice_intent.py's training notes and eval_voice_intent.py for the full detail.

Second update: an incremental continue-training round aimed at voice-intent (done on top of the update above) turned out to have a side effect elsewhere: manual testing found this model would occasionally fabricate a specific false detail not present in its given context (e.g. inventing a person's surname that was never mentioned) on the chat task. Rather than patch that narrowly on top of an already-long chain of incremental updates, this model was retrained fresh from the base in one pass on a properly-rebalanced dataset covering all 6 tasks together, with substantially richer prompt-injection coverage across draft-reply and chat (credential-phishing, wire-transfer, and data-exfiltration attempts, not just one generic case) and an explicit "the question has nothing to do with your email at all" scenario category to close the specific hallucination gap that was found. Verified through repeated testing against the real calling application's code path: no compliance with any injection attempt, and the specific hallucination case now correctly reports the missing information instead of inventing it.

It's the smallest of the three Cipher tiers (cipher-nano / cipher-air / cipher-pro), built specifically to run on modest hardware — verified at 29.4 tok/s on a 10-year-old Intel i3-6100T with 8GB RAM, no GPU, with accuracy holding essentially flat versus a high-end GPU. Cipher is the local-model engine for an unreleased larger email-assistant project — that project isn't public yet, but these weights, the training code, the eval script, and all five dataset generators are fully open now, in this repo.

Why this exists

Most email triage today means sending your inbox to a third-party API. Cipher runs entirely on your own hardware via Ollama — nothing about your email ever leaves your machine. Cipher Nano exists so that "your own hardware" can mean literally any hardware, not just a machine with a GPU.

Why Danube3-500M and not something smaller

We tried harder-to-shrink options first and hit real walls:

• SmolLM2-135M/360M (49K-token vocab) genuinely shrinks to 100-270MB, but its base pretraining wasn't strong enough to reliably learn category/importance calibration — tried LoRA vs. full fine-tune, epoch sweeps, and reshaped training data, and category accuracy stayed stuck in the 40-65% range regardless.
• Qwen2.5-0.5B / Gemma3 have strong pretraining but 152K/256K-token vocabularies that dominate disk size and don't shrink with quantization (confirmed: Q2_K barely shrank vs Q4_K_M).
• Danube3-500M is the first candidate with both a small vocabulary (32K tokens) and pretraining strong enough for the task — category accuracy jumped to 70%+ on the first fine-tuning attempt, no extensive tuning needed. Its full-parameter fine-tune (100% of weights retrained, vs. cipher-air/cipher-pro's LoRA) also gives it total plasticity to fully adopt the required output format, no competing base-model habit to fight against.

Benchmark

Evaluated on a 29-fixture triage benchmark across five machines, GPU and CPU-only:

Hardware	Tok/s	JSON-valid	Category acc	Importance-in-band
RTX 5070 (GPU)	901.0	100%	72.4%	69.0%
RX 9070 XT 16GB (GPU)	398.7	100%	69.0%	69.0%
Ryzen 9800X3D (CPU-only)	143.0	100%	72.4%	72.4%
Core Ultra 9 285K (CPU-only)	106.3	100%	69.0%	69.0%
Intel i3-6100T, 8GB RAM (CPU-only, 10 years old)	29.4	100%	69.0%	72.4%
Multi-task retrain (chat/summary/draft/compose added)	899.9	96.6%	67.9%	89.3%
Voice-intent retrain (6th task added)	867.4	96.6%	75.0%	82.1%
Full retrain (current version, injection/hallucination hardened)	106.7*	100.0%	82.8%	79.3%

* Tok/s measured while the GPU was concurrently running an unrelated training job — not a like-for-like comparison with the isolated numbers above; accuracy figures are unaffected.

Accuracy holds essentially flat across every machine tested — only throughput changes. Even the weakest machine here, a decade-old office PC with no GPU, does ~2 seconds per email, well within range for a background triage task. Reproduce with:

pip install -r requirements.txt
python eval_triage.py --models cipher-nano:latest --keep

Usage (Ollama)

Danube3's chat template does not support a system role and uses non-ChatML turn markers — the included Modelfile sets an explicit TEMPLATE to match. Don't rely on Ollama's template autodetection with this base model (confirmed live: without an explicit TEMPLATE, Ollama couldn't resolve a working chat format for this GGUF at all).

ollama create cipher-nano -f Modelfile

Query it with grammar-constrained JSON output for reliable parsing:

curl http://localhost:11434/api/chat -d '{
  "model": "cipher-nano",
  "messages": [
    {"role": "system", "content": "<system prompt from Modelfile>"},
    {"role": "user", "content": "From: alex@acme.com\nSubject: Q3 budget review\n\nBody:\nCan we sync before Friday?"}
  ],
  "format": "json",
  "options": {"temperature": 0.1}
}'

If you're integrating this into your own app rather than using Ollama: llama-server (llama.cpp's own server binary) handles this model's real chat template correctly on its own — verified directly, no override needed there, only for Ollama's simpler auto-detection.

Training

• Base: h2oai/h2o-danube3-500m-chat (plain LlamaForCausalLM, 32,000-token vocab), full fine-tune (100% of parameters, not LoRA), 3 epochs
• Data: ~4,800 triage examples + ~1,600-2,000 examples each for chat/daily-summary/draft-reply/compose, all matching production prompts exactly — generated by the five generate_*.py scripts in this repo; training text reformatted to Danube3's native <|prompt|>/<|answer|> turn format since it has no system-role support (see train_cipher_nano.py)
• Framework: Unsloth + trl.SFTTrainer
• Sequence packing was tried to speed up training (most examples are well under the 2048-token context window) — it crashed outright, an Unsloth/trl version incompatibility, not a quality tradeoff. Disabled.
• Reproduce with train_cipher_nano.py → export_gguf_cipher_nano.py
• Voice-intent retrain: added generate_voice_intent.py's data (weighted heavily toward the no-match case and exact-copy-under-distractor-pressure scenarios) to the mix across 3 rounds of oversampling. Nano's smaller full-fine-tune capacity meant this narrowed but never fully eliminated one specific habit — substituting a different plausible-looking address instead of copying the real one verbatim — see the README intro above for how this ended up getting solved at the calling-app level instead. Reproduce/verify with eval_voice_intent.py.
• Full retrain (current version): one clean pass over the base model on a single consolidated, properly-balanced dataset covering all 6 tasks, rather than continuing to layer incremental patches on top of each other — done specifically to fix both the injection-compliance issue and the hallucination regression noted above without risking a whack-a-mole cycle where each narrow fix nudges a different, previously-working case. Followed by one small, targeted continue-train pass (~700 examples, 2 epochs, lr=2e-5) reinforcing "report missing information plainly rather than inventing a plausible- sounding detail," specifically targeting the exact hallucination case found in testing.

License

Apache 2.0, inherited from the base model. Weights, training code, and eval harness are fully open.