srock44/cipher-air

Cipher Air

Cipher Air is a LoRA fine-tune of Qwen/Qwen2.5-0.5B-Instruct, trained on every LLM-backed feature of a local-first email assistant: email triage (importance/summary/category JSON), chat, daily-summary synthesis, draft reply, and compose assist — not just prompted for these tasks, actually trained on them.

It's the middle of the three Cipher tiers (cipher-nano / cipher-air / cipher-pro) — a balanced default at 40% of cipher-pro's disk size and 3x the throughput. Cipher is the local-model engine for an unreleased larger email-assistant project — that project isn't public yet, but these weights, the training code, the eval script, and all five dataset generators are fully open now, in this repo.

Why this exists

Most email triage today means sending your inbox to a third-party API. Cipher runs entirely on your own hardware via Ollama — nothing about your email ever leaves your machine.

What's in this repo

• cipher-air.Q4_K_M.gguf — the model weights, ready for Ollama
• Modelfile — the exact Ollama Modelfile (system prompt + inference params) used in training/eval
• train_cipher_air.py / export_gguf_cipher_air.py — the exact scripts used to produce this model (Unsloth LoRA on the base model above)
• generate2.py, generate_chat.py, generate_daily_summary.py, generate_draft_reply.py, generate_compose.py — the five task-specific synthetic-data generators (produces the full multi-task training set)
• eval_triage.py / eval_fixtures.json — a standalone benchmark harness (no external dependencies beyond httpx/pydantic) reproducing the numbers below

Everything needed to reproduce this model from scratch, or fine-tune your own variant, is in this repo — nothing here depends on an unreleased package.

Benchmark

Evaluated on a 29-fixture triage benchmark on an RTX 5070:

Model	Disk	Tok/s	JSON-valid	Category acc	Importance-in-band	Injection-safe
cipher-air	398 MB	520.1	96.6%	71.4%	71.4%	100%

Honest caveat: cipher-air is the tightest-capacity tier of the three (only 8.8M of 502M params are trainable via LoRA), and it shows — of the three tiers it's the one most likely to occasionally misjudge whether something genuinely needs a reminder/action versus being routine. cipher-pro and cipher-nano both handle that nuance more reliably. Reproduce with:

pip install -r requirements.txt
python eval_triage.py --models cipher-air:latest --keep

Usage (Ollama)

ollama create cipher-air -f Modelfile

Query it with grammar-constrained JSON output for reliable parsing:

curl http://localhost:11434/api/chat -d '{
  "model": "cipher-air",
  "messages": [
    {"role": "system", "content": "<system prompt from Modelfile>"},
    {"role": "user", "content": "From: alex@acme.com\nSubject: Q3 budget review\n\nBody:\nCan we sync before Friday?"}
  ],
  "format": "json",
  "options": {"temperature": 0.1}
}'

Training

• Base: Qwen/Qwen2.5-0.5B-Instruct, LoRA (r=16, alpha=32, all linear layers), 2 epochs
• Data: ~4,800 triage examples (oversampled ~2x to ~60% of the final training mix — this size tier needed a stronger triage signal than the other two to hold onto exact JSON schema output while also learning four other task formats) + ~1,600-2,000 examples each for chat/daily-summary/draft-reply/compose, all matching production prompts exactly — generated by the five generate_*.py scripts in this repo
• Framework: Unsloth + trl.SFTTrainer
• Sequence packing was tried to speed up training (most examples are well under the 2048-token context window) — it crashed outright, an Unsloth/trl version incompatibility, not a quality tradeoff. Disabled.
• Reproduce with train_cipher_air.py → export_gguf_cipher_air.py

A dead end worth knowing about

We tried quantizing this model down further (Q3_K_M, Q2_K) hoping to shrink it toward cipher-nano's size class. It barely helped (355MB / 339MB vs 398MB at Q4_K_M) — Qwen2.5's 151,936-token vocabulary embedding table dominates disk size and doesn't compress with weight quantization. If you're looking for something genuinely small, use cipher-nano instead (different base model, built specifically to solve this).

License

Apache 2.0, inherited from the base model. Weights, training code, and eval harness are fully open.