hvss/Dispatch-7B

The Dispatcher: Agent Ops 🚦

Dispatch-7B, an agent orchestrator in 7B parameters — the tiny AI manager that runs a team of expensive AI workers.

🎬 Watch it route work live →

Dispatch-7B is a fine-tuned Mistral 7B Instruct v0.3 that acts as the orchestrator in an agentic system. It reads a user request plus a catalog of available tools and outputs a structured execution plan: which tool handles each step, in what order (as a dependency DAG), and which steps genuinely deserve escalation to a frontier model.

Small model as the boss. Big models as the workers.

Most AI products send every request — even trivial ones — to the biggest, most expensive model. That's like having your highest-paid executive answer the front desk phone. Dispatch-7B is the dispatcher: it plans in milliseconds on cheap hardware, routes each step to the cheapest adequate worker, and escalates only when a step actually requires frontier-level reasoning.

Evaluation

Measured on 114 held-out tasks (unseen requests and unseen tool-catalog combinations):

Metric	Score
Valid plan rate (parseable JSON, schema-correct, acyclic DAG, catalog-consistent)	97.4%
Tool selection F1 (vs. teacher plans)	0.92
Escalation precision / recall	0.92 / 0.75
Est. cost vs. "frontier model does everything"	−86.7%

Robustness breakdowns:

Slice	Valid plan rate
Distractor-heavy catalogs (relevant tools buried among irrelevant ones)	93%
Gap catalogs (a needed tool deliberately missing)	94%
Hard tasks requiring escalation judgment	89%

Notes: routing labels are distilled from a teacher model (gpt-5-mini), so accuracy metrics measure agreement with the teacher. The cost figure is a transparent cost model over the eval plans (frontier $3/$15 per MTok, mini-tier $0.15/$0.60, self-hosted 7B router; invalid router plans charged as full frontier fallback), not a measured bill — assumptions are in the eval scripts.

The model deliberately under-escalates slightly (11.7% of plans vs. the teacher's 15.8%): it errs on the side of cheap execution, which is usually the right default for a router.

Usage

The model expects Mistral's [INST] format with the dispatcher system prompt and your tool catalog:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("hvss/Dispatch-7B", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("hvss/Dispatch-7B")

SYSTEM = '''You are a dispatcher. Read the user request and the tool catalog, then output ONLY a JSON execution plan routing each step to the cheapest adequate worker: "tool:<name>" (catalog tools only), "small_model" (simple generation), or "escalate:frontier" (genuinely hard reasoning only). Steps form a DAG via "depends_on". Output format: {"goal": ..., "steps": [{"id", "task", "assignee", "depends_on"}], "escalation_reason": ...}

Available tools:
- weather_lookup: Get the weather forecast for a location.
- calendar_create_event: Create a calendar event with title, time, and attendees.
- email_send: Send an email to one or more recipients.'''

request = "Set up a team picnic next Friday at noon if the weather looks good, and email the team an invite."

prompt = f"[INST] {SYSTEM}\n\n{request} [/INST]"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=768, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Output:

{"goal": "...", "steps": [
  {"id": 1, "task": "Check the weather forecast for next Friday.", "assignee": "tool:weather_lookup", "depends_on": []},
  {"id": 2, "task": "Create the picnic calendar event if the forecast is good.", "assignee": "tool:calendar_create_event", "depends_on": [1]},
  {"id": 3, "task": "Email the team an invite with a short fun blurb.", "assignee": "tool:email_send", "depends_on": [2]}
], "escalation_reason": null}

Three assignee classes:

Assignee	Meaning	Cost
tool:<name>	Direct tool/API call from your catalog	~free
small_model	Simple generation a small LLM handles fine	cheap
escalate:frontier	Genuinely hard reasoning — send to the big model	expensive, used sparingly

Swap in your own catalog — the model was trained on randomized catalogs (with distractors and deliberate gaps), so it plans with whatever tools you give it rather than assuming a fixed set.

Variants

Repo	Use case
hvss/Dispatch-7B	merged fp16 — vLLM / transformers
hvss/Dispatch-7B-GGUF	Q4_K_M / Q8_0 — llama.cpp, Ollama, local
hvss/Dispatch-7B-LoRA	adapter only — further fine-tuning

Training

• Base: Mistral 7B Instruct v0.3 (Apache 2.0)
• Method: QLoRA (4-bit, r=16 on all attention + MLP projections) via Unsloth, on a single free Colab T4
• Data: 2,158 (request, gold plan) pairs distilled from a teacher model over a structured taxonomy: 8 domains × 4 difficulty tiers × 3 catalog conditions (full / distractor-heavy / deliberately-missing-tool), ~16% escalation rate to teach restraint
• Loss: response-only (the model learns to write plans, not to regenerate catalogs)

Limitations

• English-only training data.
• Routing quality is bounded by the teacher's judgment; gold labels are synthetic.
• The tool catalogs are synthetic (40 tools across 8 domains); very different domains or tool-description styles may need a light further fine-tune (start from the LoRA repo).
• The model plans; it does not execute. Your harness owns tool execution, worker calls, and validation (schema-check the output — 2.6% of plans fail validation and should fall back to your default model).

License

Apache 2.0 — free for commercial use.