hayulalab/multi-specialist-framework-paper

🏗️ Multi-Specialist Model Orchestration Framework

A general-purpose training & deployment framework
One orchestrator model routes tasks to specialist models.
Reusable — plug in any purpose and any model set.

---

📌 Overview

┌─────────────────────────────────────────────────────────┐
│                    ORCHESTRATOR (Router)                 │
│              Classifies task → routes to specialist       │
└──────────┬──────────┬──────────┬──────────┬──────────────┘
           │          │          │          │
     ┌─────▼──┐ ┌────▼───┐ ┌───▼────┐ ┌───▼──────┐
     │Spec A  │ │Spec B  │ │Spec C  │ │Spec N... │
     │(Expert)│ │(Expert)│ │(Expert)│ │(Expert)  │
     └────────┘ └────────┘ └────────┘ └──────────┘

Core principle: One generalist model routes to N specialists.
Each specialist is fine-tuned on ONE domain.
The orchestraor never generates domain output — it routes.

---

Phase 1 — Define The Purpose

Before anything: what are you building?

Field	Your Value
Purpose	e.g., "Cybersecurity vulnerability analysis"
Base Model	e.g., Qwen2.5-7B, Llama-3-8B, Mistral-7B
# of Specialists	N
Specialist Names	List each one

Examples

Domain	Specialists
Cybersecurity (SAIF)	webapp, browser, vulndisc, kernel, rev_eng, mobile, network, crypto
Code Review	python, javascript, rust, go, solidity, security
Medical Diagnosis	cardiology, neurology, radiology, oncology, pediatrics
Legal Analysis	contract, litigation, regulatory, IP, tax
Translation	ar→en, en→ar, fr→en, technical, literary
DevOps	kubernetes, terraform, docker, ci/cd, monitoring, security

---

Phase 2 — The Orchestrator (Router)

2.1 Architecture

A lightweight classifier model (or a classification head on the base model) that:

1. Reads the input prompt
2. Matches keywords to specialist domains
3. Returns the specialist name + confidence score
4. Falls back to a generalist/default specialist if no match

2.2 Routing Logic (Python Pseudocode)

ROUTER_RULES = {
    "webapp":     ["sql injection", "sqli", "xss", "csrf", "ssrf", "lfi",
                   "rfi", "oauth", "jwt", "web app", "login", "api endpoint",
                   "rest", "graphql", "post", "get", "parameter"],
    "browser":    ["browser", "v8", "spidermonkey", "dom", "xss", "cors",
                   "same-origin", "chromium", "firefox", "webkit",
                   "javascript", "js engine"],
    "vulndisc":   ["vulnerability", "disclosure", "cve", "zero day",
                   "poc", "exploit", "bug bounty", "penetration test"],
    # ... add your specialists here
}

def classify(prompt: str) -> str:
    """Auto-detect specialist based on prompt keywords."""
    prompt_lower = prompt.lower()
    matches = {}
    for specialist, keywords in ROUTER_RULES.items():
        score = sum(1 for kw in keywords if kw in prompt_lower)
        if score > 0:
            matches[specialist] = score

    if not matches:
        return "generalist"  # fallback

    # Return highest scoring specialist
    return max(matches, key=matches.get)

2.3 Training the Router

The router can be:

• Rule-based (keyword matching) — simple, transparent, no training needed
• ML classifier (e.g., DistilBERT, MiniLM) — train on labeled prompt→specialist pairs
• LLM-based — just prompt the base model with "Classify this task into one of: [list]"

For rule-based, collect/maintain your keyword mapping as a YAML file:

# router_keywords.yaml
specialists:
  webapp:
    keywords: [sql injection, xss, csrf, ssrf, lfi, api, graphql]
    fallback_terms: [web, http, url, endpoint, parameter]
  browser:
    keywords: [v8, javascript, dom, browser, same-origin, cors]
    fallback_terms: [chromium, firefox, js, webkit]
  # ...

2.4 API Format

The orchestrator exposes a single endpoint:

POST /v1/completions
{
  "prompt": "...",
  "model": "router",         # ← triggers auto-classify
  "max_tokens": 500,
  "temperature": 0.3
}

• model: "router" → auto-classify and delegate
• model: "webapp" → skip routing, go directly to that specialist
• Response includes: {"specialist": "webapp", "output": "...", "latency_ms": 1234}

---

Phase 3 — Training Specialists

3.1 Data Collection

For EACH specialist, collect domain-specific training data:

data/
├── webapp/
│   ├── owasp_train.jsonl        # SQLi, XSS, CSRF examples
│   ├── real_world_bugs.jsonl    # Real bug bounty reports
│   └── edge_cases.jsonl         # Hard negative examples
├── browser/
│   ├── v8_exploits.jsonl
│   ├── dom_xss.jsonl
│   └── cve_examples.jsonl
└── ...

Data format (standardized across all specialists):

{"instruction": "...", "input": "...", "output": "...", "source": "real_bounty", "difficulty": "medium"}

3.2 The Hallucination Problem

Specialists trained on synthetic/CTF data will hallucinate on real targets.

Symptoms:

• Cloudflare /cdn-cgi/trace → classified as "RCE CWE-416"
• Standard 404 → classified as "vulnerable open endpoint"
• API 400 errors → classified as "exploitable"

Solution: Collect real-world negative examples:

• 60% positive (real vulnerabilities you found)
• 30% negative (normal responses — 200 OK, 404, 403, redirects)
• 10% edge cases (things that look vulnerable but aren't)

3.3 Fine-Tuning Method

Use LoRA (Low-Rank Adaptation) for efficient training:

# Example: Qwen2.5-7B LoRA training
python train.py \
  --base_model Qwen/Qwen2.5-7B \
  --data_path ./data/webapp/ \
  --output_dir ./models/webapp-lora/ \
  --lora_r 16 \
  --lora_alpha 32 \
  --lora_dropout 0.1 \
  --num_epochs 3 \
  --batch_size 4 \
  --learning_rate 2e-4

Key parameters per specialist:

Specialist	Lora R	Epochs	Learning Rate	Batch Size	Notes
Core (general)	16	3	2e-4	4	Good default
Complex (kernel, rev_eng)	32	5	1e-4	2	More data needed
Simple (webapp, network)	8	2	3e-4	8	Smaller domain

3.4 Validation — Before Deployment

For EACH specialist, run a validation set:

Metric	Target	How
Evidence Precision	>80%	% of outputs with concrete evidence
False Positive Rate	<15%	% of "vulnerable" on known-safe targets
Latency P95	<3s	Time to first token
Hallucination Rate	<5%	% of confident predictions on non-existent vulns

---

Phase 4 — Deployment

4.1 Model Merging

Load base model once, swap LoRA adapters:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B")

specialists = {}
for name in ["webapp", "browser", "vulndisc", ...]:
    specialists[name] = PeftModel.from_pretrained(base, f"./models/{name}-lora/")

4.2 Serving Architecture

Client → API Gateway (FastAPI)
           │
           ├── /v1/completions → Router → Specialist
           ├── /health         → {"status":"ok","models_loaded":["webapp","browser",...]}
           └── /metrics        → Prometheus

4.3 Multi-Layer Fallback

If the orchestrator + specialist pipeline times out:

Specialist → (timeout) → Fallback Generalist → (timeout) → Cloud API

---

Phase 5 — Iterative Improvement

5.1 Tracking Hallucinations

Log EVERY output with:

• Prompt fingerprint
• Specialist used
• Confidence score
• Whether user reported hallucination

5.2 Correction Pipeline

1. User reports bad output
2. Save as training example (with correct classification + output)
3. Batch collect → weekly fine-tune cycle
4. Re-deploy updated LoRA adapter

5.3 Versioning

models/
├── webapp/
│   ├── v1/          # Initial training
│   ├── v2/          # +200 hallucination fixes
│   └── v3/          # +new domain coverage
├── browser/
│   └── v1/
└── ...

---

📋 Quickstart Template

Copy this and fill it in for your project:

# 🏗️ [PROJECT NAME] — Multi-Specialist Training Plan

## Purpose
[What are you building?]

## Base Model
[Which base model? E.g., Qwen2.5-7B, Llama-3-8B]

## Specialists (N = ___)

| # | Specialist | Domain | Training Size |
|---|------------|--------|--------------|
| 1 | [name]     | [desc] | [N examples] |
| 2 | [name]     | [desc] | [N examples] |
| 3 | [name]     | [desc] | [N examples] |
| 4 | [name]     | [desc] | [N examples] |

## Router Keywords

[save as router_keywords.yaml]

## Fallback Strategy

- Primary: [specialist to use]
- Fallback: [generalist model]
- Cloud: [API name]

## Validation Targets

- Evidence Precision: >___%
- False Positive Rate: <___%
- Latency P95: <___s

## Known Pitfalls

1. [Pitfall 1]
2. [Pitfall 2]

---

⚠️ Common Pitfalls

#	Pitfall	Solution
1	URL classification — specialists hallucinate when given bare URLs	Feed specialists context (response headers, body, behavior) not just URLs
2	CTF bias — trained on CTF challenges → flags production endpoints as vulnerable	Include real-world negative examples in training data
3	Cold start — new specialist has 0% accuracy	Start with rule-based fallback until you collect 50+ real examples
4	Prompt bleed — router keywords match wrong specialist	Higher weight to exact-match keywords; add negative keywords
5	LoRA interference — multiple LoRAs loaded affect each other	Isolate LoRA loading per request; never chain LoRAs
6	Confidence overfit — model always says 0.99 confidence	Calibrate on a held-out set; cap max confidence at 0.95

---

📚 Examples (Real Deployments)

1. SAIF (Cybersecurity) — 8 Specialists

• Base: Qwen2.5-7B
• Specialists: webapp, browser, vulndisc, kernel, rev_eng, mobile, network, crypto
• Router: keyword-based classify() with 100+ trigger words
• Training: LoRA, 182 examples per specialist, 3 epochs
• After fix: Evidence precision 64.7% → 82%

2. Your Turn — Enter Purpose & Models Here:

Purpose:    ________________________________
Base Model: ________________________________
Specialists: ________________________________
_____________________________________________
_____________________________________________