lemonade-sdk/pii_policy_llm_classifier

PII Router Policy - LLM Router (Qwen3.5, three sizes)

A Lemonade collection.router policy using an LLM as the router (routing.router.type: "llm") rather than a dedicated classifier model: the router LLM reads each prompt and decides, via a system-prompt instruction, whether to route it to a local/private candidate or a cloud candidate.

This repo holds only the policy configs - not model weights. Three variants are published, one per router-LLM size; the router LLM and the local completion candidate are the same model in each (it does double duty: making the routing decision, then handling the request if routed local).

File	Router / local candidate
pii_policy_llm_classifier_9b.json	Qwen3.5-9B-GGUF
pii_policy_llm_classifier_2b.json	Qwen3.5-2B-GGUF
pii_policy_llm_classifier_0.8b.json	Qwen3.5-0.8B-GGUF

All three route to fireworks.kimi-k2p6 as the cloud candidate, and default to the local candidate when the router is uncertain (privacy-first default - the opposite convention from a classifier-based policy, where the default is normally cloud and PII detection is what moves traffic local).

Benchmark results (Nemotron-PII, 2,500-case sample)

Metric	Qwen3.5-9B	Qwen3.5-2B	Qwen3.5-0.8B
Leak rate (PII → cloud)	6.16% (154/2,500)	3.28% (82/2,500)	2.84% (71/2,500)
Recall	93.84%	96.72%	97.16%
Rationale behind routing	99.5%	59.4%	18.30%
Prompt misses (of leaks)	129/153 (84%)	61/82 (74%)	40/71 (56%)
Genuine misses (of leaks)	24/153 (16%)	21/82 (26%)	31/71 (44%)
Missed categories	not recorded	not recorded	not recorded
E2E runtime (full corpus)	6hrs	2.5hrs	2.1hrs
Routing time	not recorded	2.28hrs	1.5hrs
Prompt processing time	not recorded	21mins	18mins

Counterintuitive at a glance: the larger router (9B) has the higher leak rate and lower recall than the smaller ones. "Rationale behind routing" (how often the router LLM emitted a parseable justification alongside its decision, rather than a bare/fallback decision) drops sharply as the model shrinks (99.5% → 18.3%), so take the smaller models' numbers with that caveat in mind - they're making the same binary call with much less visible reasoning, not necessarily "better judgment" in a deeper sense. "Prompt misses" vs. "genuine misses" splits each size's leak count into misses where the source document didn't clearly surface the PII in the prompt text itself (harder to fault the router for) versus misses where the signal was present and the router still routed to cloud. Missed categories and the 9B row's timing breakdown weren't recorded for this run.

This benchmark used a 2,500-case sample, not the larger 20,001-case Nemotron corpus used to benchmark this project's ONNX-classifier policies (see lemonade-sdk/pii_policy_openmed-privacy-filter-multilingual-v2-onnx and lemonade-sdk/pii_policy_mmbert32k-pii-detector-merged-onnx) - leak rates aren't directly comparable across the two corpus sizes.

Usage

Pick one size and register it (the LLM candidates are expected to already be registered/available - no separate classifier-model registration step, unlike the ONNX-classifier policies):

hf download lemonade-sdk/pii_policy_llm_classifier --local-dir .
curl -X POST http://localhost:13305/v1/pull -H "Content-Type: application/json" \
  --data-binary @pii_policy_llm_classifier_2b.json

Then send chat completions to that file's model_name (e.g. "model": "user.PII-LLM-Router-2B") and the server routes each request per its system prompt's instructions.

Adapting this policy

The router's prompt field is the entire routing logic for an LLM-router policy - edit it to change what counts as sensitive, add categories, or adjust the privacy-first default described there. routing.candidates, default_model, and router.model need to stay in sync with whatever local/cloud models you actually have registered.

For building a routing policy from scratch, or a more sophisticated one (multiple classifiers, custom match logic, hybrid classifier+LLM routing) rather than adapting one of these, see the lemonade-router-builder skill - it turns a natural-language description of routing intent into a valid collection.router policy JSON.

License

Apache 2.0.