sanidhya1910/northwind-support-lora

Northwind Support LoRA (Qwen3-4B)

A LoRA adapter that moves a 992-token customer-support system prompt — a 20-entry policy handbook plus a strict JSON output contract — into the weights, so inference runs on a one-line system prompt instead.

Trained on a synthetic dataset generated from scratch by a seeded, rule-based generator — no teacher model, no API key, reproducible byte-for-byte. Everything needed to judge the result is on this page: the four-arm comparison, the training setup, and the limitations.

⚠️ This adapter encodes the policies of Northwind Outdoors, a fictional retailer. It is a methodology demonstration, not a usable support agent for any real company. See Limitations.

What it does

Given a customer message, it replies in 2-4 policy-grounded sentences and appends exactly one machine-readable ticket:

Clearance and final-sale items are sold as-is, so the Ridgeline rain shell on NW-771904 can't be
returned or exchanged. I know that's not the answer you wanted; it's called out at checkout.

```json
{"intent": "return_refund", "priority": "P3", "escalate": false, "policy_ids": ["RET-02"], "next_action": "answer_only"}
```

Results

200 held-out conversations, greedy decoding, rule-based scoring against gold labels derived by rule (no LLM judge). Customer phrasings in the test set never appear in training.

Metric	base + full prompt (992 tok)	+ 2-shot	base + one-line prompt	this adapter + one-line
Prompt tokens (mean)	1078.7	1394.7	98.7	98.7
Schema-valid JSON %	97.0	99.5	1.0	98.5
Intent accuracy %	64.0	70.5	3.0	98.5
Priority accuracy %	63.0	62.5	2.5	95.0
Next-action accuracy %	49.0	55.0	2.5	88.5
Escalation F1	70.0	69.2	18.5	90.3
Escalation recall	58.3	56.2	10.4	87.5
Policy-citation F1	58.6	65.1	8.3	90.5
Prose copied verbatim from training %	0.0	0.0	0.0	48.0
Seconds / example	2.93	2.43	3.92	1.47

91% fewer prompt tokens on every request, while beating the prompted baseline on every accuracy metric. The base + one-line prompt column is the ablation: without the adapter, the base model has no idea the policy IDs or the ticket format exist.

The most useful gap is escalation recall. Both prompted arms are precise (~90%) but miss ~43% of cases that require a human — injury, fraud, chargebacks, legal threats. The adapter reaches 87.5%.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "blackburn1910/northwind-support-lora")

messages = [
    {"role": "system", "content": "You are the customer support assistant for Northwind Outdoors."},
    {"role": "user", "content": "I want to return the rain shell I bought on clearance, order NW-771904."},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
out = model.generate(**tok(text, return_tensors="pt", add_special_tokens=False).to(model.device),
                     max_new_tokens=300, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

The system prompt must be exactly You are the customer support assistant for Northwind Outdoors. — that is what it was trained against. Qwen3 must run in non-thinking mode (enable_thinking=False).

Training

Base	Qwen/Qwen3-4B (Apache-2.0)
Method	LoRA r=16, α=32, dropout 0.05, on all attention + MLP projections
Data	1,400 synthetic conversations → 1,938 assistant-turn examples
Objective	causal LM, loss masked to assistant turns only
Schedule	3 epochs, 366 steps, lr 2e-4 cosine, effective batch 16, bf16
Loss	train 3.5376 → 0.0246; eval 0.0626 → 0.0345 → 0.0335 (ppl 1.034)
Hardware	1× RTX 5070 Ti (16 GB), ~1h50m

Training data is generated offline by a seeded, rule-based generator — no teacher model and no API key — so it reproduces byte-for-byte.

Limitations

• It has learned one fictional company's handbook, not customer support. Point it at a different policy set and it will be confidently wrong, in the way a prompted model would not be — the prompted baseline simply follows whatever prompt you give it. That trade is the whole point of the experiment, and it cuts both ways.
• 48% of its replies reproduce a training answer verbatim (after normalising order IDs and numbers), against 0% for the prompted baseline. The training targets are template-generated, so the adapter has substantially memorised the response bank. Read the accuracy figures as evidence about format and policy fidelity, not about writing quality.
• Prose quality is not evaluated. No human raters, no judge model — deliberately, so every reported number is rule-checkable.
• Test phrasings are unseen but come from the same generator as training. This measures generalisation across wording, not across distribution. Real ticket text has a tail of weirdness this data does not.
• Single seed, single base model. Differences of a point or two are noise.
• Not safety-tuned for production use. The escalation rules here are deliberately crisp; real escalation policy is fuzzier and would need human-labelled data.

License

Apache-2.0, matching the Qwen3-4B base model.