drstupidity/granite-3.3-2b-customer-support-lora

Customer Support Assistant (granite-3.3-2b)

LoRA fine-tune of ibm-granite/granite-3.3-2b-instruct on the Bitext customer-support dataset, trained to answer customer-support requests in a helpful support register.

Prompt template

Training, evaluation and serving all use this exact format. Use the model's own chat template with add_generation_prompt=True.

System prompt:

You are a customer support assistant. Read the customer's message, understand what they need, and reply accurately and helpfully in a professional, empathetic tone. Give concrete next steps when they apply, and never invent account details, order numbers, or policies you were not given.

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "drstupidity/granite-3.3-2b-customer-support-lora"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "I need to cancel order 12345"},
]
prompt = tok.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

The model was trained with a 15% system-prompt dropout, so it also behaves sensibly with a different system prompt or none at all.

These weights are the model, not the served stack

Everything above is all you need. The repository also runs a guardrail layer in front of the model at serve time (NeMo Guardrails plus a local out-of-domain embedding check), but none of it is baked into these weights and none of it is required to run them. Loading the checkpoint as shown gives you the model alone.

That separation is deliberate. The rails divert a small fraction of requests to predefined text instead of generating -- measured at 0.27% of real validation queries, and 9.4% on a deliberately adversarial probe set. If those fired during someone else's evaluation, the scores would describe the rails rather than the model. So the serving layer exposes an ungated /v1/completions route for exactly that purpose, and the rails can be turned off entirely with CSBOT_GUARDRAILS=0.

Training

method	LoRA (r=16, alpha=32, dropout=0.05)
target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
learning rate	0.0002
schedule	cosine, warmup 0.03
effective batch	32
max seq len	640
train rows	31,343
guardrail rows	1,235 synthetic "when not to help" examples
loss	completion-only (assistant tokens)
hardware	NVIDIA GeForce RTX 5060 Laptop GPU, peak 3.47 GB
final eval loss	0.6328

Data was split by near-duplicate cluster, not by row: the source dataset is 27 intents with roughly a thousand generated paraphrases each, so a random split leaks near-identical text across the train/test boundary. See the project repository for the leakage audit, evaluation design and base-vs-tuned results.

Why there are synthetic "decline" examples in the training data

Every one of the 23,453 Bitext rows is "customer asks -> agent helpfully assists". There is not a single example of declining. Fine-tuning on that alone taught the model always help, which is right inside the domain and wrong the moment a query leaves it: measured against a held-out behavioural probe set, its off-topic redirect rate dropped from 0.800 (base model) to 0.350.

Blending in a 5% slice of examples that decline off-topic requests, refuse instruction overrides, ask for clarification when a message is uninformative, and decline to invent account data brought that to 0.850 -- past the base model -- at no measurable in-domain cost (validation loss 0.6461 vs 0.6436 for an otherwise identical control run).

Limitations

• Trained on synthetic, English, single-turn support data. It has no multi-turn conversation training and no access to real account systems.
• It must not be relied on for account-specific facts; it is trained never to invent order numbers, policies or contact details, but that is a tendency, not a guarantee.
• Non-English input is out of distribution for the training data.