mg2660/qwen3-customer-support-lora

Qwen3 customer-support QLoRA adapter

This repository contains a QLoRA adapter fine-tuned from Qwen3-4B-Instruct-2507 for an educational customer-support experiment.

Important evaluation result

On a 1,000-example, 27-intent evaluation set, this adapter did not outperform the strong prompt-engineered baseline used in this project. The comparison evaluated the same base model in two configurations:

1. Base + Prompt: Qwen3-4B-Instruct-2507 with a fixed customer-support system prompt.
2. QLoRA: the same base model with this adapter.

The raw, unprompted base model was not evaluated.

Metric (1–5 unless noted)	Base + Prompt	QLoRA	QLoRA − Base
Relevance	4.834	4.715	-0.119
Helpfulness	4.308	3.750	-0.558
Correctness	4.743	4.425	-0.318
Unsupported-assumptions score	4.680	3.990	-0.690
Overall quality	4.466	3.968	-0.498
Pass rate	92.0%	82.8%	-9.2 percentage points
Major-error rate	3.9%	10.0%	+6.1 percentage points

All ordinal-score differences favored Base + Prompt in paired Wilcoxon signed-rank tests. The automated evaluation found recurring QLoRA issues with unresolved template placeholders, generic non-actionable responses, and unsupported company-specific claims. These findings are from one automated judge and should be complemented by human review.

See the public-safe evaluation report and aggregate summary. Per-example prompts and outputs are intentionally not published.

Intended use

This adapter is intended for educational experimentation with QLoRA fine-tuning and evaluation methodology. It is not production-ready customer-support software.

Do not use it as the sole source of customer-facing policy, account, order, payment, delivery, or refund guidance. The evaluation found that it can emit unresolved placeholders and unsupported procedures.

Training and evaluation data

The source dataset is bitext/Bitext-customer-support-llm-chatbot-training-dataset, which has 26,872 examples, 27 intents, and 10 categories. Supervised fine-tuning used the instruction → response relationship; category and intent were metadata rather than model inputs.

An instruction-grouped, leakage-aware split was created so that rows with the same instruction stayed in one split:

Split	Examples
Train	1,600
Validation	400
Test	1,000

The test set covers all 27 intents. There is no exact instruction overlap between the persisted train, validation, and test splits.

Historical reproducibility caveat: the existing adapter was produced in an earlier workflow, before this persisted 1,600-example training split became definitive. The split is leakage-controlled, but complete historical proof that this adapter never saw a final-test instruction cannot be established.

Each evaluation example includes a customer instruction, intent label, and reference response.

For the evaluation, both generated responses were independently scored against the same instruction and reference. The reference was treated as evidence rather than an exact-match target: a different response could score well if it appropriately addressed the customer request.

Evaluation methodology

The evaluator used the OpenAI Responses API with strict structured JSON output. Each response received 1–5 scores for relevance, helpfulness, correctness, unsupported assumptions, and overall quality, plus pass/fail and major-error labels.

Because both systems were judged on the same examples, the analysis used paired bootstrap confidence intervals and Wilcoxon signed-rank tests for ordinal scores, and exact McNemar tests for pass and major-error rates. See evaluation/README.md for the reproducible evaluation pipeline.

Training configuration

Setting	Value
Learning rate	0.0002
Train batch size	2
Gradient accumulation steps	8
Effective train batch size	16
Training steps	200
Seed	42
Learning-rate scheduler	Linear
Validation loss at step 200	0.7370

Training ran on a Google Colab NVIDIA Tesla T4 (approximately 14.6 GB GPU memory). The adapter used 4-bit QLoRA with NF4 quantization, bfloat16 compute, and double quantization.

LoRA setting	Value
Rank	16
Alpha	32
Dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Approximate trainable parameters	0.81%
Maximum sequence length	256
Gradient checkpointing	Enabled

Both systems generated responses for the same 1,000 instructions using deterministic decoding (do_sample=False) with approximately 160 maximum new tokens. The model received only the customer instruction; intent, category, and reference response were not generation inputs.

Loading the adapter

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model_id = "mg2660/qwen3-customer-support-lora"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoPeftModelForCausalLM.from_pretrained(model_id, device_map="auto")

Use the same tokenizer/chat template as the underlying Qwen3 base model. This repository provides an adapter; it does not replace the base model.

Limitations

• This is an experimental adapter, not a validated production system.
• The evaluation uses one automated judge; judge bias and imperfect references remain possible.
• The adapter predates the final persisted split, so complete historical proof of no test exposure is unavailable.
• Statistical significance does not by itself establish practical customer impact.
• The results apply to this held-out dataset and comparison prompt, not every customer-support domain.