customer-support-qwen2.5-0.5b

LoRA adapter for Qwen/Qwen2.5-0.5B-Instruct, supervised-fine-tuned on bitext/Bitext-customer-support-llm-chatbot-training-dataset for single-turn customer-support response generation.

Task

Given a customer request, generate the support response. Training objective: causal LM with response-only loss masking — the loss is computed on the assistant turn only, the system prompt and the customer message are masked with -100.

Training data

Dataset	bitext/Bitext-customer-support-llm-chatbot-training-dataset
Raw examples	26872
Fields used	instruction (customer message) -> response (target)
Fields used as metadata only	category, intent (stratified split + per-category evaluation; never part of the prompt)
Fields dropped	flags (linguistic-variation tags)
Train / validation / test	21499 / 2696 / 2677
Split strategy	group-aware (normalised customer message) + stratified by 'intent', seed 42

Preprocessing: NFKC normalisation, whitespace collapsing, missing/empty removal, exact (instruction, response) de-duplication. Casing, punctuation and word forms are preserved. Splitting is group-aware on the normalised customer message, so the same query cannot appear in two splits.

Prompt format

The tokenizer's native Qwen chat template is used (tokenizer.apply_chat_template):

<|im_start|>system
You are a professional customer support assistant. Answer clearly, politely and concisely.<|im_end|>
<|im_start|>user
{customer_message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

Training configuration

Hyper-parameter	Value
Strategy	lora
LoRA rank / alpha / dropout	16 / 32 / 0.05
LoRA target modules	['down_proj', 'gate_proj', 'k_proj', 'o_proj', 'q_proj', 'up_proj', 'v_proj']
Trainable parameters	8,798,208 (1.7497% of 502,830,976)
Max sequence length	416
Learning rate	0.0002
LR scheduler / warmup	cosine / 134 (5% of 2688)
Epochs	2
Batch size x grad. accumulation	4 x 4 (effective 16)
Weight decay	0.01
Max grad norm	1.0
Precision	bfloat16
Seed	42
Hardware	NVIDIA GeForce RTX 5060 Ti
Training runtime	48.0 min

Results

Generation settings: max_new_tokens=128, greedy decoding (do_sample=False) for the table below; the sampled configuration (temperature=0.7, top_p=0.9) is reported in metrics.json.

Model	ROUGE-1	ROUGE-2	ROUGE-L	BLEU	BERTScore F1	Test loss	Perplexity
Base Qwen2.5-0.5B-Instruct	0.3262	0.0933	0.2009	0.0449	n/a	1.6282	5.095
+ this adapter	0.5830	0.3235	0.4254	0.2788	n/a	0.6184	1.856

Best validation loss: 0.6129 · validation perplexity: 1.841

Test loss and perplexity are teacher-forced, token-weighted cross-entropy over the response tokens of 2677 test examples, computed on identical tokenised sequences for both models. Generation metrics use 300 test examples for the adapter and 150 for the base model.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "thealper2/customer-support-qwen2.5-0.5b"

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [
    {"role": "system", "content": "You are a professional customer support assistant. Answer clearly, politely and concisely."},
    {"role": "user", "content": "I haven't received my refund yet. Can you check its status?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

out = model.generate(**inputs, max_new_tokens=128, do_sample=True,
                     temperature=0.7, top_p=0.9,
                     pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• 0.5B parameters: limited reasoning, no reliable factual grounding.
• The training data is synthetic and template-based; responses contain placeholders such as {{Order Number}} and the model reproduces that style, including inventing placeholders that were not in the request.
• Single-turn only — no dialogue history is modelled.
• Policies, phone numbers, URLs and hours in the output are generated text, not retrieved facts.
• English only.
• max_new_tokens=128 truncates responses longer than the budget; the reference responses have a median length of 108 tokens.