Qwen3-8B-usersim-dpo

DPO-finetuned Qwen/Qwen3-8B for user simulation — i.e. the model that role-plays the customer in agent/tool-use evaluations (retail & airline customer-service scenarios). It was trained to produce more human-like and task-faithful customer turns, and to avoid breaking character (e.g. leaking simulator control tokens).

This repo contains a merged model (LoRA adapter merged into the base weights), so it can be loaded directly like any standard Qwen3-8B checkpoint.

Serve it in no-think mode. The model was trained with the Qwen3 chat template using enable_thinking=False. Use the same setting at inference, or the input distribution shifts.

Training

• Method: offline Direct Preference Optimization (DPO), sigmoid loss, β = 0.1.
• Adapter: LoRA (r=16, α=32, dropout 0.05) on all attention + MLP projections, then merged.
• Data: 753 preference pairs (prompt = sim system guidelines + scenario + dialogue so far; chosen / rejected = customer messages). Preferences were labeled by a GPT-5.x judge along human_likeness and task_fidelity perspectives. Domains: airline / retail.
• Recipe: TRL DPOTrainer, 2 epochs, lr 1e-5 (cosine, 10% warmup), effective batch 16, bf16, on a single A100 80GB. Best checkpoint selected by eval loss.

Eval (held-out 5%, by step)

step	epoch	eval_loss	reward acc	reward margin
20	0.45	0.2547	0.895	2.54
40	0.89	0.1818 ⭐	0.921	6.13
60	1.34	0.1882	0.921	6.57
80	1.78	0.1892	0.921	6.61

eval_loss bottomed at epoch ~0.9; later steps show mild overfitting, so the best checkpoint (epoch 0.89) was the one merged here. On this dataset ~1 epoch is optimal.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sngwon/Qwen3-8B-usersim-dpo"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")

messages = [
    {"role": "system", "content": "<user-simulation guidelines + scenario>"},
    {"role": "user", "content": "Hi! How can I help you today?"},  # the agent's turn
]
enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
).to("cuda")
out = model.generate(**enc, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Intended use & limitations

• Intended use: simulating a customer inside agent-evaluation harnesses. Not an assistant/agent model.
• Length signal: chosen responses are on average longer than rejected; standard DPO can pick up a mild length bias. If outputs get verbose, prefer the ~1-epoch checkpoint / lower lr.
• Control-token leakage: the training data penalizes emitting the simulator's ###STOP### control token as if it were customer text. Reduced but not fully eliminated — spot checks can still surface it under sampling. Post-filter ###STOP### from generated turns if your harness needs clean output.
• Domain scope: trained only on retail / airline customer-service scenarios; behavior outside these is untested.
• Base license: inherits Apache-2.0 from Qwen3-8B.