iSshobhit

/

bnpl-triage-small-model

like 1

Hi Austin,

Thanks for writing a note. Happy to share the details.

The key thing is that I don't generate triage data generically. The domain specification is what drives the whole dataset. In BNPL dispute handling the policy already defines everything I need: the set of dispute types, the fields that describe a case, the rules that map those fields to the correct route, priority, and remedy, and the scheme and regulatory windows that constrain them. So the generator is parameterised by that structure, not by free text.

Concretely, each synthetic case is a set of domain fields (dispute type, transaction date and amount, delivery/merchant status, days elapsed, installment state, and so on), and the ground-truth triage label is derived by applying the policy rules to those fields deterministically. That's what makes the label trustworthy: it isn't a guess, it's the policy applied to known facts. The preference pairs fall out of that. Chosen is a response that matches the policy-derived decision and justifies it with the fields that actually drove it. Rejected is a same-case response that fails in a domain-meaningful way: wrong dispute type, misapplied window, or a rationale that ignores the field that determined the call.

One example to make it concrete. The generator produces a case where the customer reports the item never arrived, the transaction was 12 days ago, the merchant has marked it delivered, and the amount is inside the chargeback-eligible window. Policy makes this a goods-not-received / merchant dispute, routed to that team under the SLA for the window, and the correct rationale has to cite the delivered-versus-not-received conflict and the date falling inside the window. A rejected candidate for that same case is one that routes it as unauthorized fraud (wrong team and SLA entirely), or one that reaches the right route but claims the dispute is out of window when the 12-day date says otherwise. That second one looks fine on the surface, but it's grounded in a fact that isn't true, which is exactly the failure I want DPO to penalize.

So the design is domain-first: encode the dispute taxonomy and policy rules into the generator, derive ground truth from them, and let the preference pairs express "policy-correct and grounded in the deciding facts" over "plausible but wrong."

Best Regards,
Shobhit

Also DPO here done is not intended to replace the Customer support where a human and tonned response are being trained for customer facing interface. Here DPO is more structured towards the Ops team which needed the grounding response with correct policy categorisation

I am currently working on another DPO for guardrail tunning on the model. It will help to block any response that shows violence or abusive language as a first guard though the AI agent build on top of it can add strong deterministic guardrails.
If needed we can connect to discuss.
reach out to me : iosshobhit@gmail.com