raphaelmansuy/bank-routing-modernbert

Bank routing ModernBERT

This is the winning model from how to train a decision model.

It reads one short English banking message and scores six queues: cards, transfers, cash, fees, account, and fraud. It does not write a reply, block a card, or decide that a payment is fraud. If sending the top queue is not cheaper than a person, the message should go to a person (needs_human).

Why this one

Five methods were compared on 2,379 held-out Banking77 messages. The score is the average pain of one message. A person on every message costs 0.20. A wrong queue costs 1.0. A real fraud message sent to another queue costs 5.0.

ModernBERT with that cost rule costs 0.052 per message and sends 1 of 235 fraud messages to the wrong team. That is cheaper than the simple word-count model (0.053, and 6 missed fraud messages) and cheaper than the same ModernBERT model with a single confidence cutoff (0.056, and 5 missed fraud messages). LoRA, zero-shot Qwen, and SetFit all cost more or are much less accurate.

The costs are illustrative. They are not a production approval.

How to use

Temperature is 1.455. Divide the six scores by 1.455, then turn them into chances with softmax. Send the top queue only when its expected cost is strictly below 0.2.

For a top queue that is not fraud:

expected cost = (1 - chance of the top queue) + 4 * chance of fraud

For a top queue that is fraud:

expected cost = 1 - chance of fraud

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "raphaelmansuy/bank-routing-modernbert"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)

text = "Why was I charged extra when I paid in euros?"
batch = tokenizer(text, return_tensors="pt", truncation=True, max_length=64)
with torch.no_grad():
    logits = model(**batch).logits
probs = torch.softmax(logits / 1.455, dim=-1)[0]
label = model.config.id2label[int(probs.argmax())]
print(label, round(float(probs.max()), 4))

An ONNX export of the same model is in onnx/. On 300 messages it picks the same queue as this file every time.

Training

• Base: answerdotai/ModernBERT-base, revision 8949b909ec900327062f0ebf497f51aef5e6f0c8
• Data: Banking77 (PolyAI, CC-BY-4.0), mapped from 77 intents to the six queues. 3,000 train, 1,000 validation, 2,379 test. Seed 42.
• Two epochs, learning rate 5e-5, max length 64, best checkpoint by validation macro-F1.
• Trained on a Modal L4 in bf16.
• Test accuracy 0.942, macro-F1 0.941, ECE 0.012 after temperature scaling.

Licence

The base model is Apache-2.0. Banking77 is CC-BY-4.0. Cite Casanueva et al., "Efficient Intent Detection with Dual Sentence Encoders", arXiv 2003.04807.