banking77-intent-classifier

A 77-class banking intent classifier, fine-tuned from BAAI/bge-small-en-v1.5 on BANKING77.

Given a customer message such as "my card still hasn't arrived after two weeks", it predicts the intent (card_arrival) so the request can be routed automatically.

Results

Metric	Value
Macro F1	0.9245
Accuracy	0.9247
Top-3 accuracy	0.974
Inference	~0.34 ms per request

Evaluated on the official BANKING77 test split (3080 requests, 40 per intent), scored once, at the end. Model selection used a stratified 10 % validation split carved out of the training data.

An honest note on what this model is for

This model was beaten by a simpler approach, and that is the interesting part.

It was trained as the third rung of a deliberate ladder, to measure what fine-tuning actually buys over cheaper alternatives on this dataset:

Approach	Macro F1	Training cost
TF-IDF (word + char n-grams) → logistic regression	0.915	23 s, CPU
Frozen bge-small embeddings → logistic regression	0.935	54 s, CPU
This model — bge-small fine-tuned end to end	0.9245	~215 s, GPU

Using the same encoder frozen, with nothing but a logistic regression on top, scores higher. The gap held across five training runs spanning three random seeds, which scored between 0.9245 and 0.9307 (mean ≈ 0.927). Runs vary by a few tenths of a point even at a fixed seed, because GPU kernel scheduling and multi-worker data loading are not bit-deterministic — so the comparison rests on the spread of runs rather than on any single number.

Two plausible reasons:

1. bge-small is contrastively pre-trained for semantic similarity. Grouping semantically similar sentences is more or less what intent classification is, so its embedding space already arrives close to the right shape — and fine-tuning distorts a geometry that was already good.
2. 10 003 examples across 77 intents is roughly 130 per class. That is thin for updating 33 M parameters, and the model reaches a memorised training loss before it generalises further.

An earlier version of this model, trained without a validation split, drove training loss to 0.037 and scored 0.9295 — marginally higher than the properly regularised model published here. That version was overfit, and comparing it against a regularised alternative would have proved nothing. The lower, honest number is the one reported.

If you want the best model for this task, use frozen embeddings with a linear head. This checkpoint is published for reproducibility and as a documented negative result.

Usage

from transformers import pipeline

classifier = pipeline("text-classification", model="functionX86/banking77-intent-classifier")
classifier("my card still hasn't arrived after two weeks")
# [{'label': 'card_arrival', 'score': 0.98}]

Training

Setting	Value
Base model	BAAI/bge-small-en-v1.5 (33 M parameters)
Max sequence length	64 tokens
Epochs	up to 15, early stopping on validation macro-F1 (patience 3)
Batch size	32
Learning rate	5e-5, 10 % warmup, weight decay 0.01
Precision	fp16
Hardware	one NVIDIA RTX 3050 Ti (4 GB)

The 64-token cap comes from the data: the 95th percentile of BANKING77 requests is 29 words, so it truncates almost nothing while running roughly four times faster than the default 256.

Limitations

• English only, and trained on retail banking requests. It will not transfer to another domain without retraining.
• Several BANKING77 intents genuinely overlap — card_arrival vs card_delivery_estimate, top_up_failed vs top_up_reverted, and the whole identity-verification cluster. A share of the residual error is label ambiguity that no model can resolve.
• Raw softmax scores are not calibrated. For any use that depends on a confidence threshold, fit a temperature on held-out data first — on the frozen-embedding variant this reduced expected calibration error from 0.110 to 0.012 without changing a single prediction.
• Trained on public research data, not on real customer messages, and never evaluated for fairness across customer segments. Not suitable for production use as-is.