takuphilchan/banking77-distilbert

DistilBERT BANKING77 intent classifier

This model classifies short English online-banking support requests into one of the 77 intents defined by BANKING77. It is a learning and research artifact, not a banking decision system.

Results

The original BANKING77 test split was kept out of training and model selection.

Split	Examples	Accuracy	Macro-F1	Loss
Validation	1,001	0.8751	0.8580	0.6973
Test	3,080	0.8662	0.8603	—

The test set is balanced with 40 examples per intent. Aggregate performance hides an important failure: virtual_card_not_working received 0.00 F1. Of its 40 test examples, 19 were confused with getting_virtual_card and 17 with get_disposable_virtual_card. Other common confusions included why_verify_identity versus verify_my_identity, and card_acceptance versus order_physical_card.

Examples from a manual sanity check:

Input	Top prediction	Score
My cash withdrawal was declined	declined_cash_withdrawal	0.7007
My card has not arrived	card_arrival	0.6600
I forgot my passcode	passcode_forgotten	0.8411

Scores are softmax probabilities, not calibrated confidence estimates.

Training data

• Original dataset: PolyAI/banking77
• Parquet mirror loaded by the training script: DeepPavlov/banking77
• Upstream data license: CC BY 4.0
• Original splits: 10,003 train / 3,080 test
• Working splits: 9,002 train / 1,001 stratified validation / 3,080 untouched test
• Validation seed: 42

The original BANKING77 dataset and authors should be cited. The dataset card does not fully document collection demographics, personal-information review, or social biases.

Training procedure

• Base model: distilbert/distilbert-base-uncased
• Method: full sequence-classification fine-tuning
• Epochs: 5
• Batch size: 32
• Learning rate: 2e-5
• Scheduler: linear with 10% warmup
• Weight decay: 0.01
• Maximum sequence length: 128
• Best-checkpoint metric: validation macro-F1
• Precision: BF16 mixed precision
• Seed: 42
• Hardware: NVIDIA GeForce RTX 5060 Laptop GPU, 8 GiB VRAM
• Transformers: 5.17.0

The run completed 1,410 optimizer steps in approximately 55 seconds. The model was selected using validation macro-F1 and evaluated once on the held-out test split.

Usage

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="takuphilchan/banking77-distilbert",
)
print(classifier("My cash withdrawal was declined", top_k=3))

The saved id2label and label2id mappings contain all 77 human-readable intent names.

Intended use and limitations

Intended for demonstrations and research involving English banking-intent classification. It is not suitable for financial advice, account actions, fraud determinations, or autonomous customer support.

Important limitations:

• Inputs outside the 77 known intents are still forced into one of those classes.
• The softmax scores are not calibrated and should not be treated as reliable confidence values.
• Performance is English-only and specific to BANKING77's banking taxonomy.
• Some closely related virtual-card, identity-verification, transfer, and card-ordering intents remain difficult.
• Production use requires current institution-specific data, privacy review, an out-of-domain rejection mechanism, monitoring, and human escalation.
• DistilBERT can inherit biases from its BookCorpus and English Wikipedia pretraining data.

Reproducibility

The training and evaluation code is in scripts/train_classifier.py in the accompanying project. The model was trained with:

uv run python scripts\train_classifier.py `
  --model D:\huggingface-cache\git-distilbert-base-uncased `
  --base-model-id distilbert/distilbert-base-uncased `
  --batch-size 32 `
  --eval-batch-size 128 `
  --output-dir outputs\banking77-distilbert

Detailed per-class metrics and the complete confusion matrix are included in evaluation.json.

Citation

@inproceedings{casanueva-etal-2020-efficient,
  title = {Efficient Intent Detection with Dual Sentence Encoders},
  author = {Casanueva, I{\~n}igo and Tem{\v{c}}inas, Tadas and Gerz, Daniela and
            Henderson, Matthew and Vuli{\'c}, Ivan},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for
               Conversational AI},
  year = {2020}
}