clef-cfpb-complaints

Base model: Cloudflare/clef @ a30162b99d68a481d8bd75d38df069a0cf83df40

Fine-tuned with clef-tune 0.0.0+gaf1a9a4708f2.

Train rows: 3000; eval rows: 300.

Results (eval set)

question	metric	baseline	fine-tuned
issue	accuracy	0.373	0.677
issue	macro-F1	0.325	0.599
issue	log-loss	1.981	0.940
product	accuracy	0.803	0.863
product	macro-F1	0.782	0.843
product	log-loss	0.533	0.340
outcome	accuracy	0.530	0.690
outcome	macro-F1	0.350	0.454
outcome	log-loss	1.177	0.647
all	exact match	0.180	0.460

Confusion matrices

issue

true \ predicted	card features or terms	closing an account	disputed purchase	fraud or scam	managing an account	opening an account	other	unwanted charge
card features or terms	3	1	2	0	1	0	5	2
closing an account	1	16	1	0	5	2	1	0
disputed purchase	1	1	36	0	2	0	2	2
fraud or scam	0	0	0	13	3	0	1	0
managing an account	0	6	6	1	74	1	5	6
opening an account	0	1	0	0	3	9	6	0
other	4	1	1	1	11	2	47	0
unwanted charge	0	0	1	0	8	0	0	5

product

true \ predicted	checking or savings account	credit card	money transfer
checking or savings account	143	10	6
credit card	16	89	0
money transfer	9	0	27

outcome

true \ predicted	explanation only	fix without money	money back
explanation only	146	0	36
fix without money	13	0	0
money back	44	0	61

Calibration

question	temperature	ECE before	ECE after	log-loss before	log-loss after
issue	1.187	0.090	0.081	0.940	0.927
product	1.173	0.061	0.044	0.340	0.334
outcome	1.163	0.060	0.046	0.647	0.643

Temperatures were fit on the same eval set used to choose the checkpoint, so post-calibration ECE and log-loss are slightly optimistic.

Run

Estimated time: 2.1 h ($5.81)

Actual time: 2.0 h ($5.47)

Benchmark

Trained on hvgupta1/cfpb-bank-complaints-benchmark: real complaints to a large US bank from the CFPB. Tested on 378 complaints received in July 2026, never used in training.

	All three right	Issue	Product	Outcome
This model	47.9% (42.9–52.9)	70.1%	92.6%	68.3%
The same model out of the box	17.7% (14.0–21.4)	38.1%	87.3%	52.4%
Claude Opus 5.5, with 30 examples	41.3% (36.2–46.3)	60.8%	91.3%	68.0%
Claude Opus 5.5, instructions only	33.3% (28.6–37.8)	56.1%	91.8%	64.0%

Method, costs, speed and limits: benchmarks/README.md.