clef-flash-cfpb-complaints

Base model: Cloudflare/clef-flash @ f0b7d3cd65296abfe459e6c228b618f5d8b26d2f

Fine-tuned with clef-tune 0.0.0+gaf1a9a4708f2.

Train rows: 3000; eval rows: 300.

Results (eval set)

question	metric	baseline	fine-tuned
issue	accuracy	0.367	0.667
issue	macro-F1	0.325	0.573
issue	log-loss	1.691	1.005
product	accuracy	0.783	0.857
product	macro-F1	0.756	0.821
product	log-loss	0.554	0.390
outcome	accuracy	0.547	0.707
outcome	macro-F1	0.325	0.513
outcome	log-loss	1.242	0.691
all	exact match	0.183	0.447

Confusion matrices

issue

true \ predicted	card features or terms	closing an account	disputed purchase	fraud or scam	managing an account	opening an account	other	unwanted charge
card features or terms	1	0	1	0	0	0	9	3
closing an account	1	15	1	1	4	3	1	0
disputed purchase	0	2	38	0	3	0	1	0
fraud or scam	0	0	0	13	2	0	2	0
managing an account	0	4	6	2	71	3	7	6
opening an account	0	1	1	1	3	10	3	0
other	4	2	4	1	9	0	46	1
unwanted charge	0	0	1	1	6	0	0	6

product

true \ predicted	checking or savings account	credit card	money transfer
checking or savings account	141	10	8
credit card	13	92	0
money transfer	12	0	24

outcome

true \ predicted	explanation only	fix without money	money back
explanation only	147	0	35
fix without money	11	1	1
money back	41	0	64

Calibration

question	temperature	ECE before	ECE after	log-loss before	log-loss after
issue	1.311	0.124	0.085	1.005	0.970
product	1.291	0.042	0.012	0.390	0.375
outcome	1.215	0.083	0.067	0.691	0.682

Temperatures were fit on the same eval set used to choose the checkpoint, so post-calibration ECE and log-loss are slightly optimistic.

Run

Estimated time: 0.7 h ($1.85)

Actual time: 0.7 h ($2.02)

Benchmark

Trained on hvgupta1/cfpb-bank-complaints-benchmark: real complaints to a large US bank from the CFPB. Tested on 378 complaints received in July 2026, never used in training.

	All three right	Issue	Product	Outcome
This model	44.4% (39.2–49.7)	69.8%	91.0%	65.3%
The same model out of the box	19.6% (15.6–23.8)	43.7%	86.5%	51.1%
Claude Opus 5.5, with 30 examples	41.3% (36.2–46.3)	60.8%	91.3%	68.0%
Claude Opus 5.5, instructions only	33.3% (28.6–37.8)	56.1%	91.8%	64.0%

Method, costs, speed and limits: benchmarks/README.md.