bigbird-base-fnd-v2

This model is a fine-tuned version of google/bigbird-roberta-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0186
• Accuracy: 0.9979
• Precision: 0.9985
• Recall: 0.9972
• F1: 0.9979

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 650
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1
2.5791	0.0509	200	0.1078	0.9659	0.9666	0.9651	0.9659
1.8145	0.1017	400	0.1081	0.9815	0.9891	0.9737	0.9813
3.1461	0.1526	600	0.0439	0.9876	0.9894	0.9857	0.9876
0.9117	0.2034	800	0.0721	0.9895	0.9929	0.9861	0.9895
1.3006	0.2543	1000	0.0571	0.9920	0.9924	0.9917	0.9920
1.1727	0.3051	1200	0.0634	0.9920	0.9898	0.9943	0.9920
1.3481	0.3560	1400	0.0368	0.9926	0.9882	0.9970	0.9926
0.7716	0.4069	1600	0.0385	0.9930	0.9915	0.9944	0.9930
1.0770	0.4577	1800	0.0307	0.9930	0.9900	0.9961	0.9931
0.5468	0.5086	2000	0.0229	0.9948	0.9963	0.9933	0.9948
0.3740	0.5594	2200	0.0349	0.9949	0.9961	0.9937	0.9949
0.4400	0.6103	2400	0.0426	0.9938	0.9906	0.9970	0.9938
0.4935	0.6611	2600	0.0338	0.9931	0.9915	0.9948	0.9932
0.6545	0.7120	2800	0.0333	0.9943	0.9922	0.9965	0.9944
1.0321	0.7628	3000	0.0335	0.9946	0.9948	0.9944	0.9946
0.3264	0.8137	3200	0.0226	0.9957	0.9970	0.9944	0.9957
0.7214	0.8646	3400	0.0429	0.9933	0.9913	0.9954	0.9933
0.3384	0.9154	3600	0.0301	0.9943	0.9906	0.9980	0.9943
0.9418	0.9663	3800	0.0362	0.9948	0.9993	0.9904	0.9948
0.5850	1.0170	4000	0.0326	0.9949	0.9908	0.9991	0.9949
0.0574	1.0679	4200	0.0307	0.9963	0.9945	0.9981	0.9963
0.3091	1.1188	4400	0.0308	0.9957	0.9970	0.9944	0.9957
0.1416	1.1696	4600	0.0261	0.9968	0.9965	0.9970	0.9968
1.0580	1.2205	4800	0.0233	0.9962	0.9965	0.9959	0.9962
0.0859	1.2713	5000	0.0243	0.9973	0.9991	0.9955	0.9973
0.2777	1.3222	5200	0.0286	0.9961	0.9946	0.9976	0.9961
0.2113	1.3730	5400	0.0249	0.9974	0.9987	0.9961	0.9974
0.5708	1.4239	5600	0.0426	0.9956	0.9989	0.9922	0.9955
0.2978	1.4747	5800	0.0244	0.9974	0.9974	0.9974	0.9974
0.4554	1.5256	6000	0.0183	0.9971	0.9968	0.9974	0.9971
0.4042	1.5765	6200	0.0183	0.9970	0.9978	0.9963	0.9970
0.6254	1.6273	6400	0.0223	0.9957	0.9956	0.9959	0.9957
1.0894	1.6782	6600	0.0249	0.9958	0.9972	0.9944	0.9958
0.4566	1.7290	6800	0.0227	0.9965	0.9978	0.9952	0.9965
0.1536	1.7799	7000	0.0218	0.9969	0.9961	0.9978	0.9969
0.5381	1.8307	7200	0.0307	0.9956	0.9946	0.9965	0.9956
0.7149	1.8816	7400	0.0195	0.9972	0.9978	0.9967	0.9972
0.2877	1.9325	7600	0.0233	0.9971	0.9987	0.9955	0.9971
0.2309	1.9833	7800	0.0222	0.9968	0.9978	0.9959	0.9968
0.1346	2.0341	8000	0.0215	0.9968	0.9968	0.9968	0.9968
0.0100	2.0849	8200	0.0199	0.9974	0.9976	0.9972	0.9974
0.2401	2.1358	8400	0.0296	0.9969	0.9957	0.9981	0.9969
0.0041	2.1866	8600	0.0224	0.9974	0.9983	0.9965	0.9974
0.0007	2.2375	8800	0.0202	0.9976	0.9978	0.9974	0.9976
0.3182	2.2884	9000	0.0242	0.9974	0.9972	0.9976	0.9974
0.0001	2.3392	9200	0.0225	0.9976	0.9976	0.9976	0.9976
0.1201	2.3901	9400	0.0186	0.9979	0.9981	0.9976	0.9979
0.0006	2.4409	9600	0.0219	0.9978	0.9987	0.9968	0.9978
0.0002	2.4918	9800	0.0209	0.9978	0.9980	0.9976	0.9978
0.1451	2.5426	10000	0.0203	0.9974	0.9981	0.9967	0.9974
0.0002	2.5935	10200	0.0181	0.9976	0.9970	0.9981	0.9976
0.0863	2.6444	10400	0.0187	0.9976	0.9980	0.9972	0.9976
0.0315	2.6952	10600	0.0197	0.9976	0.9972	0.9980	0.9976
0.2513	2.7461	10800	0.0192	0.9974	0.9978	0.9970	0.9974
0.0003	2.7969	11000	0.0169	0.9977	0.9974	0.9980	0.9977
0.0846	2.8478	11200	0.0188	0.9979	0.9985	0.9972	0.9979
0.0000	2.8986	11400	0.0210	0.9976	0.9985	0.9967	0.9976
0.0001	2.9495	11600	0.0207	0.9978	0.9985	0.9970	0.9978
0.1002	3.0	11799	0.0186	0.9979	0.9985	0.9972	0.9979

Framework versions

• Transformers 5.8.0
• Pytorch 2.12.0.dev20260407+cu128
• Datasets 4.8.5
• Tokenizers 0.22.2