GUE_EMP_H3-seqsight_16384_512_34M-L8_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_16384_512_34M on the mahdibaghbanzadeh/GUE_EMP_H3 dataset. It achieves the following results on the evaluation set:

• Loss: 0.3629
• F1 Score: 0.8684
• Accuracy: 0.8684

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.4585	2.13	200	0.3940	0.8409	0.8410
0.333	4.26	400	0.3430	0.8630	0.8631
0.2899	6.38	600	0.3729	0.8493	0.8497
0.2754	8.51	800	0.3195	0.8724	0.8724
0.2631	10.64	1000	0.3234	0.8684	0.8684
0.2546	12.77	1200	0.3287	0.8664	0.8664
0.2443	14.89	1400	0.3515	0.8594	0.8597
0.2375	17.02	1600	0.3163	0.8751	0.8751
0.2288	19.15	1800	0.3348	0.8684	0.8684
0.2243	21.28	2000	0.3513	0.8676	0.8677
0.2227	23.4	2200	0.3344	0.8656	0.8657
0.2085	25.53	2400	0.3422	0.8697	0.8697
0.2133	27.66	2600	0.3310	0.8744	0.8744
0.2036	29.79	2800	0.3745	0.8633	0.8637
0.1974	31.91	3000	0.3421	0.8664	0.8664
0.1933	34.04	3200	0.3459	0.8784	0.8784
0.1895	36.17	3400	0.3762	0.8667	0.8671
0.1828	38.3	3600	0.3801	0.8622	0.8624
0.1804	40.43	3800	0.3669	0.8682	0.8684
0.1743	42.55	4000	0.4119	0.8606	0.8611
0.1694	44.68	4200	0.3770	0.8704	0.8704
0.1669	46.81	4400	0.3873	0.8648	0.8651
0.1685	48.94	4600	0.3926	0.8667	0.8671
0.1619	51.06	4800	0.3690	0.8744	0.8744
0.1612	53.19	5000	0.4081	0.8634	0.8637
0.1555	55.32	5200	0.3844	0.8791	0.8791
0.1526	57.45	5400	0.4042	0.8717	0.8717
0.1483	59.57	5600	0.4244	0.8622	0.8624
0.1484	61.7	5800	0.3813	0.8744	0.8744
0.1465	63.83	6000	0.4256	0.8695	0.8697
0.1434	65.96	6200	0.4202	0.8675	0.8677
0.1389	68.09	6400	0.4033	0.8764	0.8764
0.1388	70.21	6600	0.4336	0.8724	0.8724
0.135	72.34	6800	0.4049	0.8764	0.8764
0.135	74.47	7000	0.4618	0.8552	0.8557
0.13	76.6	7200	0.4369	0.8663	0.8664
0.1348	78.72	7400	0.4264	0.8757	0.8758
0.129	80.85	7600	0.4316	0.8677	0.8677
0.1231	82.98	7800	0.4316	0.8717	0.8717
0.1257	85.11	8000	0.4365	0.8744	0.8744
0.1228	87.23	8200	0.4485	0.8703	0.8704
0.1195	89.36	8400	0.4391	0.8763	0.8764
0.1201	91.49	8600	0.4615	0.8689	0.8691
0.1189	93.62	8800	0.4506	0.8763	0.8764
0.1203	95.74	9000	0.4538	0.8716	0.8717
0.1166	97.87	9200	0.4507	0.8737	0.8737
0.1178	100.0	9400	0.4551	0.8737	0.8737
0.1174	102.13	9600	0.4543	0.8730	0.8731
0.116	104.26	9800	0.4593	0.8696	0.8697
0.1141	106.38	10000	0.4573	0.8703	0.8704

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2