GUE_EMP_H3-seqsight_16384_512_34M-L32_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_16384_512_34M on the mahdibaghbanzadeh/GUE_EMP_H3 dataset. It achieves the following results on the evaluation set:

• Loss: 0.3581
• F1 Score: 0.8804
• Accuracy: 0.8804

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.4331	2.13	200	0.3621	0.8516	0.8517
0.2958	4.26	400	0.3346	0.8684	0.8684
0.2657	6.38	600	0.3485	0.8597	0.8597
0.2523	8.51	800	0.3201	0.8731	0.8731
0.2368	10.64	1000	0.3464	0.8653	0.8657
0.2238	12.77	1200	0.3192	0.8771	0.8771
0.2103	14.89	1400	0.3609	0.8567	0.8570
0.1988	17.02	1600	0.3454	0.8771	0.8771
0.185	19.15	1800	0.3556	0.8764	0.8764
0.1694	21.28	2000	0.3875	0.8736	0.8737
0.1635	23.4	2200	0.3822	0.8731	0.8731
0.1469	25.53	2400	0.3999	0.8804	0.8804
0.1385	27.66	2600	0.4115	0.8677	0.8677
0.1288	29.79	2800	0.4386	0.8634	0.8637
0.1228	31.91	3000	0.4146	0.8643	0.8644
0.1082	34.04	3200	0.4470	0.8670	0.8671
0.103	36.17	3400	0.4991	0.8519	0.8524
0.0932	38.3	3600	0.5066	0.8657	0.8657
0.0896	40.43	3800	0.5448	0.8640	0.8644
0.0826	42.55	4000	0.6343	0.8518	0.8524
0.0738	44.68	4200	0.5258	0.8710	0.8711
0.072	46.81	4400	0.5121	0.8711	0.8711
0.0696	48.94	4600	0.5634	0.8683	0.8684
0.0647	51.06	4800	0.5905	0.8643	0.8644
0.0609	53.19	5000	0.6529	0.8588	0.8591
0.0559	55.32	5200	0.5790	0.8751	0.8751
0.0521	57.45	5400	0.6104	0.8716	0.8717
0.0484	59.57	5600	0.6275	0.8723	0.8724
0.048	61.7	5800	0.6447	0.8622	0.8624
0.0437	63.83	6000	0.7093	0.8578	0.8584
0.0476	65.96	6200	0.6825	0.8702	0.8704
0.0394	68.09	6400	0.6768	0.8690	0.8691
0.0404	70.21	6600	0.6940	0.8702	0.8704
0.0373	72.34	6800	0.6746	0.8751	0.8751
0.0381	74.47	7000	0.7295	0.8607	0.8611
0.0348	76.6	7200	0.7110	0.8757	0.8758
0.0318	78.72	7400	0.7322	0.8703	0.8704
0.029	80.85	7600	0.8020	0.8642	0.8644
0.0314	82.98	7800	0.7269	0.8737	0.8737
0.0283	85.11	8000	0.7380	0.8737	0.8737
0.0272	87.23	8200	0.7716	0.8710	0.8711
0.0237	89.36	8400	0.8220	0.8777	0.8778
0.0273	91.49	8600	0.7853	0.8716	0.8717
0.0255	93.62	8800	0.8045	0.8737	0.8737
0.0252	95.74	9000	0.8016	0.8723	0.8724
0.0233	97.87	9200	0.8163	0.8675	0.8677
0.023	100.0	9400	0.8253	0.8683	0.8684
0.022	102.13	9600	0.8238	0.8723	0.8724
0.0206	104.26	9800	0.8230	0.8703	0.8704
0.0221	106.38	10000	0.8229	0.8703	0.8704

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2