GUE_EMP_H3-seqsight_4096_512_46M-L32_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_4096_512_46M on the mahdibaghbanzadeh/GUE_EMP_H3 dataset. It achieves the following results on the evaluation set:

• Loss: 0.4633
• F1 Score: 0.8771
• Accuracy: 0.8771

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.3604	2.13	200	0.3293	0.8603	0.8604
0.2678	4.26	400	0.3065	0.8677	0.8677
0.245	6.38	600	0.3205	0.8777	0.8778
0.2278	8.51	800	0.2854	0.8778	0.8778
0.2116	10.64	1000	0.3084	0.8809	0.8811
0.1953	12.77	1200	0.3061	0.8824	0.8824
0.1831	14.89	1400	0.3468	0.8749	0.8751
0.1698	17.02	1600	0.3223	0.8844	0.8844
0.1547	19.15	1800	0.3590	0.8717	0.8717
0.1414	21.28	2000	0.3977	0.8729	0.8731
0.1282	23.4	2200	0.4078	0.8704	0.8704
0.1155	25.53	2400	0.4301	0.8844	0.8844
0.1067	27.66	2600	0.4789	0.8600	0.8604
0.0901	29.79	2800	0.5379	0.8539	0.8544
0.087	31.91	3000	0.4809	0.8662	0.8664
0.0801	34.04	3200	0.4592	0.8664	0.8664
0.0671	36.17	3400	0.5474	0.8640	0.8644
0.0605	38.3	3600	0.5633	0.8716	0.8717
0.058	40.43	3800	0.6149	0.8628	0.8631
0.0543	42.55	4000	0.5779	0.8743	0.8744
0.0495	44.68	4200	0.7113	0.8599	0.8604
0.0463	46.81	4400	0.7295	0.8591	0.8597
0.0458	48.94	4600	0.6465	0.8655	0.8657
0.0383	51.06	4800	0.6267	0.8704	0.8704
0.0353	53.19	5000	0.6600	0.8724	0.8724
0.0345	55.32	5200	0.7133	0.8648	0.8651
0.0321	57.45	5400	0.6932	0.8670	0.8671
0.0281	59.57	5600	0.7285	0.8697	0.8697
0.0305	61.7	5800	0.7504	0.8708	0.8711
0.0271	63.83	6000	0.7655	0.8703	0.8704
0.0224	65.96	6200	0.7983	0.8724	0.8724
0.0234	68.09	6400	0.8454	0.8709	0.8711
0.0209	70.21	6600	0.8179	0.8737	0.8737
0.0215	72.34	6800	0.8238	0.8735	0.8737
0.0203	74.47	7000	0.9210	0.8680	0.8684
0.0196	76.6	7200	0.8355	0.8730	0.8731
0.0183	78.72	7400	0.8203	0.8742	0.8744
0.0169	80.85	7600	0.9234	0.8689	0.8691
0.0169	82.98	7800	0.8332	0.8757	0.8758
0.016	85.11	8000	0.8825	0.8749	0.8751
0.0146	87.23	8200	0.9027	0.8708	0.8711
0.0145	89.36	8400	0.9219	0.8742	0.8744
0.0117	91.49	8600	0.9785	0.8696	0.8697
0.0134	93.62	8800	0.9348	0.8661	0.8664
0.0115	95.74	9000	0.9575	0.8709	0.8711
0.0126	97.87	9200	0.9394	0.8676	0.8677
0.0107	100.0	9400	0.9777	0.8662	0.8664
0.0113	102.13	9600	0.9691	0.8683	0.8684
0.0106	104.26	9800	0.9459	0.8703	0.8704
0.0105	106.38	10000	0.9487	0.8716	0.8717

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2