GUE_EMP_H3-seqsight_16384_512_56M-L8_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_16384_512_56M on the mahdibaghbanzadeh/GUE_EMP_H3 dataset. It achieves the following results on the evaluation set:

• Loss: 0.3045
• F1 Score: 0.8824
• Accuracy: 0.8824

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.4046	2.13	200	0.3698	0.8461	0.8464
0.3108	4.26	400	0.3428	0.8563	0.8564
0.2721	6.38	600	0.3552	0.8549	0.8550
0.2588	8.51	800	0.3115	0.8724	0.8724
0.2456	10.64	1000	0.3570	0.8559	0.8564
0.2343	12.77	1200	0.3222	0.8771	0.8771
0.2271	14.89	1400	0.3434	0.8655	0.8657
0.2169	17.02	1600	0.3267	0.8831	0.8831
0.2137	19.15	1800	0.3258	0.8778	0.8778
0.2015	21.28	2000	0.3579	0.8688	0.8691
0.2021	23.4	2200	0.3488	0.8769	0.8771
0.1873	25.53	2400	0.3769	0.8715	0.8717
0.1908	27.66	2600	0.3619	0.8674	0.8677
0.1793	29.79	2800	0.3864	0.8706	0.8711
0.1767	31.91	3000	0.3573	0.8797	0.8798
0.171	34.04	3200	0.3449	0.8811	0.8811
0.1678	36.17	3400	0.4275	0.8617	0.8624
0.1595	38.3	3600	0.4030	0.8701	0.8704
0.1558	40.43	3800	0.4725	0.8547	0.8557
0.1512	42.55	4000	0.4683	0.8578	0.8584
0.1473	44.68	4200	0.4366	0.8620	0.8624
0.1421	46.81	4400	0.4197	0.8708	0.8711
0.1394	48.94	4600	0.4501	0.8598	0.8604
0.1374	51.06	4800	0.4113	0.8749	0.8751
0.1323	53.19	5000	0.4698	0.8654	0.8657
0.1287	55.32	5200	0.4620	0.8648	0.8651
0.1272	57.45	5400	0.5108	0.8611	0.8617
0.119	59.57	5600	0.5212	0.8606	0.8611
0.1202	61.7	5800	0.4716	0.8694	0.8697
0.1156	63.83	6000	0.5120	0.8605	0.8611
0.1118	65.96	6200	0.5179	0.8619	0.8624
0.1127	68.09	6400	0.5186	0.8571	0.8577
0.1044	70.21	6600	0.6003	0.8523	0.8530
0.1059	72.34	6800	0.5264	0.8626	0.8631
0.1045	74.47	7000	0.5904	0.8529	0.8537
0.0996	76.6	7200	0.5376	0.8660	0.8664
0.0991	78.72	7400	0.5570	0.8646	0.8651
0.0966	80.85	7600	0.5589	0.8646	0.8651
0.0975	82.98	7800	0.5842	0.8619	0.8624
0.0927	85.11	8000	0.6082	0.8584	0.8591
0.0912	87.23	8200	0.6212	0.8598	0.8604
0.0952	89.36	8400	0.6192	0.8543	0.8550
0.09	91.49	8600	0.6004	0.8598	0.8604
0.0891	93.62	8800	0.6050	0.8626	0.8631
0.0882	95.74	9000	0.6315	0.8584	0.8591
0.0857	97.87	9200	0.6263	0.8578	0.8584
0.0872	100.0	9400	0.6448	0.8550	0.8557
0.0849	102.13	9600	0.6521	0.8543	0.8550
0.0834	104.26	9800	0.6395	0.8577	0.8584
0.0853	106.38	10000	0.6370	0.8570	0.8577

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2