GUE_EMP_H3-seqsight_4096_512_27M-L32_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_4096_512_27M on the mahdibaghbanzadeh/GUE_EMP_H3 dataset. It achieves the following results on the evaluation set:

• Loss: 0.2835
• F1 Score: 0.8864
• Accuracy: 0.8864

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.4145	2.13	200	0.3614	0.8358	0.8370
0.2922	4.26	400	0.3062	0.8737	0.8737
0.2637	6.38	600	0.3086	0.8717	0.8717
0.2479	8.51	800	0.2954	0.8817	0.8818
0.2313	10.64	1000	0.3000	0.8844	0.8844
0.2154	12.77	1200	0.3156	0.8777	0.8778
0.2039	14.89	1400	0.3139	0.8724	0.8724
0.1896	17.02	1600	0.3125	0.8784	0.8784
0.1743	19.15	1800	0.3347	0.8751	0.8751
0.1602	21.28	2000	0.3601	0.8797	0.8798
0.1542	23.4	2200	0.3792	0.8743	0.8744
0.136	25.53	2400	0.3798	0.8784	0.8784
0.1269	27.66	2600	0.4045	0.8670	0.8671
0.1183	29.79	2800	0.4342	0.8629	0.8631
0.1038	31.91	3000	0.4230	0.8651	0.8651
0.0982	34.04	3200	0.4496	0.8584	0.8584
0.0884	36.17	3400	0.4520	0.8718	0.8717
0.0818	38.3	3600	0.4904	0.8656	0.8657
0.0748	40.43	3800	0.4968	0.8622	0.8624
0.0697	42.55	4000	0.5272	0.8737	0.8737
0.0603	44.68	4200	0.5579	0.8564	0.8564
0.0584	46.81	4400	0.5943	0.8636	0.8637
0.0573	48.94	4600	0.5655	0.8704	0.8704
0.0512	51.06	4800	0.5970	0.8743	0.8744
0.0466	53.19	5000	0.6273	0.8703	0.8704
0.0448	55.32	5200	0.6674	0.8723	0.8724
0.0429	57.45	5400	0.6685	0.8689	0.8691
0.0402	59.57	5600	0.6652	0.8691	0.8691
0.0407	61.7	5800	0.6661	0.8717	0.8717
0.037	63.83	6000	0.7372	0.8622	0.8624
0.0334	65.96	6200	0.6942	0.8663	0.8664
0.0308	68.09	6400	0.6933	0.8730	0.8731
0.0302	70.21	6600	0.7081	0.8757	0.8758
0.029	72.34	6800	0.7236	0.8757	0.8758
0.0287	74.47	7000	0.7465	0.8704	0.8704
0.0249	76.6	7200	0.7735	0.8763	0.8764
0.0283	78.72	7400	0.7489	0.8744	0.8744
0.0262	80.85	7600	0.7690	0.8689	0.8691
0.0249	82.98	7800	0.7440	0.8744	0.8744
0.0244	85.11	8000	0.7504	0.8689	0.8691
0.0218	87.23	8200	0.7853	0.8697	0.8697
0.0222	89.36	8400	0.7698	0.8730	0.8731
0.0202	91.49	8600	0.7779	0.8764	0.8764
0.0194	93.62	8800	0.7931	0.8737	0.8737
0.0191	95.74	9000	0.7939	0.8757	0.8758
0.0199	97.87	9200	0.7886	0.8717	0.8717
0.0165	100.0	9400	0.8108	0.8670	0.8671
0.0197	102.13	9600	0.8093	0.8717	0.8717
0.0178	104.26	9800	0.7966	0.8724	0.8724
0.0151	106.38	10000	0.7996	0.8737	0.8737

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2