GUE_EMP_H4-seqsight_16384_512_56M-L8_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_16384_512_56M on the mahdibaghbanzadeh/GUE_EMP_H4 dataset. It achieves the following results on the evaluation set:

• Loss: 0.2737
• F1 Score: 0.9018
• Accuracy: 0.9021

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.3138	2.17	200	0.2870	0.8880	0.8877
0.2636	4.35	400	0.2759	0.8928	0.8925
0.2524	6.52	600	0.2649	0.8966	0.8966
0.2474	8.7	800	0.2766	0.8920	0.8919
0.2339	10.87	1000	0.2621	0.8897	0.8898
0.2282	13.04	1200	0.2823	0.8902	0.8898
0.2171	15.22	1400	0.2686	0.8955	0.8953
0.2085	17.39	1600	0.2772	0.8867	0.8864
0.2012	19.57	1800	0.2622	0.8958	0.8960
0.1931	21.74	2000	0.2746	0.8921	0.8919
0.1857	23.91	2200	0.2753	0.8950	0.8953
0.1829	26.09	2400	0.2679	0.8979	0.8980
0.173	28.26	2600	0.2834	0.8990	0.8994
0.1662	30.43	2800	0.2865	0.8966	0.8966
0.1585	32.61	3000	0.3245	0.8896	0.8905
0.1559	34.78	3200	0.3056	0.8907	0.8912
0.1499	36.96	3400	0.3101	0.8977	0.8980
0.1486	39.13	3600	0.2958	0.8984	0.8987
0.1419	41.3	3800	0.3143	0.8946	0.8946
0.1337	43.48	4000	0.3392	0.8877	0.8877
0.1375	45.65	4200	0.3398	0.8809	0.8816
0.1284	47.83	4400	0.3472	0.8835	0.8836
0.1238	50.0	4600	0.3613	0.8828	0.8836
0.1218	52.17	4800	0.3771	0.8831	0.8836
0.1196	54.35	5000	0.3853	0.8728	0.8734
0.1153	56.52	5200	0.3680	0.8841	0.8843
0.1127	58.7	5400	0.3492	0.8856	0.8857
0.1052	60.87	5600	0.3919	0.8751	0.8754
0.1057	63.04	5800	0.3935	0.8775	0.8775
0.1031	65.22	6000	0.4049	0.8781	0.8789
0.0991	67.39	6200	0.3886	0.8856	0.8857
0.0964	69.57	6400	0.3824	0.8787	0.8789
0.0955	71.74	6600	0.4175	0.8820	0.8823
0.0929	73.91	6800	0.4135	0.8833	0.8836
0.0905	76.09	7000	0.4160	0.8828	0.8830
0.0908	78.26	7200	0.4075	0.8790	0.8795
0.0873	80.43	7400	0.4152	0.8815	0.8816
0.0867	82.61	7600	0.4671	0.8724	0.8734
0.0867	84.78	7800	0.4273	0.8847	0.8850
0.0834	86.96	8000	0.4327	0.8799	0.8802
0.0809	89.13	8200	0.4389	0.8800	0.8802
0.0784	91.3	8400	0.4524	0.8738	0.8741
0.0773	93.48	8600	0.4755	0.8790	0.8795
0.0767	95.65	8800	0.4662	0.8825	0.8830
0.0781	97.83	9000	0.4542	0.8827	0.8830
0.0769	100.0	9200	0.4575	0.8774	0.8775
0.0726	102.17	9400	0.4654	0.8806	0.8809
0.074	104.35	9600	0.4733	0.8779	0.8782
0.0739	106.52	9800	0.4757	0.8770	0.8775
0.072	108.7	10000	0.4706	0.8792	0.8795

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2