GUE_EMP_H4-seqsight_16384_512_56M-L1_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_16384_512_56M on the mahdibaghbanzadeh/GUE_EMP_H4 dataset. It achieves the following results on the evaluation set:

• Loss: 0.2576
• F1 Score: 0.9083
• Accuracy: 0.9083

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.3259	2.17	200	0.2859	0.8885	0.8884
0.2744	4.35	400	0.2952	0.8874	0.8871
0.2658	6.52	600	0.2808	0.8907	0.8905
0.2644	8.7	800	0.2921	0.8922	0.8919
0.2556	10.87	1000	0.2708	0.8967	0.8966
0.253	13.04	1200	0.2768	0.8969	0.8966
0.2482	15.22	1400	0.2714	0.8913	0.8912
0.2444	17.39	1600	0.2728	0.8976	0.8973
0.2407	19.57	1800	0.2639	0.8932	0.8932
0.2397	21.74	2000	0.2797	0.8928	0.8925
0.2345	23.91	2200	0.2662	0.8975	0.8973
0.2327	26.09	2400	0.2734	0.8921	0.8919
0.2288	28.26	2600	0.2632	0.8953	0.8953
0.2254	30.43	2800	0.2632	0.8913	0.8912
0.2224	32.61	3000	0.2648	0.8945	0.8946
0.2193	34.78	3200	0.2640	0.8960	0.8960
0.2171	36.96	3400	0.2628	0.8960	0.8960
0.2162	39.13	3600	0.2616	0.8933	0.8932
0.2111	41.3	3800	0.2631	0.8993	0.8994
0.2072	43.48	4000	0.2666	0.8918	0.8919
0.2155	45.65	4200	0.2627	0.8972	0.8973
0.2039	47.83	4400	0.2622	0.8958	0.8960
0.2046	50.0	4600	0.2662	0.8936	0.8939
0.201	52.17	4800	0.2643	0.8978	0.8980
0.2031	54.35	5000	0.2653	0.8986	0.8987
0.1967	56.52	5200	0.2676	0.8974	0.8973
0.1968	58.7	5400	0.2658	0.8952	0.8953
0.1924	60.87	5600	0.2702	0.8972	0.8973
0.1914	63.04	5800	0.2702	0.8946	0.8946
0.1945	65.22	6000	0.2674	0.8992	0.8994
0.1906	67.39	6200	0.2662	0.8966	0.8966
0.1873	69.57	6400	0.2693	0.8971	0.8973
0.1881	71.74	6600	0.2693	0.8978	0.8980
0.186	73.91	6800	0.2660	0.8979	0.8980
0.184	76.09	7000	0.2678	0.9001	0.9001
0.1843	78.26	7200	0.2671	0.8972	0.8973
0.1847	80.43	7400	0.2657	0.8972	0.8973
0.1818	82.61	7600	0.2691	0.8957	0.8960
0.1842	84.78	7800	0.2678	0.8972	0.8973
0.1819	86.96	8000	0.2686	0.8950	0.8953
0.1822	89.13	8200	0.2681	0.8957	0.8960
0.1784	91.3	8400	0.2716	0.8936	0.8939
0.1759	93.48	8600	0.2760	0.8928	0.8932
0.179	95.65	8800	0.2755	0.8928	0.8932
0.1801	97.83	9000	0.2704	0.8943	0.8946
0.1782	100.0	9200	0.2700	0.8951	0.8953
0.1785	102.17	9400	0.2705	0.8936	0.8939
0.1781	104.35	9600	0.2707	0.8943	0.8946
0.1751	106.52	9800	0.2724	0.8935	0.8939
0.1759	108.7	10000	0.2719	0.8929	0.8932

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2