GUE_EMP_H4-seqsight_16384_512_56M-L32_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_16384_512_56M on the mahdibaghbanzadeh/GUE_EMP_H4 dataset. It achieves the following results on the evaluation set:

• Loss: 0.2550
• F1 Score: 0.9006
• Accuracy: 0.9008

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.307	2.17	200	0.2865	0.8908	0.8905
0.258	4.35	400	0.2702	0.8904	0.8905
0.241	6.52	600	0.2569	0.8986	0.8987
0.2272	8.7	800	0.2783	0.8888	0.8884
0.2056	10.87	1000	0.2594	0.9048	0.9049
0.1931	13.04	1200	0.2890	0.8887	0.8884
0.1742	15.22	1400	0.2875	0.8975	0.8973
0.1601	17.39	1600	0.3076	0.8901	0.8898
0.1488	19.57	1800	0.3107	0.8916	0.8919
0.1382	21.74	2000	0.3345	0.8918	0.8919
0.1195	23.91	2200	0.3596	0.8890	0.8891
0.1125	26.09	2400	0.3816	0.8912	0.8912
0.1	28.26	2600	0.4127	0.8835	0.8836
0.0893	30.43	2800	0.4338	0.8850	0.8850
0.0802	32.61	3000	0.4783	0.8773	0.8782
0.0735	34.78	3200	0.4466	0.8735	0.8741
0.0695	36.96	3400	0.4774	0.8773	0.8775
0.0586	39.13	3600	0.5263	0.8751	0.8754
0.0569	41.3	3800	0.5288	0.8730	0.8727
0.0496	43.48	4000	0.6031	0.8752	0.8747
0.0486	45.65	4200	0.5492	0.8718	0.8720
0.0391	47.83	4400	0.5965	0.8761	0.8761
0.0374	50.0	4600	0.6584	0.8742	0.8747
0.036	52.17	4800	0.6468	0.8813	0.8816
0.032	54.35	5000	0.6886	0.8851	0.8850
0.0304	56.52	5200	0.6704	0.8845	0.8843
0.0298	58.7	5400	0.6396	0.8810	0.8809
0.0252	60.87	5600	0.6969	0.8839	0.8836
0.0253	63.04	5800	0.6920	0.8768	0.8768
0.0222	65.22	6000	0.7377	0.8810	0.8809
0.0229	67.39	6200	0.7602	0.8731	0.8727
0.0213	69.57	6400	0.7484	0.8762	0.8761
0.0223	71.74	6600	0.7040	0.8843	0.8843
0.0189	73.91	6800	0.7103	0.8817	0.8816
0.0156	76.09	7000	0.8209	0.8806	0.8802
0.0185	78.26	7200	0.7703	0.8811	0.8809
0.0164	80.43	7400	0.7721	0.8824	0.8823
0.0165	82.61	7600	0.7630	0.8778	0.8782
0.0147	84.78	7800	0.7728	0.8845	0.8843
0.0145	86.96	8000	0.7902	0.8743	0.8741
0.0127	89.13	8200	0.8076	0.8784	0.8782
0.0131	91.3	8400	0.8044	0.8858	0.8857
0.0118	93.48	8600	0.8129	0.8817	0.8816
0.0124	95.65	8800	0.7860	0.8823	0.8823
0.01	97.83	9000	0.8226	0.8866	0.8864
0.0112	100.0	9200	0.8501	0.8812	0.8809
0.0112	102.17	9400	0.8284	0.8879	0.8877
0.0107	104.35	9600	0.8299	0.8872	0.8871
0.0096	106.52	9800	0.8253	0.8822	0.8823
0.01	108.7	10000	0.8320	0.8865	0.8864

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2