GUE_EMP_H4-seqsight_4096_512_27M-L32_f

This model is a fine-tuned version of mahdibaghbanzadeh/seqsight_4096_512_27M on the mahdibaghbanzadeh/GUE_EMP_H4 dataset. It achieves the following results on the evaluation set:

• Loss: 0.2609
• F1 Score: 0.8964
• Accuracy: 0.8966

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 128
• eval_batch_size: 128
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• training_steps: 10000

Training results

Training Loss	Epoch	Step	Validation Loss	F1 Score	Accuracy
0.3346	2.17	200	0.2842	0.8954	0.8953
0.2615	4.35	400	0.2693	0.8966	0.8966
0.2455	6.52	600	0.2684	0.9027	0.9028
0.2352	8.7	800	0.2805	0.8941	0.8939
0.2138	10.87	1000	0.2761	0.8947	0.8946
0.2049	13.04	1200	0.2838	0.8947	0.8946
0.187	15.22	1400	0.2915	0.8947	0.8946
0.172	17.39	1600	0.3155	0.8902	0.8898
0.1588	19.57	1800	0.3204	0.8877	0.8877
0.1468	21.74	2000	0.3266	0.8845	0.8843
0.1319	23.91	2200	0.3453	0.8796	0.8795
0.1229	26.09	2400	0.3427	0.8773	0.8775
0.1106	28.26	2600	0.3987	0.8792	0.8795
0.0982	30.43	2800	0.4070	0.8755	0.8754
0.0862	32.61	3000	0.4562	0.8757	0.8761
0.0801	34.78	3200	0.4331	0.8803	0.8802
0.0736	36.96	3400	0.4788	0.8724	0.8727
0.0631	39.13	3600	0.5258	0.8651	0.8652
0.0566	41.3	3800	0.5171	0.8741	0.8741
0.0535	43.48	4000	0.5513	0.8626	0.8624
0.0484	45.65	4200	0.5790	0.8693	0.8700
0.0444	47.83	4400	0.6137	0.8707	0.8706
0.041	50.0	4600	0.6488	0.8736	0.8741
0.0412	52.17	4800	0.6552	0.8739	0.8741
0.0336	54.35	5000	0.6804	0.8722	0.8727
0.0355	56.52	5200	0.6545	0.8743	0.8741
0.033	58.7	5400	0.6452	0.8725	0.8727
0.0274	60.87	5600	0.6867	0.8798	0.8795
0.0294	63.04	5800	0.6560	0.8784	0.8782
0.0287	65.22	6000	0.6701	0.8878	0.8877
0.0226	67.39	6200	0.6983	0.8748	0.8747
0.0266	69.57	6400	0.6277	0.8829	0.8830
0.0245	71.74	6600	0.7203	0.8772	0.8775
0.0231	73.91	6800	0.7011	0.8754	0.8754
0.0205	76.09	7000	0.7072	0.8795	0.8795
0.0198	78.26	7200	0.7095	0.8733	0.8734
0.0217	80.43	7400	0.7206	0.8803	0.8802
0.0194	82.61	7600	0.7410	0.8759	0.8761
0.021	84.78	7800	0.7345	0.8788	0.8789
0.018	86.96	8000	0.7149	0.8755	0.8754
0.0171	89.13	8200	0.7380	0.8761	0.8761
0.0169	91.3	8400	0.7260	0.8766	0.8768
0.0142	93.48	8600	0.7683	0.8725	0.8727
0.0141	95.65	8800	0.7640	0.8803	0.8802
0.0141	97.83	9000	0.7762	0.8776	0.8775
0.0126	100.0	9200	0.8161	0.8768	0.8768
0.0146	102.17	9400	0.8132	0.8787	0.8789
0.0121	104.35	9600	0.8014	0.8754	0.8754
0.0118	106.52	9800	0.8046	0.8794	0.8795
0.0145	108.7	10000	0.8003	0.8787	0.8789

Framework versions

• PEFT 0.9.0
• Transformers 4.38.2
• Pytorch 2.2.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2