esm2_t130_150M-lora-classifier_2024-04-26_10-31-42

This model is a fine-tuned version of facebook/esm2_t30_150M_UR50D on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.8800
• Accuracy: 0.8945

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0008701568055793088
• train_batch_size: 28
• eval_batch_size: 28
• seed: 8893
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 90
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.659	1.0	55	0.6868	0.5820
0.5296	2.0	110	0.6101	0.6777
0.5532	3.0	165	0.5395	0.7402
0.505	4.0	220	0.3482	0.8613
0.1246	5.0	275	0.3447	0.8535
0.3953	6.0	330	0.2982	0.8789
0.169	7.0	385	0.3257	0.875
0.2096	8.0	440	0.4022	0.8438
0.2339	9.0	495	0.3090	0.8848
0.1876	10.0	550	0.2723	0.8926
0.0631	11.0	605	0.3260	0.875
0.2843	12.0	660	0.4296	0.8438
0.0502	13.0	715	0.3301	0.8945
0.185	14.0	770	0.3273	0.8984
0.098	15.0	825	0.3355	0.8945
0.2528	16.0	880	0.4209	0.8926
0.2165	17.0	935	0.4467	0.8809
0.5202	18.0	990	0.4732	0.8984
0.0746	19.0	1045	0.4913	0.8926
0.0107	20.0	1100	0.4292	0.8906
0.0263	21.0	1155	0.6244	0.8555
0.011	22.0	1210	0.5228	0.8809
0.093	23.0	1265	0.5670	0.8828
0.058	24.0	1320	0.8912	0.8379
0.0007	25.0	1375	1.0458	0.8594
0.082	26.0	1430	0.7712	0.8848
0.0172	27.0	1485	0.7106	0.8945
0.1386	28.0	1540	0.9548	0.8535
0.1145	29.0	1595	0.8496	0.8945
0.001	30.0	1650	0.9245	0.8770
0.0004	31.0	1705	0.9368	0.8867
0.0039	32.0	1760	0.9754	0.8828
0.0165	33.0	1815	1.0050	0.8770
0.0028	34.0	1870	1.0051	0.8848
0.0006	35.0	1925	0.9673	0.8652
0.0	36.0	1980	0.9794	0.8906
0.0	37.0	2035	0.9294	0.8984
0.1004	38.0	2090	0.9621	0.8965
0.0	39.0	2145	0.9699	0.8965
0.0001	40.0	2200	1.0551	0.8926
0.0001	41.0	2255	0.9521	0.8965
0.1139	42.0	2310	1.0807	0.8828
0.0291	43.0	2365	0.9925	0.8965
0.0001	44.0	2420	1.0462	0.8867
0.0001	45.0	2475	0.9989	0.8848
0.0	46.0	2530	0.9005	0.8945
0.0005	47.0	2585	1.0845	0.8809
0.0	48.0	2640	0.9892	0.8965
0.0001	49.0	2695	0.9311	0.8887
0.0	50.0	2750	0.9819	0.8887
0.0	51.0	2805	1.0463	0.8887
0.0	52.0	2860	1.0672	0.8828
0.0	53.0	2915	1.0893	0.8926
0.0	54.0	2970	1.1496	0.8848
0.0002	55.0	3025	1.1330	0.8809
0.0009	56.0	3080	1.0782	0.8828
0.0046	57.0	3135	0.9937	0.8887
0.0009	58.0	3190	0.9710	0.8945
0.001	59.0	3245	1.0381	0.8848
0.0001	60.0	3300	0.9837	0.8887
0.0	61.0	3355	0.9552	0.8926
0.0002	62.0	3410	1.0600	0.8730
0.0	63.0	3465	0.9684	0.8887
0.0	64.0	3520	0.9498	0.8926
0.0003	65.0	3575	0.9644	0.8926
0.0	66.0	3630	0.9054	0.8887
0.0	67.0	3685	0.9370	0.8945
0.0001	68.0	3740	1.0082	0.8789
0.0001	69.0	3795	0.9378	0.8945
0.0048	70.0	3850	0.9371	0.8945
0.0002	71.0	3905	1.0431	0.8730
0.0007	72.0	3960	0.9235	0.8828
0.0011	73.0	4015	0.9624	0.8867
0.0	74.0	4070	0.9465	0.8926
0.0	75.0	4125	0.9266	0.8906
0.0	76.0	4180	0.9872	0.8867
0.0	77.0	4235	0.9488	0.8887
0.0002	78.0	4290	0.9376	0.8906
0.0	79.0	4345	0.9632	0.8867
0.0001	80.0	4400	0.9373	0.8926
0.0001	81.0	4455	0.9352	0.8848
0.0	82.0	4510	0.8856	0.8906
0.0001	83.0	4565	0.8813	0.8926
0.0001	84.0	4620	0.8822	0.8887
0.0	85.0	4675	0.8911	0.8887
0.0	86.0	4730	0.8834	0.8945
0.0001	87.0	4785	0.8747	0.8945
0.0	88.0	4840	0.8823	0.8926
0.0	89.0	4895	0.8824	0.8926
0.0	90.0	4950	0.8800	0.8945

Framework versions

• PEFT 0.10.0
• Transformers 4.39.3
• Pytorch 2.2.1
• Datasets 2.16.1
• Tokenizers 0.15.2