codet5p-770m-py-sanitized-codebleu-1-False-1e-05-0.1-lora

This model is a fine-tuned version of Salesforce/codet5p-770m-py on the mbpp dataset. It achieves the following results on the evaluation set:

• Loss: 0.8627
• Codebleu: 0.0882
• Ngram Match Score: 0.0055
• Weighted Ngram Match Score: 0.0171
• Syntax Match Score: 0.0936
• Dataflow Match Score: 0.1212

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 6
• eval_batch_size: 6
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Codebleu	Ngram Match Score	Weighted Ngram Match Score	Syntax Match Score	Dataflow Match Score
1.1536	1.0	20	1.1773	0.0570	0.0034	0.0105	0.0725	0.0666
1.1524	2.0	40	1.1735	0.0586	0.0049	0.0155	0.0713	0.0700
1.1551	3.0	60	1.1664	0.0622	0.0054	0.0156	0.0769	0.0734
1.1201	4.0	80	1.1548	0.0658	0.0052	0.0157	0.0792	0.0802
1.1365	5.0	100	1.1357	0.0656	0.0046	0.0142	0.0792	0.0802
1.0959	6.0	120	1.1074	0.0654	0.0045	0.0141	0.0736	0.0853
1.0749	7.0	140	1.0553	0.0876	0.0061	0.0188	0.1037	0.1092
1.0241	8.0	160	0.9992	0.0941	0.0081	0.0248	0.1093	0.1177
0.9816	9.0	180	0.9608	0.1002	0.0082	0.0247	0.1159	0.1263
0.9621	10.0	200	0.9370	0.1097	0.0099	0.0251	0.1171	0.1485
0.9475	11.0	220	0.9265	0.1077	0.0097	0.0252	0.1171	0.1433
0.9213	12.0	240	0.9187	0.1063	0.0095	0.0252	0.1171	0.1399
0.9199	13.0	260	0.9125	0.1021	0.0092	0.0252	0.1171	0.1297
0.8938	14.0	280	0.9069	0.1015	0.0091	0.0253	0.1137	0.1314
0.886	15.0	300	0.9022	0.1001	0.0091	0.0252	0.1137	0.1280
0.9008	16.0	320	0.8983	0.0989	0.0087	0.0248	0.1093	0.1297
0.874	17.0	340	0.8947	0.0969	0.0088	0.0249	0.1059	0.1280
0.8813	18.0	360	0.8914	0.0981	0.0091	0.0249	0.1070	0.1297
0.8787	19.0	380	0.8886	0.0939	0.0088	0.0257	0.0981	0.1280
0.8897	20.0	400	0.8860	0.0900	0.0086	0.0257	0.0936	0.1229
0.8728	21.0	420	0.8837	0.0898	0.0079	0.0241	0.0936	0.1229
0.8536	22.0	440	0.8816	0.0908	0.0083	0.0243	0.0925	0.1263
0.8547	23.0	460	0.8796	0.0912	0.0083	0.0243	0.0936	0.1263
0.8402	24.0	480	0.8781	0.0906	0.0085	0.0247	0.0970	0.1212
0.8334	25.0	500	0.8767	0.0906	0.0085	0.0247	0.0970	0.1212
0.8443	26.0	520	0.8753	0.0906	0.0085	0.0247	0.0970	0.1212
0.8405	27.0	540	0.8735	0.0892	0.0081	0.0246	0.0936	0.1212
0.8423	28.0	560	0.8726	0.0892	0.0081	0.0246	0.0936	0.1212
0.8305	29.0	580	0.8714	0.0892	0.0081	0.0246	0.0936	0.1212
0.8298	30.0	600	0.8705	0.0896	0.0082	0.0246	0.0948	0.1212
0.8234	31.0	620	0.8696	0.0896	0.0082	0.0246	0.0948	0.1212
0.8208	32.0	640	0.8688	0.0896	0.0082	0.0246	0.0948	0.1212
0.8242	33.0	660	0.8680	0.0896	0.0082	0.0246	0.0948	0.1212
0.8119	34.0	680	0.8673	0.0896	0.0082	0.0246	0.0948	0.1212
0.8244	35.0	700	0.8667	0.0871	0.0056	0.0171	0.0925	0.1195
0.8192	36.0	720	0.8661	0.0899	0.0082	0.0246	0.0936	0.1229
0.814	37.0	740	0.8657	0.0899	0.0082	0.0246	0.0936	0.1229
0.807	38.0	760	0.8651	0.0906	0.0083	0.0245	0.0936	0.1246
0.8009	39.0	780	0.8646	0.0915	0.0083	0.0246	0.0959	0.1246
0.8065	40.0	800	0.8642	0.0915	0.0083	0.0246	0.0959	0.1246
0.8104	41.0	820	0.8639	0.0889	0.0056	0.0171	0.0936	0.1229
0.7992	42.0	840	0.8637	0.0889	0.0056	0.0171	0.0936	0.1229
0.8061	43.0	860	0.8635	0.0889	0.0056	0.0171	0.0936	0.1229
0.7895	44.0	880	0.8633	0.0889	0.0056	0.0171	0.0936	0.1229
0.8158	45.0	900	0.8630	0.0889	0.0056	0.0171	0.0936	0.1229
0.8105	46.0	920	0.8630	0.0889	0.0056	0.0171	0.0936	0.1229
0.7951	47.0	940	0.8629	0.0889	0.0056	0.0171	0.0936	0.1229
0.8173	48.0	960	0.8627	0.0889	0.0056	0.0171	0.0936	0.1229
0.8029	49.0	980	0.8627	0.0889	0.0056	0.0171	0.0936	0.1229
0.8087	50.0	1000	0.8627	0.0882	0.0055	0.0171	0.0936	0.1212

Framework versions

• Transformers 4.30.0.dev0
• Pytorch 2.0.1
• Datasets 2.13.1
• Tokenizers 0.13.3