codet5p-770m-py-codebleu-1-True-1e-07-0.1

This model is a fine-tuned version of Salesforce/codet5p-770m-py on the mbpp dataset. It achieves the following results on the evaluation set:

• Loss: 0.6263
• Codebleu: 0.0880
• Ngram Match Score: 0.0119
• Weighted Ngram Match Score: 0.0435
• Syntax Match Score: 0.1209
• Dataflow Match Score: 0.0852

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-07
• train_batch_size: 6
• eval_batch_size: 6
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Codebleu	Ngram Match Score	Weighted Ngram Match Score	Syntax Match Score	Dataflow Match Score
0.9753	1.0	63	0.9060	0.0244	0.0000	0.0108	0.0289	0.0293
0.9732	2.0	126	0.8664	0.0781	0.0104	0.0358	0.1089	0.0747
0.9044	3.0	189	0.8430	0.0802	0.0110	0.0363	0.1132	0.0754
0.8564	4.0	252	0.8162	0.0821	0.0116	0.0384	0.1146	0.0782
0.8289	5.0	315	0.7880	0.0861	0.0135	0.0430	0.1214	0.0796
0.8171	6.0	378	0.7615	0.0862	0.0134	0.0422	0.1219	0.0796
0.7935	7.0	441	0.7390	0.0856	0.0136	0.0423	0.1204	0.0796
0.781	8.0	504	0.7206	0.0883	0.0143	0.0435	0.1219	0.0845
0.7608	9.0	567	0.7065	0.0855	0.0122	0.0396	0.1171	0.0838
0.7404	10.0	630	0.6934	0.0804	0.0094	0.0351	0.1118	0.0782
0.7388	11.0	693	0.6847	0.0787	0.0089	0.0337	0.1108	0.0754
0.7178	12.0	756	0.6786	0.0792	0.0090	0.0339	0.1113	0.0761
0.7087	13.0	819	0.6736	0.0811	0.0106	0.0388	0.1122	0.0782
0.7035	14.0	882	0.6690	0.0820	0.0109	0.0388	0.1122	0.0803
0.7005	15.0	945	0.6652	0.0842	0.0106	0.0384	0.1151	0.0831
0.688	16.0	1008	0.6620	0.0835	0.0104	0.0380	0.1156	0.0810
0.6911	17.0	1071	0.6587	0.0833	0.0106	0.0382	0.1166	0.0796
0.6782	18.0	1134	0.6559	0.0851	0.0114	0.0416	0.1156	0.0838
0.678	19.0	1197	0.6536	0.0844	0.0115	0.0416	0.1132	0.0845
0.6657	20.0	1260	0.6512	0.0856	0.0118	0.0422	0.1132	0.0873
0.6702	21.0	1323	0.6491	0.0842	0.0115	0.0416	0.1113	0.0859
0.662	22.0	1386	0.6471	0.0842	0.0115	0.0416	0.1113	0.0859
0.6569	23.0	1449	0.6453	0.0842	0.0116	0.0416	0.1113	0.0859
0.6605	24.0	1512	0.6436	0.0860	0.0114	0.0424	0.1171	0.0845
0.6589	25.0	1575	0.6420	0.0860	0.0114	0.0424	0.1171	0.0845
0.6519	26.0	1638	0.6404	0.0874	0.0118	0.0429	0.1190	0.0859
0.6568	27.0	1701	0.6390	0.0874	0.0118	0.0429	0.1190	0.0859
0.6569	28.0	1764	0.6378	0.0874	0.0116	0.0428	0.1190	0.0859
0.6455	29.0	1827	0.6365	0.0874	0.0116	0.0428	0.1190	0.0859
0.6456	30.0	1890	0.6355	0.0880	0.0116	0.0428	0.1190	0.0873
0.6503	31.0	1953	0.6345	0.0880	0.0116	0.0428	0.1190	0.0873
0.6424	32.0	2016	0.6337	0.0880	0.0116	0.0428	0.1190	0.0873
0.644	33.0	2079	0.6328	0.0880	0.0116	0.0428	0.1190	0.0873
0.6429	34.0	2142	0.6320	0.0872	0.0120	0.0435	0.1190	0.0852
0.6436	35.0	2205	0.6313	0.0872	0.0120	0.0435	0.1190	0.0852
0.638	36.0	2268	0.6307	0.0872	0.0120	0.0435	0.1190	0.0852
0.6381	37.0	2331	0.6300	0.0872	0.0120	0.0435	0.1190	0.0852
0.6307	38.0	2394	0.6295	0.0872	0.0120	0.0435	0.1190	0.0852
0.6344	39.0	2457	0.6289	0.0880	0.0119	0.0435	0.1209	0.0852
0.6296	40.0	2520	0.6285	0.0880	0.0119	0.0435	0.1209	0.0852
0.6268	41.0	2583	0.6280	0.0880	0.0119	0.0435	0.1209	0.0852
0.6315	42.0	2646	0.6276	0.0880	0.0119	0.0435	0.1209	0.0852
0.6265	43.0	2709	0.6273	0.0880	0.0119	0.0435	0.1209	0.0852
0.626	44.0	2772	0.6270	0.0880	0.0119	0.0435	0.1209	0.0852
0.631	45.0	2835	0.6268	0.0880	0.0119	0.0435	0.1209	0.0852
0.6315	46.0	2898	0.6266	0.0880	0.0119	0.0435	0.1209	0.0852
0.6309	47.0	2961	0.6264	0.0880	0.0119	0.0435	0.1209	0.0852
0.627	48.0	3024	0.6263	0.0880	0.0119	0.0435	0.1209	0.0852
0.6252	49.0	3087	0.6262	0.0880	0.0119	0.0435	0.1209	0.0852
0.632	50.0	3150	0.6263	0.0880	0.0119	0.0435	0.1209	0.0852

Framework versions

• Transformers 4.30.0.dev0
• Pytorch 2.0.1
• Datasets 2.13.1
• Tokenizers 0.13.3