codet5p-770m-py-codebleu-128-True-1e-06-0.1

This model is a fine-tuned version of Salesforce/codet5p-770m-py on the mbpp dataset. It achieves the following results on the evaluation set:

• Loss: 0.8171
• Codebleu: 0.0852
• Ngram Match Score: 0.0135
• Weighted Ngram Match Score: 0.0421
• Syntax Match Score: 0.1180
• Dataflow Match Score: 0.0810

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 6
• eval_batch_size: 6
• seed: 42
• gradient_accumulation_steps: 128
• total_train_batch_size: 768
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Codebleu	Ngram Match Score	Weighted Ngram Match Score	Syntax Match Score	Dataflow Match Score
0.4807	1.0	1	0.9113	0.0039	0.0000	0.0000	0.0048	0.0049
0.4928	2.0	2	0.9113	0.0039	0.0000	0.0000	0.0048	0.0049
0.2434	3.0	4	0.9113	0.0039	0.0000	0.0000	0.0048	0.0049
0.4813	4.0	5	0.9112	0.0055	0.0000	0.0000	0.0067	0.0070
0.2397	5.0	7	0.9111	0.0063	0.0000	0.0002	0.0072	0.0084
0.4858	6.0	8	0.9109	0.0065	0.0000	0.0002	0.0072	0.0091
0.242	7.0	10	0.9107	0.0124	0.0000	0.0012	0.0173	0.0133
0.4909	8.0	11	0.9104	0.0124	0.0000	0.0012	0.0173	0.0133
0.2437	9.0	13	0.9094	0.0136	0.0000	0.0028	0.0178	0.0154
0.4811	10.0	14	0.9091	0.0176	0.0000	0.0045	0.0212	0.0216
0.2439	11.0	16	0.9056	0.0208	0.0000	0.0057	0.0260	0.0244
0.4809	12.0	17	0.9051	0.0241	0.0000	0.0107	0.0289	0.0286
0.2428	13.0	19	0.9038	0.0334	0.0003	0.0159	0.0424	0.0370
0.4813	14.0	20	0.9034	0.0395	0.0007	0.0195	0.0477	0.0461
0.2429	15.0	22	0.8926	0.0464	0.0013	0.0214	0.0573	0.0531
0.473	16.0	23	0.8908	0.0491	0.0020	0.0221	0.0588	0.0580
0.2385	17.0	25	0.8889	0.0637	0.0060	0.0316	0.0814	0.0684
0.472	18.0	26	0.8870	0.0634	0.0053	0.0286	0.0829	0.0670
0.2385	19.0	28	0.8847	0.0744	0.0072	0.0294	0.1021	0.0747
0.4688	20.0	29	0.8838	0.0780	0.0093	0.0346	0.1079	0.0761
0.2304	21.0	31	0.8609	0.0785	0.0100	0.0357	0.1079	0.0768
0.451	22.0	32	0.8594	0.0806	0.0105	0.0363	0.1122	0.0775
0.2238	23.0	34	0.8571	0.0804	0.0107	0.0362	0.1132	0.0761
0.4484	24.0	35	0.8561	0.0804	0.0109	0.0363	0.1132	0.0761
0.2224	25.0	37	0.8525	0.0804	0.0109	0.0363	0.1132	0.0761
0.4454	26.0	38	0.8517	0.0794	0.0109	0.0362	0.1113	0.0754
0.2234	27.0	40	0.8466	0.0797	0.0118	0.0385	0.1113	0.0754
0.4458	28.0	41	0.8348	0.0797	0.0117	0.0385	0.1113	0.0754
0.2151	29.0	43	0.8318	0.0800	0.0118	0.0385	0.1113	0.0761
0.4204	30.0	44	0.8333	0.0800	0.0119	0.0385	0.1113	0.0761
0.2117	31.0	46	0.8312	0.0800	0.0118	0.0385	0.1113	0.0761
0.4165	32.0	47	0.8287	0.0808	0.0122	0.0385	0.1118	0.0775
0.2073	33.0	49	0.8216	0.0842	0.0135	0.0421	0.1171	0.0796
0.4114	34.0	50	0.8171	0.0852	0.0135	0.0421	0.1180	0.0810

Framework versions

• Transformers 4.30.0.dev0
• Pytorch 2.0.1
• Datasets 2.13.1
• Tokenizers 0.13.3