|
--- |
|
language: code |
|
tags: |
|
- code |
|
- gpt2 |
|
- generation |
|
datasets: |
|
- giulio98/xlcost-single-prompt |
|
widget: |
|
- text: "'''\nfunction to add two numbers\n'''\n###\n" |
|
example_title: "add two numbers" |
|
model-index: |
|
- name: codegen-350M-multi-xlcost |
|
results: |
|
- task: |
|
name: Code Generation |
|
type: code-generation |
|
dataset: |
|
name: "XLCost" |
|
type: code_eval_outputs |
|
metrics: |
|
- name: pass@1 |
|
type: code_eval_outputs |
|
value: 3.325 |
|
- name: pass@10 |
|
type: code_eval_outputs |
|
value: 15 |
|
- name: codebleu |
|
type: codebleu |
|
value: 20.18191 |
|
--- |
|
|
|
# CodeGen-350M-multi-xlcost-v2 |
|
|
|
CodeGen-350M-multi-xlcost is a CodeGen model fine-tuned on the Python split of XLCost dataset using Deepspeed. |
|
|
|
## Usage |
|
|
|
You can load the CodeGen-350M-multi-xlcost-v2 model and tokenizer directly in `transformers`: |
|
|
|
```Python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
tokenizer = AutoTokenizer.from_pretrained("giulio98/codegen-350M-multi-xlcost-v2") |
|
model = AutoModelForCausalLM.from_pretrained("giulio98/codegen-350M-multi-xlcost-v2") |
|
|
|
text = tokenizer.eos_token + "\'\'\'\n" + "function to add two numbers" + "\n\'\'\'\n" + "###\n" |
|
input_ids = tokenizer(text, return_tensors="pt").input_ids |
|
|
|
generated_ids = model.generate(input_ids, max_length=128) |
|
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True)) |
|
``` |
|
Output: |
|
```Python |
|
''' |
|
function to add two numbers |
|
''' |
|
### |
|
def add(a, b): |
|
return a + b |
|
``` |
|
## Training |
|
|
|
The model was finetuned on [XLCost-single-prompt](https://huggingface.co/datasets/giulio98/xlcost-single-prompt), an improved version of the original XLCost dataset [ |
|
xlcost-text-to-code](https://huggingface.co/datasets/codeparrot/xlcost-text-to-code). Below the hyperparameters. |
|
|
|
| Hyperparameter | value | |
|
|---------------------------|--------| |
|
|Per device train batch size| 16 | |
|
|Context size| 1024 | |
|
|Training steps| 259| |
|
|Gradient accumulation| 2| |
|
|Gradient checkpointing| True| |
|
|Learning rate|1.8e-05 | |
|
|Weight decay | 0.1 | |
|
|Warmup steps| 35 | |
|
|Schedule| linear | |
|
|zero stage| 2 | |
|
|
|
Below the deepspeed configuration |
|
```Python |
|
{ |
|
"fp16": { |
|
"enabled": true, |
|
"loss_scale": 0, |
|
"loss_scale_window": 1000, |
|
"initial_scale_power": 16, |
|
"hysteresis": 2, |
|
"min_loss_scale": 1 |
|
}, |
|
"optimizer": { |
|
"type": "AdamW", |
|
"params": { |
|
"lr": 0.000018, |
|
"betas": [ |
|
0.9, |
|
0.999 |
|
], |
|
"eps": 1e-8, |
|
"weight_decay": 0.1 |
|
} |
|
}, |
|
"scheduler": { |
|
"type": "WarmupLR", |
|
"params": { |
|
"warmup_min_lr": 0, |
|
"warmup_max_lr": 0.000018, |
|
"warmup_num_steps": 35 |
|
} |
|
}, |
|
"zero_optimization": { |
|
"stage": 2, |
|
"offload_optimizer": { |
|
"device": "cpu", |
|
"pin_memory": false |
|
}, |
|
"allgather_partitions": true, |
|
"allgather_bucket_size": 200000000, |
|
"overlap_comm": true, |
|
"reduce_scatter": true, |
|
"reduce_bucket_size": 200000000, |
|
"contiguous_gradients": true |
|
}, |
|
"gradient_accumulation_steps": 2, |
|
"train_batch_size": 32, |
|
"train_micro_batch_size_per_gpu": 16, |
|
"gradient_clipping": 1, |
|
"wall_clock_breakdown": false |
|
} |
|
``` |
|
|
|
The training was executed on 1 x V100 (16GB) GPU for 28min 50sec |
|
|
|
## Performance |
|
|
|
We evaluated the model on the first 400 samples of XLCOST's [XLCost-single-prompt test split](https://huggingface.co/datasets/giulio98/xlcost-single-prompt/viewer/Python/test) and comparing the outputs of the generated codes with respect to the expected output using pass@k metric. |
|
|
|
| Metric | codegen-350M-multi-xlcost-v2 | codegen-350M-multi-xlcost | codegen-350M-mono(zero-shot) | codegen-350M-mono (one-shot) | codegen-350M-mono(few-shot) |
|
|--------|-----|-----|-----|-----|-----| |
|
|pass@1 |3.325% |3.70% | 0.4% | 0.35% | 0.48% | |
|
|pass@10 |15%| 14.5% | 3.5% | 3 % | 3.75% | |
|
|CodeBLEU |20.18%| None | 15.15% | 19.42 % | 20.27% | |
|
|
|
The [pass@k metric](https://huggingface.co/metrics/code_eval) tells the probability that at least one out of k generations passes the tests. |
|
|
|
## Citations |
|
``` |
|
@article{Nijkamp2022ACP, |
|
title={A Conversational Paradigm for Program Synthesis}, |
|
author={Nijkamp, Erik and Pang, Bo and Hayashi, Hiroaki and Tu, Lifu and Wang, Huan and Zhou, Yingbo and Savarese, Silvio and Xiong, Caiming}, |
|
journal={arXiv preprint}, |
|
year={2022} |
|
} |
|
``` |
