Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <Nxcode-CQ-7B-orpo>
Root cause: Unimplemented Feature Gap

### Model introduction

An ORPO fine-tune of Qwen/CodeQwen1.5-7B-Chat on 100k samples ours datasets. The model achieves 86% and 81% on humanval and humaneval plus respectively, surpass GPT-4-Turbo (Nov 2023) on humaneval

### Model URL

https://huggingface.co/NTQAI/Nxcode-CQ-7B-orpo

### Data type

float16

### Additional instructions (Optional)

Let's use the following template:
"Complete the following Python function:\n{prompt}"

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

@ganler To speed up the evaluation process, I have included the evaluation script and results below.

### HumanEval
```
from evalplus.data import get_mbpp_plus, write_jsonl, get_human_eval_plus
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
from peft import PeftModel
import torch
from tqdm import tqdm
model_name_or_path = "NTQAI/Nxcode-CQ-7B-orpo"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)

device = "cuda"
config = AutoConfig.from_pretrained(model_name_or_path, trust_remote_code=True)
config.init_device = device
model = AutoModelForCausalLM.from_pretrained(  
        model_name_or_path, config=config,trust_remote_code=True,torch_dtype=torch.bfloat16, device_map="cpu"
        )
model.to(device)

def get_markdown_code_blocks(content):
    try:
        res = content.split("```")[1]
        res = res[6:]
        res = res.strip()
        return res
    except:
        return ""

def GEN_SOLUTION(prompt):
    prompt = f"Complete the following Python function:\n{prompt}"
    messages=[
        { 'role': 'user', 'content': prompt}
    ]
    prompt = tokenizer.apply_chat_template(messages,add_generation_prompt=True,tokenize=False)
    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
    outputs = model.generate(inputs, max_new_tokens=512, do_sample=False, top_k=50, top_p=0.95, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
    res = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)
    res = get_markdown_code_blocks(res)
    return res

samples = [
    dict(task_id=task_id, solution=GEN_SOLUTION(problem["prompt"]))
    for task_id, problem in tqdm(get_human_eval_plus().items())
]
write_jsonl("humaneval.jsonl", samples)
```
```
evalplus.evaluate --dataset humaneval --samples humaneval.jsonl
```

![humaneval](https://github.com/evalplus/evalplus/assets/17800187/973545a0-3221-40c9-9b7c-9066a727ca87)


### MBPP

```
from evalplus.data import get_mbpp_plus, write_jsonl, get_human_eval_plus
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
from peft import PeftModel
import torch
from tqdm import tqdm
model_name_or_path = "NTQAI/Nxcode-CQ-7B-orpo"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)

device = "cuda"
config = AutoConfig.from_pretrained(model_name_or_path, trust_remote_code=True)
config.init_device = device
model = AutoModelForCausalLM.from_pretrained(  
        model_name_or_path, config=config,trust_remote_code=True,torch_dtype=torch.bfloat16, device_map="cpu"
        )
model.to(device)

def get_markdown_code_blocks(content):
    try:
        res = content.split("```")[1]
        res = res[6:]
        res = res.strip()
        return res
    except:
        return ""

def GEN_SOLUTION(prompt):
    messages=[
        { 'role': 'user', 'content': prompt}
    ]
    prompt = tokenizer.apply_chat_template(messages,add_generation_prompt=True,tokenize=False)
    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
    outputs = model.generate(inputs, max_new_tokens=512, do_sample=False, top_k=50, top_p=0.95, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
    res = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)
    res = get_markdown_code_blocks(res)
    return res

samples = [
    dict(task_id=task_id, solution=GEN_SOLUTION(problem["prompt"]))
    for task_id, problem in tqdm(get_mbpp_plus().items())
]
write_jsonl("mbpp.jsonl", samples)
```
```
evalplus.evaluate --dataset mbpp--samples mbpp.jsonl
```

![mbpp](https://github.com/evalplus/evalplus/assets/17800187/9ddc3c2f-1b9e-4c4c-bc7d-ddd0a163b884)