Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - WaveCoder-Ultra-6.7B
GitHub labels: model eval

### Model introduction

WaveCoder 🌊 is a series of large language models (LLMs) for the coding domain, designed to solve relevant problems in the field of code through instruction-following learning. Its training dataset was generated from a subset of code-search-net data using a generator-discriminator framework based on LLMs that we proposed, covering four general code-related tasks: code generation, code summary, code translation, and code repair.


### Model URL

https://huggingface.co/microsoft/wavecoder-ultra-6.7b

### Data type

bfloat16

### Additional instructions (Optional)

We use the code base from magicoder (https://github.com/ise-uiuc/magicoder) and evalplus (https://github.com/evalplus/evalplus), anyone can reach the score by running the following command:
```sh
MODEL_KEY=deepseek-ai/deepseek-coder-6.7b-base
MODEL=microsoft/wavecoder-ultra-6.7b

DATASET=humaneval
SAVE_PATH=evalplus-$(basename $MODEL)-$DATASET.jsonl
SANITIZED_PATH=humaneval_result/evalplus-$(basename $MODEL)-$DATASET-sanitized.jsonl

python -m experiments.text2code \
  --model_key $MODEL_KEY \
  --model_name_or_path $MODEL \
  --save_path $SAVE_PATH \
  --dataset $DATASET \
  --temperature 0.0 \
  --top_p 1.0 \
  --max_new_tokens 512 \
  --n_problems_per_batch 28 \
  --n_samples_per_problem 1 \
  --n_batches 1

echo "$MODEL"
evalplus.evaluate --dataset $DATASET --samples $SAVE_PATH
```

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Dup of #160 

Hi @answers111 it seems that wavecoder as an instruction-following model does not have a chat prompt (https://huggingface.co/microsoft/wavecoder-ultra-6.7b/blob/main/tokenizer_config.json). Can you fix it? Otherwise the framework now runs direct generation/completion.

Hi, Jiawei. Actually we use the following prompt on HumanEval benchmark for code completion task.
```
PROMPT
You are an exceptionally intelligent coding assistant that consistently delivers accurate and reliable responses to user instructions.

@@ Instruction
Write a solution to the following problem:
```python
def flip_case(string: str) -&gt; str:
    """ For a given string, flip lowercase characters to uppercase and uppercase to lowercase.
    &gt;&gt;&gt; flip_case('Hello')
    'hELLO'
    """
```

@@ Response
```python
def flip_case(string: str) -&gt; str:
    """ For a given string, flip lowercase characters to uppercase and uppercase to lowercase.
    &gt;&gt;&gt; flip_case('Hello')
    'hELLO'
    """


```
You can reach the reported score with it.&nbsp;
We will add the chat prompt in the next version. Thank you!




------------------&nbsp;原始邮件&nbsp;------------------
发件人: "Jiawei ***@***.***&gt;; 
发送时间: 2024年4月30日(星期二) 上午9:46
收件人: ***@***.***&gt;; 
抄送: ***@***.***&gt;; ***@***.***&gt;; 
主题: Re: [evalplus/evalplus] 🤗 [REQUEST] - WaveCoder-Ultra-6.7B (Issue #169)














 
Hi @answers111 it seems that wavecoder as an instruction-following model does not have a chat prompt (https://huggingface.co/microsoft/wavecoder-ultra-6.7b/blob/main/tokenizer_config.json). Can you fix it? Otherwise the framework now runs direct generation/completion.
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

Hi, @ganler, Could we possibly provide the test code? In fact, we found in other issues that we can use the test code from magicoder for testing. I look forward to receiving your reply. 
https://github.com/evalplus/evalplus/issues/40

Hello, @Bonytu @answers111 : for convenience now EvalPlus directly use `apply_chat_template` (https://huggingface.co/docs/transformers/main/en/chat_templating) for prompt construction. That being said, if you want to apply the prompt format you can just add a chat template as I mentioned in https://github.com/evalplus/evalplus/issues/169#issuecomment-2084122098. :)

It seems that WaveCoder uses the same prompt format as Magicoder, so a very easy way you can do to enable the chat template is to use this line as the template: https://huggingface.co/ise-uiuc/Magicoder-S-DS-6.7B/blob/main/tokenizer_config.json#L183 

Oh, I actually found that you updated the chat template just a few hours ago. I will rerun the eval using the latest chat template then. :) Thanks!

@ganler Thanks for your help

Let me know if my repro is consistent with yours @Bonytu @answers111 

<img width="571" alt="image" src="https://github.com/evalplus/evalplus/assets/38074777/d32cf5eb-364f-48f8-aac4-c1f3208c248d">


This result is still inconsistent, but it's okay. We will open source the repository and test results soon and re-communicate with you about the test. Please keep the issue open. Thank you.

cc: @Bonytu 

That's fine. Just please note that getting an exact same result to match your end can be challenging due to potential inconsistencies in hardware, library versions, prompt/decoding configs, and post processing. 

As a non-profit org, for our convenience, usually what we do is to just pull the model from huggingface and test it using vLLM greedy decoding and apply chat template for prompting -- report what it is without sophisticated manual config tuning/optimization as long as it is not significantly terrible (like 0). That is what I did for the majority of other models -- and it is very common for scores that are a bit higher/lower than that in the paper (check out the leaderboard and you will see lots of instances). 

Hi, ganler. We have released our evaluation results under evalplus (https://github.com/microsoft/WaveCoder/tree/main/src/eval/evalplus). You may get the original score by running it. Thanks a lot.

cc: @Bonytu