zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: Code-llama test result.
Root cause: Documentation Deficiency

THANKS FOR YOUR NICE JOB! 
FOLLOWING YOUR INFORMATION WHICH IS INCLUDED BELOW, I GOT THREE CODE-LLAMA TEST RESULTS.
I'M CONFUSED WITH THE RESULTS I GOTED FROM  **code-llama-13B-Python** ADN **code-llama-13B-Instruction**, THEY SEEMS NO DEFFERENCES, BUT FROM META'S DESCRIPTION, **13B-Python** SHOULD BE BETTER THAN **13B-Instruction**.
[ISSUE=17](https://github.com/evalplus/evalplus/issues/17#issuecomment-1694554596)

```
code-llama-13B
Base
{'pass@1': 0.36585365853658536}
Base + Extra
{'pass@1': 0.3170731707317073}

code-llama-13B-Python
Base
{'pass@1': 0.4146341463414634}
Base + Extra
{'pass@1': 0.35365853658536583}

code-llama-13B-Instruction
Base
{'pass@1': 0.4146341463414634}
Base + Extra
{'pass@1': 0.36585365853658536}
```

So to use 13B-Instruction I guess you need to use a specific prompt. Our current prompt is quite straight-forward.
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - HumanEval/1, HumanEvalPlus-Mini-v0.1.9, imbalanced parentheses
Root cause: Validation Gap

### EvalPlus version

HumanEvalPlus-Mini-v0.1.9

### Output of running `ls ~/.cache/evalplus`

AssertionError: invalid inputs

### Task ID of the programming task

HumanEval/1

### Test input

```python
["(((()(()())())"]  # which is the 2nd plus_input for HumanEvalPlus-Mini-v0.1.9
```

```python
["(((())))()((((((()(()()((())))(())))()(((())))()((()()))((()())))()))(())((()()))"]  # which is the 2nd last plus_input for HumanEvalPlus-Mini-v0.1.9
```

### Description

These test cases have imbalanced parentheses, which according to the contract attached below, should be considered as invalid input.

``` python
assert type(paren_string) == str, "invalid inputs" # $_CONTRACT_$
cnt = 0 # $_CONTRACT_$
for ch in paren_string: # $_CONTRACT_$
  assert ch in ["(", ")", " "], "invalid inputs"  # $_CONTRACT_$
  if ch == "(": cnt += 1 # $_CONTRACT_$
  if ch == ")": cnt -= 1 # $_CONTRACT_$
  assert cnt >= 0, "invalid inputs" # $_CONTRACT_$
assert cnt == 0, "invalid inputs" # $_CONTRACT_$
```

### Other context

_No response_
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - HumanEval/28, 2nd plus_input, HumanEvalPlus-Mini-v0.1.9, not a list
Root cause: Validation Gap

### EvalPlus version

HumanEvalPlus-Mini-v0.1.9

### Output of running `ls ~/.cache/evalplus`

AssertionError: invalid inputs

### Task ID of the programming task

HumanEval/28

### Test input

```python
[{"any": 69.63616178329318, "woood": -44.62697367582944, "multiple": 28.986177867416473, "much": 52.12790878761069, "vS": 69.74335770313701, "bEC": 69.74335770313701, "\ud83e\uddd0\ud83e\uddd0": 69.74335770313701}]  # which is the 2nd plus_input for HumanEvalPlus-Mini-v0.1.9
```


### Description

According to the problem description and the contract below, the input should only be a list. However, the reported one is a dict.

``` python
def concatenate(strings: List[str]) -> str:
    """ Concatenate list of strings into a single string
    >>> concatenate([])
    ''
    >>> concatenate(['a', 'b', 'c'])
    'abc'
    """

    assert all(isinstance(s, str) for s in strings), "invalid inputs" # $_CONTRACT_$
    assert isinstance(strings, list), "invalid inputs" # $_CONTRACT_$
```

### Other context

_No response_
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - HumanEval/99, no conversion from str to float before assertion
Root cause: Validation Gap

### EvalPlus version

HumanEvalPlus-Mini-v0.1.9

### Output of running `ls ~/.cache/evalplus`

TypeError: must be real number, not str

### Task ID of the programming task

HumanEval/99

### The original wrong contract

```python
try: # $_CONTRACT_$
        assert isinstance(value, str) # $_CONTRACT_$
        float(value) # $_CONTRACT_$
    except: # $_CONTRACT_$
        raise Exception("invalid inputs") # $_CONTRACT_$
    import math # $_CONTRACT_$
    assert not (math.isinf(value) or math.isnan(value)), "invalid inputs" # $_CONTRACT_$
```


### Your proposed new contract

```python
try: # $_CONTRACT_$
        assert isinstance(value, str) # $_CONTRACT_$
        value = float(value) # $_CONTRACT_$
    except: # $_CONTRACT_$
        raise Exception("invalid inputs") # $_CONTRACT_$
    import math # $_CONTRACT_$
    assert not (math.isinf(value) or math.isnan(value)), "invalid inputs" # $_CONTRACT_$
```


### Other context

The assertion `assert not (math.isinf(value) or math.isnan(value)), "invalid inputs"` requires `value` to be a float, but the input `value` is a str and original contract doesn't convert it from str to float.   

Hi Marcus thanks for filing these issues. Hi, @Kristoff-starling do you want to fix #180, #181, #182 and bump to HumanEval+ `v0.1.10`?
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] -  AIGCodeGeek-DS-6.7B
Root cause: Unimplemented Feature Gap

### Model introduction

The model was finetuned by AIGCode based on DeepSeek-Coder-6.7B-base using open-source and private datasets.

### Model URL

https://huggingface.co/aigcode/AIGCodeGeek-DS-6.7B

### Additional information (Optional)

We have provided code for conveniently reproducing the results. 
See https://github.com/AIGCode-net/aigcodegeek_evalplus/tree/main.

### Decontamination

The datasets we used for this training:
1. Leon-Leee/Wizardlm_Evol_Instruct_v2_196K_backuped
2. ise-uiuc/Magicoder-Evol-Instruct-110K
3. openbmb/UltraInteract_sft
4. m-a-p/Code-Feedback
5. flytech/python-codes-25k
6. our private dataset

Datasets No.1~4 have been used for training models like WizardLM, Magicoder, Eurus, and OpenCodeInterpreter; we assume they are high-quality with very little or no contamination.

Then we follow a substring-matching method used by Magicoder/Bigcode (https://github.com/ise-uiuc/magicoder/tree/main/src/magicoder/decontamination) to test dataset No.5 and No.6: 
- dataset No.5 contains only a few *overlapped* solutions but we found the excluded samples were quite general code segments and this *contamination* did not contribute to the evaluation. The file is uploaded here for all to examine.
[flytech_25k_detected.json](https://github.com/evalplus/evalplus/files/15158288/flytech_25k_detected.json)
- dataset No.6 is our private enterprise dataset still in development with no contamination found by this script.

### Author

Yes

### Data

Not all

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Self-reported results are as follows:

<img width="770" alt="image" src="https://github.com/evalplus/evalplus/assets/17897916/e033a60e-59b0-448f-be5c-4b18a7375056">

<img width="742" alt="image" src="https://github.com/evalplus/evalplus/assets/17897916/73fe19dd-82bc-4ac6-a26f-39f9a8415309">

One can easily reproduce these using code from [https://github.com/AIGCode-net/aigcodegeek_evalplus/](https://github.com/AIGCode-net/aigcodegeek_evalplus/tree/main)

@ganler @FatPigeorz Thanks very much for your nice work on EvaPlus. I wonder if you could do our team a favor to assign one person to evaluate it and if the result works then update to the leader board. 
If there's anything we need to do during process, we'll keep an eye on it and coordinates closely.
Repository: evalplus/evalplus
Issue:  Nxcode-CQ-7B-orpo
GitHub labels: model eval

### Model introduction

Nxcode-CQ-7B-orpo is an ORPO fine-tune of Qwen/CodeQwen1.5-7B-Chat on 100k samples ours datasets.

- Strong code generation capabilities and competitve performance across a series of benchmarks;
- Supporting 92 coding languages
- Excellent performance in text-to-SQL, bug fix, etc.


### Model URL

https://huggingface.co/NTQAI/Nxcode-CQ-7B-orpo

### Additional information (Optional)

_No response_

### Decontamination

Information is not provided

### Author

No

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - 160 Incorrect index in contract
GitHub labels: program contract

### EvalPlus version

0.9.0

### Output of running `ls ~/.cache/evalplus`

-

### Task ID of the programming task

HumanEval/160

### The original wrong contract

```python
...
    assert not any([operand[i-1] == 0 and operator[i] == "//" for i in range(1, len(operand))]), "invalid inputs" # $_CONTRACT_$
```


### Your proposed new contract

```python
...
    assert not any([operand[i] == 0 and operator[i - 1] == "//" for i in range(1, len(operand))]), "invalid inputs" # $_CONTRACT_$
```


### Other context

_No response_
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - microsoft/Phi-3-mini-128k-instruct
GitHub labels: model eval

### Model introduction

This model is created by Microsoft as an efficient tiny model based on heavily filtered web data and synthetically created training data

### Model URL

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct

### Additional information (Optional)

_No response_

### Decontamination

unknown

### Author

No

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

I have tested the 4k version. :) I assume the 128k version should target longer-context tasks.

[Phi-4](https://huggingface.co/microsoft/phi-4)
Repository: evalplus/evalplus
Issue: Leaderboard Data Request
Root cause: Documentation Deficiency

Hi folks - awesome job on this! I was wondering if it was possible to get the raw data from the leaderboard rather than re-running a bunch of stuff myself?

I was looking to get it disaggregated, so each row would be something like {human_eval_question, model, model_was_correct_or_not}.

1. You can find the pre-generated outputs at the assets of https://github.com/evalplus/evalplus/releases/tag/v0.1.0
2. By only looking at the `temp_0.0` version (aka greedy decoding) you can run `evalplus.evaluate` (see https://github.com/evalplus/evalplus#evaluate-code-samples-with-humaneval) to get the computed pass@k in the terminal output and that's how we report the final data in the leaderboard.
3. To get the form of data structure, you are recommended to dig here: https://github.com/evalplus/evalplus/blob/master/evalplus/eval/__init__.py#L29 and here: https://github.com/evalplus/evalplus/blob/master/evalplus/evaluate.py#L241

Hi! Thanks for the greate repo. It seems like you have added more models to your leaderboard (e.g. CodeLlama), but the asserts link you provide above haven't been updated yet. I'm looking for the outputs of CodeLlama and WizzardCoder on HumanEval. Could you please share it? Thanks in advance :)

Hi @skpig per your request, I have uploaded zero-shot outptus for codet5+, codellama, and wizarcoder.

Please check the assests of https://github.com/evalplus/evalplus/releases/tag/v0.1.0
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - WaveCoder-Ultra-6.7B
GitHub labels: model eval

### Model introduction

WaveCoder 🌊 is a series of large language models (LLMs) for the coding domain, designed to solve relevant problems in the field of code through instruction-following learning. Its training dataset was generated from a subset of code-search-net data using a generator-discriminator framework based on LLMs that we proposed, covering four general code-related tasks: code generation, code summary, code translation, and code repair.

### Model URL

https://huggingface.co/microsoft/wavecoder-ultra-6.7b

### Additional information (Optional)

We use the code base from magicoder (https://github.com/ise-uiuc/magicoder) and evalplus (https://github.com/evalplus/evalplus), anyone can reach the score by running the following command:
```
MODEL_KEY=deepseek-ai/deepseek-coder-6.7b-base
MODEL=microsoft/wavecoder-ultra-6.7b

DATASET=humaneval
SAVE_PATH=evalplus-$(basename $MODEL)-$DATASET.jsonl
SANITIZED_PATH=humaneval_result/evalplus-$(basename $MODEL)-$DATASET-sanitized.jsonl

python -m experiments.text2code \
  --model_key $MODEL_KEY \
  --model_name_or_path $MODEL \
  --save_path $SAVE_PATH \
  --dataset $DATASET \
  --temperature 0.0 \
  --top_p 1.0 \
  --max_new_tokens 512 \
  --n_problems_per_batch 28 \
  --n_samples_per_problem 1 \
  --n_batches 1

echo "$MODEL"
evalplus.evaluate --dataset $DATASET --samples $SAVE_PATH
```

### Decontamination

embedding 

### Author

Yes

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Hi, @ganler, Could we possibly provide the test code? In fact, we found in other issues that we can use the test code from magicoder for testing. I look forward to receiving your reply.
https://github.com/evalplus/evalplus/issues/40

also cc @answers111

Thank you, let's concentrate the discussion thread on #169 (now there are too many WaveCoder threads...)
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - ajibawa-2023/Code-Llama-3-8B
Root cause: Unimplemented Feature Gap

### Model introduction

This Model is trained on refined version of my dataset Code-290k-ShareGPT.

Besides this it is trained on following datasets:

Code-Feedback

orca-math-word-problems-200k

CodeFeedback-Filtered-Instruction

The idea was to check how this Model will perform with both Code & Maths datasets. This model is very good with Coding. Maths outputs are also very good. You can test out this model.

It is very very good in Code generation in various languages such as Python, Java, JavaScript, GO, C++, Rust, Ruby, Sql, MySql, R, Julia, Haskell, etc.. This model will also generate detailed explanation/logic behind each code.

This Model is trained on massive datasets so the results are very good. You can check the Examples given below.

I have used ChatML prompt format.

This is Fully Finetuned Model.

### Model URL

https://huggingface.co/ajibawa-2023/Code-Llama-3-8B

### Additional information (Optional)

Datasets used for training purpose:
https://huggingface.co/datasets/ajibawa-2023/Code-290k-ShareGPT
https://huggingface.co/datasets/m-a-p/Code-Feedback
https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k
https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction

### Decontamination

Above mentioned datasets are available on the huggingface. They are not contaminated.

### Author

Yes

### Data

Yes (If so please specify in 'Additional information')

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <Sharan1712/mistralai-7-Instruct>
GitHub labels: model eval

### Model introduction

The model was finetuned on recipe instructions and ingredients to extract ingredients and recipe steps from a transcription of recipe. Fine tuning was done on a custom manually created dataset using LoRA

Model can be used to get detailed and organised recipe ingredients and steps from a transcription.

### Model URL

https://huggingface.co/Sharan1712/mistralai-7-Instruct

### Additional information (Optional)

_No response_

### Decontamination

The dataset was manually created and compared among 6 people

### Author

Yes

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: LLM-generated code
GitHub labels: question

Hi,
Thank you for sharing the replication package of your article. Within the article, you mentioned that you are open-sourcing your tool "as well as all LLM-generated code". How can we have access to those LLM-generated codes per task? Unfortunately, I couldn't find a way to download them based on the shared instructions.

@arghavanMor Thanks for the issue. We open-sourced the LLM-generated code in the attachment of our first release. Please find the link below:

https://github.com/evalplus/evalplus/releases/tag/v0.1.0

Thanks for your feedback. I made it more clear by adding a new section (https://github.com/evalplus/evalplus#-LLM-generated-code) talking about the pre-generated code and how to use them. Thanks! Feel free to reopen this issue if you witness other troubles.

GPT-J, CodeGen2, and StarCoder LLM-generated code are missing. Thanks.

Thanks for letting us know. They are all updated on the initial release. https://github.com/evalplus/evalplus/releases/tag/v0.1.0
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - AutoCoder
Root cause: Unimplemented Feature Gap

### Model introduction

The model outperforms the GPT-4 Turbo on the HumanEval base test **(90.9% vs 90.2%)**. And it provides a new kind of code Interpreter.

### Model URL

https://huggingface.co/Bin12345/AutoCoder

### Additional information (Optional)

_No response_

### Decontamination

The dataset contains some running results of the python code and several open-sourced dataset. The base model is the deepseeker 33B.

### Author

Yes

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - GPT-4o
Root cause: Unimplemented Feature Gap

### Model introduction

GPT-4o (“o” for “omni”) is a step towards much more natural human-computer interaction. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models.

### Model URL

https://openai.com/index/hello-gpt-4o/

### Additional information (Optional)

_No response_

### Decontamination

NDA/Not disclosed 

### Author

No

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Would also be interested
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - ReflectionCoder
Root cause: Unimplemented Feature Gap

### Model introduction

ReflectionCoder is a novel approach that effectively leverages reflection sequences constructed by integrating compiler feedback to improve one-off code generation performance. 

### Model URL

https://huggingface.co/SenseLLM/ReflectionCoder-DS-33B

### Additional information (Optional)

Also, the following three:
https://huggingface.co/SenseLLM/ReflectionCoder-DS-6.7B
https://huggingface.co/SenseLLM/ReflectionCoder-CL-7B
https://huggingface.co/SenseLLM/ReflectionCoder-CL-34B

The DS denotes fine-tuned DeepSeek-Coder, and CL denotes the Code Llama.

The detailed information is presented at [https://github.com/SenseLLM/ReflectionCoder](https://github.com/SenseLLM/ReflectionCoder).

The test prompt and script are presented at [https://github.com/SenseLLM/ReflectionCoder/blob/master/src/test.py](https://github.com/SenseLLM/ReflectionCoder/blob/master/src/test.py).

The generated results on our end are presented at [https://github.com/SenseLLM/ReflectionCoder/tree/master/data](https://github.com/SenseLLM/ReflectionCoder/tree/master/data).

| Model | Checkpoint | Size | HumanEval (+) | MBPP (+) 
|:-------|:------------|:------|:---------------|:----------|
| ReflectionCoder-CL-7B   | 🤗 [HF Link](https://huggingface.co/SenseLLM/ReflectionCoder-CL-7B)  | 7B   | 75.0 (68.9) | 72.2 (61.4) |
| ReflectionCoder-CL-34B  | 🤗 [HF Link](https://huggingface.co/SenseLLM/ReflectionCoder-CL-34B) | 34B  | 70.7 (66.5) | 68.4 (56.6) | 
| ReflectionCoder-DS-6.7B | 🤗 [HF Link](https://huggingface.co/SenseLLM/ReflectionCoder-DS-6.7B)| 6.7B | 80.5 (74.4) | 81.5 (69.6) | 
| ReflectionCoder-DS-33B  | 🤗 [HF Link](https://huggingface.co/SenseLLM/ReflectionCoder-DS-33B) | 33B  | 82.9 (76.8) | 84.1 (72.0) |

### Decontamination

We only use open-sourced data: [m-a-p/CodeFeedback-Filtered-Instruction](https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction)

[Magicoder-OSS-Instruct](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K) and  [Magicoder-Evol-Instruct](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K) are decontaminated by removing exact matches from HumanEval, MBPP, and DS-1000.

[Python code subset of ShareGPT](https://huggingface.co/datasets/ajibawa-2023/Python-Code-23k-ShareGPT) and [Evol-Instruct-Code](https://huggingface.co/datasets/nickrosh/Evol-Instruct-Code-80k-v1) are filtered with Qwen-72B.

### Author

Yes

### Data

Yes (If so please specify in 'Additional information')

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Hi @ganler, it's been a week since I submitted the request. I've attached the eval script and the raw outputs. Is there anything else I can do to speed up the evaluation?
Repository: evalplus/evalplus
Issue: Additional BOS token when using `make_chat_prompt`
Root cause: Algorithmic Error

Hi,

When using the `codegen/generate.py` script, I noticed this [snippet](https://github.com/evalplus/evalplus/blob/132361a616071c8f450275d55a0268425c1277c6/codegen/model.py#L73C24-L79):

```python
    prompt = tokenizer.apply_chat_template(
        [
            {"role": "user", "content": prompt},
            {"role": "assistant", "content": response},
        ],
        tokenize=False,
    ).split(_MAGIC_SPLITTER_)[0]
    return prompt
 ```
 
Introduces a BOS token to the prompt (along with the necessary chat tokens), however when the model encodes the prompt later it also adds another BOS token. This causes two sequential BOS tokens to be used instead of one. This probably does not affect performance but thought it was worth flagging nonetheless. This affects HF and VLLM generation.

Simple fix would be to remove the first BOS token from the string after applying the chat template. 
 
 ```python
    prompt = tokenizer.apply_chat_template(
        [
            {"role": "user", "content": prompt},
            {"role": "assistant", "content": response},
        ],
        tokenize=False,
    ).split(_MAGIC_SPLITTER_)[0]
    prompt = prompt.removeprefix(tokenizer.bos_token)
    return prompt
 ```

Happy to make the PR if you want


Thank you @gautierdag 

I am happy to accept your PR if you can try to fix it. :)