zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Codestral-22B-v0.1
Root cause: Unimplemented Feature Gap

### Model introduction

The model is a 22B model by Mistral AI, designed for coding. It appears to be related in some way to Mixtral-8x22B, but I'm not sure exactly how it's related.

### Model URL

https://huggingface.co/mistralai/Codestral-22B-v0.1

### Additional information (Optional)

Mistrals official repo provides a model that is only usable with Mistral's own mistral_inference library. Someone has converted it to standard HuggingFace safetensors format at <https://huggingface.co/bullerwins/Codestral-22B-v0.1-hf> so that it can be used for inference with HuggingFace's transformers library.

### Decontamination

The authors have not said anything about contamination in the training data.

### Author

No

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

The [blog post](https://mistral.ai/news/codestral/) has HumanEval and MBPP scores, but no MBPP+ or HumanEval+.

Updated.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - IBM Granite Code Models
Root cause: Unimplemented Feature Gap

### Model introduction

The models are code generation models created by IBM. They include both base models and instruct models. They are new models that have been trained from scratch, except that the 34B version is based on the 20B version.

### Model URL

https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330

### Additional information (Optional)

_No response_

### Decontamination

They have not specified any decontamination information. Information about the training data can be found at https://arxiv.org/pdf/2405.04324#page=4.

### Author

No

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Here are the reported MBPP and MBPP+ scores:
![image](https://github.com/evalplus/evalplus/assets/54595360/281946d7-b1b8-4fdf-ad86-49b98e6259d5)

They have not reported HumanEval+ scores, but they have reported HumanEvalSynthesize and MultiPL-E scores at https://arxiv.org/pdf/2405.04324#page=10.

They did not specify which version of MBPP+ they're using here, so they might be using v0.1.0.

Hello, I am also interested on the evaluation of the Granite models.
Repository: evalplus/evalplus
Issue: Evaluating on a subset of the dataset
Root cause: Unimplemented Feature Gap

In some cases evaluating only on a small subset of dataset. For e.g. in human eval there are 164 examples which can take 30-60 minutes to generate depending on hardware/model, sometimes its helpful to evaluate your model on just 10-20 examples as a sanity check. 

Currently this is not possible due to the assert statements here
https://github.com/evalplus/evalplus/blob/b7f2961f1223a1c488977ac24db5c162401ead08/evalplus/evaluate.py#L174-L175

But can be easily achieved by just taking the subset of examples out of the full dataset here
https://github.com/evalplus/evalplus/blob/b7f2961f1223a1c488977ac24db5c162401ead08/evalplus/evaluate.py#L132

I was wondering if this is a feature that can be useful, if yes maybe this can be added as a flag. What do you think? I can contribute if needed

@RahulSChand Thanks for the suggestion, this sounds like a great feature to have. Feel free to open a PR with your implemention and we can take a look. 

There is a bit of complication here since the key to each task (i.e., `task_id`) can be strings such as `HumanEval/k` which makes it inconvenient to type as CLI args and existing ground-truth uses a cache mechanism based on the MD5 fingerprint of the raw dataset file. Meanwhile, issues can be foreseen in the future when evalplus integrates other datasets using a different `task_id` mechanism. Therefore, IMO the best workaround is to simply truncate and override the dataset.

Consequently, I made d4ff6c4e4faa2ffc632e4cce6b0331dbd896a9a2 a compatible way to add this feature:
1. Create a sub-dataset file at your own flavor (by cutting the lines of tasks you are not interested in)
2. Use the env variable `HUMANEVAL_OVERRIDE_PATH` to point to that path

More concretely:

```bash
# Get top-10 tasks to quick-start.jsonl
cat ~/.cache/evalplus/HumanEvalPlus-v0.1.6.jsonl | head -n 10 > quick-start.jsonl
# Run with envar set
HUMANEVAL_OVERRIDE_PATH=$(realpath quick-start.jsonl) $YOUR_EVALPLUS_COMMAND # evalplus.evaluate... 
```

Let me know your thoughts on this workaround. Thanks.

@ganler Thanks for this. I think this workaround is enough. No need of changes in the code logic & won't be affected if the keys of the dataset change.
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - HumanEval/10 incorrect output to input 'raracece'
Root cause: Validation Gap

### EvalPlus version

0.2.1

### Output of running `ls ~/.cache/evalplus`

HumanEvalPlus-v0.1.9.jsonl

### Task ID of the programming task

HumanEval/10

### Test input

```python
raracece
```


### Description

An input / output pair for the `make_palindrome` function is:
Input: ['raracece']
Output: 'raracececarar'

Output should be: 'raracecececarar'

### Other context

_No response_

@Kristoff-starling Can you help double check the test inputs and contract?

I just realized the output is actually valid, closing this issue. Thanks!
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Qwen2 models
Root cause: Unimplemented Feature Gap

### Model introduction

The models are created by Tongyi Qianwen, part of Alibaba Cloud, and are a general-purpose model, but incorporate the techniques used to train CodeQwen. More information can be found in their [blog post](https://qwenlm.github.io/blog/qwen2/).

### Model URL

https://huggingface.co/collections/Qwen/qwen2-6659360b33528ced941e557f

### Additional information (Optional)

_No response_

### Decontamination

The authors did not mention decontamination, but it might be mentioned in the upcoming technical report.

### Author

No

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

They reported HumanEval, MBPP, and what looks to be the arithmetic mean of the HumanEval, HumanEval+, MBPP, and MBPP+ scores in their blog post.

Wonder why they are not logged in the leaderboard tho, feels very dated
Repository: evalplus/evalplus
Issue: DeepSeekCoder V2 lite
Root cause: Unimplemented Feature Gap

Please also add the lite version of new DeepSeekCoder V2 (which is already on the leaderboard) which is much faster.
Would be interesting to see how it compares to CodeQwen1.5-7b-chat

https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
Repository: evalplus/evalplus
Issue: Is the code for generating more test cases open source?
Root cause: Documentation Deficiency

Hello, this is a very good work!

I noticed that you don't seem to open source the code for generating test cases? Of course, maybe I just didn't find it. Are you going to open source this part of the code?

Looking forward to your positive response!

https://github.com/evalplus/evalplus/tree/master/evalplus/gen

Open-sourced since day-1. :)
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - InverseCoder
Root cause: Unimplemented Feature Gap

### Model introduction

InverseCoder is a series of code LLMs instruction-tuned by generating data from itself through Inverse-Instruct. It is created by Institute of Computing Technology, Chinese Academy of Sciences. More information can be found in [paper](https://arxiv.org/pdf/2407.05700).

### Model URL

https://huggingface.co/wyt2000/InverseCoder-CL-7B;
https://huggingface.co/wyt2000/InverseCoder-DS-6.7B;
https://huggingface.co/wyt2000/InverseCoder-CL-13B

### Additional information (Optional)

Dataset: 
https://huggingface.co/datasets/wyt2000/InverseCoder-CL-7B-Evol-Instruct-90K
https://huggingface.co/datasets/wyt2000/InverseCoder-DS-6.7B-Evol-Instruct-90K
https://huggingface.co/datasets/wyt2000/InverseCoder-CL-13B-Evol-Instruct-90K

### Decontamination

The same as starcoder and magicoder.

### Author

Yes

### Data

Yes (If so please specify in 'Additional information')

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: Independenly Ran Meta-Llama-3.1-8B-Instruct Run
Root cause: Documentation Deficiency

Hello, i independently ran both HumanEval and EvalPlus for the latest llama3.1 8b instruct model. Below are the results:

<img width="397" alt="Screenshot 2024-07-23 at 11 40 44 AM" src="https://github.com/user-attachments/assets/43674173-b23e-4e6d-b777-4d2587c479c7">

Would be happy to share the eval file, if needed, and could the leaderboard be updated to reflect the latest llama models?

Is this insturct or base model? Your picture says "base".

Hi @Vaibhav-Sahai , could you share your specific command to generate this result? I encountered some issues with Llama-3.1 here #220 .

@Co1lin sorry, just saw this. These are numbers for the instruct model. 

I see that the issue you linked ran the base model, which might be having issues completing the code block. I ran this independently (using evalplus to eval the final output) but i had my own cleaning/sanitization logic to clean up some of the function blocks.

Thanks! Llama3.1-Instruct has been added to the leaderboard.
Repository: evalplus/evalplus
Issue: more contracts?
Root cause: Documentation Deficiency

I feel more contracts can be added. For example, HumanEval/99 below has no contract:

```
def closest_integer(value: str) -> int:\n    '''\n    Create a function that takes a value (string) representing a number\n    and returns the closest integer to it. If the number is equidistant\n    from two integers, round it away from zero.
```

would be great to add a simple contract like this:

```
try:
    float(value)
except ValueError:
    assert False
```

@Kristoff-starling Can you help add the contract accordingly? Thanks!

Hi @crhf, I've checked HumanEval/99 in the latest version and there are already contracts for this problem. On my side it looks like this:
```python
try: # $_CONTRACT_$
    assert isinstance(value, str) # $_CONTRACT_$
    float(value) # $_CONTRACT_$
except: # $_CONTRACT_$
    raise Exception("invalid inputs") # $_CONTRACT_$
```
Could you please double-check whether you're using the latest version? Thanks a lot!

I see. @crhf could you do `pip install evalplus --upgrade` to make sure your `evalplus` is of the latest version?

Feel free to reopen if the issue remains! Thanks.
Repository: evalplus/evalplus
Issue: Does mbpp+ have base and plus version same has humaneval+?
Root cause: Documentation Deficiency

To my understanding, humaneval+ has both base version and plus version, while base version is exactly the original version.

I am wondering does mbpp+ also have base and plus version? Currently I notice the mbpp+ (which is 0.2 version) only has 378 questions. I am wondering if I can also gain the result of mbpp base version by running your evalplus project.

Thank you:)

The base version of MBPP+ still only has 378 problems to get aligned with MBPP+.
It does not align with the original MBPP but I'd still suggestion using the EvalPlus version of MBPP which removes many broken evaluation tasks from the original MBPP.
Repository: evalplus/evalplus
Issue: Integrate code generation into main package
Root cause: Unimplemented Feature Gap

* Merge code generation into `evalplus` directory
* One command to run both generation and evaluation
* Package isolation

Implemented by:

* a04f3251bcc44b151f42428f21636ea1d65e287b
* ca0acd3e6c19c323e7ff5dd5037ce04daca6f7df
Repository: evalplus/evalplus
Issue: Request for evaluating models running on TGI
Root cause: Documentation Deficiency

Hi,

I was going through this repo and I looked at the arguments that may be specified along with the model being evaluated. I see that in the engine arguments, there's no TGI but there is HuggingFace Transformers library. Are there plans to support TGI in the future?

Thank You

We support the OpenAI interface -- you can launch a TGI instance and request it via OpenAI API by setting the right base URL.

https://huggingface.co/docs/text-generation-inference/en/messages_api
Repository: evalplus/evalplus
Issue: Need help: command line execution error with evalplus
Root cause: Environment Incompatibility

Hi, thanks for the great work! I am trying to apply evalplus to my own humaneval pipeline with simple pytorch backend. When I tried command line which is the same as the tutorial page:
```
evalplus.evaluate --dataset humaneval --samples ./data/temp.jsonl --model "deepseek-ai/deepseek-coder-6.7b-base" --backend hf --greedy
```
It shows:
```
usage: evalplus.evaluate [-h] --dataset {humaneval,mbpp} --samples SAMPLES [--base-only] [--parallel PARALLEL] [--i-just-wanna-run] [--test-details] [--min-time-limit MIN_TIME_LIMIT]
                         [--gt-time-limit-factor GT_TIME_LIMIT_FACTOR] [--mini] [--noextreme]
evalplus.evaluate: error: unrecognized arguments: --model deepseek-ai/deepseek-coder-6.7b-base --backend hf --greedy
```
And then I tried `evalplus.codegen` it shows `evalplus.codegen : command not found`. However, I believe it should combine inference with evaluation just as the README said so. Did I miss something in the tutorial?

My runtime environment:
- python 3.9.2
- evalplus: 0.2.1
- transformers 4.44.2

Can you install the HEAD version of EvalPlus?

pip install --upgrade "git+https://github.com/evalplus/evalplus.git"

Thanks, problem solved. I will close this issue soon.
Repository: evalplus/evalplus
Issue: [HumanEval+ {0,3,9,148}] `plus_input` is exactly same as `base_input`
Root cause: Validation Gap

For HumanEval/0, HumanEval/3, HumanEval/9, and HumanEval/148, the 'plus_input' is  exactly same as 'base_input', which makes me quite confused about the result. (For the dataset in evalplus.data.get_human_eval_plus() )

Thanks for reporting the important issue. This is not intended as the `plus_input`  should not overlap that of the base.

After some quick investigation, I found such issues start to exist since HumanEval+ v0.1.3 (i.e., v0.1.2 is fine)

https://colab.research.google.com/drive/1UupXDR78WMIM-AOpspNkpBZbCcBAt7SX?usp=sharing

due to the dataset repair script by  @ganler :

https://github.com/evalplus/evalplus/blob/master/tools/humaneval/fix_v012.py

I am now taking a closer look to fix the issue and will get everything updated asap. 

Hi @ShuyinOuyang the bug should be fixed. Can you kindly try to clean the the cache folder with `rm -rf ~/.cache/evalplus` and retry your code? EvalPlus will automatically re-download and use the fixed versions.

Also to clarify the impact of the bug:

It will make the four programming tasks (0,3,8,148) as easy as the original HumanEval. In other words, the achieved pass@k will be slightly higher. We will also re-evaluate LLMs that using the fixed HE+ versions and update the results on https://evalplus.github.io/leaderboard.html

Thanks!

Thanks for the prompt action! By the way, I also find one test input in the 'plus_input' for HumanEval_160 (id: 303) the test case input is [['\*\*', '\*\*', '+'], [2, 7, 10, 3]]. For the ground truth solution, it takes me quite a while to compute the result for 2\*\*7\*\*10+3. I personally think this is not a good test case input to evaluate the generated code's test pass rate, since it takes too much time and might have the potential to fail the test.

That is interesting. The way we set time tolerance is via differential testing. Even if the ground-truth is slow, it does not mean HE+ will enforce the same line of latency requirements over the LLM solutions. 

![image](https://github.com/evalplus/evalplus/assets/38074777/d89cd764-f042-4ffe-aada-9f4cfda5f664)

That being said, I am neutrual to your proposed change. But let me cc: @brutalsavage for more opinons.
 

I checked, and the groundtruth solution provided in EvalPlus, does not take too long (~1-2 seconds), Although this test case seems to computing a very large number (it even exceeded the default python int to string limit) so we could remove it

@Kristoff-starling would you like to help remove such a test-case and pop HE+ up to v0.1.7? Thanks!

Hi @ShuyinOuyang, the HE+/160 inputs leading to huge answers should have been removed, and you could use `pip install "git+https://github.com/evalplus/evalplus.git" --upgrade` or pull the source code to try the latest version. Thanks for the catch!
Repository: evalplus/evalplus
Issue: How to use evalperf to evaluate generated code?
Root cause: Documentation Deficiency

The runtime environment I’ve set up should be fine, as I can run python evalplus/evaluate.py normally and get evaluation results.

However, when I run python evalplus/evalperf.py xxx.jsonl HumanEvalPlus.jsonl ./results/ --max_parallelism=1, all code evaluation results show 0 correct solutions.

P.S.: xxx.jsonl is the generated code I need to evaluate using evalperf.py, and it can be successfully evaluated with evalplus/evaluate.py, showing accuracy results.

Hi @Hwwg 

You can evaluate your models using EvalPerf via a few commands listed in https://github.com/evalplus/evalplus?tab=readme-ov-file#code-efficiency-evaluation-evalperf-nix-only
Repository: evalplus/evalplus
Issue: KeyError: 'entrypoint' while running codegen 
Root cause: Validation Gap

I am getting this issue 
I was using the same code a couple of days back, and it was functioning. 
installation 
```
!pip install --upgrade "git+https://github.com/evalplus/evalplus.git"
# commit e58663becd8896cba534485260288cadb8c8edf0
```
command executed 
`!evalplus.codegen --model "meta-llama/Meta-Llama-3.1-8B" --greedy --root /content/temp --dataset mbpp --backend vllm
`
```
  File "F:\workspace\llm\envllm\lib\site-packages\fire\core.py", line 477, in _Fire
  File "F:\workspace\llm\envllm\lib\site-packages\fire\core.py", line 693, in _CallAndUpdateTrace    
    component = fn(*varargs, **kwargs)
  File "F:\workspace\llm\evaluation\evalplus\evalplus\codegen.py", line 217, in run_codegen
    codegen(
  File "F:\workspace\llm\evaluation\evalplus\evalplus\codegen.py", line 101, in codegen
    solution, entrypoint=task["entrypoint"]
KeyError: 'entrypoint'
```

Fixed by a8a46b95793059cefc5fec53590ec7f715577934
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Claude 3.5 Sonnet & o1OpenAI
Root cause: Unimplemented Feature Gap

### Model introduction

https://www.anthropic.com/news/claude-3-5-sonnet

https://openai.com/index/introducing-openai-o1-preview/

### Model URL

https://www.anthropic.com/news/claude-3-5-sonnet

### Additional information (Optional)

_No response_

### Decontamination

n/a

### Author

No

### Data

Yes (If so please specify in 'Additional information')

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Would also be interested 

Yeah same

Updated.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - DeepSeek-V2.5
Root cause: Unimplemented Feature Gap

### Model introduction

the model is created by DeepSeek team.

### Model URL

https://huggingface.co/deepseek-ai/DeepSeek-V2.5

### Additional information (Optional)

_No response_

### Decontamination

As they did in Deepseek-v2

### Author

No

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Updated.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - deepseekv2-16b-tracewise
Root cause: Unimplemented Feature Gap

### Model introduction

Recent research revealed that Code LLMs have limitations in reasoning runtime behavior and cannot understand what the programs are actually doing. To tackle this problem, our works have been proposed to add reasoning-related semantics such as execution behaviors into the prompt to train Code LLMs accordingly. Experimental results on five Code LLMs highlight that execution behaviors highly affect the fine-tuning and inference of Code LLMs, where our designed representation Concise is the most effective one in terms of enhancing the code generation ability of Code LLMs.


### Model URL

https://huggingface.co/programExec/deepseekv2-16b-tracewise

### Additional information (Optional)

please evaluate it by vLLM;
The training data will public once paper upload to arxiv.

### Decontamination

To be rigor, we will claim in the upcoming technical report.

### Author

Yes

### Data

Yes (If so please specify in 'Additional information')

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

I'm not related with the evalplus project.

That sounds like a great idea. I will try out your model for my personal use. It's so low profile on HF with that generic model card that it would be impossible to notice it otherwise. I would love to read the paper once it's released!!

close for waiting too long. Open model-eval on SWE-bench for real-world issues.
Repository: evalplus/evalplus
Issue: Evaluation setting of the leaderboard
GitHub labels: good first issue

What is the evaluation setting on the leaderboard? 
I got a different score for bigcode/starcoder2-15b-instruct-v0.1 on both humaneval and mbpp.

Can you show more details?

I get leaderboard result from https://evalplus.github.io/leaderboard.html
I use greedy decoding and vllm as backend, but get different pass@1 results for model [bigcode/starcoder2-15b-instruct-v0.1](https://huggingface.co/bigcode/starcoder2-15b-instruct-v0.1).
| | humaneval | humaneval+ | mbpp | mbpp+ |
| ------ | ------ | ------ | ------ | ------ |
| mine | 68.29 | 57.32 | 75.93 | 63.49 |
| leaderboard | 67.7 | 60.4 | 78 | 65.1 |

My environment is
```
transformers==4.45.2
torch==2.1.2
vllm==0.3.3
```
I want know the evaluation setting on the leaderboard.
May I get the inference results in the leaderboard?
Any other details is needed for attention?

I see. I feel the diff is not large and multiple reasons can lead to such small inconsistencies:

- There have been a bit of dataset oracle fixes since the leaderboard version (half year back) which can cause a little bit of score improvements
- vLLM versions -- very likely `v0.4.0+`
- https://github.com/evalplus/evalplus/blob/master/docs/execution.md

Overall, I'd say just use the scores you reproduced as long as you compare them with other models under the same setting.

Just to add a bit why I say these are small inconsistencies:

Per [eval-arena](https://github.com/crux-eval/eval-arena)'s finding, a diff of 6.7% might be considered significant: 

> According to our analysis, a difference of 6.7% is the minimum required to achieve a p-value of 0.05 across all pairwise comparisons.

https://github.com/crux-eval/eval-arena?tab=readme-ov-file#popular-benchmarks-are-noisier-than-assumed-by-popular-papers