zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: Codegen can not run under V100 GPUs
Root cause: Architectural Constraint

For low-level GPUs, like V100.
We have to change 2 arguments in vllm.LLM setup process to make it run successfully. (Otherwise, assertion failed 😢)

At https://github.com/evalplus/evalplus/blob/master/evalplus/provider/vllm.py#L30
```
# For V100 gpu, apply these changes, reference link: https://github.com/vllm-project/vllm/issues/2729#issuecomment-2390863280
if gpu == V100:
  kwargs["enable_prefix_caching"] = False
  kwargs["enable_chunked_prefill"] = False
```


At https://github.com/evalplus/evalplus/blob/master/evalplus/codegen.py#L146
```
    dtype: str = "bfloat16" if gpu != V100 else "float16",
```

Should we adopt these changes?

we should address this more fundamentally (instead of using hacks):

-- maybe just make `enable_prefix_caching` configurable in the command line.

it's fine to by default enable or disable it -- but user has some controllability to toggle the flag when detailed hw setting needs
Repository: evalplus/evalplus
Issue: Error evaluating samples
Root cause: Validation Gap

Hello Jiawei Liu,

I am reaching out to you because I am encountering an error while evaluating the `samples.json` file. I am able to pass `sample.json` file that contains all **[164 human eval prompts X 200 samples]** without any errors. 

However, when I try to pass the `sample.json` file that has **[19 human eval prompts X 2 samples]**, I receive the following error message:

`File "/usr/local/lib/python3.10/dist-packages/evalplus/evaluate.py", line 175, in evaluate_humaneval
`
```
assert len(completion_id) == len(problems), "Missing problems in samples"
AssertionError: Missing problems in samples
```
Does the `sample.json` file have to contain **164 X 200** ? 
Could you please help me understand the error and guide me on how to resolve it?

Hi @Shlok-crypto, yes the current code assumes that sample.json should contain the same amount of problems (164), note the number of samples does not need to be 200 i.e., [164 prompts X 2 samples] will still work.

There is a work around for the exact problem you are facing, you can check this issue thread #21 for more detail 

Hi @brutalsavage, Thanks for getting back to me. I looked over #21 and was able to resolve the issue. However, I noticed it only evaluates for Pass@1 not [**1, 10, 100**]. 
```
Base
{'pass@1': 0.9263157894736841}
Base + Extra
{'pass@1': 0.8663157894736842} 
```
Could you help me resolve this issue? 

Hi, only evaluating Pass@1 is probably because you only have 2 samples per each HumanEval problem. In general pass@K can be computed if the number of samples N is greater than K. In your situation since you only have 2 samples, its not possible to compute pass@10 or pass@100.  Additionally, it is recommended that your N (number of samples) be much larger than K, (i.e., pass@100 should use 200 samples) to get a reliable results. 

I have a quick question: to fairly compare with the results in the current leaderboard, the sample size should be 200, right? 

@JiyangZhang The leaderboard setting uses greedy decoding. i.e., one sample and a temperature of zero.
Repository: evalplus/evalplus
Issue: How to calculate the Win Rate on Evalperf?
Root cause: Documentation Deficiency

Hello,

I’m conducting some evaluation on **evalperf**, but the results don’t include the **win rate**. How can I calculate it? Additionally, since I’m testing in a container environment, I can’t use the command `sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid'` to set `perf_event_paranoid` to `0`,  and its current value is `2`. Will this significantly impact the evaluation results?

Thank you!

Hi @Hambaobao 

You can compute that with the leaderboard source code:

```bash
git clone git@github.com:evalplus/evalplus.github.io.git
cd evalplus.github.io.git
# COPY your results (i.e., .brief.json) 
cp /path/to/X.brief.json results/evalperf
python results/evalperf/stats.py
```

Then you can just open the `evalperf.html` to check the results.

We will also try to optimize the user experience by showing the win-rate in the outputs soon.


Thank you very much for your response.
I have one more question. I’ve noticed that there are various prompt types available for evaluation, such as `instruct`, `perf-instruct`, and `perf-CoT`. Could you let me know which prompt is used on the leaderboard, along with the specific parameters for generation? I saw in the paper that **temperature** is set to `0.2` and **n_samples** to `50`, but I’m not entirely sure if this aligns with what’s used on the leaderboard.

Just use the default args (no need to set temp, etc) and you can align with the leaderboard config. 

The current setup in the repo improves the paper setup a bit to take into account more passing samples. You can check the EvalPerf doc for more details. 

https://github.com/evalplus/evalplus/blob/master/docs/evalperf.md

Thank you. 🙏

Just added some guide on the webpage via https://github.com/evalplus/evalplus.github.io/pull/24


>  Additionally, since I’m testing in a container environment, I can’t use the command sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' to set perf_event_paranoid to 0, and its current value is 2. Will this significantly impact the evaluation results?

I think you need to set it outside docker. e.g.,

```bash
# Code execution within Docker
sudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid' # Enable perf
docker run --cap-add PERFMON --rm --pull=always -v $(pwd)/evalplus_results:/app ganler/evalplus:latest \
           evalplus.evalperf --samples /app/evalperf/ise-uiuc--Magicoder-S-DS-6.7B_vllm_temp_1.0.jsonl
```

Thanks @ganler 

I’m using the company’s cluster, so I can’t access the host machine. However, I really appreciate your help; it has been very useful. 🤗
Repository: evalplus/evalplus
Issue: Flaky return value of None in EvalPerf
Root cause: Validation Gap

The code will report error [here](https://github.com/evalplus/evalplus/blob/c05b20b24ab6783878d9ffa1ae5496e83da0e65b/evalplus/evalperf.py#L196): 

```python
ref_solution = ptask["reference"][idx]
for _ in range(2):  # at most retry twice
  profiles = profile(                                          # <----- notice here
    ref_solution,
    entry_point,
    [pe_input],
    timeout_second_per_test=evaluation_time,
   )

  # Bad thing#1: timeout / failure happens
  if are_profiles_broken(profiles):
    print(f"{task_id}: [WARNING] Error in ref: {profiles}")
    rich.print(Syntax(ref_solution, "python"))
    print(f"{task_id}: Retrying w/ +10s timeout...")
    evaluation_time += 10
  else:
    break
avg_profile = mean(profiles)                        # <----- report error here
```

The profile function will return string
```python
def profile(
    func_code: str,
    entry_point: str,
    test_inputs: List[Any],
    timeout_second_per_test: float,
    memory_bound_gb: int = PERF_RAM_GB_PER_PROC,
    profile_rounds: int = PERF_PROFILE_ROUNDS,
    profiler: Callable = num_instruction_profiler,
    warmup_inputs: Optional[List[Any]] = None,  # multiple inputs
) -> List[int | float | str]:
    """Profile the func_code against certain input tests.
    The function code is assumed to be correct and if a string is returned, it is an error message.
    """
    timeout = timeout_second_per_test * len(test_inputs) * profile_rounds

    def _run():

        .......... # some unrelated code

        if progress.value == _STAT_SUCC:
            return compute_cost.value
        elif progress.value == _STAT_NONE:
            return "PROFILING DID NOT START"
        elif progress.value == _STAT_ERROR:
            return "SOLUTION ERROR ENCOUNTERED WHILE PROFILING"

    return [_run() for _ in range(profile_rounds)]
```

I also encountered this for one time and I just reran it to fix. Need to look closer to understand why. Thanks for the report!
Repository: evalplus/evalplus
Issue: [Leaderboard UX] Making MoE model sizes more clear
Root cause: Documentation Deficiency

Calling from the other projects, could you fix up the dashboard so that DeepSeek won't rank so high even when they were supposed to be MoE? https://github.com/bigcode-project/bigcodebench/issues/59

Hello, it's not a "bug" as in the leaderboard we had a note:

> "Size" here is the amount of activated model weight during inference.

It also says "Act. Size" not simply "Size".

Overall I agree that if we use "activated param" -- it's like a bit of over-positive for MoEs; yet if you simply use "all param", it leads to over-negative for MoEs. But the thing is we can only visualize two axes for readability so we have to pick one. :/


It's a nice note tho -- we will think about how to make it a bit more clear. Probably need to specialize MoE types.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Llama3.2 collection
GitHub labels: model eval

### Model introduction

Meta's latest LLM collection, with 1B and 3B sizes.

### Model URL

https://huggingface.co/collections/meta-llama/llama-32-66f448ffc8c32f949b04c8cf

### Additional information (Optional)

_No response_

### Decontamination

No decontamination involved.

### Author

No

### Data

No

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Hello, what kind of collection is needed here? Can you please explain more?


> Hello, what kind of collection is needed here? Can you please explain more?

with size of 1b and 3b, also the base and instruct models too.

What about 3.3 (assuming Llama 4 is bad)
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - HumanEval/32 - Issues in HuggingFace Version
Root cause: External Dependency Breakage

### EvalPlus version

latest

### Output of running `ls ~/.cache/evalplus`

none

### Task ID of the programming task

HumanEval/32

### Test input

```python
See `Description`.
```


### Description

I encountered an issue with task `HumanEval/32` in the HumanEval+ dataset on [Hugging Face](https://huggingface.co/datasets/evalplus/humanevalplus). When testing the canonical solutions, the following error occurs:
```bash
    assert _poly(*candidate(*inp), inp) <= 0.0001
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Value after * must be an iterable, not float
```
This issue has been previously identified and fixed in evalplus repository (Issue #125). The fix involves changing:
```python
assert _poly(*candidate(*inp), inp) <= 0.0001
```
to
```python
assert _poly(*inp, candidate(*inp)) <= 0.0001
```
However, two problems remain:
1. The test code on Hugging Face hasn't been updated to reflect this fix;
2. Even with the fixed assertion, the canonical solution still fail on some test cases with AssertionError;

Could you please:
1. Update the HumanEval+ dataset on Hugging Face to use the corrected test code;
2. Verify if the canonical solutions are expected to pass all test cases with the fixed assertion;

This update would help users who are using the Hugging Face version directly rather than the GitHub repository.

### Other context

_No response_

Another issue in [MBPP+](https://huggingface.co/datasets/evalplus/mbppplus) 590:
```bash
AssertionError: out: ((42.326324330254145, -2.359036256570777), (-2+2.4492935982947064e-16j)), exp: ((42.32632433025415, -2.359036256570777), (-2+2.4492935982947064e-16j))
```

i found the same issue in example 32. could you update the version in hf?
Repository: evalplus/evalplus
Issue: Evaluation request: Phind/Phind-CodeLlama-34B-v2
GitHub labels: enhancement

good first issue

I'm curious how well https://huggingface.co/Phind/Phind-CodeLlama-34B-v2 scores on your leaderboard, as it's currently one of the best offline code generation models.

Thanks for bringing this up. Kindly find the evaluation results for Phind-CodeLlama & Mistral-7b on https://evalplus.github.io/leaderboard.html.

Interestingly, if we evaluate `Phind-CodeLlama` on HumanEval, it is ranked as No 4 (worse than ChatGPT & WizardCoder). But on our rigorous HumanEval+, it is ranked as No 2! This might imply that `Phind-CodeLlama` is good at producing higher quality code which better deals with edge cases.

@ganler I did today own evalplus test with 0.7073 score !
```
(base) root@C.7286656:~/kamil/evalplus$ evalplus.evaluate --dataset humaneval --samples ../workdir/humaneval/phind-code-llama-34b-v2_temp_0.0-sanitized/
Load from ground-truth from /root/.cache/evalplus/2670b22d020cab45ede5e8285730b6ee.pkl
Reading samples...
164it [00:00, 726.60it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 164/164 [00:03<00:00, 53.42it/s]Base
{'pass@1': 0.7073170731707317}
Base + Extra
{'pass@1': 0.6707317073170732}
```
Repository: evalplus/evalplus
Issue: Why results for HumanEval and MBPP for llama 3.1 different from those in evalplus leaderboard?
Root cause: Documentation Deficiency

Based on this official blog, https://ai.meta.com/blog/meta-llama-3-1/ and https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct, the results for llama 3.1 8b instruct for humaneval and mbpp are 72.6 and 72.8 respectively. However, on the evalplus leaderboard, they are both much lower, whether for base test or evalplus tests. Why?

Even in the case of the current version, the reproduced scores are 70.7 and 77.0 for humaneval and mbpp respectively.

There could be multiple factors that lead to inconsistency -- please refer to the earlier reply below:

https://github.com/evalplus/evalplus/issues/236#issuecomment-2430997706

In general, testing timeout budgets and detailed vllm/huggingface versions can impact results. Oftentimes 2 points of discrepancy can be expected but with the upstream libraries fixing model bugs the scores can be better and better.

Regarding the official Llama results -- I am not sure if they are directly using the EvalPlus framework and usually industries adopt our datasets and run things on their own framework for engineering and performance reasons.

@jackyoung96 are you saying that you are able to get 70.7 and 77.0 for llama-3.1-8b-instruct?

@ganler are you aware of any methods that i can do to bring the results closer?

> [@jackyoung96](https://github.com/jackyoung96) are you saying that you are able to get 70.7 and 77.0 for llama-3.1-8b-instruct?

Yes I did.

> [@ganler](https://github.com/ganler) are you aware of any methods that i can do to bring the results closer?

Not sure what your actual concern is --- if you are concerned that a model could be under-presented, we can always re-evaluate them using the latest version and use whichever score is better.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] Evaluate NovaSky-AI / Sky-T1-32B-Preview
Root cause: Unimplemented Feature Gap

### Model introduction

small but high performance coding and math model,(they said it beat o1 preview in math and coding) so lets test it.

### Model URL

https://huggingface.co/NovaSky-AI/Sky-T1-32B-Preview

### Additional information (Optional)

https://github.com/NovaSky-AI/SkyThought

### Decontamination

everything is open and accessible (data, code, report, etc) https://novasky-ai.github.io/posts/sky-t1/ 
https://github.com/NovaSky-AI/SkyThought

### Author

No

### Data

Yes (If so please specify in 'Additional information')

### Security

- [X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - DeepSeek R1 (and their Distills)
Root cause: Unimplemented Feature Gap

### Model introduction

Currently DeepSeek made R1 as a deep thinking model, and with them comes Distills based on Llama and Qwen

### Model URL

https://huggingface.co/deepseek-ai/DeepSeek-R1

### Additional information (Optional)

_No response_

### Decontamination

See the following https://arxiv.org/abs/2501.12948

### Author

No

### Data

No

### Security

- [x] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [x] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Crosslink to both R1 and "R1.1" https://github.com/bigcode-project/bigcodebench/issues/100
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - <HumanEval/132> 题意理解有误 The meaning of the question is wrong.
Root cause: Validation Gap

### EvalPlus version

0.3.1

### Output of running `ls ~/.cache/evalplus`

没有这个文件夹 There is no such folder.

### Task ID of the programming task

HumanEval/132

### Test input

```python
is_nested('[][][]') == False
```

### Description

上面只是错误的测试用例之一。实际上，参考代码的编写者似乎误解了该问题的题意。问题中说，“there is a valid subsequence of brackets”，但这里的子序列似乎被误解为了子串，导致参考代码使用子串的逻辑进行求解。例如，'[][][]'中包含'[[]]'这个双层嵌套括号子序列，但不包含对应子串，所以正确输出本该为True，而按子串却是False。事实上，HumanEval的原数据集中这个问题的参考代码是正确的，但不知道是出于什么原因在EvalPlus中修改了。

The above is just one of the wrong test cases. In fact, the author of the reference code seems to have misunderstood the meaning of the question. The question says, "there is a valid subsequence of brackets", but the subsequence here seems to be misunderstood as a substring, causing the reference code to use the logic of the substring to solve. For example, ，'[][][]' contains a double-level nested parenthesis subsequence of '[[]]', but does not contain the corresponding substring, so the correct output should be True, but by substring it is False. In fact, the reference code for this question in HumanEval's original dataset is correct, but it is not known why it was modified in EvalPlus.

### Other context

_No response_
Repository: evalplus/evalplus
Issue: Test deepseek with zero score
Root cause: Environment Incompatibility

I try to evaluate deepseek through an API on HumanEval, however I got 0.00 pass@1 score, this is strange, here is the result json file I got.

[deepseek-chat_openai_temp_0.0.eval_results.json](https://github.com/user-attachments/files/19271571/deepseek-chat_openai_temp_0.0.eval_results.json)

I wonder is it because I use Windows system to run this evaluation, or maybe some other reasons?

https://github.com/evalplus/evalplus/pull/262 has been merged.
Repository: evalplus/evalplus
Issue: Will this project support Ascend NPU (e.g. 910B)?
Root cause: Unimplemented Feature Gap

This project is excellent and efficient!

Will it support the Ascend NPU? I directly downloaded this project in the 910B cluster and performed codegen with the hf backend. The model and dataset are loaded, but the inference seems stuck.

Here is my shell:
```
evalplus.codegen --model local/path/to/llama2-7b-hf --greedy --dataset humaneval --backend hf --device_map auto
```

The terminal showed 2 humaneval bars but kept 0/164 for a few minutes. And no output file is created.

How could I manage it? Or does this project not support the Ascend NPU?

Thanks for your work.


Simply use Ascend vllm and host it as a endpoint, no need to play aorund with this on transformers.

No plans on that from our side but contributions are welcome!
Repository: evalplus/evalplus
Issue: Incomplete contracts in problems involving balanced parantheses (1, 6)
Root cause: Validation Gap

### HumanEval 1 (`separate_paren_groups`)
There are two possible interpretations of the docstring.
1. `paren_string` must be balanced.
This will imply that every bracket in the input string will appear in one of the groups in the output list. Then, the complete contract is
```py
...
cnt = 0
for ch in paren_string:
    assert ch in ["(", ")", " "], "invalid inputs"  # $_CONTRACT_$
    if ch == "(": cnt += 1
    if ch == ")": cnt -= 1
    assert cnt >= 0, "invalid inputs"
assert cnt == 0 # MISSING STATEMENT
```
2. `paren_string` need not be balanced.
In this case, we don't need this part of the contract at all. This will imply that we only group those brackets which form a balanced group, and leave the rest. However, then this should allow imbalance of both kinds (`cnt > 0` and `cnt < 0`), while only one is allowed by the contract.
---
### HumanEval 6 (`parse_nested_parens`)
Same issue as above.

@mrigankpawagi Thanks for this. For `separate_paren_groups` and `parse_nested_parens` I think the entire string should be balanced, not just individual group. That would be the most natural interpretation of the docstring in my opinion.

@ganler @Kristoff-starling any thoughts?

> `paren_string` must be balanced.

Sure thing and your proposed change definitely makes more sense.

> `paren_string` need not be balanced.

I don't feel it can be interpreted this way as the description says "Separate groups are balanced (each open brace is properly closed)". If separate groups are balanced then their concatination must be balanced too. 

That being said, I will adopt your first interpretation and fix in the next revision. :) Thanks for the reports!
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - <HumanEval/8> <Incorrect input type for plus_input[446] and plus_input[595]>
Root cause: Validation Gap

### EvalPlus version

0.3.1

### Output of running `ls ~/.cache/evalplus`

NA

### Task ID of the programming task

HumanEval/8

### Test input

```python
`plus_input[446]`
`plus_input[595]`
```

### Description

for `HumanEval/8`, two cases in plus_input have incorrect type
`plus_input[446]`: input is an empty string `""`
`plus_input[595]`: input is an empty dict `{}`
instead of `List[int]`

### Other context

_No response_
Repository: evalplus/evalplus
Issue: About pass@k
Root cause: Documentation Deficiency

Hello, I would like to know whether the k value in pass@k can be adjusted? I tried changing it using the --gt-time-limit-factor parameter, but it didn’t seem to have any effect.

The same question. How to evaluate pass@5?

Sorry for the late reply. 

Using `--n-samples` can control N. 

To set a customized K, you will need to modify:

https://github.com/evalplus/evalplus/blob/d362e933265c3e7e3df8101c930a89c3c470cd9f/evalplus/evaluate.py#L329
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - HumanEval/0 , Asseration Oracle is Wrong
Root cause: Validation Gap

### EvalPlus version

[[1.0, 2.0, 3.0, 4.0, 5.0], 1.0], False

### Output of running `ls ~/.cache/evalplus`

AssertionError

### Task ID of the programming task

has_close_elements

### Test input

```python
https://github.com/evalplus/humanevalplus_release/releases
the 18th input,

0 ([[1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3], True)
1 ([[1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05], False)
2 ([[1.0, 2.0, 5.9, 4.0, 5.0], 0.95], True)
3 ([[1.0, 2.0, 5.9, 4.0, 5.0], 0.8], False)
4 ([[1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1], True)
5 ([[1.1, 2.2, 3.1, 4.1, 5.1], 1.0], True)
6 ([[1.1, 2.2, 3.1, 4.1, 5.1], 0.5], False)
7 ([[1.0, 2.5, 3.0, 4.5, 5.0, 6.5], 0.4], False)
8 ([[1.0, 2.0, 3.0, 2.9, 5.0, 6.0], 0.1], False)
9 ([[0.5, 0.6, 0.7, 0.8, 0.9], 0.05], False)
10 ([[10.5, 20.5, 30.5, 25.5, 40.5], 4.0], False)
11 ([[1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7], 0.2], False)
12 ([[0.1, 0.5, 1.0, 1.5, 2.0], 0.1], False)
13 ([[1.2, 2.4, 3.6, 4.8], 0.8], False)
14 ([[-1.0, -0.5, 0.0, 0.5, 1.0], 0.3], False)
15 ([[-2.0, 2.0, 3.0, 4.0], 1.5], True)
16 ([[1.0, 2.0, 3.0, 4.0, 5.0], 0.5], False)
17 ([[1.0, 2.0, 2.9, 5.0, 6.0], 4.4], True)
18 ([[1.0, 2.0, 3.0, 4.0, 5.0], 1.0], False)

check(has_close_elements)
assertion(candidate(*inp), exp, 0)
assert exact_match
AssertionErro
```

### Description

Here is the wrong test input for has_close_elements, which should be removed.

### Other context

_No response_