zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: [Improvement] HumanEval/111 canonical solution - ValueError on space-only input
Root cause: Validation Gap

### Description
The canonical solution for HumanEval/111 (`histogram`) in the HumanEval+ dataset throws a `ValueError` when the input string contains only spaces, which contradicts the expected behavior.

### Problem
The canonical solution attempts to call `max()` on an empty list when all words after splitting are empty strings:

### Steps to Reproduce
```python
histogram(" ")  # Single space
histogram("   ")  # Multiple spaces
```

### Expected Behavior
Ideally, the function should return an empty dictionary `{}` for strings containing only spaces.

### Actual Behavior
```
ValueError: max() iterable argument is empty
```
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Kimi K2 (and Kimi-Dev)
Root cause: Unimplemented Feature Gap

### Model introduction

Currently Kimi claims to have beaten DeepSeek and some of the SOTA proprietary weight models in programming with the SWE-Bench, but not necessarily the other ones.

### Model URL

https://huggingface.co/moonshotai/Kimi-K2-Instruct

### Additional information (Optional)

_No response_

### Decontamination

(not sure)

### Author

No

### Data

No

### Security

- [x] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [x] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: Contract misses infinities and nan (2, 99)
Root cause: Validation Gap

### HumanEval 2 (`truncate_number`)
Contract misses positive infinity. `assert number > 0` eliminates `-inf` and `nan`, but passes for `inf`. This later causes `int(number)` to fail. The complete contract should be
```py
...
assert number > 0, "invalid inputs"
assert isinstance(number, float), "invalid inputs"
assert not math.isinf(x) # MISSING STATEMENT
```
---
### HumanEval 99 (`closest_integer`)
All of `inf`, `-inf` and `nan` pass the fuzz test `float(value)`, but later cause failure due to `int(val)`. The complete contract should be
```py
...
assert not (math.isinf(value) or math.isnan(value)) # MISSING STATEMENT 
```

Thanks for the report. We will fix this in the next revision. BTW, also note that these invalid inputs are not included in our current dataset which would not hurt its current validity :).
Repository: evalplus/evalplus
Issue: .to(...) breaks when device_map="auto" is used
Root cause: Validation Gap

When using device_map="auto" with Hugging Face's Accelerate, the model is automatically split across available devices (e.g., multiple GPUs, CPU, or even disk). In this setup, manually calling .to(...) on the model causes a runtime error:

self.model = self.model.to(self.device)

Error:

RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.

This happens because Accelerate handles device placement internally, and attempting to override it interferes with its setup.
Suggested Fix

Wrap the .to(...) call with a condition to avoid breaking the device mapping:

if device_map is None:
    self.model = self.model.to(self.device)

This ensures .to(...) is only called when device mapping is not handled automatically.

I think this is fixed by https://github.com/evalplus/evalplus/pull/281

Thank you!
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Qwen3-Next
Root cause: Unimplemented Feature Gap

### Model introduction

The mixed attention model is out

### Model URL

https://huggingface.co/collections/Qwen/qwen3-next-68c25fd6838e585db8eeea9d

### Additional information (Optional)

_No response_

### Decontamination

Dunno, just check it, yout benchmark is dated anyways

### Author

No

### Data

No

### Security

- [x] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [x] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - FILL_THE_MODEL_NAME_HERE
Root cause: Unimplemented Feature Gap

### Model introduction

I’d like to request evaluation of a new Hugging Face model:
👉 https://huggingface.co/rfcoder0/qwen3-4b-custom-Sile
	•	Model size: 4B parameters
	•	Training hardware: RTX 3060 + RTX 3070 (consumer GPUs)
	•	Performance: Rivals Qwen-3 8B 
	•	Intended use: General-purpose chat / instruct

This is a custom 4B model designed to show that high performance is possible with modest hardware. Would be great to see it included in your evaluation sweeps/leaderboards for comparison.

Thank you!


### Model URL

https://huggingface.co/rfcoder0/qwen3-4b-custom-Sile

### Additional information (Optional)

_No response_

### Decontamination

I didn’t use any eval sets the data was personal high quality data created by me. The only data that was created and synthetic would be my personal chats.

### Author

Yes

### Data

No

### Security

- [x] I confirm that the model is safe to run which is not designed to produce malicious code or content.

### Integrity

- [x] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - <HumanEval/0> <Wrong TC caused by floating point error>
Root cause: Validation Gap

### EvalPlus version

0.3.1

### Output of running `ls ~/.cache/evalplus`

X

### Task ID of the programming task

HumanEval/0

### Test input

```python
([[0.01, -0.2596313061937332, 0.03, 0.7320067664244609, 0.05, 0.06], 0.01], True)
```

### Description

The answer should be `False`, but Python calculates `0.06-0.05` as `0.009999999999999995`.
 

### Other context

_No response_
Repository: evalplus/evalplus
Issue: Contract misses empty list in problem 35
Root cause: Validation Gap

# HumanEval 35 (`max_element`)
Empty lists make `max(l)` fail. The complete contract should be
```py
...
assert type(l) == list, "invalid inputs"
assert all(type(x) in [int, float] for x in l), "invalid inputs"
assert l # MISSING STATEMENT
```

Thanks for the report. We will fix this in the next revision. BTW, also note that `[]` is not included in our current dataset which would not hurt its current validity :).
Repository: evalplus/evalplus
Issue: Error when using hf.py with python 3.11
Root cause: Validation Gap

When running

```
evalplus.codegen \
  --model "$MODEL" \
  --dataset humaneval \
  --backend hf \
  --greedy
```

I was getting 

```
RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0
```

And 

```
UserWarning: You are calling .generate() with the `input_ids` being on a device type different
than your model's device. `input_ids` is on cpu, whereas the model is on cuda.
```

I believe I fixed this by replacing `input_tokens.to(self.device)` with `input_tokens = input_tokens.to(self.device)` around line 77 in hf.py.

Can anyone reproduce the error I was getting and/or confirm that this is a sound change to make?

Thanks
Repository: evalplus/evalplus
Issue: Problem 140 -- wrong solution?
GitHub labels: bug

good first issue

incorrect groundtruth

The canonical solution for problem 140 is given by
```
def fix_spaces(text):
    """
    Given a string text, replace all spaces in it with underscores, 
    and if a string has more than 2 consecutive spaces, 
    then replace all consecutive spaces with - 
    
    fix_spaces("Example") == "Example"
    fix_spaces("Example 1") == "Example_1"
    fix_spaces(" Example 2") == "_Example_2"
    fix_spaces(" Example   3") == "_Example-3"
    """
    ### Canonical solution below ###
    ans = text
    for i in range(len(text)-1, 2, -1):
        ans = ans.replace(" " * i, "-")
    return ans.replace(" ", "_")
```
Shouldn't it be `len(text)` instead?

@Randl  Thanks for letting us know the issue. I think there is indeed a bug in the ground-truth when the whole string is a sequence of whitespaces. 

I think the expected ground-truth should be:

```python
def fix_spaces_v012(text):
    ans = text
    for i in range(len(text), 1, -1):
        ans = ans.replace(" " * i, "-")
    return ans.replace(" ", "_")
```

See proof of concept: https://colab.research.google.com/drive/1M066nrbOR3dzmoH4VjT73FlDa4BQPwTU?usp=sharing

I will correct this in HumanEval+ v0.1.2 soon. Also cc: @Kristoff-starling  @brutalsavage for confirmation.

Thanks for reporting this issue!

@ganler check the problem description 
```
Given a string text, replace all spaces in it with underscores, 
and if a string has more than 2 consecutive spaces, 
then replace all consecutive spaces with - 
```

Key: "more than 2 consecutive spaces should be replace with -". The current suggestion would replace 2 or more instead of more than 2.  

Example testcase: `fix_spaces_v012(" Example  3")` returns `"_Example-3"` instead of the correct `"_Example__3"`

```python
def fix_spaces_v013(text):
    ans = text
    for i in range(len(text), 2, -1):
        ans = ans.replace(" " * i, "-")
    return ans.replace(" ", "_")
```

This should fix it. 

Thanks for the catch. Indeed the `end` of range should be 2. 

So I assume `  Examp   le` should be `-Examp-le`, but `  Examp  le` should be `__Examp__le`

It looks like the first example you showed only has 2 spaces in the prefix so it should be `__Example-le` but the second example is right. 

Basically **more than 2** consecutive spaces should be replaced with a single `-`

It is indeed how people read it, but it says **all consecutive spaces**

Ah thats a great point, I reread this part as well when looking at the issue. This is something discussed in our work which is the vagueness of the prompts leading to incorrect LLM generations. While I agree with you the wording is a bit vague, I feel like the more than 2 consecutive spaces should be replaced with a single `-` is more accurate and makes more sense.
Repository: evalplus/evalplus
Issue: [Contract] Tracking HumanEval+ `v0.1.9` non-impacting minor issues
GitHub labels: program contract

incomplete

## HumanEval/35

I accidentally found in this task one contract is not marked as `$_CONTRACT_$` and is thus missed in the contract field but included in the `canonical_solution` field. However, this should be fine since `canonical_solution` is also used to filter out invalid inputs. Mention it here as minor "bug" and will wait any other import fix to bump into a new revision (`v0.1.9`).

## HumanEval/28

Another minor issue is from `HumanEval/28` where the func sig is `def concatenate(strings: List[str]) -> str:` but we included a few strings because the contract was `assert all(isinstance(s, str) for s in strings)`. It is not impacting any correctness as far as I can see but would be good to fix the minor issue.

Tracking two cases from other issues.

## HumanEval/32

#34

## HumanEval/160

#35
Repository: evalplus/evalplus
Issue: Re-evaluate previous LLMs' results over latest HumanEval+ 
Root cause: Configuration Error

Running in process. 

The first few batches of models are evaluated on HumanEval `v0.1.0` and after 5 months we have removed/simplified a few test-cases according to community feedback. Consequently, these models must be evaluated on the same and latest version of HumanEval+ for fair comparison.

Results will be updated on the [leaderboard](https://evalplus.github.io/leaderboard.html).

Pass@k has been updated at https://github.com/evalplus/evalplus.github.io/commit/37b35f2d2897c62077aad5e12d492222715a92e5

TL;DR:
GPT4, ChatGPT, CodeGen2b, CodeGen16b and Vicuna7b get pass@1 improvement by `0.6`.
Repository: evalplus/evalplus
Issue: Open-source the scripts / inference parameters for reproducing the leaderboard results
Root cause: Documentation Deficiency

Hi, thank you for this great repo!

The scripts in `codegen/` seems to be able to generate code on humaneval with many different language models.

However, many inference parameters still need to be set for generation, like temperature, top-p, etc. Could you kindly share the parameters you used to produce the results on the leaderboard?

Thanks for the issue!

You can find the concrete parameter setting here: https://github.com/evalplus/evalplus/blob/master/codegen/model.py

The temperature/greedy-decoding is configurable in https://github.com/evalplus/evalplus/blob/master/codegen/generate.py#L102

Note that overall we follow the original Codex paper (https://arxiv.org/abs/2107.03374) for configuring top-p and thus set them to 0.95 by default. You may tweak such parameters by cloning our repo and make edits. Thanks!
Repository: evalplus/evalplus
Issue: OOM error when running evaluation test on 34b model on a single A100 GPU with 80GB VRAM
Root cause: Configuration Error

How to do an evaluation test on a 34b model? I tried on a single A100 with 80GB VRAM but got an out-of-memory (OOM) error. I saw information that this test must be run on a single GPU, so how can I do it for a large model?


I am able to run Phind's 34B model on a single A100 with 80GB VRAM:

```shell
python codegen/generate --model "phind-code-llama-34b-v2" --greedy --n_samples 1 --temperature 0 --root [PATH_TO_STORE_OUTPUTS]
```

You may find detailed implementation here: https://github.com/evalplus/evalplus/tree/master/codegen

Ok thanks.
I have to check it again when my host be again online. 



@ganler  Anyway, is it possible run evalplus on multigpu machine ?

I tried again with oobabooga/CodeBooga-34B-v0.1 using `codegen/generate` and I got OOM
```
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 688.00 MiB. GPU 0 has a total capacty of 79.15 GiB of which 54.62 MiB is free. Process 4113546 has 79.09 GiB memory in use. Of the allocated memory 78.12 GiB is allocated by PyTorch, and 120.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

```(base) root@C.7286656:~/kamil/evalplus$ nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0
```

Ok all clear now. Phind-CodeLlama-34B-v2 works fine on 80GB VRAM and on the HF there are binary files *.bin

with model from oobabooga/CodeBooga-34B-v0.1 I got OOM, but there are safetensor files. 
is it possible that safetensor will use more vram?

Hi I have not tried `oobabooga/CodeBooga-34B-v0.1`. Oftentimes for such models you may want to tweak their default precision and `max_new_tokens` etc.

From what I see in https://huggingface.co/oobabooga/CodeBooga-34B-v0.1/blob/main/config.json#L23, it should be able to support float16. So you might add a few lines to turn it into float16 like https://github.com/evalplus/evalplus/blob/master/codegen/model.py#L169
Repository: evalplus/evalplus
Issue: Incomplete contract in HumanEval #32 (`find_zero`) 
Root cause: Validation Gap

The last coefficient must be non-zero for the validity of other conditions.

```py
...
assert len(xs) > 0 and len(xs) % 2 == 0, "invalid inputs" # $_CONTRACT_$
assert xs[-1] != 0, "invalid inputs" # THIS IS MISSING
```

There's actually more to this. Since the groundtruth is using the Newton-raphson method for finding roots, it is prone to failing near (or on) stationary points. I am not fully sure, but a method like bisection may be more reliable when we don't know much about the polynomial:

```py
    a = -1e5
    b = 1e5
    tol = 1e-5
    for _ in range(1000):
        fa = func(a)
        fb = func(b)
        if abs(fa) < tol: return a
        if abs(fb) < tol: return b
        if fa * fb > 0: return None
        m = (a + b) / 2
        fm = func(m)
        if abs(fm) < tol: return m
        if fa * fm < 0:
            b = m
        else:
            a = m
    return None
```
Repository: evalplus/evalplus
Issue: Incomplete contract in HumanEval #160 (`do_algebra`)
GitHub labels: program contract

incomplete

We need to check for possible `ZeroDivisionError`s. Maybe something like the following will be okay.   

```py
...
try:
    return eval(exp)
except ZeroDivisionError:
    assert False, "invalid inputs" # $_CONTRACT_$
```

Alternatively,
```py
assert not any([operand[i-1] == 0 and operator[i] == "//" for i in range(1, len(operand))]), "invalid inputs" # $_CONTRACT_$
```
may also work.
Repository: evalplus/evalplus
Issue: Please evaluate codeBooga
GitHub labels: model eval

CodeBooga-34B-v0.1
https://huggingface.co/oobabooga/CodeBooga-34B-v0.1
Thanks!

@sabagithub Please find results here:

![image](https://github.com/evalplus/evalplus/assets/38074777/60eeced9-70d6-4569-9190-115dd9420dfe)

Overall it seems to stay between the two.

Also cc the original author @oobabooga if the results are informative. Let me know if anyone wants me to put this on the leaderboard as well.

Also FYI, now ppl can request an independent model evaluation through: https://github.com/evalplus/evalplus/issues/new/choose

Thanks for the eval @ganler, that's good to know.

Wow, that was quick @ganler! Thank you for processing it, and for the new link

 It's strange to me because although PhindV2 gives compiling code as often as CodeBooga, CodeBooga seems to follow the specs a little better. Maybe it has better reasoning skills... Anyway, thanks again!

> Wow, that was quick @ganler! Thank you for processing it, and for the new link
> 
> 
> 
>  It's strange to me because although PhindV2 gives compiling code as often as CodeBooga, CodeBooga seems to follow the specs a little better. Maybe it has better reasoning skills... Anyway, thanks again!

I saw a few cases where the merged model did not import required libraries inline but instead did that after the whole function. Anyways HumanEval+ is just one of the many aspects to evaluate LLM coders. The new model may show better results on other benchmarks or human evaluation. 😃
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - zephyr-7b-beta
Root cause: Unimplemented Feature Gap

### Model introduction

https://huggingface.co/HuggingFaceH4/zephyr-7b-beta
### Model URL

https://huggingface.co/HuggingFaceH4/zephyr-7b-beta

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

cc: @ganler 

`(e2) kamil@uCorei7:~/ml/evalplus$ python evalplus/evaluate.py --dataset humaneval --samples ./worksdir2/humaneval/mistral-7b-zephyr-7b-beta_temp_0.0-sanitized/
Load from ground-truth from /home/kamil/.cache/evalplus/2670b22d020cab45ede5e8285730b6ee.pkl
Reading samples...
164it [00:00, 580.47it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 164/164 [00:01<00:00, 131.68it/s]


**Base**
{'pass@1': 0.27439024390243905}


**Base + Extra**
{'pass@1': 0.2073170731707317}`


that's from me 

I'm able to get slightly better results using a prompt (since zephyr is a chat model from my understanding): https://github.com/evalplus/evalplus/commit/d553815ecbdff731a8754cf6e206976ae78293ad

<img width="439" alt="image" src="https://github.com/evalplus/evalplus/assets/38074777/136b6827-e6dc-47a2-8a36-0a6f67d69205">

Updated to the leaderboard. Thanks!

Sorry somehow I messed up the code where the results are actually coming from a non-chat version (https://github.com/evalplus/evalplus/blob/d553815ecbdff731a8754cf6e206976ae78293ad/codegen/model.py#L280).

However, if I use the chatting mode, it seems the performance is not very promising since the generated format seems to be uncontrollable.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - deepseek-coder-6.7b-base
Root cause: Unimplemented Feature Gap

### Model introduction

deepseek-coder-6.7b-base is a 6.7B parameter model with Multi-Head Attention trained on 2 trillion tokens.

### Model URL

https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

@guoday Thanks for the request! Please find our reproduced results below.

![image](https://github.com/evalplus/evalplus/assets/38074777/19936f77-5fbf-4357-908b-1c98969c9757)

Note that we use `bf16` and 512 max new tokens (similar to how we evaluate other models listed in our leaderboard (https://evalplus.github.io/leaderboard.html). Only 3 out of 164 produced programs have unfixable syntactical errors such as infinite enumeration shown blow:

![image](https://github.com/evalplus/evalplus/assets/38074777/ddc9c20e-81fc-49a5-9976-9bb05d1149e3)

Let us know if you have further questions. Thanks!


Thanks @ganler for evaluation. Could update the results in the leaderboard? Thanks.

I have just updated it on the leaderboard. :)
Repository: evalplus/evalplus
Issue: Tests not satisfying conditions in descriptions
Root cause: Validation Gap

Some tests contradict the conditions in the description, e.g., problem 75:
```
   """Write a function that returns true if the given number is the multiplication of 3 prime numbers
    and false otherwise.
    Knowing that (a) is less then 100. 
    Example:
    >>> is_multiply_prime(30)
    True
    30 = 2 * 3 * 5
    """
```
but there are tests with a>100.
For problem 129, there is a condition that `Every integer in the range [1, N * N] inclusive appears exactly once on the cells of the grid.` but some tests do not satisfy that. There are possibly more

Thanks for pointing this out. This comes from incomplete program contract -- in the next `v0.1.2` version will correct these contracts/conditions and filter out invalid test inputs.

Meanwhile I'd like to mention for P75 the base HumanEval inputs also includes `a >= 100`. I guess that's why it leads to write an incomplete contract. But this is not right anyways... So I am also going to remove some base HumanEval inputs.  

better check if a < 100 is really that neccessary of an constraint (we can also remove it from the prompt). Not sure which one is better

Not that really necessary in terms of computational complexity. But I felt we don't want to change the prompt as much as possible for the sake of reusability of LLM outputs.

Thanks for the reports! @Randl Please kindly use `pip install "git+https://github.com/evalplus/evalplus.git" --upgrade` to use the updated HumanEval+ dataset. An official pypi version will come a bit later when we had someone double checked the change. Thanks!
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - New GPT 3.5/4 turbo models
Root cause: Unimplemented Feature Gap

### Model introduction

- GPT-4 Turbo with 128K context and lower prices.
- The latest GPT-3.5 Turbo model with improved instruction following, JSON mode, reproducible outputs, parallel function calling.

### Model URL

https://openai.com/blog/new-models-and-developer-products-announced-at-devday

### Data type

None of above: specify the details in the 'Other context' section

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [ ] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [ ] I confirm that the model comes from unique and original work and does not contain any plagiarism.

GPT-4-Turbo results here :D

![image](https://github.com/evalplus/evalplus/assets/38074777/dd867637-1d79-4b77-abf3-5d3b793a7940)


Also MBPP+ results here:

![image](https://github.com/evalplus/evalplus/assets/38074777/73220def-573d-4512-8e6f-81cc78bbda40)