Repository: evalplus/evalplus
Issue: 🐛 [TestRemoval] - HumanEval/32 find zero incorrect assertion code
Root cause: Validation Gap

### EvalPlus version

v0.2.0

### Output of running `ls ~/.cache/evalplus`

84f4b93a1270b492e4c54d5212da7a5b.pkl  ebe95df0d5220f00cedd6f4bddea35e4.pkl  HumanEvalPlus-v0.1.9.jsonl  MbppPlus-v0.1.0.jsonl  sanitized-mbpp.json

### Task ID of the programming task

HumanEval/32

The outputs of `find_zero` are not an iterable. So when the tests are created for evaluation, defined as:
`assert _poly(*out, inp) <= atol`

You always get this error:
```
    assert _poly(*candidate(*inp), inp) <= 0.0001
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Value after * must be an iterable, not float
```


### Your proposed new contract

I changed the test case to:
```python
assert abs(candidate(*inp) - exp) <= 0.0001
```

And the ground truth solution passed all of the tests (base + plus).
```


### Other context

_No response_

Thanks for pointing this out. I think the fix is to remote the "*"

```
    assert _poly(*candidate(*inp), inp) <= 0.0001
----- to
    assert _poly(*inp, candidate(*inp)) <= 0.0001
```

Let me quickly fix this.

It should have been addressed by 832b83200089213b83f5699e67781a26a53c285a

Can you retry by reinstalling evalplus?

```bash
pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

The issue is still there even after updating.

@gabeorlanski Are you sure? I do see that this seems to work:

![image](https://github.com/evalplus/evalplus/assets/38074777/da0ed0e8-8428-4d26-9d06-df6339593fdf)

Can you verify your versions again? via `pip show evalplus`. Remember you must add ` --upgrade` to really upgrade the package.


By "updating" if you mean you manually applied your patch, the previous has a bit of typo which i fixed as 

```
    assert _poly(*candidate(*inp), inp) <= 0.0001
----- to
    assert _poly(*inp, candidate(*inp)) <= 0.0001
```

There is a "*" to `inp`.

My version is `0.3.0.dev83` and the code in the `unsafe_execute` is:

```python
if dataset == "humaneval":
    if "find_zero" == entry_point:
        assert _poly(*out, inp) <= atol 
```

But if you change it to:
```python
assert abs(out - exp) <= atol
```
It fixes it.

Can you show me the detailed error of why it does not work?

Note that you cannot change that to `assert abs(out - exp) <= atol` because if you look at the function, it allows multiple correct results, overfitting `exp` would lead to false positives (correct classfied as wrong).

> My version is `0.3.0.dev83` and the code in the `unsafe_execute` is:
> 
> ```python
> if dataset == "humaneval":
>     if "find_zero" == entry_point:
>         assert _poly(*out, inp) <= atol 
> ```
> 
> But if you change it to:
> 
> ```python
> assert abs(out - exp) <= atol
> ```
> 
> It fixes it.

Oh I see the problem. The updated version (832b83200089213b83f5699e67781a26a53c285a) is using 

```
assert _poly(*inp, out) <= atol
```

which is not what you are showing here.

You might want to uninstall evalplus and then reinstall it via

```
pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

might also be good to check if your `pip` is aligned with your `python`. a way to do that is to 

```
{PYTHON} -m pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

Yeah, it used a cached version of evalplus. Force removing it fixed it.

Glad it worked!