Repository: evalplus/evalplus
Issue: [Proposal] Using larger time-limit for slow execution environment
Root cause: Configuration Error

I generated [model outputs](https://github.com/evalplus/evalplus/files/11841987/samples.jsonl.txt) using [WizardLM/WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0) and evaluated them using HumanEval+. When running on my laptop, I get a `pass@1` of ~0.4 on HumanEval+. If I run the exact same thing on the same `samples.jsonl` file, but on [runpod](https://www.runpod.io/), I get a score of ~0.23 with very high variance, i.e. the score changes every time I run `evalplus.evaluate`.

I believe that this is due to runpod using a slower CPU which triggers the time limits in HumanEval+ that were also already in HumanEval. In [L#173 of `evalplus/eval/__init__.py`](https://github.com/evalplus/evalplus/blob/829272070656fc9b8eb353e858f8b918d534d66c/evalplus/eval/__init__.py#L173), a time limit for evaluating some code is defined. This minimum time limit is 50ms but may be higher depending on the problem. This time limit seems to be sufficient for HumanEval, but it is not for HumanEval+. The HumanEval score isn't impacted by this. It's really just the HumanEval+ score, probably because of the increased number of tests.

When I modify this time limit from 50ms to 5 seconds and don't change anything else, I also get a `pass@1` of ~0.4 on runpod. I also confirmed in other ways that the problem is really due to the the limit, e.g. on my laptop I also get slightly lower `pass@1` if I create a high CPU load while running `evalplus.evaluate`. Not as much as when running it on runpod, but still noticable.

I haven't tried out other time limits. I just increased the 0.05 in [L#173 of `evalplus/eval/__init__.py`](https://github.com/evalplus/evalplus/blob/829272070656fc9b8eb353e858f8b918d534d66c/evalplus/eval/__init__.py#L173) to 5 seconds to confirm my hypothesis. It haven't really looked at whether even more time might be required for some problems and there is also a bunch of other logic in the code that increases this minimum time, but it wasn't enough for HumanEval+ and I'm not sure whether it's worth to modify that instead of simply increasing the minimum time limit.


Hi @tju01 thanks for the issue. The timeout policy of evalplus is set here: 

https://github.com/evalplus/evalplus/blob/4a46eb1e10934f1d164f622bfe26b8956f144567/evalplus/eval/__init__.py#L174

That is, $\max(50ms, 2\times T_{gt})$ where $T_{gt}$ is the testing time used by the reference/ground-truth solution (consistent with the Appendix A of our [arXiv paper](https://arxiv.org/pdf/2305.01210.pdf)).  In other words, I doubt if it is purely due to how small the lower bound is -- if "runpod" is slow then $T_{gt}$ could be much larger than 50ms too.  And if the LLM solution is of a 2x slower than the standard one -- it is judged as wrong.

That being said, I agree that a high variance is not acceptable and we will definitely improve our implementation if it helps. Nonetheless, I hope to more precisely understand the actual reason. One way you can help me is to log the 

1. Actual test runtime
2. Reference time (or ground-truth testing time)
3. Actual time-limit

 around here:

https://github.com/evalplus/evalplus/blob/4a46eb1e10934f1d164f622bfe26b8956f144567/evalplus/eval/__init__.py#L129

And try to get a minimal log by only debugging one specific LLM solution on one programming task that you found interesting.

@tju01 Would like to kindly try out this quick & temporary solution to see if it works for you? https://github.com/evalplus/evalplus/pull/6

Or I can try that later when I can touch my test-bed.


@ganler Yes, that works, even without the flag. Just increasing it to 100ms already gives me a `pass@1` of ~40 again.

As for your other comment above, I agree that if the CPU would be (consistently) slower then it would measure that in the beginning and increase the timeout to twice that amount. That being said, the minimum of 50ms means that even if a solution is more than 2 times as slow as the reference solution, it can still pass if the time is less than 50ms. So on a faster CPU, more solutions will pass because they will be below the 50ms.

At least I think that's basically the reason. I'm not completely sure. But I'm quite sure that the problem is the time limit and I don't just get increased variance (i.e. the mean would still be ~40), but I get consistently lower `pass@1` scores (still with high variance, but also way lower mean than 40).

I can do the tests that you wrote about above tomorrow.

I see. I think you brought a fairly good point anyways -- we should be more conservative regarding the timeout -- it does no harm except the evaluation speed (but we shall just allow users to tweak it more). That being said, I have shipped #6 and hope it can at least work for your pipeline for now -- I think it is also unnecessary for you to double-check the timing details since either way using a more conservative timeout can alleviate that (but it is appreciated if you can!). 

In the immediate next step (before I bump things into `v0.1.6`), I am thinking about following steps to improve EvalPlus:

- A larger $T_\min$ to 200 milliseconds.
- Allow LLM samples to be at most K-times slower than the GT where K can be set to "4" wrt the previous "2". -- we make this configurable anyways so users could tune that according to their PC setup
- Document this in our README as "tips" to use EvalPlus.

cc: @brutalsavage @Kristoff-starling for alignments.

looks good to me