Repository: evalplus/evalplus
Issue: GPT4 reported HumanEval base significantly higher than OpenAI’s reported results
Root cause: Validation Gap

Hello,

I noticed that GPT4 HumanEval score via the EvalPlus tool reports approx an 88% HumanEval base score. This is substantially higher than OpenAI reports with the official HumanEval test harness - 67%. 

Why the discrepancy?

Thank you! 

Thanks for bring up this question!

Feel free to check the concrete GPT-4 samples at our end and re-evaluate it independently: https://github.com/evalplus/evalplus/releases/download/v0.1.0/gpt-4_temp_0.0.zip

There could be multiple reasons to the "discrepancy". When we are talking about "GPT4" we are not talking about a fixed model (e.g., starcoder-?b, llama-?b) -- essentially "ChatGPT" and "GPT4" is a product whose internal implementation can change over time very often. For example, recently people have been talking about the perf drop of GPT-4:

1. https://news.ycombinator.com/item?id=36134249
2. https://arxiv.org/pdf/2307.09009.pdf

We were using GPT-4 in May, though it is impossible to tell if OpenAI uses A/B testing, etc. which could cause inconsistency even if you run multiple experiments at the same time. To conclude, evaluation of these commercial products are expected to be neither deterministic nor reproducible over time. The best thing we can do is to store the original samples to get some level of determinism and reproducibility.

Nonetheless, potential technical reasons are:
- The way people parse the chatbot outputs -- our implementation is here: https://github.com/evalplus/evalplus/blob/master/codegen/model.py#L279
- The way people post-process the output code samples -- our implementation is here: https://github.com/evalplus/evalplus/blob/master/tools/sanitize.py

However, personally I don't see this would bring a difference of 20%. Thanks!

I encountered this as well independently and then stumbled upon this Github issue. 

I logged my findings [here](https://github.com/emrgnt-cmplxty/automata/blob/main/research/study_agency/study_human_eval/human_eval_results.md).

Strangely, I can also find evidence of a massive decline in performance when evaluating this [sample](https://github.com/GammaTauAI/leetcode-hard-gym). Where, I see 0%/2.5% success rate on the two most recent releases of GPT-4, respectively. I'm trying to make sense of these paradoxical results.

**Could it be something to do with HumanEval leakage in forward fine-tuning/training that has been performed since the initial release?

It would be great to connect and chat about these massive discrepancies.**


PS -

I decided to use Evalplus because HumanEval gave me this error [here](https://github.com/openai/human-eval/issues/27). Ty for maintaining a working alternative. 

Its definitely possible that there are some HumanEval data leakage during training. Given that people have been doing a ton of evaluation on HumanEval and similar datasets, I imagine there are tons of leaked solutions to these datasets on GitHub or other webpages. Additionally, a lot of HumanEval problems are derivatives of common programming problems with solutions available online as well (@ganler had found that geeksforgeeks contains a lot of similar problems to HumanEval). These could contribute to the improved performance on HumanEval recently (not just GPT-4 but some newly released models) 

Also thanks for running HumanEval+ using some more advanced techniques such as tool using agents. Actually in our experiments we found similar results where it seems like the advanced techniques might not neccessarily outperform the simple zero-shot baseline. Our guess is that probably a lot of HumanEval problems are just too simple: they do not require complex technqiues to solve. Think one very neccessary future direction will be to create a harder dataset that is more aligned with what developers would apply these tools on in the real-world (at the same time keeping functional correctness in mind) to better evaluate these techniques. 

Thanks for the comprehensive reply. Your in depth thoughts are very helpful. It is unfortunate if these findings are true, because it makes it difficult to do an apples-to-apples comparison with any of the recent literature results. I'm hoping that someone can point out some kind of obvious mistake. The part I find most confusing is that, to the best of my knowledge, the March snapshot should be the same one that OpenAI used in their paper which quoted ~67%

Also, I appreciate your interest in agent-based approaches. The improvement with agent based approaches does seem to scale with problem complexity. I have been designing a more difficult set of evaluations to understand coding performance in a more expansive environment.

If you are interested in collaborating on the design of benchmarks for this domain I would be pleased to do so. [Here](https://github.com/emrgnt-cmplxty/automata/tree/main/automata/config/eval) you can find some demo payloads I designed for search / code writing inside a scoped codebase which I have been using to gain informative insights. The approach I designed here is to give the agent a natural language query which requires them to retrieve or instantiate a specified object within the scope of the local repository. This forces the agent to use tools such as search / code fetching to assemble the necessary context, which can include importing and instantiating dependencies. It's preliminary, but it was the simplest next step I could think of to begin approximating more dynamic coding environments.

Thanks for the references and initial work! We would definitely love to collaborate and do more along this direction!