Repository: EleutherAI/lm-evaluation-harness
Issue: humaneval_instruct results in 0 pass@1 with openai-chat-completions and
Root cause: Configuration Error

**Description:**

The `humaneval_instruct` task is consistently reporting a `pass@1` score of 0 when using the `openai-chat-completions` model with the `gpt-4o` model specified. This is unexpected as `gpt-4o` is a highly capable model and should achieve a significantly higher score on this code generation benchmark.

**Steps to Reproduce:**

1. Ensure the `lm-evaluation-harness` is installed and configured correctly.
2. Set up the OpenAI API key as required by the `openai-chat-completions` model.
3. Run the following command:

   ```bash
   lm_eval --model openai-chat-completions \
       --model_args max_retries=5,model=gpt-4o,num_concurrent=1 \
       --tasks humaneval_instruct --apply_chat_template --confirm_run_unsafe_code

4. Observe the output, which shows a pass@1 score of 0 for the humaneval_instruct task.

**Expected Behavior:**

The humaneval_instruct task should produce a non-zero pass@1 score when evaluated with the gpt-4o model, reflecting its code generation capabilities.

**Actual Behavior:**

The evaluation consistently reports a pass@1 score of 0.

2025-05-05:15:22:46 INFO     [loggers.evaluation_tracker:272] Output path not provided, skipping saving results aggregated
openai-chat-completions (max_retries=5,model=gpt-4o,num_concurrent=1), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
|     Tasks      |Version|  Filter   |n-shot|Metric|   |Value|   |Stderr|
|-----------------|------:|-----------|-----:|------|---|----:|---|-----:|
|humaneval_instruct|     2|create_test|     0|pass@1|   |    0|±  |     0|

Would you mind taking a look into this when convenient? Any insights into the possible cause or suggestions for how to address it, would be greatly appreciated.

Thanks in advance.

Hi! The base `humaneval`

https://github.com/EleutherAI/lm-evaluation-harness/blob/fc5019ead53c45119c522c62e8eea2daa837c56e/lm_eval/tasks/humaneval/humaneval.yaml#L6

is structured for base models as instruction-tuned models tend to add unnecessary preliminary text which affects the metric calculation. Using the `humaneval_instruct` variant which adds more instructions and a `gen_prefix`, helps address this issue: https://github.com/EleutherAI/lm-evaluation-harness/blob/fc5019ead53c45119c522c62e8eea2daa837c56e/lm_eval/tasks/humaneval/humaneval_instruct.yaml#L3-L4

here `prompt` is the partial function from the [dataset](https://huggingface.co/datasets/openai/openai_humaneval/viewer/openai_humaneval/test?row=0)

Also see #2764 for additional context.

However, OpenAI does not allow prefilling the assistant context iirc, so you could try removing it (and/or modifying `doc_to_text`). Essentially, humaneval is structured to provide the model with a partial Python function and have it generate the correct executable completion. You could try modifying the `build_predictions` function to extract the correct `resp` which is used to stitch the prompt and completion together (or return the correct function from the model response depending on how you prompted the model to format the answer) before running it with `exec`:
https://github.com/EleutherAI/lm-evaluation-harness/blob/fc5019ead53c45119c522c62e8eea2daa837c56e/lm_eval/tasks/humaneval/utils.py#L26-L27

Hope this helps!

Hey @baberabb,

Appreciate for the update, that clarifies why I saw a 0 on HumanEval.

Just a quick question. Regarding the tricks used to evaluate the results, are we anticipating any significant deviation from the official OpenAI GPT-4o results?

Thanks again!

Generally not as `human_eval` has been completely saturated for a while: claude-sonnet get a perfect score in #2764. OpenAI haven't released any eval details, but it shouldn't it be too hard to modify the task to get the model to output the answer in an extractable format.

Hey @baberabb,

I think you're absolutely right, modifying tasks to get extractable answers is a good idea. When it comes to evaluating the coding ability for LLM within or out **lm_eval_harness**, any other valuable benchmarks and approaches that I can leverage?

Thank you!

We have `mbpp` on the harness. Besides that SWEBench is quite popular.

There's also quite a number of evals in this repo: https://github.com/bigcode-project/bigcode-evaluation-harness

Cool, thanks for heads up! At this point I think my original issue have been resolved, and I know what I need to for next step:)

Feel free to close this issue or let me work on that one.

@johnsonafool did you manage to solve it? If yes, would you mind sharing your solution here?

Thanks in advance.