Repository: evalplus/evalplus
Issue: Evalperf giving issue for generated code
Root cause: Validation Gap

Hi Jiawei, I have a file, `sample.jsonl` containing the generated code. I want to evaluate the **effectiveness** of the code generated using evalpref.  
`sample.jsonl` is of **HumanEval** not the Mbpp dataset.

I am running into an error `KeyError: 'Mbpp/3'`. 
```
.file "home/..../evalplus/evalperf.py", line 397, in script
    samples[task_id],
    ~~~~~~~^^^^^^^^^
KeyError: 'Mbpp/3'
```
This is the command I am using to evaluate effectiveness `!evalplus.evalperf --samples sample.jsonl --n_samples 3 --min_correct 2`

![image](https://github.com/user-attachments/assets/bf1b1d31-6130-4913-bda2-d3413b0892ee)


I know the `sample.jsonl` file is correct as It can be successfully evaluated with
` !evalplus.evaluate --dataset humaneval --samples sample.jsonl --test-details`
![image](https://github.com/user-attachments/assets/8dccb597-5f21-40a8-8315-f0d43043601d)


Unfortunately, our code efficiency evaluation only supports a selected set of performance-exercising tasks (we call these tasks "EvalPerf").

In other words, you cannot evaluate the efficiency of a customized set of samples. -- This is because not all coding tasks are meaningful for code efficiency evaluation -- including them can be a waste of time and can introduce eval noises.

You can run something like

```
evalplus.evalperf --model {model_name} --backend [vllm|hf|openai|google|anthropic]
```

It will run code gen for 100 samples per EvalPerf task and give you the profiling results. Hope this is helpful!


Thank you for your quick response.

If I understand correctly, out of the 164 prompts in HumanEval+, you are only testing a subset of 118 performance-exercising prompts. 

Is there a list of the all 118 prompts (HumanEval ID) that are being tested? I would like to update my sample.json file to include only those specific prompts.

Actually, I am able to extract the 118 prompts. 
I was wondering if there is any way to implement it so it tests on all 164 prompts of humanEval+? or test on a **subset** of prompts from the 118 prompts. i.e 118 contains Mbpp prompts also, I only want to test on HumanEval prompts
![image](https://github.com/user-attachments/assets/65d308cb-4c1f-4870-b68c-26dacdcd9822)


> I was wondering if there is any way to implement it so it tests on all 164 prompts of humanEval+? 

We did not design it in this way as our call is "some tasks are not performance-exercising so let's ignore them."

> or test on a subset of prompts from the 118 prompts. i.e 118 contains Mbpp prompts also, I only want to test on HumanEval prompts

I think I can make this happen -- but if you want to use some other metrics such as win rates (against other models), you probably need to adjust the metric computing script at https://github.com/evalplus/evalplus.github.io/blob/main/results/evalperf/stats.py



But anyways, that would still require you to generate 100 samples per task at a temperature of 1.0 though.

To do that, you can make some local edits:

Just filter out MBPP tasks by adding `ptasks  = {k: v for k, v in ptasks.items() if k.startswith("HumanEval")}` after this line:

https://github.com/evalplus/evalplus/blob/3214bbfaec9876305fa96cfe322eca7bb8af680f/evalplus/evalperf.py#L333