Repository: evalplus/evalplus
Issue: [MBPP+] Trim more ill-formed tasks for MbppPlus v0.2.0
Root cause: Validation Gap

https://github.com/evalplus/evalplus/issues/147#issuecomment-2057740796

Thanks @Romainsauvestre for raising this issue from the original MBPP dataset!

Another thing we can also do in MbppPlus v0.2.0 is to remove the URLs of prompts from specific tasks mentioned in #156.

> [#147 (comment)](https://github.com/evalplus/evalplus/issues/147#issuecomment-2057740796)
> 
> Thanks @Romainsauvestre for raising this issue from the original MBPP dataset!

I found the errorness mentioned in #147 are in `test_list` field, are we going to trim these tasks? 
I'm thinking we can simply fix the `test_list`. So I proposed the #149 
I want to know how you approached this? 

One of the rules of thumb for EvalPlus is that we don't want to modify the prompt of an original task as much as possible. So if there is a logic error:

1. in the solution -- no problem we fix it
2. in the Oracle -- no problem we fix it
3. in the prompt? ... we cannot change it so we'd rather remove it

If you change the prompt in a new task, the plus version won't have a simple relation (e.g., subset or equivalent) with the original dataset. Hope this makes sense. 

Done by 3de226896daa67b54e2683eb6e5f6840ce66018a.

Now it has fewer MBPP tasks in MBPP+ (399 -> 378).

Also because we had a bunch of fixes thanks to @Romainsauvestre, I see an improvement of ~4pp after all these fixes for both base and plus. 

Am also working on updating the leaderboard soon and uploading an HF version.

Updated HF. But I also tagged different versions that one may use the old tag to align with prior results.

![image](https://github.com/evalplus/evalplus/assets/38074777/2315bf6f-531f-40fd-b631-7aecb617e312)

cc: @Romainsauvestre

this is awesome thank you!
Just to be sure: the leaderboard will be using v0.2.0 moving forward, so if I want to compare my models with the leaderboard I should also use the v.0.2.0 version of the HF file, right?

It looks like the "test.jsonl" on hugging face on main is the "v0.1.0" version since it still has 399 rows btw.

We will make the leaderboard up to date in terms of the dataset versions (not done yet and it can take a few days).
But no worries, we have a transparent commit history (https://github.com/evalplus/evalplus.github.io/commits/main/) that you can always retrieve old results for old dataset versions. 

That being said, before we update the results towards v0.2.0 at https://github.com/evalplus/evalplus.github.io/blob/main/results.json, you should compare yours with other models on v0.1.0, and after our update ofc you should check the v0.2.0 scores as well. 

> It looks like the "test.jsonl" on hugging face on main is the "v0.1.0" version since it still has 399 rows btw.

I am going to remove that actually as I just figured out how to upload the dataset using huggingface API (https://github.com/evalplus/mbppplus_release/blob/main/hf_upload.py) haha

And yeah you can find all the source datasets at https://github.com/evalplus/mbppplus_release

My pipeline of dataset release/update:
1. make the a dataset and test it
2. ship it to a new GitHub release at https://github.com/evalplus/mbppplus_release
3. run the script to upload a copy to HF


sounds great!

Btw random question: can any of these issues, that were fixed for MBPP plus, also affect Human Eval plus?

The Popen thing in theory may affect. But I tried a bunch and did not see any change. 

The leaderboard for v0.2.0 is on.

> The leaderboard for v0.2.0 is on.

@ganler is it possible to find the previous scores on v0.1.0?