Repository: evalplus/evalplus
Issue: Problems in the HuggingFace file - discrepancies with the evalplus codebase/package
Root cause: Validation Gap

Hey! While running an evaluation with both the evalplus package and the hugging face file, I noticed some discrepancies between the 2. Listing them here in case they can be fixed :)

- Mbpp/164 and others
the following special cases are correctly handled in the evalplus codebase, but were not handled when creating the HuggingFace file:
[evalplus/evalplus/eval/__init__.py](https://github.com/evalplus/evalplus/blob/c4d5103de7fd5fc608a10a0bfb92e82bb618b17d/evalplus/eval/__init__.py#L142)

For example, the task id Mbpp/164 whose function is "are_equivalent": the hugging face file has the following base test assert `are_equivalent(23, 47) == True` which is incorrect, while the code I linked above correctly fixes this.

- Mbpp/143: 
one of the base tests was not created correctly: in the HF file it is
``` assert find_lists(([9, 8, 7, 6, 5, 4, 3, 2, 1])) == 1"  ```
while it should be
``` assert find_lists(([9, 8, 7, 6, 5, 4, 3, 2, 1], )) == 1"  ``` (note the extra comma)

- Mbpp/229

not sure what happened here, but the first base test is wrong:
```
assert re_arrange_array([-1, 2, -3, 4, 5, 6, -7, 8, 9], 9) == [-1, -3, -7, 4, 5, 6, 2, 8, 9]
```
it should rather be
```
assert re_arrange_array([-1, 2, -3, 4, 5, 6, -7, 8, 9], 9) == [-1, -3, -7, 2, 4, 5, 6, 8, 9]
```
note that when I use the evalplus package this base test passes so the output is correct. This issue is only in the HF file.

- Mbpp/407

same thing where a base test in HF file is wrong:
```
assert rearrange_bigger(10)==False
```
instead of
```
assert rearrange_bigger(10)==None
```

- Mbpp/431
```
assert common_element([1,2,3,4,5], [6,7,8,9])==None
```
instead of
```
assert common_element([1,2,3,4,5], [6,7,8,9])==False
```

- Mbpp/438
```
assert count_bidirectional([(5, 6), (1, 2), (6, 5), (9, 2), (6, 5), (2, 1)] ) == 4
```
instead of
```
assert count_bidirectional([(5, 6), (1, 2), (6, 5), (9, 2), (6, 5), (2, 1)]) == 3.0
```

- Mbpp/442
Here the GT were rounded up at the 2nd decimal, which fails since the tolerance is at e-6:
```
assert positive_count([0, 1, 2, -1, -5, 6, 0, -3, -2, 3, 4, 6, 8])==0.54
assert positive_count([2, 1, 2, -1, -5, 6, 4, -3, -2, 3, 4, 6, 8])==0.69
assert positive_count([2, 4, -6, -9, 11, -12, 14, -5, 17])==0.56
```
instead of
```
assert positive_count([0, 1, 2, -1, -5, 6, 0, -3, -2, 3, 4, 6, 8]) == 0.5384615384615384
assert positive_count([2, 1, 2, -1, -5, 6, 4, -3, -2, 3, 4, 6, 8]) == 0.6923076923076923
assert positive_count([2, 4, -6, -9, 11, -12, 14, -5, 17]) == 0.5555555555555556
```

- Mbpp/461
base tests are wrong again
```
assert upper_ctr('PYthon') == 1
assert upper_ctr('BigData') == 1
assert upper_ctr('program') == 0
```
instead of
```
assert upper_ctr("PYthon") == 2
assert upper_ctr("BigData") == 2
assert upper_ctr("program") == 0
```


Thank you!

(btw to find most of these cases I ran the GT on the base tests and inspected the ones that failed. Using the HF file I have a pass@1_gt of ~95 while I get 100 using the evalplus package).

@Romainsauvestre Thank you for the great notes! Indeed there are some corner cases, esp. those brought by the base tests originally come from the base MBPP. 

The reason we did not change the input/description is that we want to align with the base MBPP as much as possible. So instead of correcting the tests and description, I decided to relax their oracle so that the solutions can be correct in multiple ways. But you are definitely right that we should also bring these oracles to the HF file for sure!

Just passed all MBPP+ tasks using the GT!

![image](https://github.com/evalplus/evalplus/assets/38074777/42f24310-dc0a-4a0f-9788-d3ff97fc24c0)


HF version is updated. lmk if there are still more problems. Thanks for the report!

Hey @ganler  it seems like the PR fixed issue https://github.com/evalplus/evalplus/issues/148 but not this current issue. I check the file on HF and still see the problems I listed here. Am I missing something?

Could you provide a script to reproduce the error? That would be very helpful!

Sure, you can use this code:
```
import fire
import json
from tqdm import tqdm
import traceback


def exec_tests(solution, tests):
    globls = {}
    try:
        compiled = compile(solution, "<source>", "exec")
    except BaseException:
        return traceback.format_exc()
    try:
        exec(compiled, globls)
    except BaseException:
        return traceback.format_exc()
    try:
        test_c = compile(tests, "<test>", "exec")
        exec(test_c, globls)
    except BaseException:
        return traceback.format_exc()
    
    return "passed"


def main(dataset_path):
    with open(dataset_path) as f:
        data = [json.loads(x) for x in f.readlines()]

    failing_task_ids = []

    for sample in tqdm(data):
        test_imports = "\n".join(sample["test_imports"])
        base_tests = "\n".join(sample["test_list"])
        base_tests = test_imports + "\n" + base_tests
        gt_solution = sample["code"]
        out = exec_tests(gt_solution, base_tests)
        if out != "passed":
            failing_task_ids.append(str(sample['task_id']))
    
    print(",".join(failing_task_ids))
            

if __name__ == "__main__":
    fire.Fire(main)
```

For the "dataset_path", I downloaded the latest version of mbpp_plus from the hugging face hub.
When I execute this script I get the following failing task ids: 
```
83,115,117,143,164,229,249,295,396,400,407,431,438,442,461,574,582,595,640,746,783
```


I did not inspect all these task ids in details, but the ones I noted in the issue above are still present here. Quickly eyeballing the file on hugging face for these task ids, the problems I mentioned above are still in the file so it's expected the GT solution fails on these tests.

I see why now.

So in the MBPP-compatible format, we constructed the oracle in the "test" field. That being said, instead of using `"\n".join(sample["test_list"])` you might want to directly use `sample["test"]`.

But indeed I agree this format is not fully aligned with the original MBPP. I need to think about how to make this seamlessly compatible with existing frameworks that use `"test_list"`.

I remember one of the reasons that I did not add plus tests to `test_list` is because many existing frameworks also use that directly in the prompt as well. :/

And you are right that this `test_list` can be very wrong -- but it is actually made by the original MBPP dataset, not by EvalPlus... 
For the example of `Mbpp/461`, the test_list in our transformed dataset directly copies from the original MBPP dataset at: https://huggingface.co/datasets/mbpp/viewer/sanitized/test?p=2&row=241

MBPP is a crowd-sourced dataset from the web so the test list may not be perfect. That's also why when we made a substantial effort in its evalplus version to carefully double-check these tasks and either cleaned or dropped many bad ones (https://github.com/evalplus/evalplus/tree/mbpp/groundtruth/mbpp#check-implementation) -- but clearly there's still broken ones missed here. :)

However, these tests are no longer wrong if you use EvalPlus as the evaluation framework -- primarily because EvalPlus uses differential testing as the oracle and we are pretty sure about the quality of our ground-truth solutions -- therefore the oracle of using EvalPlus evaluator gives you correct evaluation. You can also get an accurate assessment if you run EvalPlus datasets at https://github.com/bigcode-project/bigcode-evaluation-harness where I specifically handled this situation (https://github.com/bigcode-project/bigcode-evaluation-harness/pull/190).

 My next plan to resolve this is to remove these tasks in MBPP+ since they lead to broken prompts in our next release/revision.