Repository: evalplus/evalplus
Issue: HumanEval/115 Prompt Error
Root cause: Documentation Deficiency

Hello, I encountered an issue with the dataset provided. Upon reviewing the data, I found that there is a prompt error in the dataset. Details are as follows:

Task: HumanEval/115 Prompt 
Error Description: Inconsistency with prompts in the original dataset [HumanEval](https://github.com/openai/human-eval/blob/master/data/HumanEval.jsonl.gz), because the imported module is in the wrong place.

EvalPlus's Prompt is:
```python
def max_fill(grid, capacity):
    import math
    """
    You are given a rectangular grid of wells. Each row represents a single well,
    and each 1 in a row represents a single unit of water.
    Each well has a corresponding bucket that can be used to extract water from it, 
    and all buckets have the same capacity.
    Your task is to use the buckets to empty the wells.
    Output the number of times you need to lower the buckets.

    Example 1:
        Input: 
            grid : [[0,0,1,0], [0,1,0,0], [1,1,1,1]]
            bucket_capacity : 1
        Output: 6

    Example 2:
        Input: 
            grid : [[0,0,1,1], [0,0,0,0], [1,1,1,1], [0,1,1,1]]
            bucket_capacity : 2
        Output: 5
    
    Example 3:
        Input: 
            grid : [[0,0,0], [0,0,0]]
            bucket_capacity : 5
        Output: 0

    Constraints:
        * all wells have the same length
        * 1 <= grid.length <= 10^2
        * 1 <= grid[:,1].length <= 10^2
        * grid[i][j] -> 0 | 1
        * 1 <= capacity <= 10
    """
```
and the original prompt is:
```python
import math

def max_fill(grid, capacity):
    """
    You are given a rectangular grid of wells. Each row represents a single well,
    and each 1 in a row represents a single unit of water.
    Each well has a corresponding bucket that can be used to extract water from it, 
    and all buckets have the same capacity.
    Your task is to use the buckets to empty the wells.
    Output the number of times you need to lower the buckets.

    Example 1:
        Input: 
            grid : [[0,0,1,0], [0,1,0,0], [1,1,1,1]]
            bucket_capacity : 1
        Output: 6

    Example 2:
        Input: 
            grid : [[0,0,1,1], [0,0,0,0], [1,1,1,1], [0,1,1,1]]
            bucket_capacity : 2
        Output: 5
    
    Example 3:
        Input: 
            grid : [[0,0,0], [0,0,0]]
            bucket_capacity : 5
        Output: 0

    Constraints:
        * all wells have the same length
        * 1 <= grid.length <= 10^2
        * 1 <= grid[:,1].length <= 10^2
        * grid[i][j] -> 0 | 1
        * 1 <= capacity <= 10
    """

```

Thanks for reporting. Would this actually cause a model to fail if the model works for the original prompt?

Actually, I think you misplaced the EvalPlus prompt and the original prompt.

Original HumanEval/115:

<img width="669" alt="image" src="https://github.com/user-attachments/assets/4877fd26-38fe-4ffb-90ca-a2f850ed673e">

EvalPlus HumanEval/115:

<img width="698" alt="image" src="https://github.com/user-attachments/assets/a2bb2777-bb8e-439b-a105-3b698f6387be">

I did the import transformation to make the parsing of docstring consistent. 

https://github.com/evalplus/evalplus/blob/4df700112e142efdebca6a7e8ce69882951f7532/evalplus/data/humaneval.py#L92-L94

I don't feel this is wrong or a bug as I think the EvalPlus version is even more natural in terms of the position of the import. If the model works with the original version and fails the EvalPlus version, it could be a contamination issue.

Thanks for your explanation!