Repository: evalplus/evalplus
Issue: HumanEval/75 & HumanEval/116 Prompt-Solution-Test Alignment
Root cause: Documentation Deficiency

Dear EvalPlus team,

Thanks for fixing the bug in this task! I have noticed this issue (https://huggingface.co/datasets/openai_humaneval/discussions/2) for a while and it seems to me the actual problem is that the **Prompt** contradicts the **Canonical Solution** and **Test Cases**.

The **Prompt**, says **"(a) is less then 100"** (yes, there's a typo). However, the **Canonical Solution** and **Test Cases** clearly show that it should be the **"3 prime numbers"** which are less than 100. Therefore, I think it makes more sense to modify the **Prompt** instead of changing the **Canonical Solution** and **Test Cases**, since in this way EvalPlus can remain aligned with the Original HumanEval on testing results of this task. Moreover, intuitively it makes more sense for **"3 prime numbers"** to be less than 100, as there're only 25 prime numbers under 100, and **"(a) less than 100"** makes the problem way too easy.

Here is a fix I proposed (https://github.com/openai/human-eval/pull/23/files#diff-63aa2f5a019591563ff023fce184aee3a9a0b404aedea28e3387193558d8370cR179) for your reference.

In addition, **the Original HumanEval has a few more typos or wrong prompts** (like the **"less then"** case in this task), but they seem still unsolved in EvalPlus. I have made a PR earlier (https://github.com/openai/human-eval/pull/23) regarding these issues. I would be happy to help solve them in EvalPlus if you think it makes sense : )

Thanks again for your time reviewing this issue. Any feedback would be much appreciated!

Sincerely,

marcusm117

**Prompt**
```
def is_multiply_prime(a):
    """Write a function that returns true if the given number is the multiplication of 3 prime numbers
    and false otherwise.
    Knowing that (a) is less then 100.
    Example:
    is_multiply_prime(30) == True
    30 = 2 * 3 * 5
    """
```

**Original Canonical Solution**
```
    def is_prime(n):
        for j in range(2,n):
            if n%j == 0:
                return False
        return True

    for i in range(2,101):
        if not is_prime(i): continue
        for j in range(2,101):
            if not is_prime(j): continue
            for k in range(2,101):
                if not is_prime(k): continue
                if i*j*k == a: return True
    return False
```

**Original Test Cases**
```
def check(candidate):
    assert candidate(5) == False
    assert candidate(30) == True
    assert candidate(8) == True
    assert candidate(10) == False
    assert candidate(125) == True
    assert candidate(3 * 5 * 7) == True
    assert candidate(3 * 6 * 7) == False
    assert candidate(9 * 9 * 9) == False
    assert candidate(11 * 9 * 9) == False
    assert candidate(11 * 13 * 7) == True
```

**EvalPlus Contract v0.1.3**
```
    assert type(a) == int, "invalid inputs" # $_CONTRACT_$
    assert a < 100, "invalid inputs" # $_CONTRACT_$
```

**EvalPlus Canonical Solution v0.1.3**
```
    if a <= 1: return False
    isprime = [True] * (a + 1)
    for i in range(2, a + 1):
        if isprime[i]:
            for j in range(i + i, a + 1, i):
                isprime[j] = False
    cnt, tmp = 0, a
    for i in range(2, a + 1):
        while isprime[i] and tmp % i == 0:
            tmp //= i
            cnt += 1
    return cnt == 3
```

**EvalPlusMini Test Cases v0.1.3**
`[[5], [30], [8], [10], [33], [-98], [2], [98], [99], [-56]]`

Thanks for bringing this up. Overall I don't have a strong preference between the existing workaround (removing invalid tests) and your proposal (tweaking the prompt better). The main reason I was not hoping to change the prompt is to respect the original HumanEval problem description as much as possible. Because the main difference between the original and plus'ed versions should be put on the additional test-cases. 

That being said, I think we can apply your proposed change for HumanEval/75 once your PR to the original HumanEval dataset is merged. If original HumanEval authors updates their dataset there is no reason that HumanEval+ won't follow. 

Meanwhile I am curious if such typos could really impact the result analysis of evaluation since I thought that most capable LLMs can tolerant such small mistakes in the prompt. (But ambiguity is another concern). 

Thanks for the lightspeed response! Yes, I fully agree the goal of EvalPlus is to enhance the test coverage.

Actually, other than HumanEval/75, there's a more urgent issue I just noticed: in **HumanEval/116**, the **Prompt** says **"sort an array of non-negative integers"**, but the **Canonical Solution** and **Test Cases** still considers negative numbers. I think this is still unsolved in EvalPlus.

I can open a new issue regarding this if it's more convenient for you : )

Reference: https://huggingface.co/datasets/openai_humaneval/discussions/1


Thanks for the report! Yeah I agree that the tests should not include negative numbers. We will fix it in our next dataset release soon. 

Just like a few wrong tests in some prior issues, our approach was misled by the base tests since we perform generation/mutation partially from the original HumanEval tests. Of course this also comes from an incomplete contract. 

Thanks a lot, Jiawei! Looking forward to the new release : )

btw, regarding the typos (and also wrong/no examples in prompts), according to my experiments with `gpt-3.5-turbo`, it barely affects the performance. However, personally, I think solving typos (and wrong/no examples) is for consistency within the dataset itself. After all, the original design objective of HumanEval is not to purposefully introduce noise. If we want to test an LLM's robustness against noise, a separate **Noisy HumanEval** would be a better idea.

Reopened for enhancement tag

@marcusm117 Thanks to @Kristoff-starling I think it is fixed in the HEAD commit. You can try to use the latest version via:

```shell
pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

Regarding the PyPI version, we usually accumulate these small changes and release a newer version every month. That being said, we will probably update the PyPI package by the end of this month. 

Meanwhile, let's keep this issue open until we hear something from https://github.com/openai/human-eval/pull/23