Repository: evalplus/evalplus
Issue: Bad Results of `meta-llama/Meta-Llama-3.1-8B` Evaluation
Root cause: Configuration Error

Hi, I used vllm as the backend to run codegen and evaluate `meta-llama/Meta-Llama-3.1-8B` base model, and the results are very bad, far away from meta's report.

My commands are like:

```shell
python codegen/generate.py --model "meta-llama/Meta-Llama-3.1-8B" --greedy --root eval_llama31_2 --dataset humaneval --backend vllm --tp 2
python codegen/merge.py eval_llama31_2/humaneval/meta-llama--Meta-Llama-3.1-8B_vllm_temp_0.0 eval_llama31_2/pred_he_vllm.jsonl
python evalplus/sanitize.py --samples eval_llama31_2/pred_he_vllm.jsonl
python evalplus/evaluate.py --dataset humaneval --samples eval_llama31_2/pred_he_vllm-sanitized.jsonl --parallel 64 --i-just-wanna-run
```

where `codegen/merge.py` merges all outputs into one jsonl file, which is appended at last.

And the results are:

```shell
humaneval (base tests)
pass@1: 0.262
humaneval+ (base + extra tests)
pass@1: 0.226
```

If I cancel this [commit](https://github.com/evalplus/evalplus/commit/db7bb427abbfeb80ead79eb190264161fe042d54), the performance could be better, but it's still very bad.

```shell
humaneval (base tests)
pass@1: 0.378
humaneval+ (base + extra tests)
pass@1: 0.317
```

As an example, for `HumanEval_10`, the generated code after `prompt` is stripped is (it gets stuck)

```python
def is_palindrome(string: str) -> bool:
    """ Test if given string is a palindrome """
    return string == string[::-1]


def make_palindrome(string: str) -> str:
    """ Find the shortest palindrome that begins with a supplied string.
    Algorithm idea is simple:
    - Find the longest postfix of supplied string that is a palindrome.
    - Append to the end of the string reverse of a string prefix that comes before the palindromic suffix.
    >>> make_palindrome('')
    ''
    >>> make_palindrome('cat')
    'catac'
    >>> make_palindrome('cata')
    'catac'
    """


    if is_palindrome(string):
        return string

    # Find the longest postfix of supplied string that is a palindrome.
    # We can do this by iterating over the string in reverse order and checking if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration and check the next substring.
    # We can use a while loop to implement this.
    # The while loop will iterate over the string in reverse order and check if the current substring is a palindrome.
    # If it is, we can stop the iteration and return the substring.
    # If it's not, we can continue the iteration
```

while without `prompt.strip()` it's

```python


def is_palindrome(string: str) -> bool:
    """ Test if given string is a palindrome """
    return string == string[::-1]


def make_palindrome(string: str) -> str:
    """ Find the shortest palindrome that begins with a supplied string.
    Algorithm idea is simple:
    - Find the longest postfix of supplied string that is a palindrome.
    - Append to the end of the string reverse of a string prefix that comes before the palindromic suffix.
    >>> make_palindrome('')
    ''
    >>> make_palindrome('cat')
    'catac'
    >>> make_palindrome('cata')
    'catac'
    """
    if not string:
        return string

    # Find the longest postfix of supplied string that is a palindrome.
    # We can do it by iterating over the string from the end and checking if the string is a palindrome.
    # If it is, we can stop the iteration and return the string.
    # If it is not, we can continue the iteration and check the next postfix.
    # The iteration will stop when we reach the beginning of the string.
    # The longest postfix that is a palindrome will be the last one that we checked.
    # We can use the `is_palindrome` function to check if a string is a palindrome.
    for i in range(len(string), 0, -1):
        if is_palindrome(string[i:]):
            return string + string[i:][::-1]

    # If we reach this point, it means that the supplied string is not a palindrome.
    # We can return the original string.
    return string
```

which shows the strip operation can affect the output much.

I also tried the instruct version `meta-llama/Meta-Llama-3.1-8B-Instruct` (using the latest code, i.e. with `prompt.strip()`) and the results on HumanEval (+) and MBPP (+) are 69.5 (63.4), 68.5 (55.6), lower than the [official report](https://ai.meta.com/blog/meta-llama-3-1/) but close to them.

```python
# codegen/merge.py
import sys, json, os
from natsort import natsorted

if __name__ == '__main__':
    
    input_dir = sys.argv[1]
    output_file = sys.argv[2]

    with open(output_file, 'w') as f:
        f.write('')

    for task in natsorted(os.listdir(input_dir)):
        task_dir = os.path.join(input_dir, task)
        if not os.path.isdir(task_dir):
            continue
        input_file = os.path.join(task_dir, '0.py')
        with open(input_file, 'r') as f:
            code = f.read()
        task_id = task.replace('_', '/')
        info = {
            'task_id': task_id,
            'solution': code,
        }
        with open(output_file, 'a') as f:
            f.write(json.dumps(info) + '\n')
```

There could be two orthogonal reasons:
1. `"""\n` is an individual token for Llama-3.1 but now we are giving a separate `"""` which is yet another token and Llama-3.1 seems to be sensitive about this.
2. Previous findings found Llama-3 base models are not very ready for code generation and I am not sure if it is also the case for Llama-3.1.

I have fixed 1 in 4df700112e142efdebca6a7e8ce69882951f7532