Repository: evalplus/evalplus
Issue: Reproduction of Meta-Llama-3-8B-Instruct results on MBPP
Root cause: Validation Gap

The test results of Llama-3-8B-Instruct on MBPP differ greatly from scores on the [official reports](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md#instruction-tuned-models) and leaderboard. I got the same result both using vllm and hf as the backend.

My results:

![image](https://github.com/user-attachments/assets/be737ff2-a2cf-470a-9eb9-4006664e216b)

The official result and leaderboard's results:

|             | MBPP+ | MBPP |
|-------------|-------|------|
| leaderboard | 59.3  | 70.1 |
| official | - | 70.6 |

The command I ran was: 

```shell
python evalplus/evaluate.py --model "meta-llama/Meta-Llama-3-8B-Instruct" --dataset mbpp --backend hf --greedy
```

The environment:
```
vllm                                     0.5.1
torch                                    2.3.0
transformers                             4.45.2
```

I just ran a repro via vllm 0.6.3.post1 but got an even worse result:

![image](https://github.com/user-attachments/assets/8fa4011f-cf56-450e-be6a-01d938657ea0)

Saw a post-processing inconsistency issue and am looking into it:

![image](https://github.com/user-attachments/assets/fce57722-0ffa-4b61-a42e-2a8c6fd7e406)


I also just ran Llama3.1-8B and it does not seem to be as good as the originally reported Llama3-8B somehow:

![image](https://github.com/user-attachments/assets/ecbc5eef-08bb-407c-a563-932d0bb059eb)


> I just ran a repro via vllm 0.6.3.post1 but got an even worse result:
> 
> ![image](https://private-user-images.githubusercontent.com/38074777/384699117-8fa4011f-cf56-450e-be6a-01d938657ea0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzEyNTE5MjMsIm5iZiI6MTczMTI1MTYyMywicGF0aCI6Ii8zODA3NDc3Ny8zODQ2OTkxMTctOGZhNDAxMWYtY2Y1Ni00NTBlLWJlNmEtMDFkOTM4NjU3ZWEwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTEwVDE1MTM0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRkNmFhZTdkMTJlZWM0Yzk1MGI3MmUxMTNkMTFjODIxZmJhYzkyZTIwYzc2NTFlYTRjOTlkMWUxZWIyYTIyZmUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.TwubaL24CX2cs9FYtO_PRX59HvYodMKDHbP5qmp9LcU)
> 
> Saw a post-processing inconsistency issue and am looking into it:
> 
> ![image](https://private-user-images.githubusercontent.com/38074777/384700361-fce57722-0ffa-4b61-a42e-2a8c6fd7e406.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzEyNTE5MjMsIm5iZiI6MTczMTI1MTYyMywicGF0aCI6Ii8zODA3NDc3Ny8zODQ3MDAzNjEtZmNlNTc3MjItMGZmYS00YjYxLWE0MmUtMmE4YzZmZDdlNDA2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDExMTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMTEwVDE1MTM0M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQzY2NkMjJmNjE5YzBmN2IzOTU2YzBkODkwNGNhYmRhZTIzYzFlNTlhNDZiZDQzZWNiYWIxZTA3M2Q5MjJjYWUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.T-aPbxJc-yHhFeqfDi0TLm3rgOQpJO8FZ_ZUw8YDxmk)

Actually it's not a parsing error but a generation issue -- the truncation is right since the bracket is not closed.

Got a higher Llama-3-8B-Instruct score using HuggingFace.

![image](https://github.com/user-attachments/assets/1deb5211-47ae-44a1-a929-edd031b22751)


I'm updating the leaderboard to reflect the latest results.

Earlier results could be higher for various reasons as I shared some other issues (https://github.com/evalplus/evalplus/issues/236#issuecomment-2430997706). 5pp difference is still strange though (may be achieved by using a stronger prompt).