Repository: evalplus/evalplus
Issue: [Query] HumanEvalPlus score not matching for magicoder-s-ds-6.7b
Root cause: Validation Gap

I generated codes for **magicoder-s-ds-6.7b** using the code generation script using this command
```
python codegen/generate.py --model magicoder-s-ds-6.7b --bs 1 --temperature 0 --n_samples 1 --resume --greedy --root result2 --dataset humaneval
```

Running humaneval on the results produced for this gives me a score of **0.585 on humaneval+**, here's the command and the result
```
evalplus.evaluate --dataset humaneval --samples result2/humaneval/magicoder-s-ds-6.7b_temp_0.0/ --i-just-wanna-run
```

> Load from ground-truth from /home/ec2-user/.cache/evalplus/84f4b93a1270b492e4c54d5212da7a5b.pkl
Reading samples...
164it [00:00, 746.90it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 164/164 [00:07<00:00, 23.29it/s]
result2/humaneval/magicoder-s-ds-6.7b_temp_0.0/eval_results.json already exists. Press [Y/N] to overwrite or exit...
Y
Backup result2/humaneval/magicoder-s-ds-6.7b_temp_0.0/eval_results.json to result2/humaneval/magicoder-s-ds-6.7b_temp_0.0/eval_results.json.bak
humaneval (base tests)
pass@1: 0.634
humaneval+ (base + extra tests)
pass@1: 0.585

Can someone help me reproduce the actual results?

Can you check this issue? @UniverseFly 

Hi, please refer to https://github.com/ise-uiuc/magicoder/tree/c900e74a577ea06d3a397b34745d0c920a91f1f7/experiments for reproducing all magicoder experiments. Thanks

Saw a clear bug made by @FatPigeorz 

https://github.com/evalplus/evalplus/blob/5c80024c69c6d48ea91d027f289c4966e59532c3/codegen/model.py#L444-L450

There are lots of unnecessary spaces after ```python


Thanks for such a quick fix

I had a small observation, variable **prompt** is being passed twice in the string now after the fix.
https://github.com/evalplus/evalplus/blob/e2d7bc691db6d5e3fe530b7bd1e59cc3d658eb78/codegen/model.py#L440-L447

Also, should I run `sanitize` and `syncheck` before running the `evaluate` command?

> Also, should I run sanitize and syncheck before running the evaluate command?

Often times results get no worse if you use `sanitize`. :)

> variable prompt is being passed twice in the string now after the fix.

Yep, this is to align with Magicoder evaluation (https://github.com/ise-uiuc/magicoder/blob/c900e74a577ea06d3a397b34745d0c920a91f1f7/experiments/text2code.py#L60-L78). You could observe slightly different numbers due to version & batch size differences. If you follow the guide here (https://github.com/ise-uiuc/magicoder/tree/c900e74a577ea06d3a397b34745d0c920a91f1f7/experiments), you should get identical numbers.

There seems to be another small bug in the input formatting, which I am fixing in https://github.com/evalplus/evalplus/pull/143. Sry for the inconvenience:)

I was able to reproduce the score with Magicoder repo, thanks @UniverseFly

@UniverseFly Btw I also checked it with the latest code in this repo after your fixes

Currently,  **Evalplus** on for code generations for **magicoder-s-ds-6.7b**(even after running `sanitize`) is **0**
<img width="1040" alt="image" src="https://github.com/evalplus/evalplus/assets/152659611/31b654ea-5280-4b86-ade1-09b7363d6328">


Thanks for the notice! There must still be some bugs.. Will check that later 

Magicoder would be a "direct completion" model now after the prompt changes by @UniverseFly. 

I was able to reproduce the score for magicoder after some quick fixes. You can check the changes here: #146