Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Nondzu/Mistral-7B-codealpaca-lora
Root cause: Validation Gap

### Model introduction

https://huggingface.co/Nondzu/Mistral-7B-codealpaca-lora

### Model URL

https://huggingface.co/Nondzu/Mistral-7B-codealpaca-lora

### Data type

float16

### Additional instructions (Optional)

require  48GB VRAM to run  in full precision

### Author

- [X] Yes
- [ ] No

### Security

- [X] I confirm

### Integrity

- [X] I confirm

Hi @Nondzu  I am able to get the following results using direct prompting (e.g., without using instructions).

![image](https://github.com/evalplus/evalplus/assets/38074777/251a961c-485d-4d29-b2e0-10344ad51778)

Let me know if you are hoping to get a instruction-prompted version evaluated. If so please indicate the prompt structure and how it is supposed to be parsed. :P

@ganler I am not sure why your results has that many difference. 
Also when I did test for original Mistral model i got higher output

here is link to my code `generate_one_completion` that I use to run evalPlus test https://huggingface.co/Nondzu/Mistral-7B-codealpaca-lora/blob/main/run-humaneval.py


But I have 24Vram GPU so I use 8bit mode. When I try use a codegen/generate.py script I got OOM and had the same result what you got. 
Do you have log from the generation ? 


<img width="1015" alt="image" src="https://github.com/evalplus/evalplus/assets/20000097/71b6e54a-afa8-4621-bf0c-5aad13c6502e">


I can see the following differences from https://huggingface.co/Nondzu/Mistral-7B-codealpaca-lora/blob/main/run-humaneval.py#L23

In EvalPlus' codegen:

1. `do_sample=False` for greedy decoding
2. `top_p=0.95` following the codex paper
3. `top_k` is unset
4. `temperature=0` for greedy decoding
5. We use a larger `max_new_tokens` (i.e., 512)
6. `skip_special_tokens` is unset by default
7. We use `float16` instead of 8-bit on A100@80G

I will try to sync your configs for 6 and do a re-try.


Notably I am also using https://github.com/evalplus/evalplus/blob/master/tools/sanitize.py to fix some trivial syntax errors.

> I can see the following differences from https://huggingface.co/Nondzu/Mistral-7B-codealpaca-lora/blob/main/run-humaneval.py#L23
> 
> In EvalPlus' codegen:
> 
> 1. `do_sample=False` for greedy decoding
> 2. `top_p=0.95` following the codex paper
> 3. `top_k` is unset
> 4. `temperature=0` for greedy decoding
> 5. We use a larger `max_new_tokens` (i.e., 512)
> 6. `skip_special_tokens` is unset by default
> 7. We use `float16` instead of 8-bit on A100@80G
> 
> I will try to sync your configs for 6 and do a re-try.

![image](https://github.com/evalplus/evalplus/assets/38074777/f2912dae-dffb-4af2-a322-18c9d43d9cf0)

This is what I achieved after the revision which is slightly better. Let me know if it looks good to you. :)

Well yes and no ;) 
I want to run other test to make sure what exactly difference is between mine implementation of `generate_one_completion  ` and `generate.py` code.

@ganler Do you use the same HFTorchDecoder like to original Mistral model ? 



@Nondzu You can check it out here: https://github.com/evalplus/evalplus/commit/d66e2f8a8562f309bfcabe3122fc6fe7cced8b6f

```shell
git clone https://github.com/evalplus/evalplus.git
cd evalplus
export PYTHONPATH=$PWD
python codegen/generate.py --model mistral-7b-codealpaca --bs 1 --temperature 0 --n_samples 1 --resume --root /scratch/humanevalp --greedy
```


I think I know why. Some of the output samples produced by your model can include `
`. By removing such now I get:

![image](https://github.com/evalplus/evalplus/assets/38074777/1f7938e8-e0b2-49f0-b41f-ada64da2fa66)

Updated to our leaderboard. Congratulations!

This might also explain why the Mistral scores I achieved is a little bit smaller than what they reported officially. ;P 

![image](https://github.com/evalplus/evalplus/assets/38074777/2d3acf44-6ea2-41e7-a120-1379f828ee21)

I have updated this to our code cleaner!

@ganler Many thanks for your work!