Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - deepseek-coder-6.7b-instruct
Root cause: Unimplemented Feature Gap

### Model introduction

deepseek-coder-6.7b-instruct is a 6.7B parameter model initialized from deepseek-coder-6.7b-base and fine-tuned on 2B tokens of instruction data.

### Model URL

https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

@guoday 6.7b instruct results are here!

![image](https://github.com/evalplus/evalplus/assets/38074777/61e6fbf0-cc19-46d9-9ad6-bd1fc8f60d46)

The results are very good! I used 512 new token size for the conversational generation and one problem (HE/68) got incomplete due to the limited context size. Consequently I manaully set the ctx size for that specific problem to 1024 and can get all produced code compilable.

![image](https://github.com/evalplus/evalplus/assets/38074777/09edec0c-0b2d-4859-8b97-8660ffcd1325)

Good job! The results are updated on the leaderboard. :)

BTW, note that i am using `bf16` instead of `fp16` as is requested here since I see in the training config the model is trained using `bf16` (https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct/blob/main/config.json#L26). Same thing to #39 



awesome results !

> @guoday 6.7b instruct results are here!
> 
> ![image](https://private-user-images.githubusercontent.com/38074777/280583411-61e6fbf0-cc19-46d9-9ad6-bd1fc8f60d46.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDI1Mjg2NTMsIm5iZiI6MTcwMjUyODM1MywicGF0aCI6Ii8zODA3NDc3Ny8yODA1ODM0MTEtNjFlNmZiZjAtY2MxOS00NmQ5LTlhZDYtYmQxZmM4ZjYwZDQ2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzEyMTQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMjE0VDA0MzIzM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM4MmQyYTNmMGEwOWUwM2I4YTAwNTk4ZDc5ZjBmYjkxNWNkNGYwODdhMDlkNGE5YWY5ZDlhMTcyYzhkNWY1N2YmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.z6L9ujjIGXNIJs-0kINpiQCVMDmsK4entYf0PeMB8yY)
> 
> The results are very good! I used 512 new token size for the conversational generation and one problem (HE/68) got incomplete due to the limited context size. Consequently I manaully set the ctx size for that specific problem to 1024 and can get all produced code compilable.
> 
> ![image](https://private-user-images.githubusercontent.com/38074777/280586182-09edec0c-0b2d-4859-8b97-8660ffcd1325.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDI1Mjg2NTMsIm5iZiI6MTcwMjUyODM1MywicGF0aCI6Ii8zODA3NDc3Ny8yODA1ODYxODItMDllZGVjMGMtMGIyZC00ODU5LThiOTctODY2MGZmY2QxMzI1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzEyMTQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMjE0VDA0MzIzM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWZiNGUwMDNiZDE3NzIxMmMwODNkYmZmNzMwYTY5Yzk3MDU1MmIyNmFmMTRiMDM1ZmY1NWU3M2Y0ZWI4MjgzMTEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.bPpBzG-W0DZj7vkpKjcJkhnxr5pOV1Of65LhLxzhwaE)
> 
> Good job! The results are updated on the leaderboard. :)

which prompt did you use for this evaluation?

@bino282 

https://github.com/evalplus/evalplus/blob/1895d2f6aa8895044a7cf69defc24bd57695e885/codegen/model.py#L290-L298

thanks, by the way, you let me know the prompt of the Magicoder-S-DS-6.7B. I don't find in the model.py. I tried the prompt above but I can't repoduce the result on leaderboard

@bino282 Hi the magicoder results are computed using code in the magicoder source code base (which uses evalplus). 

https://github.com/ise-uiuc/magicoder/tree/main/experiments

cc: @UniverseFly for details.