zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: Model  addition
Root cause: Unimplemented Feature Gap

### Model introduction

This model is trained on around 23000+ set of codes. Each set having 2 conversations. This conversation is in Vicuna/ShareGPT format. Each set, along with code, has detailed explanation. I have released the data.

Training: Entire dataset was trained on Azure 4 x A100 80GB. For 3 epoch, training took 42 hours. DeepSpeed codebase was used for training purpose. This was trained on Llama-1 by Meta.

### Model URL

ajibawa-2023/Python-Code-33B

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Thanks! Close as dup of #67

Sorry, I was supposed to add other model. I have opened other Request under: https://github.com/evalplus/evalplus/issues/73
Thank you very much!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - upstage/SOLAR-10.7B-Instruct-v1.0
Root cause: Unimplemented Feature Gap

### Model introduction

We introduce SOLAR-10.7B, an advanced large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. It's compact, yet remarkably powerful, and demonstrates unparalleled state-of-the-art performance in models with parameters under 30B.

We present a methodology for scaling LLMs called depth up-scaling (DUS) , which encompasses architectural modifications and continued pretraining. In other words, we integrated Mistral 7B weights into the upscaled layers, and finally, continued pre-training for the entire model.

SOLAR-10.7B has remarkable performance. It outperforms models with up to 30B parameters, even surpassing the recent Mixtral 8X7B model. For detailed information, please refer to the experimental table. Solar 10.7B is an ideal choice for fine-tuning. SOLAR-10.7B offers robustness and adaptability for your fine-tuning needs. Our simple instruction fine-tuning using the SOLAR-10.7B pre-trained model yields significant performance improvements.



### Model URL

https://huggingface.co/upstage/SOLAR-10.7B-Instruct-v1.0

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

These are the results I got. 

![image](https://github.com/evalplus/evalplus/assets/38074777/18b803a5-413e-400b-8f67-2aead26262f3)

Before publishing the scores, can you quickly check if the scores are consistent to your repros and check my usage scripts here: https://github.com/evalplus/evalplus/blob/7b8371379defb135e6c4cffcd9736fa4e13291ef/codegen/model.py#L178

Update: improved the parsing to get better MBPP scores:

<img width="829" alt="image" src="https://github.com/evalplus/evalplus/assets/38074777/7bacd651-ee20-46ca-8f6b-c75c0ee7f9b8">


I have updated the scores on the leaderboard but feel free to correct me if my usage is wrong. :)
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Python-Code-13B
Root cause: Unimplemented Feature Gap

### Model introduction

This model is trained on around 23000+ set of codes. Each set having 2 conversations. This conversation is in Vicuna/ShareGPT format. Each set, along with code, has detailed explanation. I have released the data.

Training: Entire dataset was trained on 4 x A100 80GB. For 3 epoch, training took 13 hours. DeepSpeed codebase was used for training purpose. This was trained on Llama-2 by Meta.

### Model URL

https://huggingface.co/ajibawa-2023/Python-Code-13B

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Check https://evalplus.github.io/leaderboard.html

Thank you!!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Code-33B
GitHub labels: model eval

### Model introduction

This model is trained on around 74000 set of codes. Each set having 2 conversations. Along with Python, Java, JavaScript, GO, C++, Rust etc. code with detailed explanation is used for training purpose. It is built upon using my existing Dataset Python-Code-23k-ShareGPT. This conversation is in Vicuna/ShareGPT format. Each set, along with code, has detailed explanation.

### Model URL

https://huggingface.co/ajibawa-2023/Code-33B

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Check https://evalplus.github.io/leaderboard.html

Thank you very much!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Code-13B
Root cause: Unimplemented Feature Gap

### Model introduction

This model is trained on around 74000 set of codes. Each set having 2 conversations. Along with Python, Java, JavaScript, GO, C++, Rust etc. code with detailed explanation is used for training purpose. It is built upon using my existing Dataset Python-Code-23k-ShareGPT. This conversation is in Vicuna/ShareGPT format. Each set, along with code, has detailed explanation.

### Model URL

https://huggingface.co/ajibawa-2023/Code-13B

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Check https://evalplus.github.io/leaderboard.html

Thank you very much!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Dolphin-2.6-mixtral-8x7b
GitHub labels: model eval

### Model introduction

The model was finetuned by Eric Hartford
This model is based on Mixtral-8x7b
The base model has 32k context,  fine-tuned with 16k.

This Dolphin is really good at coding,  trained with a lot of coding data. 

### Model URL

https://huggingface.co/cognitivecomputations/dolphin-2.6-mixtral-8x7b

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Updated at https://evalplus.github.io/leaderboard.html
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - speechless-codellama-34b-v2.0
GitHub labels: model eval

### Model introduction

The model is fine-tune codellama/CodeLlama-34B in order to improve the model's inference and planning capabilities.

### Model URL

https://huggingface.co/uukuguy/speechless-codellama-34b-v2.0

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

@uukuguy Could you kindly show me a short and self-contained example to run HE/MBPP codegen (instruction-based or direct code completion?). Thanks!

The model is fine-tuned using Alpaca prompt templates. HumanEval/MBPP code completion can be run directly and has been tested locally to successfully complete the evaluation.

Evaluated and updated on https://evalplus.github.io/leaderboard.html!

Just want to let you know that we fixed a recent post-processing bug where the score of your models has been increased. :D
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - speechless-coding-7b-16k-tora
Root cause: Unimplemented Feature Gap

### Model introduction

This model is fine-tune llm_agents/tora-code-7b-v1.0 in order to improve the model's reasoning and planning abilities.

### Model URL

https://huggingface.co/uukuguy/speechless-coding-7b-16k-tora

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Check https://evalplus.github.io/leaderboard.html
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - speechless-code-mistral-7b-v1.0
Root cause: Unimplemented Feature Gap

### Model introduction

This model is fine-tune mistralai/Mistral-7B-v0.1 in order to improve the model's reasoning and planning abilities.

### Model URL

https://huggingface.co/uukuguy/speechless-code-mistral-7b-v1.0

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Evaluated and updated on https://evalplus.github.io/leaderboard.html!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Python-Code-33B
GitHub labels: invalid

model eval

### Model introduction

This model is trained on around 23000+ set of codes. Each set having 2 conversations.  This conversation is in Vicuna/ShareGPT format. Each set, along with code, has detailed explanation. I have released the data.

Training: Entire dataset was trained on 4 x A100 80GB. For 3 epoch, training took 42 hours. DeepSpeed codebase was used for training purpose. This was trained on Llama-1 by Meta.

### Model URL

https://huggingface.co/ajibawa-2023/Python-Code-33B

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

seems the 33b versions of your models are producing explanations rather than solutions

![image](https://github.com/evalplus/evalplus/assets/38074777/a4f247c0-9c51-4385-8dfe-f59c03855aae)

consequently, the scores are even lower than smaller models.

![image](https://github.com/evalplus/evalplus/assets/38074777/ea41fab5-3676-4250-be0d-a68664ccd0c0)

i guess this is likely to be a bug so i am not going to update this on the leaderboard for now to avoid confusion




Ohh ok, this model has some bug. This is not doing good.
Thank you very much for evaluating all my models.  

Cool. I will close this issue for now but you are welcome to reopen it once you fixed the model. :)
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <WizardCoder-33B-V1.1>
Root cause: Unimplemented Feature Gap

### Model introduction

New version of WizardCoder

### Model URL

https://huggingface.co/WizardLM/WizardCoder-33B-V1.1

### Data type

bfloat16

### Additional instructions (Optional)

- We have provided the reproduced process here: https://huggingface.co/WizardLM/WizardCoder-33B-V1.1#how-to-reproduce-the-performance-of-wizardcoder-33b-v11

- We also provide all generated results, including both with and without using vllm: https://github.com/nlpxucan/WizardLM/blob/main/WizardCoder/data/humaneval_mbpp_wizardcoder33b_v1.1_results.zip

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Congratulations! I have already updated the scores on the leaderboard (https://evalplus.github.io/leaderboard.html). Nice job!
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - budecosystem/code-millenials-34b
Root cause: Unimplemented Feature Gap

### Model introduction

This is a finetuned model on CodeLLama34B. Please see the model details here. 

https://huggingface.co/budecosystem/code-millenials-34b


### Model URL

https://huggingface.co/budecosystem/code-millenials-34b

### Data type

float16

### Additional instructions (Optional)

The steps to reproduce the score is mentioned here. https://github.com/BudEcosystem/code-millenials/tree/main/scripts

The generated result for the evaluation is added here. https://github.com/BudEcosystem/code-millenials/tree/main/data

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Thanks for sending out the request!

```bash
python3 evalplus/evaluate.py --dataset mbpp --samples /scratch/humanevalp/mbpp/code-millenials-34b_temp_0.0-sanitized
mbpp (base tests)
pass@1: 0.729
mbpp+ (base + extra tests)
pass@1: 0.599
-----------------------------------------
base ❯ python3 evalplus/evaluate.py --dataset humaneval --samples /scratch/humanevalp/humaneval/code-millenials-34b_temp_0.0-sanitized
humaneval (base tests)
pass@1: 0.713
humaneval+ (base + extra tests)
pass@1: 0.677
```

The results have been updated on https://evalplus.github.io/leaderboard.html and currently, it seems to be the best model in the 34B category.

Thanks for adding the model in the leaderboard. It's not the best as per the current values you have shared.  The eval score seems less compared to the value when I tried running. Have you used your own generation script or the one I have added in the description?

@dittops I am using a unified alpaca template:

https://github.com/evalplus/evalplus/blob/233addde921ce74e5ea5c54f2c35cb229bdd1705/codegen/model.py#L228

Many other evaluated models on the leaderboard also use this template and other setups. Also notably we are reporting pass@1 based on greedy decoding. 

Thanks for the quick update. I will take a look at that.

Just want to let you know that we fixed a recent post-processing bug where the score of your models has been increased. :D
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - 33x-coder
Root cause: Unimplemented Feature Gap

### Model introduction

33x-coder is a powerful Llama based model, designed to assist and augment coding tasks. Leveraging the capabilities of advanced language models, 33x-coder specializes in understanding and generating code. This model is trained on a diverse range of programming languages and coding scenarios, making it a versatile tool for developers looking to streamline their coding process. Whether you're debugging, seeking coding advice, or generating entire scripts, 33x-coder can provide relevant, syntactically correct code snippets and comprehensive programming guidance. Its intuitive understanding of coding languages and constructs makes it an invaluable asset for any coding project, helping to reduce development time and improve code quality.



### Model URL

https://huggingface.co/senseable/33x-coder

### Data type

float16

### Additional instructions (Optional)

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("senseable/33x-coder", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("senseable/33x-coder", trust_remote_code=True).cuda()
messages = [
    {'role': 'user', 'content': "Write a Python function to check if a number is prime."}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)


### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🐛 [Explanation Required] - Average, MBPP, and HumanEval
Root cause: Documentation Deficiency

It seems the average is not the mean of MBPP and HumanEval. Then what does this average evaluation mean after all?
![image](https://github.com/evalplus/evalplus/assets/8592144/6f5dbb02-40e1-4ed3-8b5b-a9480539383d)
![image](https://github.com/evalplus/evalplus/assets/8592144/21b74b16-325b-4754-951c-154aa49abb65)
![image](https://github.com/evalplus/evalplus/assets/8592144/c00ae8dd-7cef-4c66-b218-06646797720f)



Average score = (HE+ score + MBPP+ score) / 2
76.2 = (81.7+70.7)/2 
What's wrong with that? :D

For the right-side column, please note that there are two **DIFFERENT** annotated by you: one is GPT-4-**Turbo** and another is just GPT-4. 

> Average score = (HE+ score + MBPP+ score) / 2 76.2 = (81.7+70.7)/2 What's wrong with that? :D

That is so clear, thanks a lot!

BTW, since those are averaged based on the same benchmark collections, have you considered merging those three leaderboards into one so it makes it easier to make the comparison? @ganler 

I think it is a bit visually unfriendly to cross-compare each other using three leaderboards.

Thanks for the suggestions! That would make the visualization challenging. However, we are considering moving to/syncing at the Hugging Face Hub which has the functionality you just described.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - speechless-coder-ds-6.7b
Root cause: Unimplemented Feature Gap

### Model introduction

## speechless-coder-ds-6.7b

Use the following dataset to fine-tune deepseek-ai/deepseek-coder-6.7b in order to improve the model's reasoning and planning abilities.

context window length: 8192 max_tokens > 128 && < 8192

Total 185,193 samples 426 MB

ise-uiuc/Magicoder-OSS-Instruct-75K 75,186 samples
ise-uiuc/Magicoder-Evol-Instruct-110K 110,007 samples

### Model URL

https://huggingface.co/uukuguy/speechless-coder-ds-6.7b

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Check https://evalplus.github.io/leaderboard.html
Repository: evalplus/evalplus
Issue: Zyte-1B
GitHub labels: model eval

### Model introduction

General purpose chatting, text generation and summarisation. 

### Model URL

https://huggingface.co/aihub-app/zyte-1B

### Data type

float16

### Additional instructions (Optional)

Amazing throughput vLLM and LM Studio compatible 

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

@venkycs 

MBPP(+) scores:

![image](https://github.com/evalplus/evalplus/assets/38074777/54b261cc-f877-48f6-97c9-4f356d7e3fff)

HumanEval(+) scores:

![image](https://github.com/evalplus/evalplus/assets/38074777/c70ef366-dd88-4f7e-919d-b16e215989e7)


I found such a type of syntactical error is very common in Zyte:

![image](https://github.com/evalplus/evalplus/assets/38074777/87c8f5ee-017a-48c5-9f10-30daaad043b3)

You can find the prompting here:

https://github.com/evalplus/evalplus/blob/28d05f60244841b97039369cec9427cd64400bc5/codegen/model.py#L183

which follows the repo example. the inference is powered by vLLM as Zyte's base architecture is Llama.

Let me know if you want me to post the scores on the leaderboard.


Sure, please can you post and share link ?  @ganler
Repository: evalplus/evalplus
Issue: Problem 53 - Wrong Inputs
Root cause: Validation Gap

Problem `HumanEval_53` in `evalplus` is given as below with no contracts. A test input in `evalplus` is `add('KKDnux', 'KDDnuxs')` which gives string inputs. There are multiple test cases with string inputs for this problem.

```
def add(x: int, y: int):
    """Add two numbers x and y
    >>> add(2, 3)
    5
    >>> add(5, 7)
    12
    """
    return x + y
```
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Stable Code
Root cause: Unimplemented Feature Gap

### Model introduction

Stable Code 3b was created by Stability AI's CodeAI team. It is a Small Language Model with performance similar to Code LLama 7B.

### Model URL

https://huggingface.co/stabilityai/stable-code-3b

### Data type

bfloat16

### Additional instructions (Optional)

Preferred is bfloat16

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Scores are updated on the leaderboard. Thanks for the request.
Repository: evalplus/evalplus
Issue: Reproducing Phi2 result
Root cause: Documentation Deficiency

I was trying to reproduce the Phi2 result with the following command

```
python codegen/generate.py --model phi-2 --dataset humaneval --root data --greedy 
evalplus.evaluate --dataset humaneval --samples data/humaneval/phi-2_temp_0.0/
```

But I was getting the below result.

```
humaneval (base tests)
pass@1: 0.171
humaneval+ (base + extra tests)
pass@1: 0.159
```

Could you please suggest what might be the reason here?



For models whose outputs are not strictly code-structured, you also need to run some post-processing:

```shell
python tools/sanitize.py --samples data/humaneval/phi-2_temp_0.0/ --dataset humaneval
evalplus.evaluate --dataset humaneval --samples data/humaneval/phi-2_temp_0.0-sanitized/
```

Thanks for the update. I was able to get below result with sanitize.

```
humaneval (base tests)
pass@1: 0.494
humaneval+ (base + extra tests)
pass@1: 0.433
```

Great. Because the HE scores you are showing are very close to what we had on the leaderboard, I am going to close this issue. But feel free to reopen it if you have any further questions.

![image](https://github.com/evalplus/evalplus/assets/38074777/16b34d91-624e-40f8-a6dc-f82e6eedd6ce)