zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - CodeGemma-7b
GitHub labels: model eval

### Model introduction

We've fine-tuned Gemma-2b with an additional 0.7 billion high-quality, code-related tokens for 3 epochs. This model operates using the Alpaca instruction format (excluding the system prompt).

### Model URL

https://huggingface.co/TechxGenus/CodeGemma-7b

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - uukuguy/speechless-starcoder2-7b
GitHub labels: model eval

actionable

### Model introduction

https://huggingface.co/uukuguy/speechless-starcoder2-7b

# speechless-starcoder2-7b
Code: https://github.com/uukuguy/speechless

Use the following dataset to fine-tune bigcode/starcoder2-7b in order to improve the model's reasoning and planning abilities.

Total 986k samples.

- teknium/OpenHermes-2.5
- TokenBender/python_eval_instruct_51k
- Spider
- codefuse-ai/Evol-instruction-66k

## How to Prompt the Model
This model accepts the Alpaca instruction format.

For example:

You are an intelligent programming assistant.

### Instruction:
Implement a linked list in C++

### Response:



### Model URL

https://huggingface.co/uukuguy/speechless-starcoder2-7b

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

HE+:
<img width="747" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/7369feaa-ef97-480a-935d-7bb557babcee">

MBPP+:
<img width="736" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/8a64d628-f173-4e8b-84b5-11d9fa3ef86c">
Repository: evalplus/evalplus
Issue: 01Coder-7B - Model Evaluation Request
Root cause: Unimplemented Feature Gap

### Model introduction

The model is created by Manoj Athreya A, student from University of Ottawa. It used base Mistral 7B model SFT on a combination of three datasets: ise-uiuc/Magicoder-OSS-Instruct-75K, HuggingFaceH4/CodeAlpaca_20K, and theblackcat102/evol-codealpaca-v1.

### Model URL

https://huggingface.co/Villian7/01Coder

### Data type

float32

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - uukuguy/speechless-starcoder2-15b
Root cause: Unimplemented Feature Gap

### Model introduction

https://huggingface.co/uukuguy/speechless-starcoder2-15b

<p><h1> speechless-starcoder2-15b  </h1></p>

Code: https://github.com/uukuguy/speechless

Use the following dataset to fine-tune bigcode/starcoder2-15b in order to improve the model's reasoning and planning abilities.

Total 986k samples.
- teknium/OpenHermes-2.5
- TokenBender/python_eval_instruct_51k
- Spider
- codefuse-ai/Evol-instruction-66k

## How to Prompt the Model
This model accepts the Alpaca instruction format.

For example:
```
You are an intelligent programming assistant.

### Instruction:
Implement a linked list in C++

### Response:
```

### Model URL

https://huggingface.co/uukuguy/speechless-starcoder2-15b

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

HE+:
<img width="744" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/f69bf3ae-45c9-45c0-848b-548fdc077412">

MBPP+:
<img width="748" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/1562d4c8-61ba-42b4-a5f4-7f123a8aea37">
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - ajibawa-2023/Code-Mistral-7B
GitHub labels: model eval

### Model introduction

This Model is trained on refined version of my dataset Code-290k-ShareGPT. Besides this it is trained on following datasets:

Code-Feedback
orca-math-word-problems-200k
Openhermes

The idea was to check how this Model will perform with both Code & Maths datasets. This model is very good with Coding. Maths is still hit & miss but you can test out this model.

This Model is trained on massive datasets so the results are very good. I have used ChatML prompt format.

Thank you very much for evaluating my earlier models.

### Model URL

https://huggingface.co/ajibawa-2023/Code-Mistral-7B

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - xai-org/grok-1
Root cause: Unimplemented Feature Gap

### Model introduction

The model was created by xAI and [weights were publicly released 2024-Mar-17](https://x.ai/blog/grok-os). It's a 314B MoE model.

### Model URL

https://huggingface.co/xai-org/grok-1

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

I only confirm the above to the extent that I have not inserted any plagiarism or malicious content. I do not know whether xAI has inserted plagiarismm or malicious content.

Given its generally low performance and extremely large size, it may not be worth it to evaluate this model.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - databricks/dbrx-instruct
Root cause: Unimplemented Feature Gap

### Model introduction

This is the Instruct finetune of Databricks' recently released DBRX model.

### Model URL

https://huggingface.co/databricks/dbrx-instruct

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

I only confirm the above to the extent that I have not inserted any plagiarism or malicious content. I do not know whether Databricks has inserted plagiarism or malicious content.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <LeroyDyer/Mixtral_AI_Cyber_3.m1>
GitHub labels: model eval

### Model introduction

Coding whilst playing a role; as well as medical diagnosis and simulated labs for virtual projects such as chemistry and biology , for instance you can ask to set up lab for virus examination and the model will generate a expert agent o be your guide ; ready to answer questions on the topic , whilst other requests will still go to the main agent.... For coding it as no issues with any programming language or request or designing concepts and applications . whilst it has not been trained on open interpreter style usage it functions perfectly in the environment creating python or vbs script or PowerShell etc as required... also prompted to be an actual person instead of a helpful AI

### Model URL

https://huggingface.co/LeroyDyer/Mixtral_AI_Cyber_3.m1

### Data type

float16

### Additional instructions (Optional)

works fine its a mistral model safe tensors like normal. as it is a base model 

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: [HumanEval/32] Need to remove extreme inputs
Root cause: Validation Gap

HumanEval/32 finds zero points over a polynomial. The algorithm is not expected to be precise when the polynomial is extremely sharp. For example:

If we draw $y=-36x^{3}+6x^{2}-1440\ +\ 1$ over https://www.desmos.com/calculator, we will see a very sharp line...

If we use our current GT we get a output like -0.135.... but it is not correct during validation.

IMO the simplest thing we can do is to put the validation in the groundtruth (i.e., `poly`) to filter out unreasonable inputs. 

I think the problematic input should be $y=x^3-1440x^2+6x-36$ instead. Anyway, I filtered out those extreme inputs in the most recent commit. cc @ganler
Repository: evalplus/evalplus
Issue: [BUG] Running sanitize and synccheck from CLI
Root cause: Validation Gap

Running `evalplus.sanitize` & `evalplus.syncheck` commands  while using EvalPlus as a local repo returns the following error

```
> evalplus.syncheck --samples wizardcoder-34b.jsonl --dataset humaneval                                                                                                                
Traceback (most recent call last):
  File "/Users/username/github/evalplus/venv/bin/evalplus.syncheck", line 8, in <module>
    sys.exit(main())
TypeError: main() missing 2 required positional arguments: 'samples' and 'dataset'
```

A quick workaround I used for this was modifying the code in `venv/bin/evalplus.syncheck` file 
```
#!/Users/username/github/evalplus/venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from evalplus.syncheck import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
-  sys.exit(main())
+  sys.exit(main(samples=sys.argv[2],dataset=sys.argv[4]))
```
But a more permanent fix would be parsing arguments using `argparse.ArgumentParser()` in [sanitize.py](https://github.com/evalplus/evalplus/blob/master/evalplus/sanitize.py) and [syncheck.py](https://github.com/evalplus/evalplus/blob/master/evalplus/syncheck.py) at https://github.com/evalplus/evalplus/blob/4f63a554df230d70e4c50ca845cfc8a3e62e4067/evalplus/sanitize.py#L179 
https://github.com/evalplus/evalplus/blob/4f63a554df230d70e4c50ca845cfc8a3e62e4067/evalplus/syncheck.py#L24

Similar to how it's being done in [evaluate.py](https://github.com/evalplus/evalplus/blob/master/evalplus/evaluate.py)
https://github.com/evalplus/evalplus/blob/4f63a554df230d70e4c50ca845cfc8a3e62e4067/evalplus/evaluate.py#L327-L328



Hi can you double check if you have `fire` installed?

Yes I do
```
> pip freeze | grep fire
fire==0.6.0
```

ok i think i know where the bug is. give a sec to fix it

@nalinabrol can you try 

```
pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

Sorry I did not do a complete testing for v0.2.2. I am going to yank it and bring it back in v0.2.3 or v0.3.0. But you can access this feature thru a nightly install. Thanks!

That was fast Jiawei! ⚡ 
Its working now.

Thank you @nalinabrol for reporting this important bug. We will improve our testing in future releases!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <G2b>
GitHub labels: model eval

### Model introduction

it is a fined tuned version of gemma 2b on APPS dataset

### Model URL

AdnanRiaz107/Gemma2b-APPS

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: [Feature Request] DS-1000
Root cause: Unimplemented Feature Gap

I wanted to reach out to you regarding a potential enhancement that I believe would greatly benefit the users of this project. I was wondering if you have any plans to integrate support for the DS-1000 system in the near future?

The DS-1000 has gained significant popularity in recent times due to its advanced capabilities and seamless integration with various tools and frameworks. By incorporating DS-1000 support into this project, I believe it would open up a whole new range of possibilities for users and expand the project's versatility.

EvalPlus is designed to generate test inputs for generic Python programs which might not support in the DS scenario unless we use specific mutation rules or purely rely on LLMs to generate inputs directly / indirectly. I don't see an immediate interest of support but contributions are welcome.

If you simply mean running DS-1000 using evalplus, I'd suggest you try out https://github.com/bigcode-project/bigcode-evaluation-harness

Thank you for your answer, I will close this issue.
Repository: evalplus/evalplus
Issue: wrong test case for HumanEval/{126, 6}
Root cause: Validation Gap

HumanEval/126 checks if a list of numbers is sorted. One of the plus inputs is `[""]`, which is wrong.

Hi, thanks for the report. 

@Kristoff-starling Can you help updating the contract of 126 to:

```python
    assert type(lst) == list, "invalid inputs" # $_CONTRACT_$
    assert all(x >= 0 for x in lst), "invalid inputs" # $_CONTRACT_$
    assert all(type(x) == int for x in lst), "invalid inputs" # $_CONTRACT_$
```

Thanks for the reports! @crhf Please kindly use `pip install "git+https://github.com/evalplus/evalplus.git" --upgrade` to use the updated HumanEval+ dataset for now. An official dataset on pypi will be released later.

Thanks for the fix!

Another issue: HumanEval/6 promises to provide paired parentheses as input, but there are the inputs `'(((((()()(((())))(()(())))()))()) ()(()()) ((())) ((())())())'`, `'()()(((())()) ()(()()) ((())) ((())())())'`, which have unpaired groups.

Thanks for the catch! @crhf The issue has been fixed.

Hi @ganler and @Kristoff-starling ,

I just downloaded the latest version [HumanEvalPlus-v0.1.6.jsonl.gz](https://github.com/ganler/release/releases/download/humanevalplus/HumanEvalPlus-v0.1.6.jsonl.gz) and [HumanEvalPlus-Mini-v0.1.6.jsonl.gz](https://github.com/ganler/release/releases/download/humanevalplus/HumanEvalPlus-Mini-v0.1.6.jsonl.gz)

It seems that the contract for **HumanEval/126** is still the old one:

`"contract": "\n    assert all(x >= 0 for x in lst), \"invalid inputs\" # $_CONTRACT_$\n    assert all(type(x) == int for x in lst), \"invalid inputs\" # $_CONTRACT_$\n"`

Also, for **HumanEval/126** in [HumanEvalPlus-Mini-v0.1.6.jsonl.gz](https://github.com/ganler/release/releases/download/humanevalplus/HumanEvalPlus-Mini-v0.1.6.jsonl.gz), it seems that the invalid input is still there:

`"plus_input": [[[1, 2, 3, 4, 5, 6, 7, 8]], [[1, 2, 2, 2, 2, 3, 4]], [[1, 1, 2, 2, 3, 3, 4]], [[1, 1, 2, 2, 2, 3, 4]], [""], [[1, 1, 1, 1]], [[1, 1, 2, 3, 4, 4]], [[1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2]]]`

Do you mind helping me take a look?

Huge Thanks : )

@marcusm117 Sorry that the previous fix was unexpectedly overwritten by successive commits and the problem should have been fixed now. Could you please re-download the latest version and check if everything goes well? Thanks!

Yes, they're fixed now. Thanks so much! @Kristoff-starling
Repository: evalplus/evalplus
Issue: MBPP+ HF dataset generation

What is the script that you used to generate the MBPP+ dataset on HF hub (MBPP+: https://huggingface.co/datasets/evalplus/mbppplus)?

- i'm curious about how you handled the inputs serialisation in the test column
- could we have a version where the base/plus assertions are separated?


Thank you!
 

> i'm curious about how you handled the inputs serialisation in the test column

Sure! 

To serialize MBPP inputs: https://github.com/evalplus/evalplus/blob/ce6db8f3e168853e001a3e32398b15f66ca942b7/evalplus/data/mbpp.py#L34-L44

To deserialize MBPP inputs: https://github.com/evalplus/evalplus/blob/ce6db8f3e168853e001a3e32398b15f66ca942b7/evalplus/data/mbpp.py#L47

> What is the script that you used to generate the MBPP+ dataset on HF hub

https://github.com/evalplus/evalplus/blob/master/tools/mbpp/to_original_fmt.py

Inputs are selected here: https://github.com/evalplus/evalplus/blob/master/tools/mbpp/to_original_fmt.py#L149

Let me know if you have more questions!

Thank you for all the insights. This is very helpful!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - deepseek-coder-33b-base 
Root cause: Documentation Deficiency

### Model introduction
deepseek-coder-33b-base is a 33B parameter model with Multi-Head Attention trained on 2 trillion tokens.
### Model URL

https://huggingface.co/deepseek-ai/deepseek-coder-33b-base
### Data type

float16
### Additional instructions (Optional)

This should be fairly simple, as deepseek-coder base models for all other sizes are evaluated, and deepseek-coder-33b-instruct is evaluated.
### Author

No
### Security
* [x]  I confirm that the model is safe to run which does not contain any malicious code or content.


### Integrity
* [x]  I confirm that the model comes from unique and original work and does not contain any plagiarism.

deepseek-coder-33b-base is already evaluated. pls check https://evalplus.github.io/leaderboard.html :)

> deepseek-coder-33b-base is already evaluated. pls check https://evalplus.github.io/leaderboard.html :)

Although it's evaluated on HumanEval, deepseek-coder-33b-base does not have an MBPP score on the leaderboard.

Oh my bad.

You can use the scores from the StarCoder2 paper (Table 9 was produced by me). :-)

Also will update it on the leaderboard.

updated.

> updated.

Ok, thanks!

This isn't related, but is there a leaderboard that also tests the other metrics in StarCoder2 paper, such as MultiPL-E, DS-1000, HumanEvalFix, CanItEdit, GSM8K, CRUXEval-I, CRUXEval-O, FIM-task, RepoBench, CrossCodeEval, and Asleep at the Keyboard?

@ganler Although the MBPP score is on the leaderboard, it has disappeared from the size vs MBPP and size vs average score chart.
Repository: evalplus/evalplus
Issue: Some tests are not deterministic (and incorrectly fail) due to randomness in the tuple(set()) or list(set()) operations
Root cause: Validation Gap

5 samples have some of their "plus" tests incorrectly failing due to making the assert on a tuple(set()) or a list(set()): 2, 7, 111, 140, 232.

As an example, take task Mbpp/2:
- the ground truth solution is 
```
def similar_elements(test_tup1, test_tup2):
     return tuple(set(test_tup1) & set(test_tup2))
```
the operation "tuple(set(test_tup1) & set(test_tup2))" is not deterministic as the ordering depends on the PYTHONHASHSEED. 
- the base tests correctly wrap the function call and target with a set(), hence "reverting" the "tuple" call: `assert set(similar_elements((3, 4, 5, 6),(5, 7, 4, 10))) == set((4, 5))`
- the plus tests don't do this, whether it's in the evalplus package or in the Huggingface file

The other 4 tasks exhibit similar problems, sometime it's a "list()" instead of a "tuple".

This is problematic because the targets are computed in one python process, and then the function calls on the inputs are done in another process, which means that the 2 do not always match (because different PYTHONHASHSEED). This can be seen when running the GT solution on the "plus" tests for example, some of them will fail.

I think an easy fix would be to also wrap the "plus" tests with a "set()" call, similar to the "base" tests.

Great note! Just to clarify that these happen in MBPP tasks whose task ids are 2, 7, 111, 140, 232?

Fixed, I do see some models can get +0.5 pp.

awesome! Any chance we can also make this fix in the hugging face file, to keep the 2 in sync? :)
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Command-R+
Root cause: Unimplemented Feature Gap

### Model introduction

The model was created by Cohere and shows very good performance in many benchmarks and in subjective evaluations. It is the #2 open model on Chatbot Arena Coding Leaderboard.

### Model URL

https://huggingface.co/CohereForAI/c4ai-command-r-plus

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - miqudev/miqu-1-70b
GitHub labels: model eval

### Model introduction

This is a finetune of Llama 2 70B created by Mistral and distributed to unknown corporations in a quantized form.

### Model URL

https://huggingface.co/miqudev/miqu-1-70b

### Data type

None of above: specify the details in the 'Other context' section

### Additional instructions (Optional)

It is a Q5_K_M GGUF. If you need a float16 vesion, the Q5_K_M has been dequantized here: https://huggingface.co/152334H/miqu-1-70b-sf .

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Qwen1.5-72B-Chat
Root cause: Unimplemented Feature Gap

### Model introduction

The model is created by Tongyi Qianwen and is #1 in the Chatbot Arena Coding Leaderboard.

### Model URL

https://huggingface.co/Qwen/Qwen1.5-72B-Chat

### Data type

float16

### Additional instructions (Optional)

It would be good to also evaluate the base and smaller models, so they can be compared to other smaller models such as the Deepseek models. 

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Mistral-7B-Instruct-v0.2
Root cause: Unimplemented Feature Gap

### Model introduction

The model is a finetune of Mistral-7B-v0.2 made by Mistral, and they have not documented its differences from Mistra-7B-Instruct-v0.1.

### Model URL

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - FuseChat-7B-VaRM
Root cause: Unimplemented Feature Gap

### Model introduction

This model is a merger of https://huggingface.co/NousResearch/Nous-Hermes-2-SOLAR-10.7B, https://huggingface.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO, and https://huggingface.co/openchat/openchat_3.5. It is unique in that it merges various sizes of models into a single 7.24B paramater model, and achieves much higher MT-Bench performance than other models less than 11B.

### Model URL

https://huggingface.co/FuseAI/FuseChat-7B-VaRM

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.