zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: Why the mbpp dataset randomly contains web urls in the prompt field?
Root cause: Documentation Deficiency

### EvalPlus version

recent

### Output of running `ls ~/.cache/evalplus`

NA

### Task ID of the programming task

Many

### Test input

```python
N/A
```


### Description

https://raw.githubusercontent.com/google-research/google-research/master/mbpp/sanitized-mbpp.json

There are many URLs contained within the prompt (e.g., you can search https://www.geeksforgeeks.org/python-find-minimum-k-records-from-tuple-list/).

### Other context

_No response_

It comes from the original MBPP dataset (https://github.com/google-research/google-research/tree/master/mbpp) -- not added by EvalPlus. EvalPlus only adds more test cases. :)
The reason is probably because MBPP dataset is crowd-sourced from the web such as geeks4geeks, etc. Their paper (https://arxiv.org/abs/2108.07732) also said that. :)

 @ganler 
> It comes from the original MBPP dataset (https://github.com/google-research/google-research/tree/master/mbpp) -- not added by EvalPlus. EvalPlus only adds more test cases. :) The reason is probably because MBPP dataset is crowd-sourced from the web such as geeks4geeks, etc. Their paper (https://arxiv.org/abs/2108.07732) also said that. :)

Sorry, but I find the original MBPP dataset actually doesn't contain web urls, for example the prompt of MBPP/734 in MbppPlus.jsonl(also sanitized-mbpp) is:
```
{
"task_id": "Mbpp/734", "prompt": "\"\"\"\nWrite a python function to find sum of products of all possible sublists of a given list. https://www.geeksforgeeks.org/sum-of-products-of-all-possible-subarrays/\nassert sum_Of_Subarray_Prod([1,2,3]) == 20\n\"\"\"\n"
}
```
Additional URL(https://www.geeksforgeeks.org/sum-of-products-of-all-possible-subarrays/) is included after the problem description.

But in the original MBPP dataset(https://github.com/google-research/google-research/tree/master/mbpp) the prompt for task 734 is as follows:
```
{
"text": "Write a python function to find sum of products of all possible subarrays.",
}
```
Problem ids involved I found:
```
734, 735, 736, 736, 739, 740, 743, 745, 749, 751, 752, 753, 757, 763, 771, 780, 781, 782, 783, 803, 
```
Appreciate your clarification, although I believe that this subtle difference has little impact on the overall performance.

MBPP+ tasks are a subset of MBPP sanitized. 

There are `urls` from https://github.com/google-research/google-research/blob/master/mbpp/sanitized-mbpp.json

But I guess we can still remove these URLs to avoid confusion. Added as another TODO in #159.

Thanks for clarification. Looks like this 'sanitized' version isn't really sanitized after all.😂
Repository: evalplus/evalplus
Issue:  np.testing.assert_allclose fails when subprocess.Popen = None
Root cause: Architectural Constraint

It seems like np.testing calls subprocess.Popen under the hood which makes the `assertion` function fail because the reliability guard set subprocess.Popen to None.

To reproduce the issue:
```
>>> import numpy as np
>>> import subprocess
>>> subprocess.Popen = None
>>> np.testing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/fsx-llm/shared/envs/xlformers_mm_finetune_20240311/lib/python3.10/site-packages/numpy/__init__.py", line 327, in __getattr__
    import numpy.testing as testing
  File "/fsx-llm/shared/envs/xlformers_mm_finetune_20240311/lib/python3.10/site-packages/numpy/testing/__init__.py", line 11, in <module>
    from ._private.utils import *
  File "/fsx-llm/shared/envs/xlformers_mm_finetune_20240311/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 1253, in <module>
    _SUPPORTS_SVE = check_support_sve()
  File "/fsx-llm/shared/envs/xlformers_mm_finetune_20240311/lib/python3.10/site-packages/numpy/testing/_private/utils.py", line 1247, in check_support_sve
    output = subprocess.run(cmd, capture_output=True, text=True)
  File "/fsx-llm/shared/envs/xlformers_mm_finetune_20240311/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
TypeError: 'NoneType' object is not callable
```

Because of that, some "plus" tests fail although the solution is actually correct.

Could we rather use math.isclose with rtol = 1e-7 (default in np.testign.assert_all_close) instead of np.testing.assert_allclose?

I think we either implement our own `np.testing.allclose` or we can hack the OpenAI reliability guard to throw OSError.

Let me know if the latter option can also work for your case (assuming you are using EvalPlus or can modify the reliability guard of your internal framework). :) 

I think both solutions would work :) I have a preference for implementing a new "np.testing.allclose", I guess we can just use math.isclose and iterating on the list/tuple elements when needed?

Because otherwise people who use the hugging face file will have to be careful with not using the OpenAI reliability guard, or disabling "subprocess.Popen = None". While if we change np.testing for something else, then it will work out of the box for everyone without having to know about these things :) wdyt?

I agree. My only concern is if we can really implement an equivalent to `np.testing.allclose` given that it is actually challenging to deal with compound data structures for such checking and we also want to have some way to test it.

Anyway, let me look into this and I hope this won't affect too many cases. (but the difference/gap between base and plus still holds as they share the same testing script). 

yep you're right it might not be trivial, it really depends on what cases we use it for :)
Happy to help if needed!
Repository: evalplus/evalplus
Issue: pass@k samples format?
Root cause: Validation Gap

Hello, thanks for developing this package! I want to ask what the correct format for the `samples.jsonl` file to obtain pass@k metrics for k > 1. From the document,
```from evalplus.data import get_[human_eval|mbpp]_plus, write_jsonl

samples = [
    dict(task_id=task_id, solution=GEN_SOLUTION(problem["prompt"]))
    for task_id, problem in get_[human_eval|mbpp]_plus().items()
]
write_jsonl("samples.jsonl", samples)
```

I understood the way is to pass in `solution` with a list of programs. However, the resulting pass accuracies all became 0. So what is the correct format? Thanks in advance!

The format is right but probably you want to do post-processing https://github.com/evalplus/evalplus?tab=readme-ov-file#code-post-processing

Thanks for the response! I made sure the syntax was correct. Upon closer look at the code, it seemed that the `check_correctness` function (https://github.com/evalplus/evalplus/blob/master/evalplus/evaluate.py#L82) only expects a string input for the solution, but will receive a list in my use case (https://github.com/evalplus/evalplus/blob/master/evalplus/evaluate.py#L178). I tested it out that by passing a single string to the `solution` field I could get a reasonable pass@1 but if I pass in the same string but in a list, I got 0 instead.

I see. If you want to include multiple samples per task, you can simply duplicate the json lines with repeated task id. Sorry for the confusion and will add a schema check. :) Thanks for letting me know the issue!

25e195e024b614f2671ad9ac5b8fdcd9b95a2b24 added simple schema checking. :)
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - wavecoder-ultra-6.7b
Root cause: Unimplemented Feature Gap

### Model introduction

WaveCoder 🌊 is a series of large language models (LLMs) for the coding domain, designed to solve relevant problems in the field of code through instruction-following learning. Its training dataset was generated from a subset of code-search-net data using a generator-discriminator framework based on LLMs that we proposed, covering four general code-related tasks: code generation, code summary, code translation, and code repair. 

This model is a new release from Microsoft, with impressive bench scores for a 7B, and I would love to see it tested by your Eval system.

### Model URL

https://huggingface.co/microsoft/wavecoder-ultra-6.7b

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

This looks like a finetune of Deepseek Coder 6.7b, so @evalplus can probably evaluate it the same way.

WaveCoder does not include a chat template so EvalPlus would do direct generation.

<img width="1136" alt="image" src="https://github.com/evalplus/evalplus/assets/38074777/82ddc39b-a266-4425-af28-dd771fc3fd42">


Hi, @ganler, thank you very much for helping to test our model 'wavecoder-ultra-6.7b'. In fact, we have our own prompt, which we have sent to you via email, but we have not received your reply. Could you please retest using our prompt? Thank you very much.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Artigenz-Coder-DS-6.7B
Root cause: Unimplemented Feature Gap

### Model introduction

Model by Artigenz team, intending to create family of code generation models that runs fast on local computers.

### Model URL

[Artigenz/Artigenz-Coder-DS-6.7B](https://huggingface.co/Artigenz/Artigenz-Coder-DS-6.7B)

### Data type

bfloat16

### Additional instructions (Optional)

Our model performs best with the following decoder - 


```
class Artigenz(VLlmDecoder):
    def __init__(self, name: str, **kwargs) -> None:
        super().__init__(name, **kwargs)
        self.eos += ["\n```"]

    def codegen(
        self, prompt: str, do_sample: bool = True, num_samples: int = 200
    ) -> List[str]:
        prompt = f"""You are an exceptionally intelligent coding assistant that consistently delivers accurate and reliable responses to user instructions.

@@ Instruction
Here is a Python programming problem to solve:
{prompt}
Please implement this function in a Python.

@@ Response
```python
{prompt}
"""

        return VLlmDecoder.codegen(self, prompt, do_sample, num_samples)
```

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Results produced using changes in this fork: #166.(Support for **Artigenz-Coder-DS-6.7B**)

- humaneval
![Screenshot 2024-04-18 at 1 13 30 AM 2](https://github.com/evalplus/evalplus/assets/36822111/ec1fe93a-56bf-40f5-99aa-132b2ed47c8d)

- mbpp v0.1.0 (commit ID right before mbpp v0.2.0: fe7f0598ac96e5906fb786ba27647a09c0cf01fb)
![image (9)](https://github.com/evalplus/evalplus/assets/36822111/978b2da4-9ce5-4d13-a3a4-a8377e4b73f6)


- mbpp v0.2.0
![image (8)](https://github.com/evalplus/evalplus/assets/36822111/db028e12-ec96-4017-82d7-cdbfbc172c93)

The above results were produced on an `Nvidia A100 40GB` instance in AWS(Oregon region)


@ganler can you please take a look

Looks good. We are now still working on updating scores towards v0.2.0 of the leaderboard and will look into this after that.

Added to the leaderboard using MBPP+ v0.1.0. Let me know when the sft data is open-sourced so that i can change the "open-data" field. Thanks!

Thanks for swift response @ganler! 
It seems like all models you had added on leaderboard today  llama3/Atrtigenz/qwen/command-r are accidentally overwritten in this commit https://github.com/evalplus/evalplus.github.io/commit/4f0803b2af05ad3e6b5edd76dad150104100f0f3

@nalinabrol I am so sorry for the inconvenience. I have reverted the unverified commit and will update the v0.2.0 results after careful review.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - CodeQwen1.5-7B-Chat
Root cause: Unimplemented Feature Gap

### Model introduction

The model was created by Tongyi Qianwen. It is an instruct finetune of their CodeQwen1.5-7B, which is a coding finetune of their Qwen1.5-7B.

### Model URL

https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat

### Data type

float16

### Additional instructions (Optional)

It can be run by llama.cpp using [PR#6707](https://github.com/ggerganov/llama.cpp/pull/6707). The HuggingFace Transformers inference code is on the model card, and they have released quantized versions to their HuggingFace account in separate repositories.

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

These are the reported benchmarks from [their blog post](https://qwenlm.github.io/blog/codeqwen1.5/):
![image](https://github.com/evalplus/evalplus/assets/54595360/a20077cd-462f-4dcc-a937-8b712e43a2cd)


added

@ganler Which prompt do you use for evaluating Qwen1.5-7b-chat?

CodeQwen team has this open sourced at https://github.com/QwenLM/CodeQwen1.5/tree/main/evaluation/eval_plus/released/outputs
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - CodeQwen1.5-7B
Root cause: Unimplemented Feature Gap

### Model introduction

The model was created by Tongyi Qianwen. It is a coding finetune of their Qwen1.5-7B.

### Model URL

https://huggingface.co/Qwen/CodeQwen1.5-7B

### Data type

float16

### Additional instructions (Optional)

It can be run by llama.cpp using [PR#6707](https://github.com/ggerganov/llama.cpp/pull/6707). The HuggingFace Transformers inference code is on the model card, and they have released quantized versions to their HuggingFace account in separate repositories.

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - dolphincoder-starcoder2-7b
Root cause: Unimplemented Feature Gap

### Model introduction

This is a finetune of StarCoder2 on the datasets used to finetune Magicoder, OpenCodeInterpreter, and Orca-Math.

### Model URL

https://huggingface.co/cognitivecomputations/dolphincoder-starcoder2-7b

### Data type

float16

### Additional instructions (Optional)

Since it's a finetune of StarCoder2, you should be able to evaluate it the same way you evaluated StarCoder2.

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - dolphincoder-starcoder2-15b
Root cause: Unimplemented Feature Gap

### Model introduction

This is a finetune of StarCoder2 on the datasets used to finetune Magicoder, OpenCodeInterpreter, and Orca-Math.

### Model URL

https://huggingface.co/cognitivecomputations/dolphincoder-starcoder2-15b

### Data type

float16

### Additional instructions (Optional)

Since it's a finetune of StarCoder2, you should be able to evaluate it the same way you evaluated StarCoder2.

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Llama-3-70B-Instruct 
Root cause: Unimplemented Feature Gap

### Model introduction

The model was created by Meta. It's a 70 billion parameter language model, fine tuned for chat completions.

### Model URL

https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

added

@ganler It looks like you put the model in as an 8B model on the leaderboard.

![image](https://github.com/evalplus/evalplus/assets/38074777/d379a663-ea7f-4ac1-8937-cd285c4175a3)



![Screenshot_20240422_073625_Fennec](https://github.com/evalplus/evalplus/assets/54595360/db41ac70-a9ae-40a7-8dc9-91357a062a0d)
@ganler The 70B model is in the 8B chart on the graph.

Thanks for mentioning! The issue has been fixed.

You're welcome.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Llama-3-8B-Instruct
Root cause: Unimplemented Feature Gap

### Model introduction

The model was created by Meta. It's a 8 billion parameter language model, fine tuned for chat completions.

### Model URL

https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Can see it in the leaderboard. Thanks for adding!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - AllMind AI 12B
GitHub labels: model eval

### Model introduction

AllMind AI is a hyper fine tuned model built on top of the open source model of Llama 7B for the purposes of financial analysis and investing. You can try the model for free below at the allmind website. This model can not yet be run locally through something like Ollama or an API. 

### Model URL

https://allmindinvestments.com/auth/login

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🐛 Provide the prompts for LLama3
Root cause: Documentation Deficiency

### EvalPlus version

None

### Output of running `ls ~/.cache/evalplus`

None

### Task ID of the programming task

HumanEval

### The original wrong contract

```python
None
```


### Your proposed new contract

```python
Hello, Could you provide the prompts used for llama3 on leadboard, thank you very much!
```


### Other context

_No response_

https://github.com/evalplus/evalplus/blob/02f1be465bd0b3773d33fd916133c41201e83e0e/codegen/model.py#L216-L235

The base results are reported by @huybery. Per our discussion, he used direct generation for producing Llama-3 base results.

> the original MBPP was using fewshots prompting but MBPP plus does not seem to use any fewshots, is it correct?

Yes, EvalPlus does not use few-shot as initially I was following [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness). 

But of course, I think it might be common for general-purpose base models to struggle without few-shot examples. e.g., mistral-7b.




I also did direct generation for Llama3 70B but the results are a bit too bad to be true (not sure if it's some configuration errors or vLLM bugs) so I did not put it on the leaderboard.
Repository: evalplus/evalplus
Issue: Prompt format to evaluate base models on MBPP
Root cause: Documentation Deficiency

Hey!

I'm wondering what prompt format is being used to evaluate base models on MBPP, because the original MBPP was using fewshots prompting but MBPP base/plus do not seem to use any fewshots, is it correct? For example I see a few base models on the leaderboard such as Llama3 8B Base, DeepSeek-Coder-6.7B-base etc.

Checking this [code](https://github.com/evalplus/evalplus/blob/02f1be465bd0b3773d33fd916133c41201e83e0e/codegen/model.py#L89), it seems like no special instruction beyond the prompt are provided to base models and IIUC the prompt is just the instruction + the first test.

Thanks for the help!



Thanks for the question. For base model we perform direct generation. We follow big code evaluation to include the description as well as the test list as test demonstration. 

> to include the description as well as the test list

you mean all 3 or 4 tests in the test list, or just the first one?

The first one. :)

https://github.com/evalplus/evalplus/blob/02f1be465bd0b3773d33fd916133c41201e83e0e/tools/mbpp/init_plus.py#L170-L170

This basically follows bigcode-evaluation-harness

https://github.com/bigcode-project/bigcode-evaluation-harness/blob/1b0147c50f406ff66ac4f806230479f31d19c7e6/bigcode_eval/tasks/mbpp.py#L54-L56



Got it, thanks for all the details!!

@ganler - Are the numbers for Llama3 and other models on the leaderboard using MBBPv0.2 (378) or MBPPv0.1 (399)?

It should be v0.2 as I did a global update a few mon ago but there could be a few corner cases. 

Yeah, my best guess is v0.2 because the update was made a day before the release of LLama3, and with the new version, we're almost able to reproduce the Llama3 numbers.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <WaveCoder-Ultra-6.7b>
GitHub labels: model eval

### Model introduction

WaveCoder 🌊 is a series of large language models (LLMs) for the coding domain, designed to solve relevant problems in the field of code through instruction-following learning. Its training dataset was generated from a subset of code-search-net data using a generator-discriminator framework based on LLMs that we proposed, covering four general code-related tasks: code generation, code summary, code translation, and code repair.

### Model URL

https://huggingface.co/microsoft/wavecoder-ultra-6.7b

### Data type

bfloat16

### Additional instructions (Optional)

Alpaca prompt template

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Dup of #160
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - starchat2-15b
Root cause: Unimplemented Feature Gap

### Model introduction

StarChat is a series of language models that are trained to act as helpful coding assistants. StarChat2 is the latest model in the series, and is a fine-tuned version of [StarCoder2](https://huggingface.co/bigcode/starcoder2-15b) that was trained with SFT and DPO on a mix of synthetic datasets.

### Model URL

https://huggingface.co/HuggingFaceH4/starchat2-15b-v0.1

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: Processing evaluation requests | April

Based on monthly installations, we can prioritize the evaluation of the following models:

- [x] #177 
- [x] #151 
- [x] #154
- [x] #128 
- [x] #153 

I think you should prioritize models that are optimized for code and high performance. Models for chat can be easily seen on many other benchmarks.

Hi @bino282, our computing resources are limited. But if you want to accelerate yours for #176, feel free to self-report results by posting the raw evaluation output and script.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Gemini 1.5 Pro 
Root cause: Unimplemented Feature Gap

### Model introduction

Gemini 1.5 Pro created by Google. 

### Model URL

https://aistudio.google.com/

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Updated.