zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: Integrate with LiteLLM - Evaluate 100+LLMs, 92% faster
Root cause: Unimplemented Feature Gap

Hi @jvroig  I'm the maintainer of LiteLLM. we allow you to create a proxy server to call 100+ LLMs to make it easier to run benchmark / evals . 

**I'm making this issue because I believe LiteLLM makes it easier for you to run benchmarks and evaluate LLMs (I'd love your feedback if it does not)**

Try it here: https://docs.litellm.ai/docs/simple_proxy
https://github.com/BerriAI/litellm

## Using LiteLLM Proxy Server
## Creating a proxy server 
Ollama models
```shell
$ litellm --model ollama/llama2 --api_base http://localhost:11434
```

Hugging Face Models 
```shell
$ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL]
$ litellm --model claude-instant-1
```

Anthropic 
```shell
$ export ANTHROPIC_API_KEY=my-api-key
$ litellm --model claude-instant-1
```
Palm

```shell
$ export PALM_API_KEY=my-palm-key
$ litellm --model palm/chat-bison
```

## Set api base to proxy
```
openai.api_base = "http://0.0.0.0:8000"
```

## Using to run an eval on lm harness:
```python
python3 -m lm_eval \
  --model openai-completions \
  --model_args engine=davinci \
  --task crows_pairs_english_age

```



Hi @ishaan-jaff, thanks for your interest and glad to learn about your `litellm`. 

While I think `litellm` is a great work, I feel we might not want to migrate to another framework as I think (i) our current `codegen` is already in a good shape, (ii) the maintainers are more comfortable with using the hugging face APIs which provides important transparency for evaluation, (iii) our bottleneck of evaluating model is to figure out the expected usages (e.g., prompts, data types) by inspecting any given example from the model makes; in other words, performance is not our top concern as it is fast enough to generate code over HumanEval+. Meanwhile, your pull request (#25) may only work for OpenAI models and supporting other models would take additional effort. Therefore, I am sorry that in the short term we might not consider refactoring our `codegen` module with any other APIs unless the model provider uses such tools as their offical use cases. Thanks! 

Hi @ganler thanks for the feedback 

**ii re transparency for calling the APIs - what do you want to see here? LiteLLM the library allows you to see the input args sent to the raw HF API** 

iii LiteLLM allows you to track usage for all completion calls - we return prompt & completion usage https://docs.litellm.ai/docs/completion/token_usage 

**what is the additional work you'd need to do on top of https://github.com/evalplus/evalplus/pull/25/files to add a new LLM ?**
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <deepseek-coder-1.3b-instruct>
Root cause: Unimplemented Feature Gap

### Model introduction

Can you please test the 1.3B version of deepseek-coder-instruct? I think this would be good to know if one can run this on their phone or their 4GB video-card and still have room for a large context. On the benchmark site https://huggingface.co/spaces/mike-ravkine/can-ai-code-results, it competes very well with larger models. I would like a second opinion.

### Model URL

https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-instruct

### Data type

float16

### Additional instructions (Optional)

It should be like the other 6.7B and 34B deepseek-coder models you've tested so far. 


### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

![image](https://github.com/evalplus/evalplus/assets/38074777/efb85918-571e-4c69-a4cc-f2a533ad2bb5)

Thanks!

Thank you for testing @ganler. 7th place! It's quite good for a 1.3B model!  I wonder if the WizardLM team will further fine tune it. Thank you for putting it on the board! The DeepSeek team definitely know how to train a coding model!
Repository: evalplus/evalplus
Issue: HumanEval/114 prompt disagrees with OpenAI's prompt
Root cause: Validation Gap

For HumanEval/114, the prompt provided in https://github.com/openai/human-eval is
```
def minSubArraySum(nums):
    """
    Given an array of integers nums, find the minimum sum of any non-empty sub-array
    of nums.
    Example
    minSubArraySum([2, 3, 4, 1, 2, 4]) == 1
    minSubArraySum([-1, -2, -3]) == -6
    """
```
whereas in EvalPlus, it is
```
import math

def minSubArraySum(nums):
    """
    Given an array of integers nums, find the minimum sum of any non-empty sub-array
    of nums.
    Example
    minSubArraySum([2, 3, 4, 1, 2, 4]) == 1
    minSubArraySum([-1, -2, -3]) == -6
    """
```

Is there a reason for this? Language models will typically generate different continuations for the respective prompts.

I think this is actually a bug here. 

https://github.com/evalplus/evalplus/blob/c0598d8781801de28c94eca3e5ae1d7083ad8e6a/evalplus/data/__init__.py#L215-L218

I was hoping to apply the fix for `"HumanEval/115"` whose import statement is between the signature and docstring:

![image](https://github.com/evalplus/evalplus/assets/38074777/3771e056-a2da-4846-b7b7-7a5036b15c27)

and that would affect some assumptions in our dataset making but generally should not affect the generated code.

I will double-check why the 114 index is mapped to `"HumanEval/114"` (I thought it was 0-indexed).


Thanks @ganler for clearing this up so fast!

HumanEval problems are 0-indexed so index 114 actually just point to 114. Working on a patch now.

Thanks for the report! The bug has been fixed. Please reinstall the nightly version to use the latest dataset:

```bash
pip install "git+https://github.com/evalplus/evalplus.git" --upgrade
```

The PyPI version will follow up later.
Repository: evalplus/evalplus
Issue: Keep getting IndentationError from the ground truth answers for HUMANEVAL_PLUS_VERSION = "v0.1.9", but works when set to 0.1.8
Root cause: External Dependency Breakage

Kept getting this error on all machines I tested, regardless of installation method:

```
Traceback (most recent call last):
  File "/mnt/data/FastEval/.venv/bin/evalplus.evaluate", line 8, in <module>
    sys.exit(main())
  File "/mnt/data/FastEval/.venv/src/evalplus/evalplus/evaluate.py", line 275, in main
    evaluate_humaneval(args)
  File "/mnt/data/FastEval/.venv/src/evalplus/evalplus/evaluate.py", line 136, in evaluate_humaneval
    expected_output = get_groundtruth(problems, dataset_hash)
  File "/mnt/data/FastEval/.venv/src/evalplus/evalplus/evaluate.py", line 48, in get_groundtruth
    oracle["base"], oracle["base_time"] = trusted_exec(
  File "/mnt/data/FastEval/.venv/src/evalplus/evalplus/gen/util/__init__.py", line 7, in trusted_exec
    exec(code, exec_globals)
  File "<string>", line 39
    ans = 0    
               ^
IndentationError: unindent does not match any outer indentation level
```

Was only able to fix it by changing HUMANEVAL_PLUS_VERSION = "v0.1.9" to  "v0.1.8"

@alpayariyak Thanks for the report. Can you delete your cache at `~/.cache/evalplus` and retry? It should be fixed now.

I also had a test in a fresh environment and I think the issue is sovled. Very much thank you for the report. :D

Thank you very much @ganler!
Repository: evalplus/evalplus
Issue: Where to find the MBPP evaluation metrics?
Root cause: Documentation Deficiency

### EvalPlus version

NA

### Output of running `ls ~/.cache/evalplus`

NA

### Task ID of the programming task

NA

### The original wrong contract

```python
NA
```


### Your proposed new contract

```python
NA
```


### Other context

I am checking out https://evalplus.github.io/leaderboard.html, but found:
![image](https://github.com/evalplus/evalplus/assets/8592144/c33bc193-86aa-468d-b671-6d6105547e18)


Hi. MPBB+ is not yet ready. It is just we considered that when creating the website. However we should be able to release MBPP+ soon, say in a month. :D would like you to come back and check by then. 

@ganler This sounds great! Do you plan to release MBPP code samples as well, like you did for HumanEval? (Do you already have code samples that you could share?)

@mrigankpawagi  We are still at a [stage](https://github.com/evalplus/evalplus/tree/mbpp_merge) to clean things up and perform more testing to make sure the new dataset is reliable enough. We have evaluated chatgpt over it and it seems okay, and would perform full scale evaluation soon.

> Do you plan to release MBPP code samples as well

No doubt, yes. :D
 

https://evalplus.github.io/leaderboard.html @zhimin-z Results are online now. Feel free to check it out! We will also periodically update the leaderboard. Thanks!

@mrigankpawagi Kindly find the attachment of https://github.com/evalplus/evalplus/releases/tag/v0.2.0

@ganler Thanks a lot, this is amazing! I would like to mention again that your code-samples are very helpful for people performing experiments on LLMs (generating samples is a rather time-consuming and resource-heavy task) and by releasing your samples you are playing an important role in accelerating LLM research while making it more accessible. :)
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - <Xwin-LM/XwinCoder-34B>
Root cause: Unimplemented Feature Gap

### Model introduction

Another challenger has entered the ring. A 34B coding model by the Xwin-LM team. Based on CodeLLaMA.


### Model URL

https://huggingface.co/Xwin-LM/XwinCoder-34B

### Data type

bfloat16

### Additional instructions (Optional)

https://github.com/Xwin-LM/Xwin-LM/tree/main/Xwin-Coder. On their github site they say they use vLLM.
https://github.com/Xwin-LM/Xwin-LM/tree/main/Xwin-Coder/HumanEval they say for their reported results, they used temp=0.2, pred=40. not sure if that's useful for you.

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [x] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Check https://evalplus.github.io/leaderboard.html

Thank you for testing Xwin! It performs very well for a CodeLlama fine-tune, but has since been eclipsed by Deepseek and its fine tunes.
Repository: evalplus/evalplus
Issue: Pre-generated MBPP solutions require additional post-processing
Root cause: Documentation Deficiency

Hi, I tried to reproduce the `pass@1` values at [evalplus.github.io](https://evalplus.github.io) but it seems I am unable to get the same values. Here are two examples:

```bash
$ evalplus.evaluate --dataset mbpp --samples codegen-2b_temp_0
Downloading dataset from https://github.com/ganler/release/releases/download/humanevalplus/MbppPlus-v0.1.0.jsonl.gz
100% [............................................................................] 357603 / 357603Computing expected output...
Expected outputs computed in 24.11s
Reading samples...
399it [00:00, 2206.13it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████| 399/399 [00:05<00:00, 77.59it/s]
Base
{'pass@1': 0.02506265664160401}
Base + Extra
{'pass@1': 0.017543859649122806}
```

```bash
$ evalplus.evaluate --dataset mbpp --samples zephyr-7b_temp_0/
Load from ground-truth from /home/mrigankp/.cache/evalplus/ebe95df0d5220f00cedd6f4bddea35e4.pkl
Reading samples...
399it [00:00, 1932.74it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████| 399/399 [00:05<00:00, 67.02it/s]
Base
{'pass@1': 0.02756892230576441}
Base + Extra
{'pass@1': 0.02756892230576441}
```

Am I doing something wrong here? I am able to reproduce HumanEval results though. 

This time I post the raw data without post-processing (which can be forever improved).

To perform post-processing in place:

```shell
python tools/sanitize.py --samples codegen-2b_temp_0 --dataset mbpp --inplace
```

Also see https://github.com/evalplus/evalplus#post-code-sanitizer

Thanks! The values are in much more agreement now:
```
{'pass@1': 0.44611528822055135}
Base + Extra
{'pass@1': 0.3383458646616541}
```
(codegen-2b_temp_0)



You might get a closer result if you relax the time limit a bit by adding `--min-time-limit 0.5 --gt-time-limit-factor 10`.

Specifically I noticed that for `Mbpp/599` there is a very large input such that for an iterative implementation it might take too long to run.

```python
"""
Write a function to find sum and average of first n natural numbers.
assert sum_average(10)==(55, 5.5)
"""
def sum_average(n):
    sum = 0
    average = 0
    for i in range(1, n+1):
        sum += i
        average = sum / n
    return sum, average
```

We cannot say this is logically incorrect, yet it is not quite reasonable to use such a slow implementation. I'm considering some value range over these.


Hmmm, maybe not. All `n` is within the int32 range. 🤔 The problems using over `int32` inputs are actually fine.

What does the `--gt-time-limit-factor` flag do?

See the warning block of the readme: https://github.com/evalplus/evalplus#code-evaluation

I am closing this but feel free to reopen if there is future questions. :)
Repository: evalplus/evalplus
Issue: Compile EvalPlus datasets to compatible formats of original datasets 
Root cause: Unimplemented Feature Gap

Write scripts to convert 

- [x] HumanEval+ to the self-contained form as https://github.com/openai/human-eval/blob/master/data/HumanEval.jsonl.gz
- [x] MBPP+ to the self-contained form as https://github.com/google-research/google-research/tree/master/mbpp

The hope is to allow others to use EvalPlus' dataset without EvalPlus code. There are frameworks supporting HumanEval/MBPP but not HumanEval+/MBPP+ because of engineering challenges and by porting EvalPlus' datasets we hope to allow these frameworks being able to use HumanEval+/MBPP+ out of the box. :) 

This is also partially requested here: https://github.com/evalplus/evalplus/issues/50

+1 this would be great, was just looking for this earlier

#83 has completed porting HumanEval+ for the original HE format.
Repository: evalplus/evalplus
Issue: How many shots are used for evaluation?
GitHub labels: help wanted

Do you use in-context learning the prompt?
How many examples do you use for training? 0-shot, 1-shot, or 2-shot?

Sorry. I don't understand your question. Can you define "shots" more precisely?

If you are asking about code generation during evaluation, the prompt is provided by the benchmark. Those for instruction-tuned model might include a bit more instruction. Check: https://github.com/evalplus/evalplus/blob/master/codegen/model.py

If you are asking about how we prompt to generate test inputs, check https://github.com/evalplus/evalplus/blob/master/evalplus/gen/chatgpt_gen.py   

> If you are asking about code generation during evaluation, the prompt is provided by the benchmark. Those for instruction-tuned model might include a bit more instruction. Check: https://github.com/evalplus/evalplus/blob/master/codegen/model.py
> 
> If you are asking about how we prompt to generate test inputs, check https://github.com/evalplus/evalplus/blob/master/evalplus/gen/chatgpt_gen.py

Thanks for your quick reply. I am more interested in the evaluation. The code is a bit long, where to look at?
How many shots do you use during the benchmark evaluation?

So if you look at the prompts of humaneval and mbpp, we use the exactly same prompt. I don't understand what you mean by x-shots since the prompt is already fixed and you don't have space to tune number of few shots. 

> So if you look at the prompts of humaneval and mbpp, we use the exactly same prompt. I don't understand what you mean by x-shots since the prompt is already fixed and you don't have space to tune number of few shots.

Ok, I do not find any description of what you mention here in the paper. Thus, both are zero-shot prompts.

I'm also curious about the effect of in-context learning in code tasks. Are there any code libraries that implement this feature?

Live code bench uses few shot FYI.  But in practice of program synthesis/completion users won't have few shot examples. 😃 

> Live code bench uses few shot FYI. But in practice of program synthesis/completion users won't have few shot examples. 😃

thx(
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - beowolx/MistralHermes-CodePro-7B-v1
Root cause: Unimplemented Feature Gap

### Model introduction

MistralHermes-CodePro-7B-v1 is a fine-tuned iteration of the renowned [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) model. This version has been meticulously fine-tuned using a dataset comprising over 200,000 code samples from various programming languages. It is specifically tailored to serve as a coding assistant; thus, its utility is optimised for coding-related tasks rather than a broader spectrum of applications.

### Model URL

https://huggingface.co/beowolx/MistralHermes-CodePro-7B-v1

### Data type

bfloat16

### Additional instructions (Optional)

The model is based on [OpenHermes 2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) so it should work, so it can run easily by using `transformers` with the tokeniser of Mistral.  

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Really looking forward to this!

Absolutely! Meanwhile I am doing conference travelling right now and will probably delay this to next week. Thanks for understanding! 

thank you @ganler for taking the time to evaluate the model 🙏🙏

Updated on the leaderboard!

![image](https://github.com/evalplus/evalplus/assets/38074777/6296cca9-1190-40be-846d-cb6b29cdb85f)

Let me know if my use is incorrect: 
https://github.com/evalplus/evalplus/blob/4f9d3dc8a659a7d2f63a37320114b96027d1faff/codegen/model.py#L145
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Top models on can-ai-code-results leaderboard
Root cause: Unimplemented Feature Gap

### Model introduction

These models are top-ranked on another leaderboard. It would be great to compare the scores on both leaderboards.
[uukuguy/speechless-codellama-34b-v2.0](https://huggingface.co/uukuguy/speechless-codellama-34b-v2.0)  
[OpenAssistant/codellama-13b-oasst-sft-v10](https://huggingface.co/OpenAssistant/codellama-13b-oasst-sft-v10)  
[microsoft/phi-2](https://huggingface.co/microsoft/phi-2)

### Model URL

https://huggingface.co/uukuguy/speechless-codellama-34b-v2.0

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Hi I have just updated Phi-2 and the rest models seem to be filed in other issues/requests. So I am going to close this and follow things on other requests. Thanks!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - openchat-3.5-1210
GitHub labels: model eval

### Model introduction

claims high human eval (68.9) for a "generalist" small 7B model

@article{wang2023openchat,
  title={OpenChat: Advancing Open-source Language Models with Mixed-Quality Data},
  author={Wang, Guan and Cheng, Sijie and Zhan, Xianyuan and Li, Xiangang and Song, Sen and Liu, Yang},
  journal={arXiv preprint arXiv:2309.11235},
  year={2023}
}

### Model URL

https://huggingface.co/openchat/openchat-3.5-1210

### Data type

float16

### Additional instructions (Optional)

"The server is optimized for high-throughput deployment using [vLLM](https://github.com/vllm-project/vllm) and can run on a consumer GPU with 24GB RAM."
- from the model page
https://huggingface.co/openchat/openchat-3.5-1210

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Thanks! I am closing as a dup of #61 and will follow up there!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - OpenChat 3.5 1210
Root cause: Documentation Deficiency

### Model introduction

OpenChat is an innovative library of open-source language models, fine-tuned with [C-RLFT](https://arxiv.org/pdf/2309.11235.pdf) - a strategy inspired by offline reinforcement learning.

### Model URL

https://huggingface.co/openchat/openchat-3.5-1210

### Data type

bfloat16

### Additional instructions (Optional)

We have vLLM accelerated code to run the model and convert to EvalPlus format. See "Reproducing Benchmarks" [here](https://github.com/imoneoi/openchat#%EF%B8%8F-benchmarks)

```
python -m ochat.evaluation.run_eval --condition "GPT4 Correct" --model openchat/openchat-3.5-1210 --eval_sets coding
python ochat/evaluation/convert_to_evalplus.py
```

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Hi @imoneoi may I request a short self-contained example for using OpenChat? 

Hi @ganler We've detailed instructions to use openchat in our repo https://github.com/imoneoi/openchat

Here are self-contained commands to run EvalPlus with openchat. We're glad to help if you have any questions.

```bash
# Install environment
conda create -y --name openchat python=3.11
conda activate openchat

pip3 install ochat

# Clone repo and run benchmarks
git clone https://github.com/imoneoi/openchat
cd openchat

python -m ochat.evaluation.run_eval --condition "GPT4 Correct" --model openchat/openchat-3.5-1210 --eval_sets coding
python ochat/evaluation/convert_to_evalplus.py
```

I understand but it would be nicer for me to understand your desired prompt format and run it over a setting where other models have been evaluated. In this way we can reduce the noise caused by configuration inconsistency. :D

@ganler You can try openchat server, which exposes a local HTTP interface compatible with the OpenAI API.

Great I have updated the results by using a more recent openchat version: https://huggingface.co/openchat/openchat-3.5-0106

Sorry for the delay and feel free to check it out at https://evalplus.github.io/leaderboard.html
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <xDAN-AI/xDAN-L1-Chat-dpo-qlora-v1>
Root cause: Unimplemented Feature Gap

### Model introduction

The sota MT-Bench model at 7b.

### Model URL

https://huggingface.co/xDAN-AI/xDAN-L1-Chat-dpo-qlora-v1

### Data type

float16

### Additional instructions (Optional)

alpaca type | ask for confirmation review
Defined System Prompt is better when you make a test.

Example:
`You are a helpful assistant named DAN and created by xDAN-AI . You are an expert in worldly knowledge, skilled in employing a probing questioning strategy, carefully considering each step before providing answers.`






contact with me at discord , shootime (@cryptossssun)

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Thanks for the request. The results are updated over https://evalplus.github.io/leaderboard.html