zhiminy/EvalEng huggingface dataset:
Repository: erikbern/ann-benchmarks
Issue: Run ONNG for nytimes
Root cause: Validation Gap

@maumueller 
I resolved the problem (#93) of the results due to invalid zero vectors in the queries. Sorry for troubling you, but could you rebuild NGT and recreate results for the nytimes dataset?
Thank you in advance.

@masajiro what's preventing you from doing it yourself?

@erikbern 
Sorry for multiple similar comments, because Github had big system problems yesterday.

Since I have already confirmed that the updated ONNG clearly improved its results for the nytimes dataset on my local environment, I would like you to update only ONNG results in the [figure](https://github.com/erikbern/ann-benchmarks#nytimes-256-angular).
Do you mean that you can replace the figures with figures that I made on my local environment?



@masajiro we generally re-run the benchmarks every 3-4 months or so. maybe it's time to do it again. i think it would be a lot of work to do it ad-hoc any time there's library updates.

you can also go ahead and do it yourself if you want to update the repo – can run everything on a c5.4xlarge in AWS

@erikbern Thank you for your reply.
I will consider updating the repo as well.

@masajiro @erikbern 
If there are zero vectors in the test then it is a test bed issue which affects all algorithms. 
So, I think, removing the zero vectors from the datasets is the right solution.

@yurymalkov I totally agree with you.
Repository: evalplus/evalplus
Issue: [DEV][OSS] Repository Sanitizing
Root cause: Unimplemented Feature Gap

- [x] DCE.
- [x] Structuralize enhanced dataset (i.e., HumanEvalPlus for now)
- [x] Structuralize source codes to four components
       - LLM Codegen `codegen`
       - Making of enhanced dataset  `evalplus.dataset`
       - Result analysis and evaluation `evalplus.eval` 
       - Other useful developer tools `tools`
- [x] Dataset versioning
- [x] README and documentation
- [x] Release `v0.1.0`
- [ ] Upload HuggingFace
- [x] Open-source

List some new tasks:

- [x] Make the dependency more finegrained (split `evalplus` dependency and others)


Some follow-up TODOs (working in progress) are:
- [x] refactoring `evalplus.eval` to be compatible with both prior folder-based sample generation as well as the `samples.jsonl` style following humaneval.
- [x] making `evalplus` folder a pypi package.

One last thing:

- [x] dockerize the evaluator for sandboxing
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - TheBloke/Mixtral-8x7B-v0.1-GPTQ
Root cause: Unimplemented Feature Gap

### Model introduction

Model creator: [Mistral AI_](https://huggingface.co/mistralai)

### Model URL

https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GPTQ

### Data type

float16

### Additional instructions (Optional)

VLLm acceleration will be great.

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: [Bug] Error when running generate.py with a file path containing uppercase letters
Root cause: Validation Gap

While trying to reproduce with a model downloaded to local, I found a bug where codegen/generate.py does not work if the model path contains uppercase letters.

This is confirmed to be caused by the line of code below.

https://github.com/evalplus/evalplus/blob/87e0b2e6c4d6576d99b5474e4d279eaa744e2831/codegen/generate.py#L150

Changing this code to not perform `lower()` that path if it exists, as shown below, seems to work.

```diff
- args.model = args.model.lower()
+ if not os.path.exists(args.model):
+     args.model = args.model.lower()
```

I want to create a pull request with the above diff and fix it, is it okay if I create a PR and request it?

The semantic to --model is not necessarily the entry points in huggingface model hub. To fully resolve this I'd have to make a deeper refactoring to the codebase. It is actually planned but not going to quickly happen. 

I suggest you checker earlier commits on how I support new models and you can follow accordingly to run the generation. 

Meanwhile big-code-harness supports HumanEval+ and MBPP+ as well and it maps to local folders as well as huggingface entry points. That should be the easiest way to bootstrap the evaluation if you don't have a special instruction template. 

I'm closing this issue as it is not a bug but we will try to get you updated when we make EvalPlus more huggingface native.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - uukuguy/speechless-sparsetral-16x7b-MoE
GitHub labels: model eval

### Model introduction

<p><h1> speechless-sparsetral-16x7b-MoE  </h1></p>

speechless-sparsetral-16x7b-MoE is the MoE upgraded version of [speechless-code-mistral-7b-v1.0](https://huggingface.co/uukuguy/speechless-code-mistral-7b-v1.0). The MoE fine-tuning adopts [Parameter-Efficient Sparsity Crafting (PESC)](https://arxiv.org/abs/2401.02731), which is an efficient fine-tuning architecture that uses LoRA modules as expert models, similar to the concept of [multi-loras](https://github.com/uukuguy/multi_loras). 

Specifically, Mistral-7B-0.1 is used as the base model, with 16 experts and 4 expert outputs selected for inference. The fine-tuning dataset includes codefuse-ai/Evol-Instruction-66k to enhance the model's code generation ability. The specific datasets are as follows:

- jondurbin/airoboros-2.2: Filter categories related to coding, reasoning and planning. 23,462 samples.
- Open-Orca/OpenOrca: Filter the 'cot' category in 1M GPT4 dataset. 74,440 samples.
- garage-bAInd/Open-Platypus: 100%, 24,926 samples.
- WizardLM/WizardLM_evol_instruct_V2_196k: Coding coversation part. 30,185 samples
- TokenBender/python_eval_instruct_51k: “python” in output .40,309 samples
- Spider: 8,659 samples
- codefuse-ai/Evol-Instruction-66k: 100%, 66,862 samples

## Alpaca Prompt Format

```
### Instruction:
<instruction>
### Response:
```

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name_or_path="uukuguy/speechless-sparsetral-16x7b-MoE"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto", trust_remote_code=True).eval()

system = ""Below is an instruction that describes a task.\nWrite a response that appropriately completes the request.\n\n""
prompt = f"{system}\n\n### Instruction:\n{instruction}\n\n### Response:"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
pred = model.generate(**inputs, max_length=4096, do_sample=True, top_k=50, top_p=0.99, temperature=0.9, num_return_sequences=1)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
```

### Model URL

https://huggingface.co/uukuguy/speechless-sparsetral-16x7b-MoE

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - speechless-thoughts-mistral-7b
GitHub labels: model eval

### Model introduction

<p><h1> speechless-thoughts-mistral-7b  </h1></p>

[code](https://github.com/uukuguy/multi_loras)

speechless-thoughts-mistral-7b is fine-tuned as a baseline of the [speechless-sparsetral-16x7b-MoE](https://huggingface.co/uukuguy/speechless-sparsetral-16x7b-MoE). 

The specific datasets (speechless-thoughts-252k) are as follows:

- jondurbin/airoboros-2.2: Filter categories related to coding, reasoning and planning. 23,462 samples.
- Open-Orca/OpenOrca: Filter the 'cot' category in 1M GPT4 dataset. 74,440 samples.
- garage-bAInd/Open-Platypus: 100%, 24,926 samples.
- WizardLM/WizardLM_evol_instruct_V2_196k: Coding coversation part. 30,185 samples
- TokenBender/python_eval_instruct_51k: “python” in output .40,309 samples
- Spider: 8,659 samples
- codefuse-ai/Evol-Instruction-66k: 100%, 66,862 samples

## Alpaca Prompt Format

```
### Instruction:
<instruction>
### Response:
```

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name_or_path="uukuguy/speechless-sparsetral-16x7b-MoE"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
  model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto", trust_remote_code=True).eval()

  system = ""Below is an instruction that describes a task.\nWrite a response that appropriately completes the request.\n\n""
  prompt = f"{system}\n\n### Instruction:\n{instruction}\n\n### Response:"

  inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
  pred = model.generate(**inputs, max_length=4096, do_sample=True, top_k=50, top_p=0.99, temperature=0.9, num_return_sequences=1)
  print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
```

### Model URL

https://huggingface.co/uukuguy/speechless-thoughts-mistral-7b

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <tyson0420/stack_llama_fil_ai>
GitHub labels: model eval

### Model introduction

based on llama-2 model, and finetuning by stackexchange dataset and implement DPO which dataset filtered by coding tasks

### Model URL

https://huggingface.co/tyson0420/stack_llama_fil_ai

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Code-290k-13B
GitHub labels: model eval

### Model introduction

Large Language Models (LLMs) are good with code generations. Sometimes they do make mistakes in code generation. How about if they can give detailed explanation along with the code. This is what I have tried over here. The base Llama-2 model was used for training purpose. It is trained on around 290000 set of codes. Each set having 2 conversations. Along with Python, Java, JavaScript, GO, C++, Rust, Ruby, Sql, MySql, R, Julia, Haskell, etc. code with detailed explanation is used for training purpose.
Entire dataset was trained on 4 x A100 80GB. For 3 epoch, training took 165 hours. This was trained on Llama-2 by Meta. This is a full fine tuned model.
This conversation is in Vicuna/ShareGPT format.

### Model URL

https://huggingface.co/ajibawa-2023/Code-290k-13B

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Thank you very much for evaluating my earlier models.
Repository: evalplus/evalplus
Issue: What are "extreme inputs"?

[This commit](https://github.com/evalplus/evalplus/commit/c08e5e16a67e7d1c98e412bfc4d55d2d80ac2de9) adds a flag to omit "extreme inputs" on `humaneval`. What does this mean?

Thanks.

Thanks for the question.

There are two potential reasons to falsify a solution:
1. Producing wrong output (apparently)
2. Timeout 

EvalPlus generates lots of new inputs and some inputs can be very performance-exercising for some correct but slow solutions -- failing to terminate within the timeout (See the ["warning" block](https://github.com/evalplus/evalplus?tab=readme-ov-file#code-evaluation)). For an example of `HumanEval/63`, the LLM can produce a very inefficient recursive solution:

```python3
def fibfib(n: int):
    """The FibFib number sequence is a sequence similar to the Fibbonacci sequnece that's defined as follows:
    fibfib(0) == 0
    fibfib(1) == 0
    fibfib(2) == 1
    fibfib(n) == fibfib(n-1) + fibfib(n-2) + fibfib(n-3).
    Please write a function to efficiently compute the n-th element of the fibfib number sequence.
    >>> fibfib(1)
    0
    >>> fibfib(5)
    4
    >>> fibfib(8)
    24
    """
    if n == 0:
        return 0
    elif n == 1:
        return 0
    elif n == 2:
        return 1
    else:
        return fibfib(n-1) + fibfib(n-2) + fibfib(n-3)
``` 

This solution is correct.
But because it is very inefficient it fails to complete some larger inputs within a reasonable amount of timeout (say a test can take minutes and hours to run). 

```
====================
time out inputs:
[57]
[56]
[58]
[26]
[60]
[59]
[27]
[28]
[61]
[62]
[29]
[30]
[31]
[32]
[48]
[47]
[46]
[55]
[49]
[50]
[51]
[52]
[53]
[54]
[33]
[73]
[34]
[63]
[45]
[72]
[98]
[100]
[97]
[71]
[43]
[35]
[44]
[99]
[102]
====================
success inputs:
[20]
[0]
[15]
[23]
[21]
[25]
[1]
[14]
[13]
[19]
[22]
[24]
[18]
[False]
[True]
[12]
[11]
[2]
[4]
[16]
[3]
[17]
[10]
[9]
====================
```

As such, we did a study to catch "extreme inputs", i.e., inputs that can make the possible slowest solutions hit timeout (we set 20s timeout for one test input). So by using the flag, one can exclude such "extreme inputs" for HumanEval+ and MBPP+.

Overall as you can see from the commit, we found extreme inputs in 7 HumanEval tasks and 7 MBPP tasks. It is not expected to hugely impact the scores in theory because the number of tasks with "extreme" inputs is small and it is not easy/frequent for an LLM to generate the slowest tier of solutions for most of them.
However, because we conducted the study, we enabled this "nice-to-have" option for interested users. :)




Great - many thanks for the clarification.
Repository: evalplus/evalplus
Issue: google/gemma-7b-it
GitHub labels: model eval

actionable

### Model introduction

This model card corresponds to the 7B instruct version of the Gemma model. You can also visit the model card of the [2B base model](https://huggingface.co/google/gemma-2b), [7B base model](https://huggingface.co/google/gemma-7b), and [2B instruct model](https://huggingface.co/google/gemma-2b-it).

### Model URL

https://huggingface.co/google/gemma-7b-it

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

## gemma-2b-it
HE+:
<img width="757" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/6e6c36ce-7a62-49ef-abc4-594a8c438088">


MBPP+:
<img width="755" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/476e8554-2321-41f9-b775-2e1a303369cd">





## gemma-2b
HE+:
<img width="750" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/07c45948-e6ea-444f-8ace-0cc510cf0514">

MBPP+
<img width="746" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/b2f40858-8a62-4481-80ba-f4aa9f7d093a">


## gemma-7b-it
HE+:
<img width="753" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/405a6270-b70c-494e-9cb8-398ca7710834">

MBPP+:
<img width="749" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/254e4e46-6138-48e7-9659-0df3fc9a8806">



## gemma-7b
HE+:
<img width="753" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/027dfcad-2648-4851-a0c3-8abf85d3abe5">


MBPP+:
<img width="750" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/58fd14cb-e984-446d-9a6d-b8075d670aa0">
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - OpenCodeInterpreter-DS-6.7B
GitHub labels: model eval

### Model introduction

This is the current SOTA <7B code model (claims to match GPT4)

### Model URL

https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-6.7B

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Thanks. Let's merge the threads to #109
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - OpenCodeInterpreter-CL-13B
Root cause: Unimplemented Feature Gap

### Model introduction

This is the current SOTA <13B code model (claims to match GPT4)

### Model URL

https://huggingface.co/m-a-p/OpenCodeInterpreter-CL-13B

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

I am going to evaluate the 6.7B and 33B DS versions given the launch of #120 

## 33B results

HE+:

![image](https://github.com/evalplus/evalplus/assets/38074777/604bb0a4-76b1-45ab-b77e-4f2f6c999e5a)

MBPP+:

![image](https://github.com/evalplus/evalplus/assets/38074777/48ab3e25-405d-4c24-9438-ffa3dbd07eae)


## 6.7B results

HE+:

![image](https://github.com/evalplus/evalplus/assets/38074777/dcf6754f-ef53-4dce-b002-f07a2f4f517e)

MBPP+:

![image](https://github.com/evalplus/evalplus/assets/38074777/f3948b85-901a-4c20-b5c1-30d318b7d570)


Seems using vLLM may not present the best performance. Am re-evaluating the models using the raw Huggingface.  d07182505d99aed5a0b79841a3f05fb3b4d17f3f
Repository: evalplus/evalplus
Issue: [BUG] This system prompt word will cause gpt-4 to generate json with a high probability instead of normal text.
Root cause: Validation Gap

https://github.com/evalplus/evalplus/blob/b92a382c17968abd5ed2508e143bf411cf554146/evalplus/gen/util/api_request.py#L23C1-L24C1

Notice that this prompt word is intended to be used with json_object, but it seems that the switching condition is missing.

## response of HumanEval_0 by gpt-4

Here is the JSON representation of the problem:

```json
{
  "description": "Check if in given list of numbers, are any two numbers closer to each other than given threshold.",
  "examples": [
    {
      "input": {
        "numbers": [1.0, 2.0, 3.0],
        "threshold": 0.5
      },
      "output": false
    },
    {
      "input": {
        "numbers": [1.0, 2.8, 3.0, 4.0, 5.0, 2.0],
        "threshold": 0.3
      },
      "output": true
    }
  ],
  "function": {
    "name": "has_close_elements",
    "parameters": [
      {
        "name": "numbers",
        "type": "List[float]"
      },
      {
        "name": "threshold",
        "type": "float"
      }
    ],
    "return_type": "bool"
  }
}
```

@geekan Thanks for the bug report. Can you check if 49bec7c14a75cf617e4474d176ecaba3f4c66e57 resolves the issue?

One more hotfix at 26fd3027d33c2f526ff0381ab9b2f45320153548

Awesome, looks like it's been fixed, thanks a lot @ganler
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - Code-290k-6.7B-Instruct
GitHub labels: model eval

### Model introduction

This model is trained on DeepSeek-Coder-6.7B-Instruct. I have used my existing dataset Code-290k-ShareGPT for training purpose. It is trained on around 290000 set of codes. Along with Python, Java, JavaScript, GO, C++, Rust, Ruby, Sql, MySql, R, Julia, Haskell, etc. code with detailed explanation is used for training purpose. This model utilizes Alpaca format. Besides code generation it will also give you explanation.

### Model URL

https://huggingface.co/ajibawa-2023/Code-290k-6.7B-Instruct

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

HE+:
<img width="751" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/fe76e628-165f-41da-992f-47c426c2f9de">

MBPP+:
<img width="753" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/5abd6352-eabd-497a-9de2-f9cbe351daf3">


Thank you very much for evaluating this model!
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - TokenBender/starcoder2_15B_OCI
GitHub labels: model eval

actionable

### Model introduction

The model is created by TokenBender and is a SFT on starcoder2, used for code generation purposes.

### Model URL

TokenBender/starcoder2_15B_OCI

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

HE+:
<img width="748" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/e335484e-4912-4626-b5b9-1d44bb5dfd63">

MBPP+:
<img width="746" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/d1fdb101-d708-44ee-b0a7-1c26243806ab">
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - OpenHermes-2.5-Code-290k-13B
GitHub labels: model eval

### Model introduction

OpenHermes-2.5-Code-290k-13B is a state of the art Llama-2 Fine-tune, which is trained on additional code dataset. This model is trained on my existing dataset OpenHermes-2.5-Code-290k. This dataset is amalgamation of two datasets. I have used OpenHermes-2.5 a super quality dataset made available by teknium. Other dataset is my own Code-290k-ShareGPT. Dataset is in Vicuna/ShareGPT format. There are around 1.29 million set of conversations. I have cleaned the dataset provided by Teknium and removed metadata such as "source" & "category" etc. This dataset has primarily synthetically generated instruction and chat samples.

This model has enhanced coding capabilities besides other capabilities such as Blogging, story generation, Q&A and many more.

### Model URL

https://huggingface.co/ajibawa-2023/OpenHermes-2.5-Code-290k-13B

### Data type

bfloat16

### Additional instructions (Optional)

I have used ShareGPT/Vicuna format 

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

HE+:
<img width="754" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/e808c46d-f087-41a5-a4ae-b1584c1f814f">

MBPP+:
<img width="756" alt="image" src="https://github.com/evalplus/evalplus/assets/79293503/d2ee981d-771f-47ee-8471-9313c4913372">


Thank you very much for evaluating this model!
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] -  CodeGemma-2b
Root cause: Unimplemented Feature Gap

### Model introduction

We've fine-tuned Gemma-2b with an additional 0.7 billion high-quality, code-related tokens for 3 epochs. This model operates using the Alpaca instruction format (excluding the system prompt).

### Model URL

https://huggingface.co/TechxGenus/CodeGemma-2b

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.