zhiminy/EvalEng huggingface dataset:
Repository: evalplus/evalplus
Issue: [Feature request] Better answer matching
Root cause: Unimplemented Feature Gap

Consider using a universal prompt template and answer-matching method for all open-source and API-based chat models. Currently, each open-source chat model uses a slightly different prompt template and is forced to start with a code block, while OpenAI models use JSON or direct output.

Below we provide an answer matching method using Python `ast`. We tested `gpt-3.5-turbo`, `gpt-4`, `gpt-4-turbo` and many open-source chat models, and it can find answers in almost all cases.

The matching method covers the following scenarios:

- Some models will output chain-of-thought before the code block
- Models may output the full function or only the completion
- Some completions start with the function definition directly, ignoring the `import xxx`

```python
def coding_humaneval_match_answer(task_data, response):
    # Matching utilities
    def _function_exists(code, func_name):
        tree = ast.parse(code)
        for node in ast.walk(tree):
            if isinstance(node, ast.FunctionDef) and node.name == func_name:
                return True

        return False

    def _try_match(content, prefix, entrypoint):
        # All markdown code blocks, as well as raw
        code_blocks = [m[1] for m in re.findall(r"(\`{3}.*?\n+)([\s\S]*?)(\n+\`{3})", content)] \
                    + [content]

        for block in code_blocks:
            # Check syntax
            try:
                code_completion = prefix + block
                if _function_exists(code_completion, entrypoint):
                    return code_completion
            except SyntaxError:
                pass

    # Try match with include prefix
    humaneval_task = task_data["_metadata"]
    include_prefix = humaneval_task['prompt'].split('def')[0].strip() + "\n\n"

    result = _try_match(response, include_prefix, humaneval_task["entry_point"])
    if result: 
        return True, {"task_id": humaneval_task["task_id"], "completion": result}

    # If fail then match with function signature
    result = _try_match(response, humaneval_task["prompt"], humaneval_task["entry_point"])
    if result: 
        return True, {"task_id": humaneval_task["task_id"], "completion": result}

    return False, {"task_id": humaneval_task["task_id"], "completion": response}
```

The prompt template is as below:

<p>
Provide a correct completion to the following code snippet. <br />
```python <br />
{task} <br />
``` <br />
</p>

Thanks! We are working on a tree-sitter based version and will release it soon!

Closed as we now use tree-sitter to match output code.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - deepseek-coder-7b-instruct-v1.5
GitHub labels: model eval

### Model introduction

Deepseek-Coder-7B-Instruct-v1.5 is continue pre-trained from Deepseek-LLM 7B on 2T tokens by employing a window size of 4K and next token prediction objective, and then fine-tuned on 2B tokens of instruction data.

### Model URL

https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

No

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Check https://evalplus.github.io/leaderboard.html
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - Mistral_Code
Root cause: Validation Gap

### Model introduction

The model is finetuned version of mistral model for code.

### Model URL

hiteshsom/mistral_finetuned_code

### Data type

float16

### Additional instructions (Optional)

For generation I used to supply the prompt (prompt + the actual question) like this: 

'Below is an instruction that describes a task. Write a response that appropriately completes the request.  Create a function to calculate the sum of a sequence of integers. Input: [1, 2, 3, 4, 5]'

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

It seems the `config.json` file is missing. Therefore, I cannot use your model at my end.
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - <CodeLlama-70B-Instruct>
Root cause: Unimplemented Feature Gap

### Model introduction

codellama-70B-instruction

### Model URL

https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf

### Data type

float16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

Hi, I just updated the scores at https://evalplus.github.io/leaderboard.html for the CodeLlama 70B series including base and instruction-tuned versions. Thanks for the nice work. :)

Meanwhile, I found an interesting result where 70b seems to be worse than 34b on MBPP base while 70 slightly outperforms 34b using MBPP+.

![image](https://github.com/evalplus/evalplus/assets/38074777/ffb9eb08-f61d-4f38-81f2-ed31f7760bb0)


Just noticed that the 70b model is using a different prompt template than the smaller ones: https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf#chat-prompt

I am reopening this request and will re-evaluate the 70b model on the new prompt format.

The results for 70b instruct have been updated. Using the recommended prompts leads to a significant improvement:

https://github.com/evalplus/evalplus.github.io/commit/7e731c54cbdc1def687650efbb06b805ac488360
Repository: evalplus/evalplus
Issue: [DEV] Interage code sanitizer into evalplus main package and the quick-start example
Root cause: Documentation Deficiency

Most ppl when using evalplus might not notice doing post-code sanitization and turn out to get bad scores.

Therefore we shall integrate code post-processing into the quick-start by porting the sanitizers into the standard evalplus module.

Hey @ganler , I was just wondering, were you able to update the evalplus module with the sanitizers? 

Thanks for bring this up. Current in v0.2.1, `sanitize.py` is a module included in `evalplus`: 

https://github.com/evalplus/evalplus/blob/ae747121ca8e4bf52b432f75da15cf6ab8dee89a/evalplus/sanitize.py#L57

But you are right, I think I should make it better by porting it to a commandline tool for user convenience. Will try to build as soon I as I can.

@Shlok-crypto Please check https://github.com/evalplus/evalplus?tab=readme-ov-file#code-post-processing

New wheel released at https://pypi.org/project/evalplus/0.2.2/
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - WhiteRabbitNeo-33B-v1.0
Root cause: Unimplemented Feature Gap

### Model introduction

Hey there!
WhiteRabbitNeo is a model series for cybersecurity. However, it also works as a general purpose coding assistant, using the below system command:
```"You are a helpful AI that can code. Answer with code."```

### Model URL

https://huggingface.co/WhiteRabbitNeo/WhiteRabbitNeo-33B-v1

### Data type

float16

### Additional instructions (Optional)

Yes, vLLM can run this in float16, on a 80GB VRAM GPU (A100/H100).

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

`system_prompt = "You code like a superhero!"` produces even better results on HumanEval. Keen to see the results on your leaderboard!

Hey there! Is there any update on this? Keen to see results on your leaderboard!

Migel Tissera

Hi @mtisz our lab has limited GPU resources for running 33b models and I am still waiting for my turn. 😂 But no worries, I run your models as soon as an A100 is available.  

Thank you @ganler ! Appreciate it!

![image](https://github.com/evalplus/evalplus/assets/38074777/7c8f861c-ebf2-4a1a-ac84-8dc1a7dc3e21)

Check out the results on the leaderboard! 

Thank you!
Repository: evalplus/evalplus
Issue: [Bug] Sanitizer mis-tranformed correct code to empty
Root cause: Validation Gap

It seems that using the sanitizer script may cause some issues.
For example:
After doing post-code sanitization, my solution for HumanEval/106 input will become empty.
Solution:
```
def f(n):
    def factorial(i):
        if i == 0:
            return 1
        else:
            return i * factorial(i-1)

    result = []
    for i in range(1, n+1):
        if i % 2 == 0:
            result.append(factorial(i))
        else:
            result.append(sum(range(1, i+1)))
    return result

# Test the function
print(f(5))
```

After sanitization:
{"task_id": "HumanEval/106", "solution": ""}

If i want to update my model on your leaderboard, can I refrain from using the sanitizer script and instead use my own script to extract code snippets from the model's output?

Hi @AnitaLiu98 thanks for the issue. Theoretically, it is expected for the output code snippet to be empty if the entry point is not found.

For your case, it is indeed that the entry point is `f` -- so that could be a bug in the sanitizer which I will fix and update the scores globally. :) 

Hi @AnitaLiu98 I believe the issue you mentioned is fixed. That seems to only impact the pass-fail of one task. 

Feel free to reopen if it does not fully address your concern. I will update the scores soon after re-evaluation. Would be nice if you can tell me which model is yours on the leaderboard. 

@ganler  The issue has been resolved. Thank you very much!
Repository: evalplus/evalplus
Issue: 🤗 [REQUEST] - uukuguy/speechless-mistral-hermes-code-7b
Root cause: Unimplemented Feature Gap

### Model introduction

https://huggingface.co/uukuguy/speechless-mistral-hermes-code-7b

## speechless-mistral-hermes-code-7b
Code: https://github.com/uukuguy/speechless

Use the following dataset to fine-tune mistralai/Mistral-7B-v0.1 in order to improve the model's reasoning and planning abilities.

Total 986k samples.

- teknium/OpenHermes-2.5
- TokenBender/python_eval_instruct_51k
- Spider
- codefuse-ai/Evol-instruction-66k

### Model URL

https://huggingface.co/uukuguy/speechless-mistral-hermes-code-7b

### Data type

bfloat16

### Additional instructions (Optional)

_No response_

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: [Dev] Simplifying the format of `*eval_results.json`
Root cause: Unimplemented Feature Gap

So far, the schema for `*eval_results.json` is not easy to understand and use. 

https://github.com/evalplus/evalplus/blob/3d1aa126253646b02dc33ef97c99e33c2585803f/evalplus/evaluate.py#L221

I want to propose a better-engineered schema, where for each `task_id` we have a line of JSON dictionary:

1. `task_id : str`
2. `solution : str`: the actual code being executed
3. `base_status : str`: $\in\\{pass, fail, timeout\\}$
4. `plus_status : str`: $\in\\{pass, fail, timeout\\}$
5. `base_fail_tests : Optional[List[Any]]`: the "base" test inputs that make the solution fail; None if `base_status == "pass"`
6. `plus_fail_tests : Optional[List[Any]]`: the "plus"...; None if `plus_status == "pass"`

In this way hopefully, model developers can more quickly identify what's wrong and debug/visualize them.


Could we use a list of JSON lines is this format, when it has multiple solutions for each "task_id".
Like
`[{task_id, solution, base_status, plus_status, base_fail_tests, plus_fail_tests}, ...]`

Thanks for the proposal. To support such a map-like structure I think we can consider:
Each line is a dictionary:
* Key: `task_id`
* Value: A list of solution results, each of which follows what I proposed in https://github.com/evalplus/evalplus/issues/95#issue-2125209303
Repository: evalplus/evalplus
Issue: 💡 [REQUEST] - IF-CL-MC-34B
Root cause: Unimplemented Feature Gap

### Model introduction

The model is fine-tuned with three datasets: 

- [evol-codealpaca-v1](https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1)
- [Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K)
and the dateset described in our paper:
- Instruction Fusion: Advancing Prompt Evolution through Hybridization (https://arxiv.org/abs/2312.15692)

The performance of this model is discussed in Appendix of the paper.

We have uploaded the model to huggingface, and preparing the model card.

### Model URL

https://huggingface.co/Pasta009/IF-CL-MC-34B

### Data type

bfloat16

### Additional instructions (Optional)

we use the following repo to fine-tune this model 
https://github.com/hiyouga/LLaMA-Efficient-Tuning/tree/main

During the inference, there is no system input, 
 and 
 are added at the begining and the end of each prompt:

# python code to make prompt
B_INST, E_INST = "
", "
"
prefix = "Please complete the following Python code without providing any additional tasks such as testing or explanations\n "
prompt = B_INST + prefix + prompt + E_INST

### Author

Yes

### Security

- [X] I confirm that the model is safe to run which does not contain any malicious code or content.

### Integrity

- [X] I confirm that the model comes from unique and original work and does not contain any plagiarism.
Repository: evalplus/evalplus
Issue: [Bug] Exporting `eval_results.jsonl`
Root cause: Validation Gap

Bugs introduced in #96:

1. MBPP output cannot be serialized
2. `pre-commit` not applied/skipped

The evaluation script is not working for MBPP now and I'm working on it. Also cc: @soryxie 

Thanks for the report from @natedingyifeng
Repository: evalplus/evalplus
Issue: [REQUEST] - Processing Generated GPT-4 code
Root cause: Validation Gap

I am currently using the provided **generation script** with the following command:
``python generate.py --model gpt-4-1106-preview --bs 1 --temperature 0 --root reproduce_samples --n_samples 1 --greedy  --dataset humaneval' `

Is this all that was used to generate the samples you provide? There are several cases of **duplicated function definitions**, which arises from GPT-4 redefining the function (instead of just finishing it). Is there any guidance regarding the post-processing steps taken by the team, before eval, when the provided generation script is used?

Thanks!

I think that is all you need for that. I have not been testing that for a while. But can you demonstrate the redef issue?

There are many examples. One example is HumanEval 65 - this is the file at /reproduce_samples/humaneval/gpt-4-1106-preview_temp_0.0/HumanEval_65 from command ``python generate.py --model gpt-4-1106-preview --bs 1 --temperature 0 --root reproduce_samples --n_samples 1 --greedy --dataset humaneval' `


```
def circular_shift(x, shift):
    """Circular shift the digits of the integer x, shift the digits right by shift
    and return the result as a string.
    If shift > number of digits, return digits reversed.
    >>> circular_shift(12, 1)
    "21"
    >>> circular_shift(12, 2)
    "12"
    """

def circular_shift(x, shift):
    """Circular shift the digits of the integer x, shift the digits right by shift
    and return the result as a string.
    If shift > number of digits, return digits reversed.
    >>> circular_shift(12, 1)
    "21"
    >>> circular_shift(12, 2)
    "12"
    """
    x_str = str(x)
    shift %= len(x_str)
    if shift == 0:
        return x_str
    return x_str[-shift:] + x_str[:-shift]
```

First of all, this does not affect the evaluation results as the first `function + body` is semantic/syntactic valid and is then overloaded by a valid redefinition. 

After a close look it is caused by this line: 

https://github.com/evalplus/evalplus/blob/87e0b2e6c4d6576d99b5474e4d279eaa744e2831/codegen/model.py#L535

IIRC, I added the prompt prefix to cover cases when GPT-4 only produces the body without repeating the prompt.
Repository: evidentlyai/evidently
Issue: Chore Request: Bump to scipy >= 1.6
Root cause: Architectural Constraint

Is there any reason to stay on ~1.5?
https://github.com/evidentlyai/evidently/blob/94e4c84ff3c6147d7a5364650fa20e01bb4a7b69/requirements.txt#L8

Scipy 1.6 added an option to compute one-sided p-values, which is valuable for a lot of companies. Because of the restriction above, I cannot just add evidently as a dev dependency using `poetry add -D`, but would love to!

Hi @aptlin,
Probably we can relax version constrain to `scipy>=1.5.4` so you can use other version.
We will check it, and fix it in next release.


Hi again,
We merged changes for this in new release 0.1.39.dev0. You can now try it out!

Hi @aptlin, this is working now.
Thanks again for bringing this up!
Repository: evidentlyai/evidently
Issue: ClassificationPreset bug when data is multiclassifcation and true_value doesn't contain all possible values
Root cause: Validation Gap

There is an issue with the ClassificationPreset when the data is multiclassifcation and the true_value doesn't contain all possible values. Attached the data to reproduce it and the code and the error message:
[bquxjob_5257e9c4_18db13d7387.csv](https://github.com/evidentlyai/evidently/files/14359081/bquxjob_5257e9c4_18db13d7387.csv)


```
column_mapping = ColumnMapping()
column_mapping.target = 'LABEL'
column_mapping.prediction = ['RENEWED', 'CHURN', 'FAILED']

classification_performance_report = Report(metrics=[
    ClassificationPreset(probas_threshold=None),
])

classification_performance_report.run(reference_data=None, current_data=db_data, column_mapping=column_mapping)
classification_performance_report.show(mode='inline')
```

The error is:
`ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.`
Repository: evidentlyai/evidently
Issue: NDCG Error for scoring 
Root cause: Validation Gap

I have an issue when calculating the NDCG for an scoring model. Attached the data to reproduce it

My code is:

```
report_test = Report(metrics=[
    PrecisionTopKMetric(k=5),
    RecallTopKMetric(k=5),
    FBetaTopKMetric(k=5),
    MAPKMetric(k=5),
    NDCGKMetric(k=5)])

column_mapping = ColumnMapping(recommendations_type='score', target='BEST_ACTION', prediction='RELEVANCE', user_id='SEARCH_ID', item_id='DEDUPID', datetime='DATE_OCCURENCE')
report_test.run(
    reference_data=None,
    current_data=request_data,
    column_mapping=column_mapping,
)

report_test.show(mode='inline')
```
[data_scores.csv](https://github.com/evidentlyai/evidently/files/14383618/data_scores.csv)


The error message:
`Exception: Dot product shape mismatch, (1,) vs (2,)`

If I remove the NDCG metric the rest of metrics work fine
Repository: evidentlyai/evidently
Issue: Vulnerability patching: update pyspark >= 3.3.2
Root cause: External Dependency Breakage

@thien1212381 reported in #872 a vulnerability founded in pyspark version 3.0.0:

-> Vulnerability found in pyspark version 3.0.0
Vulnerability ID: 54370
Affected spec: >=0,<3.1.3
ADVISORY: Apache Spark supports end-to-end encryption of RPC
connections via "spark.authenticate" and "spark.network.crypto.enabled"....
https://github.com/advisories/GHSA-9rr6-jpg7-9jg6
For more information, please visit
https://data.safetycli.com/v/54370/f17

-> Vulnerability found in pyspark version 3.0.0
Vulnerability ID: 54576
Affected spec: >=0,<3.2.2
ADVISORY: A stored cross-site scripting (XSS) vulnerability in Apache
Spark 3.2.1 and earlier, and 3.3.0, allows remote attackers to execute...
https://github.com/advisories/GHSA-43xg-8wmj-cw8h
For more information, please visit
https://data.safetycli.com/v/54576/f17

-> Vulnerability found in pyspark version 3.0.0
Vulnerability ID: 54694
Affected spec: <=3.0.3
ADVISORY: The Apache Spark UI offers the possibility to enable ACLs
via the configuration option spark.acls.enable. With an authentication...
https://github.com/advisories/GHSA-4x9r-j582-cgr8
For more information, please visit
https://data.safetycli.com/v/54694/f17

Update is needed:  from"pyspark" to  "pyspark>= 3.3.2"
Repository: evidentlyai/evidently
Issue: data drifts and historical drifts - Names of statistical tests

Evidently provides me with the solutions I was looking for. I have one minor question, I want to add a subtitle describing the statistical test (name) to the chart. In the documentation we can find that: "To estimate the data drift, ...,    we use statistical tests to detect if the distribution has changed significantly  ...For numerical features, we use the two-sample K-S test..." but in the chapter "How to detect, evaluate and visualize historical drifts" we can read that "We use Evidently Data Drift Profile to calculate drift". Could you please explain this problem further? is it a known statistical test or an evidently specific solution? 

Hi @tomaszek0 ,
Thanks for your interest in out project. Here documentation about used statistical tests https://docs.evidentlyai.com/reports/data-drift#how-it-works.

Hi @tomaszek0 , to clarify a bit further, "Evidently Profile" refers to the JSON Profile feature when the output of the calculations is provided as a JSON summary instead of the visual dashboard. 

Both for JSON profiles and visual dashboards Evidently uses the same default statistical tests explained in the documentation @Liraim linked to. 

You can also replace the default Evidently tests using options https://docs.evidentlyai.com/step-by-step-guides/report-customization/options-for-data-target-drift#1.-choose-a-different-statistical-test-to-detect-data-drift

Hi
Many thanks for the clarification and links.