zhiminy/EvalEng huggingface dataset:
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Humanevalpack-c++ issue
Root cause: Environment Incompatibility

Hi, 

I always generate pass@1 = 0 when running Humanevalpack-cpp
Evaluating generations...
{
  "humanevalsynthesize-cpp": {
    "pass@1": 0.0
  },


However, take #2 instances as an example, the reference & generations json file is shown as below. If I run it in the C++ compiler, the test case is passed. I'm not sure why I'm facing this issue only in C++, is there any syntax I should add when running this case?


### references.json
```
#undef NDEBUG
#include<assert.h>
int main(){
 assert (truncate_number(3.5) == 0.5); 
 assert (abs(truncate_number(1.33) - 0.33) < 1e-4);
  assert (abs(truncate_number(123.456) - 0.456) < 1e-4);
}
```
==================================================


### generations.json
```
/*
Given a positive floating point number, it can be decomposed into
and integer part (largest integer smaller than given number) and decimals
(leftover part always smaller than 1).

Return the decimal part of the number.
>>> truncate_number(3.5)
0.5
*/
#include<stdio.h>
#include<math.h>
using namespace std;
float truncate_number(float number){
    return number - floor(number);
}
```

@Gene-su-GGN Maybe the problem would be execution using c++ environments. Could you separate the phrases of code generation and code execution, and then see what happens when executing the code? We would recommend to use colabe to evaluate the model since the environment is reproducible.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Score discrepancy with humaneval
Root cause: Configuration Error

I evaluated Phind-CodeLlama-34B-v2 according to their recipe (using OAIs lib)
https://huggingface.co/Phind/Phind-CodeLlama-34B-v2/blob/main/README.md#how-to-reproduce-humaneval-results
and got a score around 70
when trying to replicate with bigcode-evaluation-harness with:
--model Phind-CodeLlama-34B-v2      
--tasks multiple-py
and
--model Phind-CodeLlama-34B-v2      
--tasks humaneval
(from my understanding those should be equivalent)
and get a scores around 34 .

From my understanding all 3 ways should yield similar scores.
Does someone have an idea or am i misunderstanding sth here ?


Hi, we've found that Phind models need to have a `\n` at the end of the prompt, while humaneval [strips](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/56ec1447b84cd1f85e59e45ec7da368de66db3bb/bigcode_eval/tasks/humaneval.py#L67) the prompt by default, try `humaneval-unstripped` task instead.

If you check the BigCode Leaderboard we got 71.95 pass@1 for this model: https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard

For MultiPL-E we explicitly made sure to have a `\n` at the end of prompts, but it's not implemented yet in main.

il check that out thx
Repository: bigcode-project/bigcode-evaluation-harness
Issue: exact commandline to reproduce leaderboard
Root cause: Documentation Deficiency

Wonderful Bigcode collaborators,

Can we add a new column to the [Big Code Leaderboard](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard) which describes the exact command line to reproduce the scores?
I saw many similar questions asking about how to reproduce the scores, and the differences against the paper.
This new column will definitely help.

Thanks!

Good suggestion, will add it to the todo list

Hi, I just saw this issue, but does the only variable is the model_name here to reproduce results? And if that is the case, do we require a whole column?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: OOM when running Multi-gpu inference using A100 40GB
Root cause: Resource Mishandling

Has anyone faced the issue of OOM when running the inference using A100 40GB? The OOM error occur when the process try to load the model shard into GPU memory. Much appreciated.

Hi, which model are you trying to evaluate and what's your execution command and accelerate config?
For instance try decreasing batch size/seq length or using fp16 or bf16 precision or even load_in_8bit/4bit to save some memory
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Evaluation speed with multi-gpu
Root cause: Configuration Error

Thx for your great work! I'm conducting evaluation under a multi-gpu scenario (2 A100x80G). But it's even slower than that with one single gpu.

Here is the output of `accelerate env`:
```
- `Accelerate` version: 0.24.1
- Platform: Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- Numpy version: 1.26.1
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 1007.35 GB
- GPU type: NVIDIA A100-SXM4-80GB
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
```

BTW, the evaluation script is copied from the README file.

can you provide the execution command and with model name, batch size, n_samples..?

Sorry for the delay. I just use greedy sampling and run with the following command:

```
accelerate launch --mixed_precision bf16 ./bigcode-evaluation-harness/main.py
    --model /mnt/models/codellama-7b-python
    --tasks humaneval
    --max_length_generation 512
    --batch_size 1
    --do_sample False
    --precision bf16
    --max_memory_per_gpu 'auto'
    --allow_code_execution
    --trust_remote_code
    --save_generations
    --use_auth_token
    --metric_output_path ./bigcode-evaluation-harness/output/humaneval_codellama.json
    --save_generations_path ./bigcode-evaluation-harness/output/generations_humaneval_codellama.json
```
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Design prompts for few-shot evaluation tasks
Root cause: Unimplemented Feature Gap

We do not have natural language prompts for all tasks in the [Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness). We would either like to find prompts which have been adopted by other research groups or design prompts that work well for the task at hand. For example, we would appreciate help with designing prompts for the following tasks:
- [APPS](https://github.com/bigcode-project/bigcode-evaluation-harness/issues/8)
- CoNaLA
- Concode
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Human_eval .yaml
Root cause: Unimplemented Feature Gap

Is there any implementation of human_eval dataset in the new version of eval_harness? (The one of EleutherAI)

In that new version, the tasks are configured via .yaml. This file is the one i need.

Thanks.

I don't think HumanEval is implemented in lm-evaluation-harness
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Use trust_remote_code for dataset load
Root cause: Configuration Error

Following https://github.com/huggingface/datasets/issues/6400 and the 2.16 release of `datasets` you now require `trust_remote_code=True` for datasets with a custom builder config.

if I read the code correctly, the `--trust_remote_code` flag gets passed to model and tokenizer already. It does not reach the `load_dataset` method [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/8d9f66716b064f1ad1598fdceb2ae4e1e749ba9d/bigcode_eval/base.py#L28).

That seems an easy fix and I can open a PR next week.
Addressing it for the finetuning examples would be more edits.

if I'm not mistaken, the only dataset that has arbitrary code in the harness now and could need this flag in the future is MultiPL-E?

It's a bit different from models where users can pass any model they want, whereas we have a limited number of carefully selected tasks so I'm not sure we need the flag for all tasks (we can add it just for MultiPL-E when it's a requirement if the authors don't update their loading script by then).

I haven't checked myself. And moving all datasets to parquet export and no builder script is the better idea anyway. Perhaps adding a notice to the new task template might be an easier "fix".

If there is just a single case it can by bypassed using the env flag much easier. 

I additionally came across this with the task I am developing. But moving to parquet is on my to-do list either way. 
Feel free to close this issue and reject the draft PR.

sounds good, I also talked to MultiPL-E authors and they will remove the python code in the dataset so it wouldn't require the flag.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Add selected tasks to the Evaluation Harness
Root cause: Unimplemented Feature Gap

We would appreciate help with adding one of the following tasks to the Evaluation Harness. See the issues below:
- [ ] [HumanEval-X](https://github.com/bigcode-project/bigcode-evaluation-harness/issues/6)
- [ ] [Transcoder task](https://github.com/bigcode-project/bigcode-evaluation-harness/issues/5)
- [ ] [CodeXGLUE-code-refinement](https://github.com/bigcode-project/bigcode-evaluation-harness/issues/4)
- [ ] [CodeXGLUE-text-to-text](https://github.com/bigcode-project/bigcode-evaluation-harness/issues/3)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: errors when ran THUDM/chatglm3-6b on this project
Root cause: Unimplemented Feature Gap

I ran the THUDM/chatglm3-6b model on this project and found that it reported an error: Error reported when running code: assert self. adding_ Side=="left" AssertionErro, if you have time, take a look. Thank you again.


It seems ChatGLM3-6B requires left padding which wasn't supported in the harness.
I implemented a fix in https://github.com/bigcode-project/bigcode-evaluation-harness/pull/184

> It seems ChatGLM3-6B requires left padding which wasn't supported in the harness. I implemented a fix in #184

thanks a lot
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Question regarding stop_at_stop_tokens method
Root cause: Documentation Deficiency

Hi all, I have a question regarding the method "stop_at_stop_tokens" implemented [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/bigcode_eval/base.py#L82) . According to my understanding, this method find the first appearance of a stop_token (which signal the end of a method/function). The code then cut off the generated text after that point. However, would this logic still work with a function/method which call upon helper function/method that is placed after it? For example:

`

        String[] x_split = x.split("/");
        String[] n_split = n.split("/");
        int x_numerator = Integer.parseInt(x_split[0]);
        int x_denominator = Integer.parseInt(x_split[1]);
        int n_numerator = Integer.parseInt(n_split[0]);
        int n_denominator = Integer.parseInt(n_split[1]);
        int lcm = lcm(x_denominator, n_denominator);
        if (lcm % x_denominator == 0 && lcm % n_denominator == 0) {
            int gcd = gcd(x_numerator, n_numerator);
            if (gcd == 1) {
                return false;
            } else {
                return true;
            }
        } else {
            return false;
        }
    }

    private static int gcd(int a, int b) {
        if (b == 0) {
            return a;
        }
        return gcd(b, a % b);
    }

    private static int lcm(int a, int b) {
        return a * (b / gcd(a, b));
    }
}


`


yes this will truncate helper functions after the main one, but the goal is to generated a self-contained solution, usually if helper functions are needed they're provided in the prompt itself.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Supporting HumanEval+ dataset
Root cause: Unimplemented Feature Gap

Hi, we have just uploaded HumanEval+ to hugging face: https://huggingface.co/datasets/evalplus/humanevalplus in a format that is 100% compatible with the original HumanEval. I am thinking of having this supported in `bigcode-evaluation-harness` as well due to the HumanEval-compatible format. 

```python
# Load from huggingface
from datasets import load_dataset
load_dataset("evalplus/humanevalplus")
```

![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/38074777/5dfbb0c7-3f4c-447b-9a92-a6f733a4cf50)

We also manually tested the validity and got the exact scores compared to using `evalplus`:
 
![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/38074777/246a935a-91be-4e04-99e6-6701595f7ff4)

A few notes regarding integration:
* Add a bit more timeouts given we have more tests: empirically I can repro the results by using max 8-10s testing time for each task
* The testing code would require `numpy` as an additional package in the execution environment.

We are also going to support MBPP+ in a compatible format soon.

PR merged, thanks for the integration! `numpy` is already a dependency of `datasets` so it should be in the environment.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Why the result of using multiple evaluation is 0
Root cause: Environment Incompatibility

![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/56470984/69f8edd8-c20f-41cc-9c4a-afcae4b3cc61)

my settings
![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/56470984/d0f7ad08-15cd-4146-9df2-5485fea0c146)


Are you running the execution inside the docker image we provide for MultiPL-E? as it requires extra dependencies
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Out-of-memory of multi-gpu evaluation
Root cause: Resource Mishandling

I configure acclerate config correctly but it gives me out-of-memory issue. I examine the GPU usage and can see that all the 4 processes are using the first gpu. I'm sure that my model fits into one card. I test one card evaluation and the process runs correctly. 
The following is my accelerate config:
```yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
gpu_ids: all
machine_rank: 0
main_training_function: main
mixed_precision: 'no'
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```
The following is my running script:
```bash
accelerate launch  main.py \
          --model meta-llama/Llama-2-7b-hf \
  --tasks mbpp \
  --max_length_generation 650 \
  --temperature 0.8 \
  --do_sample True \
  --n_samples 10 \
  --batch_size 10 \
  --allow_code_execution \
  --save_generations \
  --generation_only \
  --save_generations_path generations_llama-7b.json
```



It seems accelerate is correctly configured to use 4 GPUs, maybe the model fits in one gpu but it's not enough for a batch size of 10? can you try lowering it and using mixed precision via `--precision bf16` or `--precision fp16`?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Potentially extra slow inference when using LoRA adapter
Root cause: Configuration Error

Hello, everybody. Tried HumanEval benchmark on my custom Mistral tune today, but getting weird warning: 

```
UserWarning: Input type into Linear4bit is torch.float16, but bnb_4bit_compute_dtype=torch.float32 (default). This will lead to slow inference or training speed.
  warnings.warn(f'Input type into Linear4bit is torch.float16, but bnb_4bit_compute_dtype=torch.float32 (default). This will lead to slow inference or training speed.')
```

Dont know how to fix this, any ideas? 

My command to run the benchmark:

```bash
accelerate launch  main.py \
  --model {model_name} \
  --peft_model {peft_model_path} \
  --load_in_4bit \
  --max_length_generation 512 \
  --tasks humaneval \
  --temperature 0.2 \
  --precision bf16 \
  --n_samples 200 \
  --batch_size 32 \
  --allow_code_execution \
  --limit 25 
```

Seems to only occur when ` --load_in_4bit` passed.

This worked for me when receiving the same warning
[https://stackoverflow.com/questions/77301266/input-type-into-linear4bit-is-torch-float16-but-bnb-4bit-compute-type-torch-flo](url)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: [FEATURE REQUEST] Support HumanEval+ tests for MultiPL-E
Root cause: Unimplemented Feature Gap

We've added the support for HumanEval+ in MultiPL-E some time ago:
https://github.com/nuprl/MultiPL-E/blob/main/humaneval_plus/generate_data.py
(the queries are not stored in the repo since they're too large)
It would be nice to support it in evaluation harness too.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: AATK process_results is missing 
Root cause: Unimplemented Feature Gap

https://github.com/bigcode-project/bigcode-evaluation-harness/blob/astraios/bigcode_eval/tasks/aatk.py#L130-L131

@loubnabnl @RaymondLi0: Can you provide some pointers on this? I see that you have used this evaluation benchmark in the Starcoder2 paper. Thanks!

Hi,

Just put the answer here so that more people can see.

There is an evaluation repo provided by the authors of the benchmark: https://github.com/moyix/AsleepKeyboardDataset

We mainly rely on the evaluation scripts provided by them.

Cheers

@terryyz can you share the exact command used to run the eval for AATK?

> @terryyz can you share the exact command used to run the eval for AATK?

I just created a repo used for SC2 eval :slightly_smiling_face:
https://github.com/terryyz/AsleepKeyboardDataset
Please note that you may want to change stop_words when evaluating other models.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Support for StudentEval Dataset (Again)
Root cause: Unimplemented Feature Gap

@arjunguha It seems that #88 is not merged into the `main` branch but is somehow closed. Running the code taken from the [HuggingFace data card](https://huggingface.co/datasets/wellesley-easel/StudentEval) returns errors:

```bash
python3 main.py --model bigcode/gpt_bigcode-santacoder --tasks studenteval --max_length_generation 512 --n_samples 20 --batch_size 20 --precision bf16 --allow_code_execution
``` 

Is there any way I could run the evaluation on the StudentEval dataset any time soon?

whoops. mistake. let me try to merge it in this time. i can probably get it running in a day or so
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Add tests to the evaluation harness
Root cause: Unimplemented Feature Gap

Add tests to the existing evaluation benchmarks to make sure they are not broken by new additions. (e.g: ensure fixed generations for a specific model using greedy sampling with a fixed seed)

Maybe we can add a tiny dummy model and run it on each task for a handful of samples. Could be useful to create a small CI for this with GH actions :) 

Tests added for MBPP and HumanEval in https://github.com/bigcode-project/bigcode-evaluation-harness/pull/15

@lvwerra @Muennighoff what do you think of the current [tests](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/tests/test_generation_evaluation.py), do you have suggestions for other tests we should add aside from linters (we currently test the generation and evaluation modules with some humaneval and mbpp examples with `n_samples=2`)?

I think it's good - can close this imo
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Support for HumanEval-Infilling Benchmark
Root cause: Unimplemented Feature Gap

Thank you very much for providing the evaluation code.
I would like to inquire if there are any plans to include the [HumanEval-Infilling Benchmark](https://github.com/openai/human-eval-infilling) evaluation in the current tasks.

Not at the moment, we already support a similar benchmark SantaCoder-FIM https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/docs#santacoder-fim
Repository: bigcode-project/bigcode-evaluation-harness
Issue: How to use my local model files to run the program? I cant download the model online 
Root cause: Documentation Deficiency

thx for the work, i just want use the model downloading  from hugging face(codellama-13B-python) and i dont know how to do this

Hi, you can provide te path to your model in `--model` instead of HF repo name:
```python
accelerate launch  main.py \
    --model PATH  \
    ...
 ```

thank you for your help!  i have another quesion. may i modify the "get_promt" function to implement  few-shot in MBPP dataset? i wonder if there is someting else i need to attention  

Yes you can just change `get_prompt`. You can see how it's done for GSM8K for example https://github.com/bigcode-project/bigcode-evaluation-harness/blob/e54f33d093f342ffc0c5c057910c00aa2081515d/bigcode_eval/tasks/gsm.py#L125

thank you very much！i get it !!
Repository: bigcode-project/bigcode-evaluation-harness
Issue: max_length_generation parameter
Root cause: Documentation Deficiency

You mentioned in the readme that max_length_generation=512 is enough for tasks like HumanEval and MBPP, but when I tested phi-1.5 and deepseek-coder-1.3b-base on the mbpp task, the following problems occurred at max_length_gen = 512.

```
ValueError: Input length of input_ids is 512, but `max_length` is set to 512. This can lead to unexpected behavior. You should consider increasing `max_length` or
, better yet, setting `max_new_tokens`.
```

How should this parameter be set so that the test results can be aligned? Will the setting of this parameter have a significant impact on the results?

The default is 512 (works fine with HumanEval) but for some tasks you might need more try setting it to 1024 for mbpp. Regarding the impact on the results, if the benchmark has long prompts you want to have a higher max_length to have room for generation otherwise the solutions won't be complete.

@loubnabnl I was facing the same issue for **multiple-java** and **mulitple-cpp** while trying to reproduce the leaderboard score for **codellama-7b** using the steps given in [leaderboard README](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/leaderboard#2--generation) here 
https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/leaderboard#2--generation

is it supposed to be 1024 for **multiple-cpp** and **multiple-java** as well?

I was confused beacause in the _[leaderboad About section](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard)_ it is mentioned that 

> `All models were evaluated with the [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main) with top-p=0.95, temperature=0.2, max_length_generation 512, and n_samples=50.`


<img width="1024" alt="image" src="https://github.com/bigcode-project/bigcode-evaluation-harness/assets/152659611/15d1f688-348d-4e4e-9a4d-c44ec3d142d5">




Hi sorry for the confusion, if this happens try 1024, some tokenizers might generate more tokens than others which takes more space. Will update the "About" section of the leaderboard.

It seems MBPP has a prompt with 1700 tokens with some tokenizers, after this PR https://github.com/bigcode-project/bigcode-evaluation-harness/pull/244 you should be able to run the evaluation with a smaller max_length but you might get lower scores as the solutions to some long prompts won't be generated