zhiminy/EvalEng huggingface dataset:
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Show metric in outfile
Root cause: Unimplemented Feature Gap

```python
{
  "codexglue_code_to_text-python-left": 0.06565988797511521,
  "config": {
    "model": "bigcode/christmas-models"
  }
}
```

would be better to also have the metric imo

That makes sense, how do you suggest including it? Add an arg to the task class or output the name of the metric after calling it in [`postprocess_results`](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/d0c51273596048ad25293cd71f38e3fb19e3e847/lm_eval/tasks/humaneval.py#L72) for example?

Yeah I would return the metric name that we provide to `evaluate.load`, maybe sth like:

```python
code_metric = load("code_eval")
results, _ = code_metric.compute(
    references=references,
    predictions=generations,
)
return {
    "code_eval": results,
}

```
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Some questions about APPS 
Root cause: Documentation Deficiency

hello!  i have some problems when evaluating on APPS.  firstly ,I cant find any one-shot examples on corresponding  generation code like follows:
and i wander how should i use the locally download metrics without downloading "apps_metrics"?
thanks  a lot if anyone have some ideas !!!

Hi, I'm not sure I understand, can you provide the code you used to run the evaluation? and wdym by 

> without downloading "apps_metrics"

In docuemntation this is given for APPS. However this is never used or I cannot see it in the code? How can I evaluate in single shot for APPS for non finetuned models?

2- Few-shot: for non finetuned models, we provide one example in the prompt for each call type (Standard Input and Call-Based format). We add the examples with an instruction before the prompt above:

    one_shot_prompt = (
        "Implement answers to the following questions:\nQUESTION:\n"
        + examples["problem_type1"]
        + "\nUse Standard Input format\nANSWER:\n"
        + examples["solution_type1"]
        + "\nQUESTION:\n"
        + examples["problem_type2"]
        + "\nUse Call-Based format\nANSWER:\n"
        + examples["solution_type2"]
        + "\n"
        + prompt
    )
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Value error regarding to "--max_length_generation"
Root cause: Validation Gap

Hi, I am running MBPP on llama-2 model. However, whatever "--max_length_generation" I set, the benchmark will fail.

My command to execute: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch main.py --model meta-llama/Llama-2-7b-hf --tasks mbpp --precision bf16 --max_length_generation 1024 --allow_code_execution

The output failure message:
ValueError: Input length of input_ids is 1024, but max_length is set to 1024. This can lead to unexpected behavior. You should consider increasing max_length or, better yet, setting max_new_tokens.ValueError: Input length of input_ids is 1024, but max_length is set to 1024. This can lead to unexpected behavior. You should consider increasing ma x_length or, better yet, setting max_new_tokens.

It seems to me the length of input_ids is somehow increasing with the --max_length_generation? With that i am not able to conduct the benchmark for MBPP. Thanks!

It seems some of the prompts are long (they take all tokens of max_generation_length and then you have no room left for generation). Can you try setting it to a larger value such as 2048? (some tokenizers generate more tokens than others)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Using custom prompts and postprocessing
Root cause: Documentation Deficiency

Hi,

We want to make a submission for the leaderboard with our fine-tuned model used in www.jdoodle.com

It seems like for some of the humaneval questions, our model getting the logic right but we are having issues due to the format of the output which could be potentially fixed by postprocessing and using a slightly different prompting -- our fine-tuning process was using a different prompt template then instruction prompting--

I was wondering would it be  a problem if we use custom postprocessing steps and/or prompt template for generating responses? Would we be eligible to have a place in the leaderboard in that case?

Thanks!

Hi, you can use HumanEvalSynthesize for Python and edit the prompt similar to here https://github.com/bigcode-project/bigcode-evaluation-harness/pull/219/files I think the post-processing should work by default, what other changes did you want to introduce? 
For the other languages we're using plain MultiPL-E prompts in the leaderboard even for the chat models, you can add another stop token [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/f0f2b52ab0bac95b7fa881693e82c5ddcf2f9e95/bigcode_eval/tasks/multiple.py#L93) if that's what's messing up the post-processing
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Using the humanevalpack to test the ChatGLM3 model results in an abnormal score.
Root cause: Configuration Error

Hi
When I tried to test the ChatGLM3 model using the humanevalfixdocs-python task in humanevalpack, an abnormality occurred with a score of 0. The command used is as follows.

accelerate launch  main.py \
  --model THUDM/chatglm3-6b \
  --left_padding \
  --tasks humanevalfixdocs-python \
  --max_length_generation 2048 \
  --prompt chatglm3 \
  --trust_remote_code \
  --temperature 0.7 \
  --do_sample True \
  --n_samples 1 \
  --batch_size 64 \
  --precision bf16 \
  --allow_code_execution \
  --save_generations
  
  The prompt I used is as follows.
    elif self.prompt == "chatglm3":
              prompt = f"
{inp}
{prompt_base}"
     else:
              raise ValueError(f"The --prompt argument {self.prompt} wasn't provided or isn't supported")
              
   **The issue I encountered is that the model did not generate any answers in the generations.**
   **The answer to the first question is as follows, and it can be seen that no response from the model was obtained.**
   
   from typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n    \"\"\" Check if in given list of numbers, are any two numbers closer to each other than\n    given threshold.\n    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n    False\n    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n    True\n    \"\"\"
Repository: bigcode-project/bigcode-evaluation-harness
Issue: The evaluation results are inconsistent across different GPUs
Root cause: Resource Mishandling

command:
<img width="389" alt="image" src="https://github.com/bigcode-project/bigcode-evaluation-harness/assets/121479335/90cfbb7e-32af-41c6-8910-1511c1de6e10">
result:
<img width="574" alt="image" src="https://github.com/bigcode-project/bigcode-evaluation-harness/assets/121479335/0faa0520-505c-4586-bcd2-c112283d8273">
Why do the same model and the same command yield inconsistent results?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Need some context for certain args for Instruct Human Eval
Root cause: Documentation Deficiency

Hey all, what is the n_samples for instruct human eval about?

The docs say 200 as if its a static setting that should be used, but I can't understand why.

Also, when the turns format is structured, does this look right for chatml?

--instruction_tokens "
user\n","
\n","
assistant\n"

without quoting each string it gave an error so I assume this is how to use this arg?

I ran the instruct humaneval benchmark and the eval results.json shows this:

`"eos": "
",`

whereas the actual EOS should be 
 - don't really see why this is there 

I recommend just running humanevalsynthesize from humanevalpack which offers the same + more - instructions for running are here: https://github.com/bigcode-project/octopack?tab=readme-ov-file#run & here https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/docs/README.md#humanevalpack

you may just need to add your instruction format here: https://github.com/bigcode-project/bigcode-evaluation-harness/blob/0f3e95f0806e78a4f432056cdb1be93604a51d69/bigcode_eval/tasks/humanevalpack.py#L235
Repository: bigcode-project/bigcode-evaluation-harness
Issue: [Possibly system specific] Wild (12% vs 20%) run-to-run swings in `multiple-cpp` reported scores
Root cause: Resource Mishandling

This issue is currently blocking for me, so I am looking into it myself. I would really appreciate any potential help with investigating/isolating this issue. I'm happy to do everything I can to help reproduce.

Full logs for two consecutive runs [here](https://gist.github.com/alat-rights/95fa8c9bebab66d969508b209077a17e). The first run got `pass@1` of `12%`, the second run got `pass@1` of `20%`. They were ran on the same system with the same invocation.

**System Information**
Ubuntu Linux
Lambda Labs Instance: 1x A10 (24 GB PCIe) (30 CPU cores, 205.4 GB RAM, 1.5 TB SSD)

On a fresh instance, I'm consistently getting 38.5%. I suspect the problem was specific to my system.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: `multiple-cs`, `multiple-go`, and `multiple-d` are broken (`FileNotFound`)
Root cause: Environment Incompatibility

I wonder if this might just be user error on my end. Is there a step for installing compilers like `csc` that I skipped over before running `bigcode_eval`?

Run log / full outputs here: https://gist.github.com/alat-rights/705edc6d3aae51db7c871afa153cc3a5

```
FileNotFoundError: [Errno 2] No such file or directory: 'csc'
```

[Code point](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/bigcode_eval/tasks/custom_metrics/multiple_metrics/eval_cs.py)

**System Information**
Ubuntu Linux
Lambda Labs Instance: 1x A10 (24 GB PCIe) (30 CPU cores, 205.4 GB RAM, 1.5 TB SSD)

Here are logged failures for `multiple-go` and `multiple-d`: https://gist.github.com/alat-rights/1e0092cd9ccba9adc4057c4fa309948e

Closed, was just missing dependencies, sorry. (I didn't realize `multiple` grading was only supposed to be run in the Docker)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: [Minor] Conflicting dependencies in requirements.txt
Root cause: Configuration Error

Running `pip install -r requirements.txt` gives me

```
ERROR: Cannot install -r requirements.txt (line 1) and huggingface_hub==0.8.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested huggingface_hub==0.8.1
    transformers 4.25.1 depends on huggingface-hub<1.0 and >=0.10.0
```

Instead,
```
huggingface_hub>=0.10.0
``` 

fixed it for me and hasn't broken anything so far. 


Thanks for flagging it! It won't break anything as it's added if the user want to push generations to the hub. Feel free to open a PR to fix it or I can update it.

Just fixed in https://github.com/bigcode-project/bigcode-evaluation-harness/pull/25 🚀
Repository: bigcode-project/bigcode-evaluation-harness
Issue: HumanEval evaluation results mismatch for Codegemma-2b
Root cause: Configuration Error

I am getting a score of 7.8 when I am trying to evaluate [codegemma-2b](https://huggingface.co/google/codegemma-2b) on humaneval. The authors report a score of  31.1. Here is the command I used. 


```
accelerate launch  main.py \
  --model google/codegemma-2b \
  --tasks humaneval \
  --temperature 0.2 \
  --n_samples 20 \
  --batch_size 4 \
  --allow_code_execution \
  --use_auth_token
```


Below is the output of the code. 
```
{
  "humaneval": {
    "pass@1": 0.07804878048780488,
    "pass@10": 0.14644529185229932
  },
  "config": {
    "prefix": "",
    "do_sample": true,
    "temperature": 0.2,
    "top_k": 0,
    "top_p": 0.95,
    "n_samples": 20,
    "eos": "
",
    "seed": 0,
    "model": "google/codegemma-2b",
    "modeltype": "causal",
    "peft_model": null,
    "revision": null,
    "use_auth_token": true,
    "trust_remote_code": false,
    "tasks": "humaneval",
    "instruction_tokens": null,
    "batch_size": 4,
    "max_length_generation": 512,
    "precision": "fp32",
    "load_in_8bit": false,
    "load_in_4bit": false,
    "left_padding": false,
    "limit": null,
    "limit_start": 0,
    "save_every_k_tasks": -1,
    "postprocess": true,
    "allow_code_execution": true,
    "generation_only": false,
    "load_generations_path": null,
    "load_data_path": null,
    "metric_output_path": "evaluation_results.json",
    "save_generations": false,
    "load_generations_intermediate_paths": null,
    "save_generations_path": "generations.json",
    "save_references": false,
    "save_references_path": "references.json",
    "prompt": "prompt",
    "max_memory_per_gpu": null,
    "check_references": false
  }
}
```



Same here.

I was trying to reproduce the leaderboard score [here](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard) for CodeGemma-2b (~27). I followed the exact same procedure documented [here](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/leaderboard) but got a score much lower score (~7.5) than that reported on the leaderboard.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: What is `fine-tuning` in task submission?

Check https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/docs/guide.md#fine-tuning:
<img width="1241" alt="image" src="https://github.com/user-attachments/assets/1d875569-5d34-4379-9181-353d730fe7d0">
Repository: bigcode-project/bigcode-evaluation-harness
Issue: ImportError: cannot import name 'SyncManager' from partially initialized module 'multiprocessing.managers'
Root cause: Resource Mishandling

Hi, I met the following ImportError during generation evaluations. 

generations were saved at ./output_humaneval/humaneval_humaneval.json
Evaluating generations...
Traceback (most recent call last):
  File "/root/software/bigcode-evaluation-harness/main.py", line 414, in <module>
    main()
  File "/root/software/bigcode-evaluation-harness/main.py", line 398, in main
    results[task] = evaluator.evaluate(
  File "/root/software/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 107, in evaluate
    results = task.process_results(generations, references)
  File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/humaneval.py", line 98, in process_results
    results, _ = compute_code_eval(
  File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/custom_metrics/code_eval.py", line 154, in compute_code_eval
    result = future.result()
  File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/root/miniconda3/envs/bigcode/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/software/bigcode-evaluation-harness/bigcode_eval/tasks/custom_metrics/execute.py", line 36, in check_correctness
    manager = multiprocessing.Manager()
  File "/root/miniconda3/envs/bigcode/lib/python3.9/multiprocessing/context.py", line 55, in Manager
    from .managers import SyncManager
ImportError: cannot import name 'SyncManager' from partially initialized module 'multiprocessing.managers' (most likely due to a circular import) (/root/miniconda3/envs/bigcode/lib/python3.9/multiprocessing/managers.py)


Here is my script:
'''
#!/bin/bash
taskname=humaneval
mkdir -p output_$taskname

python /root/software/bigcode-evaluation-harness/main.py \
  --model /data00/models/meta-llama_Meta-Llama-3.1-8B-Instruct/ \
  --trust_remote_code \
  --tasks $taskname \
  --limit 8 \
  --max_length_generation 512 \
  --temperature 0.8 \
  --do_sample True \
  --n_samples 16 \
  --batch_size 16 \
  --precision fp16 \
  --allow_code_execution \
  --save_generations \
  --save_generations_path ./output_$taskname/$taskname.json
'''

Could you share some tips on how to solve this problem?

DO NOT BELIEVE MY DELETED COMMENT PLS
MY ACCOUNT GOT HACKED
IM CONTACTING GITHUB SUPPORT
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Unable to execute the `MultiPL-E` task for `python` language
Root cause: Validation Gap

Hello, I am trying to execute bigcode-eval-harness for the `MultiPL-E` task then I can't do the generation for `python` language. I can generate the responses for the other 18 languages. Because if you look into the huggingface platform they don't have the `humaneval-py` dataset But this task is trying to load the dataset.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: [Minor] Missing task template
Root cause: Documentation Deficiency

The [guide on adding a new task](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/docs/guide.md#creating-your-task-file) recommends to 

> From the bigcode-evaluation-harness project root, copy over the new_task.py template to lm_eval/tasks

but there is no `new_task.py` template in the project root. 

I suspect this is a relic from the [lm-evaluation-harness repo](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_guide.md#creating-your-task-file)?

Indeed thanks for flagging it! I forgot to push that file
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Evaluation result of bigcode/starcoder2-3b on gsm8k_pal does not matched the paper
Root cause: Validation Gap

I tried to evaluate the model  `bigcode/starcoder2-3b` on the benchmark `pal-gsm8k-greedy` using the command below

`accelerate launch --main_process_port 6789 main.py   --model bigcode/starcoder2-3b   --max_length_generation 2048   --tasks pal-gsm8k-greedy   --n_samples 1   --batch_size 1   --do_sample False   --allow_code_execution`

Then I got the result

```
  "pal-gsm8k-greedy": {
    "accuracy": 0.04624715693707354,
    "num_failed_execution": 1016
  },
```

Since the reported value in the paper starcoder2 is 27.7 (Table 14), the result from harness does not match the paper.

Is there someone else who can check out why this happened?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: ValueError: Infilling not yet supported for:/Meta-Llama-3.1-8B
Root cause: Unimplemented Feature Gap

Encountered below error on latest set of llama model 3.1 
ValueError: Infilling not yet supported for: /data07/hf_model_files/Meta-Llama-3.1-8B
[2024-09-23 14:08:53,151] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 52095) of binary: /home/kadu/venv-kb-llm-benchmark/bin/python
Traceback (most recent call last):
Repository: bigcode-project/bigcode-evaluation-harness
Issue: trust_remote_code is not been passed to dataset
Root cause: Configuration Error

At `bigcode_eval/base.py/Task/__init__`
self.dataset = load_dataset(path=self.DATASET_PATH, name=self.DATASET_NAME)
Does not pass the arg trust_remote_code here

was decided as not needed in #176 you should be able to set an env var to `HF_DATASETS_TRUST_REMOTE_CODE`=1` to bypass this
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Update docker image for HumanEvalPack-Synthesize.
Root cause: Environment Incompatibility

I would like to evaluate generated `humanevalsynthesize` code in a docker container. I'm running:
```
docker run -v ./_humanevalsynthesize-${lang}.json:/app/generations.json:ro -it --rm evaluation-harness-multiple \
      --model model \
      --tasks humanevalsynthesize-${lang} \
      --load_generations_path /app/generations.json \
      --allow_code_execution \
      --do_sample False \
      --n_samples 1 > ./${lang}_results.log
```

The "evaluation-harness" docker image does lacks the dependencies for evaluating languages such as javascript or rust.
So I'm using the latest "evaluation-harness-multiple" image.

When evaluating on [_humanevalsynthesize-cpp.json](https://github.com/user-attachments/files/17451970/_humanevalsynthesize-cpp.json),  the "pass@1" result is always 0. However, outside of the container, the same generation has a "pass@1" of about 0.3.

When evaluating on `_humanevalsynthesize-rust.json`, the tasks fails due to `FileNotFoundError: [Errno 2] No such file or directory: 'cargo'`. The full error log is in [rust_results.log](https://github.com/user-attachments/files/17451971/rust_results.log). Note that `multiple-rs` works correctly in that container, but not `humanevalsynthesize-rust`.

In short, I'd like to evaluate the 6 humanevalpack languages `python, js, java, cpp, go, rust` in a container but neither `evaluation-harness` nor `evaluation-harness-multiple` is fully working.

Thanks.


hey @zongyf02 we ran into the same issues and built our image like:

```Dockerfile
FROM ghcr.io/nuprl/multipl-e-evaluation:v3.1

# Set env vars
ENV CARGO_HOME=/.cache/cargo/
ENV GOFLAGS="-mod=mod"

# Go create dir 
RUN mkdir /go

# Rust add cargo
RUN apt-get update && apt-get install -yqq cargo

# C++ add boost lib and ssl lib
RUN apt-get update && apt-get install -yqq libboost-dev libssl-dev

# rest of default Dockerfile
...
```

> hey @zongyf02 we ran into the same issues and built our image like:

Works perfectly. Thank you!
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Continuing / Extending Previous Results from Generating and Evaluating?
Root cause: Unimplemented Feature Gap

Hi! Is there a way to extend a previous evaluation run to include more samples?

For instance, suppose I evaluate Model A with `  --n_samples 1` and then decide I want to increase `  --n_samples 10`. Rather than wasting the compute I already spent for the first sample, is there some way to continue or extend the previous sampling and evaluating results?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: mbpp pass@1 is lower than mbppplus pass@1 on starcoder2?
Root cause: Validation Gap

I evaluate pretrain model of starcoder2 3b/7b,  with the bash script 
`accelerate launch  bigcode-evaluation-harness/main.py   --model bigcode/starcoder2-3b   --max_length_generation 512  --tasks mbpp   --n_samples 200   --batch_size 100   --temperature 0.2   --precision bf16   --allow_code_execution   --use_auth_token` 
and 
`accelerate launch  bigcode-evaluation-harness/main.py   --model bigcode/starcoder2-3b   --max_length_generation 512  --tasks mbppplus   --n_samples 200   --batch_size 100   --temperature 0.2   --precision bf16   --allow_code_execution   --use_auth_token`, but get 
`"mbpp": {
    "pass@1": 0.41799000000000003
  }

"mbppplus": {
    "pass@1": 0.47734126984126984\
  }`
even I change different configs, its the same.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Adding an assistant_model Argument for Speculative Decoding
Root cause: Unimplemented Feature Gap

Hello,

I am working on distilling language models for coding tasks and want to benchmark my distilled model when used as the assistant model for speculative decoding. Currently, there doesn’t seem to be an option to specify a custom assistant model.

I think we could add an assistant_model argument, allowing users to provide the path to their assistant model and passing it to the gen_kwargs. I could make the change for my own local tests, but I was wondering if this feature would be useful enough to integrate into the project.