zhiminy/EvalEng huggingface dataset:
Repository: bigcode-project/bigcode-evaluation-harness
Issue: HumanEval-X Go evaluation raises error
Root cause: Documentation Deficiency

When trying to evaluate the go-version of humanevalsynthesize (or HumanEval-X)

```
accelerate launch --main_process_port 0 main.py --tasks humanevalsynthesize-go --load_generations_path ../.. --metric_output_path ../.. --allow_code_execution --n_samples 1
```

the following error is present in `logs.json` (the location of which is btw hardcoded):
```
failed: # command-line-arguments\nmain_test.go:5:5: no required module provides package github.com/stretchr/testify/assert: go.mod file not found in current directory or any parent directory; see 'go help modules'\n
```

A global install or get of such a module is not possible with the latest go version and I don't see a recommended Go version pinned anywhere

@nielstron Have you tried setting `ENV GOFLAGS="-mod=mod"`? That worked for us.

Thank you @timrbula that seems to fix it. Might be good to add this somewhere to the documentation :)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: HumanEval-X generation appears to not time out called subproceses
Root cause: Resource Mishandling

After running the humaneval-sythesize on a large number of samles, I noticed that many processes "a.out" and "node test.js" are runing, with runtimes of several hours and huge memory consumption. Appearently the evaluation does not kill processes after a set timeout.

Example command
```
accelerate launch --main_process_port 0 main.py --tasks humanevalsynthesize-cpp --load_generations_path ../.. --metric_output_path ../.. --allow_code_execution --n_samples 1000
```
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Select device: GPU for model eval
Root cause: Configuration Error

has the error：

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacty of 23.69 GiB of which 2.53 GiB is free. Process 3238833 has 3.43 GiB memory in use. Process 3238831 has 2.53 GiB memory in use. Process 3238830 has 2.75 GiB memory in use. Process 3238832 has 2.61 GiB memory in use. Including non-PyTorch memory, this process has 2.41 GiB memory in use. Process 3509629 has 2.41 GiB memory in use. Process 3509627 has 2.38 GiB memory in use. Process 3509630 has 2.41 GiB memory in use. Of the allocated memory 2.16 GiB is allocated by PyTorch, and 1.92 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

How can I eval the model edumunozsala/llama-2-7b-int4-python-code-20k with cuda:0 , beacause that the cuda:0 has been occupied by other tasks. Shall I change the config by (accelerate config) ?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Add CodeXGLUE-text-to-text benchmark for documentation translation
Root cause: Unimplemented Feature Gap

Add  this benchmark for documentation translation to the evaluation harness: https://huggingface.co/datasets/code_x_glue_tt_text_to_text
* few-shot setting (see CoNaLa and Spider tasks for examples)
* fine-tning

Hi, I am interested in taking this up ! 

Great, let us know if you need any help!
Repository: bigcode-project/bigcode-evaluation-harness
Issue: When executing languages such as JS and GO in Multiple-E, the generated results suddenly end
Root cause: Configuration Error

I set -- max_1ength_generation 2048 and stop word, but the evaluation result suddenly ended, and the third one in the picture can be generated normally.I am using the mamba-2.8b model.

![Image](https://github.com/user-attachments/assets/f613fdd9-4f7e-4f51-b52d-016f90e34bd5)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: testing Humaneval of qwen-2.5-7B-coder-instruct
Root cause: Configuration Error

Make some trivial changes to the code will bring a dramatical difference to the performance of Qwen-2.5-7B-coder-instruct on the Humaneval benchmark. 

The first point is line 32 in file bigcode_eval/tasks/humaneval.py, in function `create_all_tasks`, when given the parameter `True` to the function `create_task`, the fixed Humaneval prompt in the dataset will be changed, specifically, removing the last '\n'. Therefore, the last token will change, leading the dramatical output of the model.

The second point is line 52 of the same file, that is the `stop_words` list. The stop word `\n#` and `\nif` may ruin the code generated by the model as it like to add annotation and the string `if __name__ == '__main__'` to the end of the code. So removing these from the list and adding some post-processing to the generated code may make it ok. Such as add below code in the line 97 of the file bigcode_eval/evaluator.py:

```python
for i,gen_list in enumerate(generations):
            for j,gen in enumerate(gen_list):
                gen = gen.split("\nif __name__ == \'__main__\':")[0]
                generations[i][j] = gen.split('\nif __name__ == \"__main__\":')[0]
```
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Improve pass@1 Score on Humaneval
Root cause: Configuration Error

### Description:
When evaluating the `Qwen2.5-Coder-14B-Instruct` model on the Humaneval benchmark, I observed that the pass@1 score was relatively low. By analyzing the generations.json file, I found that many generated code snippets contained errors, often ending with tokens `
`. These incomplete code fragments result in syntax errors during evaluation, causing otherwise correct solutions (without such partial suffixes) to fail.

To address this issue, I added `<|` to the list of stop_words during generation to prevent the model from appending incomplete or malformed code. Experimental results show that this simple modification significantly improves the pass@1 score.

### Suggestion:
Including special tokens like `<|` in the default stop token list could help improve the completeness and correctness of generated code.

If this issue makes sense, happy to create a PR for that.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Support configurability of FIM tokens on SantaCoder
Root cause: Architectural Constraint

Currently the SantaCoder and StarCoder FIM tasks have fixed FIM tokens; however, other models may use different fim-tokens. For example to use Qwen2.5-Coder models one might have to create

```python
class SantaCoderQwen25CoderFIM(SantaCoderFIM):
    DATASET_PATH = "bigcode/santacoder-fim-task"

    def __init__(self):
        fim_prefix = "
"
        fim_middle = "
"
        fim_suffix = "
"
        stop_words = ["
", "
"]
        super().__init__(
            stop_words=stop_words,
            requires_execution=False,
            fim_prefix=fim_prefix,
            fim_middle=fim_middle,
            fim_suffix=fim_suffix,
        )
```

Allowing the FIM parameters (e.g. `--fim_tokens` and `--stop_words`) to be passed in similarly to the `--instruction_tokens` for HumanEval would allow this task to be a single class and support future FIM models.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Wrong Hugging Face link for Spider dataset in bigcode-evaluation-harness/docs/README.md line 434
Root cause: Documentation Deficiency

Hi maintainers,

Line 434 of `bigcode-evaluation-harness/docs/README.md` currently links to  
https://huggingface.co/datasets/spider when describing “Spider for SQL code generation”.  
The canonical Spider dataset on HF is actually hosted at  
https://huggingface.co/datasets/xlangai/spider.

Could you update the hyperlink so readers land on the correct page?  
(The wrong one has a nice avatar, but no data files 😄)

Thanks for the great work!

Another broken link in bigcode-evaluation-harness/docs/README.md (line 435, returns 404)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Execution-based FIM evaluation
Root cause: Unimplemented Feature Gap

The SantaCoder FIM evaluation with MultiPL-E uses exact match. We should also execute the generated code. The dataset is here:

https://huggingface.co/datasets/bigcode/santacoder-fim-task

All that is needed is is to execute `item['prefix'] + generated_solution + item['suffix'] + item['tests'`.

I recommend supporting `n` samples per item.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Add Reasoning tasks to the evaluation
Root cause: Unimplemented Feature Gap

In recent times, Code generation models have shown to be good at solving Natural language and/or math reasoning tasks ([1](https://crfm.stanford.edu/helm/latest/?group=reasoning) and [2](https://arxiv.org/abs/2211.10435)). So, it would be good to evaluate the Bigcode models on these tasks.

As discussed, in the evaluation meeting - We could explore the options of adding [PAL-datasets](https://github.com/reasoning-machines/pal/tree/main/datasets) and/or reasoning tasks from [HELM](https://crfm.stanford.edu/helm/latest/)

PAL Datasets:

- [x] [GSM8K](https://huggingface.co/datasets/gsm8k)
- [ ] [ASDIV](https://github.com/reasoning-machines/pal/blob/main/datasets/asdiv.jsonl)
- [ ] MAWPS
- [ ] [SVAMP](https://github.com/reasoning-machines/pal/blob/main/datasets/svamp.jsonl)
- [ ] [Bigbench - Penguins in a table](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/penguins_in_a_table)
- [ ] [Bigbench - Reasoning about colored objects](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/reasoning_about_colored_objects)
- [ ] [Bigbench - Date Understanding](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/date_understanding)

Here is the list of scenarios/tasks we used to evaluate code models in HELM: https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_specs_bigcode.conf#L1. I created a separate conf file for the BigCode project.

The scenarios are in the `scenarios` folder: https://github.com/stanford-crfm/helm/tree/main/src/helm/benchmark/scenarios

For example, GSM8K is at `src/helm/benchmark/scenarios/gsm_scenario.py`: https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/gsm_scenario.py.

Thank you @teetone , the conf file gives a good list of the tasks that can be covered.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Add CodeXGLUE-code-refinement benchmark 
Root cause: Unimplemented Feature Gap

Add this benchmark for code refinement to the evaluation harness: https://huggingface.co/datasets/code_x_glue_cc_code_refinement

* few-shot setting (see CoNaLa and Spider tasks for examples)
* fine-tuning

I can take a look at this task.


Great thanks! Let us know if we can help
Repository: bigcode-project/bigcode-evaluation-harness
Issue: HumanEval post-processing
Root cause: Algorithmic Error

For the HumanEval task, we remove the last block, based on the stop tokens: https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/tasks/humaneval.py#L70

If no stopword is found in the generation (for example if by chance the generation ends exactly at the function's last return statement, or before), then `remove_last_block` would remove the entire generation and return an empty string.

It seems to me that we should rather: remove anything that is after the first block, if there ever is a match with one of the stop tokens

If this issue makes sense, happy to create a PR for that. 

Actually, with the way we do the generation, there will always be an `eof_string` in the generation before it stops due to this [function](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/d60f52d11a4dd599f88f6dbdaf1ec018c973a4be/lm_eval/generation.py#L20) so `remove_last_block` always keeps the solution and removes some excess. 

~~What might happen is having some intermediate print/comment/function left in between but I think that shouldn't impact the evaluation~~ (shouldn't happen either as we stop at the first occurence). But I agree `keep_first_block` like we do in [MBPP](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/d60f52d11a4dd599f88f6dbdaf1ec018c973a4be/lm_eval/tasks/mbpp.py#L65) seems cleaner. Feel free to open a PR.

closing the issue as this was fixed in https://github.com/bigcode-project/bigcode-evaluation-harness/pull/63
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Variable max_length_generation
Root cause: Unimplemented Feature Gap

Allow `max_length_generation` to change from batch to batch to speed-up tasks where length changes a lot.
For tasks scored with exact match, we even know the maximum length for each sample, so it would be nice to just limited the max length for those samples. Need to be careful to make it work with batching.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: add TransCoder task for code translation 
Root cause: Unimplemented Feature Gap

Add this code translation (with unit test) task: https://github.com/facebookresearch/TransCoder. The C++ -> Python subsset was used in [PaLM](https://arxiv.org/pdf/2204.02311v3.pdf). This requires:
* adding the evaluation metric to HuggingFace evaluate https://huggingface.co/docs/evaluate/index
* adding the TransCoder dataset to HuggingFace hub, there already is this [dataset](https://huggingface.co/datasets/ziwenyd/transcoder-geeksforgeeks) but make sure it matches the original dataset in the GitHub repo.
* adding the benchmark to the evaluation harness in a few-shot setting (similarily to PaLM approach)

I am interested to take this task

Great! Let us know if you need any help

@KhalidAlt are you still interested in this task? The evaluation harness was refactored so it should be easier to add new tasks to, also if you don't have enough time, you can skip the first task:

>  adding the evaluation metric to HuggingFace evaluate 

and use the original evaluation code as a [custom metric](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/lm_eval/tasks/custom_metrics)

Yes, I am still interested in this task. I am going to look into it this weekend. Thanks a lot
Repository: bigcode-project/bigcode-evaluation-harness
Issue: How to evaluate the model memory efficiently?
Root cause: External Dependency Breakage

Thanks for the great work and convenient benchmarking tool!

I would like to evaluate `CodeGen-16B` model on the` humaneval` benchmark. At my disposal there is A6000 GPUs with 48Gb of memory. The evaluation script crashes due to CUDA out of memory [here](url) (i.e accelerator.prepare) even with the smallest batch size - 1. 

Since it is model evaluation I would expect that most of the memory is occupied by the model params (no optimizer states).  
Naively, this model should fit into a single GPU if loaded in half precision, since `2x 16 = 32 < 48`. However, when setting in the `accelerate launch` mixed precision with `fp16` I still face OOM error. 

What measures would you suggest to fit the model onto a single GPU? 

This is not going to be full solution. I have gotten Codegen-16B-multi to work on an A6000/48GB. The script we used to pull it off is here:

https://github.com/nuprl/MultiPL-E/blob/main/inference/codegen.py

Note the crazy code for the stopping criteria. IIRC it was necessary to get things to work.

Can you make sure that FP16 is set and follow memory consumption up until `accelerator.prepare` ?

@loubnabnl  I set fp16 in the `accelerate launch --mixed_precision fp16` but it doesn't help. There is no GPU memory consumption up to `accelerator.prepare`. 

@Godofnothing we found a bug which made the memory consumption more than necessary, can you try running evaluation with code from this PR https://github.com/bigcode-project/bigcode-evaluation-harness/pull/61? you now need to specify `--precision fp16`

Closing this issue, as I tried loading CodeGen-16B in mixed precision and it fits under 40GB of RAM

Sorry for long delay. I've pulled the latest version of the code and model successfully fits onto 40GB. Thanks for your help and response.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: add HumanEval-X metric to the HF hub and the task to the harness
Root cause: Unimplemented Feature Gap

[HumanEval-X](https://huggingface.co/datasets/loubnabnl/humaneval-x) from [CodeGeeX](https://github.com/THUDM/CodeGeeX/blob/main/codegeex/benchmark/README.md) is a multilingual version of [HumanEval](https://huggingface.co/datasets/openai_humaneval) for Java, JS, C++ and Go. In addition to code generation, it can also be used for code translation. We want to:
* add the [evaluation metric](https://github.com/THUDM/CodeGeeX/blob/main/codegeex/benchmark/humaneval-x/evaluate_humaneval_x.py) for different languages to [HF evaluate](https://huggingface.co/docs/evaluate/index) similarily to [code_eval](https://huggingface.co/spaces/evaluate-metric/code_eval)
*  add the code generation task to this evaluation harness 

Hello, I'm interested in taking this up.

Great thanks! Let me know if I can help

@pacman100 if you haven't started working on this task I think we can we close this issue, since we are now integrating MuliPL-E instead https://github.com/bigcode-project/bigcode-evaluation-harness/issues/12 which is similar but has more programming languages

Hello @loubnabnl, sure we can close this as I haven't started on it. Thank you.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: investigate discrepancy in odex implementation
Root cause: Algorithmic Error

This [PR](https://github.com/bigcode-project/bigcode-evaluation-harness/pull/45) adds [Odex](https://github.com/zorazrw/odex) benchmark to the evaluation harness, however there is a discrepancy in pass@1 for codegen-2B-mono (37% vs 41%) between the two implementations as explained in this [comment](https://github.com/bigcode-project/bigcode-evaluation-harness/pull/45#issuecomment-1529013447)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Add SantaCoder FIM task
Root cause: Unimplemented Feature Gap

Add support for this [FIM task](https://huggingface.co/datasets/bigcode/santacoder-fim-task) discussed in this [issue](https://github.com/bigcode-project/bigcode-evaluation-harness/issues/33) on HumanEval and make sure numbers match with MultiPL-E [implementation](https://github.com/nuprl/MultiPL-E/blob/fc2a0615f030c4981604a57c859fd6656eec89f3/bigcode15b.py#L86) for santacoder for example (see table 7 of this [paper](https://arxiv.org/pdf/2301.03988.pdf))
The evaluation-harness already supports FIM mode for santacoder and incoder which is used by DS-1000 task for [insertion mode](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/c3b9717276677d885693290af67a86cd91c3f15c/lm_eval/tasks/ds1000.py#L134)

There is standalone code to do this here:

https://github.com/arjunguha/santacoder_fim_benchmark

It would be great if someone tried to integrate it into the eval harness. :)

@arjunguha looking to implement this. do you have the generation args for SantaCoder used to get the results in table 7? is it the same as
```
GENERATION_ARGS = {
    "do_sample": True,
    "temperature": 0.2,
    "top_p": 0.95,
    "max_new_tokens": 25,
}
```
 Thanks 😄 

yes same ones in the file, which is that
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Llama 7B fails for Human Eval
Root cause: Configuration Error

Running `human_eval` with Llama 7B gets 0 for pass@1,10 but it does achieve the correct values (pass@1 ~ 10) in other repos.

To reproduce, simply run 

```
accelerate launch  main.py \
  --model huggyllama/llama-7b \
  --max_length_generation 512 \
  --tasks humaneval \
  --temperature 0.2 \
  --n_samples 20 \
  --batch_size 10 \
  --allow_code_execution
```
which returns 
```  
"humaneval": {
    "pass@1": 0.0,
    "pass@10": 0.0
  },
```

I reduced `n_samples` from 200 to 20 to make this run in about 1 hour on a single A100. 

Running the same `human_eval` with [CodeCapybara](https://github.com/FSoft-AI4Code/CodeCapybara)'s repo gets the correct values ` {'pass@1': 0.09542682926829267, 'pass@10': 0.12530930709402355}`

Could be related to https://github.com/huggingface/transformers/pull/22402 although I tried explicitly setting eos, bos, pad token ids same as CodeCapybara (see [here](https://github.com/FSoft-AI4Code/CodeCapybara/blob/master/main/generate.py#L114)) and didn't see a change so might be something else.

If anyone has successfully run it here, would appreciate some tips!

Can you check what the generations look like to see if there's a failure pattern? You can save them by calling `--save_generations`. 
In [CodeCapybara](https://github.com/FSoft-AI4Code/CodeCapybara) did you also use `huggyllama/llama-7b` or the one they use in the repo `decapoda-research/llama-7b-hf` ?

Closing this issue as it was fixed with https://github.com/bigcode-project/bigcode-evaluation-harness/pull/81
Your command now returns
```python
{
  "humaneval": {
    "pass@1": 0.10518292682926832,
    "pass@10": 0.1760411160613154
  },
  "config": {
    "model": "huggyllama/llama-7b",
    "temperature": 0.2,
    "n_samples": 20
  }
}
```
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Problem launching evaluation
Root cause: Validation Gap

Hi, I am trying to run the evaluation of Santacoder using the script provided, but I am getting the following error which I am not able to find out why:

    File "/home/kcdharma/ndec/eval/bin/accelerate", line 8, in <module>
      sys.exit(main())
    File "/home/kcdharma/ndec/eval/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 43, in main
      args.func(args)
    File "/home/kcdharma/ndec/eval/lib/python3.10/site-packages/accelerate/commands/launch.py", line 910, in launch_command
      simple_launcher(args)
    File "/home/kcdharma/ndec/eval/lib/python3.10/site-packages/accelerate/commands/launch.py", line 397, in simple_launcher
      process = subprocess.Popen(cmd, env=current_env)
    File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "/usr/lib/python3.10/subprocess.py", line 1762, in _execute_child
      env_list.append(k + b'=' + os.fsencode(v))
    File "/usr/lib/python3.10/os.py", line 810, in fsencode
      filename = fspath(filename)  # Does type-checking of `filename`.
  TypeError: expected str, bytes or os.PathLike object, not NoneType

Any help is appreciated.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: requirements.txt doesn't support newer models (KeyError)
Root cause: Configuration Error

(Related issue here: #73)
The requirements.txt file lists `transformers==4.25.1`, which doesn't support a lot of the newer models such as `bigcode/starcoder` and `huggyllama/llama-7b` (it gives errors such as `KeyError: 'gpt_bigcode'`). This should be simple to fix from the user side (just install a newer version of `transformers`), but just thought I'd flag it here since it's probably best if the requirements.txt can accommodate these newer models.

Thank you for flagging this! should be fixed with this [PR](https://github.com/bigcode-project/bigcode-evaluation-harness/pull/79)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Any plan for attaching release tag?

I would like to express my sincere gratitude for the ease of code eval provided by your repository. 
I see that tasks and features are being added rapidly thanks to lots of contributors.
Do you have any plan for attaching release tags for version control?

Thank you very much for the feedback! We can definitely add release tags if there's a lot of activity

Could you release the tag as of now? I want to fork the repo and manage the upstream based on the released version.

Done: https://github.com/bigcode-project/bigcode-evaluation-harness/releases/tag/v0.1.0

Thank you so much for the release. I'll try to contribute from the next version!