zhiminy/EvalEng huggingface dataset:
Repository: bigcode-project/bigcode-evaluation-harness
Issue: why change n_copies from 1 to 2?
Root cause: Algorithmic Error

【Related Code】
bigcode_eval/utils.py
```python3
class TokenizedDataset(IterableDataset):
    def __iter__(self):
        ...
        if self.n_copies == 1 and self.n_tasks % self.num_devices != 0:
            self.n_copies = 2
            warnings.warn(
                "n_copies (n_samples/batch_size) was changed from 1 to 2 because n_tasks isn't proportional to num devices"
            )
        ...
```

【My Setting】
```python
n_tasks = 164  # humaneval
num_devices = 5
n_samples = 1
batch_size = 1
```

【Results】
I get the  `UserWarning: n_copies (n_samples/batch_size) was changed from 1 to 2 because n_tasks isn't proportional to num devices`.
And the harness generated `2 samples` for `163 tasks`，and `4 samples` for `1 tasks` before `removed extra predictions to only keep nsamples=1`。
Why not keep `n_copies=1`，and just generate `1 samples` for `163 tasks` and `2 samples` for `1 tasks`？It make ```n_tasks = 1 * 163 + 2 * 1 = 165```, which also ensure ```n_tasks % num_devices == 0```.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: 请问使用vllm评测时怎么实现类似HF多卡数据并行？
Root cause: Unimplemented Feature Gap

我在评测时的模型type 为vllm，参数如下：
![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/97608046/59bcfd6b-07a7-4cef-9882-7ce56ef68479)
但是显卡占用只使用了一张卡来评测任务
![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/97608046/7ed09917-5a6b-4f5d-9b3e-b933f96eed8f)
我想让任务划分为几份分别在8张卡上评测，这样如何实现呢？
类似我如果设定为模型type为HF的话，会自动达到这种效果。
![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/97608046/e2a6c0dc-5796-465c-b0fe-31b98d034240)
![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/97608046/51d5925b-830d-43a3-a1da-1e168830c65d)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Check pass/fail count for humaneval 
Root cause: Documentation Deficiency

Im running human eval for **codellama/CodeLlama-7b-Instruct-hf** model using the folllwing command

`accelerate launch  main.py   --model codellama/CodeLlama-7b-Instruct-hf   --max_length_generation 512   --tasks humaneval   --temperature 0.2   --n_samples 1   --batch_size 1 --precision fp16 --load_in_4bit --allow_code_execution --save_generations --save_references --limit 5`

Currently I can only access the final score of my test in the file **evaluation_results.json**
```
{
  "humaneval": {
    "pass@1": 0.8
  },
  "config": {
    "prefix": "",
    "do_sample": true,
    "temperature": 0.2,
    "top_k": 0,
    "top_p": 0.95,
    "n_samples": 1,
    "eos": "
",
    "seed": 0,
    "model": "codellama/CodeLlama-7b-Instruct-hf",
    "modeltype": "causal",
    "peft_model": null,
    "revision": null,
    "use_auth_token": false,
    "trust_remote_code": false,
    "tasks": "humaneval",
    "instruction_tokens": null,
    "batch_size": 1,
    "max_length_generation": 512,
    "precision": "fp16",
    "load_in_8bit": false,
    "load_in_4bit": true,
    "left_padding": false,
    "limit": 5,
    "limit_start": 0,
    "save_every_k_tasks": -1,
    "postprocess": true,
    "allow_code_execution": true,
    "generation_only": false,
    "load_generations_path": null,
    "load_data_path": null,
    "metric_output_path": "evaluation_results.json",
    "save_generations": true,
    "load_generations_intermediate_paths": null,
    "save_generations_path": "generations.json",
    "save_references": true,
    "save_references_path": "references.json",
    "prompt": "prompt",
    "max_memory_per_gpu": null,
    "check_references": false
  }
}

```

I need to check for which all problems, my model generated the correct code(unit tests passed) & for which all problems the testcases failed(and see what was the output for the tests)

Hi you will need to change this line to save the details in addition to the score, now it's an empty variable https://github.com/bigcode-project/bigcode-evaluation-harness/blob/094c7cc197d13a53c19303865e2056f1c7488ac1/bigcode_eval/tasks/humaneval.py#L98
For example:
```python
results, details = compute_code_eval(..)
details.to_json("details.json")
```
See doc:
https://github.com/bigcode-project/bigcode-evaluation-harness/blob/094c7cc197d13a53c19303865e2056f1c7488ac1/bigcode_eval/tasks/custom_metrics/code_eval.py#L78
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Please add flag to log score for each sample (akin to Eleuther's LM Evaluation Harness)
Root cause: Unimplemented Feature Gap

Hi! I've been using EleutherAI's LM Evaluation Harness and I'd like to be able to also runs some code tasks using your Big Code Evaluation Harness. We need the scores for each sample in each benchmark and the LM Evaluation Harness has a helpful flag `log_samples` that activates logging the per-sample scores.

As best as I can tell (and please correct me if I'm wrong), Big Code's Evaluation Harness doesn't have a similar flag. If my understanding is correct, could this please be added?

Thank you!

If making this change is relatively simple, you can let me know how to make the change and I can open a PR?

Hi, I encounter the same need. 

I find that the code can be easily modified to log the per-sample score.

Take the task **humaneval** as an example, the second return value in compute_code_eval (called in process_result: [link](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/094c7cc197d13a53c19303865e2056f1c7488ac1/bigcode_eval/tasks/humaneval.py#L98)) will return the logs (including whether the output is passed) of sample as a list. 

In the original implementation, they just use a `_` to accept the returned logs. If you want to output the logs to a file, you can just save the value by `json.dump`.

Yes as @shuhao02 explained and as I mention in this issue you can dump the results returned by `code_eval` metric https://github.com/bigcode-project/bigcode-evaluation-harness/issues/211#issuecomment-2027100342 
```python
results, details = compute_code_eval(..)
details.to_json("details.json")
```
Feel free to make a PR to add a flag for the harness, but note that not all tasks use this metric
Repository: bigcode-project/bigcode-evaluation-harness
Issue: run the MBPP in the HumanEval data format 
Root cause: Configuration Error

Hello. thk you for your prior help! i want to run the MBPP dataset in the HumanEval data format.  But I found that when i do this, the generation repeat until to the max_length ,but in HumanEval dataset it doesnt happen. so I wander if there is something I shoud change .  perhaps the postprocess?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: The results of codellama-7b-hf pass@1 is worse than paper
Root cause: Configuration Error

This is my commands:
```bash
accelerate launch  main.py \
  --model codellama/CodeLlama-7b-hf \
  --tasks humaneval \
  --max_length_generation 512 \
  --do_sample False \
  --n_samples 1 \
  --batch_size 1 \
  --precision bf16 \
  --allow_code_execution \
  --trust_remote_code \
  --save_generations
```

This is my results:
<img width="546" alt="image" src="https://github.com/bigcode-project/bigcode-evaluation-harness/assets/37740540/f63353d3-aa9c-40dc-b7f6-97362a5a931c">

So, where is my wrong?


Hi, that's also the score we got for the [BigCode leaderboard](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard), it's still in a similar range to what's reported in the paper (29.8 vs 32.3). The difference could be due to using different post-processing or inference settings.

Thanks for your reply. 

I use the exact same command, yet my pass@1 score is 0. 

Command:
```bash
accelerate launch  main.py   \
--model codellama/CodeLlama-7b-hf   \
--tasks humaneval   \
--max_length_generation 512   \
--do_sample False  \
 --n_samples 1   \
--batch_size 1   \
--precision bf16   \
--allow_code_execution   \
--trust_remote_code   \
--save_generations
```

Output:

```json
{
  "humaneval": {
    "pass@1": 0.0
  },
  "config": {
    "prefix": "",
    "do_sample": false,
    "temperature": 0.2,
    "top_k": 0,
    "top_p": 0.95,
    "n_samples": 1,
    "eos": "
",
    "seed": 0,
    "model": "codellama/CodeLlama-7b-hf",
    "modeltype": "causal",
    "peft_model": null,
    "revision": null,
    "use_auth_token": false,
    "trust_remote_code": true,
    "tasks": "humaneval",
    "instruction_tokens": null,
    "batch_size": 1,
    "max_length_generation": 512,
    "precision": "bf16",
    "load_in_8bit": false,
    "load_in_4bit": false,
    "left_padding": false,
    "limit": null,
    "limit_start": 0,
    "save_every_k_tasks": -1,
    "postprocess": true,
    "allow_code_execution": true,
    "generation_only": false,
    "load_generations_path": null,
    "load_data_path": null,
    "metric_output_path": "evaluation_results.json",
    "save_generations": true,
    "load_generations_intermediate_paths": null,
    "save_generations_path": "generations.json",
    "save_references": false,
    "save_references_path": "references.json",
    "prompt": "prompt",
    "max_memory_per_gpu": null,
    "check_references": false
  }
}
```

What could the potential issues be?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Multiple-E Go test file name suffix does not contain _test.go
Root cause: Validation Gap

Hi Team,
I was using the bigcode-evaluation-harness to evaluate generation for go on Multiple-E dataset and found that, all the evaluation had output `?   	command-line-arguments	[no test files]` although `status_code = 0`. 
On debugging further, it looks like [we set self.language here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/bigcode_eval/tasks/multiple.py#L153) instead of `prompt_name['langugage']` in the problem dict to process execution downstream, and when [language is checked in evaluators here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/bigcode_eval/tasks/custom_metrics/multiple_metrics/containerized_eval.py#L47), it is appended without `_test.go` suffix leading to non detecting any test files.

To make it easy to repro this, I have added a video below which evaluate one go generation test case (used deepseek coder to generate this)

`generations_go_example.json`

```
[
	[
        "package strlen_test\n\nimport (\n    \"testing\"\n    \"fmt\"\n)\n\n// Return length of given string\n// >>> strlen(\"\")\n// 0\n// >>> strlen(\"abc\")\n// 3\nfunc strlen(myString string) int {\n    return len(myString)\n}\n"
    ]
]
```

https://github.com/bigcode-project/bigcode-evaluation-harness/assets/20701220/c57dd498-b7f8-488a-a842-f9eb405f1f0d
Repository: bigcode-project/bigcode-evaluation-harness
Issue: [FR] include "config" data in generations_only
Root cause: Unimplemented Feature Gap

feature proposal/request to include the "config" args when doing generations_only

### Motivation

I run generations on several remote systems and then run the evaluations locally on my workstation. Currently I have to keep track of my exact parameters for the generations by the filename alone. And then run a really long command as well with all the same parameters for the evals.

**pros**
- never mix up runs
- don't use filenames for data
- could be much easier to run eval only (just have to give it a .json that already includes the task, limit, model, etc)

**cons**
- likely breaks some existing downstream scripts
- tiny bit of redundant data

### relevant code
It seems to be the case that in generation_only mode, you only return save_generations. So the args are lost.
https://github.com/bigcode-project/bigcode-evaluation-harness/blob/1b0147c50f406ff66ac4f806230479f31d19c7e6/main.py#L400-L408
the `generations.json` is just a list of list of strings. But it could easily hold the same config as the `eval_results.json`.
You would also need a bit of code to read these args in eval_only mode.
Also in the case of there being a crash during the eval run - you will have generations but no eval results saved.

Alternatively there could be a config.json file that keeps track of these when in generation_only (or an empty eval_results)... but that still leaves you with multiple files - instead of having all of it contained at once.


***

bit of an RFC, before I try and implement this in a PR myself. Especially on how to make it compatible with existing formats
Repository: bigcode-project/bigcode-evaluation-harness
Issue: ignore --use_auth_token if model doesn't require it
Root cause: Validation Gap

Came across this today, and seems like it would be a little improvement. 


if `--use_auth_token` is set but no token is available (not logged in to huggingface-cli) the program will aborts with 
```py
huggingface_hub.utils._headers.LocalTokenNotFoundError: Token is required (`token=True`), but no token found.
```
Even if the model specified is not gated. So should this arg be ignored and the model loaded anyway? Essentially making it okay to always include this arg. However this somehow makes this arg redundant.

can gladly open a PR to try and implement this

or is this actual behaviour that could be addressed in transformers itself?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: The results of Llama3-8b pass@1 is worse than report
Root cause: Configuration Error

![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/56470984/fe48b9da-6832-4ee7-8ae3-cb77d07bf854)


I think the LLama3 blog post evaluates the instruct model in the blogpost not the base model. Your pass@1 is close to the numbers that the community is reporting for llama3-8B-base. See: https://twitter.com/huybery/status/1781172838361334015

Despite using Llama-8b-base, I scored 31.71 on HumanEval, which is lower than their 33.50.

I scored 57.32 on Llama-3-8B-Instruct.
![image](https://github.com/bigcode-project/bigcode-evaluation-harness/assets/56470984/7e6f4aa8-287b-49f8-9233-83eef1d811e2)


`n_samples` looks too small to get a generalized result. Try again with `n_samples >= 50`

Indeed, if you're using `n_samples=1` set the generation to greedy `do_sample=False`. To use sampling (which generalizes better with a high number of samples)  set `n_samples`  to 50. But note that there might be evaluation differences if the results are from different setups/frameworks.

I also scored 57.32 on llama3-8b-instruct, which differs from the 62.2% reported in the llama3 blog. This discrepancy could be due to the prompt template I used. Since generating 50 samples would be slow, I opted for 10 instead.
Evaluating generations...
{
  "humaneval": {
    "pass@1": 0.573170731707317,
    "pass@10": 0.6524390243902439
  },
  "config": {
    "prefix": "",
    "do_sample": true,
    "temperature": 0.1,
    "top_k": 0,
    "top_p": 0.95,
    "n_samples": 10,
    "eos": "
",
    "seed": 0,
    "model": "/huggingface/hub/models--meta-llama--Meta-Llama-3-8B-Instruct/snapshots/e5e23bbe8e749ef0efcf16cad411a7d23bd23298",
    "modeltype": "causal",
    "peft_model": null,
    "revision": null,
    "use_auth_token": false,
    "trust_remote_code": false,
    "tasks": "humaneval",
    "instruction_tokens": null,
    "batch_size": 10,
    "max_length_generation": 512,
    "precision": "fp32",
    "load_in_8bit": false,
    "load_in_4bit": false,
    "left_padding": false,
    "limit": null,
    "limit_start": 0,
    "save_every_k_tasks": -1,
    "postprocess": true,
    "allow_code_execution": true,
    "generation_only": false,
    "load_generations_path": null,
    "load_data_path": null,
    "metric_output_path": "evaluation_results.json",
    "save_generations": true,
    "load_generations_intermediate_paths": null,
    "save_generations_path": "generations.json",
    "save_references": false,
    "save_references_path": "references.json",
    "prompt": "prompt",
    "max_memory_per_gpu": null,
    "check_references": false
  }
}
Repository: bigcode-project/bigcode-evaluation-harness
Issue: If I want to add my own designed prompts before each question, how should I modify the code
Root cause: Documentation Deficiency


You can try using the `--prefix` argument which adds a prefix to the prompts, but be careful it might influence the generations style if you're using instruct models (e.g generating some text before the actual code which will cause the test to fail with the default postprocessing).

We have an "instruction-tuning" version of HumanEval under [HumanEvalPack](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/docs#humanevalpack) which supports different prompt templates depending on the models, example [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/a1b4a7949a24c8e3ef0d05a01097b2d14ffba56e/bigcode_eval/tasks/humanevalpack.py#L242)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Make `bigcode_eval` pip installable
Root cause: Configuration Error

Is it possible to make this package pip installable? Currently, when I pin some commit in my `requirements.txt` file,

`bigcode_eval @ git+https://github.com/bigcode-project/bigcode-evaluation-harness.git@a1b4a7949a24c8e3ef0d05a01097b2d14ffba56e`

I see the following error:

```
ERROR: Could not find a version that satisfies the requirement bigcode-eval (unavailable) (from versions: none)
ERROR: No matching distribution found for bigcode-eval (unavailable)
```

Is this because the `setup.py` doesn't have a package name specified?
https://github.com/bigcode-project/bigcode-evaluation-harness/blob/a1b4a7949a24c8e3ef0d05a01097b2d14ffba56e/setup.py#L30-L37

I couldn't find the published package on PyPi either.

Perhaps the error is due to mismatching Python version. I'm using python3.8 in my environment, but it's unclear what the required Python version is for bigcode. Is it possible to make the requirement explicit and streamline the package installation process? Thanks.

From the pip install logs, I also see this:
```
Collecting bigcode-eval@ git+https://github.com/bigcode-project/bigcode-evaluation-harness.git (from -r /tmp/tmpj0nno88m (line 126))
  Cloning https://github.com/bigcode-project/bigcode-evaluation-harness.git to /tmp/pip-install-2cfwowhe/bigcode-eval_e7397c9bd97643f09a0059e0caf044ae
  Running command git clone --filter=blob:none --quiet https://github.com/bigcode-project/bigcode-evaluation-harness.git /tmp/pip-install-2cfwowhe/bigcode-eval_e7397c9bd97643f09a0059e0caf044ae
  Resolved https://github.com/bigcode-project/bigcode-evaluation-harness.git to commit a1b4a7949a24c8e3ef0d05a01097b2d14ffba56e
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
  WARNING: Generating metadata for package bigcode-eval produced metadata for project name unknown. Fix your #egg=bigcode-eval fragments.
```

It seems that the call to `setup` must have name specified ([see similar issue](https://github.com/pypa/pipenv/issues/5445)), i.e.

```
 setup( 
     name="bigcode_eval", 
     description="A framework for the evaluation of autoregressive code generation language models.", 
     long_description=readme, 
     license="Apache 2.0", 
     packages=find_packages() , 
     install_requires=requirements, 
     extras_require={"ds1000": ds1000_requirements}, 
 ) 
```
Repository: bigcode-project/bigcode-evaluation-harness
Issue: The result of llama2-13b-chat(pass@1 18) is worse than paper(pass@1 37)
Root cause: Documentation Deficiency

{
  "humaneval": {
    "pass@1": 0.18658536585365854,
    "pass@10": 0.2073170731707317
  },
  "config": {
    "prefix": "",
    "do_sample": true,
    "temperature": 0.1,
    "top_k": 0,
    "top_p": 0.95,
    "n_samples": 10,
    "eos": "
",
    "seed": 0,
    "model": "huggingface/hub/models--meta-llama--Llama-2-13b-chat-hf/snapshots/a2cb7a712bb6e5e736ca7f8cd98167f81a0b5bd8",
    "modeltype": "causal",
    "peft_model": null,
    "revision": null,
    "use_auth_token": false,
    "trust_remote_code": false,
    "tasks": "humaneval",
    "instruction_tokens": null,
    "batch_size": 10,
    "max_length_generation": 512,
    "precision": "fp16",
    "load_in_8bit": false,
    "load_in_4bit": false,
    "left_padding": false,
    "limit": null,
    "limit_start": 0,
    "save_every_k_tasks": -1,
    "postprocess": true,
    "allow_code_execution": true,
    "generation_only": false,
    "load_generations_path": null,
    "load_data_path": null,
    "metric_output_path": "evaluation_results.json",
    "save_generations": true,
    "load_generations_intermediate_paths": null,
    "save_generations_path": "generations.json",
    "save_references": false,
    "save_references_path": "references.json",
    "prompt": "prompt",
    "max_memory_per_gpu": null,
    "check_references": false
  }
}


Hi, in which section did you find the 37 score? I only found the number of Llama2-13B-base
<img width="674" alt="image" src="https://github.com/bigcode-project/bigcode-evaluation-harness/assets/44069155/6ad2bae2-9afb-433a-9e1c-2c62f912c1db">

Instruct models usually need a chat format of the benchmark for optimal performance, we use `humanevalsynthesize-python` for this matter, you can find an example here https://github.com/bigcode-project/bigcode-evaluation-harness/issues/158#issuecomment-1792691690 (you might need to change the [prompt template](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/84b96da31b7f840b55c5733325346176140cdb6b/bigcode_eval/tasks/humanevalpack.py#L248) I don't think the existing ones work with llama2) 


I made a mistake and mistook the GSM8K data for HumanEval data, I'm very sorry. The data for llama2-13b-chat can be found in the last graph of figure 3 at https://arxiv.org/abs/2402.05120. The footnote on page four clarifies that it's llama2chat rather than base. According to the graph, the score is roughly around 16%. Your framework scored around 18%, which is even better, that's impressive!
Repository: bigcode-project/bigcode-evaluation-harness
Issue: API-based evaluation support (humanevalpack_openai.py is too old)
Root cause: Unimplemented Feature Gap

I am trying evaluate API base model .
（Local model working on vLLM Engine, and vLLM provide OpenAI Compatible API interface）

but tasks/humanevalpack_openai.py is not updated.
(postprocess_generation is not applied, save format is incompatible  etc)
So I can't pass generation results  to evaluate.

Do you have any plans to update it in the future?

If API-based evaluation is applied to all benchmarks, more models can be evaluated easily.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Run 70b evaluation.
Root cause: Documentation Deficiency

Hello, when I run the evaluation of the 70b model, I used 8*80G GPUs, but it still oom. How can I evaluate the the 70b model?

hello, I have met the same problem, may I ask how you solve it?

You can set `--max_memory_per_gpu` [flag](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/0f3e95f0806e78a4f432056cdb1be93604a51d69/main.py#L202) to `auto` and it will shard the model on the available gpus.  You can also reduce the batch size to 1, and use bf16 `--precision bf16` and if it still OOMs, you can try reducing`max_length_generation`
Repository: bigcode-project/bigcode-evaluation-harness
Issue: How can I pass num_beams?
Root cause: Unimplemented Feature Gap

Cannot find a way to pass num_beams. Thanks for your help!

Hi there's no flag for `num_beams` as we either use greedy or top-p sampling but you can change the code to add it here https://github.com/bigcode-project/bigcode-evaluation-harness/blob/f0f2b52ab0bac95b7fa881693e82c5ddcf2f9e95/bigcode_eval/generation.py#L68

Thanks!