zhiminy/EvalEng huggingface dataset:
Repository: bigcode-project/bigcode-evaluation-harness
Issue: MBPP eval extremely slow for CodeGen2 and Replit-Code
Root cause: External Dependency Breakage

Hi, I have been trying to evaluate CodeGen2 and Replit-Code models on the mbpp task, but the code runs extremely slow. While the corresponding eval time for other models is around 2 hours, the ETA for these 2 models varies significantly and sometimes goes up to > 90 hrs. Any help to resolve this issue? Thanks!

The Replit [model](https://huggingface.co/replit/replit-code-v1-3b) seems slow because the `use_cache` argument in their [config](https://huggingface.co/replit/replit-code-v1-3b/blob/main/config.json#L49) is set to false, you can try cloning it and changing it to true before you run inference, I opened a PR on their [repo](https://huggingface.co/replit/replit-code-v1-3b/discussions/23). For CodeGen2 which model exactly are you running?

Also what batch size and number of GPUs are you using? You can also try increasing the batch size to speed things up


Hi @loubnabnl, thanks for your reply! I am running the CodeGen2-3.7B model with a batch size of 10 and a single GPU. Here is the exact command which I am using:

```
accelerate launch main.py \
  --model Salesforce/codegen2-3_7B \
  --tasks mbpp \
  --temperature 0.1 \
  --n_samples 15 \
  --batch_size 10 \
  --allow_code_execution \
  --save_generations \
  --metric_output_path codegen2-3.7B-results.json \
  --save_generations_path codegen2-3.7B-generations.json \
  --trust_remote_code 
```

Regarding Replit  model you should be able to run evaluation in ~2h on 1 gpu in full precision. You can try fp16 or bf16 in `--precision` argument to speed things up.

As for CodeGen2, if the model inference is slow there's not much we can do about it from the evaluation harness perspective, since the same command runs fast for other models, you can try measuring the tokens/sec speed and contacting the model's authors.

I confirm that inference with the codegen2 series is extremely slow compared to other models of the same size. Codegen2.5 (7B) on the other hand is fast (but based on a different architectures)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: When I evaluated the dataset APPS, I got the error RuntimeError: stack.size() >= frames.back().function->n_inputs INTERNAL ASSERT FAILED
Root cause: Environment Incompatibility

I've tried many nodes and this error is reported. According to this [link](https://github.com/pytorch/pytorch/issues/30494) it seems that the torch version needs to be upgraded, but the highest supported torch for python 3.7 is 1.13.1, so it looks like this is a dead end. How do I avoid this problem, since someone has successfully reviewed it? My evaluation command is as follows
`accelerate launch main.py \
    --model bigcode/starcoder \
    --tasks apps-introductory \
    --max_length_generation 2048 \
    --temperature 0.8 \
    --n_samples 1 \
    --batch_size 32 \
    --save_generations \
    --precision bf16 \
    --save_generations_path generations.json \
    --metric_output_path evaluation_results.json \
    --allow_code_execution `
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Consider a refactoring

Before adding more tasks it could be a good time to take a step back and see if it makes sense to do a bit of refactoring of the code. A few aspects to consider:

- how can we make it as easy as possible to add new metrics. it's possible that we may want to add a few dozen more datasets each with some quirks. we can look at other frameworks like the lm-evaluation-harness to see how it's done there and if it make sense to build on top of it or just take inspiration. e.g. i think it would be nice if adding a new evaluation would require changes in as few places as possible.
- going for multilinguality we might need to run the code execution in different environments. maybe we should decouple generation and execution by saving the results on disk in between.
- for the execution part we probably will need to think about docker environments to execute code in different frameworks.

These are just a few thoughts, let me know if you think this makes sense @loubnabnl. 

Summary of what we discussed:
* Will be looking into how the current framework and tasks integrate with lm-evaluation-harness.
* Regarding docker environements, we can do something similar to what’s done in the [HumanEval-X benchmark ](https://github.com/THUDM/CodeGeeX/tree/main/codegeex/benchmark  )
* The generations can currently be saved on disk, I will add an option to disable the evaluation.

Refactoring in PR https://github.com/bigcode-project/bigcode-evaluation-harness/pull/19:
* build tasks in separte files like in [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)
* separate generation from evaluation
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Reproduction of HELM results
Root cause: Documentation Deficiency

How do you evaluate bigcode models on HELM benchmark?

Do you directly using their `crfm-helm` tools?

If so, can you release your bash commands for the community to rigorously reproduce your results?

Sorry we don't have the commands, members from the HELM team did the evaluation, but I believe they used the default as for other code models on reasoning tasks. Maybe you can open the issue on their repo.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Update MultiPL-E docker image
Root cause: External Dependency Breakage

The published images use a build of the harness that fails for some MultiPL-E languages (was fixed in https://github.com/bigcode-project/bigcode-evaluation-harness/pull/112 and https://github.com/bigcode-project/bigcode-evaluation-harness/pull/99)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: installation issue with pip install git+https://github.com/bigcode-project/bigcode-evaluation-harness.git
Root cause: Validation Gap

Hi, 
I would like to install with the following method, but it doesn't work.
```
pip install git+https://github.com/bigcode-project/bigcode-evaluation-harness.git
```
I get the error
```
(ipex_latest) [changwa1@aia-sdp-spr-117706 ~]$ python
Python 3.8.13 (default, Mar 28 2022, 11:38:47)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from  lm_eval import tasks
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'tasks' from 'lm_eval'
```
the root cause is `setup.py` file only write the `packages=["lm_eval"]` folder. there is no `tasks` folder in my env `site-packages` folder, so it doesn't find.
```
(ipex_latest) [changwa1@aia-sdp-spr-117706 bigcode-evaluation-harness]$ cd /home/changwa1/anaconda3/envs/ipex_latest/lib/python3.8/site-packages/lm_eval/
(ipex_latest) [changwa1@aia-sdp-spr-117706 lm_eval]$ ls
arguments.py  base.py  evaluator.py  generation.py  __init__.py  __pycache__   utils.py

```
code issue is here.
```
setup(
    description="A framework for the evaluation of autoregressive code generation language models.",
    long_description=readme,
    license="Apache 2.0",
    packages=["lm_eval"],
    install_requires=requirements,
    extras_require={"ds1000": ds1000_requirements},
)
```
the fixed PR, please review and give comments. PR #116 

@loubnabnl @arjunguha could you please help review? Thanks.

Fixed in the mentioned PR, thanks!
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Different stop words between humaneval and instruct humaneval
Root cause: Documentation Deficiency

Hi, thanks for adding the instruct_humaneval task.
Why the stop words are different between humaneval and instruct_humaneval tasks?
stop words for humaneval are ["\nclass", "\ndef", "\n#", "\n@", "\nprint", "\nif"],
while those for instruct_humaneval are ["if __name__", "\nprint", "\nclass"].


Hi. The stop words for HumanEval (HE) are mostly inherited from the [Codex paper](https://arxiv.org/pdf/2107.03374.pdf). They prevent the model from generating irrelevant text after the code it is supposed to complete. Now, for instruct_humaneval (IHE), the idea is similar. You'll notice that some stop words were removed compare to HE, namely `\n@`, `\n#` but actually we could have kept them. IHE also handles the case where we ask the LM to generate a solution just from the docstring (**Not Code Completion**). In such case we can not have `\ndef` as a stop word. Also, the model can additionally generate test cases with the help of `if __name__ == "__main__"` or it can try to embed the implementation in a class (e.g. `class Solution`). These are some scenarios that we want to avoid, because the evaluation only cares about a function implementation. 

TL; DR : compare to HE, `if __name__` kind of replaces `\nif`, `\ndef` can not be used as stop word (explained earlier), `\print` and `\nclass` are kept. We found that `\n@` and `\n#` did not occur much in failure cases, but they can be kept also.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: MultiPL-E Integration
Root cause: Unimplemented Feature Gap

As part of the integration of [MultiPL-E](https://github.com/nuprl/MultiPL-E) benchmark create Dockerfile/Docker image with all dependencies required to execute the code generations for different programming languages 

Tagging @ytzi and @canders1. Let's do this together. I actually have a little evaluation container [here](https://github.com/nuprl/MultiPL-E/tree/only_code/evaluation_container)  in progress, that only supports Racket (never mind why!)

Here my high-level sketch:

1. We build a container that runs with `--network none` and supports all 18+ languages. (Let's also get some of the others to work that are in the Stack.)
2. It receives the name of a directory/file that is read-only that contains JSON completions. ([example file])(https://github.com/nuprl/MultiPL-E/blob/mbpp/experiments/go-davinci-0.2-keep/HumanEval_0_has_close_elements.json)
3. It receives the name of a directory (writable) where it emits the result of a run. ([example file])(https://github.com/nuprl/MultiPL-E/blob/mbpp/experiments/go-davinci-0.2-keep/HumanEval_0_has_close_elements.results.json)

I *think*, Docker/Podman will allow both the directories above to be the same directory if desired.

I think we should do this in a separate repository that omits the analysis scripts, inference running, datasets, and results. BigCode will have their own analyses/inference approach.

Yes it makes sense to do this in a separate repository (maybe use the original MultiPL-E repo?), we can use this issue to discuss and track the progress of the task. 

In any case we will need to add the metrics to [evaluate](https://huggingface.co/docs/evaluate/index) and to this repo, and then add the Dockerfile when it's ready to the setup instructions

This is now in progress in here:

https://github.com/nuprl/MultiPL-E/tree/only_code/evaluation

When it is a little more mature, we will upload a container image. For now, you have to build it yourself. The makefile has commands to build the container and to test:

https://github.com/nuprl/MultiPL-E/blob/only_code/evaluation/Makefile#L5

(You can replace podman with docker, either should work.)

closing this issue as MultiPL-E was integrated in https://github.com/bigcode-project/bigcode-evaluation-harness/pull/44 🥳
Repository: bigcode-project/bigcode-evaluation-harness
Issue: bug in post processing for mbpp task
Root cause: Algorithmic Error

Hi, I found a bug in the post-processing for the mbpp task. This bug degrades the performance of StarCoder on the mbpp dataset. 

The `first_block` function is supposed to extract the code block by scanning the stop_words. However, `stop_words` is appended with `tokenizer.eos_token` in function `parallel_generations` (`generation.py`), and it includes `
` in the list.

Token `
` has `|`, which is interpreted by the regular expression and then following codes in `first_block` split based on `>`, which is wrong. I found some errors in the mbpp task are simply because of this. If this is fixed, the StarCoder performance can be better than before. 

```python
    def first_block(string, stop_words):
        """Split off first block of code by scanning for class, def etc. on newlines."""
        return re.split("|".join(stop_words), string)[0].rstrip()
```

The fix is straightforward. You can use `_stop_at_stop_token` function in `humaneval.py`. 
After this is fixed, the performance of StarCoder on mbpp task in the paper needs to be updated.


Thanks for the catch! Would you like to open a PR to update it? If not happy to do it

(The paper used MBPP from [MultiPL-E ](https://github.com/nuprl/MultiPL-E)which didn't have this bug)

I see. Thanks for the clarification. MBPP from MultiPL-E has modified the original MBPP. 

btw, I didn't see the MultiPL-E in this repo supporting MBPP yet. Is this correct?

Regarding the bug fix here, please go ahead. Thanks for the great work on this repo.

Issue fixed in https://github.com/bigcode-project/bigcode-evaluation-harness/pull/124
Regarding MultiPL-E, no MBPP isn't added yet.
Repository: bigcode-project/bigcode-evaluation-harness
Issue:  ModuleNotFoundError: No module named 'lm_eval.ds'
Root cause: Environment Incompatibility

from .ds.ds1000 import DS1000Dataset

*** ModuleNotFoundError: No module named 'lm_eval.ds'

Hi, could give more details on the command you're trying to run with the full stack trace. Also did you follow all the DS100 installation instructions [here](https://github.com/bigcode-project/bigcode-evaluation-harness#setup) it has more requirements that the other tasks (it could also be that you have a module named lm_eval from another library like the [LLM harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/master) so they are getting confused I would suggest having a separate environement)

This is solved, i was working in an offline manner, and had to manage the DS1000 dependencies correctly.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: StarCoderCommit Prompt
Root cause: Documentation Deficiency

Hi @Muennighoff, @loubnabnl,

StarCoder paper describe the commit format as:
```<commit_before>code<commit_msg>text<commit_after>code<eos>```

However, the [StarCoderCommit Prompt](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/tasks/humanevalpack.py#L219) does not have anything between `<commit_before>` and `<commit_msg>` tokens?

In my understanding, both instruction and code are part of the commit message (```{inp}```) in the current version of [StarCoder Prompt](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/tasks/humanevalpack.py#L219).
Thus the current version of StarCoder Prompt differs slightly from what was used during pre-training.

Is my understanding correct ?

Sorry, I should be looking at the prompt [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/6dafd67807366b226e5ff82dcc28ff5823384165/lm_eval/tasks/humanevalpack.py#L500)

Exactly, you need to look at the fix prompt.

The other part was for synthesize & explain where we tried doing <commit_before><commit_msg>text<commit_after> as there is no code before. Results are pretty bad in that format and not in the paper.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Cannot Reproduce SantaCoder pass@1 on HumanEval
Root cause: Configuration Error

Hi,

I tried reproducing the pass@1 result of the [SantaCoder](https://huggingface.co/bigcode/santacoder) model using this test suite on HumanEval. However, the pass@1 always be 0. 

I manually checked the model generations and found that the model always generates repeated nonsense tokens. For example, for the first test input:

```
"from typing import List\n\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n    \"\"\" Check if in given list of numbers, are any two numbers closer to each other than\n    given threshold.\n    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)\n    False\n    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)\n    True\n
```

and the model generates:

```
._._._.. is..Get..Get..)._._...._.).Can._........ is. is_._.._)...Get._ can._)_._ can_. can_ can_. is. is_). is.Get..___._ can_ is_._.Get can. is.GetCan_._._ can. can_ can. can can is. default_. is default_ can. == can. default_ can_ can_ can can can.Get). can. default_Get is_ can_ is. can. can_ default_ can. is can. is default_ default_ can. can_ default_ can_ can. can. is default_ can is default can default_ get_ default_ can can is default_ can can_ can. can. can default_Get can. can default can default_Get. can is default_ can default_Get can default_ can is default_ default_ can default_ default_ default_ is can can can is_ default_ can_ default is default_ default_ can default_GetGet, get_ default_Get). can can_ default_Get) can default_ can default_ can_ can. is default_ default_ default_ get_ can. default can can can_ default can default_ default_ default_ default_  default_ default_ default_ default_  default can can can can_ default_ default_ default_ default_ default_  can default_ default_ default_  can_ can_ default_ default_ can  default_  default_ default_ default_ can_ is default_  default_ can_ default_ get default_Get
```

My execution script is:
```bash
accelerate launch main.py \
		--model bigcode/santacoder \
		--max_length_generation 512 \
		--tasks humaneval \
		--precision bf16 \
		--temperature 0.2 \
		--top_k 0 \
		--top_p 0.95 \
		--n_samples 20 \
		--batch_size 20 \
		--seed 10 \
		--generation_only \
		--save_generations \
		--save_generations_path generations_py.json \
		--use_auth_token
```

I have also tested other models, such as `codellama/CodeLlama-7b-hf` and `meta-llama/Llama-2-7b-hf` using the exact same script above (except the model name), and I can reproduce the similar pass@1 score as reported in the papers.

Can you please help with this issue?

Thank you!

Ok, I forgot to add `--trust_remote_code`. After adding this command, the pass@1 score is reasonable.

Close this issue.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: code problem
Root cause: Configuration Error

How to specify multiple A100 models with larger parameters for this project when one A100 graphics memory is insufficient？

You can ask accelerate to use multiple GPUs when doing `accelerate config` , if your model doesn't fit in memory you can add the flag `--max_memory_per_gpu "auto"` and your model will be sharded on multiple GPUs. But ca you provide more details on which model are you running and how much RAM you have in your A100, most medium-sized models should fit with half-precision and a small batch size.

@loubnabnl 

I am running below command on VM which has 6 32GB Tesla v100 GPU card but is still failing with CUDA OOM

`accelerate launch  ./bigcode-evaluation-harness/main.py --model /data01/gemma-1.1-7b-it --task mbpp --max_length_generation 1000 --temperature 0.2 --n_samples 3 --batch_size 1 --trust_remote_code --generation_only --save_generations --save_generations_path /data01/code_eval/gemma-1.1-7b-it/gemma-1.1-7b-it_generations.json --max_memory_per_gpu "auto" --save_references --save_references_path /data01/code_eval/gemma-1.1-7b-it/gemma-1.1-7b-it_references.json --save_prompts_path /data01//code_eval/gemma-1.1-7b-it/gemma-1.1-7b-it_prompts.json `

Error:

`    new_value = value.to(device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 MiB. GPU 0 has a total capacity of 31.73 GiB of which 16.44 MiB is free. Process 39739 has 6.32 GiB memory in use. Including non-PyTorch memory, this process has 5.15 GiB memory in use. Process 39745 has 5.11 GiB memory in use. Process 39746 has 5.15 GiB memory in use. Process 39740 has 5.15 GiB memory in use. Process 39741 has 4.82 GiB memory in use. Of the allocated memory 4.85 GiB is allocated by PyTorch, and 1.95 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
`
what am i missing?
Repository: bigcode-project/bigcode-evaluation-harness
Issue: args.batch_size vs args.num_samples
Root cause: Documentation Deficiency

Looking at the implementation, it seems `batch_size` is playing the role of `num_samples`.

For instance, [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/generation.py#L111), the comment says that:
> do not confuse args.batch_size, which is actually the num_return_sequences

Similarly, I do not find `args.num_samples` being passed anywhere in calls to `model.generate()`.
Infact, calls to `model.generate()` in [utils.py](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/utils.py) set `num_return_sequences=batch_size`.

In contrast, [main.py](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/main.py#L83) describes batch size as 
> Batch size for evaluation on each worker, can be larger for HumanEval 

Is this a bug?

CC: @loubnabnl 

`args.n_samples` is the number of solutions you want to generate for each prompt however you can't always use it as `num_return_sequences` since it might OOM for large models for example, which is why we introduced the other argument `args.batch_size` that defines the effective number of sequences that you ask your model to generate, this is done until you we reach `args.n_samples` that the user requested defined by `n_copies` [here](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/68aea1805283b123a4e745e696e041c47cca4993/lm_eval/generation.py#L95C1-L95C1)

Thank you for the explanation, @loubnabnl !
Repository: bigcode-project/bigcode-evaluation-harness
Issue: code
Root cause: Resource Mishandling

When I ran Codellama-34b with two A100 cards, I was able to run it for the first time at first, but when I ran again later, I reported an error：ConnectionError: Tried to launch distributed communication on port `29500`, but another process is utilizing it. Please specify a different port (such as using the `----main_process_port` flag or specifying a different `main_process_port` in your config file) and rerun your script. To automatically use the next open port (on a single node), you can set this to `0`.

Hi, it seems like you have processes using the same GPUs, maybe run `nvidia-smi` to make sure they are free before launching the evaluation again
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Is it possible to run the harness against API hosted models?
Root cause: Unimplemented Feature Gap

I have a model that's only available through a RESTful API, and need to get some benchmarks. I'd like to run MultiPL-E benchmarks with a few languages. Has any work gone into using bigcode-evaluation-harness to perform generation with an API instead of on the local machine?

lm-evaluation-harness has the ability to run against commercial APIs, especially OpenAI. 

Hello, we currently don't support external APIs, only generations with `transformers`. Feel free to open a PR if you have something in mind.

If you're interested in HumanEvalPack benchmarks and OpenAI models there's a task that supports it(docs [here](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/docs#humanevalpack))https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/lm_eval/tasks/humanevalpack_openai.py

is there a way i could 'fake' a local model and have it call a hosted API endpoint? @pnewhook @loubnabnl 

I don't think that's possible with current setup which uses `transformers` loading that assumes you have the model checkpoint