Repository: evalplus/evalplus
Issue: Need help: my eval process is running quite slow
Root cause: Configuration Error

As described in `docs/execution.md`, the entire evaluation process should be ready in a few second with greedy decoding. However, I noticed the process has kept test script running in 13 hrs with `deepseek-6.7B-Base`. 
In my knowledge, it should call model.generate only once to calculate pass@1. But in the script `codegen.py` will call `model.codegen` and loop over `n_more_samples` times. Why do we need to this, is there any reason I missed?

Runtime env:
CPU cores: 4*26
GPU: NVIDIA A10 22G
evalplus: run in the local, I think I have got the latest version with git clone

script:
```shell
#!/usr/bin/bash
export PYTHONPATH="xxx/bin/python"
export HUMANEVAL_OVERRIDE_PATH="xxx/evalplus/evalplus/data/HumanEvalPlus.jsonl.gz"
export MBPP_OVERRIDE_PATH="xxx/evalplus/evalplus/data/MbppPlus.jsonl.gz"

python evalplus/evaluate.py --model "xxx/deepseek-coder-6.7b-base" --greedy --root "xxx/evalplus/eval_results" --dataset humaneval --backend hf --parallel 64
```
and it shows:
```
humaneval • 54% • •  88/164 • 13:47:34
```

Thanks in advance for your patience.


> But in the script codegen.py will call model.codegen and loop over n_more_samples times. 

I did get this. Can you clarify a bit more? e.g., adding some screen shot of the log may help.

looks like the generation is a bit slow. this might because you have a slow GPU, use a base model, and run on HuggingFace backend -- the EOS for the base model is sometimes limited that you are probably generating over the whole `max_new_tokens`.

My scripts `run_codegen.sh` is listed above, and here is the entire process:
<img width="1270" src="https://github.com/user-attachments/assets/e6cb0da6-8c21-4a74-a113-ad5d78f7f3e8">
<img width="719" src="https://github.com/user-attachments/assets/399ed9a5-cc01-4274-885f-5b8e48edd0ab">


Can you also paste one or two generated samples?

If they are consistently long, then the reason is likely to be what I just said.

Since the max_model_length of base model is 16K, I think is possibly to be the reason you said. So, can I limit the number of token generated with hf backend to speed up? I think most cases in humaneval is shorter than 16K tokens.
Here are some generated samples:
```
{"task_id": "HumanEval/86", "solution": "def anti_shuffle(s):\n    \"\"\"\n    Write a function that takes a string and returns an ordered version of it.\n    Ordered version of string, is a string where all words (separated by space)\n    are replaced by a new word where all the characters arranged in\n    ascending order based on ascii value.\n    Note: You should keep the order of words and blank spaces in the sentence.\n\n    For example:\n    anti_shuffle('Hi') returns 'Hi'\n    anti_shuffle('hello') returns 'ehllo'\n    anti_shuffle('Hello World!!!') returns 'Hello !!!Wdlor'\n    \"\"\"\n    return ' '.join(sorted(word) for word in s.split())"}
{"task_id": "HumanEval/87", "solution": "def get_row(lst, x):\n    \"\"\"\n    You are given a 2 dimensional data, as a nested lists,\n    which is similar to matrix, however, unlike matrices,\n    each row may contain a different number of columns.\n    Given lst, and integer x, find integers x in the list,\n    and return list of tuples, [(x1, y1), (x2, y2) ...] such that\n    each tuple is a coordinate - (row, columns), starting with 0.\n    Sort coordinates initially by rows in ascending order.\n    Also, sort coordinates of the row by columns in descending order.\n    \n    Examples:\n    get_row([\n      [1,2,3,4,5,6],\n      [1,2,3,4,1,6],\n      [1,2,3,4,5,1]\n    ], 1) == [(0, 0), (1, 4), (1, 0), (2, 5), (2, 0)]\n    get_row([], 1) == []\n    get_row([[], [1], [1, 2, 3]], 3) == [(2, 2)]\n    \"\"\"\n    return [(i, j) for i, row in enumerate(lst) for j, col in enumerate(row) if col == x]"}
{"task_id": "HumanEval/88", "solution": "def sort_array(array):\n    \"\"\"\n    Given an array of non-negative integers, return a copy of the given array after sorting,\n    you will sort the given array in ascending order if the sum( first index value, last index value) is odd,\n    or sort it in descending order if the sum( first index value, last index value) is even.\n\n    Note:\n    * don't change the given array.\n\n    Examples:\n    * sort_array([]) => []\n    * sort_array([5]) => [5]\n    * sort_array([2, 4, 3, 0, 1, 5]) => [0, 1, 2, 3, 4, 5]\n    * sort_array([2, 4, 3, 0, 1, 5, 6]) => [6, 5, 4, 3, 2, 1, 0]\n    \"\"\"\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here\n    # your code here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here here"}
{"task_id": "HumanEval/89", "solution": "def encrypt(s):\n    \"\"\"Create a function encrypt that takes a string as an argument and\n    returns a string encrypted with the alphabet being rotated. \n    The alphabet should be rotated in a manner such that the letters \n    shift down by two multiplied to two places.\n    For example:\n    encrypt('hi') returns 'lm'\n    encrypt('asdfghjkl') returns 'ewhjklnop'\n    encrypt('gf') returns 'kj'\n    encrypt('et') returns 'ix'\n    \"\"\"\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n    # Your code here\n"}

```

That's a bit weird:

I did pass `max_new_tokens`: 

https://github.com/evalplus/evalplus/blob/298465a7b3a96cdd2703c64675da89ab68669d2b/evalplus/provider/hf.py#L73

which is 768

https://github.com/evalplus/evalplus/blob/298465a7b3a96cdd2703c64675da89ab68669d2b/evalplus/provider/base.py#L13

can simply hardcode this in the local repo and test a bit?

Okay, I checked it a bit. `max_new_tokens` is surely passed into `model.generate`. However, I think it should be also received from `HuggingFaceDecoder` which is inside `__init__.py`, where `self.max_new_tokens` is determined.

Hi, ganler, thanks for your detailed comment : ). I have solved the above problem by hardcoding `max_new_tokens=512` to speedup (as mentioned in [this issue](https://github.com/evalplus/evalplus/issues/39)), however I only get 0.207 base score and 0.171 plus score with `deepseek-6.7B-base` model and the command above, which is far from the reported 47.6% base score and 39.6% plus score.
I took a look into `results.json`, which is like:
```
"HumanEval/16": [{"task_id": "HumanEval/16", "solution": "def count_distinct_characters(string: str) -> int:\n    \"\"\" Given a string, find out how many distinct characters (regardless of case) does it consist of\n    >>> count_distinct_characters('xyzXYZ')\n    3\n    >>> count_distinct_characters('Jerry')\n    4\n    \"\"\"\n    return len(set(string.lower())))", "base_status": "fail", "plus_status": "fail", "base_fail_tests": [], "plus_fail_tests": []}], 
```
Is there any specific reason to print out like compilation error to debug or it is just because of different GPU?

I did not get your question unfortunately.

Sorry, I would like to find out reasons which make the pass@1 lower than leader board reported
so, the question is: How could I get the error log to debug this even with `--test-details`?
Or are there any provided generated results by deepseek-6.7B-base to compare?

Try striping `swallow_io` in https://github.com/evalplus/evalplus/blob/master/evalplus/eval/__init__.py

太牛逼了吧 👍🏻👍🏻👍🏻👍🏻

太牛逼了吧 👍🏻👍🏻👍🏻👍🏻

Feel free to reopen if this issue still exists. I have been running a few new eval based HF and it seems fast.

Also one potential reason is likely you are running it on CPU -- you can fix that by setting `--attn-implementation` to `flash_attention_2` or set the corresponding `--device-map`.