zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Running logits / logprobs apis evaluation using local-completions
Root cause: Documentation Deficiency

Hi, I am trying to evaluate llama-405B using https://api.llama-api.com/chat/completions


now this is sample code which is working perfectly for me if I run using simple requests

```
import requests
api_key = "1234"
url = "https://api.llama-api.com/chat/completions"
headers = {"Authorization": f"Bearer 1234"}
payload = {"model": "llama3.1-405b", "messages": [{"role": "user", "content": "Hello, how are you? result in two words"}], "logprobs" : True}
response = requests.post(url, headers=headers, json=payload)
print(response.json())
```

Which outputs this result

`{'created': 1721862362, 'model': 'llama3.1-405b', 'usage': {'prompt_tokens': 72, 'completion_tokens': 4, 'total_tokens': 76}, 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': "I'm good.", 'function_call': None}, 'finish_reason': 'stop', 'logprobs': {'tokens': ['I', "'m", ' good', '.', '
'], 'token_logprobs': [-0.024414062, -0.016967773, -0.875, -0.69921875, -9.584427e-05], 'token_ids': [40, 2846, 1695, 13, 128009]}}]}`


Now I am trying to use `local-completions` to eval the llama3.1-405b using this API. I tried this code

`lm_eval --model local-chat-completions --tasks gsm8k --model_args model=llama3.1-405b,base_url=https://api.llama-api.com/chat/completions,api_key=1234,temperature=0.0 --apply_chat_template --limit 10`

this is working quite good and getting the result.

Now I want to eval the mmlu (multiple choice questions) and for that I need the log_probs which API is already returning but when I run

`lm_eval --model local-completions --tasks mmlu_college_biology --model_args model=llama3.1-405b,base_url=https://api.llama-api.com/chat/completions,api_key=1234,temperature=0.0 --limit 10
`

I got the following error

`OSError: llama3.1-405b is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>``

While checking https://github.com/EleutherAI/lm-evaluation-harness/pull/2008 I found that @baberabb is using the command like this

`lm_eval --model local-completions --model_args pretrained=pretrained=meta-llama/Meta-Llama-3-8B-Instruct,base_url=http://localhost:8000/v1/completions,concurrent=100 --tasks gsm8k --batch 1`

local model is defined but I don't see how to define the API model here?

But if I want to evaluate mmlu on https://api.llama-api.com/chat/completions API models which also support log_probs then how to do it?
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug when hashing prompt when using openai-chatcompletions and log_samples
Root cause: Validation Gap

When I use openai-chatcompletions with --apply_chat_template and log_samples, it encounters an error 
```
Traceback (most recent call last):
  File "/root/anaconda3/envs/dschat/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/evaluator.py", line 296, in simple_evaluate
    results = evaluate(
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/evaluator.py", line 525, in evaluate
    "prompt_hash": hash_string(requests[0].arguments[0]),
  File "/home/xxx/lm-evaluation-harness-flageval/lm_eval/utils.py", line 36, in hash_string
    return hashlib.sha256(string.encode("utf-8")).hexdigest()
AttributeError: 'JsonChatStr' object has no attribute 'encode'
```
It happens in 
```
"prompt_hash": hash_string(requests[0].arguments[0])
```
It seems that hash_string only receives string input rather than a list of messages. But open-chatcompletions needs an list input.

Sorry about that. The PR should fix it!

:)
Repository: EleutherAI/lm-evaluation-harness
Issue: Checkpointing Evaluation Results and Enable Resume of Evaluation
Root cause: Validation Gap

Hi, I wonder if harness is planning to support checkpointing the evaluation results of evaluation tasks during the evaluation process, so that if some error happens during the evaluation on one benchmark (e.g., end of GPU allocation walltime), the evaluation process can resume from (nearly) where it stops. Thanks!

Hi! you can  `--use_cache <DIR>` to cache the results while evaluating and skip previously evaluated samples on resumption. Caching is rank-dependent though, so restart with the same GPU count if interrupted! 

Thanks for the prompt response! A quick follow-up: how about the evaluation via a local inference server? can it be resumed?

yeah that also implements caching!

Here is a bug: if the inference is all done, but the evaluation after it fails (in my case, `"Resource punkt_tab not found"`), and after fixing it you run `lm-eval` again to resume the evaluation, it will show this error:
```
2024-09-03:20:51:09,639 INFO     [model.py:283] Cached requests: 541, Requests remaining: 0
Traceback (most recent call last):
  File ".../conda-envs/llava/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File ".../lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File ".../lm-evaluation-harness/lm_eval/utils.py", line 398, in _wrapper
    return fn(*args, **kwargs)
  File ".../lm-evaluation-harness/lm_eval/evaluator.py", line 296, in simple_evaluate
    results = evaluate(
  File ".../lm-evaluation-harness/lm_eval/utils.py", line 398, in _wrapper
    return fn(*args, **kwargs)
  File ".../lm-evaluation-harness/lm_eval/evaluator.py", line 468, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File ".../lm-evaluation-harness/lm_eval/api/model.py", line 287, in fn
    rem_res = getattr(self.lm, attr)(remaining_reqs)
  File ".../lm-evaluation-harness/lm_eval/models/api_models.py", line 522, in generate_until
    requests, all_gen_kwargs = zip(*(req.args for req in requests))
ValueError: not enough values to unpack (expected 2, got 0)
```
It seems that 0 samples to run will cause an error, and since the cache cannot be modified manually, I have to run the evaluation all over again.
Please fix this problem, thank you a lot!

> It seems that 0 samples to run will cause an error, and since the cache cannot be modified manually, I have to run the evaluation all over again. Please fix this problem, thank you a lot!

Hi! Are you using the latest commit? This error should be fixed after #2187


Oh, thank you very much for reminding! I am using an earlier version, and I will update to the new commit. Sorry for the interruption and thank you again for the effort!
Repository: EleutherAI/lm-evaluation-harness
Issue: API model: Evaluation fails when all samples are cached
Root cause: Validation Gap

When all samples are already cached, the process errors out instead (instead of skipping to the metric calculation) on the subsequent run due to lack of requests to pass on to the LM.

@baberabb which model type / task type does this seem to happen for you, if any in particular? looking at some caching-related issues rn

I thought all but looks like it's just for the API `generate_until` inputs at this part:
`requests, all_gen_kwargs = zip(*(req.args for req in requests))`

It can't handle the empty list. Should also check vllm though. 



does #2187 resolve what you were experiencing fully?
Repository: EleutherAI/lm-evaluation-harness
Issue: Is this comme\\and right?
Root cause: Documentation Deficiency

To evaluate llama3 instruct on the leaderboard dataset. i used this command:
`lm_eval --model hf \
    --model_args pretrained=meta-llama/Meta-Llama-3.1-8B,dtype="bfloat16"\
    --tasks leaderboard_gpqa\
    --device cuda:0 \
    --num_fewshot 0\
    --apply_chat_template\
    --batch_size auto:2`

But the results are too different from the leaderboard even when using normalization described  at [https://github.com/huggingface/leaderboards/blob/main/docs/source/en/open_llm_leaderboard/normalization.md](l)
 So I would like to know if this is the right command.

|           Tasks            |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|----------------------------|-------|------|-----:|--------|---|-----:|---|-----:|
|leaderboard_gpqa            |    N/A|      |      |        |   |      |   |      |
| - leaderboard_gpqa_diamond |      1|none  |     0|acc_norm|↑  |0.2778|±  |0.0319|
| - leaderboard_gpqa_extended|      1|none  |     0|acc_norm|↑  |0.2491|±  |0.0185|
| - leaderboard_gpqa_main    |      1|none  |     0|acc_norm|↑  |0.2612|±  |0.0208|





This question is probably better asked in official Open LLM Leaderboard channels where @clefourrier and team can more easily see and comment on it.

Hi! 
Some changes:
- you don't need to set few shot
- since this is a base model you should not be using the chat template (and if you were using the chat template you would also need to select the multiturn parameter)
- you should compare these results with the raw results in the leaderboard (or the result file directly).

Thanks @haileyschoelkopf for the tag! 
Indeed, please open a discussion on the leaderboard next time so we can keep track of it

(For future reference, is there a precise URL I can point people to or include in an FAQ doc?

Yes! :)
Doc is [here](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about) and discussions should be opened [here](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard/discussions)
Repository: EleutherAI/lm-evaluation-harness
Issue: Tasks not found: arc
Root cause: Documentation Deficiency

I see the benchmark support the arc task. Howerver, the follow commands will result error that ValueError: Tasks not found: arc. 
lm_eval --model hf --model_args pretrained=/root/autodl-tmp/gpt2-xl --tasks hellaswag  --device cpu --batch_size 1


Hi, the names of AI2's ARC question answering tasks are `arc_easy` and `arc_challenge` in our library.
Repository: EleutherAI/lm-evaluation-harness
Issue: Removal of trailing newline from description
Root cause: Configuration Error

Hello, I was debugging where did my trailing whitespace (\n) from the end of description disappeared. It was not present in the input prompt (logged using --log_samples) 

And it turns out jinja2 renderer removes it.

If you take a look at Jinja2 environment
https://github.com/EleutherAI/lm-evaluation-harness/blob/42dc244867889a19ae80847254a481f446f6e4b7/lm_eval/utils.py#L474

namely, parameter `keep_trailing_newline`, it is by default set to False (KEEP_TRAILING_NEWLINE = False).

In other words, trailing whitespace disappears here. Is this an expected behavior?

Thank you for providing insights. 
Best,
Martin
Repository: EleutherAI/lm-evaluation-harness
Issue: [Question] A way to run multiple evals on multiple models?
Root cause: Unimplemented Feature Gap

Hi, I was wondering if there is a way supported by Python API to run multiple eval benchmarks on multiple models (by passing in a list of models and their respective arguments in the `model_args` argument)?

For example:
```python
results = lm_eval.simple_evaluate(
    model="hf",
    model_args=["pretrained=microsoft/phi-2,trust_remote_code=True", "pretrained=microsoft/phi-3,trust_remote_code=True"],
    tasks=["hellaswag","mmlu_abstract_algebra"],
    log_samples=True,
)
```

TIA!


Hi, 

Do you know how to run [mmlu's flan_cot_zeroshot](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mmlu/flan_cot_zeroshot)? How should we specify it in the tasks arguement?

Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug in the `visualize-wandb.ipynb` example
Root cause: Documentation Deficiency

Hi, there's a bug in the [`visualize-wandb.ipynb`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/examples/visualize-wandb.ipynb):

In the example notebook's final cell, the `tasks` are comma-separated in a single string. However running that on a fresh install of `lm_eval` throws a key error. It should actually be passed in as a list of strings instead of comma separated tasks inside a string.

Basically this:
```python
results = lm_eval.simple_evaluate(
    model="hf",
    model_args="pretrained=microsoft/phi-2,trust_remote_code=True",
    tasks="hellaswag,mmlu_abstract_algebra",
    log_samples=True,
)
```

Should be this:

```python
results = lm_eval.simple_evaluate(
    model="hf",
    model_args="pretrained=microsoft/phi-2,trust_remote_code=True",
    tasks=["hellaswag","mmlu_abstract_algebra"],
    log_samples=True,
)
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Random results when parallelize=True
Root cause: Resource Mishandling

I'm encountering a strange issue with lm-eval. 
After updating to the new version and reinstalling everything from scratch, I ran MMLU with `parallelize=True` and got random accuracy results.
However, when I ran it without parallelize (i.e., without splitting the model), the results were much better. 

random results with: 
`python -m lm_eval --model hf --model_args pretrained=meta-llama/Meta-Llama-3-8B-Instruct,parallelize=True --tasks mmlu --batch_size 8`

vs:
`/home/benshoho/.conda/envs/lm_eval_new_env/bin/python -m lm_eval --model hf --model_args pretrained=meta-llama/Meta-Llama-3-8B-Instruct --device cuda:0 --tasks mmlu --batch_size 8`

How can I solve this?


Hi, Could you please share how you fix the issue? I have encontered the same issue.

> Hi, Could you please share how you fix the issue? I have encontered the same issue.

For me, the problem was resolved when I used different GPUs. Although both are RTX 6000, the issue only occurred with one of them. If you are experiencing this issue, consider opening a new issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: When using the tmmluplus dataset, I'm encountering a TypeError: init() got an unexpected keyword argument 'group_alias'.
Root cause: Interface Contract Mismatch

lm_eval --model vllm \
    --model_args pretrained=/home/models/Meta-Llama-3.1-8B-Instruct,trust_remote_code=True,tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=1 \
    --tasks tmmluplus \
    --batch_size 1 \
    --output_path /home/lm-evaluation-harness/

[rank0]: TypeError: __init__() got an unexpected keyword argument 'group_alias'

Try this cmd to remove "group_alias" line in tmmluplus_*.yaml.
```
# at lm_eval/tasks/tmmluplus/default
for file in tmmluplus_*.yaml; do
    sed -i '/group_alias/d' "$file"
done
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug about the output information.
Root cause: Algorithmic Error

When I run the 'mmlu_generative' dataset, the group score does not produce any output.
```
|Groups|Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|------|-------|------|------|------|---|-----|---|------|
```

+1
Repository: EleutherAI/lm-evaluation-harness
Issue: Running multiple evaluations on the same task
Root cause: Architectural Constraint

Prior to the "Group agg rework" commit, I was able to run multiple evaluations on the same task in the same run, like this: 

```
group: taskA
task:

  - task: taskA_fs0
    include: _taskA.yaml
    num_fewshot: 0

  - task: taskA_fs1
    include: _taskA.yaml
    num_fewshot: 1

  - task: taskA_fs3
    include: _taskA.yaml
    num_fewshot: 3

  - task: taskA_fs5
    include: _taskA.yaml
    num_fewshot: 5
```

Now, I am unable to run bencmarks in this way. Changing ```include: _taskA.yaml``` to something like ```task: _taskA``` throws ```Please call groups which overlap their constituent tasks in separate evaluation runs.```, as somewhat expected.

I tried using tag and tag_alias to no avail. How am I supposed to run tasks like above? It this still supported?

Could you share a more full repro of some kind that I can use to get this error and inspect what your group looks like a bit more?

From what I understand, in your current posted config file, because these tasks have different names, this should be the way we recommend usage like your case. So I'm not sure why this is erroring.

But yes, the reason we throw an error like this is because (for now) the different num_fewshot variants of the "same" task would compete to overwrite each other and create confusion in logged files' contents, etc.

> Could you share a more full repro of some kind that I can use to get this error and inspect what your group looks like a bit more?

Sure. My fork is here [1]. My tasks are under tasks/romanian_bench. With the exception of ro_truthfulqa they are all in the same format as stated above. Note that in order to make "include" work in the last version, I had to change a line in yaml task loading (check last commit). 

> But yes, the reason we throw an error like this is because (for now) the different num_fewshot variants of the "same" task would compete to overwrite each other and create confusion in logged files' contents, etc.

This makes sense, but as you mentioned they have different names, and prior to the group rework, everything worked just fine.

[1] https://github.com/MihaiMasala/lm-evaluation-harness

Do you have any updates on this?
Repository: EleutherAI/lm-evaluation-harness
Issue: GPT2 eval in lambada_openai, acc only 0.325
Root cause: Validation Gap

python lm_eval/\_\_main\_\_.py --model hf --model_args pretrained=openai-community/gpt2 --tasks lambada_openai --device cuda:0 --batch_size 4

|    Tasks              |Version|Filter|n-shot|  Metric  |   | Value |   |Stderr|
|lambada_openai|      1|none  |     0|acc          |↑  | 0.3256|±  |0.0065|
|                            |       |none  |     0|perplexity|↓  |40.0554|±  |1.4787|

The acc is only 0.325. In gpt2 paper, the acc is 0.4599

+1

+1
Repository: EleutherAI/lm-evaluation-harness
Issue: Better tests for API models
Root cause: Unimplemented Feature Gap

Right now the [tests](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/tests/models/test_api.py) for the API models mostly check if the payload is in the right format. Would be nice if we could mock an API and also test the responses.
Repository: EleutherAI/lm-evaluation-harness
Issue: Premature `num_fewshot` check with `fewshot_as_multiturn`
Root cause: Validation Gap

`main` raises a ValueError if `fewshot_as_multiturn` and no `num_fewshot`. However, `num_fewshot` can also be set in the task YAML, which is processed later. Think we should remove this especially because `num_fewshot` sets the global fewshots ignoring task specific values.

https://github.com/EleutherAI/lm-evaluation-harness/blob/b70af4f51481db15ec3068d26cde3d76dd0201e7/lm_eval/__main__.py#L295-L300

Thanks for raising this!

#2058 should address this, I need to get that PR finally merged!
Repository: EleutherAI/lm-evaluation-harness
Issue: mgsm task not working
Root cause: Validation Gap

### Error message :- 

`ValueError: BuilderConfig 'eng' not found. Available: ['amh', 'ewe', 'fra', 'hau', 'ibo', 'kin', 'lin', 'lug', 'orm', 'sna', 'sot', 'swa', 'twi', 'wol', 'xho', 'yor', 'zul']`


### Cause :- 

Fails [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L930-L931) when 

path = masakhane/afrimgsm-translate-test
name =  eng


eng subset does not exist on [masakhane/afrimgsm-translate-test](https://huggingface.co/datasets/masakhane/afrimgsm-translate-test)
Repository: EleutherAI/lm-evaluation-harness
Issue: After executing tmmluplus, there are no group scores displayed, only the scores for each individual task are shown.
Root cause: External Dependency Breakage

![image](https://github.com/user-attachments/assets/7c80d77f-e634-479f-bae6-bc17b8e32538)


Hi,
I encountered the same problem.
How did you fix it?

> Hi, I encountered the same problem. How did you fix it?

Install pip install lm_eval==0.4.3, ensuring that the version of lm_eval is 0.4.3, give it a try.

> > Hi, I encountered the same problem. How did you fix it?
> 
> Install pip install lm_eval==0.4.3, ensuring that the version of lm_eval is 0.4.3, give it a try.

Is the version of lm_eval that you see when using 'pip list' 0.4.3?

When I downgraded the lm_eval version from the latest to 0.4.3, the problem went way.
So might be the problem of the latest version?

Anyway, thanks for the suggestion!

I'm also experiencing similar issue with groups. I'm running it on a set of custom tasks, and just checked that I actually have the lm_eval=0.4.3
Repository: EleutherAI/lm-evaluation-harness
Issue: [BUG] Huggingface and Neuron runs fail if model revision is an integer
Root cause: Validation Gap

When the model revision identifier only consists of integers the run will fail because the revision is naively appended to a string [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/7f15cce4d759043428b3db14970677fcec6e7a45/lm_eval/models/huggingface.py#L194) in the HF model and [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/7f15cce4d759043428b3db14970677fcec6e7a45/lm_eval/models/neuron_optimum.py#L235) in the Neuron model.

Steps to reproduce:
For huggingface.

```
MODEL_NAME=some/model
REVISION=123123 
lm_eval --model_args='pretrained=MODEL,revision=REVISION' --task=xnli --limit=10 --model=hf
```
Result
```
  File "/code/research-now-llm/lm-eval-wrapper/lm-evaluation-harness/lm_eval/models/huggingface.py", line 193, in __init__
TypeError: unsupported operand type(s) for +: 'int' and 'str'` 
```
As I see it there are two possible fixes: 
1. make sure the revision is properly understood as a string when model_args are parsed.
2. ensure the revision is a string when it is appended to another string i.e ` revision = f"{revision}" + ("/" + subfolder if subfolder is not None else "")`

Hi! Thanks for reporting this!

Followed fix 1. in https://github.com/EleutherAI/lm-evaluation-harness/pull/2184 . 

As described in #2183 it might be nice to more generally fix this type of parsing type issue, but for now #2184 can fix this issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to calculate the "token_perplexity"
Root cause: Documentation Deficiency

Now in the harness, there are metrics called "byte_perplexity" and "word_perplexity". These two metrics normalize the perplexity by the length of characters and words, respectively. If we want to normalize the perplexity by tokens (i.e., the length of tokens as cut by the tokenizer), how can we calculate it effectively?

I tried modifying the function process_results in the ConfigurableTask class, but I found it difficult to obtain the token length of the target. Does anyone have a good idea on how to calculate it?

 

This should be solved by #2132 , stay tuned there!