zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: output_path not supporting direct JSON paths + OsErrror36 for long model_path
Root cause: Validation Gap

## Description
The argument parser's `--output_path` help text (`metavar="DIR|DIR/file.json"`) suggests that users can supply either a directory **or** an explicit JSON filename for `--output_path`, However, the implementation always treats `output_path` as a directory.

https://github.com/EleutherAI/lm-evaluation-harness/blob/2f03271d25db3c19e5552e19f59816bcbba07357/lm_eval/__main__.py#L132-L139

https://github.com/EleutherAI/lm-evaluation-harness/blob/2f03271d25db3c19e5552e19f59816bcbba07357/lm_eval/loggers/evaluation_tracker.py#L231-L233

## Why this matters
It can be very helpful to specify an exact output filename instead of relying on the harness’s autogenerated naming. When long model identifiers are sanitized into directory and file names, the resulting path can exceed OS limits—causing `OSError 36: Filename too long` and preventing any results from being saved.

## Steps to reproduce
```
lm_eval --model hf \
    --model_args pretrained=model-name-or-path \
    --tasks hellaswag \
    --output_path results.json
```
will create a directory called `results.json` and save results in: `results.json/results_DATE.json`
Repository: EleutherAI/lm-evaluation-harness
Issue: Move To A Saner Benchmark Definition Scheme
Root cause: Architectural Constraint

There is a lot of code duplication in YAML, the use of unsafe escape hatches, and not taking advantage of the type checker where appropriate, etc.

I came from this article, which I think explains the complaints pretty well: https://toxicpine.github.io/posts/EasyLLMBench/
  
Could we implement some alternative mechanism for specifying benchmarks in parallel and slowly transition?

Maybe we should take notes from [EasyBench](https://github.com/ToxicPine/easybench)?

We're working on improving some of these things, but some of the complaints in that blog post seem pretty detached from the needs of the hundreds of regular users we have. Most notably
> Crucially, EasyLLMBench deliberately delegates model inference entirely back to you.
is something that the vast majority of people actively don't want.

You're very welcome to contribute to the conversation about improving and unifying abstractions here: https://github.com/EleutherAI/lm-evaluation-harness/issues/3083
Repository: EleutherAI/lm-evaluation-harness
Issue: Longbench classification_score() missing 1 required positional argument: 'results'
Root cause: Interface Contract Mismatch

I was trying to evaluate on `longbench_trec_e`

The evaluation pipeline can work, except for the last step of calculating the metric. I got the error:

```
Traceback (most recent call last):
  File "/data/cl/u/yangsl66/code/lm-evaluation-harness/lm_eval/api/task.py", line 1711, in process_results
    result_score = self._metric_fn_list[metric](
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: classification_score() missing 2 required positional arguments: 'doc' and 'results'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/data/cl/u/yangsl66/code/hope-flame/eval/harness.py", line 22, in <module>
    cli_evaluate()
  File "/data/cl/u/yangsl66/code/hope-flame/lm-evaluation-harness/lm_eval/__main__.py", line 449, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/cl/u/yangsl66/code/hope-flame/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/cl/u/yangsl66/code/hope-flame/lm-evaluation-harness/lm_eval/evaluator.py", line 338, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/data/cl/u/yangsl66/code/hope-flame/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/cl/u/yangsl66/code/hope-flame/lm-evaluation-harness/lm_eval/evaluator.py", line 616, in evaluate
    metrics = task.process_results(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/data/cl/u/yangsl66/code/hope-flame/lm-evaluation-harness/lm_eval/api/task.py", line 1717, in process_results
    result_score = self._metric_fn_list[metric]([gold, result])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: classification_score() missing 1 required positional argument: 'results'
```

cc @baberabb  

Hi! There were some other errors in the longbench implementation as well, sorry about that. Making fixes in #2983, but still need to test it. #2932 for context.
Repository: EleutherAI/lm-evaluation-harness
Issue: Multimodal evals for local_completions
Root cause: Unimplemented Feature Gap

Hi folks. I'm locally serving some multimodal models and trying to evaluate them over API (`local_completions`) but seems like this is not supported as I am seeing this error message:
```bash
ValueError: Attempted to run tasks: ['chartqa'] which require multimodal input, but the selected model type does not currently implement this. Multimodal support is currently restricted to the ['hf-multimodal', 'vllm-vlm'] model type.
```

Is it correct that this is not supported, and are there any plans to support it anytime soon? If not, could you share some pointers where relevant code for this pathway lives, so I can perhaps take a stab at it.


Hi! made a start on #2981. Currently have only tested it on OpenAI's chat-completions endpoint, but should work with other `local-chat-completions` APIs with minimal changes

Hey! I just ran some tests with the PR, and the results are looking fine. Thanks a lot!
Repository: EleutherAI/lm-evaluation-harness
Issue: `local-completions` does not support sglang backend
Root cause: Interface Contract Mismatch

As per title.

`lm_eval --model local-completions --model_args model=${MODEL_ID},base_url=http://0.0.0.0:30000/v1/completions,num_concurrent=999,timeout=99999,tokenized_requests=False,max_length=6000 --tasks wikitext --gen_kwargs "do_sample=False,top_p=1,temperature=0"` does not work.

Only `/generate` endpoint in sglang supports `logprobs=1`, `echo=True`.

However this endpoint does not output `choices` which is later required in `parse_logprobs`.

> Only /generate endpoint in sglang supports logprobs=1, echo=True.

Hitting https://github.com/sgl-project/sglang/blob/c23a7072b6dcec38e54482fe3b50cfb06d1d9139/python/sglang/srt/openai_api/adapter.py#L537
Repository: EleutherAI/lm-evaluation-harness
Issue: unitxt with local-chat-completions gets stuck forever
Root cause: Interface Contract Mismatch

I am doing tests with Ollama, several evals run correctly, but with unitxt 1.23.0 and 1.22.4 it just sits forever without using CPU or GPU resources.

Using lm_eval v0.4.8 with Python 3.12:

```sh
lm_eval --limit 10 --model local-chat-completions --tasks cnn_dailymail --log_samples \
--output_path /results/qwen2.5:7b-instruct-q8_0/ --write_out --trust_remote_code \
--model_args model=qwen2.5:7b-instruct-q8_0,base_url=http://localhost:11434/v1/chat/completions,num_concurrent=10,max_retries=999,tokenized_requests=False,max_length=8192 \
--apply_chat_template
```


And I'm guessing based on your phrasing that this also worked prior to unitxt 1.22.4? Regardless, I can take a look when I'm back 'in office' week after next 

Hi, thanks for your reply.
No, I tested with both versions of unitxt and didn't work.
Didn't try older versions of lm_eval, etc.
I am new to lm_eval by the way, so I may be doing something wrong of course.

~~I think there's an issue with the dependencies. The CI tests are also timing out at dependency resolution: [stack trace](https://github.com/EleutherAI/lm-evaluation-harness/actions/runs/15110232867/job/42467886227)~~

edit: The dependency resolution problem is due to either deepsparse or sparse-ml (or both)

I just tried a few different unitxt versions and I don't think chat-completions have ever worked with the unitxt integration. I can't reproduce the hanging issue, but I do get a different failure when trying unitxt tasks with local-chat-completions. Perhaps the hang was due to @baberabb's comment but definitely keeping the issue open and will look into fixing it the core issue

sorry. the time out issue was unrelated to unitext!
Repository: EleutherAI/lm-evaluation-harness
Issue: facing error while evaluation on lora model with vllm
Root cause: Configuration Error

my command  : 
```bash 
lm_eval --model vllm \
    --model_args pretrained="${model_dir}",enable_lora=true,lora_local_path="${lora_model_dir}",gpu_memory_utilization=0.75,enforce_eager=true,trust_remote_code=true,max_length=4096\
    --tasks lm_eval/tasks/hellaswag\
    --batch_size 1\
    --output_path "$output_dir" \
    --log_samples \
```

error : 
```
/lib/python3.12/site-packages/vllm/lora/peft_helper.py", line 115, in validate_legal
    raise ValueError(f"{' '.join(error_msg)}")
ValueError: LoRA rank 256 is greater than max_lora_rank 16.
```

The model was fine-tuned with lora rank 256.
What is the cause of the problem?
I tried to find argument related to setting lora rank, but cannot find any

added `max_lora_rank=256` in model_args, and now facing new Assertion Error

```
/python3.12/site-packages/vllm/lora/ops/triton_ops/lora_shrink.py", line 151, in _lora_shrink
ERROR 05-16 07:10:58 [core.py:398]     assert token_lora_mapping.size(0) == M
ERROR 05-16 07:10:58 [core.py:398]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-16 07:10:58 [core.py:398] AssertionError
```

Hi! Are you able to run the model with `vllm` normally? i.e outside of `lm_eval`

I solved this issue by merging base model and lora trained model with export ! 
I forgot to merge them before running the command.
Thanks for your reply
Repository: EleutherAI/lm-evaluation-harness
Issue: running loglikelyhood requests error, why?
Root cause: Interface Contract Mismatch

![Image](https://github.com/user-attachments/assets/8421030b-6718-4263-b301-98ccbdfe76ec)

command:

OPENAI_API_KEY="sk-FuPxxxxxxxxxxxxx8aEb" lm_eval --model local-completions --tasks **mmlu** --model_args model=Qwen1.5-14B-Chat,base_url=https://myapi.net/v1/completions,num_concurrent=10,max_retries=3,tokenizer=Qwen/Qwen-tokenizer,tokenizer_backend=huggingface

Hi! Seems the api you're using expects `log_probs` here to be bool rather than integer:
https://github.com/EleutherAI/lm-evaluation-harness/blob/0daf28fda71dc17ca4cf93407b9601fa42cce498/lm_eval/models/openai_completions.py#L54-L62

i would refer to your docs and modify the `_create_payload` appropriately (maybe `parse_logprobs` as well).

closing now but feel free to open!
Repository: EleutherAI/lm-evaluation-harness
Issue: `mbpp` not compatible with apply_chat_template
Root cause: Interface Contract Mismatch

Running `mbpp` with `apply_chat_template` returns a score of 0.

```shell
lm_eval --model hf --model_args pretrained=Qwen/Qwen3-8B,dtype="bfloat16",device="cuda",parallelize=False --tasks mbpp  --batch_size 1 --confirm_run_unsafe_code --limit 10 --apply_chat_template

|Tasks|Version|Filter|n-shot| Metric  |   |Value|   |Stderr|
|-----|------:|------|-----:|---------|---|----:|---|-----:|
|mbpp |      1|none  |     3|pass_at_1|↑  |    0|±  |     0|
```

vs

```shell
lm_eval --model hf --model_args pretrained=Qwen/Qwen3-8B,dtype="bfloat16",device="cuda",parallelize=False --tasks mbpp  --batch_size 1 --confirm_run_unsafe_code --limit 10

|Tasks|Version|Filter|n-shot| Metric  |   |Value|   |Stderr|
|-----|------:|------|-----:|---------|---|----:|---|-----:|
|mbpp |      1|none  |     3|pass_at_1|↑  |  0.6|±  |0.1633|
```


Hi! added variants for instruct models in #2995 (slightly different prompt with assistant prefix).
Repository: EleutherAI/lm-evaluation-harness
Issue: [vllm] update `data_parallel_size` for vllm version 1
Root cause: External Dependency Breakage

Currently, the code we use for creating multiple replicas of vLLM models does not work with version 1 of vLLM. We rely on Ray to distribute the models across multiple processes, but it appears we need to modify our implementation for vLLM v1 compatibility.

The relevant code is here: https://github.com/EleutherAI/lm-evaluation-harness/blob/86c266a1b3d9f51a43d277032da9f16a632d0bc3/lm_eval/models/vllm_causallms.py#L264-L280

Note: `data_parallel_size` still works when using the `VLLM_USE_V1=0` environment variable (which forces vLLM to use the legacy engine)

vLLM also provides an example [here](https://docs.vllm.ai/en/latest/getting_started/examples/data_parallel.html), which we might be able to adapt for our use case.

Thanks! This problem tortured me for so long, and I tried to unset CUDA_VISIBLE_DEVICES but failed...

@baberabb Thanks for the solution, I want to know how to work with `tensor_parallel_size=8` or even two mode nodes, the script is 
```bash
lm_eval --model vllm --model_args pretrained="/nfs-69/Qwen3/Qwen3-30B-A3B",max_model_len=8192,tensor_parallel_size=8,dtype=auto,data_parallel_size=1 --tasks ceval-valid --batch_size 2
```

what params should i add to the command?

> [@baberabb](https://github.com/baberabb) Thanks for the solution, I want to know how to work with `tensor_parallel_size=8` or even two mode nodes, the script is
> 
> lm_eval --model vllm --model_args pretrained="/nfs-69/Qwen3/Qwen3-30B-A3B",max_model_len=8192,tensor_parallel_size=8,dtype=auto,data_parallel_size=1 --tasks ceval-valid --batch_size 2
> what params should i add to the command?

Hi! multiple nodes are currently not supported using `--model vllm`, but your command should generally work for a single node? You could also [serve](https://docs.vllm.ai/en/latest/serving/distributed_serving.html#running-vllm-on-multiple-nodes) your model with vllm and try using `local-completions` with command similar to:
```
lm_eval --model local-completions --model_args pretrained="Qwen3/Qwen3-30B-A3B",base_url=http://localhost:8000/v1/completions,max_length=8192,num_concurrent=4 --tasks ceval-valid --batch_size 2
```
full args [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/API_guide.md#templateapi-arguments)


Thanks for the solution! I passed som param when i started the container
```bash
 --privileged --ipc=host --net=host --gpus=all -m 2048g --memory-swap -1 --oom-kill-disable --memory-swappiness 30 --memory-reservation 2048g 
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the Novel Word evaluation
Root cause: Unimplemented Feature Gap

This is a dataset we need to generate ourselves. From the GPT-3 paper

> A task studied in developmental linguistics [CB78] is the ability to learn and utilize new words, for example using a word in a sentence after seeing it defined only once, or conversely inferring a word’s meaning from only one usage. Here we qualitatively test GPT-3’s ability to do the former. Specifically, we give GPT-3 the definition of a nonexistent word, such as “Gigamuru”, and then ask it to use it in a sentence. We provide one to five previous examples of a (separate) nonexistent word being defined and used in a sentence, so the task is few-shot in terms of previous examples of the broad task and one-shot in terms of the specific word. Table 3.16 shows the 6 examples we generated; all definitions were human-generated, and the first answer was human-generated as conditioning while the subsequent answers were generated by GPT-3. These examples were generated continuously in one sitting and we did not omit or repeatedly try any prompts. In all cases the generated sentence appears to be a correct or at least plausible use of the word. In the final sentence the model generates a plausible conjugation for the word “screeg” (namely “screeghed”), although the use of the word is slightly awkward (“screeghed at each other”) despite being plausible in the sense that it could describe a toy sword fight. Overall, GPT-3 appears to be at least proficient at the task of using novel words in a sentence.

- [ ] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`
Repository: EleutherAI/lm-evaluation-harness
Issue: DROP task doesn't work on 0.4.8 on PyPI
Root cause: Validation Gap

When I run the following command:

lm_eval \
    --model hf \
    --model_args pretrained=... \
    --tasks drop \
    --batch_size auto \
    --device auto \
    --output_path ... \
    --write_out \
    --show_config \

I consistently get this error:

Traceback (most recent call last):
  File "/data/afontana/my_env/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/data/afontana/my_env/lib64/python3.11/site-packages/lm_eval/__main__.py", line 389, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/afontana/my_env/lib64/python3.11/site-packages/lm_eval/utils.py", line 422, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/afontana/my_env/lib64/python3.11/site-packages/lm_eval/evaluator.py", line 308, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/data/afontana/my_env/lib64/python3.11/site-packages/lm_eval/utils.py", line 422, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/data/afontana/my_env/lib64/python3.11/site-packages/lm_eval/evaluator.py", line 645, in evaluate
    results_agg, group_agg = prepare_print_tasks(task_dict, results)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/afontana/my_env/lib64/python3.11/site-packages/lm_eval/evaluator_utils.py", line 286, in prepare_print_tasks
    task_agg[name]["alias"] = tab_string + alias
                              ~~~~~~~~~~~^~~~~~~
TypeError: can only concatenate str (not "NoneType") to str


Hi! it works if you clone from [source](https://github.com/EleutherAI/lm-evaluation-harness#install). I'll also try pushing a new version to PyPI latest by next week!

Thanks for the quick reply!

You're right — it does work when installing from [source](https://github.com/EleutherAI/lm-evaluation-harness#install), but there still seems to be an issue.

If I download the drop task folder, rename both the folder and the task to drop_copy, and then pass the new folder via the --include_path argument, I get the same error as described above.

My question is:
Is the --include_path behavior also somehow tied to the PyPI version, even when running from source?

Or should I open a separate issue specifically about include_path?

I really can't think how it would be connected. Were you using a completely different virtual environment for the source install? This would help rule out any potential interference.

Hi! It's working again now, thanks for the support.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to reproduce the Qwen2.5 base model results on GSM8K Task
Root cause: Documentation Deficiency

Hi, I am trying to reproduce the Qwen2.5 base model results on GSM8K Task. But I am getting very low scores compared to what was reported in their paper. I noticed that in the GSM8K task files, there is no YAML for reasoning models.
Thanks!

same issue here for qwen3 models..
Repository: EleutherAI/lm-evaluation-harness
Issue: backward compatibility for unitxt (and others) after adding question_suffix to `task.fewshot_context` in #2876
Root cause: External Dependency Breakage

`unitext` gives a TypeError as it doesn't expect the param, we added in #2876 

reverted PR in #3007. Leaving this issue open to track and reconsider once new version is pushed to PyPI

Feel free to assign me to this - pending a decision on a course of action ofc
Repository: EleutherAI/lm-evaluation-harness
Issue: zeno_visualize.py can't parse model_args
Root cause: Validation Gap

``` python
            model_args = re.sub(
                r"[\"<>:/\|\\?\*\[\]]+",
                "__",
                json.load(
                    open(Path(args.data_path, model, latest_results), encoding="utf-8")
                )["config"]["model_args"],
```
At zeno_visualize.py:90
The model_args can be a String, but it can also be a json, according to /lm_eval/__main__.py:96
``` python
    parser.add_argument(
        "--model_args",
        "-a",
        default="",
        type=try_parse_json,
        help="""Comma separated string or JSON formatted arguments for model, e.g. `pretrained=EleutherAI/pythia-160m,dtype=float32` or '{"pretrained":"EleutherAI/pythia-160m","dtype":"float32"}'""",
    )
```
So this script may cause error

Hi! thanks for raising the issue. Would you be interested in submitting a PR?

Hi @baberabb 
I am interested in solving this issue!
Can you please assign it to me?
Repository: EleutherAI/lm-evaluation-harness
Issue: Error  Attempted to run tasks are text-only, but used a model type which only currently supports multimodal tasks
Root cause: Validation Gap

Seems the error comes from a recent merge https://github.com/EleutherAI/lm-evaluation-harness/pull/2981/files
There is an attribute called
```MULTIMODAL = True``` in class TemplateAPI(TemplateLM) in api_models.py.
This makes any model lile ```local-completions``` or ```local-chat-completions``` incompatible with tasks like ```gsm8k```.
It will get an error here:
```if getattr(lm, "MULTIMODAL", False) != getattr(task, "MULTIMODAL", False):``` in evaluator.py
Repository: EleutherAI/lm-evaluation-harness
Issue: Docker build fails due to missing pip module in Conda environment during setup.py develop on editable install
Root cause: Environment Incompatibility

Encountering a build failure when attempting to build the Docker image for lm-eval-harness. The issue seems to originate from the use of setup.py develop, which triggers a subprocess that attempts to run pip—but pip is not found in the active Conda environment (py_3.9).

This started happening recently without any changes and seems related to either the base Conda image or changes in how pip/setuptools behave during PEP 517-based installs.

requirements.txt


```
....
-e git+https://github.com/EleutherAI/lm-evaluation-harness.git@v0.4.5#egg=lm_eval
....
```


```
Running setup.py develop for lm-eval
ERROR: Command errored out with exit status 1:
  command: /opt/conda/envs/py_3.9/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/build/src/lm-eval/setup.py'"'"'; __file__='"'"'/build/src/lm-eval/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix=
  cwd: /build/src/lm-eval/
  ...
  /opt/conda/envs/py_3.9/bin/python: No module named pip
  ...
  subprocess.CalledProcessError: Command '['/opt/conda/envs/py_3.9/bin/python', '-m', 'pip', 'install', '-e', '.', '--use-pep517', '--no-deps', '--user']' returned non-zero exit status 1.
```