zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluate GPT-OSS-20B on code benchmark datasets
Root cause: Documentation Deficiency

Hi, apologies if the question is naive. I'm trying to evaluate GPT-OSS-20B on code benchmark datsets(e.g. mbpp, humaneval).
```
export HF_ALLOW_CODE_EVAL=1
accelerate launch --num_processes 1 -m lm_eval \
  --model hf \
  --model_args pretrained=openai/gpt-oss-20b,dtype=bfloat16 \
  --tasks mbpp\
  --batch_size 32 \
  --confirm_run_unsafe_code \
```
and I got following results:
|    Tasks    |Version|   Filter   |n-shot| Metric  |   |Value|   |Stderr|
|-------------|------:|------------|-----:|---------|---|----:|---|-----:|
|mbpp|      1|extract_code|     3|pass_at_1|↑  |    0|±  |     0|
|mbpp_instruct|      1|extract_code|     3|pass_at_1|↑  |    0|±  |     0|
|humaneval|      1|create_test|     0|pass@1|   |0.2622|±  |0.0345|

which are relativly low. I'm wondering can we use thinking mode when evaluating these tasks? And if so, how should I do it?


Oops, I have solved it. Btw, I still want to use think mode to conduct code evaluation. Is there any way to do it?
```
export HF_ALLOW_CODE_EVAL=1
accelerate launch --num_processes 1 -m lm_eval \
  --model hf \
  --model_args pretrained=openai/gpt-oss-20b,parallelize=True,dtype=bfloat16 \
  --tasks mbpp_instruct \
  --confirm_run_unsafe_code \
  --num_fewshot 3 \
  --apply_chat_template \
  --gen_kwargs temperature=0,top_p=0,max_new_tokens=512 \
  --batch_size 1 \
  --limit 10 \
  --log_samples \
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Loglikelihood is not supported
Root cause: Unimplemented Feature Gap

I was running GLUE benchmarking tasks such as CoLA, QQP, and SST-2 using lm-eval.
By default, the output_type is set to multiple_choice, which internally calls loglikelihood.
However, with chat-based models (e.g., OpenAI chat completions), this fails with the following error: 



Traceback (most recent call last):
  File "/home/myusername/lm-evaluation-harness/myenv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/myusername/lm-evaluation-harness/lm_eval/__main__.py", line 450, in cli_evaluate
    results = evaluator.simple_evaluate(
  ...
  File "/home/myusername/lm-evaluation-harness/lm_eval/models/openai_completions.py", line 186, in loglikelihood
    raise NotImplementedError(
NotImplementedError: Loglikelihood is not supported for chat completions. Consider using the completions API instead.



sample yaml : 

tag: glue
task: cola
dataset_path: nyu-mll/glue
dataset_name: cola
output_type: multiple_choice
training_split: train
validation_split: validation
doc_to_text: "{{sentence}}\nQuestion: Does this sentence make sense?\nAnswer:"
doc_to_target: label
doc_to_choice: ["no", "yes"]
should_decontaminate: true
doc_to_decontamination_query: sentence
metric_list:
  - metric: mcc
metadata:
  version: 1.0



Bash Command Used: 
 

#!/bin/bash



export HF_HOME=/data
export HF_ALLOW_CODE_EVAL=1
export HF_TOKEN=
export HUGGINGFACE_HUB_TOKEN=
export MODEL_NAME="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
export BASE_URL="some url" 
export AUTH_TOKEN="sk-************"



lm_eval --model local-chat-completions \
        --tasks gsm8k \
        --apply_chat_template \
        --model_args "model=$MODEL_NAME,base_url=$BASE_URL,num_concurrent=1,max_retries=3,tokenizer=/data/local_tokenizer,tokenized_requests=False"

 



What I Tried

Changing output_type to generate_until in the YAML config.
This runs, but for CoLA, it only returns 0.0 values (not useful).

My Questions

Is it valid/appropriate to use a different output_type like generate_until for GLUE tasks?
Will this lead to wrong evaluation results compared to the intended benchmark setup?
How can I correctly configure the YAML (or adapt the task setup) so that GLUE tasks work with chat models that don’t support loglikelihood?
Repository: EleutherAI/lm-evaluation-harness
Issue: `acc_norm`: character vs byte length
Root cause: Validation Gap

Just wanted to seek some clarification about how `acc_norm` is calculated. 

In the recent paper about the Harness ([Biderman et al., 2024](https://arxiv.org/abs/2405.14782)), it is mentioned that for `acc_norm`, 'each [answer]’s loglikelihood is divided by its length in bytes'. However, in an early [blog post](https://blog.eleuther.ai/multiple-choice-normalization/), it is mentioned that this is calculated as 'average log probability per character', and in line 1584 of `lm_eval/api/task.py` (link [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/2d7cb5c31cffd3cbeb5367542ab8f4c23f4b77f4/lm_eval/api/task.py#L1584)), completion lengths are calculated like this:

```
 completion_len = np.array([float(len(i)) for i in choices])
```

To me, this looks like `completion_len` is being calculated as the length of each choice in characters rather than bytes. 

While this may not make much of a difference for English-language datasets, where each character is generally represented by one byte, this differs substantially across languages (see e.g., [Costa-jussà et al., 2017](https://aclanthology.org/W17-4123/)).

Length-normalization doesn't seem like an area where there is a single clear and universal answer for all cases, so I am not advocating for either version in particular, but I just wanted to check which is being implemented for clarity.

Hi!, you are correct in this. Would calculating the length of the `utf-8` bytes be sufficient, do you think?
Repository: EleutherAI/lm-evaluation-harness
Issue: Extending lm_eval with plugins
Root cause: Unimplemented Feature Gap

Currently, extending `lm_eval` requires modifying the source tree. For instance, adding a new model requires a change in `lm_eval/models/__init__.py` https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md#registration

It would be helpful to be able to extend `lm_eval` without needing to modify the source tree ("out of tree"). One common way to do this in Python is using plugins https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/. For instance, if the namespace package approach was adopted using the `lm_eval.plugins` namespace, one could create a separate Python distribution package, say `lm-eval-foo`, that can be independently installed. The source tree for `lm-eval-foo` might look something like
* `pyproject.toml`
* `foo`
  * `__init__.py`
  * `custom_model.py`  <-- `from lm_eval.api.registry import register_model`
* `lm_eval`
  * `plugins`
    * `foo`
      * `__init__.py` <-- `import foo.custom_model`

In `lm_eval/__init__.py`, you'd have code like ([source](https://packaging.python.org/en/latest/guides/creating-and-discovering-plugins/#using-namespace-packages))
```python
import importlib
import pkgutil

import lm_eval.plugins

def iter_namespace(ns_pkg):
    # Specifying the second argument (prefix) to iter_modules makes the
    # returned name an absolute name instead of a relative one. This allows
    # import_module to work without having to do additional modification to
    # the name.
    return pkgutil.iter_modules(ns_pkg.__path__, ns_pkg.__name__ + ".")

discovered_plugins = {
    name: importlib.import_module(name)
    for finder, name, ispkg
    in iter_namespace(lm_eval.plugins)
}
```

Then someone who wants to use `lm-eval-foo` can simply install it alongside `lm-eval` (`pip install lm-eval lm-eval-foo`) and its extensions should be available in the CLI: `python -m lm_eval --model foo ...`
Repository: EleutherAI/lm-evaluation-harness
Issue: 'str' object has no attribute 'prompt'
Root cause: Validation Gap

> @baberab  I'm getting> 
> > 
> ```> 
> Traceback (most recent call last):> 
>   File "/home/aidan/cookbook-internal/.venv/bin/lm_eval", line 10, in <module>> 
>     sys.exit(cli_evaluate())> 
>   File "/home/aidan/cookbook-internal/.venv/lib/python3.10/site-packages/lm_eval/__main__.py", line 455, in cli_evaluate> 
>     results = evaluator.simple_evaluate(> 
>   File "/home/aidan/cookbook-internal/.venv/lib/python3.10/site-packages/lm_eval/utils.py", line 456, in _wrapper> 
>     return fn(*args, **kwargs)> 
>   File "/home/aidan/cookbook-internal/.venv/lib/python3.10/site-packages/lm_eval/evaluator.py", line 357, in simple_evaluate> 
>     results = evaluate(> 
>   File "/home/aidan/cookbook-internal/.venv/lib/python3.10/site-packages/lm_eval/utils.py", line 456, in _wrapper> 
>     return fn(*args, **kwargs)> 
>   File "/home/aidan/cookbook-internal/.venv/lib/python3.10/site-packages/lm_eval/evaluator.py", line 585, in evaluate> 
>     resps = getattr(lm, reqtype)(cloned_reqs)> 
>   File "/home/aidan/cookbook-internal/.venv/lib/python3.10/site-packages/lm_eval/models/api_models.py", line 662, in generate_until> 
>     requests = tuple(> 
>   File "/home/aidan/cookbook-internal/.venv/lib/python3.10/site-packages/lm_eval/models/api_models.py", line 666, in <genexpr>> 
>     y["visual"][: self.max_images], json.loads(x.prompt)> 
> AttributeError: 'str' object has no attribute 'prompt'> 
> ```> 
> > 
> ```> 
> lm_eval --model local-chat-completions \> 
>   --tasks mmmu_val \> 
>   --model_args base_url=http://localhost:80/v1/completions> 
> ```> 
> > 
> Here - shouldn't just be json.loads(x)? 

 _Originally posted by @aidando73 in [2cfdd0a](https://github.com/EleutherAI/lm-evaluation-harness/commit/2cfdd0a294214fe156caa1b8a69da17a29c39e63#r165376902)_

@aidando73 I think you should run lm-eval with `--apply_chat_template \` to get the formatting right. That worked for me.
Repository: EleutherAI/lm-evaluation-harness
Issue: MMLUSR tasks are broken
Root cause: Configuration Error

Evaluations cannot be ran on MMLUSR tasks because the dataset names do not correspond to the [dataset schema on HuggingFace](https://huggingface.co/datasets/NiniCat/MMLU-SR).  

Suppose a user wants to run evals on a specific subtask, say, `mmlusr_answer_only_anatomy`:
```
python3 -m lm_eval --model dummy --tasks mmlusr_answer_only_anatomy --limit 1
```

The above command returns the following error:
```
...
  File "/Users/chrxu/Documents/lm-evaluation-harness/lm_eval/api/task.py", line 822, in __init__
    self.download(self.config.dataset_kwargs)
  File "/Users/chrxu/Documents/lm-evaluation-harness/lm_eval/api/task.py", line 929, in download
    self.dataset = datasets.load_dataset(
  File "/Users/chrxu/Documents/lm-evaluation-harness/.venv/lib/python3.10/site-packages/datasets/load.py", line 2062, in load_dataset
    builder_instance = load_dataset_builder(
  File "/Users/chrxu/Documents/lm-evaluation-harness/.venv/lib/python3.10/site-packages/datasets/load.py", line 1819, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/Users/chrxu/Documents/lm-evaluation-harness/.venv/lib/python3.10/site-packages/datasets/builder.py", line 343, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/Users/chrxu/Documents/lm-evaluation-harness/.venv/lib/python3.10/site-packages/datasets/builder.py", line 570, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'mmlusr_answer_only_anatomy' not found. Available: ['answer_only', 'question_only', 'question_and_answer']
```

Happy to work on this issue and I already have a draft fix here: https://github.com/christinaexyou/lm-evaluation-harness/commit/998c7ae5d1516b53f117b810b4293c53c74f98c6

Hi! Thanks for identifying that issue. And yes, will appreciate a PR!
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug: UnboundLocalError on outputs in Exception Logging
Root cause: Validation Gap

#### Error:
```
UnboundLocalError: local variable 'outputs' referenced before assignment
```
This happens when an exception is raised before outputs is defined, but it's still referenced in the error log:
```
eval_logger.error(f"Exception:{repr(e)}, {outputs}, retrying.")
```

#### Suggestion:
Either initialize `outputs = None` before the try, or safely access it with `locals().get('outputs', None)` in the log line.


Anyone else run into this?

Hi! thanks for raising this issue. PRs definitely welcome if you'd like to submit a fix, otherwise I'll get to it soon!
Repository: EleutherAI/lm-evaluation-harness
Issue: Question: Does repeats work for evaluation averaging?
Root cause: Unimplemented Feature Gap

Hi, I was evaluating on AIME24 with the following config:

```yaml
tag:
  - math_word_problems
task: AIME
repeats: 8
dataset_path: json
dataset_kwargs:
  trust_remote_code: true
  data_files: 
    train: math_eval_data/aime24/test.jsonl
    validation: math_eval_data/aime24/test.jsonl
    test: math_eval_data/aime24/test.jsonl
process_docs: !function utils.process_docs
dataset_name: null
output_type: generate_until
training_split: train
fewshot_split: train
test_split: test

doc_to_text:  "Solve the following math problem. Present the final answer in the format: Final Answer: \\boxed{your_answer}.\nProblem: {{problem}}\nAnswer:"
process_results: !function utils.process_results
doc_to_target: "{{solution}}"
generation_kwargs:
  until:
    - "Problem:"
  do_sample: true
  temperature: 0.6
  top_p: 0.95
  max_gen_toks: 16384
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: false
    regexes_to_ignore:
      - ","
      - "\\$"
      - "(?s).*#### "
      - "\\.$"
metadata:
  version: 1.0

```
Since there are only 30 questions in AIME 24, the results from a single run can vary quite a lot. That’s why I wanted to repeat the evaluation several times (with different randomness/seeds) and then average the results to get a more stable estimate.

However, the reported score was 53.33, which matches a single run result (16/30), rather than what I expected (an average over 8 runs).

My questions are:

Is repeats supposed to run the task multiple times with different randomness and then average the results?

Or is it only used for the maj@k metric to assess self-consistency?

If I want to run the task N times and report the mean ± std across runs, should I manually script this (e.g., running with different --seed values), or is there already built-in support for this use case?

Thanks!

afaik there's no support for repeats yet; you can set repeats to some value and it would run multiple times but it would only score based on the first one

check this [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/3351)
Repository: EleutherAI/lm-evaluation-harness
Issue: Integrate Custom KV Cache Compression Method
Root cause: Unimplemented Feature Gap

I am looking to integrate my custom KV cache compression method into the lm-evaluation-harness project. I have already attempted inserting the method into the _model_generate and _model_call functions. While this resulted in a change in the KV cache size, the model's accuracy remained unchanged.

Hey! I'm also interested in integrating my own KV cache compression method. Did you solve this issue?
Repository: EleutherAI/lm-evaluation-harness
Issue: local-completions model class does not support multimodal (image) tasks
Root cause: Unimplemented Feature Gap

When attempting to evaluate a multimodal model (Qwen2.5-VL) via a locally hosted API using the --model local-completions argument, the evaluation fails with an AssertionError stating that the tokenizer is not supported for multimodal requests yet.
```python
Traceback (most recent call last):
  File "/usr/local/python3.11.13/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/workspace/tools/zmj_lmeval/lm-evaluation-harness/lm_eval/__main__.py", line 455, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tools/zmj_lmeval/lm-evaluation-harness/lm_eval/utils.py", line 456, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/workspace/tools/zmj_lmeval/lm-evaluation-harness/lm_eval/evaluator.py", line 357, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/workspace/tools/zmj_lmeval/lm-evaluation-harness/lm_eval/utils.py", line 456, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/workspace/tools/zmj_lmeval/lm-evaluation-harness/lm_eval/evaluator.py", line 585, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tools/zmj_lmeval/lm-evaluation-harness/lm_eval/models/api_models.py", line 652, in generate_until
    assert self.tokenizer is None, (
           ^^^^^^^^^^^^^^^^^^^^^^
AssertionError: tokenizer is not supported for multimodal requests yet!
``` 
lm_eval command:
```
lm_eval \
  --model local-completions \
  --model_args model=/data/models/Qwen2.5-VL-7B-Instruct,base_url=http://xxx.xxx.xxx.xxx:28888/v1/chat/completions \
  --tasks mmmu_agriculture_test \
  --batch_size 1
``` 
mmmu_agriculture_test.yaml
```
task: mmmu_agriculture_test
dataset_path: parquet
dataset_name: null
dataset_kwargs:
  data_files:
    # train and test data download path
    validation: /workspace/tools/zmj_lmeval/lm-evaluation-harness/lm_eval/tasks/mmmu/Agriculture/validation-00000-of-00001.parquet
    
validation_split: validation
output_type: generate_until
doc_to_image: !function utils.doc_to_image
doc_to_text: !function utils.doc_to_text
doc_to_target: "answer"
process_results: !function utils.process_results
generation_kwargs:
  until:
    - "
"
  temperature: 0.0
  do_sample: false
  max_gen_toks: 512
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 0.0
```
Repository: EleutherAI/lm-evaluation-harness
Issue: [Dependencies] Allow for `datasets>=4.0` ?
Root cause: External Dependency Breakage

We had a user report in our repo
* https://github.com/vllm-project/llm-compressor/issues/1848

That our `datasets>=4.0` pin makes us incompatible with `lm_eval` which pins `datasets<4.0`. I noticed that `lm_eval` pinned it shortly after the 4.0.0 release:
* #3172 

I consistently hit this error on `datasets==3.6.0`:
```
File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/datasets/features/features.py", line 1474, in generate_from_dict
    raise ValueError(f"Feature type '{_type}' not found. Available feature types: {list(_FEATURE_TYPES.keys())}")
ValueError: Feature type 'List' not found. Available feature types: ['Value', 'ClassLabel', 'Translation', 'TranslationVariableLanguages', 'LargeList', 'Sequence', 'Array2D', 'Array3D', 'Array4D', 'Array5D', 'Audio', 'Image', 'Video', 'Pdf']
```

Can we loosen the datasets pin to `datasets<5.0`? Happy to create a quick PR if maintainers are board.


A minimal script to reproduce above error with `lm_eval==0.4.9.1` and `datasets==3.6.0`:

```python
import lm_eval

SAVE_DIR = "meta-llama/Meta-Llama-3-8B-Instruct"

results = lm_eval.simple_evaluate(
    model="vllm",
    model_args={
        "pretrained": SAVE_DIR,
        "add_bos_token": True,
        "dtype": "auto",
        "max_model_len": 4096,
        "gpu_memory_utilization": 0.4,
        "enable_chunked_prefill": True,
    },
    tasks=[
        "arc_challenge_llama",
        "gsm8k_llama",
    ],
    num_fewshot=5,
    apply_chat_template=True,
    fewshot_as_multiturn=True,
    batch_size=64,
)
print(lm_eval.utils.make_table(results))
```

My impression was that changing the datasets version to >4.0 would break a bunch of tasks, see issue #3171 

Thanks @jannalulu , I missed that issue in my original search. Will monitor thread there.

Hi! unpinned `datasets` in the PR. There are a few tasks still using runtime loading scripts, but most have been updated. `datasets==4.0`, also introduced the `List` type, which is probably why you're getting the error. 

I'll try making a final tally of the tasks left over before merging.

Thank you @baberabb !
Repository: EleutherAI/lm-evaluation-harness
Issue: simple_evaluate does not support support accelerate-wrapped models
Root cause: Unimplemented Feature Gap

## Problem
The `simple_evaluate()` function doesn't support models wrapped with `accelerate`, which limits its use as a library for distributed evaluation scenarios.

## Current Behavior
When passing an accelerate-wrapped model to `simple_evaluate()`, it fails because the function expects either a string model name or a base model object, as noted in [this comment](https://github.com/EleutherAI/lm-evaluation-harness/issues/1935#issuecomment-2557210170) in Issue #1935.  When setting parallelize=True, an assert is raised: [models/huggingface.py#L115](https://github.com/DhruvaKashyap/lm-evaluation-harness/blob/de496b80d60c267a2d7eea3b3c1dc40f693daee7/lm_eval/models/huggingface.py#L115).

## Expected Behavior
`simple_evaluate()` should accept and properly handle models wrapped by Accelerate.

###  Example
```python
model = AutoModel.from_pretrained(...)
tokenizer = ...
model = accelerate.prepare_model(model)
... # Do something to change the model parameters
wrapped_model = HFLM_Accelerated(model, tokenizer, accelerator, batch_size=eval_cfg.eval_batch_size)
lm_eval.evaluator.simple_evaluate(
            model=wrapped_model,
            tasks=tasks, 
            batch_size=eval_cfg.eval_batch_size,
            device=accelerator.device,
 )
```

## Proposed Solution
I have implemented a solution and would like to contribute a PR to address this limitation.

Hi! Thanks for the PR and will review next week!

Hi @baberabb, I’ve updated the PR to fix a linting issue. All checks on pre-commit are passing now.
Looking forward to your feedback when you have a chance
Repository: EleutherAI/lm-evaluation-harness
Issue: Custom aggregation ignored when metric function name doesn't match return keys
Root cause: Validation Gap

When creating a custom aggregation and metric function for a task, the aggregation function is not triggered instead it goes to default mean aggregation.
k=[1,3,5]

utils.py
```
def agg(arr):
    return sum(arr)

def pass_at_k(references: list[str], predictions: list[list[str]], k: list[int] = None):
    global compute_
    assert k is not None
    if isinstance(k, int):
        k = [k]
    res = compute_.compute(
        references=references,
        predictions=predictions,
        k=k,
    )
    return res[0] # {"pass@1":1.0,"pass@3":1.0,"pass@5":1.0}
```

humaneval.yaml
```metric_list:
  - metric: !function utils.pass_at_k
    aggregation: !function utils.agg
    higher_is_better: true```

Hi! This is a bug. The custom aggregation function is only triggered when the _metric function's name_ matches the keys returned by the metric. Since your function is named `pass_at_k` but returns keys like `pass@1`, `pass@3`, etc., the aggregation doesn't get applied and falls back to the default mean aggregation. 

Will submit a PR to fix!
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation of encoder and decoder models on SuperGLUE
Root cause: Unimplemented Feature Gap

Hi guys,

I want to evaluate models like ModernBERT, Llama and many others on SuperGLUE and my own benchmark. In my setting, every model has to be fine-tuned for the specific task, even decoder models.

Is this currently supported by Harness? Looking at the code, my feeling is that evaluations are only done by prompting.

Thanks.
Repository: EleutherAI/lm-evaluation-harness
Issue: AIME 2024 support
Root cause: Documentation Deficiency

Hello, thank you for your great work.
in the issues it is said that AIME dataset is supported, but i can not find it the task's list? can you please help? what is the exact name of the AIME2024 dataset in the tasks?
thanks


it's called `aime24`