zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue:  OOM Issues in MMLU Evaluation with lm_eval Using vllm as Backend
Root cause: Interface Contract Mismatch

I have recently encountered frequent OOM issues while running MMLU evaluation tasks using lm_eval with vllm as the backend. After some investigation, I found that this issue arises because the sampling parameters used by lm_eval are inconsistent with the default parameters used by vllm to estimate peak memory usage. In fact, the memory used by lm_eval during actual execution is much higher than the peak memory estimated by vllm (for the Meta-Llama-8B-Instruct model, using batch size auto, 10GB vs 50GB).

I have already reported [this issue](https://github.com/vllm-project/vllm/issues/10325) to vllm to see if they can provide a way for lm_eval to configure the default sampling parameters to resolve this issue. In the meantime, modifying the default [sampling parameters](https://github.com/vllm-project/vllm/blob/32e7db25365415841ebc7c4215851743fbb1bad1/vllm/worker/model_runner.py#L1047) in vllm could temporarily bypass this OOM issue, and I believe this workaround could be helpful for others encountering the same problem.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Performance significantly drop when increase the batch_size]
Root cause: External Dependency Breakage

Hello, I use the latest and v0.4.3 version of lm_eval and I find the weird phenomena on `llama-3.2-3B`
The following is my script:
```
BATCH_SIZE=256

torchrun --nproc-per-node=8 --no-python lm_eval \
    --model_args pretrained=meta-llama/Llama-3.2-3B \
    --tasks gsm8k_cot \
    --batch_size BATCH_SIZE 
```

[llama-3.2-3B] Nodes=1, GPUs=8, llama

```
batch_size=1
|  Tasks  |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|---------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_cot|      3|flexible-extract|     8|exact_match|↑  |0.2987|±  |0.0126||         
|         |       |strict-match    |     8|exact_match|↑  |0.2835|±  |0.0124|

batch_size=32
|  Tasks  |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|---------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_cot|      3|flexible-extract|     8|exact_match|↑  |0.2790|±  |0.0124|
|         |       |strict-match    |     8|exact_match|↑  |0.2616|±  |0.0121|
 
batch_size=128
|  Tasks  |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|---------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_cot|      3|flexible-extract|     8|exact_match|↑  |0.1304|±  |0.0093|
|         |       |strict-match    |     8|exact_match|↑  |0.1221|±  |0.0090|

batch_size=256
|  Tasks  |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|---------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_cot|      3|flexible-extract|     8|exact_match|↑  |0.0409|±  |0.0055|
|         |       |strict-match    |     8|exact_match|↑  |0.0364|±  |0.0052|
```
 
accelerate                1.0.1



Hi! I can't reproduce this (1 GPU). Are you using the latest `transformers`?

`hf (pretrained=meta-llama/Llama-3.2-3B), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 256`
|  Tasks  |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|---------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_cot|      3|flexible-extract|     8|exact_match|↑  |0.2980|±  |0.0126|
|         |       |strict-match    |     8|exact_match|↑  |0.2828|±  |0.0124|
Repository: EleutherAI/lm-evaluation-harness
Issue: french_bench_xnli dataset doesn't exist 
Root cause: Documentation Deficiency

Hi

For the task french_bench_xlni the dataset xlni can not be found on huggingface hub
Any idea ? should this subtask be removed ? 

Regards


Hi! its working for me! dataset's [here](https://huggingface.co/datasets/facebook/xnli)
Repository: EleutherAI/lm-evaluation-harness
Issue: Is there a good method to set the generation_kwargs for tasks so that different kv caches can be used for optimization?
Root cause: Documentation Deficiency

In this [transformers documentation](https://huggingface.co/docs/transformers/en/kv_cache), there are many kv cache methods available. However, I find it inconvenient to set `past_key_values` in the `generation_kwargs` of a `task`, for example, when using `SinkCache`. My proposed solution is to build a subclass of `Task` to implement setting `past_key_values` in the `generation_kwargs`. Is there a better way?

Hi! you can pass it on to the command for ex. `--gen_kwargs cache_implentation=....,<other kwargs to model.generate>`

> Hi! you can pass it on to the command for ex. `--gen_kwargs cache_implentation=....,<other kwargs to model.generate>`

Thank you for your reply, but something like `SinkCache` cannot use parameters directly because it needs to be initialized in advance and passed to `model.generate` through `past_key_values`.
Repository: EleutherAI/lm-evaluation-harness
Issue: Fix blimp tests
Root cause: Others


@igor-krawczuk might have something to say about this?

I don't think so? I got knocked out by health issues a while ago and have been trying to sort out my day job enough to have time to rejoin the discord and properly contribute to eleuther projects

@igor-krawczuk Oops! Wrong Igor. I meant to tag @igor0.

Sorry to hear about your health problems :( I hope things aren’t too bad.

@leogao2: any more info about what's broken with blimp tests? Did they just start failing all of a sudden?
Repository: EleutherAI/lm-evaluation-harness
Issue: Got near random guess results for GPQA.
Root cause: Documentation Deficiency

I observed this when evaluating my own models, but such behavior keeps when I switch to Meta-LLaMA-3-8B-Instruct, whose officially reported results are much higher.

**GPQA of LLaMA-3-8B-Instruct (In model card the claimed acc is 32.6)**
|           Tasks            |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|----------------------------|-------|------|-----:|--------|---|-----:|---|-----:|
|leaderboard_gpqa            |    N/A|      |      |        |   |      |   |      |
| - leaderboard_gpqa_diamond |      1|none  |     0|acc_norm|↑  |0.2626|±  |0.0314|
| - leaderboard_gpqa_extended|      1|none  |     0|acc_norm|↑  |0.2729|±  |0.0191|
| - leaderboard_gpqa_main    |      1|none  |     0|acc_norm|↑  |0.2455|±  |0.0204|

Besides, results of ARC-C are also much lower (In model card the claimed 0-shot acc is 82.4)
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |     5|acc     |↑  |0.5776|±  |0.0144|
|             |       |none  |     5|acc_norm|↑  |0.6041|±  |0.0143|

Here is my command
```
lm_eval --model hf \
        --tasks leaderboard_gpqa \ # changed according to tasks
        --model_args pretrained=path/to/model,dtype=bfloat16 \
        --write_out \
        --fewshot_as_multiturn \
        --log_samples \
        --output_path some/local/path/ \
        --apply_chat_template
```

I notice that in almost all samples logprobs of A is the highest, thus only ones whose correct answer is A are considered correct. This might explain why the acc is near to random guess. 

Also, I wonder how the logprobs are collected. If they are logprob of the first generated token (output of the last prompt token), for tasks like GPQA it might be "(" instead of certain choice letter (this explains why ARC is better, as there are no brackets around choice letters in the input). Would it affect the results?

Do you have any suggestions?

> Also, I wonder how the logprobs are collected. If they are logprob of the first generated token (output of the last prompt token), for tasks like GPQA it might be "(" instead of certain choice letter (this explains why ARC is better, as there are no brackets around choice letters in the input). Would it affect the results?

Hi! The logprobs are summed over the logprobs of each token in the target (so not just the first generated token). The likely explanation is that the [prompts](https://huggingface.co/datasets/meta-llama/Llama-3.1-8B-Instruct-evals/viewer/Llama-3.1-8B-Instruct-evals__arc_challenge__details?row=0) used by them are different from the standard ones (for example they use a generation implantation for arc_challenge, and also provide more detailed instructions in the prompt). 

They released a [library](https://github.com/meta-llama/llama-recipes/tree/main/tools/benchmarks/llm_eval_harness) which you should be able to use to reproduce their results (it's for 3.1 but you should be able to make it work for 3 as well)
Repository: EleutherAI/lm-evaluation-harness
Issue: NotImplementedError: Cannot copy out of meta tensor; no data!
Root cause: External Dependency Breakage

I use **Autoawq** to quantize Llama3.2 and Llama 3.1, and it appears problem
`NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.`

and  I try change `self._model.to(device)` to `self._model.to_empty(device)`,but it does not work

```
Traceback (most recent call last):
。...
  File "anaconda3/envs/lmeval/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1333, in convert
    raise NotImplementedError(
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.
```


Hi! Can you load the model through `transformers`, separate from this library?
Repository: EleutherAI/lm-evaluation-harness
Issue: Parallelising Metrics Calculation
Root cause: Documentation Deficiency

<Feature Request>
Hi,

I implemented a task similar to HumanEval, that requires simulations to be run for each subtask in the metrics calculation. Due to the simulations, the metrics calculation is very slow. I know from my own implementation that multithreading the metrics calculations over several independent subtasks increases the speed by substantially. 

Therefore I wanted to ask if there is a way to do this in the evaluation harness? I tried implementing it myself in the evaluation harness, with limited success until now. Any hints on if this possible and if yes where I would need to adapt this would be appreciated.

Also as a followup, I implemented a custom metric (pass@k). A set of simulations for one subtask, yields several metrics (e.g. pass@1, pass@5, ...). However, I haven't figured out how to nicely implement returning several metrics in one custom_metric functions. Any hints on this swell would be appreciated.

Thanks for your help!
 

Hi! I think the simplest way is to create a [filter](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md#filters), which accepts all the model responses as a list along with their respective sample docs. [example](https://github.com/EleutherAI/lm-evaluation-harness/blob/5680a2e6b5cf1a1621d8ff68d3d0e83e8b2731d3/lm_eval/filters/extraction.py#L28); [task](https://github.com/EleutherAI/lm-evaluation-harness/blob/5680a2e6b5cf1a1621d8ff68d3d0e83e8b2731d3/lm_eval/tasks/gsm8k/gsm8k.yaml#L32). If you transform the outputs to binary here, then you'll just need to pass them through a simple no-op metric for scoring.

The second ones going to be tricky. Currently, the way it's structured is that the metric calculation occurs at the end of the pipeline and we expect it to output a scaler. [gsm8k-cot-self-consistency](https://github.com/EleutherAI/lm-evaluation-harness/blob/5680a2e6b5cf1a1621d8ff68d3d0e83e8b2731d3/lm_eval/tasks/gsm8k/gsm8k-cot-self-consistency.yaml#L19-L22) does return several pass@k scores, but each filter is computed independently. We do allow `--predict_only`, which skips the metric calculation and logs the model generations and filtered responses to file. Maybe that can be helpful?


Amazing, that works! I now simulate in parallel in a Filter and calculate the different pass@k in separate metrics. I'll clean the code and share it later, for anyone working on similar problems. 

Thank you for your help!

Hi! Could you share your code on how to implement the pass@k? Many thanks!

Hey,
Unfortunately, I needed to add functionalities particular to my application, rendering it quite unreadable, but maybe this PR helps? #2650 or #1992
Repository: EleutherAI/lm-evaluation-harness
Issue: Loaded AutoModelForCausalLM has no grad_fn
Root cause: Validation Gap

I am working on understanding the model gradients in LM inference. I attempted to capture the gradient in _loglikelihood_tokens methods with modified Huggingface Model class and loaded AutoModelForCausalLM from pretrained huggingface checkpoints.
                    
```
# Compute Cross-Entropy Loss
logits_reshaped = logits.view(-1, logits.size(-1))  # [seq, vocab]
targets = cont_toks.view(-1)  # [seq]

loss = F.cross_entropy(logits_reshaped, targets, ignore_index=-1)
loss.requires_grad = True

# Zero the gradients of the model and enable input requie_grads
self.model.zero_grad()
self.model.enable_input_require_grads()

# Enable gradient checkpointing with custom settings
self.model.gradient_checkpointing_enable()  # Enables gradient checkpointing
self.model.config.gradient_checkpointing_kwargs = {"use_reentrant": False}

for name, param in self.model.named_parameters():
      print(param.grad_fn)

# Compute gradients using autograd.grad
grads = torch.autograd.grad(loss, self.model.parameters(), 
                                                retain_graph=False,
                                                allow_unused=True)
```

However, all model parameters have no grad_fn (printed None). I attempted on Llama and Mistral and get the same results. 

Is there anyway I can load some AutoModelForCausalLM models with enabled grad_fn?

Hi! did you overload the `_model_call` method? :
https://github.com/EleutherAI/lm-evaluation-harness/blob/5680a2e6b5cf1a1621d8ff68d3d0e83e8b2731d3/lm_eval/models/huggingface.py#L846

We also set it to eval mode here:
https://github.com/EleutherAI/lm-evaluation-harness/blob/5680a2e6b5cf1a1621d8ff68d3d0e83e8b2731d3/lm_eval/models/huggingface.py#L202-L204

Thank you for your kind response. My issue is fixed by enabling torch.grad() in the _model_call method (simply comment out lines 202 to 204 is not enough).
Repository: EleutherAI/lm-evaluation-harness
Issue: import lm_eval 
Root cause: Environment Incompatibility

After I import lm_eval, the following information is displayed and it keeps getting stuck: "2024-11-30:09:45:25,234 INFO [utils.py:149] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2024-11-30:09:45:25,234 INFO [utils.py:162] NumExpr defaulting to 16 threads."
Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate correctness and errors in multiple-choice tasks?
Root cause: Documentation Deficiency

 "gen_args_0": { 
      "arg_0": "Question: a shopkeeper sold an article offering a discount of 5 % and earned a profit of 31.1 % . what would have been the percentage of profit earned if no discount had been offered ?\nAnswer:", 
      "arg_1": " 38" 
    },
    "gen_args_1": {
      "arg_0": "Question: a shopkeeper sold an article offering a discount of 5 % and earned a profit of 31.1 % . what would have been the percentage of profit earned if no discount had been offered ?\nAnswer:",
      "arg_1": " 27.675" 
    },
    "gen_args_2": {
      "arg_0": "Question: a shopkeeper sold an article offering a discount of 5 % and earned a profit of 31.1 % . what would have been the percentage of profit earned if no discount had been offered ?\nAnswer:",
      "arg_1": " 30" 
    }

............
"filtered_resps": [
    ["-7.680142164230347", "False"],
    ["-17.299633383750916", "False"],

Why are there two gen_args, and why are the resps numerical values?
How can we determine the correct answer generated by the LLM?
How to get the model’s original response?

Did you find solution
Repository: EleutherAI/lm-evaluation-harness
Issue: How to use gpt-neox/megatron trained gpt-2 models for the evaluation?
Root cause: Documentation Deficiency

Hello, We are using the gpt-neox to train gpt-2 models, and would like to use this repository to implement custom evaluation tasks. Is there any wrapper implemented to load and use such models?

There is currently a somewhat hack-y way to do it using the `evaluate.py` function in GPT-NeoX. By calling `./deepy.py evaluate.py 20B_checkpoints/configs/ 20B.yml --eval_tasks lambada ` you will call the evaluation harness’s lambada task from within GPT-NeoX. The API is slightly different (spaces vs comma delimiting is the big one), but it’s functional.

There’s currently an ongoing effort to revamp the interface between GPT-NeoX and the evaluation harness to provide a cleaner and more natural experience. Coming soon:tm:

@StellaAthena Thank you for your suggestion.
With 20B model, I evaluated few tasks such as CB (from SuperGLUE) and PIQA but it turned out that the resulting accuracy is about random guessing. 

This is clearly unexpected. I was wondering if you have evaluated any tasks with GPT-Neox 20B model and observed reasonable result ?


> @StellaAthena Thank you for your suggestion. With 20B model, I evaluated few tasks such as CB (from SuperGLUE) and PIQA but it turned out that the resulting accuracy is about random guessing.
> 
> This is clearly unexpected. I was wondering if you have evaluated any tasks with GPT-Neox 20B model and observed reasonable result ?

I have 77.9% on PiQA. Here’s some plots and tables from a preprint currently under review:

![6AB249C3-F524-4D3F-A7D0-81DFFAC13456](https://user-images.githubusercontent.com/15899312/158385258-098baaa1-972c-48ee-8137-b5e6c030765d.jpeg)
![318CF2AC-9192-4488-9945-25CF4C6EDEBD](https://user-images.githubusercontent.com/15899312/158385568-e42d6247-d4f6-4ed9-9124-9ba861b3a96e.jpeg)
![32674C82-6042-4436-B985-967479CA429E](https://user-images.githubusercontent.com/15899312/158385571-7bcd74d6-1237-477b-8333-8b26947503c4.jpeg)
Repository: EleutherAI/lm-evaluation-harness
Issue: long-time stuck after running loglikelihood requests
Root cause: Resource Mishandling

When I run large datasets such as `mmlu` and `hellaswag`, the program gets stuck after running loglikelihood requests. It takes several hours to output the final metric scores.

lm-eval version: 0.4.5
torch: 2.4.0
transformers: 4.46.2
vllm: 0.5.4

command:
```
lm_eval --model vllm \
    --model_args pretrained=${model_path},dtype=auto,gpu_memory_utilization=0.8,max_model_len=${max_model_len},data_parallel_size=${device_num} \
    --tasks mmlu \
    --batch_size auto \
    --log_samples \
    --output_path ${output_path} \
    --apply_chat_template
```

hello, i met the same problem, have you solved it?

> hello, i met the same problem, have you solved it?

still not

Glad to see I'm not the only one, have you tried to set your data_parallel_size to 1?

I am facing the same problem. Have you find any solution?


Hi, I am having the same problem, is there a known cause for this?

same issue here, could someone take a look?

same issue, using SGLang, 
```
lm_eval --model sglang --model_args pretrained=/mnt/lvm/25-team2/DeepSeek-R1-Distill-Qwen-32B,mem_fraction_static=0.8 --tasks mmlu --batch_size auto --cache_requests true
```
stuck at
```
Running loglikelihood requests:   0%|                                                         | 0/56168 [00:00<?, ?it/s]
```
with GPU memory full, GPU-util 100%

I am having the same issue. 

+1

Same issue in v0.4.9.2 on vllm
Repository: EleutherAI/lm-evaluation-harness
Issue: Feature Request: Resume Interrupted Evaluations
Root cause: Documentation Deficiency


**Description:**
Currently, the lm-eval-harness does not support resuming an evaluation from where it was interrupted. This can be particularly challenging for large-scale evaluations or when running on systems prone to interruptions (e.g., preemptible cloud instances). Implementing a feature to resume evaluations would greatly enhance usability and efficiency.


**Proposed Feature:**
Introduce a mechanism to resume evaluations from the last successfully completed task or batch. This could involve:

1. Periodically saving the evaluation state (e.g., tasks completed, intermediate results).
2. Allowing users to specify a "resume" flag or load the state automatically.
3. Ensuring compatibility with multi-task evaluations and avoiding duplicate computations.


**Benefits:**
- Saves computational resources by avoiding re-running completed evaluations.
- Improves user experience for long-running evaluation processes.
- Facilitates use on unstable or limited-resource environments.


**Implementation Suggestions:**
- Use a checkpointing mechanism to save progress in a structured file (e.g., JSON or pickle).
- Add a command-line argument, such as --resume or --checkpoint-path.
- Log details of the resumption process for transparency.


**Use Case Example:**
1. A user runs a large evaluation task across multiple datasets but the process is interrupted due to a system crash.
2. Instead of restarting the entire evaluation, the user runs the command again with the --resume flag.
3. The evaluation seamlessly picks up from where it left off, saving time and resources.


**Additional Context:**
This feature is commonly available in benchmarking tools and would align lm-eval-harness with user expectations for robust, large-scale evaluations.

Hi! we do actually have this implemented using  `--use_cache <DIR>` to cache the model results while evaluating and skip previously evaluated samples on resumption. Caching is rank-dependent though, so restart with the same GPU count if interrupted! Also have `--cache_requests` so the dataset preprocessing steps can be saved and evaluation can resume quicker.

I should update the README to make these more prominent!
Repository: EleutherAI/lm-evaluation-harness
Issue: Answer extraction logic for Math Lvl 5 (Open LLM Leaderboard 2) may be too strict
Root cause: Validation Gap

I've been observing a huge discprepancy between models on Math Lvl 5 on Open LLM Leaderbord 2. For example, 
	
- `Qwen/Qwen2.5-72B-Instruct`: 1.21
- `Qwen/Qwen2.5-7B-Instruct`, `Qwen/Qwen2.5-3B-Instruct`: 0
- `meta-llama/Llama-3.2-3B-Instruct`: 17.15

It's hard to believe that Qwen2.5 models, which show very strong scores on GSM8K, achieve zero or very low scores.

I was looking at the answer validation logic and found that the generation must follow the exact response format to be extracted using the regular expression (i.e., "The final answer is")

https://github.com/EleutherAI/lm-evaluation-harness/blob/f49b0377bf559f5558e8cd9ebd1190218c7df2a4/lm_eval/tasks/leaderboard/math/utils.py#L199

```
def get_unnormalized_answer(text: str) -> str:
    end_seq = "I hope it is correct."
    text += end_seq
    match = re.search(
        r"Final Answer: The final answer is(.*?). I hope it is correct.",
        text,
    )
    if match:
        return match.group(1).strip()
    else:
        return INVALID_ANSWER
```

I'm afraid that's too strict. Just missing "The final answer is" in the response could turn the perfect score into zero. 


- Q1. Is this a known issue (i.e., has there been discussed somewhere)? 
- Q2. Is there a plan to update the regular expression or another version of Math Lvl5 that has a more relaxed extraction logic?


Hi!
Yes, that's a known issue, but on our side, it's a feature not a bug: since we provide the prompt with 5-shot examples, we expect the models to be able to do in context learning of the formatting instead of overfitting format from their instruction fine tuning data.