zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: add_bos_token causes very unstable results for quantized llama3-70B
Root cause: Configuration Error

https://huggingface.co/OPEA/Llama-3.3-70B-Instruct-int3-sym-inc  
arc_easy   WO BOS  **0.2643** vs  with BOS  **0.8523**

https://huggingface.co/OPEA/Llama-3.3-70B-Instruct-int2-sym-inc
mmlu WO BOS  **0.7142** vs  with BOS   **0.7606**
lambada_openai WO BOS  **0.7013** vs  with BOS   **0.7413**

is it possible to align with the 16 bits model directly in lm-eval?

Hi! Is this with the chat template? generally with a template, you don't have to include the `bos` token manually as the chat template assures it is included

> Hi! Is this with the chat template? generally with a template, you don't have to include the `bos` token manually as the chat template assures it is included

we used the cmd recommended in the homepage, I don't know whether it used chat template or not
With BOS
lm-eval --model hf --model_args pretrained=OPEA/Llama-3.3-70B-Instruct-int3-sym-inc,add_bos_token=True   --tasks mmlu --batch_size 16

Without BOS

lm-eval --model hf --model_args pretrained=OPEA/Llama-3.3-70B-Instruct-int3-sym-inc --tasks mmlu --batch_size 16
Repository: EleutherAI/lm-evaluation-harness
Issue: ValueError: Trying to set a tensor of shape torch.Size([896, 768]) in "weight" (which has shape torch.Size([896, 4864])), this looks incorrect
Root cause: Architectural Constraint

I encountered a problem where I pruned the MLP layer of qwen2.5-0.5b,
"0": 4864,
"1": 4864,
"2": 4864,
"3": 4864,
"4": 2816,
"5": 4864,
"6": 4864,
"7": 3840,
"8": 4864,
"9": 2816,
"10": 4864,
"11": 4864,
"12": 4864,
"13": 768,
"14": 4864,
"15": 4864,
"16": 4864,
"17": 768,
"18": 4864,
"19": 4864,
"20": 4864,
"21": 4864,
"22": 4864,
"23": 4864
However, in the ` config. json ` file, the mediate_size is global and fixed at 4864, so when I tested the pruned model using lm eval, I encountered an error: ValueError: Trying to set a tensor of shape torch.Size([896, 768]) in "weight" (which has shape torch.Size([896, 4864])), this looks incorrect》 Is there any solution? Or is it another issue?
Repository: EleutherAI/lm-evaluation-harness
Issue: Add o3-mini support
Root cause: Validation Gap

Hi developers,

I've encountered an issue while trying to run o3-mini on a benchmark using lm-evaluation-harness. The error states: "Unsupported parameter: 'temperature' is not supported with this model."

The problem occurs because the OpenAIChatCompletion API automatically includes the 'temperature' parameter, which isn't compatible with o3-mini. This can be seen in the code here:
https://github.com/EleutherAI/lm-evaluation-harness/blob/a40fe42aae2a5232448495c7bb5bf76aef100309/lm_eval/models/openai_completions.py#L275-L291

To reproduce the issue:
```bash
lm_eval --model openai-chat-completions --model_args model=o3-mini --tasks mmlu_flan_cot_zeroshot_college_chemistry --limit 2 --output output/openai-4o --apply_chat_template --log_samples # fails
lm_eval --model openai-chat-completions --model_args model=o1-preview --tasks mmlu_flan_cot_zeroshot_college_chemistry --limit 2 --output output/openai-4o --apply_chat_template --log_samples # works
```

Maybe we need something like:
```python
if "o3" in self.model: 
     output.pop("temperature") 
```

Other discussion: https://github.com/ai-christianson/RA.Aid/issues/70

Hi! Would you be interested in submitting a PR? we can modify the condition [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/96f5e58f6cfb48f5ae7071eb5f58fe0b1df40467/lm_eval/models/openai_completions.py#L288) to include all the `o` models
Repository: EleutherAI/lm-evaluation-harness
Issue: Fix: Inconsistent spacing in arithmetic tasks affects few-shot performance
Root cause: Algorithmic Error

Hi all, I discovered this issue while reproducing arithmetic task results. Here are the details:

### Description
The arithmetic tasks show inconsistent behavior between zero-shot and few-shot evaluations due to a spacing issue in the example construction. Few-shot examples should maintain consistent spacing with evaluation examples, leading to expected improvements in few-shot performance.

### Current behavior
Few-shot examples are constructed with double spaces before answers, while evaluation examples have single spaces, causing dramatic performance degradation in few-shot settings.

For example, for arithmetic_2da, with Phi-1.5:
- Zero-shot: ~55.1% accuracy
- Two-shot (before fix): ~0.5% accuracy
- Two-shot (after fix): 78.5% accuracy

### Steps to reproduce
1. Run arithmetic_2da task with Phi-1.5:

```
python -m lm_eval \
    --model hf \
    --model_args "pretrained=microsoft/phi-1_5" \
    --tasks arithmetic_2da \
    --num_fewshot 2
```

2. Observe the constructed examples in samplers.py's get_context method, where:
- Target values in dataset start with a space (e.g., " 143")
- task_delimiter adds another space
- Results in double-spaced examples during few-shot

### Root cause
In`ContextSampler.get_context()`, spaces are concatenated:
```
if doc_target != "":
    labeled_examples += self.target_delimiter  # Adds a space
    labeled_examples += prefix
    labeled_examples += doc_target  # Target already has space
```

### Proposed fix
Add a check to prevent double spacing:
```
if doc_target != "":
    if not (self.target_delimiter.isspace() and str(doc_target)[0].isspace()):
        labeled_examples += self.target_delimiter
```

### Additional Context

- Multiple models tested (DeepSeek-Coder-6.7B-Base, Phi-1.5, GPT-J 6B, GPT-Neo 1.3B)
- Performance improves significantly across all models after fixing

Hi! Thanks for the report. The PR should fix it. I've also added a warning for other tasks.

Thanks for the quick fix and for adding the warning for other tasks! 👍
Repository: EleutherAI/lm-evaluation-harness
Issue: Feature request: allow peft revision separate from base model revision
Root cause: Unimplemented Feature Gap

Currently, the same [revision argument](https://github.com/EleutherAI/lm-evaluation-harness/blob/c3c05b067d28ecd238d6cffef32ddcbfd085a99d/lm_eval/models/huggingface.py#L62) is oddly used for both the [base pretrained model](https://github.com/EleutherAI/lm-evaluation-harness/blob/c3c05b067d28ecd238d6cffef32ddcbfd085a99d/lm_eval/models/huggingface.py#L588) and the [peft](https://github.com/EleutherAI/lm-evaluation-harness/blob/c3c05b067d28ecd238d6cffef32ddcbfd085a99d/lm_eval/models/huggingface.py#L648) model. There should be a separate `peft_revision` argument, so that you can load e.g. base Llama + a specific peft revision you own.
Repository: EleutherAI/lm-evaluation-harness
Issue: vLLM CUDA OOM for `loglikelihood`, but not for `generate_until`
Root cause: Resource Mishandling


Hello,

I persistently get issue of CUDA OOM when trying to evaluate DeepSeek-R1-Distill-Llama-8B model (locally downloaded) on HAE-RAE benchmark.
My command is as follows:
```
lm_eval --model vllm --model_args "pretrained=...,max_model_len=4096,max_num_seqs=256" --batch_size auto --tasks haerae
```
I have to reduce `max_num_seqs` to 4 not to get the OOM error.

However, when I evaluate the model on KMMLU benchmark, there is no issue of CUDA OOM.
My command was:
```
lm_eval --model vllm --model_args "pretrained=...,max_model_len=4096,max_num_seqs=256" --batch_size auto --tasks kmmlu_direct
```

The only difference is that HAE-RAE benchmark uses `loglikelihood` while KMMLU uses `generate_until`.
Can anyone explain why this is happening?

Thank you in advance.

Hi! This is a known issue with vllm. setting `gpu_memory_utilization` quite low helps (it depends on model as it's shared by the kv-cache  and the weights). Setting the `batch_size` manually also helps.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the SAT evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper:

> To test GPT-3 on another task that is somewhat unusual relative to the typical distribution of text, we collected a set of 374 “SAT analogy” problems [TLBS03]. Analogies are a style of multiple choice question that constituted a section of the SAT college entrance exam before 2005. A typical example is “audacious is to boldness as (a) sanctimonious is to hypocrisy, (b) anonymous is to identity, (c) remorseful is to misdeed, (d) deleterious is to result, (e) impressionable is to temptation”. The student is expected to choose which of the five word pairs has the same relationship as the original word pair; in this example the answer is “sanctimonious is to hypocrisy”. On this task GPT-3 achieves 65.2% in the few-shot setting, 59.1% in the one-shot setting, and 53.7% in the zero-shot setting, whereas the average score among college applicants was 57% [TL05] (random guessing yields 20%). As shown in Figure 3.12, the results improve with scale, with the the full 175 billion model improving by over 10% compared to the 13 billion parameter model.

- [X] Data processing code implemented
- [x] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Should be available on request from Peter Turney.

https://www.apperceptual.com/

Will post here if/when we get a response. 

Got a response. PM me on the Discord if you need access.

I could do the eval here
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug: vLLM v0.7.1 throws an error when running lm_eval with TP > 1
Root cause: Configuration Error

When running lm_eval with TP > 1 using vLLM v0.7.1, it throws a CUDA-related error due to multiprocessing.
The same script works fine with TP = 1.

### Script
```
python -m lm_eval --model=vllm \
     --model_args=pretrained=meta-llama/Meta-Llama-3-8B,tensor_parallel_size=2,gpu_memory_utilization=0.75,enforce_eager=True,trust_remote_code=True,max_gen_toks=256 \
     --tasks=lambada_openai,hellaswag \
     --batch_size=4 \
     --output_path=./outputs
```

### Error message
```
(VllmWorkerProcess pid=140536) INFO 02-16 09:08:11 multiproc_worker_utils.py:227] Worker ready; awaiting tasks
INFO 02-16 09:08:12 cuda.py:235] Using Flash Attention backend.
(VllmWorkerProcess pid=140536) INFO 02-16 09:08:12 cuda.py:235] Using Flash Attention backend.
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240] Exception in worker VllmWorkerProcess while processing method init_device.
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240] Traceback (most recent call last):
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]   File "/vllm-v0.7.1/vllm/executor/multiproc_worker_utils.py", line 234, in _run_worker_process
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]     output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]   File "/vllm-v0.7.1/vllm/utils.py", line 2208, in run_method
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]     return func(*args, **kwargs)
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]   File "/vllm-v0.7.1/vllm/worker/worker.py", line 154, in init_device
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]     torch.cuda.set_device(self.device)
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 478, in set_device
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]     torch._C._cuda_setDevice(device)
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]   File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 305, in _lazy_init
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240]     raise RuntimeError(
(VllmWorkerProcess pid=140536) ERROR 02-16 09:08:12 multiproc_worker_utils.py:240] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

```

This issue is resolved by setting `VLLM_WORKER_MULTIPROC_METHOD=spawn` option.
Repository: EleutherAI/lm-evaluation-harness
Issue: vllm 0.7 support (got an unexpected keyword argument 'worker_use_ray')
Root cause: External Dependency Breakage

It seems VLLM no longer supports 'worker_use_ray', which leads cannot run evaluations with the vllm+lm_eval

```bash
lm_eval --model vllm \
    --model_args pretrained=Qwen2.5-7B-Instruct,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=8 \
    --tasks mmlu \
    --batch_size auto
```

The error message is:
```bash
ray.exceptions.RayTaskError(TypeError): ray::run_inference_one_model() (pid=4153477, ip=10.1.202.37)
  File "/home/miniconda3/envs/rl/lib/python3.10/site-packages/lm_eval/models/vllm_causallms.py", line 258, in run_inference_one_model
    llm = LLM(**model_args)
  File "/home/miniconda3/envs/rl/lib/python3.10/site-packages/vllm/utils.py", line 1051, in inner
    return fn(*args, **kwargs)
  File "/home/miniconda3/envs/rl/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 212, in __init__
    engine_args = EngineArgs(
TypeError: EngineArgs.__init__() got an unexpected keyword argument 'worker_use_ray'
```


Thanks for the bug report! The PR should fix it.
Repository: EleutherAI/lm-evaluation-harness
Issue: Importing a local module in a task included with include_path
Root cause: Architectural Constraint

I'm having problems importing a local module in a custom task that is included from a separate folder via `--include_path`.

The ifeval task contains code that imports other Python modules from that task. For instance, ifeval's `utils.py` has the following import: `from lm_eval.tasks.ifeval import instructions_registry`.

Now, I'm working on a new task, let's call it `newtask`, and I need to do something similar, but this new task is not in the harness' `tasks` folder, but in a *separate folder*, let's call it `newtasksfolder`, which is included using `--include_path newtasksfolder` in the `lm_eval` command line.

For other tasks, with no local imports, this works fine. However, I haven't been able to find the proper way to do an import similar to the one above. The most straightforwad option, `from . import instructions_registry`, fails with `ImportError: attempted relative import with no known parent package`.

Ultimately, I guess I can just move the new task to the normal `tasks` folder of the harness and use `from lm_eval.tasks.newtask import instructions_registry`, but first I'd like to try to have the new task totally separate from the harness, as it is now.

Am I missing something or is the module loading (sys.path and so on) not dealing well with tasks that are loaded from a separate folder via include_path?

I'll have a look!

An update: I've been using a workaround that seems to work. It's not the ideal solution, but at least I can keep my configuration in a separate folder included with `--include_path`.

What I've done is add the following piece of code to the utils.py:
```
import sys
import os.path
sys.path.append(os.path.dirname(__file__))
```

I can then import from this utils.py the other modules in the same folder, like ifeval does, with `import instructions_registry` (but not, note, using the alternative format `from . import instructions_registry`; for that I guess I'd need to create a `__init__.py` and do some more stuff).



Glad you got it working! I looked into it as well, but wasn't able to come to a solution. Some background:
1. the yaml configs are parsed in this [file](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/__init__.py). It's a bit dense, but basically the main method used to parse and import the configs is [_load_individual_task_or_group](https://github.com/EleutherAI/lm-evaluation-harness/blob/1b357a68e576317aaac1bc4fa80ea41ba665d78c/lm_eval/tasks/__init__.py#L257), which in turn calls
2. this utility function `load_yaml_config` [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/1b357a68e576317aaac1bc4fa80ea41ba665d78c/lm_eval/utils.py#L470)

The logic for loading the function constructors is contained in `import_function` https://github.com/EleutherAI/lm-evaluation-harness/blob/1b357a68e576317aaac1bc4fa80ea41ba665d78c/lm_eval/utils.py#L448

Hope this is useful. Would appreciate a PR or further update if you decide to look into it more!
Repository: EleutherAI/lm-evaluation-harness
Issue: Different models on same tasks gives same results when cache is active
Root cause: Validation Gap

Executing the same task with same configs on two different models produces same identical resulting metrics when using cache. If I don't use cache I get the correct results (different metrics). Is cache working properly?


To reproduce the problem:
```
lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen2.5-3B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 1 \
    --limit 10 \
    --output_path output/test \
    --use_cache cache/cache.db

lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-3.2-3B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 1 \
    --limit 10 \
    --output_path output/test \
    --use_cache cache/cache.db
```

I'm using the latest version of the library installed directly from this repo.

will have a look! probably a hashing error

> will have a look! probably a hashing error

Is there any progress?

Hi! yeah, sorry meant to provide an update. So the current implementation simply checks if the inputs exist in the provided cache file, and does not take the model into account. Would be trivial to extend this to create unique cache db files for different `model_args`, but couple of issues:
1. Currently we allow passing both a model str or an already initialized model to `simple_evaluate`, and would run into some issues in the latter case.
2. Not clear which args should use different caches. Obviously we want it for unique `pretrained`. But other args like `max_length` or precision could also have an affect on performance, depending on the task.

Was thinking we always add a warning to use a different file for different models and leave it to user to decide.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to preprocess a document with the assistance of a tokenizer from a specific Model
Root cause: Unimplemented Feature Gap

For example, I encountered difficulties when integrating ```needle_in_a_haystack``` benchmark into ```lm-evaluation-harness```. During the preprocessing of documents, I needed to first tokenize the ```haystack document``` and then insert the ```needle``` at different positions, which required the involvement of a tokenizer. However, both the ```process_docs``` and ```doc_to_text```  unction only have a single doc parameter, making it impossible to pass in the tokenizer.


Hi! I've been working on the ruler benchmark in #2629. You can now use `download_dataset` : !function utils... in the config and that function will have access to the `tokenizer` or `pretrained` from model_args, as well as custom arguments passed to `--metadata`
Repository: EleutherAI/lm-evaluation-harness
Issue: Get acc_norm for HF models in log_samples
Root cause: Unimplemented Feature Gap

Hello, I am using the acc_norm metric to analyze the score of multiple options normalized by the length. However, I believe the unnormalized scores are displayed in .jsonl file generated by log_samples=True. Is it possible to display normalized scores since the decision is based on that instead of unnormalized scores?

Line 1241 in models/huggingface.py: 
`answer = (float(logits.sum()), bool(max_equal))`


I have modified it to  `(float(logits.sum())/logits.shape[-1], bool(max_equal))` and set the metric to acc instead of acc_norm for now. Since we only want the sum of log probs of continuations (normalized by the continuation length), we think that this is acceptable instead of adding another feature. Any help will be greatly appreciated!
Repository: EleutherAI/lm-evaluation-harness
Issue: Batching and generate_until special tokens
Root cause: Configuration Error

`models.huggingface.generate_until` has a step to cut off response suffixes that occur after end tokens in: https://github.com/EleutherAI/lm-evaluation-harness/blob/0bf9f4eafcde8e1bde61c188e8e30084a324bc6c/lm_eval/models/huggingface.py#L1370 

...which should help with cases of batching where the batch generation of:
https://github.com/EleutherAI/lm-evaluation-harness/blob/0bf9f4eafcde8e1bde61c188e8e30084a324bc6c/lm_eval/models/huggingface.py#L1355

...is leaving other sequences too long otherwise.

The problem is that using, say a special token `
` as an EOS token, we would want this decode to preserve that token:
https://github.com/EleutherAI/lm-evaluation-harness/blob/0bf9f4eafcde8e1bde61c188e8e30084a324bc6c/lm_eval/models/huggingface.py#L1368

...but it doesn't because `skip_special_tokens` defaults to True. Should it be set to false here? I don't have the bigger picture to decide if that would break other things, but we are encountering this issue of things not being truncated, so if you have any other ideas for how to remedy that well, please let me know!



Hi! do you mean if we are using a special token other than the EOS? Otherwise we always add the EOS token to the stop token list so generally the sequence should stop generating when it hits that. 

It does in the unbatched case, but as soon as we generate an entire batch, that is returned as a single tensor (second snippet), stopping only when all sequences of the batch have hit an EOS token, right? So only the longest sequence is thus cut at the right position, the shorter sequences need special cutting which is what I assumed the first linked snippet is for---only it doesn't work because the EOS token is stripped in the third snippet. Am I missing something here?

Generally each sequence _should_ be independent as we use [`transformers.StoppingCriteriaList`](https://github.com/EleutherAI/lm-evaluation-harness/blob/29971faaede3e0459eef6d1b4ec0f58460bed510/lm_eval/models/utils.py#L256). The second condition is mainly a fallback in case we run into tokenizer weirdness (for example `\n\n` vs. `\n` `\n`).

Are you encountering sequences not being truncated? A reproducible example would really be helpful to debug.
Repository: EleutherAI/lm-evaluation-harness
Issue: IFEval Error
Root cause: Validation Gap

On the newest main, I seem to have an error when running IFEval. I think it might be due to the logging changes made recently.

Error:
```
5, ip=10.202.0.27) [rank0]:   File "/usr/local/lib/python3.10/site-packages/yaml/constructor.py", line 143, in construct_mapping
(launch pid=595, ip=10.202.0.27) [rank0]:     value = self.construct_object(value_node, deep=deep)
(launch pid=595, ip=10.202.0.27) [rank0]:   File "/usr/local/lib/python3.10/site-packages/yaml/constructor.py", line 100, in construct_object
(launch pid=595, ip=10.202.0.27) [rank0]:     data = constructor(self, node)
(launch pid=595, ip=10.202.0.27) [rank0]:   File "/opt/lm_eval/lm-evaluation-harness/lm_eval/utils.py", line 442, in import_function
(launch pid=595, ip=10.202.0.27) [rank0]:     spec.loader.exec_module(module)
(launch pid=595, ip=10.202.0.27) [rank0]:   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
(launch pid=595, ip=10.202.0.27) [rank0]:   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
(launch pid=595, ip=10.202.0.27) [rank0]:   File "/opt/lm_eval/lm-evaluation-harness/lm_eval/tasks/ifeval/utils.py", line 5, in <module>
(launch pid=595, ip=10.202.0.27) [rank0]:     from lm_eval.utils import eval_logger
(launch pid=595, ip=10.202.0.27) [rank0]: ImportError: cannot import name 'eval_logger' from 'lm_eval.utils' (/opt/lm_eval/lm-evaluation-harness/lm_eval/utils.py)
```