zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: pass@k doesn't work for MBPP
Root cause: Validation Gap

Currently pass@1 is hard-coded in https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mbpp/utils.py#L15.

I modified the code for pass@k with k=[1,5,10] and it returned the metrics. But acc. would drop to 0 for all pass@k values. Looks like something is wrong with handling multiple generations in MBPP and MBPP+.

```
|    Tasks     |Version|  Filter   |n-shot|Metric |   |Value |   |Stderr|
|--------------|------:|-----------|-----:|-------|---|-----:|---|-----:|
|mbpp_plus     |      1|none       |     3|pass_at_1|↑  |0.4683|±  |0.0257|

|    Tasks     |Version|  Filter   |n-shot|Metric |   |Value |   |Stderr|
|--------------|------:|-----------|-----:|-------|---|-----:|---|-----:|
|mbpp_plus     |      1|create_test      |     3|pass_at_1|↑  |0|±  |0|
|mbpp_plus     |      1|create_test      |     3|pass_at_5|↑  |0|±  |0|
|mbpp_plus     |      1|create_test      |     3|pass_at_10|↑  |0|±  |0|
```

The scripts I'm using are added here: https://gist.github.com/sijial430/ef1f1d72d05bc5ca91a61527dadfa8c5
Did I miss anything? Thanks in advance.

The `build_predictions` filter should just be passthrough (we still require at least one filter otherwise `lm_eval` defaults to the `take_first` [filter](https://github.com/EleutherAI/lm-evaluation-harness/blob/3816796ebdcb1e7102e2964fc23d8e7a1082eba3/lm_eval/filters/selection.py#L23) which returns the first repeat):
```python
def build_predictions(resps: list[list[str]], docs: list[dict]) -> list[list[str]]:
    return resps
```

With `humaneval`, we prepend the `prompt` to the generations because the model is expected to complete a partial function. In contrast, `mbpp` requires the model to generate the entire function from scratch.

`code_eval`, in turn, concatenates a prediction string with the reference and [executes](https://github.com/huggingface/evaluate/blob/5aa3982a9a8c86e506860e381d428a64b0cce73b/metrics/code_eval/code_eval.py#L172) it to check if it runs correctly.

thank you, this works for me. can i open a PR to facilitate pass@k metrics for mbpp/mbpp+?
Repository: EleutherAI/lm-evaluation-harness
Issue: Question about requirements specific to a dataset
Root cause: Documentation Deficiency

Hello, 

We have a custom evaluation script for a [new dataset](https://arxiv.org/abs/2503.24378) that we released. I see that tasks allow custom metric, but our evaluation would require some python libraries too. What are your guidelines for tasks that has some dependencies? Can we submit the requirements file in the task folder?

Regards,
Harsha Kokel.

Hi! Generally, the practice for tasks with dependencies is to import them normally in the utils file in a try clause, and raise an import error if any dependencies aren't installed. For example, [see](https://github.com/EleutherAI/lm-evaluation-harness/blob/773dcd7f8fe95ae7ae73c687dec0a4dc1a6174b9/lm_eval/tasks/minerva_math/utils.py#L13-L24). This is just to ensure that any error will be raised when the file is imported (when the config is parsed), rather than arbitrarily later when a function requiring the dependency is used. 

The optional-dependencies can be added [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/773dcd7f8fe95ae7ae73c687dec0a4dc1a6174b9/pyproject.toml#L73). A requirements file in the task folder will would also be fine.



Thank you. The reference provided is very useful. Closing this issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: Do the current leaderboards use model-specific prompt templates (system prompt etc.)?

Folks,

I'm trying to compare some benchmark results I'm compiling with the leaderboard, and I'm wondering if I should switch to using lm-eval-harness. I've gone through the documentation and this issue: https://github.com/EleutherAI/lm-evaluation-harness/issues/1098, as well as the dataset someone compiled here: https://docs.google.com/spreadsheets/d/1b5s0LuhAQLbtzexxasLBA47gQ-R3HOuIKFRfIgJflf0/edit?gid=0#gid=0.

I'm still unclear whether the current small model leaderboards incorporate model-specific templates and standard system prompts, or if they don’t.

Does this vary by task? Could you please clarify?

Hi! If you're referring to the OpenLLM Leaderboard, they use this [fork](https://github.com/huggingface/lm-evaluation-harness/tree/main). For instruct models they also add `--apply_chat_template` and `--fewshot_as_multiturn` ([ref](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about#reproducibility)).



Super helpful thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: Startup Sglang Service Error
Root cause: Validation Gap

Hi，big guys！
Recently I followed the official documentation of sglang and quantized the Qwen2.5-0.5B model into three different classes (W4A16, W8A8-Int8, and W8A8-FP8), but with the same quantization code, the same runtime environment, and only the model files are different, only the model of W8A8-FP8 is able to be started through the Sglang service, the other However, with the same quantization code and the same runtime environment, only the model files are different, only the model of W8A8-FP8 can be started by Sglang service, while the others will report errors. Is there any solution?

Here is the code I quantified, the commands executed by the service startup, and the errors reported (W4A16 and W8A8-Int8)

![Image](https://github.com/user-attachments/assets/eee19e9c-2d7f-41e5-8fbc-cf0432deede0)

W8A8-Int8 Error：

![Image](https://github.com/user-attachments/assets/efd2f578-b31d-4c4f-af40-272da3d1f4af)

W4A16 Error：

![Image](https://github.com/user-attachments/assets/df535c16-0309-41c8-be8f-429f779f3ae6)
Repository: EleutherAI/lm-evaluation-harness
Issue: Indexing cont_tokens in _loglikelihood_tokens
Root cause: Documentation Deficiency

Hi,
I'm struggling to understand the indexing in `_loglikelihood_tokens()` in the `huggingface` model.
It's `logits = self._select_cont_toks(logits, contlen=contlen, inplen=ctx_len)`. I would think that you have to decrement `ctx_len` by 1 because the logits obtained from huggingface seem to be shaped like that [1]. I debugged through it and it doesn't seem that that's the case. But maybe I'm missing something? 

[1] eg https://github.com/huggingface/transformers/issues/14498#issuecomment-2229109826

Hi! We [shift](https://github.com/EleutherAI/lm-evaluation-harness/blob/19ba1b16fef9fa6354a3e4ef3574bb1d03552922/lm_eval/models/huggingface.py#L1134) our input tokens by removing the last one (with `[:-1]`) so we can extract exactly the continuation logits by starting at position `ctx_len - cont_len` without any additional off-by-one adjustment.

Oh, thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: `TypeError` from missing `yaml_path` in `lm_eval.utils.load_yaml_config` when task uses `include`
Root cause: Validation Gap

I am trying to run the public [NorEval tasks](https://github.com/ltgoslo/noreval) on the latest version of `lm_eval` (as opposed to v0.4.2, which it runs with without any modifications). However, when the `lm_eval.tasks.TaskManager._load_individual_task_or_group` method calls `lm_eval.utils.load_yaml_config` [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/__init__.py#L266), I end up with `yaml_path=None` and `yaml_dir=None`, which leads to a `TypeError` being raised in [`lm_eval.utils.load_yaml_config`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/utils.py#L487):
```raw
 File "[...]/.venv/lib/python3.11/site-packages/lm_eval/utils.py", line 461, in load_yaml_config
    yaml_dir = os.path.dirname(yaml_path)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 152, in dirname
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

If I understand the code correctly, then any task that enters [this](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/__init__.py#L264) if-block will lead to a `TypeError`? In that case, this might be a bug?

After playing around with the debugger, I managed to get around the bug by changing [this code](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/__init__.py#L265-L272)
```python
                config = {
                    **utils.load_yaml_config(
                        yaml_path=None,
                        yaml_config={"include": config.pop("include")},
                        mode="full",
                        yaml_dir=os.path.dirname(self.task_index[parent_name.group]["yaml_path"])
                    ),
                    **config,
                    "task": task,
                }
```
I also had to add the `"task": task,` line as without it, I got an error where there were duplicate tasks with the name `None`. Also, there might be a better way to fetch the `yaml_dir` (`parent_name` had the type `ConfigurableTask`), but I'm not too familiar with this codebase.

Hi! Thanks for the detailed report, and apologies for the tardiness. I'm currently working on cleaning up task loading in #3133, would you happen to recall the specific command/subset you used to recreate this issue?
Repository: EleutherAI/lm-evaluation-harness
Issue: LLaMA-4 evaluation
Root cause: External Dependency Breakage

I tried to evaluate Llama-4-Scout-17B-16E-Instruct from Huggingface. When I tried batch size =16, there is an error as follows:

```
File "~/anaconda3/envs/lm-eval-harness/lib/python3.12/site-packages/transformers/models/llama4/modeling_llama4.py", line 359, in forward
    attn_scales = attn_scales.view((*input_shape, 1, 1))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: shape '[16, 968, 1, 1]' is invalid for input of size 968
```
Then I tried to set batch-size = 1, there is no error reported. 

Is there any way to run batch-size > 1?

I'm having the same issue with BS > 1. Any ideas what's the reason for this error ?

Maybe try to upgrade the transformers module to the latest version?
Repository: EleutherAI/lm-evaluation-harness
Issue: low eval accuracy with gguf
Root cause: Interface Contract Mismatch

I followed https://github.com/EleutherAI/lm-evaluation-harness/issues/1254#issuecomment-1881161560 to start a llama.cpp server locally and use lm-eval do evaluation:

```
python -m llama_cpp.server --model Meta-Llama-3-8B-Instruct.Q4_K_M.gguf --n_gpu_layers -1
lm_eval --model gguf --model_args base_url=http://localhost:8000 --tasks hellaswag --device cuda:0 --batch_size 8 --limit 100
```
this gives:
```
|  Tasks  |Version|Filter|n-shot| Metric |   |Value|   |Stderr|
|---------|------:|------|-----:|--------|---|----:|---|-----:|
|hellaswag|      1|none  |     0|acc     |↑  | 0.21|±  |0.0409|
|         |       |none  |     0|acc_norm|↑  | 0.16|±  |0.0368|
```

while I'd expect around 0.6 for acc, wondering what is the issue here?

cc @haileyschoelkopf @StellaAthena @baberabb 

The same issue is observed: Phi-4 Q4-K-S result is very low, and all four tested models show very poor performance at q2_k_s. In contrast, generation quality appears acceptable, and the qdq model achieves reasonable accuracy.
Repository: EleutherAI/lm-evaluation-harness
Issue: Very high ROUGE-L score on LongBench
Root cause: Algorithmic Error

Hi,

There seems to be some hidden bugs in LongBench summarization tasks.

I am running `longbench_gov_report` etc to evaluate LlaMA3.1-8B-Instruct model from huggingface using the following commands:
```
lm-eval --model hf --model_args pretrained=${MODEL_PATH},attn_implementation=flash_attention_2,dtype=bfloat16,use_cache=True --gen_kwargs do_sample=False,num_beams=1 --tasks longbench_gov_report --device cuda:0 --batch_size 1 --num_fewshot 0 --seed 42 --apply_chat_template=True --output_path ${OUTPUT_PATH}
```

However, I always got a ROUGE-L score close to 1.0 (i.e. 0.985) using LlaMA3.1-8B-Instruct from huggingface. I think it should not be as high as this, as I find it to be around 0.23 in some NLP papers. 

Could you please check if there is anything wrong in lm-evaluation-harness' LongBench implementation?

Many thanks!



Hi! do you have a reference for this?

> Hi! do you have a reference for this?

Fig 6 in [this paper](https://arxiv.org/pdf/2412.16187)

There was a bug in the metric calculation (for all longbench tasks!). Fixed in #2895. I'll merge after testing it out a bit more thoroughly now.

Hi,

The fixed longbench works well. However, for `longbench_trec` and `longbench_lsht`, it always raises the following error:
```
  File "/lm-evaluation-harness/lm_eval/tasks/longbench/metrics.py", line 132, in classification_score
    all_classes = kwargs["all_classes"]
KeyError: 'all_classes'
```

Could you please check whether there are any bugs in the metrics related to these two tasks?

Many thanks.

Sorry. Pushed a fix!
Repository: EleutherAI/lm-evaluation-harness
Issue: Injecting custom variables/values into a task via YAML
Root cause: Documentation Deficiency

Hi and thanks for a great library. I was wondering if there is a way to inject custom key-value pairs into the `config` object using the YAML task configuration setup? I'm aware of the possibility of doing something like this with a custom `task.py` as outlined in the [new task guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md):

<blockquote>
You can also pass a custom argument to your class by accepting `config` in the custom class constructor.
Here's how to do it:

```yaml
task: 20_newsgroups
class: !function task.Unitxt
recipe: card=cards.20_newsgroups,template=templates.classification.multi_class.title
```

In this example, `recipe` is the custom argument for the `Unitxt` class.
</blockquote>

However it's not clear to me if the same is possible via the YAML configuration.

#### My particular use case

I'm trying to create a general version of the [FLORES](https://huggingface.co/datasets/facebook/flores) translation task and need to pass in `source_language` and `target_language`. My goal is to have each doc have a `source_sentence`, and a `target_sentence` after `process_docs` is called. So instead of hardcoding the language codes in the YAML prompt like this

```
[...]
doc_to_text: 'German sentence: {{sentence_deu_Latn}}

  English sentence:'
doc_to_target: '{{sentence_eng_Latn}}'
```

I could use a common `doc_to_text` and `doc_to_target` regardless of the language

```
[...]
doc_to_text: '{{source_language}} sentence: {{source_sentence}}

  English sentence:'
doc_to_target: '{{target_sentence}}
```

This would also help me compute modern machine translation metrics like COMET, which require source sentences, target sentences and the model hypotheses.

Any idea of how to make this happen using the YAML config would be greatly appreciated! Thanks in advance.



One way I thought of doing this would be to pass `source_language` and `target_language` in through `metadata` in the YAML:

```
[...]
metadata:
    source_language: deu_Latn
    target_language: eng_Latn
```

but it's unclear to me how I can access this in `process_docs` if I simply define it in `utils.py`? Seems like `process_docs` is only ever called with one argument (the document). 

Hi! The typical approach we follow is to create one common template (e.g., [1](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/gpqa/cot_n_shot/_gpqa_cot_n_shot_yaml), [2](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/arc/arc_easy.yaml)) and then use `include: <file>` to import the shared fields while overriding the subtask-specific ones. So you could, for example, create your template and then change the `dataset_path` in the individual configs.

We also allow overloading in a group config like [so](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/benchmarks/flan/flan_held_in.yaml), but then you can't run the individual tasks individually.

Does this help your use-case?
Repository: EleutherAI/lm-evaluation-harness
Issue: Additional data for evaluation
Root cause: Documentation Deficiency

Hello,

Is there a way to provide a custom-metric access to additional information (other than predictions and references)? We need this to help us identify the example index in our task, as the response evaluation is a little more complicated than just a comparison with a single answer. 

Hi! Have you considered using a [filter](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md#filters) pipeline? It allows for processing all the model responses along with their respective docs (optionally with statefulness). An [example](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/filters/extraction.py#L33) for regex processing, or if you just want quickly apply a [custom](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/filters/custom.py) [function](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/tasks/humaneval/humaneval.yaml#L25-L28). The outputs are then sent to the metric function for score calculation (in `process_results`)

You can also overload the `process_results` by the usual `!function` method. [Example](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/tasks/minerva_math/utils.py#L74). `process_results` is called on each (doc, result) pair individually.

Let me know if this helps with your use case! Happy to elaborate further.
Repository: EleutherAI/lm-evaluation-harness
Issue: Question- Anyone tried IFEval over Llama-4-Scout?
Root cause: Documentation Deficiency

Not sure if this is the right place to discuss this.

I'm trying to test Llama-4-Scout for IFEval and BFCL, in order to gauge its agentic capabilities (tool-calling, structured-output generation).

* `IFEval` (steerability) is not captured in the latest release notes [here](https://ai.meta.com/blog/llama-4-multimodal-intelligence/).
* As per [link](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about#task-evaluations-and-parameters)- The reportable IFEval is the average of Strict Accuracy at Instance and Prompt Levels (`inst_level_strict_acc`, and `prompt_level_strict_acc`) ~= 70.5.
* While, IFEval for Llama-3.1-70b and Llama-3.3-70b [here](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct#instruction-tuned-models) seem to be much better (87 and 92 resp).

```bash
lm_eval \
    --model vllm \
    --model_args pretrained="meta-llama/Llama-4-Scout-17B-16E-Instruct",download_dir="./hf/",max_model_len=32768,tensor_parallel_size=8,gpu_memory_utilization=0.85
```

* ![Llama-4-Scout-IFEval-Results](https://github.com/user-attachments/assets/9642758e-3376-4e4c-a87a-296b63d37a7e)


* Side note- BFCL (tool-calling) eval support for Llama-4 models is an open issue [TBD](https://github.com/ShishirPatil/gorilla/issues/978).

Am I evaluating incorrectly here, or is Meta's latest Llama-4-Scout truly not more steerable than its predecessors?

Hi! You should add `--apply_chat_template` to wrap the prompt in their chat template. Also according to the eval details they released for [3.1](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/eval_details.md#ifeval), the reported result is the average of all four scores, from my understanding.

Thanks that worked @baberabb!
Llama-4-Scout consumes 640GB of GPU memory (one entire H100 AWS node) over the `IFEval` task with `–apply_chat_template` as you suggested:

![Image](https://github.com/user-attachments/assets/c31f9207-593f-4b12-a9b0-2bc545345f42)

![Image](https://github.com/user-attachments/assets/1ec65610-bcd6-4c59-aea0-59945098f58c)

As per Meta's Llama-3.3 [release](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct#-1), the `3.3-70b` model scores `92%` on IFEval, `3.1-70b` scores `87%`, and this latest `Llama-4-Scout` scores `88.25%` (average of all 4 scores in my screenshot).
Repository: EleutherAI/lm-evaluation-harness
Issue: Improve the behavior of progress bars when using huggingface
Root cause: Architectural Constraint

## Issue Description
I've been using lm_evaluation_harness to evaluate my huggingface model. Specifically, the code looks like follows:
```python
lm_eval_model = models.huggingface.HFLM(pretrained=model)
results['results'][task] = evaluator.simple_evaluate(
      model=lm_eval_model,
      tasks=tasks,
      max_batch_size=64,
      limit=calib_size,
 )
```
which works fine when I'm evaluating on accuracy-based tasks, with a progress bar showing the progress of the evaluation. However, when I use this interface to evaluate perplexity, I get tons of progress bars with length 1, which is annoying and not informative:
<img width="646" alt="Image" src="https://github.com/user-attachments/assets/156c1aad-6ee2-4e59-b197-ceb3390cfd4b" />
Therefore, I wonder if it's possible to improved the behavior of progress bars when evaluating huggingface models?
## Possible Solution
After some investigation, it seems it's [this line](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/models/huggingface.py#L1092) throwing numerous pbars. My suggestion is to modify [this line](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/models/huggingface.py#L988) to set `disable_tqdm` to `True` and add tqdm to the outer forward loop. However, I have little knowledge of how tasks and requests are handled so this may not be the optimal way.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the News Article Generation evaluation
Root cause: Unimplemented Feature Gap

This is rather intensive to carry out due to the need for human input and may be skipped.

From the GPT-3 paper:
> To gauge the quality of news article generation from GPT-3 (which we believe is likely to be correlated with conditional
sample generation quality in general), we decided to measure human ability to distinguish GPT-3-generated articles
from real ones. Similar work has been carried out by Kreps et al. [KMB20] and Zellers et al. [ZHR+19]. Generative
language models are trained to match the distribution of content generated by humans, so the (in)ability of humans to
distinguish the two is a potentially important measure of quality. In order to see how well humans can detect model generated text, we arbitrarily selected 25 article titles and subtitles from the website newser.com (mean length: 215 words). We then generated completions of these titles and subtitles from four language models ranging in size from 125M to 175B (GPT-3) parameters (mean length: 200 words). For each model, we presented around 80 US-based participants with a quiz consisting of these real titles and subtitles followed by either the human written article or the article generated by the model. Participants were asked to select whether the article was “very likely written by a human”, “more likely written by a human”, “I don’t know”, “more likely written by a machine”, or “very likely written by a machine”.
> The articles we selected were not in the models’ training data and the model outputs were formatted and selected programmatically to prevent human cherry-picking. All models used the same context to condition outputs on and were pre-trained with the same context size and the same article titles and subtitles were used as prompts for each model. However, we also ran an experiment to control for participant effort and attention that followed the same format but involved intentionally bad model generated articles. This was done by generating articles from a “control model”: a 160M parameter model with no context and increased output randomness.

- [ ] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`