zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Why does ArcC pass in the answer to the model as part of the prompt?
Root cause: Documentation Deficiency

I added a hook to my model to print out the tokens passed in for the ArcC task. The input to the model includes the answer, but I am getting sensible ArcC zeroshot numbers (ie not 100% as passing in the answer would imply). For example, one of the inputs is 

```
['Question', ':', ' Devil', ' facial', ' tumor', ' disease', ' (', 'D', 'FT', 'D', ')', ' is', ' a', ' disease', ' that', ' is', ' dec', 'imating', ' the', ' population', ' of', ' Tas', 'man', 'ian', ' dev', 'ils', '.', ' The', ' disease', ' passes', ' from', ' one', ' animal', ' to', ' another', ' through', ' bites', ' and', ' is', ' caused', ' by', ' parasites', '.', ' The', ' parasites', ' cause', ' cancer', 'ous', ' tumors', ' that', ' spread', ' throughout', ' an', ' infected', ' animal', "'s", ' body', ' and', ' kill', ' it', '.', ' What', ' is', ' the', ' best', ' description', ' of', ' D', 'FT', 'D', '?\n', 'Answer', ':', ' a', ' non', '-in', 'fect', 'ious', ',', ' cell', '-cycle']
```

Why is the answer part of the prompt and why does this not cause the model to get 100%? Also, I'm not seeing the standard A B C D choices in the model input. Is Arc supposed to be like this?
Repository: EleutherAI/lm-evaluation-harness
Issue: auto-detect batchsize finding too large a batchsize to fit in VRAM at the end when used with multi-gpu
Root cause: Algorithmic Error

when running lm_eval across 8xH100 using accelerate and auto-detect batchsize I'm getting include/alloc.h:103 NCCL WARN Cuda failure 2 'out of memory' at the end after running all loglikelihood requests and it crashes due to OOM
seems to be at the end when it's trying to put everything together again...
if I manually set a smaller batchsize it works fine

command was like:
`accelerate launch -m lm_eval --model hf --model_args pretrained=/local_model,trust_remote_code=True --tasks lambada_openai --batch_size auto`
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluate local tasks failed when using lm-eval --tasks <local-folder>
Root cause: Validation Gap

When I evaluate local tasks using  lm-eval --tasks <local-folder>, then it will failed

![image](https://github.com/user-attachments/assets/82b50029-a6a3-47bf-b72f-8821c3b1c886)
Repository: EleutherAI/lm-evaluation-harness
Issue: task load return error
Root cause: External Dependency Breakage

Hi, I got an error when I ran the lm-eval command:
`Traceback (most recent call last):
  File "/vol3/ctr/.conda/envs/hzx1/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/evaluator.py", line 235, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 618, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 414, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 398, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 398, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 314, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 280, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/api/task.py", line 818, in __init__
    self.download(self.config.dataset_kwargs)
  File "/vol3/ctr/lm-evaluation-harness/lm_eval/api/task.py", line 925, in download
    self.dataset = datasets.load_dataset(
  File "/vol3/ctr/.conda/envs/hzx1/lib/python3.10/site-packages/datasets/load.py", line 2132, in load_dataset
    builder_instance = load_dataset_builder(
  File "/vol3/ctr/.conda/envs/hzx1/lib/python3.10/site-packages/datasets/load.py", line 1890, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable`

Below is my command:
`#!/bin/bash

lm_eval --model hf \
    --model_args pretrained=/vol3/ctr/hzx/OpenLongCoT-Base-Gemma2-2B \
    --tasks mmlu \
    --device cuda:4 \
    --batch_size 4`

the same error

Do you have any suggestions for solving it?

same


same

same

same problem...

same

same

same

same

same

same

same
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue with Perplexity Score Using `max_length > 1024` in lm-evaluation-harness
Root cause: Algorithmic Error

I am using a custom model in the lm-evaluation-harness script and have configured the tokenizer to use `GPT-2` with a `max_length` of 2048. However, I’ve noticed that when I increase the max_length beyond 1024, the perplexity score unexpectedly decreases. This reduction in perplexity doesn’t align with my expectations, as the perplexity should not inherently decrease with a larger sequence length.

Could someone kindly investigate and help understand why this is happening?

@fnusid Same question here! 

I found that the PPL@WikiText would be change if the max_length is different, is this as expected?

- max_length=4096
```shell
> lm_eval --model hf \
    --model_args pretrained=meta-llama/Meta-Llama-3-8B-Instruct,max_length=4096 \
    --tasks wikitext \
    --batch_size 1
......
| Tasks  |Version|Filter|n-shot|    Metric     |   | Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|------:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  | 0.6269|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  | 1.5442|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |10.2122|±  |   N/A|
```

- max_length=1024
```shell
> lm_eval --model hf \
    --model_args pretrained=meta-llama/Meta-Llama-3-8B-Instruct,max_length=1024 \
    --tasks wikitext \
    --batch_size 1
......
| Tasks  |Version|Filter|n-shot|    Metric     |   | Value |   |Stderr|
|--------|------:|------|-----:|---------------|---|------:|---|------|
|wikitext|      2|none  |     0|bits_per_byte  |↓  | 0.6695|±  |   N/A|
|        |       |none  |     0|byte_perplexity|↓  | 1.5905|±  |   N/A|
|        |       |none  |     0|word_perplexity|↓  |11.9581|±  |   N/A|
```
Repository: EleutherAI/lm-evaluation-harness
Issue: some issues for MMLU benchmark when loading local dataset
Root cause: Configuration Error

Hi 
I have some issues for MMLU benchmark when loading [localdataset](https://huggingface.co/datasets/hails/mmlu_no_train/tree/main). Could you help to take a look?
1: python3.12 + lm-evaluation-harness-0.4.5
2.  run command
```
lm_eval --model hf --model_args pretrained=${local_model_path} --tasks mmlu_human_aging ........
```
3: yaml: I have changed the dataset_path and dataset_kwargs to load local dataset
```
dataset_path: {PRIVATE_LOCAL_PATH}/hails/mmlu_no_train/ # a copy of `cais/mmlu` with no auxiliary_train split
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_choice: ["A", "B", "C", "D"]
doc_to_target: answer
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0
dataset_kwargs:
  data_dir: all
```
4: error with dataset name not found
```
ValueError: BuilderConfig 'human_aging' not found. Available: ['default']
```
5. I also tried to modify some code as below to bypass this check, but will encounter another error
```
lm_eval/api/task.py
@@ -922,11 +922,14 @@ class ConfigurableTask(Task):
                     )

     def download(self, dataset_kwargs: Optional[Dict[str, Any]] = None) -> None:
         self.dataset = datasets.load_dataset(
             path=self.DATASET_PATH,
-            name=self.DATASET_NAME,
+            # name=self.DATASET_NAME,
             **dataset_kwargs if dataset_kwargs is not None else {},
         )
```

another error:
````
KeyError: 'dev'
```


This issue was resolved by using the online datasets, let's close this issue now, thanks.
Repository: EleutherAI/lm-evaluation-harness
Issue: Custom data folder caching is relative instead of absolute
Root cause: Architectural Constraint

Most os the datasets rely on HF, and in particular its cache handling. However there are a few tasks when the dataset download/caching is done manually. https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/lambada.py#L11-L23

This is usually not a problem if one continually uses the `main` script at the root of the repo to run evaluation. But as soon as we run it from another path, then the caching doesn't work anymore and generates a new data folder. For example, we tried using this repo and create our custom bindings to run the harness on a Megatron-Deepspeed model. https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/212#discussion_r766754427 . The reason is that the caching mechanism right now uses relative paths instead of an absolute path. 

I think we should replace `data/*` with an absolute path, in order to have a single cache. HF datasets have a default to `~/.cache`, and it exposes a overridable flag to point to a specific path `HF_DATASETS_CACHE` https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/212#discussion_r766754427 . Is that something you'd be willing to add to the repo? @leogao2 

I second that request as I have just run into weird failures until Thomas explained that it's a relative path issue! Thank you!

That sounds like a good idea; I'll prioritize getting it done.

Sorry for the late response. All caching is now handled by HuggingFace `datasets`. You can instantiate each `Task`  with a specific `cache_dir` or override the `HF_DATASETS_CACHE` environment variable to set the cache location across all `Task`s of a given process.

Thanks for raising this issue.

No worries! Thank you for taking care of this. It looks like come datasets could actually be pushed to `datasets` directly? WDYT?

Oh yeah! That was definitely in the plans - I just haven't set aside the time to do it 😅. The main candidates to upstream over there are:

- `GSM8K`
- `MuTual`
- `SAT_Analogies`
- `TruthfulQA`

I'll create separate PRs for each by the end of the week.  
@thomasw21 is it preferred to add datasets via the Hub following https://huggingface.co/docs/datasets/share or as PRs to the repo? Thanks!

Maybe @lhoestq is the best one to answer such questions. 

Hi ! As you want ^^ If you put it yourself on the Hub it will be under your username. And if you open a PR then it would not be under any namespace (same as `squad` for example), and the `datasets` team will have to review your PR. It also requires a bit more work (see [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) for all the details)

I'd be happy to help with the reviews and to answer questions if you open a PR ;)

I suppose I'll open some PRs. See ya'll over there :)
Repository: EleutherAI/lm-evaluation-harness
Issue: [QUESTION] Does lm-eval support direct evaluation from json?
Root cause: Unimplemented Feature Gap

Without loading and serving the model?

No, the purpose of this library is to run the model and grade its outputs. If you have model outputs saved to a JSON, you should be able to hack it with a fake "model" class that loads the outputs from disk though.
Repository: EleutherAI/lm-evaluation-harness
Issue: IFEval: the package pkg_resources is deprecated and removed in Python 3.12
Root cause: External Dependency Breakage

A warning is thrown when using IFEval. 

```
DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
```

The relevant code is as follows:

https://github.com/EleutherAI/lm-evaluation-harness/blob/0845b588303f1f59af98dd1c5bdbd78a9e75a1e2/lm_eval/tasks/ifeval/instructions_util.py#L21-L47

This warning was added in [setuptools version 67.3.0](https://setuptools.pypa.io/en/latest/history.html#v67-3-0). It is because the package `pkg_resources` is removed in [Python 3.12](https://docs.python.org/3/whatsnew/3.12.html#ensurepip). 

According to the [migration guide](https://importlib-metadata.readthedocs.io/en/stable/migration.html) and [metadata's documentation](https://docs.python.org/3.12/library/importlib.metadata.html), I think we can use [the version function of importlib.metadata](https://docs.python.org/3.12/library/importlib.metadata.html#importlib.metadata.version) to get the version information of the nltk package.

The reference code is as follows:

```python
import immutabledict
import nltk
from packaging import version


# Downloading 'punkt' with nltk<3.9 has a remote code vuln.
# see  https://github.com/EleutherAI/lm-evaluation-harness/issues/2210
# and https://github.com/nltk/nltk/issues/3266
# for more information.
NLTK_MIN_VERSION = "3.9.1"


def download_nltk_resources():
    """Download 'punkt' if not already installed"""
    try:
        import importlib.metadata
        nltk_version = importlib.metadata.version("nltk")
    except ImportError:
        import pkg_resources
        nltk_version = pkg_resources.get_distribution("nltk").version
    assert (
        version.parse(nltk_version) >= version.parse(NLTK_MIN_VERSION)
    ), f"`nltk` version {nltk_version} is not >= {NLTK_MIN_VERSION}. Please update `nltk` before proceeding--older versions are vulnerable to a remote code execution vulnerability."

    try:
        nltk.data.find("tokenizers/punkt_tab")
    except LookupError:
        nltk.download("punkt_tab")


download_nltk_resources()
```
Repository: EleutherAI/lm-evaluation-harness
Issue: MMLU-Pro Generation -> Generative
Root cause: Configuration Error

In the Generative version of MMLU Pro the task's group name is `mmlu_generation` whereas [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/4155ec7fb9695d73d8301bc0323972fcb0f0d31e/lm_eval/tasks/mmlu/generative/_mmlu.yaml#L1) the group name is `mmlu_generative` 

Thank you for pointing this out!
Repository: EleutherAI/lm-evaluation-harness
Issue: [QUESTION] hello, I has a problem…

device: Chromebook
version of ChromeOS: 130.0.6723.101, 64 bit

hello, I has a problem… 
I press this keys: Ctrl+Alt+Refresh to write on the command line for install and start a Linux on my Chromebook, that me demand for the localhost login, but I don't know, what the localhost login… so… what the localhost login? yes, I know, this question is so stupid… thanks for the answer!
Repository: EleutherAI/lm-evaluation-harness
Issue: Why Different Versions Make a Big Difference  in HellaSwag zero-shot
Root cause: Configuration Error

Could you please explain why there is such a significant difference in the zero-shot experimental results reported on hellaswag between the 0.3.0 version and the new 0.4.4 version for the same model? For instance, with LLaMA-2-13B, the accuracy (acc) was 60.06 in the former, and it's 79.39 in the latter.

hmm. Not sure. The implementation seems the same ([old](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hellaswag.py) vs [new](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/hellaswag/utils.py)). A good way to check would be to compare with [Open LLM leaderboard](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard), where they used 0.3.0 (10-shot for hellaswag)
Repository: EleutherAI/lm-evaluation-harness
Issue: Cannot reproduce LLaMA 3 8B on hendrycks_math
Root cause: Configuration Error

Hi! Im trying to reproduce the results of  llama3-8b on MATH (`hendrycks_math` in lm_eval, right?). 
The lm_eval version is `version 0.4.5`, and 4-shot for hendrycks_math.

Then I get the results below
```
{
  "hendrycks_math": {
    "exact_match,none": 0.1342,
    "exact_match_stderr,none": 0.004774784542910501,
    "alias": "hendrycks_math"
  },
  "hendrycks_math_algebra": {
    "alias": " - hendrycks_math_algebra",
    "exact_match,none": 0.13900589721988205,
    "exact_match_stderr,none": 0.01004556473230226
  },
  "hendrycks_math_counting_and_prob": {
    "alias": " - hendrycks_math_counting_and_prob",
    "exact_match,none": 0.12869198312236288,
    "exact_match_stderr,none": 0.015396819954085057
  },
  "hendrycks_math_geometry": {
    "alias": " - hendrycks_math_geometry",
    "exact_match,none": 0.13987473903966596,
    "exact_match_stderr,none": 0.015864871092013833
  },
  "hendrycks_math_intermediate_algebra": {
    "alias": " - hendrycks_math_intermediate_algebra",
    "exact_match,none": 0.07198228128460686,
    "exact_match_stderr,none": 0.008605729055597207
  },
  "hendrycks_math_num_theory": {
    "alias": " - hendrycks_math_num_theory",
    "exact_match,none": 0.1,
    "exact_match_stderr,none": 0.012921914767618418
  },
  "hendrycks_math_prealgebra": {
    "alias": " - hendrycks_math_prealgebra",
    "exact_match,none": 0.2261768082663605,
    "exact_match_stderr,none": 0.014183560974528395
  },
  "hendrycks_math_precalc": {
    "alias": " - hendrycks_math_precalc",
    "exact_match,none": 0.11355311355311355,
    "exact_match_stderr,none": 0.013590255815449908
  }
}
```
`exact_match` on hendrycks_math is 13.42, which is much different from the results 20.3 reported in llama3 tech report.
![image](https://github.com/user-attachments/assets/844cb42b-c257-4bc9-89b0-f337efcdfb99)



They [say](https://github.com/meta-llama/llama3/blob/main/eval_details.md) they use the same implementation as https://arxiv.org/pdf/2206.14858, which is `minerva_math` here, but that only gives `0.159` acc for `meta-llama/Meta-Llama-3-8B`. Given the limited information provided, it's tough to verify the exact evaluation setup and formatting used.

Also they use 11 shot instead of 4, that could help a bit...
Repository: EleutherAI/lm-evaluation-harness
Issue: Review all GLUE + SuperGLUE tasks.
Root cause: Unimplemented Feature Gap

Time to review all the tasks, see if there are any formats we want to change based on what we've learned in the last year from other LM/projects. Also using this to track related issues.

https://github.com/EleutherAI/lm-evaluation-harness/pull/243

Hey @zphang do you maybe know of particular GLUE issues that might be causing https://github.com/EleutherAI/lm-evaluation-harness/issues/320?
Repository: EleutherAI/lm-evaluation-harness
Issue: wandb logger hits printer error due to `get_printer` change in wandb API
Root cause: External Dependency Breakage

Following [PR 8658 of wandb](https://github.com/wandb/wandb/pull/8658) calling the wandb logger crashes as `get_printer(...)` has been replaced by `new_printer()`, giving the following error:

```
[...]
  File "/path/to/venv/lib/python3.10/site-packages/lm_eval/loggers/wandb_logger.py", line 18, in get_wandb_printer
    from wandb.sdk.lib.printer import get_printer
ImportError: cannot import name 'get_printer' from 'wandb.sdk.lib.printer'
```

To reproduce, just do something like this:
```
lm_eval \
    --model hf \
    --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype="float" \
    --tasks arc_easy \
    --device cuda:0 \
    --wandb_args project=lm-eval-logger-repro
```

I think it just needs lines 18 and 21 in `lm_eval/loggers/wandb_logger.py` updating to use `new_printer()`. A quick test suggests this seems to work, however I've not extensively tested it.

Cheers,
Alex

Thanks for reporting in!
Repository: EleutherAI/lm-evaluation-harness
Issue: No such file or directory: 'data/unscramble/mid_word_1_anagrams.jsonl.gz'
Root cause: Validation Gap

Using @DanielHesslow `tasks/eval_harness/download.py` [eval harness integration](https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/212) into Megatron-Deepspeed, it works for some tasks and fails in others, e.g. for all:

```
$ python ./tasks/eval_harness/download.py
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107k/107k [00:00<00:00, 946kbyte/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107k/107k [00:00<00:00, 1.19Mbyte/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107k/107k [00:00<00:00, 2.48Mbyte/s]
Traceback (most recent call last):
  File "./tasks/eval_harness/download.py", line 20, in <module>
    main()
  File "./tasks/eval_harness/download.py", line 17, in main
    tasks.get_task_dict(task_list)
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 232, in get_task_dict
    return {
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 233, in <dictcomp>
    task_name: get_task(task_name)()
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/lm_eval/tasks/unscramble.py", line 23, in __init__
    super().__init__()
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/lm_eval/base.py", line 123, in __init__
    self.download()
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/lm_eval/tasks/unscramble.py", line 33, in download
    extract_gzip(gz=rawfile, to=file)
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/site-packages/lm_eval/tasks/unscramble.py", line 11, in extract_gzip
    with gzip.open(gz, 'rb') as fin:
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/home/stas/anaconda3/envs/py38-pt110/lib/python3.8/gzip.py", line 173, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'data/unscramble/mid_word_1_anagrams.jsonl.gz'
```

It should have failed earlier, since I don't have any files under unscramble

```
$ ls -l data/unscramble/
total 0
```
so some download failed or was missed.

these work just fine:
```
python ./tasks/eval_harness/download.py --task_list hellaswag,mrpc,piqa
```

It was suggested I create an issue here.

Tried with both `lm-eval==0.0.1` and `master` with the same results.

Additionally it proved that we actually don't care for all tasks and the ones we need work just fine. So absolutely no urgency on this one.

Thanks.

#250 fixed this

Thank you!