zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Global-MMLU
Root cause: Unimplemented Feature Gap

We just released [Global-MMLU ](https://huggingface.co/datasets/CohereForAI/Global-MMLU) and would like to integrate it with eval-harness. I'd be happy to submit a PR to support this.  What would be the best way to structure this ? 

I was planning on setting it up similar to [okapi-mmlu](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/okapi/mmlu_multilingual) task. Let me know if that'd work ?

Hi! That would be great! And congrats on the release - love the cultural bias analysis. okapi does sound like a good starting point!
Repository: EleutherAI/lm-evaluation-harness
Issue: Inquiry about the feature to continue evaluation after abnormal termination
Root cause: Validation Gap

Thank you for your kind reply.

In the code, it seems that when use_cache is enabled, the task results for the same model are saved after the evaluation is completed successfully.

The "save_to_cache" function is called after the evaluation is completed in the code, but if it ends abnormally, there seems to be no way to continue the evaluation from where it left off.

For example, if you are evaluating 10 tasks and it ends abnormally after 80% progress, you have to start the evaluation from the beginning again.

I would greatly appreciate it if you could guide me once again if there is such a feature.

==========================================================


              Hi! we do actually have this implemented using  `--use_cache <DIR>` to cache the model results while evaluating and skip previously evaluated samples on resumption. Caching is rank-dependent though, so restart with the same GPU count if interrupted! Also have `--cache_requests` so the dataset preprocessing steps can be saved and evaluation can resume quicker.

I should update the README to make these more prominent!

_Originally posted by @baberabb in https://github.com/EleutherAI/lm-evaluation-harness/issues/2533#issuecomment-2516689607_
            

Hi! for example if you run with:
`lm_eval --model hf --model_args pretrained=EleutherAI/pythia-14m --tasks gsm8k,arc_easy --use_cache test_dir` ,
the model outputs will get cached as they are generated. If the evaluation ends prematurely or you just want to calculate the metrics, the next time you run the command with `--use_cache test_dir`, it will check which samples have already been completed and skip them.

> Hi! for example if you run with: `lm_eval --model hf --model_args pretrained=EleutherAI/pythia-14m --tasks gsm8k,arc_easy --use_cache test_dir` , the model outputs will get cached as they are generated. If the evaluation ends prematurely or you just want to calculate the metrics, the next time you run the command with `--use_cache test_dir`, it will check which samples have already been completed and skip them.

Thank you for the kind guidance. 😊

It seems there is a bug in the save_to_cache function located in caching/cache.py.
If file_name contains a space (' ') or a slash ('/'), the .pickle file is not created.
In my case, the issue occurred because the cache_key generation process included special characters in tokenizer_name. 

Thanks :)

You're welcome. Will take a look, but note `save_to_cache` is used to cache pre-processing steps (mostly data), so the program can start quicker next time around. Results caching is handled by [CachingLM](https://github.com/EleutherAI/lm-evaluation-harness/blob/bcb4cbf464668450f1969a4f1bef28f8638d1648/lm_eval/api/model.py#L228):
Repository: EleutherAI/lm-evaluation-harness
Issue: BUG: TypeError: create_from_arg_string() takes 2 positional arguments but 3 were given
Root cause: Interface Contract Mismatch

I am getting this error when trying the dummy model on master branch (commit ed6931e7e6c422bd17029c13a61cbbeef11e453c)
```
(venv) julius@big-bertha:~/Documents/projects/lm-evaluation-harness$ python main.py --model dummy --tasks hellaswag                    WARNING: headqa is deprecated. Please use headqa_es or headqa_en instead. See https://github.com/EleutherAI/lm-evaluation-harness/pull/240 for more info.
Traceback (most recent call last):
  File "main.py", line 71, in <module>
    main()
  File "main.py", line 43, in main
    results = evaluator.simple_evaluate(
  File "/home/julius/Documents/projects/lm-evaluation-harness/lm_eval/utils.py", line 156, in _wrapper
    return fn(*args, **kwargs)
  File "/home/julius/Documents/projects/lm-evaluation-harness/lm_eval/evaluator.py", line 50, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(model_args, {
TypeError: create_from_arg_string() takes 2 positional arguments but 3 were given
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Inconsistent responses for the same case with different limit parameters
Root cause: Algorithmic Error

Hello,  I meet this problem when using the same input parameters but varying the limit value, the responses differ. This behavior is unexpected, as changing the limit should only affect the number of results returned, not the response content for identical cases.

This is my code:
``lm_eval --model hf --limit 0.01  --model_args pretrained=../../../llms/Meta-Llama-3.1-8B-Instruct   --tasks gsm8k_cot   --device cuda:4 --log_samples    --batch_size 1  --output_path ./results/gsm8k_cot/hf_test_0.01_v2.json --gen_kwargs max_gen_toks=512  do_sample=true temperature=0.5 --seed 42,42,42,42``

However, when the limit is changed to 0.015, the response is different in the same case.  And nearly all responses in the two settings differ.
![image](https://github.com/user-attachments/assets/012c2f1f-8894-464a-940b-d5c3188fa519)

How can I fix this bug and get the same response? Thanks.
Repository: EleutherAI/lm-evaluation-harness
Issue: Trust_remote_code argument nowhere to be passed
Root cause: Architectural Constraint

Hi, I am following the ```interface.md``` instructions to use eval harness as an external library. I am currently using simple_evaluate:
```python
task_manager = lm_eval.tasks.TaskManager()

results = lm_eval.simple_evaluate(
    model=wrapper,
    tasks=[task],
    num_fewshot=0,
    task_manager=task_manager, 
    device=device,
    cache_requests=True
)
```

and I am trying to evaluate on the task ```boolq```. There's an error that says
```
ValueError: The repository for super_glue contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/super_glue.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
```

But there's no way to set this parameter externally anywhere. Any tips?

Quick workaround as mentioned in [#1467](https://github.com/EleutherAI/lm-evaluation-harness/pull/1467#issuecomment-1964282427)

Thank you! This seems to have fixed the issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: Hendrycks Math extraction rule seems too strict
Root cause: Validation Gap

Hi thanks for the library! It seems that the way how math answer is extracted, i.e. https://github.com/EleutherAI/lm-evaluation-harness/blob/bcb4cbf464668450f1969a4f1bef28f8638d1648/lm_eval/tasks/hendrycks_math/utils.py#L20-L24, may be too strict.

For example, the following answer: `... some reasoning logic ... Thus the answer is \[ \boxed{42} \]` is not extracted, because it is not a `$`.

Hi! This is based on the original code, but we can add another metric to show alongside by adding a [flexible-extract](https://github.com/EleutherAI/lm-evaluation-harness/blob/bcb4cbf464668450f1969a4f1bef28f8638d1648/lm_eval/tasks/gsm8k/gsm8k.yaml#L38-L43) filter, as `gsm8k` does it. PR welcome!

Hi @baberabb 
I am interested in solving this issue!
Can you please assign it to me?
Repository: EleutherAI/lm-evaluation-harness
Issue: reproduce llama 3 evals
Root cause: Unimplemented Feature Gap

llama 3.{1,2,3} have released most of their eval details here in [HF](https://huggingface.co/collections/meta-llama/llama-32-evals-66f44b3d2df1c7b136d821f0) and also some in this [repo](https://github.com/meta-llama/llama-recipes/tree/b5f64c0b69d7ff85ec186d964c6c557d55025969/tools/benchmarks/llm_eval_harness). Would be great if we can upstream them here. I'm adding some in #2556

added `llama_arc_challenge` (base) and `mgsm_chat` #2556, and (`arc_challenge_chat`, `mmlu_llama`) in #2615

Hi, @baberabb do you plan to add the benchmark that can reproduce the gpqa result as well?

added as `llama_gpqa`. Still have to test it out! Note, looks like they use the main subset and not diamond

Also @baberabb do you know why the max_gen_toks is set to 10 for the mmlu_llama here https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/llama3/instruct/mmlu/_continuation_template_yaml? It only needs to generate max 2 tokens right? (Which will also make this test run faster)

Also @baberabb do you know if the mmlu_llama with `--fewshot_as_multiturn --apply_chat_template` also works for non-llama models? I try it on mixtral and it seems that the result is lower than the mmlu benchmark

> Also [@baberabb](https://github.com/baberabb) do you know if the mmlu_llama with `--fewshot_as_multiturn --apply_chat_template` also works for non-llama models? I try it on mixtral and it seems that the result is lower than the mmlu benchmark

The [prompt](https://github.com/EleutherAI/lm-evaluation-harness/blob/53504a969dd66e7191c41747601b684f2d122adf/lm_eval/tasks/llama3/instruct/mmlu/_continuation_template_yaml#L7) should work well with most instruct models, I think. Have you looked at the outputs (saved with `--log_samples`)? Right now it assumes the model will generate the answer letter right away, which might not be true for all models and we could make the answer extraction more flexible.

And the generation tokens are 10 as thats whats used in llama evals (they do not provide the stop tokens though, but I added `.` here) 

Are there plans to add support for how Meta reports Llama 3.x scores fore `gpqa_diamond_cot_zeroshot`? I recently ran this and got 
```
|          Tasks          |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-------------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gpqa_diamond_cot_zeroshot|      1|flexible-extract|     0|exact_match|↑  |0.1515|±  |0.0255|
|                         |       |strict-match    |     0|exact_match|↑  |0.0000|±  |0.0000|
```
which is well below Meta's reported scores.

I've noticed this is a general pattern when using generative metrics when calculating evaluation scores for Llama.
Repository: EleutherAI/lm-evaluation-harness
Issue: partially initialized module
Root cause: Validation Gap

ImportError: cannot import name 'cli_evaluate' from partially initialized module 'lm_eval.__main__' (most likely due to a circular import) (/mnt/data-disk0/project/lm-evaluation-harness/lm_eval/__main__.py)


Why is there a circular import?
Repository: EleutherAI/lm-evaluation-harness
Issue: wikitext task cannot be evaluated [link to dataset is down]
Root cause: External Dependency Breakage

It looks like wikitext task takes the data from https://state.smerity.com

However in recent times the website has been down and hence the task can no longer be evaluated.

Steps to recreate

```
# model = ... (e.g. load from llama)
lm = HFLM(pretrained=model, tokenizer=tokenizer, dtype=torch.bfloat16, max_length=tokenizer.model_max_length,
                batch_size=64, trust_remote_code=True)
tasks = ["wikitext"]
  print("evaluating on tasks: ", tasks)
  results = lm_eval.simple_evaluate(
      model=lm,
      tasks=tasks,
      task_manager=lm_eval.tasks.TaskManager(),)
```

gives the error:
```
FileNotFoundError: https://wikitext.smerity.com/wikitext-2-raw-v1.zip
```

Perhaps we need to change the URL to somewhere more stable? to somewhere like: https://huggingface.co/datasets/Salesforce/wikitext?


Reference commit previously which was changed: https://huggingface.co/datasets/EleutherAI/wikitext_document_level/commit/3ad9e13102af0ea3a47d6b0907694d908f01be05

Converted the dataset to parquet. Should work now, but feel free to open this issue if you run into any problems
Repository: EleutherAI/lm-evaluation-harness
Issue: When can support MATH/HumanEval datasets eval
Root cause: Unimplemented Feature Gap


Hi we already have two MATH variants: the original `hendrycks_math` and `minerva_math`. for humaneval, there is #1992 which I'll try merging this week!

@baberabb Any updates on merging humaneval?

humaneval added in #1992!
Repository: EleutherAI/lm-evaluation-harness
Issue: CaseHOLD Task Implementation
Root cause: Validation Gap

Hello,

I have been working on implementing the CaseHOLD task within the lm-harness-evaluation framework, as introduced in [Pull Request #2570](https://github.com/EleutherAI/lm-evaluation-harness/pull/2570). While I managed to create the necessary files (casehold.yaml and utils.py) to preprocess and evaluate the dataset, I encountered significant challenges during evaluation. Specifically, all models I tested consistently performed at or below random chance, which indicates an issue I have been unable to resolve.

Below are the results of an example run using the `lm_eval` with the `Llama-3.2-1B-Instruct`:
```
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.2-1B-Instruct --task casehold --device cuda:1 --batch_size 1 --apply_chat_template

| Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|--------|------:|------|-----:|--------|---|-----:|---|------|
|casehold|      1|none  |     0|acc     |↑  |0.1472|±  |0.0049|
|        |       |none  |     0|acc_norm|↑  |0.2633|±  |0.0060|
|        |       |none  |     0|f1      |↑  |0.1472|±  |   N/A|
```

I’ve created the pull request to share my implementation and to seek help with identifying and fixing the root cause of these subpar results. If you have experience working with the caseHOLD dataset or lm-harness-evaluation, your insights would be greatly appreciated.

For context, here are the resources relevant to the task:

[The CaseHOLD dataset](https://huggingface.co/datasets/casehold/casehold)
[Paper on CaseHOLD](https://arxiv.org/abs/2104.08671)
Any guidance or feedback would be greatly appreciated. Thank you in advance for your support!

Best regards,
David
Repository: EleutherAI/lm-evaluation-harness
Issue: v0.4.6 not published to pypi
Root cause: Configuration Error

The latest version of lm-eval failed to publish to pypi.

https://github.com/EleutherAI/lm-evaluation-harness/actions/runs/12010552042/job/33477658734

Thanks for reporting this! Forgot to increment the version :S. Have pushed a new version!
Repository: EleutherAI/lm-evaluation-harness
Issue: train, val-test split: How to select the right split on demand
Root cause: Documentation Deficiency

Hello, i would like to be able to select the split i wish to evaluate my model on. but i am not seen such an option .
how do i do that for dataset such as mmnlu, mmlu-pro, hellaswag with llaba3.2-1B-instruct

Hi! We check `test_split`> `validation_split` > `training_split` to determine which evaluation split to use. You can change them in the task configs:
https://github.com/EleutherAI/lm-evaluation-harness/blob/8558b8d4998991220cb5483246b2ae4878c92a1f/lm_eval/tasks/gsm8k/gsm8k.yaml#L7-L9
Repository: EleutherAI/lm-evaluation-harness
Issue: Qwen2.5-14B-Instruct eval shape wrong.
Root cause: Validation Gap

my command is 
`CUDA_VISIBLE_DEVICES=2,3 torchrun --nproc-per-node=2 --master_port 8866 --no-python lm_eval --model hf --model_args pretrained=Qwen/Qwen2.5-14B-Instruct, --tasks hellaswag  --batch_size auto:4`

but i miss the wrong:
ValueError: Trying to set a tensor of shape torch.Size([152064, 5120]) in "weight" (which has shape torch.Size([151646, 5120])), this look incorrect.

additionally, it will be right if i replace the model with Qwen2.5-3B-Instruct.

Stupid cry. i changed my config.json in Qwen2.5-14B-Instruct, when i rewrite the vocab_size, things be ok.
Repository: EleutherAI/lm-evaluation-harness
Issue: Chat template not being used despite passing `--apply_chat_template` option
Root cause: Configuration Error

I am having an issue when evaluating several instructed models using the LM Evaluation Harness and the `--apply_chat_template` option. I am passing this option and I ensured that the models have chat templates available in their tokenizer configurations, but when I check the results JSON file, the `chat_template` field is null, which, as I understand, means that the chat template was not used in the evaluation to obtain such results. 

For example, I am evaluating the [utter-project/EuroLLM-1.7B-Instruct](https://huggingface.co/utter-project/EuroLLM-1.7B-Instruct) model, which has a chat template defined in its [`tokenizer_config.json`](https://huggingface.co/utter-project/EuroLLM-1.7B-Instruct/blob/a25c7fa65fc2a644e6270b8940dbe295b51da681/tokenizer_config.json#L2096). I'm using a local clone of this model and I've confirmed the `chat_template` field is there:

```
$ cat tokenizer_config.json | grep chat_template
  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'
' + message['role'] + '\n' + message['content'] + '
' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '
assistant\n' }}{% endif %}"
```

I ran evaluation on the [`catcola`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/catalan_bench/catcola.yaml) task with the following command:
```
accelerate launch -m lm_eval --model hf \
    --model_args pretrained=EuroLLM-1.7B-Instruct,trust_remote_code=True \
    --tasks catcola \
    --num_fewshot 5 \
    --batch_size 1 \
    --output_path chat_template_test_results.json \
    --log_samples \
    --trust_remote_code \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --seed 1234
```

But in the results file, the chat template is null.
```
]$ cat chat_template_test_results.json | grep chat_template
  "chat_template": null,
  "chat_template_sha": null,
```

I'm attaching a zip file with the results file and the two log files from the Slurm job, with stdout in `chat_template_test.out` and stderr in `chat_template_test.err`. There doesn't seem to be anything about chat template issues in the logs.

Would appreciate any help to solve this issue!

[chat_template_test.zip](https://github.com/user-attachments/files/18180448/chat_template_test.zip)

Hi! I'll have a look. Can you also confirm by having a look at the samples file? Should be obvious if the `args` are wrapped up in chat formatting

Hello. I am having the same issue with [utter-project/EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct). Did you solve this problem?
Repository: EleutherAI/lm-evaluation-harness
Issue: Weird results for 70b models
Root cause: Validation Gap

I have forked the repository and created custom tasks in the lm_eval -> tasks directory. All of them are MMLU-like tasks and are in the Kazakh language.

I am running evaluations using the following command:

`lm_eval     --model hf     --model_args pretrained=meta-llama/Meta-Llama-3-8B-Instruct    --batch_size 2    --num_fewshot 0  --apply_chat_template    --tasks mmlu_translated_kk,kazakh_and_literature_unt_mc,kk_biology_unt_mc,kk_constitution_mc,kk_dastur_mc,kk_english_unt_mc,kk_geography_unt_mc,kk_history_of_kazakhstan_unt_mc,kk_human_society_rights_unt_mc,kk_world_history_unt_mc     --output output    --device cuda:0`

The problem is that for the LLama 3 8B models, I get an average accuracy of around 0.3–0.4, which is correct. I have verified this by running each model manually with the generation part.

However, for the larger models like LLama instruct 3.3-70B, LLama 3.1-70B, gemma-2-27b-it and others, I keep getting an accuracy score of ~0.1 across all tasks.

What am I doing wrong? its
Repository: EleutherAI/lm-evaluation-harness
Issue: Using ElutherAI to create a  interactive evaluation app
Root cause: Documentation Deficiency

I am trying to create an app for my company that sends the model path and the tasks that they want the model to be evaluated on and pass it for evaluation to elutherAI, is there any API which i can call with this data and get the response if not then how can i do it?
Repository: EleutherAI/lm-evaluation-harness
Issue: Couldn't detect gpu when generation using ray data_parallel_size > 1
Root cause: Environment Incompatibility

I'm trying to use data_parallel to generation, this is my command: 
```
CUDA_VISIBLE_DEVICES="0,1"
lm_eval --model vllm \
    --model_args pretrained=./downloads/llama3.2-1b-instruct,trust_remote_code=true,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.3,data_parallel_size=2 \
    --tasks truthfulqa_gen \
    --batch_size auto
```
I got an error:
![image](https://github.com/user-attachments/assets/670a3ae9-be5f-4928-a8bb-62558f0a680c)
And I try to print cuda devices like this:
![image](https://github.com/user-attachments/assets/637def3b-9a99-42a3-9e57-8c12f37c8878)
ray: 2.39.0
vllm: 0.6.4.post1
torch: 2.5.1+cu124
I wonder is there something wrong with my environment and how to solve this problem.

I reset my environment and it's ok now.

Hi there, I ran into a similar problem of getting a ValueError saying no GPU available when it literally prints out the two GPUs that I requested. Can I know how you reset your environment to make everything work? I tried to do the same thing but this error still shows up.

`ValueError: Current node has no GPU available. current_node_resource={'bundle_group_e8bc9d1a4e9c3b75ca8af7086f2b01000000': 2000.0, 'GPU_group_e8bc9d1a4e9c3b75ca8af7086f2b01000000': 2.0, 'node:10.0.8.9': 0.999, 'bundle_group_1_e8bc9d1a4e9c3b75ca8af7086f2b01000000': 1000.0, 'GPU_group_0_e8bc9d1a4e9c3b75ca8af7086f2b01000000': 1.0, 'node:__internal_head__': 1.0, 'node:10.0.8.9_group_0_e8bc9d1a4e9c3b75ca8af7086f2b01000000': 0.001, 'bundle_group_0_e8bc9d1a4e9c3b75ca8af7086f2b01000000': 1000.0, 'CPU': 12.0, 'object_store_memory': 76673319321.0, 'accelerator_type:A40': 1.0, 'memory': 168904411751.0, 'node:10.0.8.9_group_e8bc9d1a4e9c3b75ca8af7086f2b01000000': 0.001, 'GPU_group_1_e8bc9d1a4e9c3b75ca8af7086f2b01000000': 1.0}. vLLM engine cannot start without GPU. Make sure you have at least 1 GPU available in a node current_node_id='61912222bec3e4a6e27043a94fb7162fc186a6830292f0f4e67ae25f' current_ip='10.0.8.9'.`

Any update on this issue? I'm also facing it but it happens to me quite randomly (sometimes after a while, sometimes at the very beginning, and sometimes it just works)