zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: fail to evaluate piqa
Root cause: External Dependency Breakage

hello
I am trying to evaluate llama-2-13b-hf with v0.4.7 on dataset piqa but get error

  **File "/home/xwj/llm/lm-evaluation-harness/lm_eval/api/task.py", line 819, in __init__
    self.download(self.config.dataset_kwargs)
  File "/home/xwj/llm/lm-evaluation-harness/lm_eval/api/task.py", line 926, in download
    self.dataset = datasets.load_dataset(
  File "/usr/local/anaconda3/envs/xwj_transformers/lib/python3.12/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/usr/local/anaconda3/envs/xwj_transformers/lib/python3.12/site-packages/datasets/load.py", line 2265, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable**

and the command is
lm_eval --model hf  --model_args pretrained="/home/llama-2-13b-hf" --tasks piqa --device cuda:0 --batch_size 8

can anyone help please?

I seem to solve this problem by replacing
**dataset_path: piqa** to **dataset_path: nthngdy/piqa**
in **tasks/piqa.yaml**

but I don't know whether it is correct

Same problem here... This problem affects on hellaswag, piqa, social_iqa...

Same problem☹️

Hi! I can't reproduce this. If it's a network issue then some people have had success with https://github.com/EleutherAI/lm-evaluation-harness/issues/1634#issuecomment-2019517937,
might be other mirrors as well. 

thanks for your comment!
and maybe one solution is replacing the dataset_path which has data rather than .py file on hug.

@baberabb same problem☹️

Same problem, how to fix on hellaswag dataset

> Same problem, how to fix on hellaswag dataset

does `export HF_ENDPOINT=https://hf-mirror.com/` not help?

Same problem, export HF_ENDPOINT=https://hf-mirror.com/ is not help.

> Same problem, export HF_ENDPOINT=https://hf-mirror.com/ is not help.
downgrade datasets to 2.21.0 fixed hellaswag  and winogrande issues for me, but not piqa,


Hi! I've copied the parquet files to [a new HuggingFace repo](https://huggingface.co/datasets/baber/piqa/) and updated the path in #2790. If someone could confirm they can run it successfully, I'll proceed with the merge.

I think the issue was that the original dataset sources files from storage.googleapis.com ([see](https://huggingface.co/datasets/ybisk/piqa/blob/2e8ac2dffd59bac8c3c6714948f4c551a0848bb0/piqa.py#L58)), which may be blocked depending on your location.

> Hi! I've copied the parquet files to [a new HuggingFace repo](https://huggingface.co/datasets/baber/piqa/) and updated the path in [#2790](https://github.com/EleutherAI/lm-evaluation-harness/pull/2790). If someone could confirm they can run it successfully, I'll proceed with the merge.
> 
> I think the issue was that the original dataset sources files from storage.googleapis.com ([see](https://huggingface.co/datasets/ybisk/piqa/blob/2e8ac2dffd59bac8c3c6714948f4c551a0848bb0/piqa.py#L58)), which may be blocked depending on your location.

It works for me, thanks, really appreciated
Repository: EleutherAI/lm-evaluation-harness
Issue: Passing a limit doesn't randomly sample, but rather takes dataset[:limit], introducing dataset bias
Root cause: Algorithmic Error

Hi,

I've noticed that the limit passed to evaluate method doesn't randomly select from the dataset, as I would expect, but rather just takes the [first n samples where n is the limit ](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L462)

I understand how this works better with cached results, but it does introduce a sampling bias when calculating results, b/c although the guidance is to only use for testing, it is very useful for datasets which have a large amount of samples (e.g. MMLU). 

Would it possible to pass an additional param that allows for random sampling instead of first N? Alternatively, pass in a HuggingFace dataset to the evaluate method?
Repository: EleutherAI/lm-evaluation-harness
Issue: Significant Discrepancy in ARC-Challenge Accuracy: Llama-3.2-3B Official vs lm-evaluation-harness
Root cause: Configuration Error

## Description
I've observed a substantial difference between the officially reported ARC-Challenge accuracy and my local evaluation results using lm-evaluation-harness.

## Official Results

Reported 25-shot arc_challenge accuracy on HuggingFace repo (meta-llama/Llama-3.2-3B): 69.1%

## Local Evaluation Results
### Base Model Test
My command line:
`accelerate launch -m lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-3.2-3B,trust_remote_code=True \
    --tasks arc_challenge \
    --batch_size 8 \
    --num_fewshot 25`
    
Results:
| Tasks | Version | Filter | n-shot | Metric | | Value | | Stderr |
|-------|---------|--------|--------|---------|---|-------|---|--------|
| arc_challenge | 1 | none | 25 | acc | ↑ | 0.4667 | ± | 0.0146 |
| | | none | 25 | acc_norm | ↑ | 0.5068 | ± | 0.0146 |

### Instruct Model
My hypothesis is that, the Llama model requires a chat template. Therefore, I tested the meta-llama/Llama-3.2-3B-Instruct with the official chat template:
`accelerate launch -m lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-3.2-3B-Instruct,trust_remote_code=True,dtype="bfloat16" \
    --tasks arc_challenge \
    --num_fewshot 25 \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --batch_size 8 `

However, this produces similar results:
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr |
|-------|---------|---------|--------|---------|--------|---------|
| arc_challenge | 1 | none | 25 | acc | 0.4983 | 0.0146 |
| arc_challenge | 1 | none | 25 | acc_norm | 0.5341 | 0.0146 |

I'm wondering about what caused this inconsistency.

Hi! They use the MMLU format for this task (`<question><A. answer_choice>,...><Answer: ><"A">`) rather than the standard cloze format (`<question><Answer: ><"answer_choice">`).

You can run it on #2556 (`llama_arc_challenge`)

Hi! I also foundthat lm-eval uses the standard cloze format instead of the origin MMLU format on some datasets.  I'm curious about the reason because sometimes using MMLU format will lead to a better performance. So the reason is to be consistent with other works or ??? @baberabb
Repository: EleutherAI/lm-evaluation-harness
Issue: Are the function names mixed up in preprocess_winogrande.py？
Root cause: Documentation Deficiency

https://github.com/EleutherAI/lm-evaluation-harness/blob/1044db955c9072e6fb7e15da08114005ba9d91cc/lm_eval/tasks/winogrande/preprocess_winogrande.py#L1-L14

The def of **doc_to_text** and **doc_to_target** should be inverted

Hi! not quite! winogrande is implemented in a slightly non-standard way here in that we keep the continuation the same and switch over the contexts.  Theres a good discussion about it in #978

Thank you! 
I think the reason why I got confused is my false assuming of the prompt construction logic about doc_to_text  doc_to_choice and doc_to_target. It's clear now after reading your ref and the implementation in https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py 

For anyone who is also confused: 
https://github.com/EleutherAI/lm-evaluation-harness/blob/370e2f9e5bbe59912644b1b6e052e17be31d6858/lm_eval/tasks/winogrande/preprocess_winogrande.py#L1-L3

https://github.com/EleutherAI/lm-evaluation-harness/blob/370e2f9e5bbe59912644b1b6e052e17be31d6858/lm_eval/api/task.py#L892-L893
Repository: EleutherAI/lm-evaluation-harness
Issue: Question about the evaluation of OpenBookQA
Root cause: Configuration Error

It seems that the fact is not provided to give a answer, as a openbook qa, it may be better to use the fact, otherwise it become "closebookqa"
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/openbookqa/openbookqa.yaml

task: openbookqa
dataset_path: openbookqa
**dataset_name: main** **#here maybe additional**
output_type: multiple_choice
training_split: train
validation_split: validation
test_split: test
**doc_to_text: question_stem**  **#here may need use fact1**
doc_to_target: "{{choices.label.index(answerKey.lstrip())}}"
doc_to_choice: "{{choices.text}}"
should_decontaminate: true
doc_to_decontamination_query: question_stem
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: acc_norm
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0
Repository: EleutherAI/lm-evaluation-harness
Issue: Tag for mgsm_cot_native and mgsm_cot_en same?
Root cause: Configuration Error

The yaml files for mgsm with CoT in same language as dataset and with English have the same tag - mgsm_cot_native.
See here:
[with english](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mgsm/en_cot/cot_yaml)
and this:
[with native language](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mgsm/native_cot/cot_yaml)

Hi! thanks for the report. Seems to be an oversight. Should be different
Repository: EleutherAI/lm-evaluation-harness
Issue:  output_path arg is no longer available in lm_eval/evaluator.py
Root cause: Documentation Deficiency

The output_path arg is no longer available in lm_eval/evaluator.py. Is this a feature or a bug? I can't spot the paper trail leading to its disappearance after this PR (https://github.com/EleutherAI/lm-evaluation-harness/pull/1766)

Hi! That PR consolidated the logging a bit. Now have to pass in an instance of [EvaluatorTracker](https://github.com/EleutherAI/lm-evaluation-harness/blob/bb098f13b05e361f01a5afe7b612779ce362b3f2/lm_eval/loggers/evaluation_tracker.py#L106), and can you its methods like [so](https://github.com/EleutherAI/lm-evaluation-harness/blob/bb098f13b05e361f01a5afe7b612779ce362b3f2/lm_eval/__main__.py#L431-L439).
Repository: EleutherAI/lm-evaluation-harness
Issue: When I set device=CPU, long-time stuck after running loglikelihood requests.
Root cause: Resource Mishandling

When I set device=CPU, long-time stuck after running loglikelihood requests. But when set device=cuda, it can work.

This is cmd:
lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B,trust_remote_code=True \
    --tasks piqa \
    --device cpu \
--batch_size 64\
--output_path ./eval_out --use_cache ./eval_cache


I have a similar problem also when using GPU.

+1
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug with loglikelihood_rolling: TypeError: 'float' object is not subscriptable
Root cause: Interface Contract Mismatch

When I implement the `loglikelihood_rolling` for calculating the perplexity of answers in #258 
```
ll_true = rf.loglikelihood_rolling(ctx.replace("[MASK]", "Yes"))[0]
ll_neither = rf.loglikelihood_rolling(ctx.replace("[MASK]", "Also"))[0]
ll_false = rf.loglikelihood_rolling(ctx.replace("[MASK]", "No"))[0]
```
I've encounter the following bug in the file `lm_eval/evaluator.py`.
```
Traceback (most recent call last):
  File "main.py", line 71, in <module>
    main()
  File "main.py", line 52, in main
    description_dict=description_dict
  File "/gpfs/data/sbach/zyong2/mt0/data/external/lm-evaluation-harness/lm_eval/utils.py", line 156, in _wrapper
    return fn(*args, **kwargs)
  File "/gpfs/data/sbach/zyong2/mt0/data/external/lm-evaluation-harness/lm_eval/evaluator.py", line 69, in simple_evaluate
    description_dict=description_dict
  File "/gpfs/data/sbach/zyong2/mt0/data/external/lm-evaluation-harness/lm_eval/utils.py", line 156, in _wrapper
    return fn(*args, **kwargs)
  File "/gpfs/data/sbach/zyong2/mt0/data/external/lm-evaluation-harness/lm_eval/evaluator.py", line 187, in evaluate
    resps = [x if req.index is None else x[req.index] for x, req in zip(resps, reqs)]
  File "/gpfs/data/sbach/zyong2/mt0/data/external/lm-evaluation-harness/lm_eval/evaluator.py", line 187, in <listcomp>
    resps = [x if req.index is None else x[req.index] for x, req in zip(resps, reqs)]
TypeError: 'float' object is not subscriptable
```

The current fix is to use `resps = [x for x, req in zip(resps, reqs)]` instead of `resps = [x if req.index is None else x[req.index] for x, req in zip(resps, reqs)]`. 

Try again without doing `[0]` on the loglikelihood_rolling - that should just return one value
Repository: EleutherAI/lm-evaluation-harness
Issue: Add HRM8K (new math benchmark)
Root cause: Unimplemented Feature Gap

[HRM8K](https://huggingface.co/datasets/HAERAE-HUB/HRM8K) dataset is just out! 

This benchmark is a bilingual math reasoning benchmark for Korean and English. HRM8K comprises 8,011 instances for evaluation, sourced through a combination of translations from established English benchmarks (e.g., GSM8K, MATH, OmniMath, MMMLU) and original problems curated from existing Korean math exams.

I want to add this benchmark similar to Hendrycks Math.
Repository: EleutherAI/lm-evaluation-harness
Issue: fatal: not a git repository with library usage
Root cause: Validation Gap

Hello, I'm trying to use lm_eval within python interpreter instead of CLI. Therefore, I referred "[External Library Usage](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage)".

### CLI Usage

First, this is what I'm trying to do in the python interpreter. 

```bash
lm_eval --model hf \
  --model_args pretrained=~/Llama-3.2-1B,trust_remote_code=True \
  --tasks arc_challenge --batch_size 8 --num_fewshot 25

hf (pretrained=Llama-3.2-1B,trust_remote_code=True), gen_kwargs: (None), 
      limit: None, num_fewshot: 25, batch_size: 8
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |    25|acc     |↑  |0.3660|±  |0.0141|
|             |       |none  |    25|acc_norm|↑  |0.3933|±  |0.0143
```

### Library Usage (My question)

What I want to evaluate is my customized hugging face model. Therefore, I should pass updated model object.

```python
from lm_eval.utils import make_table
import lm_eval

from my_custom_llama import LlamaForCausalLM
model = LlamaForCausalLM.from_pretrained('Llama-3.2-1B')
model = lm_eval.models.huggingface.HFLM(pretrained=model)

results = lm_eval.simple_evaluate(
    model,
    tasks=['arc_challenge'],
    num_fewshot=25,
    batch_size=8,
)

print (make_table(results))
```

This is the printed stdout.
```bash
fatal: not a git repository (or any of the parent directories): .git
|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |    25|acc     |↑  |0.3635|±  |0.0141|
|             |       |none  |    25|acc_norm|↑  |0.3976|±  |0.0143|
```

Seems that it is same with the one from CLI Usage. Except the fatal git error.

Have I done something wrong?

It seems like something is set up a bit wonky locally, but that it's not impacting your ability to run the library or get correct results. I'm going to close this for now, unless you find code that isn't working or substantial difference in evaluation results between the two modes.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add global_mmlu full version
Root cause: Unimplemented Feature Gap

Current global_mmlu is using `CohereForAI/Global-MMLU-Lite` not `CohereForAI/Global-MMLU`. `CohereForAI/Global-MMLU-Lite` only supports 14 languages but `CohereForAI/Global-MMLU` supports 42 languages. Also, it can provide more detailed comparison.
Repository: EleutherAI/lm-evaluation-harness
Issue: revise `mbpp` prompt
Root cause: Configuration Error

based on [original code](https://github.com/google-research/google-research/tree/master/mbpp), prompt format is:
`You are an expert Python programmer, and here is your task: {prompt} Your code should pass these tests:\n\n{tests}\n[BEGIN]\n{code}\n[DONE]`
but current implementation has no '\n' after last [BEGIN].

to fix this, add `\n` on `doc_to_text` at the end, replace `target_delimiter` `"\n"` with `""`:
```diff
-doc_to_text: "You are an expert Python programmer, and here is your task: {{text}} Your code should pass these tests:\n\n{{test_list[0]}}\n{{test_list[1]}}\n{{test_list[2]}}\n[BEGIN]"
-target_delimiter: "\n"
+doc_to_text: "You are an expert Python programmer, and here is your task: {{text}} Your code should pass these tests:\n\n{{test_list[0]}}\n{{test_list[1]}}\n{{test_list[2]}}\n[BEGIN]\n"
+target_delimiter: ""
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Question about humaneval
Root cause: Configuration Error

I tried to evaluate humaneval on meta-llama-3.1-instruct, but got a score close to 0. I printed the output and found
```json
{
    "resps": [
        [
            "Here's a Python function that implements the required functionality:\n\n```python\nfrom typing import List\n"
        ]
    ]
}
```
I think this may be due to `generation_kwargs.until` in the configuration. So what is the correct way to evaluate?

Hi! what model are you using?

I use `Meta-Llama-3.1-8B-Instruct` and run with
```bash
lm_eval --model vllm --model_args="pretrained=${model_name},dtype=auto,tensor_parallel_size=${GPUS_PER_NODE},max_model_len=16384,gpu_memory_utilization=0.9,enable_chunked_prefill=True" --tasks=humaneval --batch_size=auto --output_path results --apply_chat_template --fewshot_as_multiturn --gen_kwargs="stop_token_ids=[128009]"
```

Looks like a prompting/answer extraction issue. Added the prompt from llama evals (as `humaneval_instruct`) in the PR, but score still lower then official (0.5976 vs. 0.726 )
Repository: EleutherAI/lm-evaluation-harness
Issue: few-shot examples are not presented by their subjects for Global-MMLU-Lite (`global_mmlu_{language}`)
Root cause: Architectural Constraint

Current implementation for `global_mmlu_{lang}` use random subjects because they do not separate each subject; instead they merge all subjects as one task. I want to separate each subject so provide few-shots from same domain for each subject as original mmlu. (global_mmlu_full_{lang} works as mmlu does)

I tried separate each subject but some subjects have very few examples so I decided to separate category not subject.
Repository: EleutherAI/lm-evaluation-harness
Issue: maximum sequence length
Root cause: Validation Gap

the error below occurs with some models running on evaluation is there any solutions :
Token indices sequence length is longer than the specified maximum sequence length for this model (7614 > 2048). Running this sequence through the model will result in indexing errors. 


Hi! What model is this? seems like the model only supports length up to 2048

how do we truncate the sequence then?
Repository: EleutherAI/lm-evaluation-harness
Issue: Issues running any bigbench tasks
Root cause: External Dependency Breakage

I am getting the following error while trying to run any bigbench tasks:
`TypeError: TaskConfig.__init__() got an unexpected keyword argument 'group'. `

I am using the correct nomenclature of the task bigbench_<subtask_name>_<generate_until/multiple_choice>, but this issue seems to persist.

Any idea about the solution? 

Hi! Sorry about this. The PR should fix this.
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue running Squadv2 in LM-Evaluation-Harness
Root cause: Interface Contract Mismatch

Hi, I am facing the following error while trying to run squadv2:

`TypeError: Instance.__init__() got an unexpected keyword argument 'apply_chat_template'`

It stems from this [line](https://github.com/EleutherAI/lm-evaluation-harness/blob/1208afd34ce132e598fcd7e832762630a35d01c6/lm_eval/tasks/squadv2/task.py#L121).

Any solution?

Hi! The PR should resolve this!

Hi, is there a solution to this?

Sorry, I thought I already merged this
Repository: EleutherAI/lm-evaluation-harness
Issue: FileNotFoundError: https://storage.googleapis.com/ai2-mosaic/public/winogrande/winogrande_1.1.zip
Root cause: External Dependency Breakage

Hello,

Seems like Winogrande and PIQA cannot be downloaded today. Is there any workaround?

https://storage.googleapis.com/ai2-mosaic/public/winogrande/winogrande_1.1.zip

Thank you!

It's fixed
Repository: EleutherAI/lm-evaluation-harness
Issue: Support `processor_kwargs` for `hf-multimodal`
Root cause: Unimplemented Feature Gap

We're  using LM-Harness to evaluate `Qwen/Qwen2-VL-2B-Instruct` on `task_name: mmmu_val` using 4 `A100` GPUs w/ 40GB VRAM.

The evaluation starts and progresses OK until the middle of the dataset where it fails with CUDA OOM (out-of-memory). It looks like it may be caused by a single example in the dataset which requires more GPU memory than usual, leading to OOM. 

This Qwen2-VL OOM problem was resolved for _training_, by passing the following params as Qwen2-VL `processor_kwargs`:
```
{
  "min_pixels": 256 * 28 * 28,
  "max_pixels": 1280 * 28 * 28,
}
``` 
Context: https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct#image-resolution-for-performance-boost

For LH-Harness evaluation, I haven't found a way to configure `processor_kwargs` for `hf-multimodal` (only `model_args` seem configurable): https://github.com/EleutherAI/lm-evaluation-harness/blob/5c006ed417a2f4d01248d487bcbd493ebe3e5edd/lm_eval/models/hf_vlms.py#L134

Could you please advise if setting `processor_kwargs`  is supported? If not, would it be possible to add it for `hf-multimodal` ?



The PR should let you pass `min_pixels`, `max_pixels` to the model args, but just so I don't forget, need to think of a better way of passing sub-method kwargs
Repository: EleutherAI/lm-evaluation-harness
Issue: Use AWS Bedrock Models
Root cause: Unimplemented Feature Gap

I am trying to use lm-eval-harness to evaluate Claude Sonnet, Haiku, Mistral, Meta Llama, and Titan models. I am accessing these models through AWS Bedrock. In anthropic_llms.py, there is a function called self.client.get_tokenizer() that gets the Anthropic tokenizer. But, I am using an AWS Bedrock client instead of an Anthropic client. How do I get access to this tokenizer without having an Anthropic OpenAI key? The tokenizer does not appear to be part of tiktoken or HuggingFace. Is there a different way to run lm-eval-harness on Amazon Bedrock Models

Is this possible?
Repository: EleutherAI/lm-evaluation-harness
Issue: Running Global MMLU
Root cause: Documentation Deficiency

Hi! I'm trying to run eval on the global mmlu benchmark (https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/global_mmlu) but for some reason don't see this as part of the latest package download (double checked I'm on v0.4.7). I did a clean install etc and still am not able to find this task in the task folder. Do you know how I can include this for running eval?

Hi! This is on a recent commit, and not pushed to pypi yet. Planning to cut a new release this week, but in the meanwhile you can clone directly

Got it, thank you for the quick response!
Repository: EleutherAI/lm-evaluation-harness
Issue: Assert "vLLM only supports CUDA" is not true
Root cause: Validation Gap

The assert that is placed in [vllm_causallms.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/vllm_causallms.py#L78C8-L78C77) doesn't look right for me. There are many platforms supported by vLLM project: https://docs.vllm.ai/en/latest/getting_started/installation/index.html, but mentioned assert blocks usage of `lm-evaluation-harness` with them.

Simple proof why this assert is not necessary:
I have ran lm-eval with vLLM on different platform than CUDA, but didn't specify the device argument. The assert was not called and the rest of execution works properly (vLLM automatically detected which platform I used). However, it could be confusing for users who specifies the platform as "xpu", "tpu" or "hpu" and will receive this AssertionError.

Hi! Thanks for the report. The PR should fix this :)