zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Reproduce QWen 2.5-14B-Instruct and LLaMa-3.1-8B-Instruct Results
Root cause: Documentation Deficiency

Hi, thank you all for very nice work. I would like to use this wonderful work to re-produce the evaluation performance reported by qwen 2.5 and llama 3.1. However, I encounter several problems as shown below:

the performance of llama3.1-8b-instruct on gsm8k-cot-zeroshot
<img width="806" alt="image" src="https://github.com/user-attachments/assets/4680d6eb-baf4-452a-ab20-a55a845c7570">
the performance of qwen-2.5-14b-instruct on gsm8k-cot-zeroshot
<img width="585" alt="image" src="https://github.com/user-attachments/assets/543c225d-534a-4ac0-bb52-aa79bea64f90">

the scores are much lower compared with original reports, especially for llama3.1-8b-instruct

I directly use command like 

lm_eval --model hf --tasks gsm8k_cot_zeroshot --model_args pretrained=/pathto/Qwen2.5-14B-Instruct,parallelize=True --batch_size 64 --output_path ./results --log_samples

is there anything wrong? should I change dtype, batch_size, and also template / max_tokens or others?
besides that, when i use this command to run gsm8k, the program just get stuck without running the evaluation after loading the model.

I faced the same problem.

Same problem here. 


Same problem here.
Repository: EleutherAI/lm-evaluation-harness
Issue: tasks RACE only high not "middle"
Root cause: Unimplemented Feature Gap

I understand that the RACE task includes both ‘high’ and ‘middle’ levels. 
In various papers, experiments are conducted separately on RACE-high and RACE-middle to measure performance. 
However, in this package, only RACE-high is available, and RACE-middle is missing. 
Is there a reason for this? Do you have any plans to add this task as well?
Repository: EleutherAI/lm-evaluation-harness
Issue: The base model and chat model have no difference when using generate_until, loglikelihood, loglikelihood_rolling,right?
GitHub labels: asking questions

The base model and chat model have no difference when using generate_until, loglikelihood, loglikelihood_rolling,right?
generate_until, loglikelihood, loglikelihood_rolling,right is related to the dataset I use,right?
If i use a long answer dataset,i config it generate_until, while I use multi-choice dataset, i config it multi-choice,right?

yes, thats correct. You probably want to use `--apply_chat_template` (and optionally `--fewshot_as_multiturn`) for the instruct models to use the chat templates.
Repository: EleutherAI/lm-evaluation-harness
Issue: squadv2 task occurred "AttributeError: module 'datasets' has no attribute 'load_metric'" 
Root cause: External Dependency Breakage

Here is the command I executed:
```
lm_eval --model hf --model_args pretrained=gpt2 --tasks squadv2 --limit 1
```

and here is the error
```
Traceback (most recent call last):
  File "/media/chengpong/Repository/Dehalluciation/.venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/media/chengpong/Repository/Dehalluciation/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/chengpong/Repository/Dehalluciation/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/media/chengpong/Repository/Dehalluciation/lm-evaluation-harness/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/media/chengpong/Repository/Dehalluciation/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/media/chengpong/Repository/Dehalluciation/lm-evaluation-harness/lm_eval/evaluator.py", line 595, in evaluate
    task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)
  File "/media/chengpong/Repository/Dehalluciation/lm-evaluation-harness/lm_eval/evaluator_utils.py", line 104, in calculate_aggregate_metric
    self.agg_metrics[metric_key] = agg_fn(items)
                                   ^^^^^^^^^^^^^
  File "/media/chengpong/Repository/Dehalluciation/lm-evaluation-harness/lm_eval/tasks/squadv2/task.py", line 47, in _squad_agg
    return _squad_metric(predictions=predictions, references=references).get(key, 0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/chengpong/Repository/Dehalluciation/lm-evaluation-harness/lm_eval/tasks/squadv2/task.py", line 40, in _squad_metric
    squad_metric = datasets.load_metric("squad_v2")
                   ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'datasets' has no attribute 'load_metric'
```

Expected: no errors.
Repository: EleutherAI/lm-evaluation-harness
Issue: I have a question about the postprocess
Root cause: Documentation Deficiency

Hi! thank you for a great project.  
I have a question about the postprocess.   
  
When solving the following tasks with zero-shot:  
```
"The amazing performance of GPT-3!" Which section should the title of this article 
fit into IT science, economy, and society?
```

The model is ideal for releasing one of IT science, economy, and society, but because it is a generative model, it may not come up with one of them. or example, "The answer is IT science" can be predicted by the model.  
  
How do you deal with this? If it's wrong answer, how do you calculate the F1-score?



@StellaAthena Can you answer me?

Unfortunately this is a fundamental limitation of free-answer generation tasks. There is no known way to evaluate whether an arbitrary string answers an arbitrary natural language question. This is why people generally prefer multiple choice evaluation metrics, where this is (less of) a problem.
Repository: EleutherAI/lm-evaluation-harness
Issue: Setting limit_mm_per_prompt for vllm_vlm fails argument parser
Root cause: Validation Gap

Using the latest vllm release and current main of lm-eval, I see this error:

```
lm_eval --model vllm-vlm --model_args pretrained=llava-hf/llava-1.5-7b-hf,limit_mm_per_prompt="image=50" --tasks mmmu_val

Traceback (most recent call last):
  File "/home/mgoin/venvs/vllm/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/utils.py", line 398, in _wrapper
    return fn(*args, **kwargs)
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/evaluator.py", line 199, in simple_evaluate
    f"Initializing {model} model, with arguments: {simple_parse_args_string(model_args)}"
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/utils.py", line 107, in simple_parse_args_string
    args_dict = {
  File "/home/mgoin/code/lm-evaluation-harness/lm_eval/utils.py", line 108, in <dictcomp>
    k: handle_arg_string(v) for k, v in [arg.split("=") for arg in arg_list]
ValueError: too many values to unpack (expected 2)
```

I printed out the intermediate arguments before that line and the issue seems to be the overuse of "=" in the value of the argument
```
[['pretrained', 'llava-hf/llava-1.5-7b-hf'], ['limit_mm_per_prompt', 'image', '50']]
```

Maybe we could get past this by directly passing a dict in the parameter and using another parser, such as `limit_mm_per_prompt="{'image':50}"`

We've worked around this for now with #2387 ! -- `limit_mm_per_prompt="{'image':50}"` isn't a bad idea but looks a little irritating for a user to specify.
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation of MMLU tasks using a fined tuned Gemma 2 model 
Root cause: Architectural Constraint

Hi, 

I’m currently working on evaluating a fine-tuned Gemma 2 model that was built using Keras. The model is saved as a single file (model.keras). However, I’ve noticed that the evaluation harness seems to expect models with multiple configuration files (e.g., config.json, etc.).

Since Keras models typically don't have these separate configuration files, I’m running into difficulties when trying to run the evaluation harness on my fine-tuned model.

Is there a workaround or an alternative way to adapt the evaluation harness for models that are saved in this format? Any guidance or suggestions would be greatly appreciated!

Thanks in advance!
Repository: EleutherAI/lm-evaluation-harness
Issue: --tasks mmlu
Root cause: Documentation Deficiency

lm_eval --model hf     --model_args pretrained=./mfs-7b-32k     --tasks mmlu     --device cuda:3     --batch_size 8    --output_path ./eval_out_5shot_mmlu   --num_fewshot 0  --limit 5  --predict_only


so how to set --tasks  to use mmlu or subtask of mmlu, it maybe the newest version is not the same as the old version in loading *.yaml file

Hi, `DATASET_NAME` refers to the `name` parameter in `datasets.load_dataset()`, which indicates the *subset* of a given dataset found under `DATASET_PATH` (local path or name of dataset, e.g. `hails/mmlu_no_train` on the hub).

Are you using a locally saved MMLU dataset and have you edited the YAML files for MMLU to point to these (parquet) files? if so, then this looks like correct output assuming that `dataset_kwargs` contains the right filepath to your saved MMLU dataset on-disk.
Repository: EleutherAI/lm-evaluation-harness
Issue: Improve `docs/model_guide.md` with skeleton template code + description of utils like `Collator` and `Reorderer`
Root cause: Documentation Deficiency

As per title, some of the machinery we use in HFLM around the ordering, batching, and caching of requests is still opaque to most users. 

It'd be great if we could expand `model_guide.md` with:
- in `templates/new_template_model`, a filestub containing the rough outline of a fully-featured HF-like local model impl that uses reorderer, collator, partial caching, and so on
- Content that explains key footguns / common mistakes when writing a new model class (/ a mini FAQ)
- more description and documentation of what the different request types are and what they do (perhaps turn this into a blogpost we can link out to instead?)
Repository: EleutherAI/lm-evaluation-harness
Issue: boolq trust remote code
Root cause: Configuration Error

Evaluation on BoolQ fails due to trust_remote_code=False during hf dataset downloading.

Looks like boolq task config lacks this argument.

```yaml
tag:
  - super-glue-lm-eval-v1
task: boolq
dataset_path: super_glue
dataset_name: boolq
output_type: multiple_choice
training_split: train
validation_split: validation
doc_to_text: "{{passage}}\nQuestion: {{question}}?\nAnswer:"
doc_to_target: label
doc_to_choice: ["no", "yes"]
should_decontaminate: true
doc_to_decontamination_query: passage
metric_list:
  - metric: acc
metadata:
  version: 2.0
```

for example PIQA config has it

```yaml
...
dataset_kwargs:
  trust_remote_code: true
```


please, check if i am correct, and i will make a PR.

Hi!

We don't apply `trust_remote_code: true` to all remote-code-requiring configs because we expose a flag `--trust_remote_code` to allow users to run any remote-code-needed dataset without manual yes/no confirmation on the console. 

If `--trust_remote_code` does not allow you to run this dataset with no manual confirmation needed, please let us know!

I see. I am not using CLI, but directly calling `simple_evaluate`. There is no such flag there.

Do you suggest setting `HF_DATASETS_TRUST_REMOTE_CODE=1` manually before running `simple_evaluate` in this case?
Wouldn't it be more consistent to update task config?

Thanks.
Repository: EleutherAI/lm-evaluation-harness
Issue: Which filter value should be used among the accuracy test results?
Root cause: Documentation Deficiency

For example, in the accuracy test results of gsm8k_cot, there are two filters: 'flexible-extract' and 'strict-match'. I don't know which of the various filters meta, mircrosoft, or academia select when reporting accuracy.

<img width="929" alt="제목 없음" src="https://github.com/user-attachments/assets/682fb15d-639c-4614-97cb-009d1c7d1811">


Same question

+1
Repository: EleutherAI/lm-evaluation-harness
Issue: Extracting vLLM metrics
Root cause: Unimplemented Feature Gap

I'm using a vLLM model with simple_evaluate function on a benchmark task and I'm very much interested in extracing vLLM metrics such as time to first token, time in queue, etc. I've been reading through the source code to see if this is currently supported and I found the following in lm_eval/models/vllm_causallms.py

```
            for output, context in zip(cont, context):
                generated_text = output.outputs[0].text
                res.append(generated_text)
                ...
```

I was wondering if the following could be added as an option for vLLM models:

```
            for output, context in zip(cont, context):
                generated_text = output.outputs[0].text
                generated_metrics = output.metrics        # extracting vLLM metrics
                res.append(generated_text)
                res.append(generated_metrics)             # appending metrics to results
                ...
```
Thanks! This feature would be greatly appreciated.
Repository: EleutherAI/lm-evaluation-harness
Issue: How can we connect to azure openai deployed models?
Root cause: Documentation Deficiency


Please can anyone share some reference if any.

You can directly use 'local-chat-completions' by passing the relevant Azure HTTP URL to the 'base_url' parameter.

i want to know how to solve it,have you solved this problem? @patilpriyadarshini 

@tmracy Yes solved
In lm_eval->models->openai_completions.py->openai-completions you update the base_url as per your azure openai api.
Repository: EleutherAI/lm-evaluation-harness
Issue: set trust_remote_code when loading tokenizer in api_model
Root cause: Configuration Error

Hi, when evaluating internlm/internlm2_5-1_8b-chat with local inference server (vllm) I'm getting an error, because lm-eval can't load tokenizer:

```
2024-10-01:12:40:20,544 INFO     [evaluator.py:198] Initializing local-completions model, with arguments: {'base_url': 'http://localhost:8080/v1/completions', 'num_concurrent': 1, 'max_retries': 3, 'tokenized_requests': False, 'model': 'internlm/internlm2_5-1_8b-chat', 'trust_remote_code': True}
2024-10-01:12:40:20,544 INFO     [api_models.py:108] Using max length 2048 - 1
2024-10-01:12:40:20,544 INFO     [api_models.py:111] Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2024-10-01:12:40:20,544 INFO     [api_models.py:121] Using tokenizer huggingface
The repository for internlm/internlm2_5-1_8b-chat contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/internlm/internlm2_5-1_8b-chat.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] Traceback (most recent call last):
  File "/scratch/slurm_tmpdir/job_1472282/venv-cot-eval/lib64/python3.11/site-packages/transformers/dynamic_module_utils.py", line 651, in resolve_trust_remote_code
    answer = input(
             ^^^^^^
EOFError: EOF when reading a line
```

The problem seems to be this code here:

https://github.com/EleutherAI/lm-evaluation-harness/blob/15ffb0dafa9c869c7436ba9a3cf3067ac4c9d846/lm_eval/models/api_models.py#L130-L132

Do we need to pass trust_remote_code=True, provided that has been set?


Hi! thanks for catching this. The PR should allow you to pass `trust_remote_code=True` in the model_args
Repository: EleutherAI/lm-evaluation-harness
Issue: regex filter strips whitespace
Root cause: Algorithmic Error

Hello, thank you for maintaining this project! Super valuable. Wanted to make one small issue.

The "regex" filter removes whitespace from the response, which is unnecessary. It should be up to the user to provide a regex that includes or removes whitespace. Alternatively, if the user wants to remove whitespace, they can just use the "remove_whitespace" filter on top of the "regex" filter, but as it stands, it's impossible to use the regex filter and maintain the leading whitespace, which is important for code generation tasks like humaneval, which, because it's Python, needs the leading whitespace.

https://github.com/EleutherAI/lm-evaluation-harness/blob/15ffb0dafa9c869c7436ba9a3cf3067ac4c9d846/lm_eval/filters/extraction.py#L41

@thnkinbtfly @haileyschoelkopf -- tagging you both because you're on the blame for this line
Repository: EleutherAI/lm-evaluation-harness
Issue: [QUESTION] Setting the number of data samples
Root cause: Documentation Deficiency

Hi, I found that some datasets require long evaluation times, even for sub-groups/tasks.
Is there an option for setting the maximum number of test samples for evaluation?

Thank you for sharing the great work by the way :)

Found the "limit" argument, which I think is what I was looking for
Repository: EleutherAI/lm-evaluation-harness
Issue: Error upon running coqa
Root cause: Unimplemented Feature Gap

While running coqa task, i faced an error as follows

![image](https://user-images.githubusercontent.com/42966248/147260572-9bb39dab-c39a-4633-8011-63c140afe68c.png)

If I modify `primary_until, _` into `primary_until` this generates another issue 

![image](https://user-images.githubusercontent.com/42966248/147260853-4187ac94-fd2f-4b7b-aa80-e8a3d6076997.png)

Any solutions ?
Thanks

I hit this recently as well. I believe that the CoQA dataset only works with GPT-3, not with GPT-2. The problem is that:

**1. CoQA uses a multi-token EOS marker. In coqa.py:**
`rf.greedy_until(ctx, ['\nQ:'])`

**2. The harness only supports `greedy_until(..)` with a single token. All other callers only provide a single token, like '.' or a newline.**
```
    def greedy_until(self, requests):
        # TODO: implement fully general `until` that handles untils that are 
        #       multiple tokens or that span multiple tokens correctly
```
If you pass in a multi-token string, you get "ValueError: too many values to unpack (expected 1)". That's because only a single token is expected by the implementation.

**3. GPT-3 uses a different implementation of `greedy_until(..)` that calls into the OpenAI APIs, and that seems to support multi-token `greedy_until(..)`**

To confirm this, you can modify the call to `rf.greedy_until(ctx, ['\nQ:'])` in coqa.py to only pass in a single token, which will avoid the crashes, but presumably won't do the right thing as far as the benchmark is concerned.

So, to get CoQA working on GPT-2, a multi-token greedy_until will need to be implemented in `base.py`.
Repository: EleutherAI/lm-evaluation-harness
Issue: how to reduce verbosity text in versin 0.4.3

Hello, I’m using a forked version of `lm_eval` from Hugging Face, but the verbose output is too long.  Is there a way to reduce the length of the verbose text?
` lm_eval_model = HFLM(model,
                             device=device,
                             tokenizer=tokenizer,
                             batch_size=1,
                             dtype=torch.bfloat16,
                             verbosity=40,
                             )

        task_manager = lm_eval.tasks.TaskManager()`
 this is  how i instantiated my model

thank you
Repository: EleutherAI/lm-evaluation-harness
Issue: Gemma2 BOS token for likelihood_rolling
Root cause: Unimplemented Feature Gap

Hi,

creating an issue to track progress on optional BOS prefix in all windows for llh_rolling methods. This is useful for models such as Gemma2 family.

Based on original issue reported in discord
>  I investigated some strange numbers on language modeling tasks (evaluated with ppl), and I assume BOS is added just once  (not in subsequent frames of loglikelihood_rolling, when document is windowized). Is this the case? And shouldn't BOS be added for gemma everytime? 

> I stopped model call randomly in debugger and verified its inputs to see, that BOS is actually not there in llh_rolling everytime
Repository: EleutherAI/lm-evaluation-harness
Issue: Tasks not found when using vllm
Root cause: Validation Gap

I'm trying to use vllm and evaluate some leaderboard tasks but I get this error:

` ValueError: Tasks not found: leaderboard_mmlu_pro, leaderboard_bbh, leaderboard_gpqa, leaderboard_ifeval, leaderboard_musr, leaderboard_math_hard. Try `lm-eval --tasks list` for list of available tasks, or '--verbosity DEBUG' to troubleshoot task registration issues.`

Any idea why this is happening?

Hi! what is the exact command you are using. I cannot reproduce this

Hello,

This is the command:

`lm_eval --model vllm \
    --model_args pretrained={model_name},tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.8 \
    --tasks leaderboard_mmlu_pro, leaderboard_bbh, leaderboard_gpqa, leaderboard_ifeval, leaderboard_musr, leaderboard_math_hard \
    --batch_size auto`
Repository: EleutherAI/lm-evaluation-harness
Issue: How can I run the mmlu task in offline mode?
Root cause: Configuration Error

In the #1223, there are solution for offline mode.
So, I try run mmlu task using below yaml setting.

However, in the case of mmlu, it is difficult to transfer data_files to data_kwargs because it is mapped to group configuration.

Usually, datasets are imported in the following way.

`for name in ['all', 'abstract_algebra', 'anatomy', 'astronomy', 'business_ethics', 'clinical_knowledge', 'college_biology', 'college_chemistry', 'college_computer_science', 'college_mathematics', 'college_medicine', 'college_physics ', 'computer_security', 'conceptual_physics', 'econometrics',  ... ]:
    dataset = load_dataset("hails/mmlu_no_train", f'{name}')
    dataset.save_to_disk(f"dataset/mmlu/{name}")`

So, is there any way to load this save_to_disk file into load_dataset? I would like to import it from harness as is without going through hf_hub, but errors always occur and difficulties arise.



+1

Indeed when having : 
```
export HF_HUB_OFFLINE=1
```
MMLU gives : 
```
huggingface_hub.errors.OfflineModeIsEnabled: Cannot reach https://huggingface.co/api/datasets/cais/mmlu/revision/main: offline mode is enabled. To disable it, please unset the `HF_HUB_OFFLINE` environment variable.
````
doesn't append with other tasks

But the all point is to make it run without internet

same query