zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: local-completion: Passing Messages as List to Prompt Instead of Required String Format
Root cause: Interface Contract Mismatch

We've encountered an issue while using a local endpoint deployed by Text Generation Inference (TGI) and the vLLM API. When evaluating using the **local-completion** format, we receive an error indicating that the input cannot be parsed into the required string format. The root cause of the problem is that the TGI and vLLM APIs expect the input prompt to be in string format, but currently, the messages are being passed in a list format.
You can find the specific line causing the issue [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/7ad7c5b9d0f1c35c048af0ce8b197ebc2021dbd3/lm_eval/models/openai_completions.py#L47).

To resolve this issue, we have found that applying a chat template with a tokenizer and converting it to a string before passing it as a prompt works effectively. We would like to know if there is already a solution for this issue, or if we should proceed by implementing this fix and submitting a pull request.

If you have any suggestions for alternative solutions or feedback on our proposed approach, please let us know.


Can you provide the full command? the requests should be [passed](https://github.com/EleutherAI/lm-evaluation-harness/blob/7ad7c5b9d0f1c35c048af0ce8b197ebc2021dbd3/lm_eval/models/api_models.py#L183) as a string if bsz = 1 and `tokenized_requests=False`. And vllm's completion endpoint supports both `list[str, ...]` or tokenized`list[list[int], ...]` inputs (or at least it used to).
#2174 fixes a bug where the string was processed incorrectly, but it should still have gone through.
Repository: EleutherAI/lm-evaluation-harness
Issue: Test BBH using local dataset
Root cause: Documentation Deficiency

https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#using-local-datasets
From the link above it shows how to use local datasets but for BBH, I download all the subsets and organize them into one folder. I set dataset_path="arrow" but I am not sure how to set local path because the subset is in another yaml file. Could any one give some hints? Thanks

The BBH dataset is from [lukaemon/bbh](https://huggingface.co/datasets/lukaemon/bbh), which contains 27 subsets. I think a easier way is to split those subsets into individual arrow files and load them independently in each subtask yaml. For example: delete [this line](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/bbh/cot_zeroshot/_cot_zeroshot_template_yaml#L1) in the template then change [this line](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/bbh/cot_zeroshot/boolean_expressions.yaml#L1)  `"dataset_name": "boolean_expressions"` to 
```
dataset_path: arrow
dataset_kwargs:
  data_files:
    train: /path/to/arrow/boolean_expressions.arrow
```

Thanks! I have a try but fails. Do I need to revise the lm-evaluation-harness/lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml file? I think the file should be revised but I am not sure how to revise.

Can you tell me more about what did you do? what does your yaml file look like and what is the error message? 


I made a mistake on the other part and it works now. Thanks for your help!
Repository: EleutherAI/lm-evaluation-harness
Issue: call model instead of forward
Root cause: Documentation Deficiency

You will eventually hit this: https://github.com/pytorch/pytorch/issues/114299#issuecomment-2266352419

Most of your users are not on FSDP2 yet, so it will only become a problem later.

yea, sorry about this

I guess calling `model(...)` instead of `model.forward(...)` is generally a good practice in PyTorch nowadays exactly due to `model.forward(...)` not running hooks

I looked through a few models inside this repo, and they already use self.model(). It's only GPT-NeoX, in https://github.com/EleutherAI/gpt-neox/blob/1cee5b7c7074302de4867ad5cac3f1ea26f7a7d7/eval_tasks/eval_adapter.py#L42, which stores the forward function. That's Quentin's issue, so I'm migrating this issue there.

Actually, looking at https://github.com/EleutherAI/gpt-neox/blob/main/eval_tasks/eval_adapter.py, it would take a lot of work to make it FSDP2-compatible, and the model call is the smallest of the issues. Since the adapter code is 7 months old, I assume it's broken by now and nobody uses it, so it's not worth filing a bug on.

@ad8e thanks for filing an issue! is there some context here I'm missing? We indeed should be safe here I think, since we don't use `model.forward()` anywhere in the Harness that I'm aware of. Maybe I'm not up-to-date on GPT-NeoX development, but I don't believe it would be planned to add FSDP2 support there unless DeepSpeed were to do so first

lm-eval is safe. The context is that external (non-HF) models have to go through an adapter, which nowadays must be pulled from a project like GPT-NeoX because it's no longer practical to write the adapter from scratch. But issues with these adapters are all out-of-scope for lm-eval.
Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError: argument 'ids': 'NoneType' object cannot be converted to 'Sequence'
Root cause: Validation Gap

I face this error whenever I trying to evaluate a model with a benchmark type generation or (Open Question)  like triviaqa, nq_open and truthfulqa

Is there anyone face this problem ?

Without more information we can't help you solve this issue.

What command are you running when you receive this error? What model type and specific model are being used for evaluation? What are any other relevant system details?

I tried this command in the picture to evaluate model: huawei-noah/TinyBERT_General_4L_312D
also I tried it with model: Intel/dynamic_tinybert
and I tried it with these tasks: triviaqa, nq_open and truthfulqa

in every time I got this error
 
![image](https://github.com/user-attachments/assets/5c90ea67-b04f-4137-aafd-e0c967ad62e2)


Is there any update?

> I tried this command in the picture to evaluate model: huawei-noah/TinyBERT_General_4L_312D我在图片中尝试了这个命令来评估模型：huawei-noah/TinyBERT_General_4L_312D also I tried it with model: Intel/dynamic_tinybert还尝试了模型：Intel/dynamic_tinybert and I tried it with these tasks: triviaqa, nq_open and truthfulqa并且我用这些任务尝试了：TriviaQA、NQ_open 和 TruthfulQA
> 
> in every time I got this error每次都遇到了这个错误
> 
> ![image](https://github.com/user-attachments/assets/5c90ea67-b04f-4137-aafd-e0c967ad62e2)

same

same (for PleIAs models (https://huggingface.co/collections/PleIAs/common-models-674cd0667951ab7c4ef84cc4))
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for multi-turn conversation benchmark
Root cause: Unimplemented Feature Gap

Would harness plan to support multi-turn conversation benchmark such as MT-bench?

Hi! We likely will not support general multi-LM call evals, because the complex case would need e.g. asynchronous building and dispatching of requests and would require significant internal reworking to do efficiently. There has, however, been interest in LLM-as-a-Judge which we'll be considering adding as a special case.

This might change depending on user interest and if maintainer bandwidth freed up at all, though I think in general it might be preferable to consolidate the library around supporting static evals and e.g. first expanding to improve long-context eval support and code generation task support. 

One library you might find useful for more complex multi-call control flow is Inspect AI, which has been designed around this use case. https://github.com/UKGovernmentBEIS/inspect_ai




Thank you for your feedback! While multi-LM integration may not be my main focus, multi-turn conversation is a pressing need for us.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add long context evaluation benchmarks such as LongBench and LEval.
Root cause: Unimplemented Feature Gap

Add long context evaluation benchmarks such as LongBench and LEval.

We'd very much like to support a broader range of long-context eval tasks that can be implemented in our framework! This is something we would like to get to eventually ourselves--if contributors are interested, please hop in the EleutherAI discord in the **#lm-thunderdome** channel, where **there is a sub-thread of that channel for discussing long-context eval tasks**.

Hi @txchen-USTC, did you make any progress on this?
Repository: EleutherAI/lm-evaluation-harness
Issue: Better Arg Typechecking / Validation
Root cause: Validation Gap

A common error is for a `--model_arg` value that is meant to be a string to be cast/passed as an int, or for an arg that should be an int or float to stay a string.

This requires manual casting + handling of various model kwargs for edge cases. It would be ideal if we could more extensibly handle all these cases so that a user doesn't have to worry about model kwargs being the expected types. Not hugely high priority but would be quite nice to have.

#2167 is one instance of this. 

how do you feel about pydantic? obviously much less flexible, but should be a way to partially validate some fields

I've not worked with Pydantic before, embarassingly. (I know Inspect AI uses it pretty heavily if I'm recalling correctly?) Will check it out and follow up
Repository: EleutherAI/lm-evaluation-harness
Issue: OOM for gpqa on A100
Root cause: Environment Incompatibility

I met OOM when running the gpqa on A100.
If I use "--apply_chat_template", it works. However, if I delete this command, the OOM error will occur.

Command: `lm_eval --model hf --model_args pretrained=mistralai/Mistral-7B-v0.1,peft=./runs_1e-3/final_checkpoint,trust_remote_code=True,dtype=bfloat16 --tasks leaderboard_gpqa --num_fewshot 0 --batch_size 6 --apply_chat_template`

I have even tried to make batch_size to 1 and the problem continues.

Hi! This seems like really surprising behavior (chat template should only make inputs longer...)

Does this OOM occur even when not using a PEFT? If so I can try to repro.

Hey Haliey,

Without PEFT, it would work.

I think it is mainly because of the issue of memory not retrived when stoping the job. I stop the AWS instance and rerun it again and it now works.



Thanks, glad it's now working for you!

why dtype=bfloat16, not torch_dtype=torch.bfloat16 or dtype=torch.bfloat16 ?
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding AdvGLUE as an evaluation
Root cause: Unimplemented Feature Gap

We propose adding AdvGLUE. [AdvGLUE](https://adversarialglue.github.io.) is a benchmark that provides implementations of various adversarial attacks and defenses for natural language processing models, specifically designed for the GLUE benchmark tasks.

Would this be of interest? If so, we will add in an upcoming PR that either I or my co-collaborator (Steven) will submit.
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding RMS Calibration Error as a Metric
Root cause: Unimplemented Feature Gap

We propose adding Root Mean Squared (RMS) Calibration Error as a metric, as well as temp-tuned Brier Score and temp-tuned RMS Calibration Error.

Would this be of interest? If so, we will add in an upcoming PR that either I or my co-collaborator (Steven) will submit.
Repository: EleutherAI/lm-evaluation-harness
Issue: CoQA evaluation
Root cause: Algorithmic Error

In the original paper of CoQA (https://arxiv.org/abs/1808.07042), the f1 score is averaged across $n$ sets of $n-1$ answers to make it fair when comparing the model performance and human performance (given n human answers, each human answer can only be compared with $n-1$ answers). 
However, when evaluating any LLM, should we use the max f1 score from $n$ f1 scores (without partitioning the $n$ answers to $n$ sets)? The reason being we are not comparing with human performance as the author did in the paper, we are just evaluating the capability of LLM to predict any valid answer.

Original code from lm_eval/tasks/coqa/utils.py:
```
def compute_scores(gold_list, pred):
    # tests for exact match and on the normalised answer (compute_exact)
    # test for overlap (compute_f1)
    f1_sum = 0.0
    em_sum = 0.0
    if len(gold_list) > 1:
        for i in range(len(gold_list)):
            gold_answers = gold_list[0:i] + gold_list[i + 1 :]
            # predictions compared against (n) golds and take maximum
            em_sum += max(squad_metrics.compute_exact(a, pred) for a in gold_answers)
            f1_sum += max(squad_metrics.compute_f1(a, pred) for a in gold_answers)
    else:
        em_sum += max(squad_metrics.compute_exact(a, pred) for a in gold_list)
        f1_sum += max(squad_metrics.compute_f1(a, pred) for a in gold_list)

    return {
        "em": em_sum / max(1, len(gold_list)),
        "f1": f1_sum / max(1, len(gold_list)),
    }
```

Proposed em/f1 computation:
```
def compute_scores(gold_list, pred):
    em = max(squad_metrics.compute_exact(a, pred) for a in gold_list)
    f1 = max(squad_metrics.compute_f1(a, pred) for a in gold_list)

    return {
        "em": em,
        "f1": f1,
    }
```
Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate very large models (>= 70b) ?
Root cause: Documentation Deficiency

Hi, 
I would like to evaluate Llama-3.1-405B model (https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct) on 8xH100s with `lm-evaluation-harness` and `hf` backend. My command looks as follows:
```bash
lm_eval \
  --model hf \
  --model_args pretrained="meta-llama/Meta-Llama-3.1-405B-Instruct",dtype=auto,parallelize=True \
  --tasks gsm8k \
  --num_fewshot 5 \
  --batch_size auto
```
but it seems like this is going to take ages to finish (~5% progress after a day or so). By looking at `nvidia-smi`, I have noticed that only GPU-0 is being utilized, and other GPUs seem to be at 0%. 

Am I running this command correctly, and is there something that I might have missed?

(For example, running evals for the same config with `vllm` backend takes ~3 hours)

Hi! I am curious how you can load the `meta-llama/Meta-Llama-3.1-405B-Instruct` without running into OOM issue. Are you using any quantization, like fp8? It seems that 405B requires at least 14 x H100 80G to run with bf16. 

Hi! 

We support a few variants of parallelism/sharding with HF (see our main README for more details), but ultimately the best recommendation will be to use `vllm` or to use multiple machines with some hosted inference server like VLLM's OpenAI-like server (https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) and then use `local-completions` or `local-chat-completions`. 

*Using HF with parallelize=True and a *massive* model you can't load under any conditions will end up trying to use CPU or disk offloading to make things run, but that's going to be very slow.* Launching using `accelerate launch --num_processes=8`, along with both `parallelize=True` and some form of `load_in_4bit` or `load_in_8bit` in the model args might allow you to fit Llama 3 405B on 8x H100s, but HF only implements "naive" pipelining (only 1 GPU actually performing work at a time) and so I highly recommend trying e.g. VLLM with fp8 to use this model.

Hope this helps!
Repository: EleutherAI/lm-evaluation-harness
Issue: Feature request: `4.4.0` Pypi release with `leaderboard`
Root cause: Unimplemented Feature Gap

Hi,
The current lm-eval on Pypi seems to not contain `leaderboard` tasks, one need to build lm-eval from source to run lm-eval with the CLI with leaderboard evals
```bash
pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git
```
Thank you in advance!

Hi! Planning to put out a new version release soon!

Thank you !
Repository: EleutherAI/lm-evaluation-harness
Issue: mmlu_pro fewshot_config
Root cause: Configuration Error

I'm afraid that fewshot_config.doc_to_text is not used when generating fewshot_ctx for mmlu_pro tasks.
```
fewshot_config:
  sampler: first_n
  doc_to_text: !function utils.fewshot_to_text
  doc_to_target: ""
```

Instead, I think we can map doc["cot_content"] to doc["answer"] in process_docs:
```
def doc_to_text(example):
    prompt = "Question:\n"
    question = example["question"]
    options = example["options"]
    prompt += question + "\n"
    prompt += "Options:\n"
    for i, opt in enumerate(options):
        prompt += "{}. {}\n".format(choices[i], opt)
    prompt += "Answer: Let's think step by step."
    return prompt

def cot_as_answer(example):
    if example['cot_content'] != "":
        example['answer'] = example['cot_content'].replace("A: Let's think step by step. ","")
    return example

def process_docs(dataset, subject):
    dataset = dataset.filter(lambda x: x["category"] == subject)
    return dataset.map(cot_as_answer)
```

This way we do not need a separate doc_to_text in fewshot_config and doc_to_target will take care of the answers in fewshot_ctx

closed by #2237
Repository: EleutherAI/lm-evaluation-harness
Issue: AttributeError: module 'lm_eval.tasks' has no attribute 'ALL_TASKS'
Root cause: External Dependency Breakage

```Traceback (most recent call last):
  File "/mnt/sharedata/hdd/zhouxn/Pruner-Zero/main.py", line 134, in <module>
    main()
  File "/mnt/sharedata/hdd/zhouxn/Pruner-Zero/main.py", line 116, in main
    results = eval_zero_shot(args.model, model, tokenizer, task_list, num_shot, accelerate)
  File "/mnt/sharedata/hdd/zhouxn/Pruner-Zero/lib/eval.py", line 141, in eval_zero_shot
    task_names = pattern_match(task_list, tasks.ALL_TASKS)
AttributeError: module 'lm_eval.tasks' has no attribute 'ALL_TASKS'
```
My origin code is :
```
def eval_zero_shot(model_name, model, tokenizer, task_list=["boolq","rte","hellaswag","winogrande","arc_challenge","arc_easy","openbookqa"], 
        num_fewshot=0, use_accelerate=True, add_special_tokens=False):
    from lm_eval import tasks, evaluator 
    def pattern_match(patterns, source_list):
        task_names = set()
        for pattern in patterns:
            for matching in fnmatch.filter(source_list, pattern):
                task_names.add(matching)
        return list(task_names)
    task_names = pattern_match(task_list, tasks.ALL_TASKS)
    model_args = f"pretrained={model_name},cache_dir=./llm_weights"
    limit = None 
    if "70b" in model_name or "65b" in model_name:
        limit = 2000
    if use_accelerate:
        model_args = f"pretrained={model_name},cache_dir=./llm_weights,use_accelerate=True"
    results = evaluator.simple_evaluate(
        model="hf-causal-experimental",
        model_args=model_args,
        tasks=task_names,
        num_fewshot=num_fewshot,
        batch_size=None,
        device=None,
        no_cache=True,
        limit=limit,
        description_dict={},
        decontamination_ngrams_path=None,
        check_integrity=False,
        pretrained_model=model,
        tokenizer=tokenizer, 
        add_special_tokens=add_special_tokens
    )

    return results 
```
followed [https://github.com/pprp/Pruner-Zero/blob/main/lib/eval.py](url)
I install lm_eval via 
```
pip install lm_eval
```
I would like to know how to modify it so that I can conduct zero shot evaluation?

Best regards

How I solved this problem was

 from lm_eval.tasks import TaskManager

tm = TaskManager()
tasks = tm.all_task  instead of tasks.ALL_TASKS

Hi @Shinning-Zhou !

It looks like the project you're using is based off of a significantly older lm-eval version--installing using the `master` branch from this repo should work with no problems!
Repository: EleutherAI/lm-evaluation-harness
Issue: AttributeError: module 'torch' has no attribute 'uint16'
Root cause: External Dependency Breakage

I met
```
Traceback (most recent call last):
  File "/home/xuanning/.conda/envs/prune/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/home/xuanning/project/LLMPruning/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xuanning/project/LLMPruning/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/xuanning/project/LLMPruning/lm-evaluation-harness/lm_eval/evaluator.py", line 198, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xuanning/project/LLMPruning/lm-evaluation-harness/lm_eval/api/model.py", line 147, in create_from_arg_string
    return cls(**args, **args2)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/xuanning/project/LLMPruning/lm-evaluation-harness/lm_eval/models/optimum_lm.py", line 38, in __init__
    super().__init__(
  File "/home/xuanning/project/LLMPruning/lm-evaluation-harness/lm_eval/models/huggingface.py", line 183, in __init__
    self._create_model(
  File "/home/xuanning/project/LLMPruning/lm-evaluation-harness/lm_eval/models/optimum_lm.py", line 57, in _create_model
    from optimum.intel.openvino import OVModelForCausalLM
  File "/home/xuanning/.conda/envs/prune/lib/python3.12/site-packages/optimum/intel/openvino/__init__.py", line 19, in <module>
    from .utils import (
  File "/home/xuanning/.conda/envs/prune/lib/python3.12/site-packages/optimum/intel/openvino/utils.py", line 76, in <module>
    "u16": torch.uint16,
           ^^^^^^^^^^^^
  File "/home/xuanning/.conda/envs/prune/lib/python3.12/site-packages/torch/__init__.py", line 1932, in __getattr__
    raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
AttributeError: module 'torch' has no attribute 'uint16'
```
when runing 
```
lm_eval --model openvino \
    --model_args pretrained=/data/xuanning/llmpruning/deepseek-coder-7b-base-v1.5_wanda_unstructured_0.5,revision=step100000 \
    --tasks gsm8k \
    --device cuda:0 \
    --num_fewshot 0 \
    --output_path ~/project/LLMPruning/evaluate/ \
    --batch_size 16
```

my library version is python==3.12 torch==2.2.0 lm_eval==0.4.2 optimum==1.21.3
I want to know whicih version should I choose?

@Shinning-Zhou I think for this torch needs to be `pip install torch>=2.3.1`

Hi @Shinning-Zhou , did upgrading torch solve your issue?



yes it is, thanks for your reply.
Repository: EleutherAI/lm-evaluation-harness
Issue: Question about IFEval on LeaderBoard
Root cause: Documentation Deficiency

There are two metrics for IFEval, inst_level_strict_acc and prompt_level_strict_acc, but the leaderboard only shows one number, how to get that number based on these two metrics?

Hi! The number is the average of  `Strict Accuracy at Instance and Prompt Levels (inst_level_strict_acc,none and prompt_level_strict_acc,none)` as stated in this [about page](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about#task-evaluations-and-parameters)

I notice in the raw Hugging Face submission results, there are four metrics for each IFEval question; for example:

```
                 metric                score
prompt_level_strict_acc                False
  inst_level_strict_acc  [False, True, True]
 prompt_level_loose_acc                False
   inst_level_loose_acc  [False, True, True]
```

@wukaixingxp from what you are saying, it sounds like the `*_strict_*` metrics are used for calculation -- is that the case? If so, how is `inst_level_strict_acc` reduced to a single value?
Repository: EleutherAI/lm-evaluation-harness
Issue: [rank1]: huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url:
Root cause: Architectural Constraint

When using lm-evaluation-harness in DDP setting with accelerate launch command, all ranks try to the load the dataset 
https://github.com/EleutherAI/lm-evaluation-harness/blob/0571eeb14d4e48aac51956a726c62cd8b382b3d8/lm_eval/api/task.py#L278 at the same time, however, if the dataset is not in the HF datasets cache dir, all the ranks make requests to download the dataset at once, which is signalling a spamming activity with HF dataset hub making it reject all the requests.

Can we complete loading of the dataset in rank 0 first and then signal other ranks to load as well?