zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: What is the output_type in the metric for?
Root cause: Unimplemented Feature Gap

Every metric has an `output_type`. I can't find the place where it is accessed.
```python
@register_metric(
    metric="acc",
    higher_is_better=True,
    output_type=["loglikelihood", "multiple_choice"],
    aggregation="mean",
)
```
The register metric decorator does not access this from the args.
```python
def register_metric(**args):
    # TODO: do we want to enforce a certain interface to registered metrics?
    def decorate(fn):
        assert "metric" in args
        name = args["metric"]

        for key, registry in [
            ("metric", METRIC_REGISTRY),
            ("higher_is_better", HIGHER_IS_BETTER_REGISTRY),
            ("aggregation", METRIC_AGGREGATION_REGISTRY),
        ]:
            if key in args:
                value = args[key]
                assert (
                    value not in registry
                ), f"{key} named '{value}' conflicts with existing registered {key}!"

                if key == "metric":
                    registry[name] = fn
                elif key == "aggregation":
                    registry[name] = AGGREGATION_REGISTRY[value]
                else:
                    registry[name] = value

        return fn

    return decorate
```

It's currently unused. We're planning to reconfigure the metrics.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add a way to instantiate from HF.AutoModel (again)
Root cause: Documentation Deficiency

Hi,

I was looking to run `simple_evaluate` with my own `transformers` model and found #521 / #601, but it looks like they're only merged to `master`. Was this feature lost in the 0.4.0 migration? Should I port it?

Hi! This should be supported via the HF model type in `main` as well -- https://github.com/EleutherAI/lm-evaluation-harness/blob/f257d38b66f00f544aab5817fe66eb9fe78a8b0f/lm_eval/models/huggingface.py#L81 and we have people who use this feature. Does using this work for you?

Ohh, thank you, I was confused by the API there.

For anyone who was confused like me, it's

```python
model = transformers.AutoModelForCausalLM.from_pretrained(...)
lm_eval.simple_evaluate(
    model=lm_eval.models.huggingface.HFLM(model)
)
```

And here's what you would do to run inference with unsloth:

```python
from unsloth import FastLanguageModel
import lm_eval

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3-8b-Instruct-bnb-4bit",
    max_seq_length = 8192,
    load_in_4bit = True,
)

results = lm_eval.simple_evaluate(
    model=lm_eval.models.huggingface.HFLM(
        pretrained=model,
        tokenizer=tokenizer,
    ),
)
```

@dmitrii-palisaderesearch - have you run the script via accelerate ?

Nope, the above was plain Colab. I believe you can leverage accelerate by passing `FastLanguageModel.from_pretrained(device_map=XXX)` though.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to enable trust_remote_code when encountered programmatically via get_task_dict?
Root cause: Configuration Error

              Hi ! FYI the `datasets` release that requires explicit `trust_remote_code=True` for datasets with a python loading script will be out today. Feel free to ping me if you encounter any issue with your CI or from users !

_Originally posted by @lhoestq in https://github.com/EleutherAI/lm-evaluation-harness/issues/1135#issuecomment-2165689312_

Hi folks, prior to the change we were able to run the following without setting the flag.
Q: What is the suggested way of setting the "trust_remote_code" flag when programmatically calling into from lm_eval code? (specifically get_task_dict)

```
from eval import get_task_dict
task  = get_task_dict(["wikitext"])
```

This now fails with the following

```
The repository for EleutherAI/wikitext_document_level contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/EleutherAI/wikitext_document_level.
  Please pass the argument `trust_remote_code=True` to allow custom code to be run.
```

            

the same issue.  `export HF_DATASETS_TRUST_REMOTE_CODE=1` can improve.


Hi, have you tried updating `lm-evaluation-harness` ? This issue has been taken care in a recent version of `EleutherAI/lm-evaluation-harness/lm_eval/tasks/wikitext/wikitext.yaml` already afaik

Thanks that resolves it
Repository: EleutherAI/lm-evaluation-harness
Issue: Long time testing Qwen2-72B
Root cause: Configuration Error

When testing Qwen/Qwen2-72B `parallelize=True,max_length=4096` on `generate_until` task I am getting warnings:
```
Both `max_new_tokens` (=2048) and `max_length`(=882) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
```
Number after `max_length` is getting lower in time.

The problem is the test is taking few times more time than e.g. using llama3-70B. In the task there is `max_gen_toks: 50`.

Is the long time related to the warning above?

I see that this model has `"max_new_tokens": 2048` in generation_config.json, but llama3 does not have.

There have been a couple reports of odd scores or behavior when evaluating Qwen2 models. I hope to check this out soonish, if others aren't able to.

Looks like `max_new_tokens=2048` is taking precedence over `max_tokens`(len(inputs) + max_gen_toks)
Repository: EleutherAI/lm-evaluation-harness
Issue: `piqa` task need add trust_remote_code true in piqa.yml
Root cause: Configuration Error

when I run `piqa` tasks with datasets 2.20.0 and  enable `trust_remote_code`, 
```
lm-eval --model hf --model_args pretrained=facebook/opt-125m,trust_remote_code=True --tasks piqa --device cpu --batch_size 8 --limit 24
```
the warning raised
```
(Pdb) c
The repository for piqa contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/piqa.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] 
```

I found the similar issue has been fixed by #1487 , I saw in this PR the author has listed the trust_remote_code needed tasks https://gist.github.com/veekaybee/9fa165f5f41fc57d6e2ccd8c7644ca5d, `piqa` works before but failed now, 

inspired from https://github.com/EleutherAI/lm-evaluation-harness/pull/1487 and #1135 , I fixed the issue by PR #1983 , Could you have verify and review it? @veekaybee
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the SQuAD evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive, multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset.
> GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI+18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions. On DROP [DWD+19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL+19]. On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper. On RACE [LXL+17], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.

- [X] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Note: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/squad_v2

I'll take this.

Similar to RACE #21 , HuggingFace splits the questions into one-per-passage, as opposed to the multi-question-per-passage setup from the paper.
Repository: EleutherAI/lm-evaluation-harness
Issue: Incorrect Multilingual arc implementation
Root cause: Algorithmic Error

The okapi translated arc uses preprocessing for no reason (likely oversight as the preprocessing if for hellaswag). I am not sure if it destroys some instructions, but I don't think there is any preprocessing needed.

Line in question:
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/okapi/arc_multilingual/utils.py#L6
Repository: EleutherAI/lm-evaluation-harness
Issue: Implementing lessons from OLMES
Root cause: Unimplemented Feature Gap

The OLMES paper is a pretty interesting read and is complementary to LM-Eval. I think there are a few features we can consider implementing based on lessons and recommendations from the paper.

1. **Implement mode-switch between Multiple-choice formulation and Completion/cloze formulation.** Empirically models respond to both version differently depending on how much tokens trained with the former showing a stronger signal in later stages (400B tokens and above). The recommendation is to evaluate on both and take the highest score. Lm-eval would benefit from having the ability to write 1 prompt format and have it automatically be used in both formulations.
2. **Normalization should be a configuration feature.** We use both non-normalized accuracy and normalized accuracy, specifically by dividing the log-probability by the length or characters. It would be great to be able to choose the normalization and add more (normalization like based on token length or pointwise-mutual information)
3. **Further add support for fewshot selection.** We have some support for hardcoding fewshot samples, but we've never really supported a way to make it easier to select fewshots with conditions like "make sure the answer choices are not all A" or set a list of index number that points to the samples directly.

cc: @haileyschoelkopf @StellaAthena
Repository: EleutherAI/lm-evaluation-harness
Issue: mmlu evaluation fail
Root cause: Validation Gap

I try to evaluate model in mmlu, however i get 

`ValueError: No tasks specified, or no tasks found. Please verify the task names.`

for example,

```
lm_eval \
    --model hf \
    --model_args pretrained=microsoft/phi-2,trust_remote_code=True \
    --tasks mmlu \
    --device cuda:0 \
    --batch_size 8 \
    --output_path output/phi-2 \
    --limit 10 \
```

it only works for a subset of mmlu,

```
lm_eval \
    --model hf \
    --model_args pretrained=microsoft/phi-2,trust_remote_code=True \
    --tasks mmlu_abstract_algebra \
    --device cuda:0 \
    --batch_size 8 \
    --output_path output/phi-2 \
    --limit 10 \
```

i think it is a huge issue. could you please to take a look? it does not appear previously.


    
    

Hi @jxiw , what was the problem?

it because i have a fold called 'mmlu' in the current path. it automatically reads config from that folder instead of using the standard mmlu, but that folder is not for a new config.
Repository: EleutherAI/lm-evaluation-harness
Issue: Wrong calculation of score when there are ties?
Root cause: Algorithmic Error

Even though this seems very unlikely, for the multiple choice task the models might return the same scores for some of the options. If these options have the highest score, and the reference label is among these labels, the result will be treated as true positive. 

I think this is not a good strategy, since the model is not sure which of the options is the correct one.

I would suggest changing the algorithm, to treat the result as false negative whenever there's a tie between the top-scoring answers.

I have run some test with Quality dataset (my own definition) and openchat and PHI models. Seems like it's not such an issue I thought it is. For 100 samples there were 13 cases for PHI model with the same values, but only 2 of them were cases when the value was the same for the highest scores.

Why is tiebreaking "incorrect" better than tiebreaking "correct"?
Repository: EleutherAI/lm-evaluation-harness
Issue: Supporting Multimodality
Root cause: Unimplemented Feature Gap

Add support to multimodal models as discussed with @haileyschoelkopf 

- This PR #1832 would be a great starting point.
- List all tasks we want to support for the first iteration.


Do we have an ETA for this? Happy to help contribute in any way I can. For reference we are doing part of our VLM evaluation suite in [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and some of it in some other frameworks and are looking to unify it. 

We're aiming in the next week or 2 to have a prototype API for multimodality w/ HF VLMs in a branch that people can try out! 

Feedback once that is ready, and contributing task implementations, would be very appreciated! We'll update this issue as new updates arise.

Working branches for @lintangsutawika and myself are `{lintang,hailey}-multimodal-prototyping` and "ground truth" prototype will be in `multimodal-prototyping` !

We'll update again when there's something it's useful for others to take a look at--namely HF `AutoModelForVision2Seq` support + MMMU.

Do we have any updates on this one? to support vision lanugage models.  Thanks.
Repository: EleutherAI/lm-evaluation-harness
Issue: Running on custom model, getting 'TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'
Root cause: Validation Gap

Running:
`lm_eval --model <custom model name> --tasks lambada_openai,hellaswag --device cuda:0 --batch_size 8`
I am getting error:

> INFO:lm-eval:Running loglikelihood requests
> Traceback (most recent call last):
>   File "/opt/conda/bin/lm_eval", line 8, in <module>
>     sys.exit(cli_evaluate())
>   File "/home/francois/lm-evaluation-harness/lm_eval/__main__.py", line 375, in cli_evaluate
>     results = evaluator.simple_evaluate(
>   File "/home/francois/lm-evaluation-harness/lm_eval/utils.py", line 395, in _wrapper
>     return fn(*args, **kwargs)
>   File "/home/francois/lm-evaluation-harness/lm_eval/evaluator.py", line 277, in simple_evaluate
>     results = evaluate(
>   File "/home/francois/lm-evaluation-harness/lm_eval/utils.py", line 395, in _wrapper
>     return fn(*args, **kwargs)
>   File "/home/francois/lm-evaluation-harness/lm_eval/evaluator.py", line 477, in evaluate
>     metrics = task.process_results(
>   File "/home/francois/lm-evaluation-harness/lm_eval/api/task.py", line 1333, in process_results
>     **({"acc": int(is_greedy)} if "acc" in use_metric else {}),
> TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'


hello, I am trying to evaluate the customized gpt2 model (adapted from the original model in huggingface.transformers ) on _lambada_openai_ dataset, too.  But I don't know how to implement it with command like

>  lm_eval --model <custom model name> --tasks lambada_openai,hellaswag --device cuda:0 --batch_size 8

and what changes do I need to make to the code in 

> lm-evaluation-harness/lm_eval/

Thank you very much, and wish you have a nice weekend : )
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding Permutation/Masked Language Models
Root cause: Unimplemented Feature Gap

First, it's a great library as a one-stop place for all LM evaluations. In the same vein, is the addition of permutation/masked language models also on the agenda? That would really increase the adoption/usefulness of this library. Thanks

See [here](https://github.com/EleutherAI/lm-evaluation-harness/issues/293)
Repository: EleutherAI/lm-evaluation-harness
Issue: YAML config was updated, but the project still remains the same as before
Root cause: Configuration Error

Hello, I am confused about config yaml, can anyone help me? I change the configuration file yaml for the task, but the evaluation log file was not updated to the new configuration and keep the  former configuration .  I am sure the new yaml was saved. Thanks!

![1719375351919](https://github.com/EleutherAI/lm-evaluation-harness/assets/53637341/5d9165f1-bd64-46d9-84e5-8549e217e90b)
![1719375393038(1)](https://github.com/EleutherAI/lm-evaluation-harness/assets/53637341/5213123d-5e91-4220-ba08-97ff9b4c631d)


Hi, how are you installing the package? in particular, are you installing it in editable form (`pip install -e .` from the git repo top-level)?

Thank you for your prompt reply. I have installed all the packages, although some versions of the libraries do not strictly follow the requirements, I can run tasks such as gsm8k normally. This bug is very strange. Sometimes there is, sometimes there is not, and I cannot reproduce this problem. Let's see if anyone else has encountered the same problem.

What fixed your issue?
Repository: EleutherAI/lm-evaluation-harness
Issue: Duplicate `sample` entries; actually not duplicate but separate entry for each filter
Root cause: Documentation Deficiency

Looking at the samples generated from `gsm8k` and it seems like there is a separate entry for each filter now, except there is no indication of _which_ filter each entry corresponds to.

Hi @baberabb, we just noticed this too for the recent addition of the `metabench_gsm8k*` tasks, which mirror the `gsm8k` task. Is there any update on the cause of the duplication or whether it is intended?

Hi! @kozzy97. So actually not duplicate but a separate entry logged for each filter used. I have been meaning to add an identifier to indicate which entry belongs to which.

edit: oh, have already said that in the OP :0. Will try submitting a PR this week!
Repository: EleutherAI/lm-evaluation-harness
Issue: Test Open LLM Leaderboard 2
Root cause: Documentation Deficiency

I am wondering which task command should I use to test with new leaderboard, including IFEval, BBH, MATH, GPQA, MuSR and MMLU-PRO? Thanks a lot!

From the leaderboard's [about page](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about): "To reproduce our results, here are the commands you can run, our fork of the Eleuther AI Harness (link coming soon!)", I guess that [this is the fork](https://github.com/huggingface/lm-evaluation-harness)?

Yes, "Official" leaderboard task configs will be coming upstream very soon!

Any idea when the tasks would be added and if there are any changes in the existing tasks? Like BBH, GPQA and IFEVAL?

There will be separate canonical configs (which will remain static) for all leaderboard tasks' variants! (some of these might be the same as the existing configs we have for the corresponding dataset, but others may not.)

(cc @clefourrier to make sure I'm not mischaracterizing anything!)

When can we expect the new tasks?

BBH is now framed as multiple-choice, so the task definitely changes. For now, it would be super helpful to have access to `lighteval/MATH-Hard`, which I cannot find the HF hub.

Hi!
As @haileyschoelkopf mentionned (thanks for the ping!), we are working on adding them asap - to run the exact same thing as us, you'll first need to use our fork, and once all PRs are in the harness, you'll be able to use them as is.

@Mugariya iirc, all implementation and tasks change. We constrain which examples are used in few shot and do not resample, we made sure that the end of generation tokens were working well for all models, etc.

@RicardoDominguez Yep, this is the fork! We are checking that all our PRs are correct (we had to extract all individual feature changes as specific PRs) and we'll 1) open them on the harness and 2) merge them in our fork.


Tagging @nathanhb who's wrapping it up :)

Thanks @clefourrier @haileyschoelkopf that's helpful


#2023 tasks are now usable!
Repository: EleutherAI/lm-evaluation-harness
Issue: --log_samples not saving all inference output
Root cause: Validation Gap

 The output file didn't include all inference results. How can I save the all results?
Here is my command:
lm_eval --model vllm \
    --model_args pretrained=/data/Meta-Llama-3-70B-Instruct/,tensor_parallel_size=8,dtype=auto,gpu_memory_utilization=0.8 \
    --tasks boolq\
    --batch_size 1\
    --output_path output/llama3-70b-boolq\
    --log_samples

output:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/42319139/4b2aac79-26eb-47f3-a55c-8a0112778973)


Created a fix for this in #2027 .

the issue: doc 5 of boolq has some non-ascii characters which, when being logged, hit 

```
'ascii' codec can't encode character '\u02c8' in position 103: ordinal not in range(128)
```
this error gets suppressed by [this](https://github.com/EleutherAI/lm-evaluation-harness/blob/be01651cf6b611c09f088c336db74ea625227716/lm_eval/loggers/evaluation_tracker.py#L345) try-except . Using utf-8 again for logged sample files fixes the issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: Using chat template with vllm engine
Root cause: Unimplemented Feature Gap

Is the chat template flag --apply_chat_template not supported for vllm models? 

I tried running the following command and it doesn't work, even though I have the template in my tokenizer config. The same works for hf model though

```
lm-eval --tasks gsm8k --model vllm \
        --model_args pretrained=<path_to_my_model>,tensor_parallel_size=2 \
        --output_path=<output_path> \
        --log_samples --num_fewshot 0 --batch_size auto:N --apply_chat_template
```
Repository: EleutherAI/lm-evaluation-harness
Issue: The problem of generate responses with my own trained model
Root cause: Interface Contract Mismatch

hi, thank you for your wonderful work.

I fintuned a llama3-8b-instruct on GSM8K dataset with LLama-Factory. 
But when I use lm-evaluation-harness to generate the response and get the evaluation score on GSM8K, it came with the following problem:

Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/mnt/public02/usr/yuanpeiwen/lm-evaluation-harness/lm_eval/__main__.py", line 375, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/mnt/public02/usr/yuanpeiwen/lm-evaluation-harness/lm_eval/utils.py", line 395, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/public02/usr/yuanpeiwen/lm-evaluation-harness/lm_eval/evaluator.py", line 329, in simple_evaluate
    add_tokenizer_info(results, lm)  # additional info about tokenizer
  File "/mnt/public02/usr/yuanpeiwen/lm-evaluation-harness/lm_eval/loggers/utils.py", line 118, in add_tokenizer_info
    "tokenizer_pad_token": [lm.tokenizer.pad_token, lm.tokenizer.pad_token_id],
AttributeError: 'function' object has no attribute 'pad_token'

I use python 3.10.6, torch 2.3.1, cu121
and I run the program with the following command：
lm_eval --model hf     
--model_args pretrained=./LLaMA-Factory/llama3-8b-instruct-alpaca-oriGSM8K-1e-5-3epoch/checkpoint-823
--tasks gsm8k
--batch_size auto
--output_path ./eval_out/llama3-8b    
--use_cache ./eval_cache

I hope you can give me some advice, thank you!