zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Missing Tasks in Leaderboard
Root cause: Documentation Deficiency

The `readme.md` page says that the `leaderboard_bbh` group includes the `leaderboard_bbh_dyck_languages` task, but the setting `--tasks "leaderboard"` does not include this task's result.
The following tasks are also missing: `leaderboard_bbh_dyck_languages`, `leaderboard_bbh_multistep_arithmetic_two`, `leaderboard_bbh_word_sorting`, and `leaderboard_math_precalc_hard`.

hi ! Thanks for the catch. we do not include those tasks because they require bbh to be in generative mode and we use loglikelihood when evaluating bbh for the leaderboard. the tasks should be removed from the readme

fixed in #2219 :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the Contract Understanding Atticus Dataset (CUAD)
Root cause: Unimplemented Feature Gap

> Contract Understanding Atticus Dataset (CUAD) v1 is a corpus of 13,000+ labels in 510 commercial legal contracts that have been manually labeled under the supervision of experienced lawyers to identify 41 types of legal clauses that are considered important in contact review in connection with a corporate transaction, including mergers & acquisitions, etc. 
> 
> CUAD is curated and maintained by The Atticus Project, Inc. to support NLP research and development in legal contract review.

Dataset: https://www.atticusprojectai.org/cuad
Paper: https://arxiv.org/abs/2103.06268
Repository: EleutherAI/lm-evaluation-harness
Issue: nltk pickle
Root cause: External Dependency Breakage

I am quoting from nltk repo: The "punkt" package consists in unsafe pickles, so it is deprecated and not used in NLTK 3.8.2. It is replaced by "punkt_tab". Users should definitively avoid the https://github.com/nltk/nltk/issues/3266#issuecomment-2284001819 and upgrade to NLTK 3.8.2.

Ifeval task downloads 'punkt' which seems won't work in nltk 3.8.2. I think we need an update on this.

Hi @bezir , thanks for reporting this! Fixing today after I confirm it does not result in different scores for IFEval.
Repository: EleutherAI/lm-evaluation-harness
Issue: The results in the new and old versions differ from one another.
Root cause: Configuration Error

This screenshot shows the results of Llama2-7B; the results are not the same as what I did before with the previous version. Could someone please explain the reason?
![image](https://github.com/user-attachments/assets/756bd3a3-8139-4682-9524-854213e33b53)
 

Hi, could you give a bit more detail on 1) which codebase versions you are comparing and 2) if any parameters such as batch size changed?

The vast majority of these discrepancies are extremely tiny -- if just the third decimal place changes that's well within the sort of variation we'd expect from minor nondeterminacies and so isn't a bug per se.

For MMLU, #2116 adjusted the whitespace slightly so that the prepended task description for MMLU now has 2, not 1, newline in between the questions and it, to reflect the intended whitespacing of the MMLU original prompt that was affected due to a bug on our part. We've incremented the versions of all affected tasks--I highly recommend reporting or keeping track of the task versions when you report results!

Hope this helps.

I compare the old version of the LM harness (without YAML files) and the new LM harness (with YAML files). 

Is the change in "batch size" having an effect on the results? 
What do you think other factors that are affecting the results, especially for locally added tasks that I added to both versions, have different results between the two versions? for the same tasks, matrices, and number of shots.

I see, it was not clear to me what commits "new" and "old" are.

For all but MMLU, differences on that order are often just due to numeric indeterminacies which can be introduced by changing the batch size. I'd compare these differences (on the order of 3rd or 4th decimal point, often) to the `stderr` we report to see how they appear insignificant.

For specifically MMLU, if you ran on `v0.3.0` as opposed to the most recent `master`, then MMLU's standard prompt was changed to now reflect the original paper's "canonical" prompt format: see https://huggingface.co/blog/open-llm-leaderboard-mmlu for a description of what changed and its effect! Hope this helps. If it doesn't help then feel free to reopen this issue and let me know the precise commit hash that "old" refers to.
Repository: EleutherAI/lm-evaluation-harness
Issue: Invalid response for loglikelihood for GGUF model
Root cause: Validation Gap

I have a llama-2-7b.gguf quantized in Q4_K_M that I run with llama.cpp server. The inference works fine. 

This is the code (found from [here](https://github.com/EleutherAI/lm-evaluation-harness/issues/1637#issuecomment-2021783648), who experiences similar issue) I use to do a test eval (hence the `limit` param): 

```
import json
from lm_eval.models.gguf import GGUFLM
from lm_eval import simple_evaluate

lm = GGUFLM(base_url="http://localhost:8080")  
results = simple_evaluate(model=lm,tasks=["french_bench_hellaswag"],device="mps",limit=10) 
filtered_results = results.copy()  
filtered_results = {key: value for key, value in results.items() if key != "samples"}  
json_filtered_results = json.dumps(filtered_results, indent=4)  
with open("results.json", "w") as json_file:
    json_file.write(json_filtered_results)
```

Every time I run this, I get the following error: 

`ERROR    [gguf.py:96] Invalid response for loglikelihood.`

Stack trace: 

```
Traceback (most recent call last):
  File "/Users/jasminelatendresse/exp-os-assistant-redaction/francisation-llm/scripts/eval.py", line 6, in <module>
    results = simple_evaluate(model=lm,tasks=["french_bench_arc_challenge"],device="mps",limit=5) 
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jasminelatendresse/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/jasminelatendresse/lm-evaluation-harness/lm_eval/evaluator.py", line 296, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/Users/jasminelatendresse/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/jasminelatendresse/lm-evaluation-harness/lm_eval/evaluator.py", line 468, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jasminelatendresse/lm-evaluation-harness/lm_eval/models/gguf.py", line 99, in loglikelihood
    assert False
           ^^^^^
AssertionError
```

I have tried with a different task, but I get the same error. Anyone else with this issue? Thank you. 

I used the code suggested by [here](https://github.com/EleutherAI/lm-evaluation-harness/issues/1637#issuecomment-2021783648) in the gguf.py file and it fixed it. However, I feel like this should still be considered a bug since it should run out of the box. 


looking into this here: https://github.com/EleutherAI/lm-evaluation-harness/issues/1472#issuecomment-2301827684_
Repository: EleutherAI/lm-evaluation-harness
Issue: Extremely SLOW, even slower than training
Root cause: Algorithmic Error

```python
def eval(model,peft):
    result = lm_eval.simple_evaluate(
                model="hf",
                model_args=f"pretrained={model},tokenizer={model},peft={peft},dtype=bfloat16,load_in_4bit=True,bnb_4bit_quant_type=nf4,bnb_4bit_compute_dtype=bfloat16,bnb_4bit_use_double_quant=True",
                tasks=["gsm8k"],
                num_fewshot=5,
                task_manager=task_manager,
                batch_size=4)
```

where the model is Mixtral-8x7B

In training, step time = 1.636, in lm_eval, step time = 18.72. What?

I am also experiencing this 

Hi, is this specific to Mixtral and to running with quantization? How fast do non-generative tasks like `mmlu` or `arc_easy` run?

Because freeform generation, as done in GSM8k, requires running the model sequentially repeatedly to generate text, it can take longer than running a single forward + backward pass as might be done during SFT. Hope this helps!

If the slowness persists for other settings, especially non-generative tasks, please let me know and we can look into it further!
Repository: EleutherAI/lm-evaluation-harness
Issue: Classification into categories with generate-until and metrics
Root cause: Documentation Deficiency

Hello, can anyone recommend the best practice for setting up a task for lm-eval-harness with the generate-until type? I need to solve classification into categories, but I can't use multiple-choice because I need to make predictions on closed-source models (like OpenAI). Currently, I'm using a workaround in the prompt so that the model responds with only one word without further explanation, but I'm concerned that this might not work in all cases. I'm using exact-match as the metric. Does anyone have tips on how to solve this problem for most models like LLaMA 3.1, Gemma 2, and OpenAI models?

Hi! Unfortunately when it comes to classification tasks with generative outputs, it's not really possible in (unstructured) generation to force the model to never deviate from your intended output format and set of classes, and parsing code needs to handle locating a model's answer and translating it into a class if that mapping can be performed for a given answer.

Exact match can work, and for more advanced models like OAI ones which can semi-reliably follow a format you instruct them to use, that can work, but this is the primary historical reason that we implement multiple-choice as loglikelihood-based.

We are also in the early stages of considering adding LLM-as-a-Judge for some cases, where an LLM could potentially handle the parsing and conversion of free-form output to answer class.
Repository: EleutherAI/lm-evaluation-harness
Issue: test_version_stable.py does not exist?
Root cause: Validation Gap

Hi, 
in the file lm_eval/evaluator_utils.py in the function "find_test_root" on line 507, it is searching for a test_stable_version.py and it seems like this file is used to run the test, but this file does not exist in the current repository. What other file is the right file to be able to run tests using simple_evaluate? 
Please let me know. 
Thank you! 


I solved this by passing check_integrity=False
Repository: EleutherAI/lm-evaluation-harness
Issue: trust_remote_code error in simple evaluate for hellaswag
Root cause: Configuration Error

Getting a remote code error but doesn't seem to be an argument in simple_evaluate that can set trust_remote_code=True. Anyone facing a similar error? 

Error:
```
2024-08-18:12:13:13,353 INFO     [task.py:411] Building contexts for hellaswag on rank 0...
 29%|██▉       | 2903/10042 [00:01<00:02, 2888.73it/s]
  File "python3.8/site-packages/datasets/load.py", line 102, in _raise_timeout_error
    raise ValueError(
ValueError: Loading this dataset requires you to execute custom code contained in the dataset repository on your local machine. Please set the option `trust_remote_code=True` to permit the loading of this dataset.

```

Script:
 ```
   results1 = simple_evaluate( 
            model=lm_obj,
            tasks=["hellaswag", "winogrande", "arc_easy", "arc_challenge", "piqa"],
            num_fewshot=0,
            batch_size=batch_size,
            cache_requests=None,
            log_samples=False,
            bootstrap_iters=0,
            gen_kwargs="max_new_tokens=40",
        )
```



Fixed it by adding `datasets.config.HF_DATASETS_TRUST_REMOTE_CODE = True` (from https://github.com/EleutherAI/lm-evaluation-harness/pull/1467#issuecomment-1964282427)
Repository: EleutherAI/lm-evaluation-harness
Issue: Fix the zeno_visualizer.py to work with Phi3 Model
Root cause: Algorithmic Error

Hi team,

First, I want to thank you all for the incredible work on the Phi3 model!

While running the evaluation pipeline, I encountered the following error [at this line](https://github.com/EleutherAI/lm-evaluation-harness/blob/cd35aecb9cf0ebb8c7f6c0031d434e9507e3717a/scripts/zeno_visualize.py#L186):

```python
instance = [x["arguments"][0][0] for x in data]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <listcomp>
KeyError: 0
```

After reviewing the output, I realized the code should be:

```python
x["arguments"]["gen_args_0"]["arg_0"]
```
Once I made this change, the pipeline for Zeno ran smoothly.

If this update doesn't interfere with other models, would it be alright if I submitted a PR for it?

Thanks again for your hard work!

Thank you.



Hi! Thanks for the feedback and a PR would be appreciated! The format for the logged outputs was changed recently and looks like this script was overlooked.
Repository: EleutherAI/lm-evaluation-harness
Issue: Why no results for closed-sourced models?

Hi!

Why don't you benchmark closed models? Is there any project that does that?

Best regards

No results? This framework is not providing any result by itself, AFAIK. Though there are multiple projects that use it as a workhorse to benchmark different models. 

For now, there is 2 API provided for closed models in https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/models and there is a template to add more.
There are following PRs waiting to be added (or reviewed or some other decision):
https://github.com/EleutherAI/lm-evaluation-harness/pull/395
https://github.com/EleutherAI/lm-evaluation-harness/pull/834
https://github.com/EleutherAI/lm-evaluation-harness/pull/936
https://github.com/EleutherAI/lm-evaluation-harness/pull/1996
Repository: EleutherAI/lm-evaluation-harness
Issue: Can I incorporate my own benchmark as a new task within lm-evaluation-harness?
Root cause: Documentation Deficiency

I would like to design my own set of benchmarks, but evaluating mainstream models is a challenging task. I want to use lm-evaluation-harness to assist me. Does lm-evaluation-harness support the expansion of tasks with custom benchmarks?

Hi! There is information on how to create your own benchmark [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs), and some existing benchmarks in this [folder](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/benchmarks) and [here](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/leaderboard).

closing the issue now but feel free to open.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement SCAN
Root cause: Unimplemented Feature Gap

[SCAN](https://github.com/brendenlake/SCAN) is a set of simple language-driven navigation tasks for studying compositional learning and zero-shot generalization.

SCAN consists of a set of commands and their corresponding action sequences. These are the actions an agent should perform to execute the commands successfully. The commands and actions are defined compositionally based on primitives ("jump", "walk", "run", "turn left", etc.) and modifiers such as "twice", "thrice", "and", "after", "around left", etc. Here are some examples:

|Command | Action sequence |
| --- | --- | 
| IN: jump                |                       OUT: JUMP |
| IN: jump left            |                       OUT:  LTURN JUMP | 
| IN: jump around right       |                   OUT: RTURN JUMP RTURN JUMP RTURN JUMP RTURN JUMP |
| IN: turn left twice          |                  OUT: LTURN LTURN |
| IN: jump thrice               |                 OUT: JUMP JUMP JUMP | 
| IN: jump opposite left and walk thrice   |      OUT: LTURN LTURN JUMP WALK WALK WALK | 
| IN: jump opposite left after walk around left | OUT: LTURN WALK LTURN WALK LTURN WALK LTURN WALK LTURN LTURN JUMP |
Repository: EleutherAI/lm-evaluation-harness
Issue: OpenAICompletionsAPI does not implement the chat_template function, causing a TypeError when evaluating. 
Root cause: Interface Contract Mismatch

When using lm-eval's OpenAICompletionsAPI for evaluation (e.g. evaluating a checkpoint being served with vLLM), an error occurs because `lm_eval.models.openai_completions.OpenAICompletionsAPI` does not implement `chat_template`, even though this is required by the TemplateAPI interface ([see here](https://github.com/EleutherAI/lm-evaluation-harness/blob/259b756afe66ae39ffbd410a13aa6b475dd6f15d/lm_eval/models/api_models.py#L227).)

The error [occurs here](https://github.com/EleutherAI/lm-evaluation-harness/blob/259b756afe66ae39ffbd410a13aa6b475dd6f15d/lm_eval/evaluator.py#L295), where `simple_evaluate` tries to call `lm.chat_template()` even though `lm.chat_template` is a string. 

As an example, if I run a vLLM instance and try to evaluate my model.
```
export MODEL_CHECKPOINT_PATH="..."

OPENAI_API_KEY="dummy_key" lm_eval \
--model openai-completions \
--tasks mmlu \
--model_args model="$MODEL_CHECKPOINT_PATH",base_url=http://localhost:8000/v1,tokenizer_backend=huggingface,tokenizer="$MODEL_CHECKPOINT_PATH" \
--num_fewshot 0 \
--batch_size 16
```
I get this error:
```
Traceback (most recent call last):
  File "/home/yonatan/miniconda3/envs/lm-eval-2/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/yonatan/workspace/lm-evaluation-harness/lm_eval/__main__.py", line 389, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/yonatan/workspace/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/yonatan/workspace/lm-evaluation-harness/lm_eval/evaluator.py", line 297, in simple_evaluate
    chat_template=lm.chat_template(apply_chat_template),
TypeError: 'str' object is not callable
```

I put a PDB trace in `simple_evaluate` and you can see that `lm.chat_template` is a string:
```
(Pdb) lm
<lm_eval.models.openai_completions.OpenAICompletionsAPI object at 0x153737f82980>
(Pdb) lm.chat_template
''
```

However, if I run lm-eval with an HF model as in
```
lm-eval --model hf --model_args pretrained=gpt2 --tasks gsm8k
```

then I get this:
```
(Pdb) lm
<lm_eval.models.huggingface.HFLM object at 0x150232969930>
(Pdb) lm.chat_template
<bound method HFLM.chat_template of <lm_eval.models.huggingface.HFLM object at 0x150232969930>>
```

Hi! We've hotfixed this in #2250 and will shortly be fixing it in more generality.
Repository: EleutherAI/lm-evaluation-harness
Issue: Allow Prefilling Assistant Response w/ Chat Templates
Root cause: Unimplemented Feature Gap

Models that are open-source and/or used via `local-completions`, as well as Claude, allow one to "prefill" the start of the assistant's response to a given input: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response

We don't currently support this with the current chat templating. We should consider supporting this via a new `doc_to_text_response_prefill` (name TBD...) field in the config file, for portions of input that are post-pended after applying a chat template. 

One downside is that we would have to make models that can't accept such prefilled responses error out or ignore the prefill when evaluating on a task that uses this, and also this would further complicate the construction of contexts. So somewhat a tough decision. But Llama-3 uses this for tasks such as evaluation on MBPP so it's worth considering

They also use it for [MMLU](https://huggingface.co/datasets/meta-llama/Meta-Llama-3.1-8B-Instruct-evals/viewer/Meta-Llama-3.1-8B-Instruct-evals__mmlu__details):
```'gen_prefix': 'The best answer is '```

added in #2630
Repository: EleutherAI/lm-evaluation-harness
Issue: local-completions error with dataset longer than max_length
Root cause: Validation Gap

While testing meta-llama/Meta-Llama-3-8B model perpexity via vLLM, there is error on assessing context length. Here are steps to reproduce.
- start vLLM with model meta-llama/Meta-Llama-3-8B
- run evaluation
`lm-eval --model local-completions --tasks wikitext --model_args model=meta-llama/Meta-Llama-3-8B,base_url=http://localhost:8000/v1/completions,num_concurrent=1,tokenized_requests=True,max_length=8190 --verbosity DEBUG --log_samples --output_path test/`
It is causing an error on some requests:
```
  File "/lm-evaluation-harness/lm_eval/models/api_models.py", line 510, in _loglikelihood_tokens
    self.parse_logprobs(
  File "/lm-evaluation-harness/lm_eval/models/openai_completions.py", line 68, in parse_logprobs
    assert ctxlen > 0, "Context length must be greater than 0"
AssertionError: Context length must be greater than 0
```
It happens when len(context_enc) + len(continuation_enc) - (self.max_length) is equal to 1.

Hi! Thanks for reporting the issue. #2262 should fix it!
Repository: EleutherAI/lm-evaluation-harness
Issue: What are `mmlu_continuation` and `mmlu_generative`?
Root cause: Documentation Deficiency

What are `mmlu_continuation` and `mmlu_generative`? Where can I find their description?

I am going to test `mmlu` in the `cloze` way. Like the following illustration: 
![image](https://github.com/user-attachments/assets/8488ca43-de06-47bf-a36b-7d07dfd5f044)

Thank you!

Hi! mmlu_generative is a generation variant of mmlu (so can be used when log probs aren't available). mmlu_continuation is cloze style multiple-choice but it does _not_ put the choices in the context. So for example the prompt will be `<question> <answer_choice_a>, <question> <answer_choice_b>...` (compared to default MMLU where it's `<question> <choices> <A.>, <question> <choices> <B.>`).  
If you want the choices in context as well, I think the simplest way is probably replacing the doc_to_choice in the [default](https://github.com/EleutherAI/lm-evaluation-harness/blob/ebe7226ebfb8d11a9fb8d6b53eb65891f895c633/lm_eval/tasks/mmlu/default/_default_template_yaml#L8) config  with `{{choices}}`

I see. It is quite clear. Thank you so much!

Hi,
Just to confirm, is my following command correct?

```
dataset_path: hails/mmlu_no_train # a copy of `cais/mmlu` with no auxiliary_train split
test_split: test
fewshot_split: dev
fewshot_config:
  sampler: first_n
output_type: multiple_choice
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
doc_to_choice: {{choices}}
doc_to_target: answer
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0
dataset_kwargs:
  trust_remote_code: true
```

Hi @shizhediao , this looks correct to me, although if you wanted the

"The following are multiple-choice questions (with answers) about ..." you would need to add that to the `description` config field!

Thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: Regarding metric chrf's implementation
Root cause: Unimplemented Feature Gap

Hi,

In this repo, the `chrf` metric [implementation](https://github.com/EleutherAI/lm-evaluation-harness/blob/ebe7226ebfb8d11a9fb8d6b53eb65891f895c633/lm_eval/api/metrics.py#L92C1-L103C52) calls `sacrebleu.corpus_chrf()` with default [parameters](https://github.com/mjpost/sacrebleu/blob/0f351010b8b641aaa59fe75b98d7cc522bf221eb/sacrebleu/compat.py#L94): character order 6 and word order 0. Perhaps in `metric.py` it would be nice to include those two optional parameters so users can specify this in `task.yml`, for example to facilitate `chrf++` which uses word order 2?

Thanks!

Hi! That sounds reasonable. Would you like to open a PR?

Hi can i take this up?

Hi @JINO-ROHIT , yes sure, I have not implemented this.

Yo @PinzhenChen !
if this is still open, can I take this up..?


> Yo [@PinzhenChen](https://github.com/PinzhenChen) ! if this is still open, can I take this up..?

yes! that would be great

I noticed that this feature hasn’t been implemented yet, so I went ahead and tried to solve it. Hope that’s okay!