zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Per-sample perplexity of a continuation?
Root cause: Documentation Deficiency

This could be related to this question but I want to ask a separate one for clarity: https://github.com/EleutherAI/lm-evaluation-harness/issues/1595

Suppose that I have a dataset of the following format, 

| context   | continuation           | 
|-----------|------------------------|
| This is a |  very good movie       |
| This is a |  fascinating book      | 
| This is a |  long but boring story | 

I have three instances, each with a context and a continuation. I want to compute the per-sample perplexity of each continuation given the context (not really interested in any aggregation). 

The perplexity is defined by the PPL(X) formula [here](https://huggingface.co/docs/transformers/en/perplexity). Specifically, I compute the log likelihood of each token in the continuation given all the preceding text (i.e., context + earlier continuation tokens), take the average by the number of words in the continuation, and exponentiate its negative. Is there a way to do this automatically?

For per-sample perplexity, the current `perplexity` metric seems to be just doing sum of loglikelihood, rather than taking the average over all words. 

The aggregation referenced in the answer of the issue above is about computing average perplexity across all samples. However, this average is still the *sum* of log probability, just averaged across all instances. So according to the PPL(X) definition, I don't think this is the actual notion of perplexity? 

You're right--Lambada `perplexity` is unique in that we assume there is only one word, so no averaging over length is needed. This should be changed for the general case

I see. Thanks. I think some caution on the documentation would be helpful.
Repository: EleutherAI/lm-evaluation-harness
Issue: `lm_eval --tasks list` return nothing?
Root cause: Environment Incompatibility

Somehow installing from `main` only give me this output

```bash
Downloading builder script: 100%|██████████████████████████████████████████████████| 5.67k/5.67k [00:00<00:00, 8.07MB/s]
```

The second time doing it returns nothing

Hi @fahadh4ilyas --I tried to freshly install the library and was not able to reproduce the behavior that you are seeing. Could you try freshly creating a new environment and trying again?

Anecdotally, I've rarely had it happen where all messages printed via the logger don't show up (that is what is happening in your case), but a reinstall always fixes this. I've not been able to track down why this happens yet--it seems like when I'm trying to use an env that I'd created as a different user it sometimes occurs? But have not found a way to cleanly replicate.

Closing as I can't replicate, please lmk if there are more details you can share!

Yes, return nothing on version 0.4.3
Repository: EleutherAI/lm-evaluation-harness
Issue: MuTual scores not matching with values in paper
Root cause: Algorithmic Error

Both detokenized and non-detokenized have the same problem

![image](https://user-images.githubusercontent.com/54557097/122661559-e1bc5a00-d148-11eb-86cb-ebdca833e8b8.png)
current impl on master

plus has higher score, presumably because it isn't all-lowercased

@leogao2 can we close this?
Repository: EleutherAI/lm-evaluation-harness
Issue: Limiting scipy integration
Root cause: Architectural Constraint

Context: The `scipy` package can be difficult to redistribute as it requires fortran and some other custom libraries. 

Looking at the code, `scipy` is only used in two places:
1. The `_align_bags` func in `utils.py` for DROP: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/drop/utils.py#L125
2. The `calculate_z_value` func in `model_comparator.py` script: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/scripts/model_comparator.py#L30

Would the maintainers be amenable to making `scipy` a local import for these functions so other aspects of the library can be used without needing to install `scipy`?

This seems very reasonable!

So @nathan-weinberg `scipy` is installed by way of `scikit-learn`--while we don't use `sklearn` in *too* many places within `lm_eval.api.metrics`, it might be annoying for many users to have to manually install a scikit-learn extra for any task using F1.


If scipy being installed is a big dealbreaker though, can likely just make sklearn an optional dependency as well. But curious if there are any in-betweens commonly taken here or that could still be useful

@haileyschoelkopf (responding from the same team as @nathan-weinberg) I don't know of any in-between options and so making sklearn an optional dep seems like it would need to be part of the solution.  If there is an alternative you are thinking of please let us know.  Really appreciate the consideration!

Hi @haileyschoelkopf 

I'm also the same team. We haven't considered the dependency on `sklearn`. I can see how the additional dependency makes the situation more complicated.

I have another idea. We are rebuilding all wheels from sources anyway. Our build pipeline has the capability to modify a package to e.g. drop a dependency. It would help us a lot of you could either consolidate `import scipy, sklearn` into one module or use function-level imports instead of module-level imports for `scipy` and `sklearn`. That would reduce the complexity of our downstream changes.

Example:
```
def cb_multi_fi(items):
    from sklearn.metrics import f1_score

    preds, golds = zip(*items)
    preds = np.array(preds)
    golds = np.array(golds)
    f11 = f1_score(y_true=golds == 0, y_pred=preds == 0)
    f12 = f1_score(y_true=golds == 1, y_pred=preds == 1)
    f13 = f1_score(y_true=golds == 2, y_pred=preds == 2)
    avg_f1 = np.mean([f11, f12, f13])
    return avg_f1
```

What do you think?

@haileyschoelkopf I'm also from the same team as well. I'd be happy to open up a PR consolidating the imports into one module or moving the imports around to being function-level as long as you'd be receptive to reviewing and accepting such a change.

Function-level imports seem like a good idea! Would be happy to accept that change : )

@haileyschoelkopf feel free to reassign the issue to me if the above PR is satisfactory on y'alls side!
Repository: EleutherAI/lm-evaluation-harness
Issue: Can I see all raw inputs to models and raw outputs from models?
Root cause: Documentation Deficiency

When I get log from lm-eval with `--log_samples` argument, it creates inputs and outputs.
Are the inputs in the log samples applied by chat template, system instruct and fewshots?
In other words Are the inputs to models and the outputs from models logged?

This could be confirmed by manual inspection, but for `--model hf` and `--model vllm` types, then if you use the chat templating feature then yes, the inputs after being wrapped by a chat template should be logged!
Repository: EleutherAI/lm-evaluation-harness
Issue: Inconsistent format of `doc_to_text` in the task.yaml files?
Root cause: Documentation Deficiency

For evaluating multi-choice datasets, the choices should be given in the input prompt for LLM(which is `doc_to_text` in task.yaml file).

While [some task files](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/belebele/_default_template_yaml) includes choices in the input prompt, [some](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/bigbench/multiple_choice_template_b_yaml) does not.

I'm curious if task files are just meant to show one example.

But in the second example
```
doc_to_choice: "{{multiple_choice_targets}}"
```
choices are read from dataset...

@andrew0411 there are multiple ways to frame multiple choice questions and it's not clear that there is "one true answer" for every case. The choice of what completions to score over, and whether to list out answer choices in the prompt, can vary

For more context on what those ways are I'd suggest checking out
https://arxiv.org/abs/2406.08446
https://arxiv.org/abs/2405.14782
Which one is "best" is going to be application-dependent.

In the Eval Harness, we intend to support the prompt style that is originally implemented by the benchmark authors or the first person to popularize the task (for example, for historical reasons, a number of tasks in this repo use a "cloze" style and the prompt style implemented by GPT-3 in the GPT-3 paper, because that paper adapted a number of NLP tasks to the prompted setting). Task configs can be changed as desired to fit your purposes, so long as when reporting benchmark scores you provide the config so that others could get the same numbers you did.

I hope this helps to clear things up!
Repository: EleutherAI/lm-evaluation-harness
Issue: Running evaluation on Gemma-2 27B model
Root cause: Resource Mishandling

Hi,

I'm trying to run evaluation code with the Gemma-2 27B model. I have 2 GPUs, each with 80 GB of memory. However, I encountered an out-of-memory error when using the parallel command. Has anyone successfully used multiple GPUs for evaluation, and if so, how? Can you help me, please?

I have tried running the following command:

`lm_eval --model hf --model_args pretrained=google/gemma-2-27b,parallelize=True --tasks mmlu,arc,gsm8k,hellaswag,truthfulqa,winogrande --output google-gemma-2-27b --batch_size 8`

I also tried using the accelerate command and as Python code:

`accelerate launch -m lm_eval`

but that didn't work either.

**Note**: I have tried exporting CUDA_VISIBLE_DEVICES=0,1.

Thank you
Repository: EleutherAI/lm-evaluation-harness
Issue: Error Running New Open LLM Leaderboard Tasks
Root cause: Environment Incompatibility


### Error Running New Open LLM Leaderboard Tasks: "Tasks were not found"

**Error Encountered:**
```
Tasks were not found: leaderboard
Try `lm-eval --tasks list` for list of available tasks
```

I am encountering an error with the newly added Open LLM Leaderboard tasks under the [leaderboard](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/leaderboard/README.md) task group. The error occurs when attempting to execute the following command:

```bash
lm_eval --model hf --model_args "$MODEL_ARGS" --task="$TASK" --batch_size auto --output_path "$MODEL_DIR/$TASK"
```

### Steps Taken:
1. Attempted to run the command using the entire leaderboard task group.
2. Attempted to run the command using individual leaderboard subgroups and specific tasks, such as `leaderboard_bbh` and `leaderboard_bbh_boolean_expressions`.

Despite these attempts, the same error persists.

### Additional Information:
- The command being used is the same as listed on the [Hugging Face documentation for the Open LLM Leaderboard reproducibility](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about#reproducibility).
- The leaderboard tasks are not present in the tasks list returned by `lm-eval --tasks list`.

### Request:
I would appreciate any guidance on resolving this issue and successfully running the new leaderboard tasks.


If you can launch any other task, it's likely that it's an install/version problem.
Did you make sure to install from main in a clean env?

@dhiaEddineRhaiem This is an unrelated error to the main one, you should most likely open another discussion

@clefourrier Thank you for your prompt response and the helpful clarification.

As correctly mentioned here: https://github.com/EleutherAI/lm-evaluation-harness/issues/2064#issuecomment-2210255389 , the issue was due to my installation using `lm-eval==0.4.3`, which does not include the leaderboard tasks, as they are only available on the main branch at this moment. Installing the harness directly from the source i.e. as outlined in the [installation guide](https://github.com/EleutherAI/lm-evaluation-harness?tab=readme-ov-file#install), resolves the issue.

I appreciate all your assistance here, closing this issue, Thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: package version conflict while launching leaderboard2 eval
Root cause: External Dependency Breakage

what version of omegaconf , hydra-core and antlr4-python3-runtime should we use ! i got error precising antlr4-python3-runtime should have version 4.11 , what is the consequent versions of omegaconf and hydra-core please ? @clefourrier 

similar issue here https://github.com/facebookresearch/hydra/issues/2699#issuecomment-1635165162

It looks like `omegaconf` can be installed from main to get compatibility: https://github.com/omry/omegaconf/pull/1114

and same with `hydra-core`: https://github.com/facebookresearch/hydra/pull/2733 --no releases including this yet

It seems like `omegaconf==2.4.0.dev2` and `omegaconf==2.4.0.dev3` should do the trick (pre-release versions), though no equivalent for hydra-core is there yet.



I solved the issue today , 
Antlr4 must have version 4.11, as a consequence , i agree omegaconf pre-release versions 2.4.0.dev3/2 are compatible with antlr 4.11.

for hydra-core , i cloned the repo locally , changed the jar file to point to 4.11 not 4.9.3 and changed the correspondant affected files ( see this ref https://github.com/facebookresearch/hydra/issues/2699#issuecomment-1639058159)

then installed hydra-core in my docker file using the local ammended repo folder of hydra.
Repository: EleutherAI/lm-evaluation-harness
Issue:  LLM leader board setting for mmlu.
Root cause: Documentation Deficiency


I have pre download the MMLU dataset [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train), and I want to reproduce the result on MMLU dataset in [LLM leader board (old-version)](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard). In the leader board, MMLU setting is ```5-shot, hendrycksTest-abstract_algebra,hendrycksTest-anatomy,hendrycksTest-astronomy,hendrycksTest-business_ethics,hendrycksTest-clinical_knowledge...```

However, in `lm_eval/tasks/mmlu` folder, there are several variants are supported. I want to know what parameters I need to set for my `tasks`. Thanks.


lm_eval version=0.4.3
```
Name: lm_eval
Version: 0.4.3
Summary: A framework for evaluating language models
Home-page: 
Author: 
Author-email: EleutherAI <contact@eleuther.ai>
License: MIT
Location: /home/pclv100s-1/.local/lib/python3.9/site-packages
Editable project location: /home/pclv100s-1/Desktop/dsj/lm-evaluation-harness
Requires: accelerate, datasets, dill, evaluate, evaluate, jsonlines, more-itertools, numexpr, peft, pybind11, pytablewriter, rouge-score, sacrebleu, scikit-learn, sqlitedict, torch, tqdm-multiprocess, transformers, word2number, zstandard
Required-by: 
```

`leaderboard_mmlu_pro` (available in `main`, but not `v0.4.3` or any PyPI version yet) can be used!
Repository: EleutherAI/lm-evaluation-harness
Issue: TinyBenchmark/TinyMMLU broken?
Root cause: External Dependency Breakage

There was a commit 21 hours ago that deleted some of the parquet files from the TinyMMLU dataset on the hub (https://huggingface.co/datasets/tinyBenchmarks/tinyMMLU), since then the TinyBenchmark task fails:
```
datasets.exceptions.DataFilesNotFoundError: No (supported) data files found in tinyBenchmarks/tinyMMLU
```
I've tried changing the dataset_name from 'all' to 'default' as suggested in the updated readme on the dataset page but that didn't help. I've also tried to pin a previous commit via the revision passed by dataset_kwargs but that still lead to the same issue.

Should either of those approaches have worked or does the dataset owner need to do some further changes to make this compatible again?

Hi @LucWeber ! do you happen to know what's going on with changes to the tinyMMLU dataset?


@skramer-dev I'm surprised that passing a revision through `dataset_kwargs` doesn't fix this. would you be willing to dig in a bit just to triple-check we are indeed passing that kwarg to the download method as expected?

A few things happened, there were a load of commits over the weekend and i ultimately got it to work again today. I'd have to look at the exact configuration tomorrow but i think i both removed the dataset_name and the fewshot_split fields from the config yaml.

I also believe that there were some sort of caching issues with the way i executed the run code in our cloud evironment. I added a print to get the job config that is passed down to datasets and saw some values there that weren't set in the yaml of the respective task so the fact that the revision part didn't work could have come down to that. I'd have to check it again with the print of the assembled config to see if it indeed takes the revision kwarg and does something with it or not.

Hey both,
thanks a lot for pinging me, we will look into this!

I think tinyMMLU should run as usual again. Let me know if there are still any problems @skramer-dev

> I think tinyMMLU should run as usual again. Let me know if there are still any problems @skramer-dev

I've used the original config now and it works again 👍
Repository: EleutherAI/lm-evaluation-harness
Issue: Add LAMA + MLAMA
Root cause: Unimplemented Feature Gap

LAMA: https://github.com/facebookresearch/LAMA
Multilingual LAMA: https://github.com/norakassner/mlama

This paper, X-FACTR (https://arxiv.org/abs/2010.06189), is better than mLAMA in terms of more coverage of languages and better quality.
Repository: EleutherAI/lm-evaluation-harness
Issue: max_new_tokens and max_length conflict
Root cause: Configuration Error

Setting up the Hugging Face Open LLM Leaderboard to use the realtoxicityprompts task through `simple_evaluate`.
Keep receiving this warning:

`Running generate_until requests:   0%|          | 13/99442 [10:42<1539:30:14, 55.74s/it]
Both `max_new_tokens` (=2048) and `max_length`(=302) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)`

I have not set `max_new_tokens` nor `max_length` on my end; it looks to me like `max_new_tokens` is being set with a default value of 2048 in the Harness code. 
This is overriding `max_length`, which looks to be a more reasonable value defined per-instance when running the realtoxicity task.

This can also occur if setting `max_new_tokens` within the `generation_kwargs` in some tasks' yaml.
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding a metric and an aggregation requires knowledge of input
Root cause: Documentation Deficiency

When creating a custom aggregation and metric function for a task, it is not clear what the input is. Furthermore, **printing out the input shape does not work.** Therefore, it is very difficult for someone to add their own custom metric and aggregation for a custom task.

1. What is the input?
2. Can the code be rewritten to allow the printing of inputs and shapes?

Reproduce:

utils file for a task:
```
def agg(arr):
    print(arr)
    return arr

def metric(items):
    print("METRIC FUNCTION")
    print(f"Items type: {type(items)}")
    print(f"Number of items: {len(items)}")
    print(f"Example item: {items[0]}")  # Print an example item for inspection
    return items
```

Task:
```
task: "discrim_eval"
dataset_name: "explicit"
dataset_path: "Anthropic/discrim-eval"
doc_to_text: '{{filled_template}}\nPlease answer the above question with either "yes" or "no."'
doc_to_choice: ["no", "yes"]
doc_to_target: "yes"
test_split: "train"
output_type: multiple_choice
metric_list:
  - metric: !function utils.metric
    aggregation: !function utils.agg
    higher_is_better: true
```

Didn't do metrics correctly

Hi, it would be nice to explain your solution for the records :)

I just wrote a function to export all of the results from discrim_eval into a pkl file, and then implemented the metric outside of lm-eval-harness through my own python function
Repository: EleutherAI/lm-evaluation-harness
Issue: Invalid modification to task YAML file
Root cause: Validation Gap

I am running llama2 model in wikitext dataset. I just want try some other metrics so I modify the default YAML file(`lm-evaluation-harness/lm_eval/tasks/wikitext/wikitext.yaml`) to the following, just deleting perplexity and adding acc.
```yaml
task: wikitext
dataset_path: EleutherAI/wikitext_document_level
dataset_name: wikitext-2-raw-v1
output_type: loglikelihood_rolling
training_split: train
validation_split: validation
test_split: test
doc_to_text: ""
doc_to_target: !function preprocess_wikitext.wikitext_detokenizer
process_results: !function preprocess_wikitext.process_results
should_decontaminate: true
doc_to_decontamination_query: "{{page}}"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 2.0
dataset_kwargs:
  trust_remote_code: true
```
But when I run the evaluation again, I get an error like this:
```shell
Traceback (most recent call last):
  File "/home/xxx/environment/miniconda/miniconda3/envs/hf-transformers/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/data/xxx/lm-evaluation-harness/lm_eval/__main__.py", line 385, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data/xxx/lm-evaluation-harness/lm_eval/utils.py", line 395, in _wrapper
    return fn(*args, **kwargs)
  File "/data/xxx/lm-evaluation-harness/lm_eval/evaluator.py", line 308, in simple_evaluate
    results = evaluate(
  File "/data/xxx/lm-evaluation-harness/lm_eval/utils.py", line 395, in _wrapper
    return fn(*args, **kwargs)
  File "/data/xxx/lm-evaluation-harness/lm_eval/evaluator.py", line 579, in evaluate
    task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)
  File "/data/xxx/lm-evaluation-harness/lm_eval/evaluator_utils.py", line 96, in calculate_aggregate_metric
    agg_fn = self.task.aggregation()[metric]
KeyError: 'word_perplexity'
```

**Running script:**
```shell
lm_eval --model hf \
    --model_args pretrained=/data/xxx/cache/models--baffo32--decapoda-research-llama-7B-hf/snapshots/aa18b48a1330572a6dd5f5d5619ed19838ca285c \
    --tasks wikitext \
    --device cuda:1 \
    --batch_size 8 \
```

It seems the modification to wikitext YAML file does not work. 

Hi! What's going on here is that you're trying to use a metric `acc` that isn't set up to work with the `loglikelihood_rolling` metric. It's on our todo list to make metrics more clear and to prevent silent failures made in cases like this. 

When you insert `acc`, what do you expect to be computed? Token-level accuracy as a % ? 

> Hi! What's going on here is that you're trying to use a metric `acc` that isn't set up to work with the `loglikelihood_rolling` metric. It's on our todo list to make metrics more clear and to prevent silent failures made in cases like this.
> 
> When you insert `acc`, what do you expect to be computed? Token-level accuracy as a % ?

Yes, what about token-level accuracy? It seems not easy to modify this file correctly. At least the `output_type` and `doc_to_xxx` should be set properly.
Repository: EleutherAI/lm-evaluation-harness
Issue: The evaluation results are inconsistent across different GPUs
Root cause: Algorithmic Error

command:
<img width="744" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/121479335/524b642b-88b6-4317-82b0-02971e3eb016">
result:
<img width="824" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/121479335/0eb48944-2d89-44d6-9b5d-5aa2e835c930">
Why do the same model and the same command yield inconsistent results?

I am noticing a similar problem as well when using lm-harness to evaluate Llama3 model.

+1, this is happening for multiple models to me