zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Question: Does this library provide learning when evaluating?
Root cause: Documentation Deficiency

I use `b281b09` branch
Evaluate through the tasks below
`arc_challenge`
`hellaswag`
`truthfulqa_mc`
`hendrycksTest-*` (All hendrycksTest)
`winogrande`
`gsm8k`

(huggingface leaderboard metrics)

Do the above tasks learn before evaluating?

example) python main.py --model=hf-causal-experimental --model_args="pretrained=meta-llama/Llama-2-7b-hf,use_accelerate=True" --tasks=arc_challenge --num_fewshot=25 --batch_size=1 --output_path=/workspace/output/origin_arc.json

I'm curious about the meaning of `--num_fewshot=25`

Hi,

1. There is no finetuning involved in this library as of now. So no learning in that sense is done before evaluating.
2. `--num_fewshot` is basically how many examples a model is shown before being given the sample it needs to predict/answer. For example, the case of an arithmetic problem, an input with 2-shot example would be `Q: 1 + 1 = ... A: 2\n\nQ: 3 + 8 = ... A: 11\n\nQ: 3 + 3 = ... A:`

hope this helps.

Thank you.
But I still couldn't answer the question.
Is it correct to tell the model how to answer the question through the few-shot?

So how do you tell the model the answer to the question?

In my experience, the PreTrained model(not fine-tuned) doesn't know how to answer questions.

You can refer here as a starter for how language models can answer queries given few-shot examples
https://arxiv.org/abs/2005.14165



Thank you. I'll read it :)
Have a nice day
Repository: EleutherAI/lm-evaluation-harness
Issue: Speed up + streamline prompt template rendering runtime
Root cause: Unimplemented Feature Gap

In some situations (for example, N-shot with very large N) we currently spend an annoying amount of time redundantly rendering our Jinja2 prompt template strings. 

There are two ways we can significantly cut down on this runtime, without impacting code complexity significantly (e.g., no multiprocessing when generating requests / Instances, etc.)
1. At the start, render `doc_to_target`, `doc_to_text`, `doc_to_choice` once each for every doc, and cache that result so that we don't need to repeatedly render our prompts when we want to call these methods multiple times for every test instance.
2. *cache* these attributes, or the actual Instance objects, created when a task is run once, so that we can skip this whole step's processing in future. The easiest way to do so would be to leverage HF datasets' `dataset.map()` function which can cache results based on pickling a function. *However,* we'd want to be extremely cautious of whether this might create weird bugs when someone is trying to change + rerun a task but the cached intermediate input objects get silently used, preventing them from seeing the expected changes in behavior when they change code.

Change 1 should suffice for some pretty nice QoL improvements.
We should however also make sure this doesn't negatively affect runtime when we'd have to perform it on a large train set that we're drawing few-shot examples from.



I'll probably take on change 1 sooner rather than later--but if no PR can be seen that references this issue yet, then anyone interested can comment here volunteering to add it too!

Hi ! Happy to help if you have any question regarding `datasets` hashing for `map()` functions. :)

FYI it dumps the function using `dill` (an alternative to `pickle` that can handle e.g. lambda functions) and `recursive=True` to also take into account the globals used in the function. The hash is computed using the `dill` dump to build the fingerprint of the resulting dataset and store it in the cache, see https://huggingface.co/docs/datasets/v2.16.1/en/about_cache#fingerprint

Though you can override this mechanism by passing your own `new_fingerprint=` to `map()` but in this case you want to make sure that the fingerprint is not the same whenever a processing parameter changes.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate using custom prompt template?
Root cause: Unimplemented Feature Gap

My model prompt template is this:

```
{system_prompt}
User: {user_prompt}
Assistant: {ai_prompt}

```

I already add `chat_template` to my tokenizer. Is it enough to do evaluation without set any parameters?

Hi, we are working on supporting custom chat / prompt templates in #1277 !

For the time being, you could use this with a task you want by editing its `doc_to_text` field in the YAML config, though this is not very convenient. 

Closing as we have #1098 to centralize discussion of this feature.
Repository: EleutherAI/lm-evaluation-harness
Issue: Different score when using `accelerate`
Root cause: Resource Mishandling

I'm observing discrepancy between when running lm-eval with and without accelerate. I've only realized it just now.

With accelerate
```
hf (pretrained=lintang/t5-v2-base-sglue,truncation=True,max_length=512), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
|         Tasks          |Version|     Filter      |n-shot| Metric |Value |   |Stderr|
|------------------------|------:|-----------------|-----:|--------|-----:|---|-----:|
|super_glue-wsc-t5-prompt|      0|wsc_postprocessor|     0|accuracy|0.6346|±  |0.0474|
```

Without 
```
hf (pretrained=lintang/t5-v2-base-sglue,truncation=True,max_length=512), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
|         Tasks          |Version|     Filter      |n-shot| Metric |Value |   |Stderr|
|------------------------|------:|-----------------|-----:|--------|-----:|---|-----:|
|super_glue-wsc-t5-prompt|      0|wsc_postprocessor|     0|accuracy|0.7308|±  |0.0437|
```

~Could this be limited to seq-to-seq models? Might be used too infrequently to pick up the bug.~
Tried pythia-1b on gsm8k and there's a discrepency as well. But there the fewshot samples are different (with or without accelerate) so it is hard to tell. 

But on `wsc_postprocessor`, the same outputs seem to be scored differently.

@baberabb would you be interested to take a look into this?

Yeah I can take a look again. Do you have a suspicion about the main cause?

Not at the moment. But for sanity check, I would check the output value for both versions to be sure that the model outputs are the same. If that's the case, then it could be how the outputs are collected.
Repository: EleutherAI/lm-evaluation-harness
Issue: Accuracy not matched for Llama1-7b
Root cause: Configuration Error

The evaluated results for Llama-7b on zero-shot common sense reasoning tasks are not matched with the paper.

I use the following command

```
lm-eval 
    --model hf 
    --model_args pretrained=huggyllama/llama-7b,trust_remote_code=True 
    --tasks boolq,piqa,social_iqa,hellaswag,winogrande,arc_easy,arc_challenge,openbookqa 
    --output_path results.json 
    --num_fewshot 0
    --batch_size 16 
```

and the results are shown below

```
hf (pretrained=huggyllama/llama-7b,trust_remote_code=True), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 16
|    Tasks    |Version|Filter|n-shot| Metric |Value |   |Stderr|
|-------------|-------|------|-----:|--------|-----:|---|-----:|
|arc_challenge|Yaml   |none  |     0|acc     |0.4181|±  |0.0144|
|             |       |none  |     0|acc_norm|0.4471|±  |0.0145|
|arc_easy     |Yaml   |none  |     0|acc     |0.7525|±  |0.0089|
|             |       |none  |     0|acc_norm|0.7294|±  |0.0091|
|boolq        |Yaml   |none  |     0|acc     |0.7511|±  |0.0076|
|hellaswag    |Yaml   |none  |     0|acc     |0.5692|±  |0.0049|
|             |       |none  |     0|acc_norm|0.7618|±  |0.0043|
|openbookqa   |Yaml   |none  |     0|acc     |0.3440|±  |0.0213|
|             |       |none  |     0|acc_norm|0.4440|±  |0.0222|
|piqa         |Yaml   |none  |     0|acc     |0.7867|±  |0.0096|
|             |       |none  |     0|acc_norm|0.7916|±  |0.0095|
|social_iqa   |Yaml   |none  |     0|acc     |0.3291|±  |0.0106|
|winogrande   |Yaml   |none  |     0|acc     |0.7009|±  |0.0129|
```

However, some results are different from paper. (https://arxiv.org/pdf/2302.13971.pdf)

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/14341413/68e63ba4-369b-4db7-a7f4-9d890e9dfc78)

For example, SIQA is 48.9 but I got 32.91; OBQA is 57.2 but I got 44.4; arc_challenge is 47.6 but I got 44.71.



Hi, thanks for opening this issue! 

As also mentioned in #1292 , it's difficult / potentially not possible to fully replicate Llama's eval setup without access to their prompts. However, one thing we do know is that for some tasks, they prepend a task description like "Answer these questions:\n\n" which seems to help models zero-shot, often significantly, though the gap can often close when doing few-shot evaluation. In many cases, we don't use this description option if the original task implementation / prompt we take from did not provide one. 

However, we expose the ability to use a description via `description: .....` in the YAML config file for a task. 

I hope that this might be helpful!
Repository: EleutherAI/lm-evaluation-harness
Issue: Can I use the lm-evaluation-harness to evaluate a quantized or pruned model, where I am working with a saved weight model that has already undergone quantization?
Root cause: Documentation Deficiency

I have quantized a model using the intel-extension-for-transformers on the CPU, can I use the lm-evaluation-harness to evaluate it?

Hi, we haven't tested support for this library before. However, we do support both 1. passing already-initialized HF PretrainedModel classes into the init for `HFLM` and 2. the ability to eval HF models you've saved locally, by passing `--model_args pretrained=<path/to/model/goes/here>`. 

Given this, I think that if you install that package, and follow the instructions in their readme: https://github.com/intel/intel-extension-for-transformers?tab=readme-ov-file#int4-inference-cpu-only
(AKA passing `--model_args pretrained=<path/to/save/location/of/your/model/on/disk>,load_in_4bit=True` and `--device cpu` ) that this will work as intended.

Feel free to reopen and let us know if this is not in fact the case.

apologize for reopening this issue, as I am not yet fully proficient in using this tool. In order to conduct evaluations with Intel® Extension for Transformers, I need to begin by importing the library using the command from intel_extension_for_transformers.transformers import AutoModelForCausalLM. However, I have not found a method to properly pass 'intel_extension_for_transformers.transformers' into my project. Could you kindly guide me on how to do this? I would greatly appreciate it (as I am genuinely new to these and still learning).
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue downloading TriviaQA
Root cause: External Dependency Breakage

After running the following code I get the error:

> lm_eval.tasks.initialize_tasks()
> task = lm_eval.tasks.get_task_dict(['triviaqa'])['triviaqa']

I get the following error

`datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=12749651131, num_examples=138384, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=49068770211, num_examples=728780, shard_lengths=[5323, 5323, 5323, 5323, 5323, 5323, 5323, 5323, 5323, 5323, 5323, 5323, 5322, 5322, 5322, 10644, 10644, 10644, 10644, 10644, 147733, 8053, 8052, 4026, 4026, 4026, 4026, 8052, 8052, 8052, 8052, 8052, 85338, 17682, 17682, 17682, 65618, 3730, 3730, 3730, 3730, 1865, 1865, 1865, 1865, 1864, 1864, 1864, 1864, 1864, 1864, 1864, 3728, 3728, 3728, 3728, 3728, 3728, 3728, 1864, 3728, 3728, 3728, 3728, 3728, 1864, 87622], dataset_name='parquet')}, {'expected': SplitInfo(name='validation', num_bytes=1662321188, num_examples=17944, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=6399889055, num_examples=94402, shard_lengths=[4486, 4486, 8972, 24578, 21261, 11226, 1616, 3232, 3232, 11313], dataset_name='parquet')}, {'expected': SplitInfo(name='test', num_bytes=1577710503, num_examples=17210, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='test', num_bytes=6071557757, num_examples=90504, shard_lengths=[4303, 4303, 8604, 23550, 20379, 11313, 1805, 3610, 12637], dataset_name='parquet')}]`

Before running the code I deleted the existing cache for TriviaQA. I noticed on HuggingFace Hub that the most recent commit is 

> [Convert dataset to Parquet](https://huggingface.co/datasets/trivia_qa/commit/0f7faf33a3908546c6fd5b73a660e0f8ff173c2f)

Could this be the source of the issue?



It works fine with `datasets==2.16.1`
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/38466901/6089980c-7b2f-47d4-8ac2-b39a9b78aaf2)
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the adversarially-mined Winogrande evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> The Winograd Schemas Challenge [LDM12] is a classical task in NLP that involves determining which word a pronoun refers to, when the pronoun is grammatically ambiguous but semantically unambiguous to a human. Recently fine-tuned language models have achieved near-human performance on the original Winograd dataset, but more difficult versions such as the adversarially-mined Winogrande dataset [SBBC19] still significantly lag human performance. We test GPT-3’s performance on both Winograd and Winogrande, as usual in the zero-, one-, and few-shot setting.
> On Winograd we test GPT-3 on the original set of 273 Winograd schemas, using the same “partial evaluation” method described in [RWC+19]. Note that this setting differs slightly from the WSC task in the SuperGLUE benchmark, which is presented as binary classification and requires entity extraction to convert to the form described in this section. On Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings, showing no clear in-context learning but in all cases achieving strong results just a few points below state-of-the-art and estimated human performance. We note that contamination analysis found some Winograd schemas in the training data but this appears to have only a small effect on results (see Section 4).
> On the more difficult Winogrande dataset, we do find gains to in-context learning: GPT-3 achieves 70.2% in the zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting. For comparison a fine-tuned RoBERTA model achieves 79%, state-of-the-art is 84.6% achieved with a fine-tuned high capacity model (T5), and human performance on the task as reported by [SBBC19] is 94.0%.

- [X] Data processing code implemented
- [x] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

_Note_: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/winogrande

Format of the evaluations is different, since this task uses the "partial evaluation" method, where they have the network rate the probability of the context with a missing word/phrase filled in.
Repository: EleutherAI/lm-evaluation-harness
Issue: huggingface_hub.utils._errors.HfHubHTTPError: 401 Client Error: Unauthorized for url: https://hf-mirror.com/api/datasets/hellaswag/hellaswag.py
Root cause: Documentation Deficiency

when i want to use the dataset hellaswag, the following question happend,then i download the dataset and put it in the local dir,but the question is still exists.
![QQ截图20240117130142](https://github.com/EleutherAI/lm-evaluation-harness/assets/96739096/84848022-2e09-45ba-9442-dd0215707c3b)
![QQ截图20240117130200](https://github.com/EleutherAI/lm-evaluation-harness/assets/96739096/655d9ce5-8875-4aaf-8bb4-d7953cf26572)


huggingface_hub.utils._errors.HfHubHTTPError: 401 Client Error: Unauthorized for url: https://hf-mirror.com/api/datasets/hellaswag/hellaswag.py (Request ID: Root=1-65a75e1e-1e8848a653c8f48d7122e5ea;64ab5edb-2078-4206-9051-d34f2280caa6)
Invalid username or password.

@Mistobaan @richhankins @pminervini @jeffhsu3 Could you help me take a look at this issue

My guess is that you need to get a HF_TOKEN from hugging face website and set is as env variable

@qxpBlog you posted your HF api key so I deleted the comment.

Yes, I would check that you have an internet connection on your machine, and if not, then download the dataset ahead of time manually and load it following the local dataset instructions we provide: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#using-local-datasets

> @qxpBlog you posted your HF api key so I deleted the comment.
> 
> Yes, I would check that you have an internet connection on your machine, and if not, then download the dataset ahead of time manually and load it following the local dataset instructions we provide: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#using-local-datasets

thanks

Did this fix?

Closing as this must be either a connection error or else was likely fixed in #1314 . Please let us know if this occurs for other users or persists frequently.
Repository: EleutherAI/lm-evaluation-harness
Issue: Missing tasks
Root cause: Documentation Deficiency

I am trying to evaluate a quantized LLM on some tasks such as `winogrande`, however I get the following:

```
2024-01-17:12:34:41,208 INFO     [__init__.py:219] Available tasks:
2024-01-17:12:34:41,208 INFO     [__init__.py:220] ['squadv2', 'scrolls_qasper', 'scrolls_quality', 'scrolls_narrativeqa', 'scrolls_contractnli', 'scrolls_govreport', 'scrolls_summscreenfd', 'scrolls_qmsum']
Traceback (most recent call last):
  File "/home/user/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 217, in get_task
    return TASK_REGISTRY[task_name](config=config)
KeyError: 'winogrande'
```

Any idea why?

Hey! Are you using `evaluate()` or `simple_evaluate()` outside of the library `__main__.py` or CLI?

If so, you'll need to call `lm_eval.tasks.initialize_tasks()` to load all the YAML tasks into the registry.

That solves it thanks!

If I had more time I'd PR this, but if something could be done to add that to the readme, that'd be big ups, spent some time running the code through the debugger to find out what was going on and then stumbled across this.

That or setting up the tasks to be populated earlier on in the code's execution. (I don't know the underlying details, I'd assume it's probably not as simple as I'm making it out to be.)

@inf3rnus we currently document this in https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md , but makes sense that it’s not as visible as it should be, will look to improve this!

O much appreciated!

Yeah, I mean if it was just put in the repo's main README I think that'd help a ton, or if it's already there and I've missed it, putting it next to the CLI usage, or puting a link to the more extensive docs in the main README that's more front and center, at the very top or bottom.
Repository: EleutherAI/lm-evaluation-harness
Issue: polemo2_in task error with datasets 2.16.1
Root cause: External Dependency Breakage

polemo2_in task error with datasets 2.16.1:
```
Traceback (most recent call last):
  File "~/venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "~/lm-evaluation-harness/lm_eval/__main__.py", line 231, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "~/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "~/lm-evaluation-harness/lm_eval/evaluator.py", line 122, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "~/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 275, in get_task_dict
    task_name: get_task(task_name=task_element, config=config),
  File "~/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 217, in get_task
    return TASK_REGISTRY[task_name](config=config)
  File "~/lm-evaluation-harness/lm_eval/api/task.py", line 619, in __init__
    self.download(self.config.dataset_kwargs)
  File "~/lm-evaluation-harness/lm_eval/api/task.py", line 714, in download
    self.dataset = datasets.load_dataset(
  File "~/venv/lib/python3.10/site-packages/datasets/load.py", line 2523, in load_dataset
    builder_instance = load_dataset_builder(
  File "~/venv/lib/python3.10/site-packages/datasets/load.py", line 2232, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "~/venv/lib/python3.10/site-packages/datasets/builder.py", line 371, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "~/venv/lib/python3.10/site-packages/datasets/builder.py", line 592, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'klej-polemo2-in' not found. Available: ['default']
```

but it works better with datasets 2.15.0.

Hi ! `klej-polemo2-in` is not a valid subset for https://huggingface.co/datasets/allegro/klej-polemo2-in

I opened a PR to fix this issue in the yaml task file
Repository: EleutherAI/lm-evaluation-harness
Issue: Different fewshot with and w/o `accelerate` DDP
Root cause: Resource Mishandling

The fewshot splits are different for the same `doc_id` when using/not using DDP on accelerate.

Edit:

The issue is that each DDP process runs its own instance of the `rnd`. Could have the `rnd` no longer be stateful and offset the seed by `doc_id` on each call. But if there's a way to share it across the processes, that would be better.
https://github.com/EleutherAI/lm-evaluation-harness/blob/97a67d27c09857e5698cbae730750cf84cd987f3/lm_eval/api/samplers.py#L66-L71
Repository: EleutherAI/lm-evaluation-harness
Issue: datasets.utils.info_utils.NonMatchingSplitsSizesError:
Root cause: External Dependency Breakage

when I try use 'python main.py opt-1.3b --wbits 4 --abits 4 --eval_ppl --tasks lambada_openai,piqa,arc_easy,arc_challenge,openbookqa,boolq’  to get the result,it return the error as follows:
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=1305088, num_examples=4358, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='test', num_bytes=5176698, num_examples=17432, shard_lengths=None, dataset_name='parquet')}, {'expected': SplitInfo(name='train', num_bytes=546500949, num_examples=1801350, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=1113622699, num_examples=3676136, shard_lengths=[1650675, 1661350, 364111], dataset_name='parquet')}, {'expected': SplitInfo(name='validation', num_bytes=1159288, num_examples=3760, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=4607450, num_examples=15040, shard_lengths=None, dataset_name='parquet')}]
in all process I only change the dir because of perssion deny,How to solve the error?

It seems you are using an old version of `datasets`, can you try to update `datasets` and try again ?

I just tried and I'm able to load all these tasks without errors with the latest `datasets` 2.16.2