zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: how to do when i need “from intel_extension_for_transformers.transformers import AutoModelForCausalLM”
Root cause: Documentation Deficiency



_Originally posted by @WCSY-YG in https://github.com/EleutherAI/lm-evaluation-harness/issues/1295#issuecomment-1913639288_
            

 apologize for reopening this issue, as I am not yet fully proficient in using this tool. In order to conduct evaluations with Intel® Extension for Transformers, I need to begin by importing the library using the command from intel_extension_for_transformers.transformers import AutoModelForCausalLM. However, I have not found a method to properly pass 'intel_extension_for_transformers.transformers' into my project. Could you kindly guide me on how to do this? I would greatly appreciate it (as I am genuinely new to these and still learning).

You could do so via something essentially like the following:

```
from intel_extension_for_transformers.transformers import AutoModelForCausalLM

from transformers import AutoTokenizer

from lm_eval.models.huggingface import HFLM
from lm_eval import simple_evaluate
from lm_eval import tasks


intel_model = AutoModelForCausalLM.from_pretrained("your-model-name")
tokenizer = AutoTokenizer.from_pretrained("your-model-name")

lm = HFLM(
    pretrained=intel_model,
    tokenizer=tokenizer,
    batch_size=....,
    ...,
)

tasks.initialize_tasks()

results = simple_evaluate(
    model=lm,
    tasks=["taskname1", "taskname2"],
)
``` 

Hope this helps!

> You could do so via something essentially like the following:
> 
> ```
> from intel_extension_for_transformers.transformers import AutoModelForCausalLM
> 
> from transformers import AutoTokenizer
> 
> from lm_eval.models.huggingface import HFLM
> from lm_eval import simple_evaluate
> from lm_eval import tasks
> 
> 
> intel_model = AutoModelForCausalLM.from_pretrained("your-model-name")
> tokenizer = AutoTokenizer.from_pretrained("your-model-name")
> 
> lm = HFLM(
>     pretrained=intel_model,
>     tokenizer=tokenizer,
>     batch_size=....,
>     ...,
> )
> 
> tasks.initialize_tasks()
> 
> results = simple_evaluate(
>     model=lm,
>     tasks=["taskname1", "taskname2"],
> )
> ```
> 
> Hope this helps!

thankyou verymuch
Repository: EleutherAI/lm-evaluation-harness
Issue: Parameter Injection
Root cause: Validation Gap

When we developing a model and trying to give it a shot with lm-evaluation-harness, I have an issue with the file path that includes an equal sign, even if that will be legal on the Linux File System.
Here's the error we got:
```
File "/usr/share/bin/lm_eval/utils.py", line 92, in <dictcomp> 
    k: handle_arg_string(v) for k, v in [arg.split("=") for arg in arg_list]
ValueError: too many values to unpack (expected 2)
```

> Currently, we solve it by renaming the file path without the '=' sign

Here's an example that causes errors.
`lm_eval --model hf --model_arg "pretrained=$WORK/mistral-7b-v0.1_tokenizer=v2|data=m|step=34500|epoch=18975"` --tasks "tmmluplus"

Hi! This is due to the simple parsing utility we use to split the CLI strings into key-value pairs. Where the value part contains additional equal signs, the parser fails to correctly split these into just two elements, leading to the error. 

https://github.com/EleutherAI/lm-evaluation-harness/blob/488759d2c9a55ad3ed55d55417b7733e95e37667/lm_eval/utils.py#L81-L94

> Hi! This is due to the simple parsing utility we use to split the CLI strings into key-value pairs. Where the value part contains additional equal signs, the parser fails to correctly split these into just two elements, leading to the error.

Thanks for responding. I know this issue exists because the argument parser uses the equal sign to separate.
But I'm wondering if there are any plans to solve this problem. 
And to be honest, it's not necessary to solve it.
I just want to submit this problem to you guys.

It may be worth considering allowing for `=` signs within a parameter name *if it appears within `"..."`*, as one thing that's been requested is the ability to pass dicts and lists into the library as arg values via the CLI.

However, this might lead to annoying bugs if it means that we end up silently failing when a user typos their kwargs and this no longer errors immediately.

> It may be worth considering allowing for `=` signs within a parameter name *if it appears within `"..."`*, as one thing that's been requested is the ability to pass dicts and lists into the library as arg values via the CLI.

How about a yaml config? Would be easier to maintain compared to creating all these increasingly complicated parsing rules. 


Maybe yeah. There may be a point at which it's easier to just have people write it out in a python script by passing the correct kwargs direct to the LM

As discussed in #1412 with @Am1n3e , we may want to provide another launcher aside from the CLI that makes it easier for users to provide any arbitrary argument combos to simple_evaluate. However, we currently won't provide for this in `__main__.py`.
Repository: EleutherAI/lm-evaluation-harness
Issue: kernel version requirement too recent
Root cause: Environment Incompatibility

I'm using lm-harness to evaluate local fine-tuned models. However, my kernel version is 5.4.0, older than the recommended 5.5.0, and the process hangs. I'm considering using previous lm-harness commits to bypass this problem. Which previous commit works for 5.4.0?
<img width="1431" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/30493956/65db9315-179b-4cb7-80fe-d0a448056e72">


Hi, it's very difficult to say what is causing this issue without more info. 

Can you share:
- Full outputs from this log, and what command you ran?
- what your environment consists of, and run `pip freeze`

maybe-relevant:
https://github.com/huggingface/datasets/issues/6363
https://github.com/huggingface/accelerate/issues/314#issuecomment-1565259831

When does the process hang? Possibly related: #1404 

Closing as can't replicate with more info--please reopen or comment here if you also experience this issue!

#1404 might fix.

I think this problem is related to https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/other.py#L322
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the EpicQA Evaluation
Root cause: Unimplemented Feature Gap

I'm adding the [EpicQA dataset](https://bionlp.nlm.nih.gov/epic_qa/), as per the issue split suggested in #78.

Any progress on this?

![Screenshot_2509](https://user-images.githubusercontent.com/45185629/114738722-3ccf6c00-9d16-11eb-801c-618805b23049.png)
Somewhat, will have it ready by indicated date prolly

@leogao2 EDIT: actually I prolly won't have this done until like a week from Monday. Maybe we should have someone else do this instead, if the delay is being a problem. Sorry for taking it up :(
Repository: EleutherAI/lm-evaluation-harness
Issue: Request for files to be placed in 'path/containing/training/set/ngrams'.
Root cause: Unimplemented Feature Gap

Hello, I sincerely appreciate your excellent work.

Due to copyright issues with the pile, it's no longer possible to download the 825GB of data, so I can't use the decontamination feature of harness.

If anyone can share the 825GB data, or temporarily and unofficially share the files for `path/containing/training/set/ngrams` in https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/decontamination.md#decontamination, please send them to parkminwoo1991@gmail.com or share them with me. 

The data will be used solely for academic purposes to examine contamination in the dataset. 

If it's possible to share in any way, I would greatly appreciate it.

Thank you.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to add a new custom metric?
Root cause: Documentation Deficiency

Is it possible to add a new custom metric (for eg: efficiency metrics for inference) easily to this framework?

Yes, _but_ I suspect you shouldn't in your use-case. This library is not a good way to run inference efficiency tests. We support adding custom metrics for alternative scoring systems [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/metrics.py).
Repository: EleutherAI/lm-evaluation-harness
Issue: Using dataset_path with / character and --log_samples command line option causes failure
Root cause: Validation Gap

When using a dataset path with a "/" such as:

dataset_path: EleutherAI/asdiv

where the yaml is placed in a directory and passed in the cli with directory

 --task=my/directory

If you include --log_samples, it will fail, because it is trying to write a file which contains / in the filename. 
Without --log_samples it works fine.

Due to this code in

[__main__.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/__main__.py)

```
re.sub("/|=", "__", args.model_args), task_name
```

To recreate:

```
python ./lm_eval --model hf     --model_args pretrained=google/flan-t5-base --limit 1 --output_path output --log_samples  --device cuda:0 --task lm_eval/tasks/asdiv 
```


Suggested solution: replace the special characters in the task_name as well.


I think the standard way is to include the folder containing the configs through `--include_path lm_eval/tasks/asdiv` and specify the tasks using `--tasks asdiv,...`

Hi @baberabb .  Thank you. This solved my problem.
Repository: EleutherAI/lm-evaluation-harness
Issue: `generation_kwargs` in `nq_open` 
Root cause: Configuration Error

Hi, some of the answers in the `nq_open` datasets contain full stops and commas, e.g.,

![Screenshot 2024-02-01 at 17 13 06](https://github.com/EleutherAI/lm-evaluation-harness/assets/227357/4e46b450-3a40-4ced-ae2d-acb031be01d0)

![Screenshot 2024-02-01 at 17 14 16](https://github.com/EleutherAI/lm-evaluation-harness/assets/227357/77e7588e-1ba4-4d55-bce6-786e211748b4)

Could this be a reason to remove some of the `generation_kwargs` in https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/nq_open/nq_open.yaml#L13 ?

Thanks for raising this! Yes, definitely a reason to remove these, although we'd then need to make sure that there's not a score drop from the model then ending up trailing off and not ending quickly enough.

Thanks @haileyschoelkopf, looking into this on the fly

@haileyschoelkopf Here is a quick experiment in a 8-shot setting: https://colab.research.google.com/drive/1eD9bwlRISi6IcsV3v7GH7INRC8i0uTi1?usp=sharing

The command is `!lm_eval --model hf --model_args pretrained=mistralai/Mistral-7B-v0.1 --tasks nq_open_v2 --device cuda --batch_size 1 --num_fewshot 8`

Original `nq_open`:

```
hf (pretrained=mistralai/Mistral-7B-v0.1), gen_kwargs: (None), limit: None, num_fewshot: 8, batch_size: 1
| Tasks |Version|     Filter      |n-shot|  Metric   |Value |   |Stderr|
|-------|------:|-----------------|-----:|-----------|-----:|---|-----:|
|nq_open|      3|remove_whitespace|     8|exact_match|0.3011|±  |0.0076|
```

New `nq_open`:

```
hf (pretrained=mistralai/Mistral-7B-v0.1), gen_kwargs: (None), limit: None, num_fewshot: 8, batch_size: 1
|  Tasks   |Version|     Filter      |n-shot|  Metric   |Value |   |Stderr|
|----------|------:|-----------------|-----:|-----------|-----:|---|-----:|
|nq_open_v2|      3|remove_whitespace|     8|exact_match|0.2911|±  |0.0076|
```

There is a small drop; shall we leave `nq_open` as-is? I can try to verify this with other models as well.

Let's leave as-is for now (but see if we can come up with a better fix instead)--thanks for checking on this!
Repository: EleutherAI/lm-evaluation-harness
Issue: Eval on SQuAD v2 stopped working
Root cause: Validation Gap

Steps to reproduce the issue: https://colab.research.google.com/drive/152sh0oZNL8GzYQHWoA5QsO4WVfGZ0xN8?usp=sharing

```
!lm_eval --model hf --model_args pretrained=mistralai/Mistral-7B-v0.1,revision=main --tasks squadv2 --device cpu --batch_size 1 --num_fewshot 8
[..]
Downloading readme: 100% 8.18k/8.18k [00:00<00:00, 25.6MB/s]
Downloading data: 100% 16.4M/16.4M [00:00<00:00, 34.0MB/s]
Downloading data: 100% 1.35M/1.35M [00:00<00:00, 11.0MB/s]
Generating train split: 100% 130319/130319 [00:00<00:00, 402958.21 examples/s]
Generating validation split: 100% 11873/11873 [00:00<00:00, 407780.51 examples/s]
2024-02-01:22:22:23,037 WARNING  [evaluator.py:171] Overwriting default num_fewshot of squadv2 from None to 8
Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/content/lm-evaluation-harness/lm_eval/__main__.py", line 240, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/content/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/content/lm-evaluation-harness/lm_eval/evaluator.py", line 174, in simple_evaluate
    task_obj.override_config(key="num_fewshot", value=num_fewshot)
AttributeError: 'SQuAD2' object has no attribute 'override_config'
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Multiple prompts
Root cause: Documentation Deficiency

Is it possible to have multiple prompts for a task and aggregate the results?
If not, do you plan to support this?

Hi, yes, this should be possible either with Promptsource templates or as is done here, for FLAN: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/benchmarks/flan/flan_held_in.yaml
Repository: EleutherAI/lm-evaluation-harness
Issue: Where is fewshot placed in `doc_to_text`?
Root cause: Documentation Deficiency

For example, in MMLU `doc_to_text` default to
```
"{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:"
```

If I set `num_fewshot > 0`, where is the fewshot located? Is it prepended before `doc_to_text`? What if I want to add it at the middle?

My `doc_to_text` for MMLU will be 
```
"User: {{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:
Assistant:"
```
which means fewshot must be put after `User:` and can not be before it. So, how to add fewshot here?

Fewshot will be placed before doc_to_text , with fewshot_delimiter separating every example.


The `description` config field can be used to prepend a string before all fewshot examples!

Note that no delimiter is placed between description and the first fewshot’s doc_to_text.

So, my parameter will be like this?

```yaml
description: "User: "
doc_to_text: "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:
Assistant:"
```

That’s correct!

Thank you. I will close it as completed.
Repository: EleutherAI/lm-evaluation-harness
Issue: Invalid operations: power of a string
Root cause: Validation Gap

This will happen with --limit 1
Upon evaluating a benchmark, I realized that the test case did not permit using only one test case.
that will cause the stderr to return a string 'N/A' if the benchmark is using accuracy.

https://github.com/EleutherAI/lm-evaluation-harness/blob/7411947112117e0339fe207fb620a70bcec22690/lm_eval/evaluator.py#L512-L515

Is that supposed to be a 0, or should it return N/A instead?

There could be an unsupported operation to calculate the power of a string.

https://github.com/EleutherAI/lm-evaluation-harness/blob/7411947112117e0339fe207fb620a70bcec22690/lm_eval/evaluator.py#L537-L543

Returning `"N/A"` is correct, but it seems like we didn't catch all of the issues that that recent change caused. Please open a PR with the improved logic.

Closing as we've handled all such edge cases that I am aware of.
Repository: EleutherAI/lm-evaluation-harness
Issue: ModuleNotFoundError: No module named 'lm_eval.base'
Root cause: External Dependency Breakage

Hi team,

I am trying to run a project repo that depends on lm_eval.base (https://github.com/chancefocus/PIXIU). Even after installing all of the required libraries, I'm getting an error that lm_eval.base is missing. 

I have tried searching online for anyone else having the same issue, but I have not found anyone else having the same issue. 
Now, I'm thinking that 'lm_eval.base' is just a deprecated function that is no longer present in the current lm-evaluation-harness; could someone give me confirmation if this is the case? And if yes, how should I modify the lm_eval.base call for the new function that is supported.

Thank you!

`lm_eval.base`  was at [v0.3.0](https://github.com/EleutherAI/lm-evaluation-harness/tree/v0.3.0) of this package 
and PIXIU has https://github.com/chancefocus/financial-evaluation/tree/a2a3bec98dde558224825efb7931de65768cc1d2/lm_eval as a submodule, having lm_eval.base too.

@LSinev is right, `lm_eval<=0.3.0` uses `lm_eval.base` ! It has been replaced by a number of sub-modules in `lm_eval.api` encompassing the base classes for various components.
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding `hf_transfer` dependency for download
Root cause: Unimplemented Feature Gap

When using `hf_transfer` our download speed goes from ~80MB/s to ~1.8GB/s on AWS. 

hf_transfer comes with almost no dependencies. I would recommend adding the following to main.py or similar, while adding hf_transfer to the dependencies

```python
import os
try:
    import hf_transfer
    os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
except ImportError:
    pass
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the PhysicalQA evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> Next we consider three datasets which attempt to capture physical or scientific reasoning, as distinct from sentence completion, reading comprehension, or broad knowledge question answering. The first, PhysicalQA (PIQA) [BZB+19], asks common sense questions about how the physical world works and is intended as a probe of grounded understanding of the world. GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot (the last measured on PIQA’s test server). This compares favorably to the 79.4% accuracy prior state-of-the-art of a fine-tuned RoBERTa. PIQA shows relatively shallow scaling with model size and is still over 10% worse than human performance, but GPT-3’s few-shot and even zero-shot result outperform the current state-of-the-art. Our analysis flagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark the result with an asterisk. See Section 4 for details.

- [X] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Original data available at the link below.

https://yonatanbisk.com/piqa/data/

implemented in #103
Repository: EleutherAI/lm-evaluation-harness
Issue: log_samples writting issues with multi_GPUs utilization in gsm8k test
Root cause: Documentation Deficiency

The experiment only outputs the sample results from the gpu:0 device. The sample results and accuracy metrics from other GPU devices are not properly written out.

I used the following command to launch the experiment:

accelerate launch --num_machines 1 --num_processes 8 --main_process_port $free_port --multi_gpu --gpu_ids 0,1,2,3,4,5,6,7 --no_python lm-eval 
--model hf 
--model_args pretrained=$1,dtype="bfloat16" 
--tasks gsm8k 
--num_fewshot 5 
--batch_size 8 
--write_out 
--log_samples 
--show_config 
--output_path ./output

The above problem occurs when the number of GPUs is greater than 1 in both 0.4.0 and 0.4.1 versions
![60f0214a-2d83-4db0-8afd-d4a5ac6e3f6a](https://github.com/EleutherAI/lm-evaluation-harness/assets/22727295/67634cee-9e96-477c-bc65-1964eee95da8)

I would appreciate it if you could investigate this issue and provide a solution.

I think you should re-check as `accelerate` chunks the the docs by stride = number of gpus. These are the samples handled by the first rank.  The other docs should be located further down.

> I think you should re-check as `accelerate` chunks the the docs by stride = number of gpus. These are the samples handled by the first rank. The other docs should be located further down.

Right, appreciate your assistance.

Best regards,
Repository: EleutherAI/lm-evaluation-harness
Issue: How to use local datasets in lm-eval?
Root cause: Documentation Deficiency

Hello! I've used some methods to convert a dataset(for example hellaswag) into a different version and I want to test whether the conversion changes the results of model. So I need to use this local dataset to test, and how can I achieve this target? Thanks! 

You may find this useful https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#using-local-datasets.

+1 to @leocnj !

Please let us know if this does not solve your issue or you encounter errors.
Repository: EleutherAI/lm-evaluation-harness
Issue: lm-eval:It is just staying here like this without other outputs.
Root cause: Environment Incompatibility

Why do i get this?
It is just staying here like this without other outputs.
```
(base) root@autodl-container-eab742abf6-e879826f:~/autodl-tmp/Qwen/lm-evaluation-harness# lm-eval --task list
2024-02-07:23:05:40,098 INFO     [utils.py:145] Note: detected 256 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2024-02-07:23:05:40,098 INFO     [utils.py:148] Note: NumExpr detected 256 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-02-07:23:05:40,265 INFO     [config.py:58] PyTorch version 2.1.2 available.
```
I am using `python=3.10.8, torch=2.1.2, cuda=11.8, lm-eval=0.4.1, vllm=0.3.0`

Hi, thank you for reporting this! I quickly tried myself, and wasn't able to replicate this, but there might be something I'm overlooking.

We recently merged a PR that should see pretty noticeable speedups at library startup (#1321 )--what commit of the codebase are you using?

@lintangsutawika , have you observed this recently at all?

> Hi, thank you for reporting this! I quickly tried myself, and wasn't able to replicate this, but there might be something I'm overlooking.您好，感谢您报告此事！我很快就自己尝试了，但无法复制这一点，但可能有一些我忽略的事情。
> 
> We recently merged a PR that should see pretty noticeable speedups at library startup (#1321 )--what commit of the codebase are you using?我们最近合并了一个 PR，它在库启动时应该会看到相当明显的加速（#1321）——您使用的是代码库的哪个提交？
> 
> @lintangsutawika , have you observed this recently at all? ，你最近有观察到这一点吗？

Thank you for answering,I tried on my another computer and had no problem like this.I guess it's because the version of my packages and now this problem has been solved.Thank you all the same!

I’m glad it works out for you now!
Repository: EleutherAI/lm-evaluation-harness
Issue: How does repeat work exactly?
GitHub labels: asking questions

I'm wondering how does repeat work exactly. I've read the doc that says it can be used for majority_voting and I've tried to read the evaluation code but I don't think I understand how it works. Specifically, 

1  Is it using temperature/top-k/beam search generation on the generate() function to get more than 1 solution? 
2  How can we ensure the generated results are different?
3  Would parameters like temperature affect the diversity of repeated generation?

Hi @Some-random !

`repeat` itself just means that you make `repeat` separate calls to the model, using the specified `generation_kwargs`. We assume that one will use, typically, temperature sampling + top-k or top-p to ensure diversity + different model generations, and that these parameters will impact the diversity. In the case of beam search, one could optimize the code further to use num_return_sequences or something similar to not have to perform an entirely separate call to the model to receive `repeat` outputs.

I hope this is helpful!