zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Prompt Engineering
Root cause: Unimplemented Feature Gap

Is there any way to add a prompt template as input while using a dataset on a specified model for inference?
I would like to experiment with different prompt templates and do extensive prompt engineering.

You may be interested in the fork we collaborated on with BigScience Workshop: https://github.com/bigscience-workshop/lm-evaluation-harness , which allows for writing prompt templates in Promptsource and using them for evaluation!


We are currently in the middle of a redesign of this library in the background, and prompt templating is one of the features we intend to have first-class support for going forward. That redesign is currently slated to be usable and be a new version release within the next month, tentatively!

@haileyschoelkopf Are there any updates on this? Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Create task groups
Root cause: Unimplemented Feature Gap

Create task groups to run grouped tasks together. All the tasks that have more than one config should be grouped. There are some tasks with many different configurations and we have to write them individually. These groups could be defined in the same file where the task registry is defined. I have been doing it in bash, but I think it should be integrated into the code. Some examples:

```bash
glue=cola,mnli,mnli_mismatched,mrpc,rte,qnli,qqp,sst,wnli
superglue=boolq,cb,copa,multirc,record,wic,wsc
lambada_openai_mt=lambada_openai_mt_en, lambada_openai_mt_de,lambada_openai_mt_es,lambada_openai_mt_fr,lambada_openai_mt_it
```

An argument could be added to the main function to specify task groups:
```bash
python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=EleutherAI/gpt-j-6b,peft=nomic-ai/gpt4all-j-lora \
    --tasks_groups glue,superglue,lambada_openai_mt \
    --device cuda:0
```
    
We could also create groups by task type (question answering, machine-translation, common sense reasoning...). Examples:

```bash
question_answering=triviaqa,squad2,truthfulqa_mc,webqs
machine_translation_gpt3=wmt14-en-fr,wmt14-fr-en,wmt16-de-en,wmt16-en-de,wmt16-en-ro,wmt16-ro-en
common_sense_reasoning=boolq,piqa,winogrande,hellaswag,arc_easy,arc_challenge,openbookqa,copa,wsc273
```


Hi, thanks for opening an issue! I think that this makes sense to add in some capacity. 

Have you used the support for wildcards in task names in the command line? For example, you can run all of `lambada_openai_mt` via `--task lambada_openai_mt_*`. 

This would not apply in all cases (certainly, running all of glue or superglue together seems relevant) but would doing something like 
- adding the ability to run all tasks of a given type
- Retaining wildcard support as is currently used
- Adding aliases for glue and superglue component tasks such that the aliases match the expression `glue_*` or `superglue_*` 

Suffice for all use cases you're requesting?

Hi, thanks! I didn't know that wildcards are supported, that solves most of the issues. I think it should be added to the README.

Implementing aliases is also a good idea. Aliases could also be defined for the last examples I give: `qa`, `mt`, `csr`. More examples of task types that are used: Language Modeling, Reading Comprehension, Mathematical Reasoning, and Safety/Alignment. The results of each task type are usually reported together on papers, so it makes sense to evaluate them together.

I’ve added a note about this to the readme under “advanced usage” along with a link to the BigScience repo for prompted evals https://github.com/EleutherAI/lm-evaluation-harness/commit/14043a0f762702bbb356073f50c066bcd9b275b2

This was merged in https://github.com/EleutherAI/lm-evaluation-harness/pull/612 and can also be done via the `groups` field in task YAMLs, both on `big-refactor` branch. 

We will better document this feature prior to version release of `big-refactor`.
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding a new task failed
Root cause: Validation Gap

我尝试添加一个新任务，名为cblue_qic，但是报错了。
具体错误：
Traceback (most recent call last):
  File "/opt/conda/envs/lmeval/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/lmeval/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/nas/LLM/lm-evaluation-harness/scripts/write_out.py", line 6, in <module>
    from lm_eval import tasks
  File "/root/nas/LLM/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 56, in <module>
    from . import cblue_qic
  File "/root/nas/LLM/lm-evaluation-harness/lm_eval/tasks/cblue_qic.py", line 24, in <module>
    import datasets.cblue_qic.qic
ModuleNotFoundError: No module named 'datasets.cblue_qic'

我在lm-evaluation-harness/lm_eval/tasks/cblue_qic.py文件中写了：
import sys
sys.path.append("..")
import datasets.cblue_qic.qic

我在lm-evaluation-harness/lm_eval/datasets/cblue_qic文件夹下仿照其他任务写好了qic.py和__init__.py。
![Y2HW8{_S3RKN JY}KZ HS7O](https://user-images.githubusercontent.com/65701007/234043660-1918077f-b893-462a-9fd5-b1ef42479b20.png)


Hi! Do you think you could share the code you've written somewhere, so I can take a look at your implementation?

In particular, are you using ` DATASET_PATH = inspect.getfile(lm_eval.datasets.cblue_qic.qic)` in your `cblue_qic.py` task file? 

Additionally, can you share more about how you are installing the repository and how you are attempting to run evaluation?

Thank you. Problem solved.
I didn't import it as required in `lm-evaluation-harness/lm_eval/tasks/cblue_qic.py`
before:
`import sys`
`sys.path.append("..")`
`import datasets.cblue_qic.qic`

after:
`import sys`
`import lm_eval.datasets.cblue_qic.qic`

And then the problem was solved

Awesome!!
Repository: EleutherAI/lm-evaluation-harness
Issue: How are questions longer then the context window handled for gpt2/hf-causal style models (specifically in QASPER)? 
Root cause: Documentation Deficiency


Hi, I believe that extra-long questions are left-truncated to the size of models' max length! I will look into this, as I'm not very familiar with the QASPER task. This is also something we might want to be configurable in future.

They are left-truncated, meaning that only the last n_ctx tokens are actually fed into the model. This is standard practice with decoder-only language models, and implicitly encouraged by their design. I’m less confident that it’s standard practice for encoder-decoder models, but making this model-dependant is a huge can of worms I don’t want to open.

Thanks
Repository: EleutherAI/lm-evaluation-harness
Issue: ValueError: invalid literal for int() with base 10: 'cuda:2'
Root cause: Validation Gap

when I runing the main.py script, the below error raised, it seems like there is different between doc demonstraction and current code.

my raised Exception
> Traceback (most recent call last):
  File "/home/yuyaxiong/work/lm-evaluation-harness/main.py", line 108, in <module>
    main()
  File "/home/yuyaxiong/work/lm-evaluation-harness/main.py", line 79, in main
    results = evaluator.simple_evaluate(
  File "/home/yuyaxiong/work/lm-evaluation-harness/lm_eval/utils.py", line 182, in _wrapper
    return fn(*args, **kwargs)
  File "/home/yuyaxiong/work/lm-evaluation-harness/lm_eval/evaluator.py", line 64, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/yuyaxiong/work/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/yuyaxiong/work/lm-evaluation-harness/lm_eval/models/gpt2.py", line 28, in __init__
    device = int(device)
ValueError: invalid literal for int() with base 10: 'cuda:2'

command
```bash
python main.py \ 
    --model hf-causal \  
    --model_args pretrained=EleutherAI/gpt-j-6B \  
    --tasks lambada_*,hellaswag \  
    --device cuda:0  
```

The documentation mentions the string format of the `--device` (e.g. `cuda:0`). `--model hf-causal-experimental` mode uses this format, but `--model hf-causal` for some reason expects an integer, e.g., this should work:  

```
python main.py \ 
    --model hf-causal \  
    --model_args pretrained=EleutherAI/gpt-j-6B \  
    --tasks lambada_*,hellaswag \  
    --device 0
```

This is a bug, though, I reckon. All modes should use string format (e.g., "cuda:0").

Can confirm using 0 instead of cuda:0 works

Thanks for noting this. As @oshev says, we should be expecting  `"cuda:0"` as the input for all modes. This is a bug and is being added to the to-do list.

Tried a fix for this issue - [PR Link](https://github.com/EleutherAI/lm-evaluation-harness/pull/447)

Great work! Thanks.
Repository: EleutherAI/lm-evaluation-harness
Issue: huggingface model warning
Root cause: Validation Gap

hi, I meet the warning when I evaluate the Pythia model in gsm8k.

```
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
```

Does this affect the model performance?


I test the pythia2.8b in gsm8k, and get the results 0 :
![image](https://user-images.githubusercontent.com/42565075/234855648-dba74561-7ec0-4713-9be5-aa06aa1fe8bd.png)
Is there something wrong with this？


@StellaAthena Raised PR with a potential fix for this issue - [PR Link](https://github.com/EleutherAI/lm-evaluation-harness/pull/448)
The pre-commit check seems to be failing despite the small change any help there would be highly appreciated.
The build check also seems to have canceled itself.

 [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/448) should fix issue
Repository: EleutherAI/lm-evaluation-harness
Issue: Any plans for MMLU and BBH?
Root cause: Documentation Deficiency

I think it would be pretty simple to add MMLU and BBH datasets to LM eval. Any reason it is not added yet?

https://huggingface.co/datasets/lukaemon/mmlu
https://huggingface.co/datasets/lukaemon/bbh

Maybe I can do it? 

MMLU is already implemented, https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hendrycks_test.py. There is a PR to add BBH: https://github.com/EleutherAI/lm-evaluation-harness/pull/406

I see thanks
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation on training set?
Root cause: Documentation Deficiency

Hi, thanks for your nice project. 
I have some  questions:



1 Some tasks (e.g., arc) download the training set, **Is the training set used to evaluate the model?** 
![image](https://user-images.githubusercontent.com/42565075/235071128-ec9a11e6-5f92-49b9-a3c9-0a2a81cb50f6.png)


2 In addition, I ran the ARC-easy, and I find that the evaluation includes 9496 steps. 
![image](https://user-images.githubusercontent.com/42565075/235073146-15b7ff43-f7ce-47ad-8a5e-10b983e266b2.png)


However, in the task table, the `Val/Test Docs` of arc-easy is 2376. 

I also find in the hugging face, the whole arc-easy includes 5.2k instances.
![image](https://user-images.githubusercontent.com/42565075/235071904-23b43553-cfa8-4e52-bf41-0affd2f14dc0.png)

**I am wondering where is the step 9496 from.**



Hi! The behavior is to evaluate on the test set if it exists, and on the validation set if it does not. Train set is used for fewshot examples in context.

Here, this is evaluating on the test set of ai2_arc ARC-Easy, but each document must use one forward pass per answer choice (with 5 answer choices in each doc.) And so this leads to the 2.38k * 5 ~= 9.5k forward passes through the model.

Hi, thanks for your explanation. However, why it is `~=` rather than `=`

Sorry, meant to say 4 possible choices -> 2376 * 4 = 9504 ! The reason I said `~=` was because in your screenshot, the number of requests is off by 8 compared to 9504, which should be because if we have a request where the arguments fully match, then we don't run on that request twice in our code. (so it appears there are 8 duplicated requests here it seems.)

Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding model generation_kwargs as input argument?
Root cause: Unimplemented Feature Gap

Hi! I just discovered this repo and think it's amazing that someone has built something so complete and open.

I wanted to ask a question about making `generation_kwargs` either an input argument or allowing additional `kwargs` to be added to it. Take, for example,

https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/models/gpt2.py#L112

Here, if someone wanted to use a different decoding/sampling strategy, it would be super useful to have the ability to alter `generation_kwargs`.

Is such a feature worth adding? If so, how would one go about adding it? We'd be happy to implement it if (1) we knew that it would be useful and if (2) we had some guidance on the right way to go about doing so.

Seems like a useful feature, I think you could just add a `generation_kwargs` argument to `HFLM.__init__()`?

Welcome! We are currently in the midst of a major internal revamp that will make many things like this substantially easier. You can check out the new internals here https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor

@StellaAthena thank you! At the risk of asking an annoying question, where might I find documentation for the revamp or information on the timeline? I'm happy to put in the work to add this feature, but I don't know what's the right approach. Would here make sense?

https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/lm_eval/models/gpt2.py#L111-L114

@RylanSchaeffer that is a question for @haileyschoelkopf. You can also drop by our discord server where most of the work is coordinated.

Thank you! Just messaged Hailey on Discord

Closing because this is implemented in `big-refactor`, and that branch should be used for this feature. We won't be implementing this in `master` before the new version release of `big-refactor`.
Repository: EleutherAI/lm-evaluation-harness
Issue: Running evaluation on already downloaded models
Root cause: Documentation Deficiency

I may have overlooked it, but I cannot understand how to evaluate an already downloaded local model rather than having to download from huggingface again. 

Can someone point to what command I use to give it a path to the model or if this is even possible? (I'd rather not have to redownload my 30-ish models just to run evaluations on them...)

python main.py \
    --model hf-causal \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000 \
    --tasks lambada_openai,hellaswag \
    --device cuda:0

pretrained改为本地模型地址

Replace `--model_args pretrained=EleutherAI/pythia-160m,revision=step100000 ` with `--model_args pretrained=path/to/files`. The `--model_args` are fed directly into the HF constructor, and you can provide any arguments you like to the HF constructor that way
Repository: EleutherAI/lm-evaluation-harness
Issue: Do I have to download the data set every time I run the model
Root cause: Documentation Deficiency

Do I have to download the data set every time I run the model, can I put the data set locally

It should cache the dataset automatically. Do you have the caching disabled?

Does the current framework only automatically cache data sets, not download them in advance?

It downloads them before running requests, so as to not bottleneck evaluation by download speeds. I do not believe we currently offer the ability to pre-run downloads without running evals, that would be interesting. What usecase makes that desirable?
Repository: EleutherAI/lm-evaluation-harness
Issue: To evaluate (causal) models across multiple GPUs
Root cause: Documentation Deficiency

To evaluate (causal) models across multiple GPUs, use `--model hf-causal-experimental`
But there is no such key `hf-causal-experimental` in MODEL_REGISTRY
`MODEL_REGISTRY = {
    "hf": gpt2.HFLM,
    "hf-causal": huggingface.AutoCausalLM,
    "hf-seq2seq": huggingface.AutoSeq2SeqLM,
    "gpt2": gpt2.GPT2LM,
    "gpt3": gpt3.GPT3LM,
    "textsynth": textsynth.TextSynthLM,
    "dummy": dummy.DummyLM,
}`

Hi, are you on the most up to date commit of the codebase? This was a recent change.

How to evaluate models across multiple GPUs, I do not see the model is distributed for multiple gpus in main.py?

python -u main.py \
    --model hf-xx \
    --model_args pretrained=/xx/chatglm-6b,use_accelerate=True \
    --tasks xx \
    --device cuda:0 \
    --no_cache，
用`use_accelerate=True`即可使用多gpu

> 如何评估跨多个 GPU 的模型，我在 main.py 中没有看到模型是为多个 gpus 分布的？
Repository: EleutherAI/lm-evaluation-harness
Issue: AttributeError: module 'importlib.resources' has no attribute 'files'
Root cause: Environment Incompatibility

# Description

The following error appears using Python versions lower than 3.9 even though the requirements in the setup say that the minimum Python version is 3.6:

```
Traceback (most recent call last):
  File "main.py", line 6, in <module>
    from lm_eval import tasks, evaluator
  File "/home/angel/Documents/projects/bsc/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 314, in <module>
    **bigbench.create_all_tasks(),
  File "/home/angel/Documents/projects/bsc/lm-evaluation-harness/lm_eval/tasks/bigbench.py", line 232, in create_all_tasks
    resources_dir = importlib.resources.files("lm_eval.datasets") / "bigbench_resources"
AttributeError: module 'importlib.resources' has no attribute 'files'
```

# Current solution:

I have substituted the following line in lm_eval/tasks/bigbench.py:

`resources_dir = importlib.resources.files("lm_eval.datasets") / "bigbench_resources" `

for the following one:

` resources_dir = os.path.join(os.path.dirname(__file__), "../datasets/bigbench_resources")`

obtaining the same result.



I believe this issue is no longer relevant as the project now requires Python 3.9.

Closing as we've revamped this part of the codebase making this issue irrelevant.
Repository: EleutherAI/lm-evaluation-harness
Issue: Python 3.9 Support
Root cause: Environment Incompatibility

Recently the Python requirement was changed to 3.9 from 3.6. The package is not installing via pip. A try except block for the Python version should be added. 

I also encounter the same issue, what is going on..?

import setuptools

with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name="lm_eval",
    version="0.3.0",
    author="Leo Gao",
    author_email="lg@eleuther.ai",
    description="A framework for evaluating autoregressive language models",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/EleutherAI/lm-evaluation-harness",
    packages=setuptools.find_packages(),
    classifiers=[
        "Development Status :: 3 - Alpha",
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires=">=3.9",
    install_requires=[
        "datasets>=2.0.0",
        "jsonlines",
        "numexpr",
        "openai>=0.6.4",
        "omegaconf>=2.2",
        "peft>=0.2.0",
        "pybind11>=2.6.2",
        "pycountry",
        "pytablewriter",
        "rouge-score>=0.0.4",
        "sacrebleu==1.5.0",
        "scikit-learn>=0.24.1",
        "sqlitedict",
        "torch>=1.7",
        "tqdm-multiprocess",
        "transformers>=4.1",
        "zstandard",
        "accelerate>=0.17.1",
    ],
    extras_require={
        "dev": ["black", "flake8", "pre-commit", "pytest", "pytest-cov"],
        "multilingual": ["nagisa>=0.2.7", "jieba>=0.42.1"],
        "sentencepiece": ["sentencepiece>=0.1.98", "protobuf>=4.22.1"],
        "auto-gptq": ["auto-gptq[triton] @ git+https://github.com/PanQiWei/AutoGPTQ"],
        "anthropic": ["anthropic"],
    },
)


3.9 should work now


why i still encounter errors while install lm-eval on python>=3.9

```bash
ERROR: Package 'lm-eval' requires a different Python: 3.8.10 not in '>=3.9'
```

I have python version 3.10 , getting same error
Repository: EleutherAI/lm-evaluation-harness
Issue: 这个框架可以评测ChatGLM吗
Root cause: Documentation Deficiency

这个框架可以评测ChatGLM吗，如果可以`--model`参数要选哪个，选择`--model hf-causal`吗

Hi! ChatGLM is supported in Huggingface I believe (https://huggingface.co/THUDM/chatglm-6b ), so it should work with `hf-causal`. If it does not work, then feel free to reopen this!
Repository: EleutherAI/lm-evaluation-harness
Issue: Single True Multiple Choice Evaluation Metrics
Root cause: Documentation Deficiency

I followed the guide and template of the Single True Multiple Choice task but the return metric is acc. How to modify it for other metrics such as F1 score. I did not find any guide about it. Should I use the normal task template?

Hi, thanks for your interest! The proper way to do this would be to subclass `MultipleChoiceTask` but in place of `process_results(self, doc, results)` to implement the additional logic for that metric. In the case of F1 score, you'd also have to implement a custom aggregation function to calculate f1 score based on the overall precision and recall on your dataset.

Let me know if this helps!

@QishengL Hey, do you have any follow up questions? Was Hailey’s response helpful?

Yes. Thanks, Hailey and Stella.
Repository: EleutherAI/lm-evaluation-harness
Issue: Pulling loglikelihood tasks from cache crashes
Root cause: Validation Gap

Pulling loglikelihood tasks from the cache currently crashes. To reproduce:

`python -m main.py --model hf-causal-experimental --model_args pretrained=EleutherAI/pythia-410m-deduped --tasks winogrande`

Then, run again:

`python -m main.py --model hf-causal-experimental --model_args pretrained=EleutherAI/pythia-410m-deduped --tasks winogrande`

```
  File "C:\Users\Owner\jeffq\lm-evaluation-harness\lm_eval\base.py", line 257, in _loglikelihood_tokens
    _, context_enc, continuation_enc = re_ord.get_reordered()[0]
IndexError: list index out of range
```

I believe this is caused by #394. When pulling from the cache, `re_ord.get_reordered()` is empty since there are no new requests to run.

Thank you for the PR!
Repository: EleutherAI/lm-evaluation-harness
Issue: Summarization tasks
Root cause: Unimplemented Feature Gap

Hello, 
There is plenty of summarization tasks and [datasets](https://paperswithcode.com/datasets?task=text-summarization&page=1) out there. 
Am I missing something or are there no summarization tasks among the ones supported? 
Thank you. 

You are correct that there are currently no summarization tasks supported. If you'd like to open PRs for summarization tasks you use we would greatly appreciate it.
Repository: EleutherAI/lm-evaluation-harness
Issue: 请问这个框架目前可以运行生成任务吗，运行greedy_until时函数报错
Root cause: Interface Contract Mismatch

在运行wmt20-zh-en之类的生成任务时报错。
运行到`Running greedy_until requests`，之后就报错，报错如下：
### 报错信息
Traceback (most recent call last):
  File "/root/nas/LLM/lm-evaluation-harness/main.py", line 118, in <module>
    main()
  File "/root/nas/LLM/lm-evaluation-harness/main.py", line 86, in main
    results = evaluator.simple_evaluate(
  File "/root/nas/LLM/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/root/nas/LLM/lm-evaluation-harness/lm_eval/evaluator.py", line 86, in simple_evaluate
    results = evaluate(
  File "/root/nas/LLM/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/root/nas/LLM/lm-evaluation-harness/lm_eval/evaluator.py", line 247, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/root/nas/LLM/lm-evaluation-harness/lm_eval/models/huggingface.py", line 377, **in greedy_until**
    stop_sequences = request_args["stop_sequences"]
**TypeError: list indices must be integers or slices, not str**

我应该怎么解决这个问题

Closing as solved in v0.4.0 onward, alongside an improvement to our stop sequence handling in general.