zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: (Question) Can we use in-code evaluation through the APIs, instead of using lm-eval in bash terminal?

(Question) Can we use in-code evaluation through the APIs, instead of using lm-eval in bash terminal?

There is an example in the previous issue:
```python
import lm_eval

device:str ='cuda'  # 'cuda' or 'cpu'

lm_eval.tasks.initialize_tasks()
results = lm_eval.simple_evaluate(
    model="hf",
    model_args=f"pretrained={<PATH_TO_HF_REPO>}," + \
        f"tokenizer={<PATH_TO_HF_REPO>}",
    tasks=['mmlu'],
    num_fewshot=0,
    device=device)
```

Is this the API call you are asking for?

Thank you @LSinev for providing the example from the previous issue. Yes, that is the API call I was referring to. I appreciate your help :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Image Dataset
Root cause: Unimplemented Feature Gap

I am wondering if there is a way to evaluate models on image datasets using lm-evaludation-harness. For example, evaluating vision language models on datasets that include both image prompts and text prompts.

You can subscribe and also add ideas to the discussion thread https://github.com/EleutherAI/lm-evaluation-harness/issues/1155

Closing as future discussion will take place in #1155 .
Repository: EleutherAI/lm-evaluation-harness
Issue: How can I evaluate on an output file?
Root cause: Unimplemented Feature Gap

When I run the `lm_eval` with `--log_samples`, I got the output file.
I want to only rerun the metrics on the output file, how can I do?
I searched the issues, however I could not find a way.

Hi! 

We don't currently support this but we really should. Adding it as a TODO!

+1 for this functionality.
Repository: EleutherAI/lm-evaluation-harness
Issue: after install， it  is unknown
Root cause: Environment Incompatibility

susie.sun@yizhu-amd8:~/lm-evaluation-harness$ sudo pip install -e .
[sudo] password for susie.sun:
Obtaining file:///home/susie.sun/lm-evaluation-harness
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: UNKNOWN
  Running setup.py develop for UNKNOWN
Successfully installed UNKNOWN-0.0.0


Check these out:

1. https://github.com/lief-project/LIEF/issues/108
2. https://stackoverflow.com/questions/53077360/pip-install-issue-with-egg-fragments-that-lead-to-an-unknown-installation-of-a-p

And also don't forget to check `python`, `pip`, and `setuptools` versions called with and without sudo (and also check this please https://stackoverflow.com/questions/29310688/sudo-pip-install-vs-pip-install-user).

thanks, pip install setuptools --upgrade  has fixed the issue

And I  want to ask another question. Could lm_eval run on AMD gpu?

You can run lm-eval on AMD GPUs. This is good resource on how to install some of the dependencies that `lm-eval` depends on https://rocm.docs.amd.com/en/latest/how-to/llm-fine-tuning-optimization/model-acceleration-libraries.html
Repository: EleutherAI/lm-evaluation-harness
Issue: After installing lm_eval, can't import and can't run the example lm_eval --model hf etc..
Root cause: Environment Incompatibility

I have installed it successfully.
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/85290523/f66186e6-4358-4cea-8da1-2926d8327d44)
But I can't import it, and if I run the "import lm_eval" code, it will keep for a long time without any output.
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/85290523/83ba93d1-c3d9-4093-a41c-fad31e7af8f5)

the import error
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/85290523/48e68059-a2d4-4b82-a769-ab2c447466aa)


related issue #1071 

> related issue #1071

@gerayking  Thank you very very very very much!!!
I solve the problem by set "export HF_ENDPOINT=https://hf-mirror.com".
I think someone in China wants to eval LLM must set it.
record.
Repository: EleutherAI/lm-evaluation-harness
Issue: Unnecessary installation: Nemo
Root cause: Architectural Constraint

The most recent commit on branch main requires installing nemo even when I'm going to use vllm, I think It's unnecessary.

What do you think, may this be because [this exception](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/nemo_lm.py#L49-L53) is raised every time `lm_eval` used, because `nemo_lm` module [loaded every time](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/__init__.py#L7)?

Any ideas on the solution?
Repository: EleutherAI/lm-evaluation-harness
Issue: understanding context length behaviors
GitHub labels: asking questions

Hi, I have a quick question. What is the behavior of the harness if the input examples exceed the model's sequence length in long-document tasks (and how do few-shot examples influence this)? Thank you!


Hi!

The current (intended) behavior is to simply *left-truncate* so things won't exceed (model max length). 

- for perplexities, no context is discarded--we chunk into nonoverlapping blocks of length (max length), with a BOS token prefixed to the first chunk. This is the same as strided PPL evaluation with a stride of (max length)--we might expose the option for shorter sliding window sizes / strides soon.
- for generative tasks, we truncate to (max length + max tokens to generate) 
- for multiple-choice, and `loglikelihood` type tasks (lambada), we truncate the (context + continuation) to (max length).

(PS. this is all described for HFLM, but the other local model impls are meant to match HF as much as possible in behaviors like this.)

There aren't currently any optimizations we're doing w.r.t intelligently truncating beyond these--because LMs' tokenizers are (not currently) exposed elsewhere to the tasks or construction of string inputs, it's a pain to figure out what would be truncated beforehand and e.g. only provide the max number of shots that fit while maintaining the prefixed task description and fewshot format. Would like to change this in future though.

Would like to improve this behavior in future, or at minimum make it clear via logging when requests are being truncated!

Hope this helps!
Repository: EleutherAI/lm-evaluation-harness
Issue: How do I set a directory for HuggingFace weights to be downloaded to?
Root cause: Documentation Deficiency

Currently, they are being downloaded to my docker container when I'm running vllm, but I'd like to pass a volume/directory so they are persisted. Thanks.

Hi, this can be controlled by the HF_HOME, TRANSFORMERS_CACHE, and HF_DATASETS_CACHE environment variables!
Repository: EleutherAI/lm-evaluation-harness
Issue: Minvera MATH

why is MATH called minvera_math? 
Minerva is a model that happened to use MATH, but it doesn't define the dataset. The current naming makes it hard to find.
https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/minerva_math

Hi! This was done because it uses the 4-shot prompt disclosed by Minerva and uses SymPy to check answer equivalence as described in the Minerva paper.

Will get the `math_*` variant added, to support both!

another question on this - is the exact-match metric now defined in v0.4.2 the same as the maj@1 metric reported in the llama paper? Same for the GSM8K dataset.
Repository: EleutherAI/lm-evaluation-harness
Issue: Different batch sizes give different results
Root cause: Algorithmic Error

```
accelerate launch -m lm_eval --model hf --model_args pretrained=mistralai/Mistral-7B-v0.1 --tasks hellaswag --batch_size 1
```
gives
```
hf (pretrained=mistralai/Mistral-7B-v0.1), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
|  Tasks  |Version|Filter|n-shot| Metric |Value |   |Stderr|
|---------|------:|------|-----:|--------|-----:|---|-----:|
|hellaswag|      1|none  |     0|acc     |0.6128|±  |0.0049|
|         |       |none  |     0|acc_norm|0.8110|±  |0.0039|
```

while

```
accelerate launch -m lm_eval --model hf --model_args pretrained=mistralai/Mistral-7B-v0.1 --tasks hellaswag --batch_size 16
```
gives
```
hf (pretrained=mistralai/Mistral-7B-v0.1), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 16
|  Tasks  |Version|Filter|n-shot| Metric |Value |   |Stderr|
|---------|------:|------|-----:|--------|-----:|---|-----:|
|hellaswag|      1|none  |     0|acc     |0.6124|±  |0.0049|
|         |       |none  |     0|acc_norm|0.8106|±  |0.0039|
```

Here the difference is quite small, but I remember trying some other cases there before where the difference is somewhat big. Why do different batch sizes give different results? Is this the intended behavior?

#1625 has some related discussion! 

TL;DR: full determinism on GPUs can be hard! If you see differences in score greater than this or greater than the stderrs, definitely file another issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: Could lm_eval run on AMD gpu?
Root cause: Documentation Deficiency

Could lm_eval run on AMD gpu?

Yes, HF Transformers and vLLM should both support AMD gpus, the former without any code change, as far as I am aware: https://huggingface.co/blog/huggingface-and-optimum-amd
Repository: EleutherAI/lm-evaluation-harness
Issue: Wrong (no) instruction for QQP evaluation in GLUE
Root cause: Configuration Error

The QQP template does not mention any instruction. This is incorrect I think. 

https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/glue/qqp/default.yaml#L8

Currently, the evaluation prompt would read something like the following (there is also a newline at the beginning, which I don't know should be there or not): 

```

Sentence 1: I want to pursue PhD in Computer Science about social network,what is the open problem in social networks?
Sentence 2: I handle social media for a non-profit. Should I start going to social media networking events? Are there any good ones in the bay area?
Answer: [yes/no]
```

I think the template should be `"Are the two questions paraphrases of each other?\nSentence 1: {{question1}}\nSentence 2: {{question2}}\nAnswer:"`. 

Hi, thanks for catching this! I've made a patch fixing QQP's prompt to match what it was in v0.3.0.
Repository: EleutherAI/lm-evaluation-harness
Issue: What is likelihood-difference in CrowS-Pairs? Is a lower value better for this indicator?


Hi! This refers to the difference in loglikelihood between the stereotyped and non-stereotyped sentences. 

A theoretically "unbiased" model (one that assigns equal probability to the stereotyped and non-stereotyped sentences) would have this value closer or equal to 0.

Got it! Really thank you :)

For further reference, this implementation is sourced from: https://arxiv.org/pdf/2304.01373.pdf
Repository: EleutherAI/lm-evaluation-harness
Issue: BitsandBytesConfig in CLI?
Root cause: Unimplemented Feature Gap

How to add `quantization_config` inside `--model_args`? There is only `load_in_4bit` parameters but I need `llm_int8_skip_modules` parameters and it's not working because the value it's needed is a list of string.

Hi! To pass such nested or complicated argument structures you'd currently have to call `simple_evaluate()` or `cli_evaluate()` programmatically in a python script. Adding the ability to parse such args from the CLI would be very complex for users to remember and prone to misentry/typos.

We'd like to make it possible to pass more complicated arguments, including looping over several settings: #1622 but I don't have an ETA on when we'll be able to add this. 

Closing in the hopes of centralizing related discussion in #1622 ! Feel free to chime in there or share more about desired use cases / ask questions you may have.
Repository: EleutherAI/lm-evaluation-harness
Issue: running error
Root cause: Environment Incompatibility

I pip intall -e . running lm_eval after ld.so reported an error ld.so detected an inconsistency: ... /sysdeps/x86_64/dl-machine.h: 534: elf_machine_rela_relative: assertion `ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE' failed!



lm_eval --tasks list
Inconsistency detected by ld.so: ../sysdeps/x86_64/dl-machine.h: 534: elf_machine_rela_relative: Assertion `ELFW(R_TYPE) (reloc->r_info) == R_X86_64_RELATIVE' failed!


check this https://github.com/boltgolt/howdy/issues/357#issuecomment-1832996571

> 检查这个 [boltgolt/howdy#357 （评论）](https://github.com/boltgolt/howdy/issues/357#issuecomment-1832996571)

Thank you, but I've rebooted and it didn't work.
Repository: EleutherAI/lm-evaluation-harness
Issue: ERROR: file or directory not found: /data/liuhuanbin/code/assessments/lm-evaluation-harness/tests/test_version_stable.py
Root cause: Validation Gap

Hi, I did the following, but got an ERROR: file or directory not found: /data/liuhuanbin/code/assessments/lm-evaluation-harness/tests/test_version_stable.py
git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

lm_eval --model hf \
    --model_args pretrained=~/llama-2/ \
    --tasks minerva_math,gsm8k \
    --gen_kwargs top_k=1 \
    --batch_size auto:4 \
    --output_path result/ \
    --device cuda:0 \
    --limit 10 \
    --check_integrity \
    --log_samples \
    --use_cache cache_db/ \
    --verbosity DEBUG \
    --trust_remote_code \
    --seed 42

Can you help me? Looking forward to your reply.
Thanks！



Hi! 

You should be able to forgo `--check_integrity` and things will run fine (and scores won't be affected). (That flag runs some of our test files, which might have since been updated.)

I will look into fixing the root cause of this problem!
Repository: EleutherAI/lm-evaluation-harness
Issue: HOWTO: Perform evaluation that requires model responses on two inputs
Root cause: Documentation Deficiency

I am wondering how to (or if it is possible to) perform the following evaluation: each instance has two prompts, `p1` and `p2`. The model should be called twice, on `p1` and `p2` respectively, to get the two responses `r1` and `r2`. The scoring metric is a function of the two inputs (e.g., number of overlapping words or edit distance). Is it possible with this library? 

Hi!

Assuming `r1` and `r2` are generated text, I'd recommend implementing this via a Python `Task` or `ConfigurableTask` subclass. 

You'd have to: 
- Override `construct_requests()` to build the list of requests per dataset instance: one `generate_until` type request with `p1` as input, and one generate_until type request with `p2` as output.
- Override `process_results()` to take the generated `r1` and `r2`, and compute your metric dict per instance with it.

Hope this helps!
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluating multiple choice questions with GPT (OpenAI Chat Completion API)
Root cause: Unimplemented Feature Gap

I am wondering is there a way to evaluate GPT (OpenAI Chat Completion API) on a set of multiple choice questions such as the mmlu tasks. I know that the chat completion api does not support log likelihood, but is there a built-in way that allows getting text/json answers from GPT and then post-process to get multiple choice evaluations?

I was wondering the same thing. Has there been any progress?

> I was wondering the same thing. Has there been any progress?

There is an open PR about it: https://github.com/EleutherAI/lm-evaluation-harness/pull/2601

But I don't see any progress so far 🥲

we really need this feature
Repository: EleutherAI/lm-evaluation-harness
Issue: Anthropic message api not supported
Root cause: Unimplemented Feature Gap

Hi, 

As far as I can see, the anthropic message api is not supported atm, only the legacy api ([compare](https://docs.anthropic.com/claude/docs/upgrading-to-the-messages-api)). This means that lm-eval does not support the claude 3 models.
If no one is working on this atm, I can create a PR for this.

Best,
Albert

Hi! #1594 adds this--let us know if you encounter any difficulties with it!
Repository: EleutherAI/lm-evaluation-harness
Issue: log_samples with multi args in model_args
Root cause: Validation Gap

i obtain an error when lm_eval try to save result in file.
lm_eval uses model_args to do the file name but it don't remove the ':' character 

the error: 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "c:\python311\Scripts\lm_eval.exe\__main__.py", line 7, in <module>
  File "C:\Projects\transpose\lm-evaluation-harness\lm_eval\__main__.py", line 400, in cli_evaluate
    filename.write_text(samples_dumped, encoding="utf-8")
  File "C:\Python311\Lib\pathlib.py", line 1078, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\pathlib.py", line 1044, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 22] Invalid argument: 'output\\mistral-7b-instruct-v0.2-LG\\model__sn__mistral-7b-instruct-v0.2-LG,base_url__http:____10.2.42.198:1234__v1_gsm8k.jsonl'

the command:
lm_eval --model local-chat-completions --tasks gsm8k --model_args model=sn/mistral-7b-instruct-v0.2-LG,base_url=http://10.2.42.198:1234/v1 --log_samples --output_path output/mistral-7b-instruct-v0.2-LG/ --limit 4 


perhaps you can modify :
output_name = "{}_{}".format(
                        re.sub("/|=", "__", args.model_args), task_name
                    )
by :

output_name = "{}_{}".format(
                        re.sub("/|=|:", "__", args.model_args), task_name
                    )


same thing with zeno_visualize.py 

            model_args = re.sub(
                "/|=|:",
                "__",
                json.load(
                    open(Path(args.data_path, model, "results.json"), encoding="utf-8")
                )["config"]["model_args"],

Hello, thank you for raising this issue!

Since you've pointed out a fix for this, perhaps you could create a PR with this fix applied?
Repository: EleutherAI/lm-evaluation-harness
Issue: cannot eval wikitext2
Root cause: External Dependency Breakage

I met the following error when I eval Llama2 7B on wikitext2-raw-v1:
```
Generating test split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1726, in _prepare_split_single
    for key, record in generator:
  File "/root/.cache/huggingface/modules/datasets_modules/datasets/wikitext_document_level/fd2b48221bb0d1e01acb17a9dc22e7278c03f8ff32e9147a39c701d5a7749435/wikitext_document_level.py", line 238, in _generate_examples
    with open(data_file, encoding="utf-8") as f:
  File "/usr/local/lib/python3.10/dist-packages/datasets/streaming.py", line 75, in wrapper
    return function(*args, download_config=download_config, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/datasets/download/streaming_download_manager.py", line 507, in xopen
    return open(main_hop, mode, *args, **kwargs)
NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/94be2a7b3fff32ae7379658c8d3821035b666baddad3a06d29b55ab3a4ab3115/wikitext-2-raw/wiki.test.raw'
```

the run command is:
```
accelerate launch -m lm_eval --model hf --model_args pretrained="/workspace/model/Llama-2-7b-chat-hf",dtype="bfloat16" --task wikitext
```

Looking forward to your reply.

Hello, is there any update?

Hi! The original hosting for Wikitext is no longer accessible. Looking into making the zip files directly in that HF repo, or else sourcing from somewhere else such as https://huggingface.co/datasets/wikitext/tree/main . 

Hi, I've updated the wikitext dataset on Huggingface to use the new hosted location of the wikitext files: https://huggingface.co/datasets/EleutherAI/wikitext_document_level/commit/3ad9e13102af0ea3a47d6b0907694d908f01be05. 

Let me know if there are any problems!

@haileyschoelkopf I still see the issue with loading the dataset:

datasets==2.18.0

```python
from datasets import load_dataset
dataset = load_dataset(
    path='EleutherAI/wikitext_document_level',
    name='wikitext-2-raw-v1',
    data_dir=None,
    cache_dir=None,
    download_mode=None,
)
```

```
ConnectionError: Couldn't reach https://wikitext.smerity.com/wikitext-2-raw-v1.zip (SSLError(MaxRetryError("HTTPSConnectionPool(host='wikitext.smerity.com', port=443): Max retries exceeded with url: /wikitext-2-raw-v1.zip (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')))")))
```

Could you try 
1) entering this url (https://wikitext.smerity.com/wikitext-2-raw-v1.zip) into your browser and seeing if you can access it? Works for me
2) does your machine have internet connection?

Thanks! @haileyschoelkopf. There was some issue with proxy. It works for me now.
Repository: EleutherAI/lm-evaluation-harness
Issue:  id_column contains duplicate values
Root cause: Validation Gap

I tried to send data to zeno visualisator but i have this error:

py ./scripts/zeno_visualize.py --data_path ./output/ --project_name "LG Model Test"
Successfully updated project.
Access your project at  https://hub.zenoml.com/project/460c-8698-59a31942ea41/LG%20Model%20Test_gsm8k
Traceback (most recent call last):
  File "C:\Projects\transpose\lm-evaluation-harness\scripts\zeno_visualize.py", line 219, in <module>
    main()
  File "C:\Projects\transpose\lm-evaluation-harness\scripts\zeno_visualize.py", line 103, in main
    project.upload_dataset(
  File "C:\Python311\Lib\site-packages\zeno_client\client.py", line 117, in upload_dataset
    pa_table = df_to_pa(df, id_column)
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\zeno_client\util.py", line 22, in df_to_pa
    raise ValueError("ERROR: id_column contains duplicate values")
ValueError: ERROR: id_column contains duplicate values


it's because result are duplicate in the log file.

command is:
lm_eval --model local-chat-completions --tasks gsm8k,babi --model_args model=sn/mistral-7b-instruct-v0.2-LG,base_url=http://10.2.42.198:1234/v1 --log_samples --output_path output/mistral-7b-instruct-v0.2-LG --limit 4 

it's concerning only gsm8k task


I got the same error. Did you resolve the issue?

gsm8k contains multiple filters, which the current script doesn't support. I'm working on a PR.

https://github.com/EleutherAI/lm-evaluation-harness/pull/2599
Repository: EleutherAI/lm-evaluation-harness
Issue: [Question] How I can evaluate safety of my LLM model with lm-evaluation-harness
Root cause: Documentation Deficiency

What benchmarks are available in this context ? i.e. toxicity benchmarks.

You can have a look at the current list of benchmarks (or tasks). The format of the list is being worked on in issue #1727 but you could see this [list](https://github.com/Harryalways317/lm-evaluation-harness/blob/main/lm_eval/tasks/tasks.md) and then search the README (under lm_eval/tasks/<task_family> for additional information.