zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: How do I send the api key when using local-chat-completions?
Root cause: Documentation Deficiency

I've notice you can change base_url of the remote host, but how can I set the api_key?

Whomever else finds this issue, api key is specified via `OPENAI_API_KEY` env var (regardless of the chat completions kind)

For example:
```
!OPENAI_API_KEY=<api-key> && lm_eval \
  --model openai-chat-completions \
  --model_args base_url=<api_url>/v1/chat/completions,num_concurrent=4,tokenized_requests=False,model=gpt-4.1-nano\
  --tasks gsm8k \
  --num_fewshot 0 \
  --limit 2 \
  --apply_chat_template
```
Repository: EleutherAI/lm-evaluation-harness
Issue: The response is too short to extract answer on GPQA. What should I set to extend it?
Root cause: Configuration Error

`lm_eval --model local-chat-completions --tasks gpqa_main_cot_zeroshot --model_args model=Qwen/Qwen2-72B-Instruct,base_url=https://api.together.xyz/v1 --output_path ./gpqa/result/Qwen2 --use_cache ./gpqa/cache/Qwen2 --log_samples --limit 10 --gen_kwargs temperature=0.7,max_tokens=8192`
Using this command, The Qwen2's result just end sooo weirdly like the image below
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/109973290/32a2e9b9-c6ec-4a56-bacc-87d214492535)

To be specific, only 256 tokens are generated. I'm wondering why this happens, is there any problem with max_tokens? 

same issue here, maybe try add max_gen_toks: 2048 in _gpqa_cot_zeroshot_yaml:
![image](https://github.com/user-attachments/assets/c2ad3197-cef4-44e3-ba1c-1034a2fc19ad)
Repository: EleutherAI/lm-evaluation-harness
Issue: How to RUN BENCHMARK on GGUF Models ?
Root cause: Documentation Deficiency


Hi!!
I want to run the benchmarks like MMLU or ARC on my gguf converted model.
Does lm-evaluation-harness support doing so?

If Yes then can I get any reference or tutorial to do so?

I really appreciate any help you can provide.
Repository: EleutherAI/lm-evaluation-harness
Issue: How is the score calculated in open-llm-leaderboard
Root cause: Documentation Deficiency

Hi ！ Thank you for your outstanding work.
I noticed that the 'resps' calculated in the choice questions of open-llm-leaderboard is like this: "[ [ "-17.890625", "False" ], [ "-26.65625", "False" ], [ "-19.109375", "False" ] ]". 
Could you please explain the number of the choices?

`docs/model_guide.md` should help to clear up a bit some of the return types of the various request types you can run against an LM!

In this case, each multiple-choice answer string has type `(float, bool)` where the float is the loglikelihood of the model greedily producing that string, and the bool is True if and only if every one of that answer string's tokens would be produced by taking the argmax among the vocabulary probabilities. Here, the length of the list indicates that there are 3 choices that given document can select from.

Hope this clears things up a bit!
Repository: EleutherAI/lm-evaluation-harness
Issue: Same accuracy and std with different seeds
Root cause: Documentation Deficiency

we are trying to check stability of the model using different seeds. But getting same accuracy and std even if seeds are different` lm_eval --model hf --model_args pretrained=<model_path> --tasks mmlu_abstract_algebra --batch_size --seed <some_xyz_number>`. 

Hi!

What are the settings you're running under? If you are running 0-shot, for example, and using multiple-choice or greedy generation without random sampling, then I'd expect results to not change much if at all.

(Stderrs are calculated via bootstrapping, and so are solely dependent on the resulting metric value--they won't capture any notion of variance over prompts, and aren't obtained by running multiple times)

Closing since this is likely expected behavior, but please feel free to reopen with more details if not!
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the RACE evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive, multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset.
> GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI+18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions. On DROP [DWD+19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL+19]. On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper. On RACE [LXL+17], a multiple choice dataset of middle school and high school English examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.

- [X] Data processing code implemented
- [x] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Note: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/race

One big issue with HF's implementation of this dataset: it makes a separate document for each question; meanwhile, in the GPT3 paper it is shown that one document is made per passage.

https://github.com/huggingface/datasets/blob/master/datasets/race/race.py#L106
Repository: EleutherAI/lm-evaluation-harness
Issue: The problem about the overall score of BBH and GPQA datasets
Root cause: Documentation Deficiency

Hi! Thank you very much for your helpful and outstanding work!

The BBH dataset and the GPQA dataset both have multiple subsets, but on the open-llm-leaderboard, what is displayed is the overall score of each model on BBH and GPQA. I wonder how these overall scores are obtained? Are they simply the average of the scores the model achieves on each subset? Or they are the score of the concat of all subsets?
<img width="280" alt="image" src="https://github.com/user-attachments/assets/a0664864-beff-4e35-bc01-4984798ff766">

Thank you very much!

cc @clefourrier @NathanHB for confirmation, but I believe that these aggregate scores on the Leaderboard are obtained by *macro* averaging over the subsets in the task (so just taking per-subset accuracies and averaging them out.)

Scores on multiple-choice tasks are also rescaled so that random guessing (e.g. perhaps 25% accuracy) is 0 in the leaderboard.

@alozowski is writing a doc page on this atm, the PR is [here](https://github.com/huggingface/leaderboards/pull/21)
Repository: EleutherAI/lm-evaluation-harness
Issue: Error when running tinyMMLU 
Root cause: Validation Gap

![image](https://github.com/user-attachments/assets/a2bc8a52-aed8-43aa-952a-b310226bfd9b)

Getting an error - IndexError: index 5 is out of bounds for axis 0 with size 5  when running tinyMMLU on my gguf model.

Does the same fail if you try running using `--model hf` ? 

@haileyschoelkopf  Sorry, I have not tried that my main objective is only for GGUF converted models.
Repository: EleutherAI/lm-evaluation-harness
Issue: Getting "jinja2.exceptions.TemplateError: System role not supported" exception with some tasks using --apply_chat_template
Root cause: Validation Gap

This is happening on the main branch as of time of this writing:

```bash
$ git rev-parse HEAD
1adab703e9320ea23b2a7008ec3377a3a4d82cfd
```

This isn't an issue with all tasks:

```bash
$ lm_eval --model hf \
          --model_args pretrained=UCLA-AGI/Gemma-2-9B-It-SPPO-Iter3,dtype="float",add_bos_token=True \
          --tasks mmlu_clinical_knowledge --batch_size 48 --device mps --apply_chat_template 
[..snip..]
2024-07-16:11:35:26,780 INFO     [task.py:423] Building contexts for mmlu_clinical_knowledge on rank 0...
[..snip..]
Traceback (most recent call last):
  File "/path/to/mmlu-eval/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/path/to/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/path/to/lm_eval/evaluator.py", line 296, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/path/to/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/path/to/lm_eval/evaluator.py", line 412, in evaluate
    task.build_all_requests(
  File "/path/to/lm_eval/api/task.py", line 446, in build_all_requests
    fewshot_ctx = self.fewshot_context(
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/path/to/lm_eval/api/task.py", line 1119, in fewshot_context
    return chat_template(labeled_examples)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/lm_eval/models/huggingface.py", line 1286, in apply_chat_template
    return self.tokenizer.apply_chat_template(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[..snip..]
  File "/path/to/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1899, in raise_exception
    raise TemplateError(message)
jinja2.exceptions.TemplateError: System role not supported
```

Duplication of #2069
Repository: EleutherAI/lm-evaluation-harness
Issue: Citing this repo
GitHub labels: documentation

Hi!

How should we cite this repo in order to give it proper credit incase we use it for academical purposes?

Thanks!
Yuval



I plan on eventually™ having a preprint up about eval harness, but I don't have any promises about when that will happen. 

For now, use the following list (everything after first 2 is alphabetical):

> Gao, Leo and Tow, Jonathan and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and McDonell, Kyle and Muennighoff, Niklas and Phang, Jason and Reynolds, Laria and Tang, Eric and Thite, Anish and Wang, Ben and Wang, Kevin and Zou, Andy


There is now a zenodo DOI, to cite use:

```@software{leo_gao_2021_5371629,
  author       = {Leo Gao and
                  Jonathan Tow and
                  Biderman, Stella and
                  Black, Sid and
                  DiPofi, Anthony and
                  Foster, Charles and
                  Golding, Laurence and
                  Hsu, Jeffrey and
                  McDonell, Kyle and
                  Muennighoff, Niklas and
                  Phang, Jason and
                  Reynolds, Laria and
                  Tang, Eric and
                  Thite, Anish and
                  Wang, Ben and
                  Wang, Kevin and
                  Zou, Andy},
  title        = {A framework for few-shot language model evaluation},
  month        = sep,
  year         = 2021,
  publisher    = {Zenodo},
  version      = {v0.0.1},
  doi          = {10.5281/zenodo.5371628},
  url          = {https://doi.org/10.5281/zenodo.5371628}
}
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Empty --log_samples  outputs
Root cause: Validation Gap

_I checked similar issues_

While working with this tool for machine translation `generate_until` task all calculation give correct output but after doing `--log_samples ` in wandb i see the following where most of the precitions give `null` response. i am using llama-3-8B model.
any idea what the problem is?

<img width="901" alt="Screenshot 2024-07-19 at 08 01 18" src="https://github.com/user-attachments/assets/ef90b381-e74f-45f4-bf73-b569706b7a9b">



Hi! Could you share the command you're running and what the locally logged files look like?

Could you also share the codebase version you are using to run these commands? (e.g. PyPI version number, or a git commit from the main branch)
Repository: EleutherAI/lm-evaluation-harness
Issue: WMT JP/ZH/KO Tokenization
Root cause: Unimplemented Feature Gap

We don't tokenize JP, ZH & co prior to computing the BLEU Score, which means its computed on the entire sentence, as JP & ZH don't use spaces; 

I can fix this when I have some time, just noting it here for now~

essentially we'd need to add the packages for chinese, jap, kor, thai listed somewhere on this page: https://spacy.io/usage/models

& then check for the lang and tokenize it

I'm hesitant to add additional packages just for those langs to do the tokenization (To calculate BLEU score correctly, we need 私はシンガポールが好きです。　 --->  私　は　シンガポール　が　好き　です　。)

I was thinking if it wouldn't be possible to just train a tiny gpt to do the space tokenization; we could then just download it from hug face and use it before calculating the bleu score - Do you that would make sense @leogao2?
I'd love to try it out


That sounds significantly more complicated than just using existing segmentation packages. I think adding more dependencies for segmentation instead is fine.

Even if they're GPL licensed?

Can't we use jieba/nagisa for zh/jp? Both of those are MIT

Yep, but for Korean I was gonna use https://github.com/LuminosoInsight/mecab-ko-dic which requires Mecab which is GPL

KoNLPy is also GPL (https://konlpy.org/en/latest/#license)

Does ko need segmentation? I thought it uses spaces

Hm my knowledge of Korean is unfort. immensely limited, but in [this repo](https://github.com/Muennighoff/hf-translation-benchmark) using `python main.py --model m2m --weights facebook/m2m100_418M --data en-ko --sample 500`
I get 
BLEU: 36.651343611373036 with tokenization & BLEU 2.9324395566566337 without (rmving it in the data folder).

E.g. a sentence like `그래서 이것은 우리가 알고있는 것을 어떻게 알고 있는지에 대한 이야기입니다.` is turned into 
`그래서 이것 은 우리 가 알 고 있 는 것 을 어떻게 알 고 있 는지 에 대한 이야기 입니다 .`

I think tho we can ignore it for now - even though the BLEU may be a bit lower for Korean, relative comparisons should still make sense as long as the BLEU is calculated the same way.

I'll open a PR for Jap & Zh - Other languages like Thai, we can do lateron ad-hoc i think.






Merged in [214](https://github.com/EleutherAI/lm-evaluation-harness/pull/214).
Note: Consider tokenization for Thai in the future when available in WMT & possibly Korean if it's raised as an issue
Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval --model hf --model_args pretrained=/home/lmz/glm-4-9b-chat --tasks /home/lmz/ceval-exam/a.yaml --device cuda:0 --batch_size 8 In this command, can the --tasks option only download the dataset from the hub? Can I use a local dataset?
Root cause: Documentation Deficiency

lm_eval --model hf --model_args pretrained=/home/lmz/glm-4-9b-chat --tasks /home/lmz/ceval-exam/a.yaml --device cuda:0 --batch_size 8 In this command, can the --tasks option only download the dataset from the hub? Can I use a local dataset?
Below are the tasks I listed using lm-eval --tasks list. I found that they are all yaml files, so I also edited the yaml file on my dataset, as shown in the figure below, but an error will be reported.
![截屏2024-07-20 11 41 53](https://github.com/user-attachments/assets/9a516d95-1909-447a-b85a-1a31c8dc4cf6)
![截屏2024-07-20 11 42 34](https://github.com/user-attachments/assets/839fc756-2b79-4303-8cb9-c3cc2a2e76ab)
![截屏2024-07-20 11 43 15](https://github.com/user-attachments/assets/a5b70320-19d2-4bac-985e-695d8dc1e1c6)


I also have the same problem, my environment cannot be connected to the internet, so I want to use a local dataset for evaluation. If you have any progress, I hope you can share it

I have configured VPN on the server and can download the dataset
Repository: EleutherAI/lm-evaluation-harness
Issue: RecursionError: maximum recursion depth exceeded while calling a Python object
Root cause: Validation Gap

When I run the code "CUDA_VISIBLE_DEVICES=3 TRANSFORMERS_OFFLINE=1 lm_eval --model hf  --model_args pretrained=/public/MountData/yaolu/LLM_pretrained/LLAMA2_7B/,trust_remote_code=True  --tasks mmlu,cmmlu  --device cuda:0  --batch_size auto  --num_fewshot 0" for llama2-7b, it reports "RecursionError: maximum recursion depth exceeded while calling a Python object". However, it works well on llama3-8b and baichuan2-7b? So how can I fix this problem.

By the way, the version of lm-eval is 0.4.3

What Llama 2 download is this?

Could you post the *entire traceback?*
Repository: EleutherAI/lm-evaluation-harness
Issue: ValueError when task name collide with local directory names
Root cause: Validation Gap

# Steps to reproduce
```bash
$ cd lm-eval-harness
$ pip install -e .[vllm]
$ mkdir hellaswag
$ lm-eval --tasks hellaswag --model vllm --model_args pretrained=deepseek-ai/deepseek-coder-1.3b-instruct,dtype=float16,tensor_parallel_size=1 --limit 5
```

Observed error:
```
ubuntu:~/evalharness$ lm-eval --tasks hellaswag --model vllm --model_args pretrained=deepseek-ai/deepseek-coder-1.3b-instruct,dtype=float16,tensor_parallel_size=1 --limit 5
/usr/lib/python3/dist-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.25.2
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
2024-07-20 13:09:49.266468: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-07-20 13:09:49.309275: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX512F AVX512_VNNI, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/lm-eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/ubuntu/evalharness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/ubuntu/evalharness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/evalharness/lm_eval/evaluator.py", line 163, in simple_evaluate
    raise ValueError(
ValueError: No tasks specified, or no tasks found. Please verify the task names.
```

# Why this (maybe) matters

The way I came across this bug was when I tried to run log `mmlu` samples to a folder called `mmlu`. This seems like a normal thing to try to do, and the error could be confusing.

# Mitigations

I think it would be nice if there was just a clearer error message for when this happens.

# Code points
- https://github.com/EleutherAI/lm-evaluation-harness/blob/feff1b55c57993c4d42c8f913a22eeec395cd690/lm_eval/__main__.py#L333

Hi! Thanks for opening this, definitely high-priority to fix. Thought I'd handled some other edge cases relating to tasks shadowing other tasks but evidently not this one.--am away at a conference but can try to push a fix ASAP.



This issue is still not resolved. 

As a workaround, rename your local directory to something that doesn't conflict with any task name in lm_eval. That fixed it for me.
Repository: EleutherAI/lm-evaluation-harness
Issue: Error "trust_remote_code=True"
Root cause: Configuration Error

lm_eval --model_args pretrained=/data/hf_model/$model,dtype="float16" \
        --tasks wsc,winogrande \
        --batch_size 1 \
        --trust_remote_code

ValueError: The repository for winogrande contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/winogrande.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.
what should i do?

How did you resolve this issue?

I would also like to know

@lyx-hush @TommyClemenzaChen  @erick648  Try this: 
`export HF_DATASETS_TRUST_REMOTE_CODE=1`
Repository: EleutherAI/lm-evaluation-harness
Issue: Unrecognized argument for --apply_chat_template
Root cause: Unimplemented Feature Gap

`(xenv-0017-phi3) (xenv-0017) aginart@ip-10-1-123-117:~/dev$ lm_eval --model hf     --model_args pretrained=EleutherAI/gpt-j-6B,trust_remote_code=True     --tasks hellaswag     --device cuda:0     --batch_size=8 --apply_chat_template
2024-07-23:01:51:15,897 INFO     [utils.py:148] Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-07-23:01:51:15,897 INFO     [utils.py:161] NumExpr defaulting to 8 threads.
2024-07-23:01:51:16,268 INFO     [config.py:58] PyTorch version 2.2.0 available.
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-12.1'
usage: lm_eval [-h] [--model MODEL] [--tasks task1,task2] [--model_args MODEL_ARGS] [--num_fewshot N] [--batch_size auto|auto:N|N] [--max_batch_size N] [--device DEVICE] [--output_path DIR|DIR/file.json]
               [--limit N|0<N<1] [--use_cache DIR] [--decontamination_ngrams_path DECONTAMINATION_NGRAMS_PATH] [--check_integrity] [--write_out] [--log_samples] [--show_config] [--include_path DIR]
               [--gen_kwargs GEN_KWARGS] [--verbosity CRITICAL|ERROR|WARNING|INFO|DEBUG]
lm_eval: error: unrecognized arguments: --apply_chat_template`


- Clone master
- run basic command with apply_chat_tempalte
- Somehow arg is unrecognized?

hi did you solve this?
Repository: EleutherAI/lm-evaluation-harness
Issue: getting error with llama3-1-8B
Root cause: External Dependency Breakage

i tried the new llama but i got error .
here is the command
`lm_eval --model hf \
    --model_args pretrained=meta-llama/Meta-Llama-3.1-8B \
    --tasks leaderboard_gpqa \
    --device cuda:0 \
    --num_fewshot 0\
    --batch_size auto:2`
 and the error last lines
`lib/python3.12/site-packages/transformers/models/llama/configuration_llama.py", line 182, in _rope_scaling_validation
    raise ValueError(
ValueError: `rope_scaling` must be a dictionary with two fields, `type` and `factor`, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}
`




You may want to check if `transformers` is of latest version and also if it has similar issue

thank you i update the transformers and it seems working
Repository: EleutherAI/lm-evaluation-harness
Issue: choices of verbosity in simple_evaluate
Root cause: Documentation Deficiency

In simple_evaluate, what are the choices for verbosity? It's not mentioned in the function documentation. Currently, it is at "INFO" by default. Is there a verbosity level below this? INFO level prints out the model, progress bar etc. 

did you solve the issue? i am facing a similar issue where I prefer not to have verbose because too unwanted logs

You can try these levels: https://docs.python.org/3.8/library/logging.html#levels
`ERROR` in particular