zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the Adversarial Natural Language Inference (ANLI) evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper:

> Natural Language Inference (NLI) [Fyo00] concerns the ability to understand the relationship between two sentences. In practice, this task is usually structured as a two or three class classification problem where the model classifies whether the second sentence logically follows from the first, contradicts the first santence, or is possibly true (neutral).
> SuperGLUE includes an NLI dataset, RTE, which evaluates the binary version of the task. On RTE, only the largest version of GPT-3 performs convincingly better than random (56%) in any evaluation setting, but in a few-shot setting GPT-3 performs similarly to a single-task fine-tuned BERT Large. We also evaluate on the recently introduced Adversarial Natural Language Inference (ANLI) dataset [NWD+19]. ANLI is a difficult dataset employing a series of adversarially mined natural language inference questions in three rounds (R1, R2, and R3). Similar to RTE, all of our models smaller than GPT-3 perform at almost exactly random chance on ANLI, even in the few-shot setting (∼ 33%), whereas GPT-3 itself shows signs of life on Round 3. Results for ANLI R3 are highlighted in Figure 3.9 and full results for all rounds can be found in Appendix H. These results on both RTE and ANLI suggest that NLI is still a very difficult task for language models and they are only just beginning to show signs of progress.

- [x] Data processing code implemented
- [ ] Evaluation implemented

This should be modeled after the `BoolQ` task in `lm_eval/tasks/suerglue.py`

_Note_: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/anli
Repository: EleutherAI/lm-evaluation-harness
Issue: OSError: You are trying to access a gated repo.
Root cause: Documentation Deficiency

I am trying to run evaluations on a set of models from HF transformers. Some of these repos - such as llama 3.1 8B instruct - are gated models. In transformers I would pass my auth token to the pipeline or the from_pretrained method. 

What is the correct way to pass this token when running lm-harness from the command line?



The easiest way is (from the terminal) `huggingface-cli login`. Otherwise setting an enviorment variable with `export HF_TOKEN=xxx`

Suggestion: please put this in the documentation. I couldn't find it there and I had to search multiple keywords before I finally found this issue in the search results. 

> Suggestion: please put this in the documentation. I couldn't find it there and I had to search multiple keywords before I finally found this issue in the search results. 


I'd second this, definitely useful information that should be in the docs.
Repository: EleutherAI/lm-evaluation-harness
Issue: mgsm tasks not found when using Accelerate
Root cause: Configuration Error

If I try to run mgsm task using accelerate, I have the tasks not found error. But if I run it outside accelerate, it works fine.

Any idea how I can incorporate all tasks with Accelerate?

This is the command I was using:
`   accelerate launch --multi_gpu --num_processes 4 -m lm_eval
      model: hf
      model_args: pretrained=${model}
      tasks: mgsm_direct
      device: cuda
      batch_size: auto`

Hi! not sure what the underlying cause could be. Can you run it with `--verbosity DEBUG` and see if any `mgsm` specific warning/error shows up. I would also double check the environments. 

If I use `tasks: mgsm_direct`, it does not work but if I separate the tasks like this, tasks: `mgsm_en_cot_bn,mgsm_en_cot_de,mgsm_en_cot_en` , it accepts the task but throws this error:

`TypeError: TaskConfig.__init__() got an unexpected keyword argument 'group'`
I'm using the main branch, so it should have tags instead if groups if I understand correctly.

> If I use tasks: mgsm_direct, it does not work but if I separate the tasks like this, tasks: mgsm_en_cot_bn,mgsm_en_cot_de,mgsm_en_cot_en , it accepts the task but throws this error:

> TypeError: TaskConfig.__init__() got an unexpected keyword argument 'group'

I am getting the same error when running with vllm.


@jmkuebler we can replace `group` with `tag` in the yaml file and that works - I think it is because of the change from groups top tags recently in lm-eval-harness
Repository: EleutherAI/lm-evaluation-harness
Issue: Can we get the detailed results of a task?
Root cause: Documentation Deficiency

Hi~, thanks for your great contributions!
Existing code seems to report the final results (EM, F1 Score, or others) for the task. But I'm wondering how can I get the detailed results that which sample is correct and which sample is wrong?
Does existing code support this? I have tried to set `output_path`, but the output file just saves the final score.
Looking forward to your reply. Thank you so much!

Hi!~, Has your issue been resolved? I'm encountering the same problem. If you've found a solution, could you please share it with me? Thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: Multilingual StoryCloze
Root cause: Unimplemented Feature Gap

From @todpole3's recent [multilingual autoregressive modeling paper](https://arxiv.org/abs/2112.10668).

> We professionally translated the StoryCloze dataset (Spring 2016 version) into 10 typologically diverse non-English languages (ru, zh Simplified, es Latin America, ar, hi, id, te, sw, eu, my).

The dataset is currently being reviewed for release but the anticipated release is in January.

Just want to follow up here that we now have an official link to download this dataset: https://dl.fbaipublicfiles.com/xstorycloze.zip

Previously we passed on the dataset to authors via email communication. And hope that has unblocked things!

> Just want to follow up here that we now have an official link to download this dataset: https://dl.fbaipublicfiles.com/xstorycloze.zip
> 
> Previously we passed on the dataset to authors via email communication. And hope that has unblocked things!

Thank you very much. I’m sad I didn’t make your distribution list though… I had chalked this up to get another thing that never got released lol.

Thanks and sorry about the previous miscommunication!

@StellaAthena I'm working on this dataset (and more multilingual datasets) at https://github.com/EleutherAI/lm-evaluation-harness/pull/426. I've evaluated models on XStoryCloze previously with my own code, so I have other results to compare with. Maybe we should create issues and add them to the project so that others know that I'm working on them.
Repository: EleutherAI/lm-evaluation-harness
Issue: Can I use lm-eval for training?
GitHub labels: asking questions

Hi, I appreciate your solid work. Since there are many tasks, I wonder if you can provide code to train on these datasets.

You can take a look to the Bagel dataset, incorporates plenty of the tasks train sets:
https://huggingface.co/datasets/jondurbin/bagel-v0.5

just keep in mind that tests and validation splits are not to be used for training purposes, thats contamination.

> You can take a look to the Bagel dataset, incorporates plenty of the tasks train sets: https://huggingface.co/datasets/jondurbin/bagel-v0.5
> 
> just keep in mind that tests and validation splits are not to be used for training purposes, thats contamination.

Hi [fblgit](https://github.com/fblgit), thank you for your answer. You are so kind, but bagel cannot meet our needs. We hope to perform separate training on each task instead of mixing multiple datasets to improve model performance. Therefore, we hope to have a general dataset processing framework to do this.

A. Use datasets.map to split it out. --> fastest way imho.
B. U can go thru the citations of Bagel where each Dataset is cited and available in HF hub so u can access it separately.
C. U can check on `lm-eval/tasks/$folder` each task has a YAML with the dataset/split used.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluation openai model?
Root cause: Documentation Deficiency

I want to evaluate a model like gpt 3.5 turbo. How can I evaluate it?

I wonder if it is possible to evaluate API models that are not local models. Can you tell me the command?

Hey, the documentation for this is here: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/API_guide.md

Example command: `lm_eval --model openai-chatcompletions --model_args model=gpt-4o-mini --tasks gsm8k`
Repository: EleutherAI/lm-evaluation-harness
Issue: vllm mode shows much slow inference speed than normal (hf) mode.
Root cause: Configuration Error

below evaluation is vllm settings for llama3.2 evaluation

```

lm_eval --model vllm \
    --model_args pretrained=/home/jovyan/data-vol-1/models/meta-llama__Llama3.2-1B-Instruct,dtype=auto,gpu_memory_utilization=0.8 \
    --tasks leaderboard\
    --batch_size auto\
    --output_path result/meta-llama__Llama3.2-1B-Instruct.json


Processed prompts:  14%|███▉                         | 20791/152671 [15:38<1:23:11, **26.42it/s, est. speed input: 24555.79 toks/s,** output: 22.14 toks/s]

```

below evaluation is hf settings for llama3.2 evaluation

```
lm_eval --model hf \
    --model_args pretrained=/home/jovyan/data-vol-1/models/meta-llama__Llama3.2-1B-Instruct,dtype=auto \
    --tasks leaderboard\
    --batch_size auto\
    --output_path result/meta-llama__Llama3.2-1B-Instruct_hf.json

Passed argument batch_size = auto:1. Detecting largest batch size
Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
Determined largest batch size: 4
Running loglikelihood requests:   8%|█████▊                                                                    | 12011/152671 [01:23<11:20, 206.83it/s]

```


This is shows not only loglikelihood request, but also generation request.
Is this wrong with my evaluation setting?. I using A100 80GB single-GPU.

Thanks for help



Hi, I met the same problem that using vllm is even slower than the normal hf model. Have you got any ideas?

Is the setting for `max_model_len` the same?

+1
Repository: EleutherAI/lm-evaluation-harness
Issue: Question related to how to use the validation and training  splits.
Root cause: Documentation Deficiency

Hello, I would like to know how I can use the validation split to evaluate the models, and similarly, how to use the training split for evaluation if needed. I haven't found an option where the user can specify the dataset split they want to use for model evaluation. Could you provide guidance on how to set this up?
thank you

Hi! You can switch up the sets used in the task yamls. We use the test split if provided, otherwise the validation split is used. Example: https://github.com/EleutherAI/lm-evaluation-harness/blob/1185e89a044618b5adc6f0b9363b629a19fffdc4/lm_eval/tasks/arc/arc_easy.yaml#L7-L9

If a `fewshot_split` is not provided, then the priority is training > val > test to extract the fewshot examples.

I see. So the validation set is never used when both the training and test sets are present. Is the Open LM leaderboard following the same approach?
Thank you.




> I see. So the validation set is never used when both the training and test sets are present. Is the Open LM leaderboard following the same approach? Thank you.

yes!
Repository: EleutherAI/lm-evaluation-harness
Issue: Llama3.1-8B-Instruct evaluation fails 
Root cause: Validation Gap

Running the following command with Llama-3.1-8B-Instruct fails with a `AttributeError: 'function' object has no attribute 'pad_token'` error. I am using the adding_all_changess branch to replicate the leaderboard results. Any help is greatly appreciated! 

```
lm_eval \
--output_path ./outputs \
--model hf \
--tasks leaderboard_gpqa \
--model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16 \
--batch_size auto \
--use_cache ./cache \
--log_samples \
--trust_remote_code \
--verbosity DEBUG
```

Hi! might be able to help if you provide the full stack trace.
Repository: EleutherAI/lm-evaluation-harness
Issue: GPU with GGFU LLM 
Root cause: Documentation Deficiency

Hi,
I am running some accuracy tests with quantized models. I run the following commands but it takes a long time to run a small test and apparently NVIDIA GPU is not utilized while its available:

python -c "import torch; print(torch.cuda.is_available())"
True

python -m llama_cpp.server --model ./model/Yi-Coder-1.5B-Chat-Q4_0_4_4.gguf --host 0.0.0.0
lm_eval --model gguf --tasks arc_challenge --output_path ./model/yi/ --device cuda:0 --model_args model=./model/Yi-Coder-1.5B-Chat-Q4_0_4_4.gguf,base_url=http://localhost:8000

Any guidance will be appreciated. Thanks!

Hi again,
I have found this issue resolving the slowness of running gguf models using cuda: https://github.com/EleutherAI/lm-evaluation-harness/issues/1437#issuecomment-2085638559
I have tried the same method too. But no GPU is utilized on my system.

Is there any additional information needed from my side? I would appreciate any guidance on the matter. Thanks for your time and support!

i meet same issue, have any solutions?

I have the same issue. @pasky @orsharir @kailashbuki 

Same issue
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Support for Untrusted Certificates in local-completion
Root cause: Unimplemented Feature Gap

When querying an endpoint that has a self-signed TLS certificate the local-completion API model will crash with:

```python
  File "/var/home/core/.local/lib/python3.9/site-packages/lm_eval/models/api_models.py", line 335, in model_call
    response = requests.post(
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /v1/chat/completions (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))
```

This issue can be solved by amending the argument `verify=False` to the requests call here:

https://github.com/EleutherAI/lm-evaluation-harness/blob/7882043b4ee1ef9577b829809c2f4970b0bdba91/lm_eval/models/api_models.py#L335-L345

If desired, the request argument can be dependent on a model_arg such as `insecure=True` or `request_args=verify=True`. Let me know what is acceptable and I am happy to open a PR.

Hi! Thanks, a PR will be appreciated! I think something like `verify_certificate` might be more clear.
Repository: EleutherAI/lm-evaluation-harness
Issue: Improve preprocessing for paws-x and xnli tasks
Root cause: Unimplemented Feature Gap

Some tasks such as paws-x and xnli do not have any kind of data preprocessing. For instance, some examples of few-shot to illustrate this problem are: 

- paws_en: _" Teams had to use a traditional tribal club from a distance to hit the pots ., right? No, The teams had to use a traditional tribal club from a distance to hit the pots .”_
- xnli_en: _" And he said, Mama, I'm home., right? Yes, He called his mom as soon as the school bus dropped him off.”_

In the spanish_bench we have included some preprocessing in the utils.py file in order to remove extra whitespaces, remove the final punctuation mark in the first sentence, lowercase the first letter in the second sentence (to be used after "Yes, ..." or “No, …”). With this preprocessing, the Spanish instances have the form:

- paws_es: _“Los equipos tuvieron que usar un club tribal tradicional desde la distancia para golpear los botes, ¿verdad? Sí, los equipos tuvieron que usar un club tribal tradicional desde la distancia para golpear los botes.”_
- xnli_es: _"Y él dijo: Mamá, estoy en casa, ¿correcto? Sí, llamó a su madre tan pronto como el autobús escolar lo dejó.”_

Maybe it could be useful to include these preprocessing techniques for the languages that follow the same rules (Spanish, English, French, German).
Repository: EleutherAI/lm-evaluation-harness
Issue: Wrong format of the few-shot examples in mgsm_direct tasks
Root cause: Configuration Error

In the 'mgsm_direct' task, the examples given in the few-shot to the model include the whole CoT reasoning, while the model is expected to output just the number corresponding to the answer. For instance, see the example below for ‘mgsm_direct_en’.

_"Question :  If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? \nAnswer : There are 3 cars in the beginning, 2 more arrive, so now there should be 3 + 2 = 5 cars. The answer is 5.”_

We believe that for the direct task, the few-shot instances should only contain the answer number instead of the whole CoT. So for the previous example, it should be:

_"Question :  If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? \nAnswer : 5.”_

This occurs because the current yaml is including the whole CoT answer as the 'doc_to_target' for the few-shot examples.

`doc_to_target: '{% if answer is not none %}{{answer[21:]}}{% else %}{{answer_number|string}}{% endif %}'`

For the Catalan, Spanish, and Basque variants of mgsm (tasks inside the catalan_bench, spanish_bench, and basque_bench), the YAML configurations have been modified so that the few-shot examples contain only the final number, in line with the expected output of the task. We think it would make sense to apply this change to all 'mgsm_direct' tasks to maintain consistency between the few-shot examples and the task objective.

`doc_to_target: '{{answer_number|string}}'`
Repository: EleutherAI/lm-evaluation-harness
Issue: Why is using vLLM via lm-eval-harness slower than using vLLM directly?
Root cause: Documentation Deficiency

Why is inference more than twice as fast when using vLLM directly compared to using vLLM via lm-evaluation-harness?
<img width="1214" alt="Snipaste_2024-10-30_21-38-11" src="https://github.com/user-attachments/assets/37267f88-3b84-4c57-b721-86dfe721536b">
<img width="1205" alt="Snipaste_2024-10-30_21-40-20" src="https://github.com/user-attachments/assets/6f7c4018-5d64-4416-979f-2e1cb598acd0">


Hi! are you using `--batch_size auto`? that works best with vllm

@baberabb Oh, I forgot about that. I’ll give it a try soon—thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: Few Shot configurations for multilingual tasks
Root cause: Documentation Deficiency

For multilingual arc, the [original implementation](https://github.com/nlp-uoregon/mlmm-evaluation/blob/main/lm_eval/tasks/multilingual_arc.py) has 25 shots but in lm_eval, it doesn't
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/okapi/arc_multilingual/_arc_yaml

Same is the case for mmlu_multilingual, [original implementation](https://github.com/nlp-uoregon/mlmm-evaluation/blob/main/lm_eval/tasks/multilingual_arc.py) has 25 shots and lm_eval has zero shots: 
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/okapi/mmlu_multilingual/_default_yaml


Hi! you can set the fewshots with `--num_fewshots 25`!
Repository: EleutherAI/lm-evaluation-harness
Issue: how to compute tasks' metrics by their sub-tasks
Root cause: Documentation Deficiency

As the picture shows below, the lm-evaluation-harness computes the metrics of sub-tasks (such as leaderboard_gpqa_diamond/extended/main), how can i get the metrics of leaderboard_gpqa?
![issue](https://github.com/user-attachments/assets/5f53d5b4-485a-4b82-964b-83ca8f32d248)

There are other examples, I want know the metrics of bbh, but I got only the sub-tasks score.
![image](https://github.com/user-attachments/assets/8ef1a5da-4bae-4d6f-9496-7f9978990248)



Hugging Face Blog discusses the idea behind normalization of  scores -
https://huggingface.co/spaces/open-llm-leaderboard/blog

The exact method and scripts can be found here
https://huggingface.co/docs/leaderboards/en/open_llm_leaderboard/normalization
Repository: EleutherAI/lm-evaluation-harness
Issue: AssertionError: Error: number of fewshot samples requested exceeds the 8 that are available.
Root cause: Documentation Deficiency

When I try to use **few-shots** in CommonsenseQA, I wrote a YAML file in this location [[I'm the YAML file]](https://github.com/Here1sWqW/seeeeee/blob/main/commonsense_qa_cot_8shot.yaml). After that, I used this command [[I'm the command]](https://github.com/Here1sWqW/seeeeee/blob/main/run_llamma3OnCsqa_8shot.sh) to run the process, but it **asserts that the number of few-shot samples requested exceeds the 8 that are available**. If I **do not specify num_fewshots** in the YAML file, **it ignores the examples I included.**


Hi! looks like it should work if you set a `test_split` to the one you want to evaluate. Also need to add the samples in the same format as the original dataset. So for e.g.
```yaml
test_split: validation
fewshot_config:
  sampler: first_n
  samples:
  - question: "What do people use to absorb extra ink from a fountain pen?"
    answerKey: "A"
    choices:
      text: [ "shirt pocket", "calligrapher's hand", "inkwell", "desk drawer", "blotter" ] 
```

> Hi! looks like it should work if you set a `test_split` to the one you want to evaluate. Also need to add the samples in the same format as the original dataset. So for e.g.
> 
> ```yaml
> test_split: validation
> fewshot_config:
>   sampler: first_n
>   samples:
>   - question: "What do people use to absorb extra ink from a fountain pen?"
>     answerKey: "A"
>     choices:
>       text: [ "shirt pocket", "calligrapher's hand", "inkwell", "desk drawer", "blotter" ] 
> ```

Oh! I got it. Thank you so much for your detailed response! Your explanation about setting the test_split and formatting the samples was really helpful. I appreciate your guidance and the example you provided. It clarified a lot for me.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Question]How could I get the same output as when doing the GSM8K task while working on multiple-choice questions?
Root cause: Documentation Deficiency

Hi~, dear maintainer of lm-evaluation-harness.
I encountered an issue while trying to **obtain the output from my model**. Specifically, when the model performs a **multiple-choice task**, the response is limited to:
```json
"resps": [
  [["-28.943561553955078", "False"]], 
  [["-32.30293655395508", "False"]], 
  [["-29.412311553955078", "False"]], 
  [["-30.683795928955078", "False"]], 
  [["-33.34199905395508", "False"]]
], 
"filtered_resps": [
  ["-28.943561553955078", "False"], 
  ["-32.30293655395508", "False"],
   ["-29.412311553955078", "False"], 
  ["-30.683795928955078", "False"], 
  ["-33.34199905395508", "False"]
],
```
However, for tasks like **gsm8k**, the model provides **detailed output**:
```json
"resps": [["Janet eats 3 eggs and bakes 4 eggs, so she uses 3 + 4 = 7 eggs.\nShe lays 16 eggs per day, so she has 16 - 7 = 9 eggs left.\nShe sells 9 eggs at $2 each, so she makes 9 x $2 = $<<9*2=18>>18 per day.\n#### 18"]], 
"filtered_resps": ["18"]
```
I would like to know **how I can obtain similarly detailed output for multiple-choice tasks**. Could you please provide guidance on how to achieve this? Any advice would be greatly appreciated!