zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the DROP evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive, multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset.
> GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI+18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions. On DROP [DWD+19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL+19]. On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper. On RACE [LXL+17], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.

- [X] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Note: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/drop

added in ec4d361514ec0779e502460a705bd875cc11299c , ended up not using HuggingFace's implementation since they only included answer spans as labels, leaving out a ton of labels

Also, should we prepend 'Passage: ' for each passage? It seems OA did not do this, not sure if it was intentional. 

My vote is “no” because I see no reason to do so.

@anishthite I don't see a line for this dataset in [lm_eval/tasks/__init__.py](https://github.com/EleutherAI/lm_evaluation_harness/blob/master/lm_eval/tasks/__init__.py). Am I missing something, or is this not quite finished?

@jon-tow How is the work on this coming?
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the ASDiv Evaluation
Root cause: Unimplemented Feature Gap

> We present ASDiv (Academia Sinica Diverse MWP Dataset), a diverse (in terms of both language patterns and problem types) English math word problem (MWP) corpus for evaluating the capability of various MWP solvers. Existing MWP corpora for studying AI progress remain limited either in language usage patterns or in problem types. We thus present a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem types taught in elementary school. Each MWP is annotated with its problem type and grade level (for indicating the level of difficulty). Furthermore, we propose a metric to measure the lexicon usage diversity of a given MWP corpus, and demonstrate that ASDiv is more diverse than existing corpora. Experiments show that our proposed corpus reflects the true capability of MWP solvers more faithfully.


https://www.aclweb.org/anthology/2020.acl-main.92/
https://github.com/chaochun/nlu-asdiv-dataset

@rokosbasilisk Has there been any progress on this?

> @rokosbasilisk Has there been any progress on this?

https://github.com/EleutherAI/lm-evaluation-harness/pull/244

> > @rokosbasilisk Has there been any progress on this?
> 
> #244

Great! I’ve linked the issue and pinged Leo to rereview
Repository: EleutherAI/lm-evaluation-harness
Issue: OpenaiCompletionsLM invokes the completions API with max_tokens set to 0
Root cause: Validation Gap

As per the title, the completions API is invoked with max_tokens = 0, which, if properly interpreted by the server, will cause it not to generate anything (according to the [API documentation](https://platform.openai.com/docs/api-reference/completions/create#completions-create-max_tokens), that value defaults to 16 and there is no indication that a value of 0 has any special meaning).

It seems the value of self._max_gen_toks (via the **max_gen_toks** property) should have been used instead.

I have the same issue when using the local completions model (vLLM with OpenAI server) I am getting the following error:
`openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': 'max_tokens must be at least 1, got 0.', 'type': 'BadRequestError', 'param': None, 'code': 400}`

Additionally, the `logprobs` parameter is set to `10` even though the maximum value as per the OpenAI documentation is `5`.
https://github.com/EleutherAI/lm-evaluation-harness/blob/070d31df4edad2d8263b64b3ace7ec0cbaeb7ae8/lm_eval/models/openai_completions.py#L215 
https://platform.openai.com/docs/api-reference/completions/create#completions-create-logprobs
Repository: EleutherAI/lm-evaluation-harness
Issue: Load sentencepiece tokenizer for evaluation
Root cause: External Dependency Breakage

Hi,

My tokenizer was trained using Google's [sentencepiece](https://github.com/google/sentencepiece/) library which produces `.model` and `.vocab` file. lm-eval requires a `.json` file in the ` transformers.AutoTokenizer.from_pretrained` in [this line](https://github.com/EleutherAI/lm-evaluation-harness/blob/b4cd85d406938f94ee5d451840a0d69bbda27006/lm_eval/models/huggingface.py#L640).
Few discussions suggested to convert [spm tokenizer to hf](https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326/5) and use `save_pretrained` [function](https://discuss.huggingface.co/t/loading-sentencepiece-tokenizer/20814/3).
However, these suggestions are not working.

I am getting the following error while running **without** passing json file

`OSError: trained-tokenizer does not appear to have a file named config.json`

Hi! I think this may be a HF tokenizers issue--have you tried asking them, or in the HF discord?

We do also support passing in an already-initialized HF tokenizer object, which may help--but still requires loading the tokenizer into a HF Tokenizer class. So you could use the `from_spm` suggestion, and pass that tokenizer into `HFLM(tokenizer=my_spm_initialized_tok)`. Hope this helps.

Thanks Hailey for the reply. We have resolved the issue now. Documenting the solution here.

The issue is with [this line](https://github.com/huggingface/transformers/blob/6bd511a45a58eb02bd59cf447141a2af428747a4/src/transformers/convert_slow_tokenizer.py#L578) which was treating BPE as a unigram tokenizer.

We modified the [SPMConverter class](https://github.com/huggingface/transformers/blob/6bd511a45a58eb02bd59cf447141a2af428747a4/src/transformers/convert_slow_tokenizer.py#L544) to ensure that BPE tokenizer goes to the proper conditional statement.
Repository: EleutherAI/lm-evaluation-harness
Issue: Fewshot seed only set when overriding num_fewshot
Root cause: Algorithmic Error

- #1616 introduces a configurable seed for few shot sampling.
- However, the random seed is only set when `simple_evaluate` is called with `num_fewshot` to override the task default.
- Shouldn't the seed also be set for tasks that set a default `num_fewshot`? (E.g. in the YAML file)

I would appreciate if someone could confirm whether this behavior is correct? :pray: 

https://github.com/EleutherAI/lm-evaluation-harness/blob/14221c84aba34560515822d2aff136ef47a43c71/lm_eval/evaluator.py#L234-L244

You're correct, good catch!

Would you be willing to PR a fix for this? Simply moving it after the `num_fewshot` conditionals should suffice.
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation for MegatronT5 Model
Root cause: Unimplemented Feature Gap

Hi I have noticed that there are existing support for some nemo models. But it does not seem that there is a support for MegatronT5 Model. Anyone has ideas how to evaluate this model?

Hi! We don't currently support this, though would welcome contributions adding it.

Perhaps @sergiopperez has input on how simple it would be to extend the current NeMo integration into Megatron-T5?

Hi any updates on how to evaluate T5? Could you please give some clue on how to extend to Megatron-T5?

> Hi any updates on how to evaluate T5? Could you please give some clue on how to extend to Megatron-T5?

If you are interested in evaluating a pretrained model such as [NeMo Megatron T5](https://huggingface.co/nvidia/nemo-megatron-t5-3B) that has been released on Hugging Face, the Hugging Face version is compatible with this library. However we still lack support for models native to the Megatron library.

Closing as NeMo Megatron-T5 support won't be prioritized, though we'd welcome a contribution for it.
Repository: EleutherAI/lm-evaluation-harness
Issue: social_iqa choices do not use actual answers 
Root cause: Configuration Error

Looking at the logged samples 

{
    "doc_id": 0,
    "doc": {
      "context": "Tracy didn't go home that evening and resisted Riley's attacks.",
      "question": "What does Tracy need to do before this?",
      "answerA": "make a new plan",
      "answerB": "Go home and see Riley",
      "answerC": "Find somewhere to go",
      "label": "3"
    },
    "target": 2,
    "arguments": [
      [
        "Q: Tracy didn't go home that evening and resisted Riley's attacks. What does Tracy need to do before this?\nA:",
        " {{answerA}}"
      ],
      [
        "Q: Tracy didn't go home that evening and resisted Riley's attacks. What does Tracy need to do before this?\nA:",
        " {{answerB}}"
      ],
      [
        "Q: Tracy didn't go home that evening and resisted Riley's attacks. What does Tracy need to do before this?\nA:",
        " {{answerC}}"
      ]
    ]
    
    We can see that Siqa doesn't use the answers properly. How can this be fixed? This is the yaml file
    
task: social_iqa
dataset_path: social_i_qa
dataset_name: null
output_type: multiple_choice
training_split: train
validation_split: validation
doc_to_text: "Q: {{context}} {{question}}\nA:"
target_delimiter: " "
doc_to_choice:
  - "{{answerA}}"
  - "{{answerB}}"
  - "{{answerC}}"
doc_to_target: "{{ (label|int) - 1 }}"
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 0.0
  
  @haileyschoelkopf This can explain Accuracy not matched for Llama1-7b #1294 for Siqa. Since the choices are like {{answerC}}, the model is equally likely to predict any one of the choices leading ~0.33 acc.

switching to 

`doc_to_choice: "{{[answerA, answerB, answerC]}}"` I believe should resolve this.

Yes, it does. Thank you very much.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement XQuAD
Root cause: Unimplemented Feature Gap

https://github.com/deepmind/xquad

Can i implement XQuAD?

Hi, any updates on whether anyone is attempting to work on this, if not, can i take it up?

@Sanchit-404 yes, feel free to take this. Also note, we've gone through a refactor so you might want to check the `main` branch for documentation.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add TensorRT-LLM support
Root cause: Unimplemented Feature Gap

I am trying to run the benchmarking on an Nvidia Orin 64GB machine due to lack of GPU resources, but it is too slow, so I would appreciate it if you could apply TensorRT-LLM. 🤣

Hello!

We don't currently support TRT-LLM, though we do support VLLM which should improve over HF performance.

We'd however welcome a contribution adding TRT-LLM!

Hello, would be greatly appreciated here as well!

Seems to work if you change `apply_chat_template` in [`lm_eval/models/api_models.py:347`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/api_models.py#L347) from `[{**item, "type": "text"} for item in chat_history]` to `[{**item} for item in chat_history]`

Tested with `deepseek-ai/DeepSeek-R1-0528`, `nvcr.io#nvidia/tensorrt-llm/release:1.1.0rc2.post2` on 8xH200
Repository: EleutherAI/lm-evaluation-harness
Issue: accuracy precision
Root cause: Documentation Deficiency

Hi, running the evaluation I get acc with 1 decimal digit and I was looking for a way to see more precise results.

```{'results': {'arc_easy': {'acc,none': 0.3, 'acc_stderr,none': 0.15275252316519464, 'acc_norm,none': 0.1, 'acc_norm_stderr,none': 0.09999999999999999, 'alias': 'arc_easy'}}```


I am running this:
```model="hf"
task_name = ['arc_easy']
limit=10
model_args = 'pretrained=EleutherAI/pythia-14m,dtype=float32,device=cpu'
bootstrap_iters=0

from lm_eval import tasks, evaluator, utils, api
tasks.initialize_tasks()

e1 = evaluator.simple_evaluate(
        model=model,
        tasks=task_name,
        limit=limit,
        model_args=model_args,
        bootstrap_iters=bootstrap_iters,
    )
```

Is there a way to show more digits after the decimal point?

I found the problem - although I think it should be better documented:
limit=10 causes this issue.
but this setting appears in the example https://github.com/EleutherAI/lm-evaluation-harness/blob/main/examples/lm-eval-overview.ipynb and it is not pointed there what it means and what are the outcomes.
It only appears as a warning in the log: "--limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT."

"PR welcome" ;)


Hi @lernerjenny , `--limit 10` causes only the first 10 documents to be evaluated (and for this reason, is only recommended for debugging purposes). The reason only one digit is therefore reported is because it's only possible to get an accuracy with 0.1 increments when you have 10 docs--we don't right-pad with zeros in this case.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add New Benchmark 
Root cause: Documentation Deficiency

Hi, 
I want to ask a question in case if I want to add This benchmark [Alghafa](https://gitlab.com/tiiuae/alghafa/-/tree/main) to the llm-eval framework, but I chose only the datasets that have been used with [Arabic leaderboard](https://huggingface.co/blog/leaderboard-arabic) which are 9 datasets, hellaswag_ar has already added to the framework. 
I have some questions  @haileyschoelkopf:  
    1- if I didn't use all the data in the original paper can we still call The Alghafa benchmark? if not can I put all datasets under one group  
    2- should I put all chosen datasets in one task folder or a different folder for each task, or doesn't really matter?
 
I already prepared the data to use with the llm-eval framework, I hope to answer me as fast as possible.

1. @khalil-Hennara you could put the specific Alghafa subtasks in `tasks/alghafa/` then write an additional benchmark yaml (could be placed in `tasks/benchmarks/`) for the Arabic Learderboard.
2. It's just a matter of personal preference and association. Subtasks that make up SuperGLUE are originally individual tasks but here we just group them under the `superglue` directory. If you feel thematically it would be easier to put all tasks in 1 directory than that would be fine.

Ok thanks I will do it, as you mention
Repository: EleutherAI/lm-evaluation-harness
Issue: output_path may break postprocessing
Root cause: Architectural Constraint

after recent updates (~few weeks ago) Evaluation Tracker has been introduced and also the have been a change in storing outputs. Now outputs_path is not the FINAL destination, where the outputs are stored. There is one mode (sub) directory is created. This causes some problems:
- postprocessing becomes harder. There is one more dir inside output_path. You do not now its name till end of running the code. Or you have to include the code for finding model_name_sanitized inside of your processing script. Once this logic for defining model_name_sanitized changes, all scripts will fail
- there could be more errors for users, when someone uses the same output_path for running two different models. This way two diff subfolders are created
- if I accidentaly run the same lm_eval (for the same model and tasks + output_path), the files are not overwritten. The names now contain the exact time. So, there would be 2 copies of the same file with no reason

What are the future plans for using `model_source` param of  Evaluation Tracker?

https://github.com/EleutherAI/lm-evaluation-harness/issues/1842

Same issue

Hi @artemorloff!
- I have addressed problems raised here and in #1842 in the new PR #1926, where I have moved functions for handling the new results files to utils, so that they can be conveniently used in other parts of the codebase. For example, the Zeno script or the EvaluationTracker class.
- In general, it's now better to just provide the results path without the model name, and for each model a separate directory will be created inside, making it easier not to accidentally mix model results.
- Yes, now results won't be overwritten, and this is done intentionally, so it's harder to lose results. Some users prefer this option, e.g. [here](https://github.com/EleutherAI/lm-evaluation-harness/pull/1766#issuecomment-2146858265)

> if I accidentaly run the same lm_eval (for the same model and tasks + output_path), the files are not overwritten. The names now contain the exact time. So, there would be 2 copies of the same file with no reason

If you don't desire this behavior, you can use the --use_cache flag to avoid rerunning tasks that are already run and have results stored.
Repository: EleutherAI/lm-evaluation-harness
Issue: build commit_id=b281b09, I cannot find lm-eval command.
Root cause: Documentation Deficiency

according to the open llm leaderboard's ABOUT tag。
now the data in the board come from llm-evaluation-harness commit_id=b281b09。
but when i download the project, execute the command: “pip istall -e .”
i cannot find lm-eval command.but when i build the master branch, everything is ok.
why?


b281b09 is rather old branch (seems like 0.3.0 with some patches), check documentation of that state to find out the ways to run it.
This is an example from that branch and commit, no lm-eval available (many new api and models are not available too)
```
python main.py \
    --model hf-causal \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks lambada_openai,hellaswag \
    --device cuda:0
```
Repository: EleutherAI/lm-evaluation-harness
Issue: OOM Issue
Root cause: Resource Mishandling

Hi! I am running evaluations but keep getting OOM errors. Here is my script:

```bash
TASKS="mmlu"
BATCH_SIZE=1
NUM_SHOTS=5


MODEL=Qwen/Qwen1.5-4B
API=vllm
lm_eval \
    --model ${API} \
    --model_args pretrained=${MODEL},dtype="float",gpu_memory_utilization=0.6,max_model_len=1024 \
    --tasks ${TASKS} \
    --device cuda:0 \
    --batch_size ${BATCH_SIZE} \
    --num_fewshot ${NUM_SHOTS} \
    --trust_remote_code \
```

I am using a 80GB A100. I have already decreased the `gpu_memory_utilization` and `max_model_len`, but the problem persists. I have tried Llama3-8b with the same hyperparameters and everything was just fine. Can anyone please tell me why this happens and how I can solve it? Thanks!

https://github.com/EleutherAI/lm-evaluation-harness/issues/1894 check this one too


Try running nvidia-smi to see which processes are using up your gpu memory. 

not sure why but my vram usage does not seem to imply that gpu_memory_utilization is being respected - i also have issues running a 7B model on an A40 (40GB vram) despite having no issues in using the HF backend (~28gb vram utilized there). specifically: it OOMs even before running any requests. Have verified that there is nothing else running on the GPU.

However, passing `enforce_eager=True` in model_args into the vllm backend fixes this issue for me by preventing cuda graphs being built.

@johnwee1 when you pass `enforce_eager=True` to vllm, does it seem then like gpu_memory_utilization is being respected?

@johnwee1 confirmed on Discord that when using `enforce_eager=True`, vllm seems to respect the `gpu_memory_utilization` value. Am intending to create an FAQ docs page, and will include this fact when I do
Repository: EleutherAI/lm-evaluation-harness
Issue: --hf_hub_log_args causes IndexError
Root cause: Validation Gap

Brief description:

 Trying to upload to HF causes the script to fail at the end (after running evaluations). The results are still saved in file, I just don't see the formatted results printed on the terminal. Removing the --hf_hub_log_args flag however fixes this issue. could it be due to recent changes? i just updated yesterday.

logfile:
```
Checking cached requests: 100%|███████████████████████████████████| 317/317 [00:00<00:00, 755.55it/s]



Running generate_until requests: 0it [00:00, ?it/s]

Running generate_until requests: 0it [03:34, ?it/s]

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/home/jovyan/lm-evaluation-harness/lm_eval/__main__.py", line 448, in <module>

    cli_evaluate()

  File "/home/jovyan/lm-evaluation-harness/lm_eval/__main__.py", line 432, in cli_evaluate

    evaluation_tracker.recreate_metadata_card()

  File "/home/jovyan/lm-evaluation-harness/lm_eval/loggers/evaluation_tracker.py", line 472, in recreate_metadata_card

    last_results_file = [

                        ^

IndexError: list index out of range
```


@haileyschoelkopf I think this can be closed now. From what I’ve checked, it’s related to the change of the extension from .json to .jsonl in the result files, which prevented the function from finding them - already fixed here: https://github.com/EleutherAI/lm-evaluation-harness/blob/2b26690fe5d06551030031116d073656c8dc0f33/lm_eval/utils.py#L162-L166

Thanks for tracking this @KonradSzafer !
Repository: EleutherAI/lm-evaluation-harness
Issue: Parallel GPU evaluation using simple_evaluate /evaluate functions?
Root cause: Unimplemented Feature Gap

I want to evaluate my models directly within python (and not via command line), by using the  [simple_evaluate function](https://github.com/EleutherAI/lm-evaluation-harness/blob/40f5458f57f2acee1a2ba86aed16c8a37d99fffc/lm_eval/evaluator.py#L42). Is there functionality to parallelize this function across multiple GPUs?
Repository: EleutherAI/lm-evaluation-harness
Issue: Parallel GPU evaluation using simple_evaluate /evaluate functions? #1934
Root cause: Documentation Deficiency

I want to evaluate my models directly within python (and not via command line), by using the [simple_evaluate function](https://github.com/EleutherAI/lm-evaluation-harness/blob/40f5458f57f2acee1a2ba86aed16c8a37d99fffc/lm_eval/evaluator.py#L42). Is there functionality to parallelize this function across multiple GPUs?

Hi! By passing `parallelize=True` to model args / to the `lm_eval.models.huggingface.HFLM` constructor, you can run within a single python script while parallelizing your model across several devices. VLLM can also be made to use multiple GPUs while only running a single Python script.

For small models, this can be slower than data parallelism using `accelerate` on the command line, however.

Hi @haileyschoelkopf , thank you for the explaination.
From the code of [HFLM](https://github.com/EleutherAI/lm-evaluation-harness/blob/6ccd520f3fb2b5d74c6f14c05f9d189521424719/lm_eval/models/huggingface.py#L101), I noticed that `parallelize=True` only works for models passed via model name.
If I have a model customized from huggingface and parallelized with `accelerate.accelerator.prepare` for multiple nodes/GPUs, how can I integrate it with `HFLM` and `simple_evaluate`? Does it work by directly passing the AutoModel to the `HFLM` and setting `parallelize=False`?