zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Error when limit is not specified (possibly issue with requirements?)
Root cause: Validation Gap

Hello guys! Thanks for the great work.

I was trying to run a fresh install of lm harness (I've used this library in the past before the newer updates). For testing purposes I'm using

```
accelerate launch  -m lm_eval --model hf --model_args pretrained=EleutherAI/pythia-160m --tasks arc_challenge --output_path ./results --batch_size 8
```

A simple evaluation on `pythia` 160M on `arc_challenge` running this generates an error 

```
  File "/home/username/lm-evaluation-harness/lm_eval/evaluator.py", line 598, in <dictcomp>
    "effective": min(limit, len(task_output.task.eval_docs)),
```

Looking into this it seems this stems from not passing limit (hence limit = None).

Is there something required to solve this ? Could this stem from the current requirements.txt file given that I did a fresh conda environment ?


Edit: 
A possible fix could be setting the default value to 1.0 maybe ?


This is from a recent change (#1766). #1785 will fix it.

Thanks (: !!
Repository: EleutherAI/lm-evaluation-harness
Issue: Support loading slices of a split from a dataset
Root cause: Unimplemented Feature Gap

### What
Hugging Face Datasets support [Slice Splits](https://huggingface.co/docs/datasets/v2.19.0/loading#slice-splits)

```py
datase_10pc = datasets.load_dataset("mydataset", split="test[:10%]")
```

Therefore I assumed that when creating a new task, I could express the dataset split as:

```yaml
task: mytask
dataset_path: user/mydataset
dataset_name: null
training_split: null
validation_split: null
test_split: 'test[:50%]'
doc_to_text: abc
doc_to_target: def
metric_list:
  - metric: ...
```

However, that fails in:
https://github.com/EleutherAI/lm-evaluation-harness/blob/30c060d208e2c445fdf02a8c4dd1fd047064cc3e/lm_eval/api/task.py#L901

with `KeyError: 'test[:50%]'`


Same question
Repository: EleutherAI/lm-evaluation-harness
Issue: Math or minerva_math not generating any samples via scripts.write_out
Root cause: Validation Gap

I installed `lm-evaluation-harness` from main as of May 6th 2024.

I try to run 

```bash
python -m scripts.write_out --output_base_path ~/workspace --tasks minerva_math
```

The generated files are all empty.

Hi!

This is because `minerva_math`  task has no validation split specified and write_out.py defaults to `--sets val` for printing only the val set. I've added an error message for this case in #1795 .
Repository: EleutherAI/lm-evaluation-harness
Issue: Add NPU support for huggingface.py
Root cause: Unimplemented Feature Gap

## Feature request
It would be nice if the lm_eval  could be used directly on the Ascend NPU without modifying the source code of `huggingface.py`.

## Motivation
In China, Ascend NPU is the second choice after Nvidia GPU and has been adpoted by many companies, such as Alibaba, ByteDance, Meituan, etc. Huawei officially released an adapter called [torch_npu](https://github.com/Ascend/pytorch/blob/master/README.en.md), to adapt PyTorch on Ascend NPU. torch_npu is user friendly to developers, so that we can still enjoy the same PyTorch experience that we accustomed to today.

And for transformers suites, currently `accelerate`, `transformers`  and other repos has supports NPU devices. Here are some links:

accelerate:  https://github.com/huggingface/accelerate/releases/tag/v0.26.0
transformers: 

## Contribution
I can assist in adding support if you want,, `accelerate ` has already natively support NPU device, so there are only few fixs on existing code.

I have already make a PR(https://github.com/EleutherAI/lm-evaluation-harness/pull/1787), it's better to show how to adapt a NPU to huggingface.py

@haileyschoelkopf Hi, would you mind having a check?
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the QuAC evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive, multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset.
> GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI+18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions. On DROP [DWD+19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL+19]. On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper. On RACE [LXL+17], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.

- [X] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Original training and validation sets are JSON files at the link below:

https://quac.ai/

Have been working on this.

Hi, has the evaluation code of QuAC been implemented yet? Thanks! @StellaAthena 

Thanks for flagging this! It looks like it hasn’t been, so I’ll re-open it.

If you’re interested in using this task, we would greatly appreciate having it implemented and a PR submitted :)

Hi, can i take the QuAC evaluation task, i see it has not been implemented yet?

Closing as duplicative of https://github.com/EleutherAI/lm-evaluation-harness/issues/827
Repository: EleutherAI/lm-evaluation-harness
Issue: Exclude all current tasks
Root cause: Unimplemented Feature Gap

Is it possible to exclude all current tasks in the `lm_eval/tasks` folder? Then I will include the tasks that I want via the `--include-path` flag. I want to do this mainly for speed reasons (not having the code process hundreds of tasks that I won't use anyway), but also for testing customization and modification of existing tasks. 

This is not currently supported but certainly should be. Will do this, may have it require a specific flag so as to not break backward-compatibility.

Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Hugging Face: Open LLM Leaderboard: how do I reproduce results for details_gpt2 repository
Root cause: Documentation Deficiency

I'm trying to reproduce the results for GPT2 on the Open LLM Leaderboard found here: https://huggingface.co/datasets/open-llm-leaderboard/details_gpt2/blob/main/results_2024-03-23T06-18-16.565546.json

The `config_general` json seems to list many other parameters that don't seem to be supported by the lm-evaluation-harness even though the Open LLM Leaderboard uses this harness to generate the results.

What command and commit should I use reproduce the results in the above link?

Solved. Had the wrong commit.

> 已解决。提交有误。

can you give me the correct commit number? thx!
Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate a large model like llama-65B?
Root cause: Documentation Deficiency

I tried use 
 lm_eval \
      --model hf \
      --model_args pretrained=$model_path, parallelize=True \
      --tasks ${tasks} \
      --output_path $savepath \
      --batch_size auto \

and 

lm_eval --model vllm \
    --model_args pretrained=$model_path,tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.8,data_parallel_size=4 \
    --tasks lambada_openai \
    --batch_size auto

However, it loads the llama 65B model directly into one GPU and return CUDA OOM. 

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB. GPU 0 has a total capacty of 79.35 GiB of which 23.19 MiB is free. Process 178649 has 79.32 GiB memory in use. Of the allocated memory 78.41 GiB is allocated by PyTorch, and 352.00 KiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I used A100 80G GPUs. The llama 30B model can be evaluated normally.

How can I use model parallelize to evaluate llama 65B?

@fayuge what was your solution?

> @fayuge what was your solution?

The following command works, I used the wrong script before:
lm_eval \
    --model hf \
    --model_args pretrained=$model_path,parallelize=True \
    --tasks ${tasks} \
    --output_path $model_path \
    --batch_size auto:4 \
Repository: EleutherAI/lm-evaluation-harness
Issue: Is caching large evaluation dataset like MMLU supported?
Root cause: Documentation Deficiency

I am trying to evaluate MMLU accuracy on about 100 HF models and I noticed the process of creating the MMLU evaluation dataset is pretty slow. Is there any way I can cache the dataset and hence speed up evaluation of multiple models?

I am currently using command like
lm_eval --model vllm \                                                                      
    --model_args pretrained=NousResearch/Meta-Llama-3-8B,tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.95,max_model_len=4096,enforce_eager=True \
    --tasks mmlu \
    --batch_size auto:10

Hi! `--cache_requests true` should allow you to save on a substantial amount of time here.
Repository: EleutherAI/lm-evaluation-harness
Issue: how to run all the bigbench tasks at once?
Root cause: Documentation Deficiency

python -m lm_eval --model hf --model_args pretrained=/hf_model_files/Llama-2
-7b-chat-hf/,parallelize=True   --tasks bigbench --batch_size 16 --output_path /out/Llama-2-7b-chat-hf/

i am getting below error:
2024-05-04 08:44:14 EDT llm_evals line: 74: [INFO] 2024-05-04:08:44:14,457 ERROR    [__main__.py:301] Tasks were not found: bigbench

Follow up: i am getting same error for ,mbpp,mgsm,arc tasks.. how should we run all the subtasks for these group tasks
Repository: EleutherAI/lm-evaluation-harness
Issue: Out-Of-Memory Error for same batch size but different dataset
Root cause: Resource Mishandling

Hi,

I was running evaluation on MMLU and GSM8K datasets using the following command:
lm_eval --model vllm \                                                                      
    --model_args pretrained=NousResearch/Meta-Llama-3-8B,tensor_parallel_size=4,dtype=auto,gpu_memory_utilization=0.95,max_model_len=4096,enforce_eager=True \
    --tasks mmlu \
    --batch_size auto:10 \
    --cache_requests true

However it seems that even with the same batch size (for example I tried batch_size 128) and fixed max_model_len, with MMLU I would get OOM error and with GSM8K everything runs fine. I was guessing that the code you used would transfer the whole dataset onto CUDA before running the evaluation? Is there a way to move each batch onto GPU to avoid this error?

I have the same problem, does anyone know how to solve it? 🌹

Batching is handled internally by vllm so it shouldn't matter much. Have you tried reducing the gpu_memory_utilization? If you aren't using data parallel, then you could also try using the latest release of vllm if still on 0.3.2.  

Could you explain how data_parallel works? I don't seem to see it in vLLM's documentation.

> Could you explain how data_parallel works? I don't seem to see it in vLLM's documentation.

We create multiple vllm models across different devices so each can process a subset of the data concurrently. But we need to update it for vllm > 0.3.2. Latest vllm versions should work without it though. 

Thanks for the clarification. And also why do we want to lower the gpu_memory_utilization? I thought if we make it close to 1 it won't reserve any memory which means it uses more memory to do the inference. Is it not true?

> Thanks for the clarification. And also why do we want to lower the gpu_memory_utilization? I thought if we make it close to 1 it won't reserve any memory which means it uses more memory to do the inference. Is it not true?

I think it's the [opposite](https://github.com/vllm-project/vllm/blob/be0c5180ac0832a0b285d0845d458798bb3f0f4f/vllm/entrypoints/llm.py#L58) 😅. Setting it lower helps with OOM most of the time.

My bad you are absolutely right 🧎‍♀️. Let me try these new settings and see if that helps. Thanks a lot!
Repository: EleutherAI/lm-evaluation-harness
Issue: Support Mamba based models for evaluation tasks
Root cause: Documentation Deficiency

Hi,

Great repo, just love it <3.

I would like to know if there's any plan to support Mamba based model to the current pipeline. Example usage would be - 

`lm_eval --model hf --model_args pretrained=state-spaces/mamba-130m --tasks wikitext --device cuda:0 --batch_size 16`

Thanks

This is already supported via `--model mamba_ssm` to use the state-spaces/mamba package with compatible models, and one can also use HF-compatible mamba checkpoints using `--model hf` with recent enough versions of transformers!
Repository: EleutherAI/lm-evaluation-harness
Issue: Multi-round evaluation for chat models
Root cause: Unimplemented Feature Gap

Is it possible to conduct multi-round evaluations for chat models? For example, I want to study how a chat model can take hints to solve math problems. Say, I have a multiple-choice math problem, with one correct choice, and one hint for each wrong choice. I first ask the question to the model, get a model answer (using generate_until and some parsing), in the following workflow: 

```
User: Solve the following math question, and output a single letter corresponding to your selection. [question]
Agent: B
```

If this is correct, I stop here. If not, I continue, where the agent left off: 

```
User: Solve the following math question, and output a single letter corresponding to your selection. [question]
Agent: C
User: This is not correct. Consider the hint and try again: [hint corresponding to choice C]
Agent: B
```

In the evaluation, I want to calculate the percentage of the first round correct answer, as well as the percentage of the second round correct answer when the first round answer is incorrect. 

Is it possible to do in the current framework? Or is the library not suitable for this kind of evaluation? 

right now I'm working on solution for your problem!
take a look at #1571 
here I suggest the way multi-step and multi-round tasks may be handled. The magic lies in `update_request` and `update_storage` funcs. You can customize them to cover your problem
it would be great, if you can take a look at this PR and leave your feedback, so that I can make it better and also draw @haileyschoelkopf @lintangsutawika attention to the desired feature
Repository: EleutherAI/lm-evaluation-harness
Issue: Task description newline characters removed by Jinja templating, affecting generated requests and performance
Root cause: Algorithmic Error

### What is the problem?
The code changes from PR #1553 improperly remove trailing newline character `\n` from task description when provided, resulting in unintended changes to the generated request and potential performance implications..

For example, the subtask `temporal sequence` from `bbh_cot_fewshot` includes a [task description](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/bbh/cot_fewshot/temporal_sequences.yaml#L2) with a trailing newline: 
`Task description: Answer questions about which times certain events could have occurred.\n\n`. 

Ideally, the generated request should preserve the description content as it is. However, after calling function [`apply_template`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L973) , the trailing newline characters are removed, altering the content to:
`Task description: Answer questions about which times certain events could have occurred.\n`. 

### Why?
This behavior is due to how Jinja2 handles whitespace control and newline characters in templates. By default, Jinja2 removes trailing newline characters from templates. This is done to allow better control over whitespace in the rendered output, especially when working with HTML templates where extra newlines can affect the rendering and formatting of the content. 

### How to fix?
We have two main options to address the issue:
1. Revert back to previous implementation
We could remove `apply_template` function to make the description filed unmodified. This is a low-risk solution as we simply revert back the original implementation. 
2. Modify the `apply_template` function to properly handle whitespaces and newlines
If we want to keep the templating feature for description, then we need to modify the function to handle whitespaces and newline characters carefully. One candidate solution be to use Jinja's `trim_blocks` and `lstrip_blocks` options to control whitespace trimming. However, the risk is unclear since this code change might affect the rendering behavior in other ways. 

```
env = Environment(loader=BaseLoader, 
                  undefined=StrictUndefined,
                  lstrip_blocks=True,
                  trim_blocks=True)
env.filters["regex_replace"] = regex_replace


def apply_template(template: str, doc: dict) -> str:
    rtemplate = env.from_string(template)
    return rtemplate.render(**doc)
```

Thank you for tracking this down!

I think the preferred fix would be: 

- revert back to previous implementation (description does not have apply_template applied)
- have tasks which required this simply generate a `description` column using `process_docs` and use that (along with whatever desired trailing whitespace) for custom per-document description fields.

Hi, looks like we are still using "utils.apply_template" to generate the description column. Is someone working on it?
Repository: EleutherAI/lm-evaluation-harness
Issue: when MMLU eval, num_few_shot=5, more GPU overhead
Root cause: Resource Mishandling

when MMLU eval, same single GPU, it success:
lm_eval --model hf \
	--model_args pretrained="Qwen/Qwen1.5-7B-Chat",dtype=bfloat16 \
        --tasks mmlu \
        --device cuda \
	--batch_size 32 \
        --trust_remote_code \
        --cache_requests true 

but when --num_fewshot 5  , batchcsize masu <-=4,  otherwise OOM:

 it seems more GPU overhead, any suggestions for reduce GPU mem and accelerate?

This is expected because fewshot examples will result in larger model inputs. I'd recommend checking out vllm for faster inference!
Repository: EleutherAI/lm-evaluation-harness
Issue: SyntaxError when import lm_eval
Root cause: External Dependency Breakage

Hello, I met the following error when tried to import lm_eval:
```
python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lm_eval
Downloading builder script: 3.69kB [00:00, 12.7MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/__init__.py", line 1, in <module>
    from .evaluator import evaluate, simple_evaluate
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 11, in <module>
    import lm_eval.api.metrics
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/metrics.py", line 168, in <module>
    exact_match = hf_evaluate.load("exact_match")
  File "/usr/local/lib/python3.10/dist-packages/evaluate/loading.py", line 751, in load
    evaluation_cls = import_main_class(evaluation_module.module_path)
  File "/usr/local/lib/python3.10/dist-packages/evaluate/loading.py", line 76, in import_main_class
    module = importlib.import_module(module_path)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/009c8b5313309ea5b135d526433d5ee76508ba1554cbe88310a30f85bb57ec88/exact_match.py", line 16
    }
    ^
SyntaxError: closing parenthesis '}' does not match opening parenthesis '(' on line 14
```
The version is 0.4.2. Both pip and installing from source all encountered the error.

You may use hf-mirror.com instead of huggingface.co as your HF_ENDPOINT.

Since hf-mirror.com could turn down when the load is high, you may temporally use the main huggingface.co site by `unset HF_ENDPOINT` 

替换exact_match.py文件即可解决，参考https://github.com/huggingface/evaluate/issues/590

Thanks all.
Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError: 'NoneType' object is not iterable when using cache and loglikelihood_rolling
Root cause: Validation Gap

Hi, I receive following exception when using caching and loglikelihood_rolling:

> 
> Traceback (most recent call last):
>   File "...micromamba_stuff/envs/harness/lib/python3.9/runpy.py", line 197, in _run_module_as_main
>     return _run_code(code, main_globals, None,
>   File "...micromamba_stuff/envs/harness/lib/python3.9/runpy.py", line 87, in _run_code
>     exec(code, run_globals)
>   File "...harness_vanilla/lm_eval/__main__.py", line 416, in <module>
>     cli_evaluate()
>   File "...harness_vanilla/lm_eval/__main__.py", line 347, in cli_evaluate
>     results = evaluator.simple_evaluate(
>   File "...harness_vanilla/lm_eval/utils.py", line 321, in _wrapper
>     return fn(*args, **kwargs)
>   File "...harness_vanilla/lm_eval/evaluator.py", line 256, in simple_evaluate
>     results = evaluate(
>   File "...harness_vanilla/lm_eval/utils.py", line 321, in _wrapper
>     return fn(*args, **kwargs)
>   File "...harness_vanilla/lm_eval/evaluator.py", line 406, in evaluate
>     resps = getattr(lm, reqtype)(cloned_reqs)
>   File "...harness_vanilla/lm_eval/api/model.py", line 253, in fn
>     rem_res = getattr(self.lm, attr)(remaining_reqs)
>   File "...harness_vanilla/lm_eval/models/huggingface.py", line 893, in loglikelihood_rolling
>     string_nll = self._loglikelihood_tokens(
>   File "...harness_vanilla/lm_eval/models/huggingface.py", line 1136, in _loglikelihood_tokens
>     self.cache_hook.add_partial("loglikelihood", request_str, answer)
>   File "...harness_vanilla/lm_eval/api/model.py", line 193, in add_partial
>     hsh = hash_args(attr, req)
>   File "...harness_vanilla/lm_eval/api/model.py", line 178, in hash_args
>     dat = json.dumps([attr] + list(args))
> TypeError: 'NoneType' object is not iterable
> 

Arguments:

> '--model', 'hf', '--model_args', 'pretrained=BUT-FIT/CSTinyLlama-1.2B,dtype=bfloat16,max_length=2048,truncation=True,trust_remote_code=True', '--tasks', 'benczechmark_histcorpus', '--batch_size', '8', '--use_cache', 'tcache', '--output_path', 'results/eval_cstllama_historical', '--log_samples', '--verbosity', 'DEBUG'


Task config:

> 
> task: benczechmark_histcorpus
> task_alias: Historical Corpus
> dataset_path: CZLC/benczechmark_histcorpus
> output_type: loglikelihood_rolling
> test_split: test
> doc_to_text: "Historický text: "
> doc_to_target: "{{text}}"
> metric_list:
>   - metric: word_perplexity
>     aggregation: weighted_perplexity
>     higher_is_better: false
>   - metric: acc
>     aggregation: mean
>     higher_is_better: true
> metadata:
>   version: 1.0

It seems to be related to the  [line](https://github.com/EleutherAI/lm-evaluation-harness/blob/1980a13c9d7bcdc6e2a19228c203f9f7834ac9b8/lm_eval/models/huggingface.py#L879) as there is the None instead of Tuple[str, str].

I have also encountered this problem, how did you solve it?