zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the MathQA evaluation
Root cause: Unimplemented Feature Gap

> We introduce a large-scale dataset of math word problems and an interpretable neural math problem solver that learns to map problems to operation programs. Due to annotation challenges, current datasets in this domain have been either relatively small in scale or did not offer precise operational annotations over diverse problem types. We introduce a new representation language to model precise operation programs corresponding to each math problem that aim to improve both the performance and the interpretability of the learned models. Using this representation language, our new dataset, MathQA, significantly enhances the AQuA dataset with fully-specified operational programs. We additionally introduce a neural sequence-to-program model enhanced with automatic problem categorization. Our experiments show improvements over competitive baselines in our MathQA as well as the AQuA dataset. The results are still significantly lower than human performance indicating that the dataset poses new challenges for future research. Our dataset is available at: https://math-qa.github.io/math-QA/

https://arxiv.org/abs/1905.13319

It seems like MathQA reuses aquarat data:

> Our dataset is gathered by using a new representation language to annotate over the AQuA-RAT dataset. AQuA-RAT has provided the questions, options, rationale, and the correct options. https://math-qa.github.io/math-QA/

They only modify the options in MathQA & if we don't use them as mentioned in #84, then there's no point in adding MathQA, is there?
Repository: EleutherAI/lm-evaluation-harness
Issue: `evaluator.py` should only depend on the contract of LM, actually depends on other model attributes
Root cause: Interface Contract Mismatch

Based on the code comments, I'd expect to be able to pass any valid instance of `LM` as `model` in evaluator.simple_evaluate. In reality the function depends on attributes not provided by the LM interface.

Specifically, I don't think it should rely on `model.model.config._name_or_path` here: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L165

Maybe better would be a hasattr check and default to something like `type(lm).__name__` ?
Repository: EleutherAI/lm-evaluation-harness
Issue: Pile dataset not found
Root cause: External Dependency Breakage

Hi,

I'm trying to run the `pile` group through `lm_eval.simple_evaluate`, but I am getting the following error.

```python
Traceback (most recent call last):
  File "/scratch/renatolg/tokens/harness.py", line 72, in <module>
    res = lm_eval.simple_evaluate(**eval_args, bootstrap_iters=0)
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/evaluator.py", line 122, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 255, in get_task_dict
    task_obj = get_task_dict(task_name)
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 275, in get_task_dict
    task_name: get_task(task_name=task_element, config=config),
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/tasks/__init__.py", line 217, in get_task
    return TASK_REGISTRY[task_name](config=config)
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/api/task.py", line 622, in __init__
    self.download(self.config.dataset_kwargs)
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/api/task.py", line 717, in download
    self.dataset = datasets.load_dataset(
  File "/home/renatolg/.local/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/renatolg/.local/lib/python3.10/site-packages/datasets/load.py", line 1852, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/renatolg/.local/lib/python3.10/site-packages/datasets/builder.py", line 373, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/renatolg/.local/lib/python3.10/site-packages/datasets/builder.py", line 539, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'pile_arxiv' not found. Available: ['all', 'enron_emails', 'europarl', 'free_law', 'hacker_news', 'nih_exporter', 'pubmed', 'pubmed_central', 'ubuntu_irc', 'uspto', 'github']
```

I thought `pile` (and its subsets, like `pile_arxiv`) was included in `lm-evaluation-harness`?

Just to clarify, I did successfully initialize tasks with `lm_eval.tasks.initialize_tasks()`.

Thanks

Hi! 

to run the Pile tasks, you'll need to use the fix described in https://github.com/EleutherAI/lm-evaluation-harness/issues/731 and have access to the Pile locally, since it is no longer downloadable via the Eye.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the Childrens Book Test evaluation
Root cause: Unimplemented Feature Gap

> We introduce a new test of how well language models capture meaning in children's books. Unlike standard language modelling benchmarks, it distinguishes the task of predicting syntactic function words from that of predicting lower-frequency words, which carry greater semantic content. We compare a range of state-of-the-art models, each with a different way of encoding what has been previously read. We show that models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words, although this advantage is not observed for syntactic function words. Interestingly, we find that the amount of text encoded in a single memory representation is highly influential to the performance: there is a sweet-spot, not too big and not too small, between single words and full sentences that allows the most meaningful information in a text to be effectively retained and recalled. Further, the attention over such window-based memories can be trained effectively through self-supervision. We then assess the generality of this principle by applying it to the CNN QA benchmark, which involves identifying named entities in paraphrased summaries of news articles, and achieve state-of-the-art performance.

this should be two seperate evals: CBT-CN and CBT-NE (from gpt2 paper)

https://arxiv.org/abs/1511.02301
Repository: EleutherAI/lm-evaluation-harness
Issue: Undefined symbol: _ZN3c104cuda9SetDeviceEi while using vLLM
Root cause: External Dependency Breakage

Hi, thanks for maintaining this wonderful project!
I am trying to load models with vLLM and I have successfully installed required dependencies by 
```bash
pip install -e ".[vllm]"
```
However, when I tried to launch lm-eval with vLLM, the following undefined symbol error occurred:
```
2024-01-24:04:13:44,829 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2024-01-24:04:13:44,829 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-01-24:04:13:44,829 INFO     [utils.py:160] NumExpr defaulting to 8 threads.
2024-01-24:04:13:45,610 INFO     [config.py:58] PyTorch version 2.0.1 available.
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/lmeval/bin/lm-eval", line 5, in <module>
    from lm_eval.__main__ import cli_evaluate
  File "/home/user/lm-evaluation-harness/lm_eval/__init__.py", line 1, in <module>
    from .evaluator import evaluate, simple_evaluate
  File "/home/user/lm-evaluation-harness/lm_eval/evaluator.py", line 11, in <module>
    import lm_eval.models
  File "/home/user/lm-evaluation-harness/lm_eval/models/__init__.py", line 7, in <module>
    from . import vllm_causallms
  File "/home/user/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 22, in <module>
    from vllm import LLM, SamplingParams
  File "/home/user/anaconda3/envs/lmeval/lib/python3.9/site-packages/vllm/__init__.py", line 3, in <module>
    from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
  File "/home/user/anaconda3/envs/lmeval/lib/python3.9/site-packages/vllm/engine/arg_utils.py", line 6, in <module>
    from vllm.config import (CacheConfig, ModelConfig, ParallelConfig,
  File "/home/user/anaconda3/envs/lmeval/lib/python3.9/site-packages/vllm/config.py", line 9, in <module>
    from vllm.utils import get_cpu_memory, is_hip
  File "/home/user/anaconda3/envs/lmeval/lib/python3.9/site-packages/vllm/utils.py", line 8, in <module>
    from vllm._C import cuda_utils
ImportError: /home/user/anaconda3/envs/lmeval/lib/python3.9/site-packages/vllm/_C.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi
```

I am using `python=3.9.18, torch=2.0.1, cuda=12.1, lm-eval=0.4.0, vllm=0.2.5`. Any idea on this? Thanks in advance!


Hey! [Looks](https://github.com/vllm-project/vllm/issues/1807#issuecomment-1829712585) like it requires torch >= 2.1.

Solved by reinstalling my packages, and now it works well. Thank you @baberabb !

@sdc17 What packages did you use?
Repository: EleutherAI/lm-evaluation-harness
Issue: Contributing AWS Inferentia Code
Root cause: Unimplemented Feature Gap

We at Gradient are using primarily the AWS inf2 instances. Our code is based of the `big-refactor` (v0.4.0) branch and adds a new model. 

Tested them for `generate_until`, `loglikelihood` requests. There is currently no support for some generation_kwargs such as beam search afaik.

So far, we have a implementation based on the `pip install transformers-neuronx` package in a separate model class `hf-neuronx`. What do you recommend to contribute? Is this contribution welcome?

# Motivation
Especially for the `generate_until` inference, inf2 setup runs at a decent speed + you can easily load a 70B model and bigger at batch_size=4 in a single inf2.x48large node with no pain points + there are usually high quotas available (over H100/A100 GPUs), including AWS spot instances.

# Results:
```bash
# Running on single A100
100%|██████████| 1319/1319 [1:18:42<00:00,  3.58s/it]
(demo-eval, pid=70834) hf (pretrained=mistralai/Mistral-7B-v0.1,dtype=float16), limit: None, num_fewshot: None, batch_size: 1
(demo-eval, pid=70834) |  Tasks   |Version|  Filter  |  Metric   |Value |   |Stderr|
(demo-eval, pid=70834) |----------|-------|----------|-----------|-----:|---|-----:|
(demo-eval, pid=70834) |gsm8k_yaml|Yaml   |get-answer|exact_match|0.3859|±  |0.0134|
```

```bash
# Running on AWS Inferentia inf2.x48large
1319it [41:10,  1.87s/it]█████████████████████████████████████████████████████████████████████████████| 1319/1319 [41:10<00:00,  1.00it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [41:10<00:00,  1.87s/it]
hf-neuronx (pretrained=mistralai/Mistral-7B-v0.1,dtype=float16,tp_degree=24), limit: None, num_fewshot: None, batch_size: 1
|  Tasks   |Version|  Filter  |  Metric   |Value |   |Stderr|
|----------|-------|----------|-----------|-----:|---|-----:|
|gsm8k_yaml|Yaml   |get-answer|exact_match|0.3836|±  |0.0134|
```

Hi! Thank you for your interest in the framework!

Yes, I think we'd be happy to support this model type. If you all could also contribute tests for this model type along with it to minimize the extra maintenance / overhead that supporting this model would require then that would be ideal.

I see `transformers-neuronx` looks to also support non-Inferentia devices--that's great!

> There is currently no support for some generation_kwargs such as beam search afaik.

With respect to this, the current must-have kwargs are to handle `do_sample: true/false`, `temperature`, and `max_gen_toks`--for others, if the library does not support them, then it's fine if it raises an error and does not fail silently.

Awesome, I'll aim to add a integration beginning next week or over the weekend. Likley using the `optimum[neuronx]` library in the end. If the library does now allow cpu inference, I'll mock the required requests.
Repository: EleutherAI/lm-evaluation-harness
Issue: Where is advanced_task_guide.md
Root cause: Documentation Deficiency

In [`docs/new_task_guide.md`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md) there are two references to `docs/advanced_task_guide.md`, but I can't find such file.  

Hi! This file has been renamed to task_guide.md. I've updated the docs to point to this instead in #1348 .
Repository: EleutherAI/lm-evaluation-harness
Issue: Question aboud evaluating MMLU
Root cause: Documentation Deficiency

Hi, thanks for the great work!  
I want to evaluate the MMLU benchmark. Because I could not access the huggingface hub, I download the 'hails/mmlu_no_train' on huggingface. But when I run this command 'lm_eval --model hf --model_args pretrained=/root/paddlejob/workspace/env_run/huitingfeng/models/llama-2-7b-chat-hf --tasks mmlu --device cuda:4 --batch_size 8', there has some tracebacks:
```
ValueError: BuilderConfig 'logical_fallacies' not found. Available: ['default']
```
My datasets version is 2.16.1. When I downgrade the datasets version to 2.15.0, there is another traceback:
```
column names don't match
```
I want to know how could I organize the downloaded MMLU benchmark and its corresponding yaml file.

Hi!

could you share more how you're downloading the dataset manually beforehand?

We describe a few ways to load from a local dataset here: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#using-local-datasets which may be helpful. For instance, the latter example should work with `dataset.save_to_disk()` 




Hi, please let us know if you're continuing to experience issues not solved by following the instructions in the linked docs page.

i have the same problems. I use the "dataset.save_to_disk()" to save gsm8k dataset into "llm/dataset/gsm8k". however when i set gsm8k.yaml as 
"
task: try_gsm8k
dataset_path: /mnt/nfs/vault/jiangp/llm/dataset/gsm8k  
dataset_name: main
"
or 
"
task: try_gsm8k
dataset_path: gsm8k
dataset_kwargs:
  data_dir: /mnt/nfs/vault/jiangp/llm/dataset/gsm8k/
dataset_name: main
"
it doesn't work neither, and show the same bug info "  File "/home/jiangp/.conda/envs/llm2/lib/python3.8/site-packages/datasets/builder.py", line 371, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/jiangp/.conda/envs/llm2/lib/python3.8/site-packages/datasets/builder.py", line 592, in _create_builder_config
    raise ValueError(
ValueError: BuilderConfig 'main' not found. Available: ['default']"

Want any help if possible

Hi @Jp-17 , could you open a new issue for this documenting what you're running into + steps to replicate? It sounds like the issue may be related to the use of `save_to_disk()` in datasets

thanks for your reply, i have just open a new issue, which contain more detailed info. https://github.com/EleutherAI/lm-evaluation-harness/issues/1829
Repository: EleutherAI/lm-evaluation-harness
Issue: Question - Is there a way to run evaluation on different benchmarks with generated model outputs ? 
Root cause: Documentation Deficiency

I am interested in using lm-eval with model outputs generated offline. If it's not already supported, is this something worth supporting ? 

Is there a specific reason or usecase you're targeting for this? 

I think exposing an extra utility / script in `scripts/` to (re-) score tasks using a dump of the config and samples, in file formats matching what we log in the library, might make sense.

The primary use case is to be able to run inference on other hardware accelerator ( for eg, AWS Inferentia  accelerator) machines instead of GPUs. 





Would it be possible to share some pointers to where in the repo I can find the file-formats & config dumps that is used to score samples ? I would like to go over it and understand the effort required. 

We don't have these documented in quite as much detail as we should but here is an example of some output dumps:

[mistral-gsm8k-hf.zip](https://github.com/EleutherAI/lm-evaluation-harness/files/14068498/mistral-gsm8k-hf.zip)

And this is an example of a script that performs a utility using the output dumps: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/scripts/zeno_visualize.py
Repository: EleutherAI/lm-evaluation-harness
Issue: Question: # of forward passes in multiple choice tasks
Root cause: Architectural Constraint

# Question
Looking at [task.py:972](https://github.com/EleutherAI/lm-evaluation-harness/blob/34cded30e6c0f13ad682f37f932e1573a36cc410/lm_eval/api/task.py#L972) it looks like an instance is created for each choice in multiple choice tasks and the `ctx` are identical for all choices. 
Later in [evaluate.py:318](https://github.com/EleutherAI/lm-evaluation-harness/blob/34cded30e6c0f13ad682f37f932e1573a36cc410/lm_eval/evaluator.py#L318) I see that the model is runs for each instance. 
Does this mean if we have 4 choices, the model is run 4 times on exactly the same context? If yes, what is the rationale?

Attempt at answering my own question: Is the reason behind the implementation because choices can be longer than one token (e.g. full sentences of multi-token words) ? 

That's correct, in the multi-token continuation case, to compute loglikelihood of a multi-token continuation we can't simply select the logit from the vocab, we need to feed in the (continuation+context) and get the logit at each continuation position, to compute the loglikelihood of continuation token N+1 conditional on the *context and first N continuation tokens*. 

It would be possible to reuse the KV cache across same-context requests, or detect when all choices are single-token, but it's not currently an optimization we're doing.
Repository: EleutherAI/lm-evaluation-harness
Issue: How is the summarized score of MMLU computed?
Root cause: Documentation Deficiency

I evaluated my model on MMLU with 5 shots using the command below. 

```
lm-eval 
    --model hf 
    --model_args pretrained=./my_model,trust_remote_code=True 
    --tasks mmlu
    --output_path result.json 
    --num_fewshot 5
    --batch_size 16
```

Here is the result.

```
|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.3161|±  |0.0562|
| - humanities     |N/A    |none  |     5|acc   |0.2931|±  |0.0444|
| - other          |N/A    |none  |     5|acc   |0.3502|±  |0.0669|
| - social_sciences|N/A    |none  |     5|acc   |0.3341|±  |0.0505|
| - stem           |N/A    |none  |     5|acc   |0.2994|±  |0.0563|
```

I'm confused about the acc score of `mmlu`.
I found that the score of `mmlu` is not the average of all MMLU subtasks. How is it computed? 
And why the `n-shot` of `mmlu` is 0?




Hi, 

MMLU average score is computed by averaging over MMLU *documents*, not by averaging over (differently-sized) subtasks! This matches the original MMLU codebase.

Regarding `n-shot` being `0` for the MMLU group, this is a printing bug, and we will fix it!
Repository: EleutherAI/lm-evaluation-harness
Issue: Suboptimal Performance on Generation Tasks
Root cause: Architectural Constraint

Hi @haileyschoelkopf 

I find one general problem for generation tasks when I am developing the [selfcheckgpt](https://github.com/EleutherAI/lm-evaluation-harness/pull/1080)  task.

The problem is that:
1. For each task, we can set one prompt format in task.yaml. But different models have their own system prompts or starting token. So one task can not be compared across multiple models.
2. For each task, we can set stopping tokens in the {"until": ["\n\n"}. But different models have their own ending tokens. So the model will generate something like this:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/43887288/c4de0db0-ae84-42ce-9d77-42832edc9751)

I think this is a general problem and more people who are using this framework should be aware of it when they are trying to compare different models in on generation task.

Thank you very much. 


We are currently working on supporting HF Chat Templating, see https://github.com/EleutherAI/lm-evaluation-harness/pull/1287

Thanks @erenup for the feedback! With respect to the latter point--we currently support the passing of generation hyperparameters via the  `--gen_kwargs` CLI flag. I can see about adding the ability to provide extra stop sequences via the CLI using this flag.

Thank you very much for the quick reply!

hi @haileyschoelkopf 

I think this could be a potential way to fix the [huggingface.py ](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/huggingface.py) model generation performance with apply_chat_template:
add `self.tokenizer.apply_chat_template(messages, tokenize=False)` to tok_batch_encode function in huggingface.py
<img width="1021" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/43887288/dca36676-e19b-4b3d-874f-fdd821899816">

How do you think about it?


Is this still an issue after #1873?
Repository: EleutherAI/lm-evaluation-harness
Issue: Display `n-shot` better for groups and for hardcoded-fewshot tasks
Root cause: Architectural Constraint

For tasks like `minerva_math` where we hardcode `n-shot` to be 0, and provide the model with a hardcoded few-shot prompt, fewshot is erroneously displayed as 0 . We should allow configs to contain 

```
metadata:
  num_fewshot: 4
```
to override what will print in `n-shot` column of our results tables.

it would also be nice to allow task groups to get a version field perhaps, though would need to decide how this should be inherited.

this was also mentioned in #1352 .
Repository: EleutherAI/lm-evaluation-harness
Issue: Question: system prompt in the few-shot setting
Root cause: Documentation Deficiency

# Question:
Hi, 
Is it possible to have a system prompt that only appears **once** at the begging of a few-shot sample?
So a sample would look something like:
`{system prompt}\n{fewshot example-1}\n{fewshot example-2}\n...{question}\nThe answer is:`

If that's not build-in, I would appreciate recommendations about what the best approach is to implement that. 

same question as you 


Hi! Chat template is still WIP in #1287 but if you just want a system prompt you could add a `description: "..."` field in the task yaml which will prepend the string to the prompt before all fewshot samples.

@baberabb is correct, `description` exposes this!

Note that there is no delimiter set to go between `description` and the first few-shot example, so prepend `\n\n` (same as `fewshot_delimiter` default) to the end of `description` if you desire this.
Repository: EleutherAI/lm-evaluation-harness
Issue: Hello, I would like to know if there is a method to use "generate_until" to evaluate on the ceval or cmmlu dataset. I'm using a chat model, which adds a prompt template to make it answer questions. However, the model's answer choices (like A, B, C, D) may not necessarily be the first generated token.
Root cause: Documentation Deficiency


Hi!

We don't currently have a generative counterpart for all loglikelihood / multiple choice tasks in a one-size-fits-all way.

However, for CEval or CMMLU, you could make a variant that is zero-shot CoT, following `_mmlu_flan_cot_zeroshot_template_yaml` in this PR: https://github.com/EleutherAI/lm-evaluation-harness/pull/1356/files#diff-77e4a56ffcde165e312239a2abcef4943035c61df2eb09249d8872ce91f29658