zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] output_path
Root cause: Validation Gap

Right now the output_path expects a string in a specific format of `dir_name/file_name`. Otherwise you get an error: `No such file or directory: ''`, if you provide a path such as `results` or `results.jsonl`, _after_ the evaluation has already been run. Maybe add an assertion beforehand or create the file/dir. Think it also copies over if the file exists so maybe add a warning?

I could fix this but not sure how you want it handled.

Could we adjust this so that it takes either `dir_name` or `dir_name/file_name`, and:

1. If it is in format `dir_name/file_name`, save to `dir_name/file_name` and save the samples jsonls from `--log_samples` to a file in `dir_name/` as is currently done?
2. If it is in format `dir_name/`, save to `dir_name/results.json` and save the samples jsonls to `dir_name`

And then it would be great to have a warning that's raised before running `simple_evaluate()` that prints the paths the results.jsonl and maybe sample jsonls will be saved into! I think we might want this to be a full error so that people don't copy over top accidentally? 

Thanks so much!
Repository: EleutherAI/lm-evaluation-harness
Issue: ['hellaswag' has no attribute 'Validation']
Root cause: Environment Incompatibility

Hi, thank you for this awesome repository.

I am facing one issue regarding evaluating hellaswag. When I am running the integrity check, it says `ValueError: Not all tests for the specified tasks (['hellaswag']) ran successfully!`

The reason is no keyword called` 'Validation' `into the dataset.

```
tests/test_version_stable.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
lm_eval/utils.py:242: in _wrapper
    return fn(*args, **kwargs)
lm_eval/evaluator.py:218: in evaluate
    task_docs = list(task_doc_func())
lm_eval/tasks/hellaswag.py:51: in validation_docs
    return map(self._process_doc, self.dataset["validation"])
self = DatasetDict({
    train: Dataset({
        features: ['prompt_3', 'prompt_0', 'logit_1', 'acc_norm', 'prompt_2', 'logit_2', 'acc', 'logit_3', 'doc_id', 'truth', 'prompt_1', 'logit_0'],
        num_rows: 10042
    })
})
k = 'validation'

    def __getitem__(self, k) -> Dataset:
        if isinstance(k, (str, NamedSplit)) or len(self) == 0:
>           return super().__getitem__(k)
E           KeyError: 'validation'

```


Your response is highly appreciated.

Best regards,
Abdelrahman.

Hi! What command are you running when you observe this error, so that I can reproduce it?

On my end, I see the tests for Hellaswag pass, and the HF dataset does have a `"validation"` split: https://huggingface.co/datasets/hellaswag/viewer/default/validation . 

The columns printed in your dataset seem to differ from the Hellaswag dataset. did you override the logic here locally?

If there’s a local file with the correct name, I believe that the HF loader prioritizes that over the copy on the hub. That could explain what’s been seen.

That would make sense! Closing as I'm unable to reproduce. Please reopen if this issue persists on a new machine/install, @Amshaker !
Repository: EleutherAI/lm-evaluation-harness
Issue: Unable to evaluate falcon model with the big-refactor branch
Root cause: Interface Contract Mismatch

I used the big-refactor branch to evaluate the falcon model as such

python main.py --model hf-auto \
       --model_args pretrained=tiiuae/falcon-7b,trust_remote_code=True \
       --tasks hellaswag --device cuda:0

I am encountering the following traceback:

Traceback (most recent call last):
  File "lm-evaluation-harness/main.py", line 182, in <module>
    main()
  File "lm-evaluation-harness/main.py", line 137, in main
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lm-evaluation-harness/lm_eval/utils.py", line 318, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/lm-evaluation-harness/lm_eval/evaluator.py", line 93, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lm-evaluation-harness/lm_eval/api/model.py", line 119, in create_from_arg_string
    return cls(**args, **args2)
           ^^^^^^^^^^^^^^^^^^^^
  File "/lm-evaluation-harness/lm_eval/models/huggingface.py", line 172, in __init__
    self._model = self.AUTO_MODEL_CLASS.from_pretrained(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda/envs/<>/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 487, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.tiiuae.falcon-7b.378337427557d1df3e742264a2901a49f25d4eb1.configuration_RW.RWConfig'> for this kind of AutoModel: AutoModelForSeq2SeqLM.
Model type should be one of BartConfig, BigBirdPegasusConfig, BlenderbotConfig, BlenderbotSmallConfig, EncoderDecoderConfig, FSMTConfig, GPTSanJapaneseConfig, LEDConfig, LongT5Config, M2M100Config, MarianConfig, MBartConfig, MT5Config, MvpConfig, NllbMoeConfig, PegasusConfig, PegasusXConfig, PLBartConfig, ProphetNetC
onfig, SwitchTransformersConfig, T5Config, XLMProphetNetConfig.


When switching back to the master branch, we do not see this error. 

Hi, thanks for reporting this issue!

I've made a patch that fixes this error in PR #686 , hope this helps!
Repository: EleutherAI/lm-evaluation-harness
Issue: Upgrade `sacrebleu` dependency to V2.0.0+
Root cause: External Dependency Breakage

Hi team,  it seems like we fix the `sacrebleu` to `1.5.0`.
However, this is a quite old version which make it impossible to install `lm_eval` with latest V2 `sacrebleu` in other projects.
V2 `sacrebleu` supports many more features in the below, including `ko` tokenizer: https://github.com/mjpost/sacrebleu/blob/master/CHANGELOG.md#release-notes.
Not sure if we can plan to upgrade `sacrebleu` in the dependency list?

Hi, thanks for raising this! It should be no problem, though I'd like to confirm there aren't any breaking changes before preventing the version from being fixed.

I will do such testing next week when I'm back from ICML next week, or if you're willing to test the tasks TruthfulQA and WMT which I believe are the ones that rely on sacrebleu to confirm that scores at 1.5.0 vs. recent releases of `sacrebleu` then happy to make the change earlier than that.

Thanks @haileyschoelkopf! 

Allowing anything higher than 1.5.0 in `big-refactor` (the branch that will be our next version release)! See #775 . 

However, having issues downloading the datasets we previously used for translation tasks, so no longer relying on `sacrebleu` as much as before.

@haileyschoelkopf is "big-refactor" merged as I understand it when seeing #775 ? because it still fix sacrebleu to 1.5.0 in requirements.
EDIT: obviously it's not merged.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] `num_fewshot` in yaml is unused during eval.
Root cause: Configuration Error

1. When we specify e.g. `num_fewshot: 5` in a YAML, but don't run with `--num_fewshot`, this ends up being set to 0
2. Have a way for when users specify `num_fewshot` in their yaml, raise a warning or error when someone tries to run that task with a different `num_fewshot` value passed via command line
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the HellaSwag evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> The HellaSwag dataset [ZHB+19] involves picking the best ending to a story or set of instructions. The examples were adversarially mined to be difficult for language models while remaining easy for humans (who achieve 95.6% accuracy). GPT-3 achieves 78.1% accuracy in the one-shot setting and 79.3% accuracy in the few-shot setting, outperforming the 75.4% accuracy of a fine-tuned 1.5B parameter language model [ZHR+19] but still a fair amount lower than the overall SOTA of 85.6% achieved by the fine-tuned multi-task model ALUM.

- [X] Data processing code implemented
- [x] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

_Note_: HuggingFace includes this in its datasets package. 

https://huggingface.co/datasets/hellaswag

Working on adding this for dedupe.
Repository: EleutherAI/lm-evaluation-harness
Issue: Runtime Error Running Benchmark for `TheBloke/falcon-40b-instruct-GPTQ`
Root cause: Interface Contract Mismatch

Good day!

Been trying to do a test benchmark for `TheBloke/falcon-40b-instruct-GPTQ`, but I'm running into a `RuntimeError` while doing so. Below are some details to replicate the issue:

<hr />

Branch: `master`

Command:
```
python main.py --model hf-causal-experimental --model_args pretrained=TheBloke/falcon-40b-instruct-GPTQ,trust_remote_code=True,dtype=float32,load_in_4bit=True,quantized=True,gptq_use_triton=True --check_integrity --tasks hellaswag --batch_size 8 --limit 50 --write_out --device cuda:0
```

Response:
```
RuntimeError: Expected query, key, and value to have the same dtype, but got query.dtype: float key.dtype: float and value.dtype: c10::Half instead.
```

Any idea what's causing the issue? Thanks!

From @haileyschoelkopf :

```
this issue can be solved by dropping the dtype=float32 from model_args
```

Thanks for the suggestion! Been experimenting with different values for `model_args` and one of them is trying to drop the `dtype` argument and I'm still getting the same `RuntimeError`. 

Closing as we've revamped this part of the codebase making this issue irrelevant.
Repository: EleutherAI/lm-evaluation-harness
Issue: GPTQ LLaMA v2 getting mat multiplication error
Root cause: Interface Contract Mismatch

Hi there!

Attempting to run lm eval harness on a GPTQ LLaMA v2 model with `gptq-4bit-128g-actorder_False` (https://huggingface.co/TheBloke/StableBeluga2-GPTQ) gets the following error:

```
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1480x8192 and 524288x10240)
```

Any help with be greatly appreciated!!

Closing as we've revamped this part of the codebase making this issue irrelevant.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Feature] Modular quantization using AutoGPTQ
Root cause: Unimplemented Feature Gap

Hi @StellaAthena, @haileyschoelkopf 

I was wondering if your repo supports modular quantization. For example, the Llama architecture looks like the following:

```
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32000, 4096, padding_idx=0)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (v_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (down_proj): Linear(in_features=11008, out_features=4096, bias=False)
          (up_proj): Linear(in_features=4096, out_features=11008, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm()
        (post_attention_layernorm): LlamaRMSNorm()
      )
    )
    (norm): LlamaRMSNorm()
  )
  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)
)
```
Now, what if I want to quantize only the `mlp` module or `self_attn` module?

**Usecase**: I would like to run AutoGPTQ on few datasets in [lm-evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness) dataset and understand the effects of quantizing modules for a study.

**P.S:** I am not sure if this feature is useful and if so, can be requested in this repo? Do let me know your thoughts and I am happy to close the issue if this doesn't work! 

Hi, this sounds like a feature that (if not already supported) should be implemented in autoGPTQ.

I'd recommend either making a `gptq_kwargs` argument to the `HFLM` init that can take in a set of args which are themselves parsed and passed into AutoGPTQForCausalLM, or just hacking the initialization of the `AutoGPTQForCausalLM` yourself if this is something more custom than that library supports.
Repository: EleutherAI/lm-evaluation-harness
Issue: Problem in reproducing accuracy of llama-7B on GLUE dataset from results folder
Root cause: Algorithmic Error

Hi, 
   Thank you for this excellent codebase! I tried to reproduce the results of `llama-7B` on GLUE dataset (https://github.com/EleutherAI/lm-evaluation-harness/blob/master/results/llama/llama-7B/llama-7B_glue_0-shot.json).

 I ran a few GLUE tasks:

```
python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=huggyllama/llama-7b \
    --tasks mrpc,sst,wnli,mnli_mismatched,qnli,rte \
    --device cuda:0 \
    --batch_size auto
```

The result below shows accuracy higher than the one in json file.

```
{
  "results": {
    "mnli_mismatched": {
      "acc": 0.4387713588283157,
      "acc_stderr": 0.005004840028486087
    },
    "mrpc": {
      "acc": 0.6862745098039216,
      "acc_stderr": 0.022999936277943434,
      "f1": 0.8134110787172011,
      "f1_stderr": 0.016204705075759232
    },
    "qnli": {
      "acc": 0.5116236500091524,
      "acc_stderr": 0.006763582165762025
    },
    "rte": {
      "acc": 0.6642599277978339,
      "acc_stderr": 0.028426021205818792
    },
    "sst": {
      "acc": 0.8314220183486238,
      "acc_stderr": 0.012685339891674497
    },
    "wnli": {
      "acc": 0.5211267605633803,
      "acc_stderr": 0.05970805879899504
    }
  },
  "versions": {
    "mnli_mismatched": 0,
    "mrpc": 0,
    "qnli": 0,
    "rte": 0,
    "sst": 0,
    "wnli": 1
  },
  "config": {
    "model": "hf-causal-experimental",
    "model_args": "pretrained=huggyllama/llama-7b",
    "num_fewshot": 0,
    "batch_size": "auto",
    "batch_sizes": [],
    "device": "cuda:0",
    "no_cache": false,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal-experimental (pretrained=huggyllama/llama-7b), limit: None, provide_description: False, num_fewshot: 0, batch_size: auto
|     Task      |Version|Metric|Value |   |Stderr|
|---------------|------:|------|-----:|---|-----:|
|mnli_mismatched|      0|acc   |0.4388|±  |0.0050|
|mrpc           |      0|acc   |0.6863|±  |0.0230|
|               |       |f1    |0.8134|±  |0.0162|
|qnli           |      0|acc   |0.5116|±  |0.0068|
|rte            |      0|acc   |0.6643|±  |0.0284|
|sst            |      0|acc   |0.8314|±  |0.0127|
|wnli           |      1|acc   |0.5211|±  |0.0597|
```

I would appreciate it if you have any idea about this. 

Thank you for your time!


Hi! I'm sorry for the confusion this may have caused-- `results/` contains results run on an earlier commit of the codebase, in which our Llama tokenization suffered from erroneous extra whitespace tokens that decreased scores. 

We'd like to maintain results on common tasks + models in the future, but however removing the pre-computed now-incorrect scores is preferable for the time being. #686 and #718 will fix this in both active branches. 

Thanks for using the codebase and sorry for the confusion!
Repository: EleutherAI/lm-evaluation-harness
Issue: Update Anthropic Client
Root cause: External Dependency Breakage

Anthropic changed their python sdk - making this code line outdated.

https://github.com/EleutherAI/lm-evaluation-harness/blob/4fbbd60fa3573dcbf61eb79492f772adeb969157/lm_eval/models/anthropic_llms.py#L50

---

Would love to know if this might help - https://github.com/BerriAI/litellm

~Simple I/O library, that standardizes all the llm api calls to the OpenAI call

```
from litellm import completion

## set ENV variables
# ENV variables can be set in .env file, too. Example in .env.example
os.environ["OPENAI_API_KEY"] = "openai key"
os.environ["ANTHROPIC_API_KEY"] = "anthropic key"

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# anthropic call
response = completion("claude-v-2", messages)

```

cc: @lintangsutawika @guijinSON
Repository: EleutherAI/lm-evaluation-harness
Issue: Pile tasks on big-refactor use dataset_names from old dataset loader that don't exist on HF
Root cause: Configuration Error

Task example: https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/lm_eval/tasks/pile/pile_arxiv.yaml#L7

HF dataset: https://huggingface.co/datasets/EleutherAI/pile

Original dataset loader prior to `big-refactor`: https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/datasets/pile/pile.py

@haileyschoelkopf mentioned that using this loading script should work if we upload it to HF and point the Pile tasks to that new dataset.


Adding the file "pile.py" at "lm-evaluation-harness/EleutherAI/the_pile/the_pile.py" does indeed fix the issue. Additionally changing the test split to "test" in pile_arxiv.yaml (line 9)

This recipe works pretty fast, but I observe this strange trend where the first few samples are processed slow (which is understandable), the middle samples are processed at an extremely fast speed, and then in the end the last few samples again take a lot of time. When using "accelerate launch" this almost halts forever (I eventually killed the process after waiting for a few minutes), whereas using a single GPU does allow me to get final output.

Just an update to the above. Since PILE is no longer public now, you may want to modify the _URLS to your local path to pile. This is line 44 of the current file pile.py

`_URLS = {
    "validation": "/data/the_pile/val.jsonl.zst",
    "test": "/data/the_pile/test.jsonl.zst",
}`

Also, there have been some changes to the repo since the last comment. The file should be placed at 
`"lm-evaluation-harness/EleutherAI/pile/pile.py"`
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] Add OpenAI Chat Completion API 
Root cause: Unimplemented Feature Gap

Right now the library only supports the completion endpoint and all the newer models use the chat completion API. This doesn't provide the log probabilities so cannot use it for that but would be great if we added support for it for `greedy_until`.

Also would be great if we add async support.

Merged in #839 , #1008 !
Repository: EleutherAI/lm-evaluation-harness
Issue: New Evaluation: Legal
Root cause: Unimplemented Feature Gap

We would like to find or create an eval dataset that tests legal knowledge. Note that the LSAT exams do not actually test any legal knowledge.

- [ ] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

[This dataset](https://arxiv.org/abs/1911.12011) is a good place to start for this issue.

https://arxiv.org/pdf/2009.03300.pdf has ~2000 US multistate bar exam questions (a bar exam has <200 graded questions).
The most updated form of the paper is here: https://openreview.net/pdf?id=d7KBjmI3GmQ
In around a month I should have a legal contract review dataset ready. Covering case law and contract review should cover the majority of legal NLP.

Excellent! I look forward to it.
Repository: EleutherAI/lm-evaluation-harness
Issue: Question in CrowS-Pairs accuracy metric
Root cause: Algorithmic Error

If a stereotyped sentence is predicted, then `pct_stereotype` should be lower, right? In line 135, it suggests that higher is better.

https://github.com/EleutherAI/lm-evaluation-harness/blob/f9eca2c8160be8c20ecc956b7ff545f880160d0e/lm_eval/tasks/crowspairs.py#L135

Hi, thanks for raising this! 

You're correct, although technically it's "50% is better" for this benchmark, which I'm uncertain how we want to handle atm.

Patched this in #765 .
Repository: EleutherAI/lm-evaluation-harness
Issue: New Evaluation: Math
Root cause: Unimplemented Feature Gap

We would like to find or create an eval dataset that tests mathematical knowledge. The GRE exams may be a good source of material. 

- [ ] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

MathQA: https://arxiv.org/pdf/1905.13319.pdf
<img width="560" alt="image" src="https://user-images.githubusercontent.com/18056781/103669513-70278f80-4f9e-11eb-8226-2e42a813ebed.png">



AquaRAT - https://github.com/deepmind/AQuA




This is probably of much interest to us: https://deepmind.com/research/open-source/mathematics
> A large-scale extendable dataset which generates question and answer pairs from a range of question types at roughly school-level difficulty. It is designed to test the mathematical learning and algebraic reasoning skills of learning models.
Mathematics Dataset - https://github.com/deepmind/mathematics_dataset

The DeepMind mathematics dataset is in the pile.

> The GRE exams may be a good source of material.

Some GRE exams (biology, chemistry, maths, physics, cs) are covered in https://arxiv.org/pdf/2009.03300.pdf

Here's a competition maths dataset: https://drive.google.com/file/d/1RafScUe8O6MxE4K3COnOT4CFR4zvQdXv/view?usp=sharing
I'll have an arXiv paper for it in a month.

Again, excellent work! Can I make you down as responsible for the processing code?

OK, though it might be an undergraduate collaborator of mine to add the code.

> OK, though it might be an undergraduate collaborator of mine to add the code.

Understood. Thanks again!
Repository: EleutherAI/lm-evaluation-harness
Issue: New Evaluation: Biology
Root cause: Unimplemented Feature Gap

We would like to find or create an eval dataset that tests biological and medical knowledge. [This dataset](https://bionlp.nlm.nih.gov/epic_qa/) for evaluating questions about COVID is a good place to start.

- [ ] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

PubMedQA -  https://arxiv.org/pdf/1909.06146.pdf 

<img width="571" alt="image" src="https://user-images.githubusercontent.com/18056781/103668017-a532e280-4f9c-11eb-80b5-11f469650bac.png">

<img width="1116" alt="image" src="https://user-images.githubusercontent.com/18056781/103668085-bda2fd00-4f9c-11eb-9de3-b7b6e862b096.png">




Electronic Medical Records QA - https://arxiv.org/abs/1809.00732
<img width="558" alt="image" src="https://user-images.githubusercontent.com/18056781/103668376-065ab600-4f9d-11eb-8dbe-57492ffcd213.png">


Further datasets to look at:
BioMRC: https://arxiv.org/pdf/2005.06376.pdf
HeadQA: https://arxiv.org/pdf/1906.04701.pdf
BioASQ: http://bioasq.org

I can do this

@StellaAthena Looking at epic-qa, their license/usage (under "Document Collection") is compatible with our stuff, right?

> @StellaAthena Looking at epic-qa, their license/usage (under "Document Collection") is compatible with our stuff, right?

Yeah, the usage restriction boils down to

> We therefore pledge to make our intellectual property available free of charge for use in ending the COVID-19 pandemic and minimizing the impact of the disease.

and we are making everything freely available for everyone.

Perfect, thanks!

We shouldn't be having these big omnibus issues; for each dataset, please make an issue and add it to the project board. I've made separate issues for the following:

Pubmed: #125 
emrQA: #115 
BioMRC: #126 
HeadQA: #127
BioASQ: #114