zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Unable to Reproduce HellaSwag ZeroShot Performance Reported In GPT-NeoX Paper For GPT-J-6B
Root cause: Documentation Deficiency

In the GPT-NeoX paper (https://arxiv.org/pdf/2204.06745.pdf), in Table 2, I noticed that the HellaSwag zeroshot accuracy is reported as 51.8% percent. When evaluating the same model using the lm-eval-harness, neither the length normalized accuracy nor the raw accuracy matches.

```
gpt2 (pretrained=EleutherAI/gpt-j-6B), limit: None, provide_description: False, num_fewshot: 0, batch_size: 8
|  Task   |Version| Metric |Value |   |Stderr|
|---------|------:|--------|-----:|---|-----:|
|hellaswag|      0|acc     |0.4954|±  |0.0050|
|         |       |acc_norm|0.6625|±  |0.0047|
```

Does anyone know what could be causing this?

Thank you so much in advance!

I also could not repro the 0-shot evals running from the main branch:

<img width="640" alt="Screen Shot 2022-12-29 at 22 32 38" src="https://user-images.githubusercontent.com/41410219/210031726-d0da743a-f828-4f4c-8f8e-25b0931858a1.png">

I was responsible for evaluating the 5-shot results reported in the paper and those are reproducible:

<img width="640" alt="Screen Shot 2022-12-29 at 22 33 24" src="https://user-images.githubusercontent.com/41410219/210031758-2f098ce0-6f0a-4523-862b-b861776c8180.png">

I'm not so sure what the process was for the 0-shot setting. Maybe @StellaAthena could shed some light on this or point to someone who may have more info.

Thanks for pointing it out 🙏 


The numbers reported in the paper are all unnormalized.

I have no idea where that number came from, but it looks like it must be a transcription error. In the blog post announcing the model the number is reported as 36.53%, which I also have no explanation for. The files that had the results should be on my other computer, and I can inspect them directly on Monday. Right now my inclination is that we somehow messed the numbers up and should correct the paper. For completeness, it's worth mentioning that the version number for the task hasn't change since we wrote the paper.

The GPT-J repository reports 69.7%, though I happen to know that's normalized. That doesn't exactly agree with the number the eval harness is reporting, but I happen to know this was run in Mesh Tensorflow and there was a small performance loss in the conversion to HuggingFace which explains the 69.7 -> 66.3 I think.

<img width="824" alt="Screen Shot 2022-12-30 at 12 52 39 AM" src="https://user-images.githubusercontent.com/15899312/210039048-1df4001b-9b5e-4541-ab06-889570fc67bf.png">
Repository: EleutherAI/lm-evaluation-harness
Issue: How to load pretrained ckpts from local machine
Root cause: Documentation Deficiency

How to load a ckpt from local machine such as this:

python main.py \
    --model gpt2 \
    --model_args pretrained=EleutherAI/gpt-neo-2.7B \
    --tasks lambada_openai,hellaswag \
    --device 0

Hi @sanyalsunny111, you should be able to pass the path to your checkpoint through the `pretrained` model args option. For example,
```
--model_args pretrained="/path/to/checkpoint",tokenizer="tokenizer-name"
```
Note, you may be required to pass in the tokenizer name. Let us know if you come across any issues.

Thank you for answering @jon-tow just 2 last questions
1. Can we pass a local checkpoint with ckpt.pth and not ckpt.db as I think this code is using .db checkpoints?
2. How to pass checkpoints and tokenizers of different steps ?  such as this 
"EleutherAI/pythia-19m-deduped",
  revision="step3000",
  cache_dir="./pythia-19m-deduped/step3000"
  
  as shown in pythia example https://github.com/EleutherAI/pythia#quickstart

1. I'm not exactly sure how HuggingFace handles loading checkpoints of different file types but best to give it a try or check out the HuggingFace forums.
2. For different steps of a Pythia model you would pass in the `revision` model args option. For your example this would look like this:
    ```
    --model_args pretrained="EleutherAI/pythia-19m-deduped",revision="step3000"
    ```
   where again you can set the pretrained option to a local path. Furthermore, `cache_dir` is not a supported arg at this moment so things will be cached to the default HuggingFace cache path unless you manually override it (see [here](https://huggingface.co/docs/datasets/cache#cache-directory)).

Thanks a lot @jon-tow very helpful.

Hey @jon-tow 

Is there a better way to run distributed evaluation for very big checkpoints 12b onwards?
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Anthropic Model-written Eval datasets to harness?
Root cause: Unimplemented Feature Gap

Anthropic recently released a number of evaluation datasets, some partially model-generated.

https://github.com/anthropics/evals
Files also uploaded as JSONL files on github: https://huggingface.co/datasets/Anthropic/model-written-evals/tree/main

To my knowledge all datasets from this paper should be multiple choice, so the implementation shouldn't differ too much from other multiple choice tasks like MMLU.

Would require:

- [ ] Implementing a Huggingface dataset loading script for all evaluation subsets (should require only one script with a number of dataset subsets)
- [ ] Implementing task classes for each dataset

I will take care of this. 

Have you made any progress？

Hi [haileyschoelkopf](https://github.com/haileyschoelkopf), I have successfully implemented a loading script for the Model-written Eval dataset, available at: https://huggingface.co/datasets/khalidalt/model-written-evals. I have also added new columns to capture behavioral and topical attributes. For instance, the "persona" subset contains multiple behavioral tests, such as anti-immigration, agreeableness, and beliefs-abortion-should-be-illegal. These categories have been integrated as columns in the dataset. I would like clarification on whether these behavioral categories should be considered as subsets, given that each one includes multiple instructions and responses stored in .jsonl format. Please advise on the best approach to handle this matter.

Hi @KhalidAlt , thanks so much for this contribution! If it is not too much trouble, I think separating out the separate behavioral categories into their own dataset subsets would be fantastic, so that separate evaluations could be evaluated. 

Thanks so much! Let me know if you need any more help or guidance on this issue. 

Hi @haileyschoelkopf, It is an easy fix. I will do it ASAP. Thanks a lot for your reply and I will let you know if I need any help. Thanks again


Hi @haileyschoelkopf, I have separated the dataset into distinct subsets based on behavioral categories. Each category now has its own dedicated subset, which should facilitate more accurate evaluations and analysis. You can access the updated dataset https://huggingface.co/datasets/khalidalt/model-written-evals . Please let me know if everything is right.
Repository: EleutherAI/lm-evaluation-harness
Issue: URL `http://eaidata.bmk.sh/data/triviaqa-unfiltered.tar.gz` seems inaccessible
Root cause: External Dependency Breakage

Maybe we should replace it with `http://nlp.cs.washington.edu/triviaqa/data/triviaqa-unfiltered.tar.gz`?

Oh I see that it has been fixed by https://github.com/EleutherAI/lm-evaluation-harness/pull/373
Repository: EleutherAI/lm-evaluation-harness
Issue: How to use lm-evaluation-harness for distributed evaluation for extremely large size models
Root cause: Unimplemented Feature Gap

Can somebody please provide me with a way to use distributed evaluation on single or multiple tasks?

Hi @sanyalsunny111 ! Distributed eval (by way of `transformers` native pipeline parallelism support) is currently only supported in the BigScience fork [here](https://github.com/bigscience-workshop/lm-evaluation-harness) and that fork is incompatible with main. The process of porting it over here is still ongoing; see related issue #360 and port https://github.com/EleutherAI/lm-evaluation-harness/pull/381 - you could try running from the 381 branch but no guarantees yet 😄 


@sanyalsunny111 Moving discussion from the PR here to keep the PR focused:

> Can you suggest a main.py and running script for this branch for big models like EleutherAI/Pythia-12b-deduped for some task, say wikitext or lamda_openai? 

You should be able to call it just as you would in the main branch but with the proper model type name `hf-causal`:

```
python main.py \
    --model hf-causal \
    --model_args pretrained="EleutherAI/Pythia-12b-deduped" \
    --tasks lambada_openai
```

Hey @jon-tow, Just to let you know your suggestion works for both small 1b and large 12b models for lamda_openai, wikitext and wnli tasks. Thank you very much sir :)

Just found this warning for wikitext : Token indices sequence length is longer than the specified maximum sequence length for this model (7614 > 2048). Running this sequence through the model will result in indexing errors. 


Glad to hear! Just to reiterate: We don't guarantee this is 100% working so I would advise not to publish results from this branch yet if you're considering doing so 😄 (it could still be useful for sanity-checking downstream perf though).

Those warnings can be ignored; the tokens are manually truncated to avoid indexing errors.
Repository: EleutherAI/lm-evaluation-harness
Issue: kobest_copa,kobest_hellaswag tasks
Root cause: Documentation Deficiency

in model card of EleutherAI/polyglot-ko-5.8b, they use kobest_copa,kobest_hellaswag tasks, but not found error. how can I install these tasks?

I'm not able to find the exact branch/fork but this looks to be the implementation: https://github.com/EleutherAI/lm-evaluation-harness/pull/329/files

CC @Taekyoon for confirmation

KOBEST benchmark is in the "polyglot" branch: https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot
Repository: EleutherAI/lm-evaluation-harness
Issue: Add function to list all generation / log probs task separately.
Root cause: Unimplemented Feature Gap

Something that would be useful imo would be a function to separate out generation from log likelihood tasks / be able to tell at a glance whether a task calls `_loglikelihood_tokens` or `greedy_until` or both. 

We find that generation tasks are often a lot slower / more bug prone, and we want to skip them in certain contexts (e.g during training)

I guess the simplest way to implement this would be to add a property to tasks (`task.is_generation_task` or `task.request_type` or something. 

If the maintainers agree this is necessary, and decide how they would like this implemented, I will submit the PR ASAP.  

Closed as we have solved this problem.

Hey Stella - working a little with LM eval again, and finding myself coming back to this issue. What is the canonical solution here? I don't see an obvious way to do this still
Repository: EleutherAI/lm-evaluation-harness
Issue: Carper models don't work, when GPT-J does
Root cause: Validation Gap

When I run

```
python main.py     --model hf-causal     --model_args pretrained=CarperAI/openai_summarize_tldr_ppo --tasks pubmedqa     --device cuda:0 --num_fewshot=5
```

I get 

```
Selected Tasks: ['pubmedqa']
Traceback (most recent call last):
  File "/mnt/ssd-0/arthurworkspace/repos/lm-evaluation-harness/main.py", line 101, in <module>
    results = evaluator.simple_evaluate(
  File "/mnt/ssd-0/arthurworkspace/repos/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/ssd-0/arthurworkspace/repos/lm-evaluation-harness/lm_eval/evaluator.py", line 64, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/mnt/ssd-0/arthurworkspace/repos/lm-evaluation-harness/lm_eval/base.py", line 115, in create_from_arg_string
    return cls(**args, **args2)
  File "/mnt/ssd-0/arthurworkspace/repos/lm-evaluation-harness/lm_eval/models/huggingface.py", line 155, in __init__
    self.tokenizer = self._create_auto_tokenizer(
  File "/mnt/ssd-0/arthurworkspace/repos/lm-evaluation-harness/lm_eval/models/huggingface.py", line 374, in _create_auto_tokenizer
    tokenizer = super()._create_auto_tokenizer(
  File "/mnt/ssd-0/arthurworkspace/repos/lm-evaluation-harness/lm_eval/models/huggingface.py", line 221, in _create_auto_tokenizer
    tokenizer = self.AUTO_TOKENIZER_CLASS.from_pretrained(
  File "/home/conjecture/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 679, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/conjecture/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1788, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'CarperAI/openai_summarize_tldr_ppo'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'CarperAI/openai_summarize_tldr_ppo' is the correct path to a directory containing all relevant files for a GPT2TokenizerFast tokenizer.
```

Even though when I use `EleutherAI/gpt-j-6b` everything works. Any idea what's up?

Fixed by adding `"tokenizer": "gpt2"` to the here: https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/base.py#L113 . I may come back to this to find a better, permanent fix

@ArthurConmy The issue is that the `HFLM` class instantiates the tokenizer with the pre-trained model name if it is not explicitly passed in and since the tokenizer config was never pushed to the hub for `CarperAI/openai_summarize_tldr_ppo` it will error out. https://github.com/EleutherAI/lm-evaluation-harness/blob/83dbfbf6070324f3e5872f63e49d49ff7ef4c9b3/lm_eval/models/gpt2.py#L40-L48

 One way to handle this from the command line is to set the appropriate `model_args` option e.g. 
 ```bash
 --model_args pretrained="CarperAI/openai_summarize_tldr_ppo",tokenizer="EleutherAI/gpt-j-6B"
 ```
Repository: EleutherAI/lm-evaluation-harness
Issue: HuggingFaceAutoLM greedy_until mismatch
Root cause: Interface Contract Mismatch

Hi!
For HuggingFaceAutoLM, the definition of greedy_until is 'def greedy_until(self, requests: List[Tuple[str, dict]]) -> List[str]:'
which does not match with  BaseLM 'def greedy_until(self, requests):'
This will cause failure when using HF models for greedy_until request.

to reproduce:
python main.py  --model hf-causal --model_args pretrained=EleutherAI/gpt-j-6B --tasks gsm8k --device cuda:0 --limit 10
I

Hi, thank you for reporting this!

Until I get this fixed, if you're running on a single device (as it seems from your example command) then you can run successfully ( and with the same result you would get once this is fixed) via `--model gpt2` instead to use the older `HFLM` class. Hope this helps in the meantime!

> Hi, thank you for reporting this!
> 
> Until I get this fixed, if you're running on a single device (as it seems from your example command) then you can run successfully ( and with the same result you would get once this is fixed) via `--model gpt2` instead to use the older `HFLM` class. Hope this helps in the meantime!

Got it. Thanks!

https://github.com/EleutherAI/lm-evaluation-harness/pull/427 should fix this, tested locally! Note that the model names change in this PR, so now hf-causal calls what was formerly known as --model hf or --model gpt2 and --model hf-causal-experimental calls the former hf-causal.

Please let me know if there continue to be issues and I'll follow up right away. Sorry for the trouble!
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement WIkitext for GPT-2 replication
Root cause: Unimplemented Feature Gap

It's a HF dataset, so should be straightforward.  I can add it in.

- [x] Data processing code implemented
- [ ] Evaluation implemented

This should be modeled after the `BoolQ` task in `lm_eval/tasks/suerglue.py`



> what the heck is up with wikitext? wikitext2 and wikitext103 have the exact same test set but gpt2 paper reports different scores on both; moreover, it doesnt even seem like there's a fixed normalization constant between both numbers. how the heck did that happen and how can we implement it?

the hf implementation for wikitext is kinda questionable -- it has one paragraph per document, rather than one wiki page per document. see explorer below for example. this doesnt seem like what we want

https://huggingface.co/datasets/viewer/?dataset=wikitext
Repository: EleutherAI/lm-evaluation-harness
Issue: ModuleNotFoundError: No module named 'omegaconf'
Root cause: Environment Incompatibility

After install:

```
pip install lm-eval
```

i got:

```
(model_eval) root1@root1-System-Product-Name:~/lx/lm-evaluation-harness$ python main.py --model hf-causal --model_args pretrained=decapoda-research/llama-7b-hf --device cuda  --tasks boolq
Traceback (most recent call last):
  File "/home/root1/lx/lm-evaluation-harness/main.py", line 6, in <module>
    from lm_eval import tasks, evaluator
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 5, in <module>
    import lm_eval.base
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/base.py", line 16, in <module>
    from lm_eval import utils
  File "/home/root1/lx/lm-evaluation-harness/lm_eval/utils.py", line 10, in <module>
    from omegaconf import OmegaConf
ModuleNotFoundError: No module named 'omegaconf'
```

Hi,

You can fix this via `pip install -e .` in your lm-evaluation-harness repository folder to install the most up-to-date dependencies!

installing via 
```bash
pip install lm-eval
```
installs version 0.3.0 of the Evaluation Harness from PyPI but this version is older than the `master` branch of this repository, and v0.3.0 did not yet have `omegaconf` as a dependency.

Looks like Github's having issues with Pull Requests (https://www.githubstatus.com/) but I've created a patch for the README which I will push as soon as it allows:

-------------------
To install `lm-eval` from the github repository main branch, run:

```bash
git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
```

To install additional multilingual tokenization and text segmentation packages, you must install the package with the `multilingual` extra:

```bash
pip install -e ".[multilingual]"
```

> Hi,
> 
> You can fix this via `pip install -e .` in your lm-evaluation-harness repository folder to install the most up-to-date dependencies!
> 
> installing via
> 
> ```shell
> pip install lm-eval
> ```
> 
> installs version 0.3.0 of the Evaluation Harness from PyPI but this version is older than the `master` branch of this repository, and v0.3.0 did not yet have `omegaconf` as a dependency.
> 
> Looks like Github's having issues with Pull Requests (https://www.githubstatus.com/) but I've created a patch for the README which I will push as soon as it allows:
> 
> To install `lm-eval` from the github repository main branch, run:
> 
> ```shell
> git clone https://github.com/EleutherAI/lm-evaluation-harness
> cd lm-evaluation-harness
> pip install -e .
> ```
> 
> To install additional multilingual tokenization and text segmentation packages, you must install the package with the `multilingual` extra:
> 
> ```shell
> pip install -e ".[multilingual]"
> ```

got it, thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Where is the application of utils.select_continuation_from_batch_left_padding()
Root cause: Unimplemented Feature Gap

In lm_eval/models/huggingface.py line 417, implementation of utils.select_continuation_from_batch_left_padding() can't be found

I also meet this problem.....

@haileyschoelkopf Hello, I see this code is written by you, can you tell me how to fix it

Hi, sorry for the trouble, I apologize for the issue! I'll try to push a fix for this as soon as I'm able, hopefully tomorrow if I have time. 

for the time being, running with `--model gpt2` can be used as a substitute for any autoregressive causal decoder architectures, if this helps!

Working on this today.

Hailey, thank you for reaction, this change is much anticipated. Could you also look at #412, please? It seems also about the behavior of the huggingface module slightly differs from the surrounding code. Thanks!

This should be addressed by PR #427 ! Please let me know if any issues persist.
Repository: EleutherAI/lm-evaluation-harness
Issue: Support For Peft Adapters
Root cause: Unimplemented Feature Gap

Hi all!

Thanks for a great repo, it's really really an awesome piece of software :)

I made a quick change to the `HuggingFaceAutoLM` class to allow for [peft](https://github.com/huggingface/peft) adapters to be added to a base model. 

Is this something that would you al would be interested in having in the repo? If so, I'm happy to open a PR for this.
Repository: EleutherAI/lm-evaluation-harness
Issue: load_in_8_bit supported?
Root cause: Unimplemented Feature Gap

Is it possible to pass the `load_in_8_bit` argument to the model constructor?

Looking at 
https://github.com/EleutherAI/lm-evaluation-harness/blob/221186c7aeaa977978a335615bc84023216d3254/lm_eval/models/huggingface.py#L58
I don't see anything obvious.

It is now!
Repository: EleutherAI/lm-evaluation-harness
Issue: Support simple text-completion based API endpoint
Root cause: Unimplemented Feature Gap

I think the OpenAI Model could be evaluated with this repository.

They only provide an API which is agnostic for the model specification.

The only thing is that they gave us output text and log-probs when we give input text.

It seems to be generalized to support any API endpoints that accepts "input_text" and outputs "output_text" and "log_probs".

Is there any move to support these kind of features? If it is, how to apply it?

I think it could be nice to follow the API spec of OpenAI.
Repository: EleutherAI/lm-evaluation-harness
Issue: 运行llama_7b报错，TypeError: Descriptors cannot not be created directly.
Root cause: Environment Incompatibility

运行llama_7b报错，
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

请问要怎么解决

Hi! Have you tried running `pip install protobuf==3.20.0` and rerunning? This should fix the error message reported here.

Thank you. Problem solved.

> Hi! Have you tried running `pip install protobuf==3.20.0` and rerunning? This should fix the error message reported here.
Repository: EleutherAI/lm-evaluation-harness
Issue: how add new tasks？
Root cause: Documentation Deficiency

May I ask how to add testing tasks and how to test local tasks

Hi! We have documentation written for how to add a new task here: https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_guide.md 

If you still have questions after reading this, feel free to post again under this issue! This would be helpful to improve our documentation, which we intend to do over the course of the next month.

Our laboratory is planning to develop software based on your code. May I ask when your upload test set function will be implemented?

I would greatly appreciate it if you could answer!
Repository: EleutherAI/lm-evaluation-harness
Issue: Unable to Reproduce RACE Zero-shot accuracy reported in LLaMA paper [LLaMA-7B]
Root cause: Others

RACE Zero-shot accuracy reported in the paper:
<img width="456" alt="image" src="https://user-images.githubusercontent.com/6135246/233840133-1e96f913-c3b1-45a1-b284-6e70318c7b31.png">

My results are as follows:
<img width="744" alt="image" src="https://user-images.githubusercontent.com/6135246/233840172-69f34957-18ea-4c21-b36d-7f853f851caf.png">

Anybody help?

This issue is also reported at https://github.com/EleutherAI/lm-evaluation-harness/issues/443 so I’m going to close it as a duplicate.