zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: How to make sure the harness utilizes ONLY the gpu mentioned with --devices
Root cause: Documentation Deficiency

I'm running a bunch of experiments on a machine with 4 GPUs. So, I divide the experiments in 4 and route them to each GPU. Surprisingly, they lead to cuda OOM. But when I try to reproduce the errors by running those 4 sets one by one, I don't run into any trouble. My guess is this happens because when I mention --device cuda:3, even though the 'main' work happens on cuda:3, there is something that happens in cuda:0 as well. I saw that when I run the program with --device cuda:3, both cuda:0 and cuda:3 get significant utilization. So, I want to know 2 things :
1. What does rank 0 gpu do ?
2. How to make sure that all the work is done ONLY on the GPU mentioned in the --devices tag ?

Thanks!
 

Setting CUDA_VISIBLE_DEVICES aptly worked for me (apparantly doing the above is not enough). So, if I want to exclusively use gpu 1, I had to set CUDA_VISIBLE_DEVICES=1 and then use --devices cuda:0
Repository: EleutherAI/lm-evaluation-harness
Issue: [big refactor]mmlu is needed
Root cause: Documentation Deficiency

Where can i find mmlu benchmark in big refactor branch?

I have implement some kind of rwkv world benchmark adapted from huggingface model,
https://github.com/xiaol/rwkv-hf-lm-evaluation-harness

not so sure how to proceed to improve the benchmark result based on the evaluation code.

Hi! I believe you're looking for [this](https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor/lm_eval/tasks/mmlu)
Repository: EleutherAI/lm-evaluation-harness
Issue: PubMedQA evaluation is done incorrectly
Root cause: Algorithmic Error

The PubMedQA evaluation is done with the labeled subset of 1000 examples currently, but the actual test set is 500 examples from that labeled subset so it's actually being evaluated on more examples than it is typically done.

I can fix this and put in a PR since I'll need to anyway...
Repository: EleutherAI/lm-evaluation-harness
Issue: [big-refactor] Error in wildcards for task names
Root cause: Algorithmic Error

Wildcards feature to evaluate tasks with `*` is no longer working with the new changes to main.py on lines 138-144

```
        if task_missing != []:
            missing = ", ".join(task_missing)
            eval_logger.error(
                f"Tasks were not found: {missing}\n"
                f"{SPACING}Try `lm-eval -h` for list of available tasks",
            )
            raise ValueError(f"Tasks {missing} were not found.")
```
The raise erroneously exists even when we have correct tasks in the `task_names` list. `task_names` does get the correct overlap via `task_names = utils.pattern_match(tasks_list, ALL_TASKS)` but the new change to catch missing tasks incorrectly includes the wildcard phrase and exists with an error.
eg: 
```
python main.py --model hf --model_args="facebook/opt-350m" --task xnli_*

2023-10-02:17:50:42,116 ERROR    [main.py:140] Tasks were not found: xnli_*
                                               Try `lm-eval -h` for list of available tasks
Traceback (most recent call last):
  File "/home/toolkit/workspace/research-now-llm/evaluation/research-lm-evaluation-harness/main.py", line 216, in <module>
    main()
  File "/home/toolkit/workspace/research-now-llm/evaluation/research-lm-evaluation-harness/main.py", line 144, in main
    raise ValueError(f"Tasks {missing} were not found.")
ValueError: Tasks xnli_* were not found.
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Prompt Templating
Root cause: Documentation Deficiency

So this is kind of a pretty basic inquiry. I was wondering if the model inputs(for instruction following models) shouldn't be wrapped around the template that it was trained on, say for example, the Alpaca prompt template. And without this wouldn't the model evaluation kind of be in error?

Please do let me know how to wrap the model inputs around a custom template that the model actually expects.

This is supported in the Big-Refactor batch, coming to main soon!

Can you show me an example of doing this, please. Maybe wrapping around the alpaca template?

@lintangsutawika is currently putting together a demo of how this works! Should be out by the end of the week.

@StellaAthena could you put the link to the demo here? 

@lintangsutawika What's the current status of this? IIRC you showed me a working demo... does it just need to be uploaded?

The original request of this issue was a bit different. Essentially it was if it's possible to wrap a prompt with another template like the Alpaca template.

But to show how templating works, (and the general overview of how the configs in lm-eval work), yes, just need to upload it. Will make a PR for that.
Repository: EleutherAI/lm-evaluation-harness
Issue: CoQA no longer works
Root cause: External Dependency Breakage

```
Traceback (most recent call last):
  File "/fsx/BlinkDL/conda/lib/python3.10/site-packages/datasets/builder.py", line 1629, in _prepare_split_single
    for key, record in generator:
  File "/admin/home-blinkdl/.cache/huggingface/modules/datasets_modules/datasets/coqa/26139fc8533060e3fa9352d04f06a83822d89c60c5589c80592a6f7aa36cd375/coqa.py", line 184, in _generate_examples
    data = json.load(f)
  File "/fsx/BlinkDL/conda/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/fsx/BlinkDL/conda/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/fsx/BlinkDL/conda/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/fsx/BlinkDL/conda/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

This is now working! I'm not sure when it was fixed, but it works on both master and big-refactor
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the Natural Questions evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> In this section we measure GPT-3’s ability to answer questions about broad factual knowledge. Due to the immense amount of possible queries, this task has normally been approached by using an information retrieval system to find relevant text in combination with a model which learns to generate an answer given the question and the retrieved text. Since this setting allows a system to search for and condition on text which potentially contains the answer it is denoted “open-book”. [RRS20] recently demonstrated that a large language model can perform surprisingly well directly answering the questions without conditioning on auxilliary information. They denote this more restrictive evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better and we test this hypothesis with GPT-3. We evaluate GPT-3 on the 3 datasets in [RRS20]: Natural Questions [KPR+19], WebQuestions [BCFL13], and TriviaQA [JCWZ17], using the same splits. Note that in addition to all results being in the closed-book setting, our use of few-shot, one-shot, and zero-shot evaluations represent an even stricter setting than previous closed-book QA work: in addition to external content not being allowed, fine-tuning on the Q&A dataset itself is also not permitted.

- [X] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

_Note_: HuggingFace includes this in its datasets package. 

https://huggingface.co/datasets/natural_questions

_Warning_: This dataset is super big.

> _Warning_: This dataset is super big.

How big is “super big”?

97G.

what the fuck. Why are we not training on this. 

Ah, dev set is only 1G. But we should add train set to the pile. 

We would need to dedupe this with Wikipedia, since the bulk of it is just the HTML of Wikipedia pages.

I can claim this

> I can claim this

Assigned!

would love to take this on if help on implementing the evaluation is still needed?

> would love to take this on if help on implementing the evaluation is still needed?

Yes this would be quite helpful. Thanks!

I think Natural Questions is implemented already?  https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/naturalqs.py

@haileyschoelkopf Some methods are not implemented, they raise `NotImplementedError`

Ah you're right sorry!--I'm not sure why this was originally merged then. It's not in the task registry though so it should be alright to keep in the repo until the refactor is done, at which point we can decide what to do with it

I wonder what the progress of NQ eval is and if any help is needed?

@memray I am under the impression that is hasn't been implemented and help is need.

+1

+1

Closed by #789 which implements the NaturalQs dataset split used by Llama and (possibly, unconfirmed) used by PaLM and more!
Repository: EleutherAI/lm-evaluation-harness
Issue: Installation leaks the `scripts` directory to site-packages
Root cause: Architectural Constraint

Steps to reproduce

```shell
$ virtualenv venv
$ source venv/bin/activate
$ ls venv/lib/python3.10/site-packages | grep scripts
$ pip install pip install https://github.com/EleutherAI/lm-evaluation-harness/archive/refs/heads/master.zip
$ ls venv/lib/python3.10/site-packages | grep scripts
scripts
```

This is causing issues in Lit-GPT where we have a local `scripts` directory: https://github.com/Lightning-AI/lit-gpt/issues/558

You should be able to tune what https://github.com/EleutherAI/lm-evaluation-harness/blob/master/setup.py#L15-L17 includes by manually excluding it as described in https://stackoverflow.com/questions/608855/excluding-a-top-level-directory-from-a-setuptools-package

Thank you!
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation on local dataset 
Root cause: Documentation Deficiency

Hi, Thanks for maintaining and creating this awesome LM benchmarking framework. However, I have some questions and would appreciate it if I get some directions

1. Right now, I see Harness has a way to add a newer dataset. However, if I have a dataset on S3 bucket, can I register that evaluation dataset using a harness?

2. Are models other than hugging face supported (example: GGML, GGUF)? Also, are all models supported here (encoder-decoder-like models: T5 and Decoder-like models like Llama?)

Hi, thanks for your interest in this framework!

1. We currently support any HF dataset format datasets. In the `big-refactor` branch, which is soon to be, we also allow people to pass any kwargs to the HF dataset constructor, so in this way we definitely support locally stored-on-disk datasets, though I'm less familiar with the extent to which HF datasets supports S3.
2. All model types in HF are supported. We have a GGML PR in progress here: #617 , however it is not yet completed--help here would be appreciated, else it is on our roadmap down the line once we release a new version.

Please feel free to reopen this if there are any further questions on this!

Awesome thanks @haileyschoelkopf for the update.
Repository: EleutherAI/lm-evaluation-harness
Issue: GPT-2 Scores on lambada_openai Don't Match Paper
Root cause: Validation Gap

While trying to replicate the GPT-2 scores on the lambada_openai benchmark, I found there was a discrepancy in my results relative to what's reported in the paper. Specifically, I'm getting PPL = 10.63 vs. the PPL of 8.63 that was published. 

For ease of replication, here's the command I'm running: 

```
python -m lm_eval \
    --model hf \
    --model_args pretrained=gpt2-xl \
    --tasks lambada_openai \
    --output_path /outputs/10-10-23-gpt2-xl-lambda_openai \
    --device cuda:0 \
    --batch_size 8
```

Digging deeper into the [GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), I noticed that the paper states "We observe gains of 2.5 to 5 perplexity for GPT-2 with these de-tokenizers."  Perhaps the delta here is because de-tokenizers are not used. 

Note the first author, Alec Radford, shares PPL scores for GPT-2 without use of de-tokenization [here](https://www.reddit.com/r/MachineLearning/comments/oye64h/r_struggling_to_reproduce_perplexity_benchmarks/); unfortunately, scores for the lamabda dataset weren't mentioned in his reply. 

Running lambada with the `_openai` suffix is supposed to call the same detokenized dataset that OpenAI used. At one point the dataset was made publicly available, but it isn't anymore as far as I know.

Are you using the `main` branch or `big_refactor`?

I'm using big_refactor 

@jasonkrone can you reproduce the numbers from the [GPT-NeoX-20B paper](https://arxiv.org/abs/2204.06745)?

Yes, I'm getting the same scores as shown in the paper for GPTJ-6B on lambada. 

I also noticed that the [GPT-Neo repo](https://github.com/EleutherAI/gpt-neo/tree/v1.1) list the same result I'm getting on lambada for GPT-2 (i.e. PPL = 10.63 vs. the published PPL of 8.63). 

> Yes, I'm getting the same scores as shown in the paper for GPTJ-6B on lambada.
> 
> I also noticed that the [GPT-Neo repo](https://github.com/EleutherAI/gpt-neo/tree/v1.1) list the same result I'm getting on lambada for GPT-2 (i.e. PPL = 10.63 vs. the published PPL of 8.63).

Okay, so that means that we've always been consistent in how we calculate it. It's just that we didn't notice we weren't reproducing their numbers exactly. We didn't notice because who cares about GPT-2 and since the GPT-3 API models aren't the same models as the GPT-3 paper, even if we did have it exactly we'd still see some variance. Interesting.
Repository: EleutherAI/lm-evaluation-harness
Issue: Readme not clear

This sentence in the README.md is not clear:

```
Models that are loaded via either transformers.AutoModelForCausalLM (autoregressive, decoder-only GPT style models) or transformers.AutoModelForSeq2SeqLM (such as encoder-decoder models like T5) in Huggingface are supported via Support for this model type is currently pending.
```

what does it mean by "via Support for this model type is currently pending" -- perhaps this could be rewritten to be more clear.

Thanks! I've fixed this.
Repository: EleutherAI/lm-evaluation-harness
Issue: can we get validation loss?

Hi, I tried to integrate your fabulous evaluation framework into my training scripts based on HF trainer. 
I modified `HFLM` to support loaded HF model and triggers `evaluate` with custom callbacks.  Currently, the evaluation works like a charm and I get to see evaluation metrics throughout the training.  And it would be a lot nicer to see validation loss curve. 

So, is it possible to get validation loss?



You can use the JsonPerplexity task to evaluate perplexity on an arbitrary text dataset.
Repository: EleutherAI/lm-evaluation-harness
Issue: Hellaswag in other languages?
Root cause: Documentation Deficiency

Hi, I need to make some test with some tasks like hellaswag in some languages, I read how to make a new task but it's a bit difficult. I need to know if there is a simple way to create a little dataset, something like 20 rows, and adding it to the tasklist without uploading it to huggingface. Many thanks

If you have it locally you can load it from a local path by supplying that local path to the data loader.

I don't know how to load it from a local path. A little help?

It uses the [datasets.load_dataset()](https://huggingface.co/docs/datasets/loading) method so you can either edit the dataset related options in the yaml (dataset_path, dataset_name and dataset_kwargs) or edit the method [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/cf617ab1ee83e6ea24a33ba535fdfa8b25557267/lm_eval/api/task.py#L702) directly.
Repository: EleutherAI/lm-evaluation-harness
Issue: PAWS-X Scores Significantly Differ between `master` vs `big-refactor`
Root cause: Algorithmic Error

Score difference between:

main:

<img width="752" alt="MicrosoftTeams-image (2)" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/910450/f135f5a6-c0fe-4a23-8cb3-c75f4cde498b">

big-refactor

<img width="740" alt="MicrosoftTeams-image (1)" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/910450/da3b32f4-a04b-4e5e-ae7c-350acdb68106">

checking the (obvious) differences
- Looking at the implementation's master uses: `loglikelihood_rolling` vs big-refactor uses `loglikelihood` for multichoice.

cc: @lintangsutawika

Thanks @nitsanluke , this is intentional. The master branch uses `loglikelihood_rolling` and observe the `[MASK]` token. Since most language evaluated are decoder-only models that don't use a `[MASK]` token it might be better to evaluate the whole sentence, i.e '{{[sentence1+", right? Yes, "+sentence2, sentence1+", right? No, "+sentence2]}}' instead of `sentence1+", right? [MASK], "+sentence2`.

If you think we should still retain the master version. We may need a new `output_type` to support the use of `[MASK]` (and also allow users to set their own mask token).

Thanks @lintangsutawika for the clarification!
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement GLUE STSB
Root cause: Unimplemented Feature Gap

Everything else in glue should be implemented now

Relevant resources

https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark
https://www.aclweb.org/anthology/S17-2001/

Deferring because STSB is *weird*
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
Root cause: Validation Gap

I'm trying to see if I can get some of my old lm-evaluation-harness jobs to work with the big refactor branch, and when I try to run the following command:

`python -m lm_eval --model hf --model_args pretrained=EleutherAI/pythia-70m --tasks squadv2 --num_fewshot 0 --device cuda:0 --batch_size 2 --output_path ../output/pythia-70m-squad2-0-shot-refactor`

I get the following stacktrace:

`Traceback (most recent call last):
  File "C:\Users\emily.vanark\.conda\envs\lm_eval\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\emily.vanark\.conda\envs\lm_eval\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\emily.vanark\git\generative_ai\lm-eval\lm-evaluation-harness\lm_eval\__main__.py", line 227, in <module>
    cli_evaluate()
  File "C:\Users\emily.vanark\git\generative_ai\lm-eval\lm-evaluation-harness\lm_eval\__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "C:\Users\emily.vanark\git\generative_ai\lm-eval\lm-evaluation-harness\lm_eval\utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "C:\Users\emily.vanark\git\generative_ai\lm-eval\lm-evaluation-harness\lm_eval\evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "C:\Users\emily.vanark\git\generative_ai\lm-eval\lm-evaluation-harness\lm_eval\api\model.py", line 141, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until`

Is this somehow user error, or is there maybe something actually wrong with the code?  (Does this happen to anyone else with that command?  Is there something obviously wrong with my command?)

I'm getting the same error and this is the command I'm trying to use:
`accelerate launch -m lm_eval --model hf --model_args pretrained=/local/path/to/model --tasks hellaswag --num_fewshot 10 --batch_size 2 --output_path /local/path/to/otuput/directory`

Hi, I'm so sorry for the bug! This is closed by #931 .
Repository: EleutherAI/lm-evaluation-harness
Issue: GPT2-XL does not work in ToxiGen
Root cause: Resource Mishandling

When trying to evaluate `gpt2-xl` on ToxiGen (all other sizes of GPT-2 work fine), the CLI simply stops with a quit command after the model and the task are loaded.

```bash
python main.py \
    --model hf-causal \
    --model_args pretrained=gpt2-xl \
    --tasks toxigen  \
    --device cuda:0
```

Is this a known problem with the 1.5B version of GPT-2?

This is the output:

```bash
2023-10-19 01:11:02.566168: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Selected Tasks: ['toxigen']
Using device 'cuda:0'
Downloading (…)lve/main/config.json: 100% 689/689 [00:00<00:00, 4.31MB/s]
Downloading pytorch_model.bin: 100% 6.43G/6.43G [00:37<00:00, 169MB/s]
^C
```

This is wild, I have no idea what could be causing it. This doesn't happen with any other model or task?

I wasn't able to replicate this bug from the most recent commit on `master`. Could this issue potentially be an OOM while the model is being loaded from disk? 

For reference, evaluation runs with 16GB RAM and an 8GB GPU. Using commit `906ef94...`




OK. I think it was an OOM problem. I tried this time on a Collab notebook, and I had the same problem using the T4 GPU, but with the V100, everything works. But this means that it actually takes more than 8GB of VRAM since the T4 has around 16 GB of RAM to give, right?
Repository: EleutherAI/lm-evaluation-harness
Issue: Loading GPTQ model, unexpected keyword argument 'quantized'
Root cause: Documentation Deficiency

I'm trying to run the CMMLU evaluation on a GPTQ model I have saved locally with the following command:

python main.py --model hf-causal --model_args pretrained=C:/Users/admin/Documents/models/TheBloke_StableBeluga-13B-GPTQ, quantized=model.safetensors --tasks cmmlu-clinical_knowledge --device cuda:0

This matches the format for running GPTQ show in the repo readme. However, I am getting the following error:

main.py: error: unrecognized arguments: quantized=model.safetensors

To try and solve this, I wrapped all the model_args in parenthesis as follows:

python main.py --model hf-causal --model_args "pretrained=C:/Users/admin/Documents/models/TheBloke_StableBeluga-13B-GPTQ,quantized=model.safetensors" --tasks cmmlu-clinical_knowledge --device cuda:0

This then causes the following error:

 File "C:\Users\admin\Documents\LLMs\evaluation\lm-evaluation-harness\lm_eval\base.py", line 115, in create_from_arg_string
    return cls(**args, **args2)
TypeError: HFLM.__init__() got an unexpected keyword argument 'quantized'

I asked ChatGPT and its response regarding this was:

This error message suggests that while you've structured the command correctly based on the README, the underlying code in lm-evaluation-harness doesn't support or recognize the quantized keyword argument for the HFLM class.

I was hoping to get some help on how to load GPTQ models for evaluation. Thanks in advance for any help.


Ok I found the issue, model needs to be --model hf-causal-experimental to use GPTQ models.
Repository: EleutherAI/lm-evaluation-harness
Issue: Prompt Structure
Root cause: Documentation Deficiency

Hey Team,

Thank you for your excellent work. I'm having trouble locating information regarding the recommended prompt structure for fine-tuning when using this repository for LLM evaluation. Alternatively, if I've used a format like "### Question: .., ### Answer: ...," how can I specify it for evaluation? I am using the Hugging Face TRL library for fine-tuning. 

Regards,
Yash

We already support this on `big-refactor`! Check out the [task guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/docs/task_guide.md) for details.