zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: PERSPECTIVE_API_KEY is not set is not set
Root cause: Validation Gap

I am able to evaluate the model and run codes a few days ago. After I pulled the latest version (759da8d53bae2d1d1f7796aa9dddcf8c84aaa161). I hit the error `PERSPECTIVE_API_KEY is not set` whatever the task or model I set.

```
accelerate launch --model hf --model_args  pretrained=xlm-roberta-large,low_cpu_mem_usage=True --tasks paws-x --batch_size 16
```

What does this API key mean here? I don't think simply using xlmr needs that.

The issue is from this line: https://github.com/EleutherAI/lm-evaluation-harness/blob/759da8d53bae2d1d1f7796aa9dddcf8c84aaa161/lm_eval/tasks/realtoxicityprompts/metric.py#L12C17-L12C17 It will try to find an API key before running evaluation. 

A simple fix is set this to any value in the env if you don't use it, e.g.,  `export PERSPECTIVE_API_KEY=None`

Thanks for raising this, and sorry about the bug! Have pushed a hotfix at #787 .
Repository: EleutherAI/lm-evaluation-harness
Issue: HellaSwag testing on validation
Root cause: Documentation Deficiency

Hi, I was checking some details in the HellaSwag evaluation and I noticed that the evaluation [runs on the validation](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/hellaswag.py#L41) split of the dataset, even if the test set is available on the [HuggingFace dataset](https://huggingface.co/datasets/hellaswag/viewer/default/test). I can imagine that there's a good reason for it, but I don't know what it is. Any suggestions? 

Hi! The reason for this is because the test set does not make labels public, so it can't be evaluated automatically by us. You'd have to run, save the predictions on the test set, and upload them to the Hellaswag creators' leaderboard to get test set scores.
Repository: EleutherAI/lm-evaluation-harness
Issue: Running multiple evaluation processes causes database locked error
Root cause: Resource Mishandling

Error: sqlite3.OperationalError: database is locked




Hi! If you are not in need of caching your model's generations, then you can run with the `--no_cache` argument to avoid this error. 

If you do want to cache, the `big-refactor` branch and upcoming release require passing a cache location, which allows each evaluation process to use a different database. (this separation isn't ideal, but avoids the database locked error).
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the StoryCloze evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> We next evaluate GPT-3 on the StoryCloze 2016 dataset [MCH+16], which involves selecting the correct ending sentence for five-sentence long stories. Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot setting (with K = 70). This is still 4.1% lower than the fine-tuned SOTA using a BERT based model [LDL19] but improves over previous zero-shot results by roughly 10%.

- [X] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Form to get access is here:

https://cs.rochester.edu/nlp/rocstories/

I have the download links and can provide them to anyone who DM’s me on Discord.

Implemented in #300
Repository: EleutherAI/lm-evaluation-harness
Issue: Multi-device evaluation with data parallel
Root cause: Unimplemented Feature Gap

Hi team.

I followed [this issue](https://github.com/EleutherAI/lm-evaluation-harness/issues/573) to use multiple gpus for evaluation. However, the multi-gpu you implemented seems to be in model parallel, instead of data parallel, which actually makes the evaluation even slower... How to use data parallel in lm-eval?

Thanks!

I have the same problem here! 

OpenComapss provides efficient evaluation support for LLM, https://github.com/InternLM/opencompass 
Feel free to try OpenCompass.

Hi! we currently support data-parallel evaluation in the `big-refactor` branch, which will be our new version release once we have finished porting all currently supported branches to it. This branch is much faster than the current main branch!
Repository: EleutherAI/lm-evaluation-harness
Issue: Will the harness support vLLM in the future?
Root cause: Unimplemented Feature Gap

As the vllm has a very good inference performance. Will it be supported by  harness as backend in the future?

Hi, yes we'd love to support vLLM in the future, and it's on our roadmap to support it (though for now we don't have a timeline on finding time to add it, so contributions are welcome!)

A partial integration of vllm can be found here: https://github.com/wellecks/lm-evaluation-harness/blob/master/lm_eval/models/vllm.py though I believe this does not support Tensor parallelism. 

Ideally however, we'd have support for the API version of vllm so that one can host a vllm endpoint on their machine and then run evaluations that query this locally hosted api.
Repository: EleutherAI/lm-evaluation-harness
Issue: Perplexity Evaluation broken with "Big refactor" branch (huge differences on 'wikitext')
Root cause: Algorithmic Error

I`m trying to evaluate perplexity with the lm-eval-harness and getting confused:

When running with the [Huggingface LLM Leaderboard commit ](https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463) (i can reproduce LLM leaderboard results for Hellaswag and arc_challenge just fine):
```bash
python main.py --model=hf-causal --model_args="pretrained='meta-llama/Llama-2-7b-hf'" --tasks=wikitext --device cuda:0

  "results": {
    "wikitext": {
      "word_perplexity": 8.792073612124492,
      "byte_perplexity": 1.5015845643537054,
      "bits_per_byte": 0.5864857250704371
    }
  }
```

When running with commit [b5b01a4](https://github.com/EleutherAI/lm-evaluation-harness/tree/b5b01a497e698c78cc8bc1eb575b2c25bca64e35) from the big-refactor-branch:

```bash
python main.py --model=hf-auto --model_args="pretrained='meta-llama/Llama-2-7b-hf'" --tasks=wikitext --device cuda:0

|  Task  |Version|Filter|    Metric     | Value |   |Stderr|
|--------|-------|------|---------------|------:|---|------|
|wikitext|Yaml   |none  |word_perplexity|13.0065|   |      |
|        |       |none  |byte_perplexity| 1.5264|   |      |
|        |       |none  |bits_per_byte  | 0.6102|   |      |
```

And online, I see values of 7.0-7.5 posted for llama2-7b on wikitext.

What is the correct way to evaluate perplexity on wikitext with the eval harness currently and why is there such a huge difference? Is this due to context/stride length? 

Wikitext is even mentioned in the docs as an example task, so I guess there should be some benchmark values?

Thanks!

EDIT:
With the code from the [Huggingface article](https://huggingface.co/docs/transformers/perplexity) and max_length/stride = 512, I get 7.1781.
When adding max_length=512 (for big refactor branch), word_perplexity goes up to 17.755.....


The perplexity scores you quote of 7.0-7.5 are for token-level perplexity (which is not the same thing as word_perplexity!). This is a tokenizer-dependent metric, and its not possible to compare token-level perplexities between models, which is why eval-harness doesn't report it. For reference, `bits_per_byte` is the standard metric to use as it allows a more direct comparison between models with different tokenizers (see Section 3.1 of [The Pile](https://arxiv.org/abs/2101.00027)) paper. 

Still, it's weird that there is a score divergence between `big-refactor` and `master`. I've noticed this with other models too so I'll open a separate issue for this topic. 

It seems like there was some task-specific logic relating to how words and bytes are counted for wikitext specifically! I've fixed this in #833 .
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] IndexError if label > num_of_choices for multiple choice
Root cause: Validation Gap

Ran into an issue for a task in AGIEval where the number of choices were in the range A-C (0-2) but the label was D (3) (wrongly). This isn't an issue for the `acc` as we just check`(pred = argmax(lls)) == gold` but get an index error while computing exact match as `exact_match = int(is_greedy[gold])` and `gold > len(is_greedy) - 1` here. Should fix this so that exact_match and acc are symmetrical . `L1019` in `task.py`

@baberabb I don't yet quite understand, shouldn't label be <= num_of_choices-1?

I'm talking about an edge case where a row is mislabelled. Choices were `0-2`and `gold_label = 3`. Not a problem for acc (always 0), but `main` function broke while indexing exact_match (task metric was just `acc`).
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] Wikitext score divergence
Root cause: Algorithmic Error

There is a minor divergence between Wikitext on `master` and the current implementation of Wikitext on `big-refactor`.

# `master`:
Run with:
```bash
python main.py     --model hf-causal     --tasks wikitext
```
Output is: 
|  Task  |Version|    Metric     | Value |   |Stderr|
|--------|------:|---------------|------:|---|------|
|wikitext|      1|word_perplexity|37.3698|   |      |
|        |       |byte_perplexity| 1.9682|   |      |
|        |       |bits_per_byte  | 0.9769|   |      |

# `big-refactor`:
Run with:
```bash
python main.py     --model hf     --tasks wikitext
```
Output is:
|  Task  |Version|Filter|    Metric     | Value |   |Stderr|
|--------|-------|------|---------------|------:|---|------|
|wikitext|Yaml   |none  |word_perplexity|71.7454|   |      |
|        |       |none  |byte_perplexity| 2.0227|   |      |
|        |       |none  |bits_per_byte  | 1.0163|   |      |

Divergence is also noted on issue #806 for LLAMA. 

I can't tag issues but this is probably a "good first issue"!

Fixed in #833 ! Thanks for reporting this :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for transformers quantization via bitsandbytes config.  
Root cause: Documentation Deficiency

Are there any plans to support bitsandbytes config for huggingface models? With the transformers library, I can load a model in a large number of different quantization configs by passing in a bitsandbytes config as per the following example  

```
from transformers import BitsAndBytesConfig

nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model_nf4 = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=nf4_config)
```

It would be very useful to evaluate different quantization configs without having to push each version of the model to the hub as a separate revision. I would be happy to contribute to the library if this was of interest!    

I've just realised that you do support this type of functionality if I use HuggingFaceAutoLM directly! 

Hi, may I ask how can you achieve this, I want to evaluate the performance of qlora
Repository: EleutherAI/lm-evaluation-harness
Issue: How to compute metrics per subcategory within one task ?
Root cause: Unimplemented Feature Gap

I have a multi-choice question answering task where each question has a sub-category. My goal is to compute both overall accuracy and per subcategory accuracy (in the style of the image bellow). The number of subcategory is in the range 10-30. 
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/43207538/25cae433-80e9-4a89-8afe-52ed0b92ffc6). 

One way I see is to create a `benchmark`, with a `task` & `dataset` for each of the sub-categories, as well as a union one to get overall accuracy. Yet, this is less user friendly than being able to design directly such fine-grained metrics and less scalable. More generally, I'd be interested to group accuracy by characteristics of questions (eg longer questions, questions on such topic, keyword...). Do you have any advices on how to achieve this or ways I could contribute if it's of interest ?

Many thanks in advance for your help !


@lintangsutawika is working on this and has an example of this done up for MMLU [here](https://github.com/EleutherAI/lm-evaluation-harness/pull/922)
Repository: EleutherAI/lm-evaluation-harness
Issue: We needs features that adds template used for finetuning 

First of all, Thanks for constructing and maintaining this large project for research & industry field. It is very usefull and every-one-needed repo for llm research & engineers.
---
We need new features that adds template used for finetuning for every docs. 

Like ![alpaca-lora](https://github.com/tloen/alpaca-lora/tree/main), which is largely used code template for llm finetuning, there are many LLMs that finetuned with pre-defined template like ![alpaca-lora](https://github.com/tloen/alpaca-lora/tree/main/templates)

As I know, there is no feature now on this repo(if exists, let me know plz. I really need that feature). 

How about add that feature for bit-refactor branch?

Hi, thanks for your appreciation of this project!

We have generally defaulted to intentionally *not* allowing for model-specific prompt templates because it prevents comparisons between models (changing the way the task is posed or how it is prompted can impact the "easiness" of the task and make raw numeric scores impossible to compare apples-to-apples).

However, seeing as we will likely have to add variants of tasks that are done via greedy decoding, for compatibility with recent closed-source APIs, maybe we'll be more open to supporting variants of tasks with templates adapted. Would have to discuss with @lintangsutawika .

We now support this.
Repository: EleutherAI/lm-evaluation-harness
Issue: Quac Dataset
Root cause: Unimplemented Feature Gap

When will you support evaluation on the quac dataset? I found the results of the Llama2 paper difficult to reproduce. Especially regarding how to segment the answer for the base model's F1 score.

are there any solutions? i am confused 

If nobody is already working on it, I can try this feature.

> If nobody is already working on it, I can try this feature.

That would be great!

Hi @glerzing are you working on quac? If not, i can take that.

Yes, I'm on it. It's a bit more complicated than I expected, it will probably take weeks.

Actually, I will not have the opportunity to finish it, sorry. I had other important things to do.
The implementation should be quite similar to the one of CoQA, but to avoid having the same problem as with #1231 , you need a way to make a list of predictions for each document, probably by implementing construct_requests. @Sanchit-404, if you are still motivated, feel free to pick this issue.

Hi, any updates on this ? 

@glerzing if you have a partial implementation or high level plan, please share it, will be helpful for anyone to pick it up 

The Python script is too much of a draft to share. It's not worth much, but here is the [README.md](https://github.com/EleutherAI/lm-evaluation-harness/files/14625457/README.md). I would have liked to have a yaml file like this [like this](https://github.com/EleutherAI/lm-evaluation-harness/files/14628199/default.txt), with the ability to redefine  `construct_requests` inside the yaml file, like with `doc_to_text` or `process_results`. That would be cleaner than having to implement it with a class that inherits from `Task` like in squadv2.
Repository: EleutherAI/lm-evaluation-harness
Issue: Possible Bug?: argmax in sat.py comparison
Root cause: Algorithmic Error

In `tasks/sat.py`, the `process_results()` method contains the line `acc = 1. if np.argmax(results) == gold else 0.`. However, this is comparing the highest value in the loglikelihood list for the options, which is always a number... to `gold = doc["answer_key"]`. Were the result or answer_key formats reworked [later](https://github.com/EleutherAI/lm-evaluation-harness/pull/80)?

`answer_key` format was updated.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
Root cause: Unimplemented Feature Gap

Belebele would be a great addition to evaluate models in many languages.

> Belebele is a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. This dataset enables the evaluation of mono- and multi-lingual models in high-, medium-, and low-resource languages. Each question has four multiple-choice answers and is linked to a short passage from the [FLORES-200](https://github.com/facebookresearch/flores/tree/main/flores200) dataset. The human annotation procedure was carefully curated to create questions that discriminate between different levels of generalizable language comprehension and is reinforced by extensive quality checks. While all questions directly relate to the passage, the English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. Belebele opens up new avenues for evaluating and analyzing the multilingual abilities of language models and NLP systems.

Paper: [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://arxiv.org/abs/2308.16884)
GitHub [Repo for the Belebele dataset, a massively multilingual reading comprehension dataset](https://github.com/facebookresearch/belebele)
Repository: EleutherAI/lm-evaluation-harness
Issue: RACE dataset error？
Root cause: Algorithmic Error

I am not sure that in your `lm_eval/tasks/race.py`
is this function an error?

```def doc_to_text(self, doc):
        text = "Article: " + doc["article"] + "\n\n"
        for problem in doc["problems"][:-1]:
            if problem["question"][-6:] == "  _  .":
                text += (
                    problem["question"][-5:] + self.get_answer_option(problem) + "\n"
                )
            else:
                question = "Question: " + problem["question"] + "\n"
                answer = "Answer: " + self.get_answer_option(problem) + "\n"
                text += question + answer
        text += self.last_problem(doc)["question"]
        return text
```
        
```
text += (problem["question"][-5:] + self.get_answer_option(problem) + "\n")
```    
should be like
```
text += (problem["question"][:-5] + self.get_answer_option(problem) + "\n")
```
to get the whole text

[:-6] maybe is better


I think `text += problem["question"][:-5] ` is correct. Thanks for catching this, can you open a PR with the change? In the PR, please add some examples showing how it improves formatting.

oh yeah, i will do this soon

@RanchiZhao Hey, just wanted to check in and find out if this was on the horizon still.

Hi, I can help to open a PR for this change

> Hi, I can help to open a PR for this change

I want do confirm with @haileyschoelkopf that this is still desirable, but AFAIK that would be very helpful.
Repository: EleutherAI/lm-evaluation-harness
Issue: Support of OpenVINO models from optimum-intel
Root cause: Unimplemented Feature Gap

Hi, thank you for this project. I have the fork of lm-evaluation-harness with supporting of optimum.intel.openvino.OVModelForCausalLM which has similar interface for text generation and inference like the transformers.AutoModelForCausalLM. I run some local test on my fork and it works. Can I create PR for your repository with supporting mentioned model type ?

Yes please! Thank you for this contribution.
Repository: EleutherAI/lm-evaluation-harness
Issue: Should num_fewshot be type list?
Root cause: Unimplemented Feature Gap

Currently, the command line interface supports the flag `--tasks` with type `List[str]`. However `--num_fewshot` is of type `int`. It is desirable to schedule multiple tasks, with varying examples provided. It seems that this is currently not possible on the command line interface.

This is a good idea! We can accept either `int` or `List[int]` and add some checking code to make sure things are handled correctly. You're welcome to open a PR with this feature!
Repository: EleutherAI/lm-evaluation-harness
Issue: openai.error.InvalidRequestError: '$.stop' is invalid.
Root cause: Validation Gap

I'm trying to run evals using OpenAI but getting this Invalid Request Error:

```
> python main.py \
    --model gpt3 \
    --model_args engine=davinci \
    --tasks squad2 \
    --write_out \
    --output_base_path output
...
openai.error.InvalidRequestError: '$.stop' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference.
```

Note this example in the Readme:
```
> python main.py \
    --model gpt3 \
    --model_args engine=davinci \
    --tasks lambada_openai,hellaswag \
    --check_integrity
```

Works OK for me. So maybe the issue is with squad2?
Repository: EleutherAI/lm-evaluation-harness
Issue: `parallelize=True` does not work on some tasks.
Root cause: Algorithmic Error

Python 3.10
lm-evaluation-harness b8d1cef915ff82ec5d5c27a39c6c2b58a9171fac

I have a multi GPU setup which I was trying to use to load parts of a model to benchmark them. During testing I came across this issue where running CoQA didn't parallelize the model while something like PiQA did. 

PiQA:
```
python main.py \
    --model hf-auto \
    --model_args pretrained=meta-llama/Llama-2-7b-hf,parallelize=True \
    --tasks piqa \
    --batch_size auto:2
```
CoQA:
```
!python main.py \
    --model hf-auto \
    --model_args pretrained=meta-llama/Llama-2-7b-hf,parallelize=True \
    --tasks coqa \
    --batch_size auto:2
```
While the PiQA task executes successfully, CoQA errors out due to insufficient GPU memory:
```
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.99 GiB (GPU 0; 14.75 GiB total capacity; 13.45 GiB already allocated; 1.06 GiB free; 13.47 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

It is very weird to me that you might have this issue in a dataset-dependent way. Can you check if it's still a problem on the `big-refactor` branch?

Closing as abandoned.
Repository: EleutherAI/lm-evaluation-harness
Issue: Question about acc_norm
Root cause: Documentation Deficiency

Why use `(PPL / char_len)` instead of `(PPL /  token_len)` to calculate and find the lowest PPL choice as the final result?

refer to [`MultipleChoiceTask.process_results`](https://github.com/EleutherAI/lm-evaluation-harness/blob/00209e10f6e27edf5d766145afaf894079b5fe10/lm_eval/base.py#L758)

a good question, but maybe tokenizer needs time, just approximately

We've done this out of a design decision to keep scoring tokenizer-independent, in the same way as we only report word-level, byte-level, and bits_per_byte for perplexity tasks--the intent is for all `LM` classes implemented to be string-to-string (or string-to-loglikelihood) in interface.
Repository: EleutherAI/lm-evaluation-harness
Issue: Question about automatic detection of max batch size
Root cause: Documentation Deficiency

In base.py we have the following piece of code to determine max batch size 

`@find_executable_batch_size(starting_batch_size=self.max_batch_size)

        def forward_batch(batch_size):

            test_batch = torch.ones((batch_size, max_length), device=self.device).long()

            for _ in range(5):

                _ = F.log_softmax(self._model_call(test_batch), dim=-1).cpu()

            print("Detected batch size = ", batch_size)

            return batch_size
`

I want to know how you came up with the number 5 ? Shouldn't it be 1 ? Maybe 2 just to be on the safe side. The current configs may underestimate the max batch size I think. 


We picked it pretty arbitrarily. The point of this is that GPU load fluctuates sometimes and running several test runs helps increase confidence that the GPU can really handle the batch size in question.