zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Loading mmlu eval not efficient
Root cause: Architectural Constraint

`MMLU` bench has 57 tasks, seems like this code below will load 57 `auxiliary_train `, I think it's not efficient, because all tasks' training set is the same one, maybe you can load these 57 tasks into one `dataset class`.
```python
tasks = ['mmlu']
task_dict = lm_eval.tasks.get_task_dict(tasks)
```
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/31238754/0a59cbe9-e5b0-436e-9636-277d4bc5e5d4)


That's because a common training set is duplicated for each sub-task in the dataset [loading script](https://huggingface.co/datasets/cais/mmlu).

@haileyschoelkopf Quick workaround could be to clone the dataset without the train set. Might be worth it as currently it re-downloads the training set for each `load_dataset` call.

> That's because a common training set is duplicated for each sub-task in the dataset [loading script](https://huggingface.co/datasets/cais/mmlu).
> 
> @haileyschoelkopf Quick workaround could be to clone the dataset without the train set. Might be worth it as currently it re-downloads the training set for each `load_dataset` call.

Thanks for your reply!!! BTW, if I start `mmlu` eval using my `yaml` config, e.g. modify the configs at `lm_eval/tasks/mmlu/default`, how can I get the average score in report?

Think you should get the average score as long as you add the `group` tag in the yaml(s) and initiate the run through that. You can also create a [benchmark](https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/docs/task_guide.md#benchmarks) which is similar to group but might be more convenient.

> Think you should get the average score as long as you add the `group` tag in the yaml(s) and initiate the run through that. You can also create a [benchmark](https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/docs/task_guide.md#benchmarks) which is similar to group but might be more convenient.

thank you for your reply

Created a PR addressing this inefficiency: #953
Repository: EleutherAI/lm-evaluation-harness
Issue: About the number of bbh task
Root cause: Unimplemented Feature Gap

Hi

I fing the official bbh contrains 23 tasks and I just find 20 tasks in this repo. Do you plan to add more tasks in this repo?



Which three are we missing?

> Which three are we missing?

Hi
I noticed a discrepancy in the file names and found that 6 tasks are missing.
boolean_expressions、multistep_arithmetic_two、object_counting、penguins_in_a_table、web_of_lies、word_sorting
https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main/bbh

@StellaAthena also it seems the number of examples doesn't match. Most datasets listed at original BBH repo ([BIG-Bench-Hard](https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main/bbh)) have 250 data points, but, for example, current dyck_languages has 1000 examples. 
One possible reason is the current resources are from [bigbench](https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/dyck_languages/task.json) instead of the [BBH](https://raw.githubusercontent.com/suzgunmirac/BIG-Bench-Hard/main/bbh/dyck_languages.json).

Hi! In the `big-refactor` branch (soon to be next major version release) we support BBH as implemented in the BBH paper, such as 3-shot CoT with the 250 subselected examples and matching their prompt: https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor/lm_eval/tasks/bbh/flan_cot_fewshot
Repository: EleutherAI/lm-evaluation-harness
Issue: request pad not work as intended and potential solution
Root cause: Algorithmic Error

https://github.com/EleutherAI/lm-evaluation-harness/blob/12fe236de5871be01121a63e238aae46402f9c97/lm_eval/evaluator.py#L282C57-L282C57

original:
```python
padding_requests[task.OUTPUT_TYPE] += numpad
```
change to
```python
padding_requests[reqtype] += numpad
```

@wanglamao2 Thank you for opening this issue! Can you elaborate on why this is desired, ideally with examples using both the current and your newly recommended format?

@StellaAthena Hello, I integrated your fabulous evaluation framework into my HF based training scripts so that it can work with loaded models and worked like charm. However, since I use Deepspeed zero-3 mostly, evaluation gets stuck at the end when I evaluate on tasks like boolq and mmlu. 
The problem is caused by unequal number of batches across processes.
I CI a little bit and found the padding part didn't work as intended because the dict is accessed with wrong key. 

The following code is quoted from https://github.com/EleutherAI/lm-evaluation-harness/blob/12fe236de5871be01121a63e238aae46402f9c97/lm_eval/evaluator.py#L267C3-L295C21
```python
        reqtype = (  #  reqtype of "multiple_choice" is changed to "loglikelihood"
            "loglikelihood"
            if task.OUTPUT_TYPE == "multiple_choice"
            else task.OUTPUT_TYPE
        )  # TODO: this is hacky, fix in task.py
        requests[reqtype].extend(task.instances)

        if lm.world_size > 1:
            instances_rnk = torch.tensor(len(task._instances), device=lm.device)
            gathered_item = (
                lm.accelerator.gather(instances_rnk).cpu().detach().numpy().tolist()
            )

            # compute number of pseudobatches to pad with (FSDP/DDP require even batches among ranks)
            numpad = max(gathered_item) - gathered_item[lm.rank]
            padding_requests[task.OUTPUT_TYPE] += numpad # dict key is assigned as "multiple_choice"

    ### Run LM on inputs, get all outputs ###
    # execute each type of request
    for reqtype, reqs in requests.items():
       # deleted for clarity


        if (lm.world_size > 1) and (padding_requests[reqtype] > 0): # dict is acessed with "loglikelihood"
            for _ in range(padding_requests[reqtype]):                   # which doesn't exist in key set
                cloned_reqs.extend([req] * req.repeats)
```
After the following modification, everything is fine.

> 
> original:
> 
> ```python
> padding_requests[task.OUTPUT_TYPE] += numpad
> ```
> 
> change to
> 
> ```python
> padding_requests[reqtype] += numpad
> ```



I also would love to contribute my adaptation for loaded model. 
Is it interesting for you?

Yes, we would love a PR with this contribution.
Repository: EleutherAI/lm-evaluation-harness
Issue: chatglm2 acc=0 on lambada_openai dataset, is it correct?
Root cause: Interface Contract Mismatch

I also validate chatglm and chatglm3, they are works, could you have fix the root cause?
https://huggingface.co/THUDM/chatglm2-6b/discussions/97
python main.py --model hf-causal --model_args pretrained=THUDM/chatglm2-6b,trust_remote_code=True --tasks lambada_openai --limit 10 --batch_size 1 --no_cache

When I run this I get `AttributeError: property 'pad_token_id' of 'ChatGLMTokenizer' object has no setter`. You're seeing it run though?

as the code show, https://huggingface.co/THUDM/chatglm2-6b/blob/main/tokenization_chatglm.py#L91
yes, correct, the `pad_token`  or `pad_token_id` is `property` and has `no setter`.
I comment `tokenizer.pad_token = tokenizer.eos_token`, I can get the acc about chatglm&chatglm3, but acc=0 for chatglm2.
Could you give me some help?

ChatGLM-6B and ChatGLM3-6b both seem to work fine. I'm still unsure as to what's going wrong on ChatGLM2 and would very much like to figure out how to fix it.
Repository: EleutherAI/lm-evaluation-harness
Issue: globally normalized models
Root cause: Unimplemented Feature Gap

Globally normalized models score sequences using the sum of unnormalized logits, as opposed to locally normalized ones which take a log_softmax at each token position before summing to compute the sequence score. They are strictly more expressive then locally normalized models (i.e. they can express a superset of probability distributions over sequences). It would take a one-line change in lm-evaluation-harness/lm_eval/base.py to allow evaluation of globally normalized models:
```
            # multi_logits = F.log_softmax(                                                                                            
            #     self._model_call(batched_inps), dim=-1                                                                               
            # ).cpu()  # [batch, padding_length, vocab]                                                                                
            multi_logits = self._model_call(batched_inps).cpu()
```
I just wanted to start a discussion to see if there is any interest and if there is a way we could make this an option for the evaluator. I can submit a pull request. (And I have some globally normalized models I'd like to share on hf-leaderboard ;).

Sure, I don't see why we shouldn't include this.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] Allow for some tasks to force zero-shot
Root cause: Unimplemented Feature Gap

some tasks either define a custom few-shot prompt already in `doc_to_text` (and thus should not be passed additional few-shot examples), or are ill-defined in the fewshot setting (for example, BLIMP.) We should allow `num_fewshot: -1` to be set in a yaml and have these tasks raise a warning + ignore the setting, if `--num_fewshot X` is passed for `X > 0` via the CLI.

@haileyschoelkopf I can take a look at this if you haven't
Repository: EleutherAI/lm-evaluation-harness
Issue: can you support trt-llm backends?
Root cause: Unimplemented Feature Gap


You're welcome to open a PR with support if you like! It looks like it's still in alpha so I don't think we're going to prioritize it, but if it would be useful for you and others we would be down.

Closing as not planned. If someone wants to add it that's great, but we probably won't get around to it.
Repository: EleutherAI/lm-evaluation-harness
Issue: fine-tuning LLaMA on common sense  reasoning datasets such as  PIQA  and HellaSwag

Thanks for the great effort you made in this work. 
I know that LLM such as LLaMA is typically being evaluated in common sense reasoning datasets to measure its zero-shot capability. I also believe that fine tuning LLM on downstream tasks can improve its performance.  
My primary objective is to develop an algorithm that optimizes LLaMA for specific data distributions, which could include tasks or diverse datasets. Consequently, I plan to fine-tune the LLaMA model on various datasets to assess its performance. I have a couple of questions that I hope you can assist me with:

Question 1: Do you support or can point to any work that fine tunes the LLaMA model on common sense reasoning datasets such as PIQA and HellaSwag to ensure reproducibility ?

Question 2:  In the event that the implementation of common sense reasoning datasets is unavailable or challenging, do you have any recommendations for alternative datasets that are readily accessible and suitable for the fine-tuning process?. 

Thanks so much 

Closing as outside the scope of this library.
Repository: EleutherAI/lm-evaluation-harness
Issue: Python 3.8 support on the main branch
Root cause: Unimplemented Feature Gap

Hello everyone! I hope everything is going well with you.

Would there be much refactor to support Python 3.8 on the `main` branch?

I noticed the support was added to the `big_refactor`, however we still have some legacy code that uses the `main`, and this is preventing some uses cases (NVIDIA-based images <= 23.04).

Thanks for your attention and best regards,
Gustavo.

I think it would require very little. Might be some edge cases but the main functionality should work fine imo. 

Sorry for the late reply, but thanks for the inputs! We will work on that.
Repository: EleutherAI/lm-evaluation-harness
Issue: Build failure?
Root cause: Validation Gap

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/8592144/116aa631-fada-46f0-b6c8-2a5d5cc851c4)
Does anyone confirm it is still the thing?


This is because of some vestigial stuff we need to clean up before our next release and can be safely ignored.
Repository: EleutherAI/lm-evaluation-harness
Issue: Average score metric isn't normalized whatsoever
Root cause: Unimplemented Feature Gap

As of writing, the maximum scores for each benchmark as measured by the open LLMs are:

- Max ARC Score: 76.79
- Max HellaSwag Score: 88.89
- Max MMLU Score: 78.66
- Max TruthfulQA Score: 65.81
- Max Winogrande Score: 86.9
- Max GSM8K Score: 60.27
- Max DROP Score: 64.2

The disparities between the maximum possible scores for each benchmark means that the average score of the model is heavily skewed towards the benchmarks that have higher 'ends'.

A potential improvement could involve taking the theoretical maximum of each respective benchmark, and then using that as the normalization factor. Alternatively, the highest observed score of each benchmark could be used if the theoretical maximum is too far from the maximum observed real world value for certain benchmarks.

For example, if we take the alternative approach and use the highest observed score for each benchmark (as currently observed) on Yi-34b, and then average those normalized scores, there's a value of:
0.9180

This would be significantly more interpretable and understandable to use as a metric for averaging different benchmarks.



Hi, I believe this should be opened on the HF Open LLM Leaderboard--as the benchmarking and score averaging is handled by them and is done by HF independently of us and this repo.
Repository: EleutherAI/lm-evaluation-harness
Issue: llama 2 70b gptq use too much cpu memory 
Root cause: Configuration Error

I want to test https://huggingface.co/TheBloke/Llama-2-70B-Chat-GPTQ. I can run the following code in an a100 40GB gpu:
```
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_name_or_path = "TheBloke/Llama-2-70B-chat-GPTQ"
# To use a different branch, change revision
# For example: revision="main"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                             device_map="auto",
                                             trust_remote_code=False,
                                             revision="main")

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

prompt = "Tell me about AI"
prompt_template=f'''
 

You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.


{prompt}


'''

print("\n\n*** Generate:")

input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
print(tokenizer.decode(output[0]))

```
But when I use the following command to test:
```
python main.py     --model hf-causal     --model_args pretrained=/nas/lili/models_hf/Llama-2-70B-chat-GPTQ/    --tasks hellaswag     --device cuda:0
```

It don't use gpu, but it use cpu and cause OOM of cpu.

I am also observing the same issue. My CPU is being used rather than GPU

python main.py --model hf-causal-experimental --model_args pretrained=openlm-research/open_llama_3b,dtype=float16 --tasks truthfulqa_gen --batch_size 4 --output_path "out/results.json" --device cuda:0



Does this issue persist on the `big-refactor` branch?

Closing as stale.
Repository: EleutherAI/lm-evaluation-harness
Issue: it seems like a bug in winogrande.py
Root cause: Documentation Deficiency

Example of winograde dataset:
<img width="1083" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/10429114/cbd20077-ea42-4df5-83ee-a44f2c609c61">

code to calculate target loglikelyhood given ctx:
<img width="418" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/10429114/99a459ca-5b78-48bc-8fbb-65cea048b5b5">

The "target" does not contain "answer" replaced at "_", but only the postfix of sentence:
<img width="460" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/10429114/9676b1b3-4e75-41b7-8c6f-6c2878130823">

In another word, current target: "always got the easier cases."
Expected target: "Maria always got the easier cases."

Am I right?


As discussed at https://github.com/EleutherAI/lm-evaluation-harness/issues/978#issuecomment-1807433058, this is the intended feature. While it diverges from the original Winogrande implementation, it follows common usage for causal decoder-only models (in particular, was used by GPT-3). I also talk a bit about our philosophy for deciding what to support as the default option for evals in that thread.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to interpret generated results for truthful_qa test
Root cause: Documentation Deficiency

Hello, I executed truthful_qa benchmarks on a few models.

However, the documentation does not explain how to interpret results.

Below is a sample result I achieved in one of my benchmarks

|     Task     |Version|  Metric   | Value |   |Stderr|
|--------------|------:|-----------|------:|---|-----:|
|truthfulqa_gen|      1|bleurt_max |-0.5266|±  |0.0171|
|              |       |bleurt_acc | 0.4590|±  |0.0174|
|              |       |bleurt_diff|-0.0073|±  |0.0229|
|              |       |bleu_max   |30.2628|±  |0.7998|
|              |       |bleu_acc   | 0.4382|±  |0.0174|
|              |       |bleu_diff  | 0.5205|±  |0.9126|
|              |       |rouge1_max |59.4502|±  |0.8515|
|              |       |rouge1_acc | 0.4370|±  |0.0174|
|              |       |rouge1_diff| 1.5690|±  |1.1883|
|              |       |rouge2_max |44.9280|±  |1.0439|
|              |       |rouge2_acc | 0.4051|±  |0.0172|
|              |       |rouge2_diff| 0.9125|±  |1.3574|
|              |       |rougeL_max |56.0939|±  |0.8891|
|              |       |rougeL_acc | 0.4186|±  |0.0173|
|              |       |rougeL_diff| 0.9057|±  |1.2061|

Which value do I use as my truthful_qa benchmark score on a hugging face repo?
I'll be glad if anyone could help me out with this


+1

+1

I am also confused with the metrics here, just did a quick search, the people in the following discussion states that llama2 used mc2 in their paper : 
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/95
Also, see the reference results here : 
https://huggingface.co/meta-llama/Llama-2-70b-hf
Repository: EleutherAI/lm-evaluation-harness
Issue: Model is not a local folder and is not a valid identifier.
Root cause: Documentation Deficiency

I have download model in the local path and It is loading perfectly.

import torch

model_identifier = "NousResearch/Llama-2-7b-chat-hf"
model =  AutoModelForCausalLM.from_pretrained(model_identifier)
tokenizer = AutoTokenizer.from_pretrained(model_identifier)

Entered into below error when i ran the cmd for evaluation

**cmd:**

!python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=model,trust_remote_code=True,load_in_4bit=True \
    --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
    --device cuda:0

**Error:**
OSError: model is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`


Hi, you would have to run 

```
python main.py
--model hf-causal-experimental
--model_args pretrained=./NousResearch/Llama-2-7b-chat-hf,trust_remote_code=True,load_in_4bit=True
--tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq
--device cuda:0
```

You cannot pass `pretrained=model` to the command, but must instead replace `model` in the above with the actual path to the model you have downloaded.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: cmmmu
Root cause: Documentation Deficiency

when using '--tasks cmmmu' I got some wrongs.

assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names.

Hi, when I tried `lmms_eval --tasks list`, I can see cmmmu on the list and this means cmmmu can be loaded successfully.

<img width="453" alt="image" src="https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/92624596/bc77bfc7-33f0-4a91-8db5-09e8e578487c">

If you are encountering this issue, maybe you can try `lmms_eval --tasks list --verbosity DEBUG` and see what error causing  the tasks can not be loaded.

@kcz358 Thanks for your reply.
Another question: My server is unable to access the HF dataset directly. How can I use the local dataset downloaded from 'lmms-lab/CMMMU' to evaluate the model?

@phellonchen 

I haven't tried this but based on my understanding of the code and huggingface `load_dataset` api. 

You may want to change the `dataset_path` in the [_default_template_cmmmu_yaml](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/tasks/cmmmu/_default_template_cmmmu_yaml) to your local dataset path. 

Our code loads huggingface dataset using the `load_dataset` method. For each tasks, the code logic is like this `dataset = load_dataset(<dataset_path>, name = <dataset_name> (if there is one else None for this), split = <split>)`. Thus, as long as you clone our dataset repo and can successfully load the dataset through `load_dataset` method, this should be able to work.