zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug in Leaderboard IFEval Code
Root cause: Validation Gap

https://github.com/EleutherAI/lm-evaluation-harness/blob/8138fd52437dcd8c76ac87bdc9d684840e794c42/lm_eval/tasks/leaderboard/ifeval/instructions.py#L1384

the updated IFEval dataset (https://www.oxen.ai/wis-k/instruction-following-eval/file/main/instruction-following-eval_train.parquet) now includes letters like "!", "#", which are considered true under the conditions mentioned. As a result, the letters may be changed to random characters. 

ccing @NathanHB @clefourrier for discretion over changing the official leaderboard IFEval task definition!

#2218 from @lewtun updated the dataset to `google/ifeval` for our `ifeval` non-leaderboard task--perhaps that should be carried over to the leaderboard variant regardless of whether that fixes this issue as well?
Repository: EleutherAI/lm-evaluation-harness
Issue: Cannot evaluate on unitxt-related tasks such as unfair-tos, ledgar...
Root cause: Configuration Error

I want to evaluate on law-related tasks to examine the effect of my LAW LLM, and my commands are as follows:

cd lm-evaluation-harness
pip install -e.
pip install lm_eval[unitxt]
CUDA_VISIBLE_DEVICES=0,1,2,3 bash eval_law_tasks.sh ../model/Qwen2-7B law-tasks-Qwen2-7B

I got the following reported errors:
ValueError: The repository for unitxt/data contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/unitxt/data.

Any ideas to solve this issue? Many thanks!

I think that you need to add the param `trust_remote_code=True` when you load the dataset.

I use 
`lm_eval --model hf  --model_args pretrained=modelpath,trust_remote_code=True` for a model, but I'm not sure if it's implemented for datasets, or how to pass it.

otherwise you could always download and save the dataset first and then evaluate on it from a local file too.

Hi! You can pass `--trust_remote_code` arg, for e.g.:
`lm_eval --trust_remote_code --model hf ...`
and that should generally work. Alternatively you can all use the `HF_DATASETS_TRUST_REMOTE_CODE=1` environment variable.

closing as `--trust_remote_code` should resolve this, but please reopen this issue if you continue facing problems!

Hi! I use the recent version of lm_eval, but get the following errors:

Repo card metadata block was not found. Setting CardData to empty

Any ideas to solve this issue? Many thanks!

> Hi! I use the recent version of lm_eval, but get the following errors:
> 
> Repo card metadata block was not found. Setting CardData to empty
> 
> Any ideas to solve this issue? Many thanks!

H! I think you can ignore that. It's a HF warning, letting you know there is no readme for the dataset iirc
Repository: EleutherAI/lm-evaluation-harness
Issue: fatal: not a git repository (or any parent up to mount point /kaggle)
Root cause: Validation Gap

Thanks for your valuable contribution to this project.

**There is the issue**:
I was running the command line on Kaggle. And I got the error, logs see below.
```
!lm_eval --model hf \
    --model_args pretrained=${MODEL_NAME} \
    --tasks mmlu \
    --device cuda:0 \
    --num_fewshot 0 \
    --batch_size 4 \
    --output_path eval \
    --use_cache True\
    --limit 10 \
    --hf_hub_log_args hub_results_org=aisuko,hub_repo_name=eval-smolLM-135M,push_results_to_hub=True,push_samples_to_hub=True,public_repo=False
```

The logs
```
100%|██████████████████████████████████████████| 10/10 [00:00<00:00, 458.18it/s]
Checking cached requests: 100%|███████████| 2280/2280 [00:00<00:00, 4466.65it/s]
Running loglikelihood requests:   0%|                  | 0/2280 [00:00<?, ?it/s]We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
Running loglikelihood requests: 100%|██████| 2280/2280 [00:09<00:00, 232.61it/s]
fatal: not a git repository (or any parent up to mount point /kaggle)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Traceback (most recent call last):
  File "/opt/conda/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/opt/conda/lib/python3.10/site-packages/lm_eval/__main__.py", line 438, in cli_evaluate
    evaluation_tracker.recreate_metadata_card()
  File "/opt/conda/lib/python3.10/site-packages/lm_eval/loggers/evaluation_tracker.py", line 470, in recreate_metadata_card
    last_results_file = [
IndexError: list index out of range
```

Hi! you also need to add `--log_samples` if you are adding `push_samples_to_hub` to the HF log args.

HI @baberabb thanks for your quickly replying. After I added `--log_samples` the issue still occurred. Please check the notebook below.

https://www.kaggle.com/code/aisuko/evaluating-language-models/notebook

Thank you. Thats helpful. I'll have a closer look later but if you install the harness through (first cell):

`!git clone https://github.com/EleutherAI/lm-evaluation-harness`
`%cd lm-evaluation-harness`
`!pip install -e .`

I think it should work

> Thank you. Thats helpful. I'll have a closer look later but if you install the harness through (first cell):
> 
> `!git clone https://github.com/EleutherAI/lm-evaluation-harness` `%cd lm-evaluation-harness` `!pip install -e .`
> 
> I think it should work

Thank you. It works now. Feel free to close this issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: Tasks of type `python_task` are not listed in `lm-eval --tasks list`
Root cause: Validation Gap

## Problem Description
When calling `TaskManager._initialize_tasks` I noticed that tasks which are managed by their own python class enter the `_task_index` with `type="python_task"`: 

https://github.com/EleutherAI/lm-evaluation-harness/blob/928e8bb6f50d1e93ef5d0bcaa81f8c5fd9a6f4d8/lm_eval/tasks/__init__.py#L452-L457

As a consequence, when printing the task list `lm-eval --tasks list` these are effectively omitted from any constructed table, which only consider types `group, task` or `tag`

https://github.com/EleutherAI/lm-evaluation-harness/blob/928e8bb6f50d1e93ef5d0bcaa81f8c5fd9a6f4d8/lm_eval/tasks/__init__.py#L39-L47

## How to reproduce
Simple way to notice this, take a task that is managed by a Python class, such as `fda` managed by `task.FDA`, I can run the following:
```bash
lm_eval --model hf  \
    --model_args pretrained=EleutherAI/pythia-160m,dtype="float" \
    --tasks fda \
    --device mps \
    --limit 10
```

Allthough, when I try to see `fda` in the list of tasks, e.g. `lm-eval --tasks list | grep fda` this doesn't appear in the table.

I would be happy to work on this but I'd like to hear a maintainer's opinion regarding whether these tasks should appear in tables and be treated the same way `task`-type tasks are treated, including their inclusion in the tag handling.

Hi! This check

```
 self._all_subtasks = sorted( 
     [x for x in self._all_tasks if self._task_index[x]["type"] == "task"] 
 ) 
```
should indeed be
```
 self._all_subtasks = sorted( 
     [x for x in self._all_tasks if self._task_index[x]["type"] in ["task", "python_task"]] 
 ) 
```
.

If you have the bandwidth to PR and test a fix, we'd be grateful, but will fix this ourselves if not!
Repository: EleutherAI/lm-evaluation-harness
Issue: Metrics that require probability scores (y_scores)
Root cause: Documentation Deficiency

Hi, I want to use the `PR-AUC`  (or `ROC-AUC`) metrics for a few-shot classification problem where the test data is imbalanced. 
Therefore, I need the positive (yes) **probability** to calculate this via `scikit-learn`, not the `y_pred`
How can I do that with lm-eval? 


Hi! you can get the model outputs as well as the metric score for each each example using `--log_samples` (use it with `--output_path`).
Repository: EleutherAI/lm-evaluation-harness
Issue: May be parse LAST numbers in GSM8K "flexible-extract" filter?
Root cause: Documentation Deficiency

In GSM8K config: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/gsm8k/gsm8k.yaml
```yaml
filter_list:
  - name: "strict-match"
    filter:
      - function: "regex"
        regex_pattern: "#### (\\-?[0-9\\.\\,]+)"
      - function: "take_first"
  - name: "flexible-extract"
    filter:
      - function: "regex"
        group_select: -1
        regex_pattern: "(-?[$0-9.,]{2,})|(-?[0-9]+)"
      - function: "take_first"  # i think problem in this line
```

On the flexible-extract filter, we select the first number from the model prediction. However, most models provide a similar answer:
```
To find the total cost, we need to calculate the cost of each type of pastry and then sum them up.

First, let's calculate the cost of the donuts:
- Toula bought 3 dozen donuts.
- The cost per dozen is $68.
- The total cost for the donuts is 3 * $68 = $204.

Next, let's calculate the cost of the mini cupcakes:
- Toula bought 2 dozen mini cupcakes.
- The cost per dozen is $80.
- The total cost for the mini cupcakes is 2 * $80 = $160.

Finally, let's calculate the cost of the mini cheesecakes:
- Toula bought 6 dozen mini cheesecakes.
- The cost per dozen is $55.
- The total cost for the mini cheesecakes is 6 * $55 = $330.

Now, let's sum up all the costs:
- Total cost = $204 (donuts) + $160 (mini cupcakes) + $330 (mini cheesecakes) = $694.

Therefore, the total cost is $694.
```

And the answer from the model (without any formatting from a few shots) is the LAST number.

Hi! We actually select the last number from the regex hence the `group_select: -1`. The [take_first](https://github.com/EleutherAI/lm-evaluation-harness/blob/7a1614eb90d29b2983ffa027a7974b7ef53fba19/lm_eval/filters/selection.py#L21) selects the first model response in case of multiple `n`.

Hello, I apologize for not having fully studied the code of this filter. You're right, thank you for the answer.
I close this issue
Repository: EleutherAI/lm-evaluation-harness
Issue: Multi-node MMLU support ? 
Root cause: Unimplemented Feature Gap

Is there support for multi-node MMLU evaluation ? 

I'd be ok with trying experimental stuff, etc. 

Hi! We don't support multi-node natively but you should be able to use it with an [API](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/API_guide.md).

Thanks @baberabb ! Is there any plan to support multi-node ?
Repository: EleutherAI/lm-evaluation-harness
Issue: External API - same results different models
Root cause: Interface Contract Mismatch

Hi, I noticed that when I use the following I get the same results even with different models of different families.
```
    lm = HFLM(pretrained=model_path, peft=model_path, trust_remote_code=True, batch_size=4)
    task_manager = lm_eval.tasks.TaskManager()
    res=lm_eval.simple_evaluate(
        model=lm,
        tasks=task,
        num_fewshot=n_fewshot,
        use_cache=cache_path,
        task_manager=task_manager
        )
```
where, `model_path` is the huggingface model ID, `model_path` is the local `peft` path.

whereas, I get better i.e. different results when I simply pass the `hf` as the model:
```
    task_manager = lm_eval.tasks.TaskManager()
    res=lm_eval.simple_evaluate(
        model="hf",
        model_args = f"pretrained={model_id},tokenizer={model_id},peft={model_path}",
        tasks=task,
        num_fewshot=n_fewshot,
        use_cache=cache_path,
        task_manager=task_manager
        )
```

Best,

Hi! Which model were you testing and how much was the difference?

closing now but feel free to open
Repository: EleutherAI/lm-evaluation-harness
Issue: What's going on with swde or squadv2 tasks ? 
Root cause: Validation Gap

I am using the 0.4.4 version and able to evaluate the model on these tasks:

` wikitext,lambada_openai,piqa,hellaswag,winogrande,arc_easy,arc_challenge,mmlu`

But I am not able to evaluate the model on swde or squadv2 tasks. Here's the error I get:

`TypeError: The current model class (HF_GPTForCausalLM) is not compatible with .generate(), as it doesn't have a language model head. Please use one of the following classes instead: {'HF_GPTForCausalLM'}`

This is really confusing as the model does have the same lm head as suggested. 

Why is this happening and is there a solution ? 

Hi!

Does the task `gsm8k` work with your model? `swde` and `squadv2` tasks involve `generate_until`-type responses (aka generating text from a model autoregressively), whereas the others you've tested on only require a forward pass and extracting the resulting logits.

Could you share more about your `HF_GPTForCausalLM` ? In particular especially, if you load your model in a fresh script and try to call `model.generate()`, does that throw this same error? If it does error out, then the culprit is likely your modeling code--if it doesn't error and can generate text via this `.generate()` method, then it may be a problem with our library.



Thanks @haileyschoelkopf !
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue with openai completions API - related to logprobs
Root cause: Algorithmic Error

Hello, I believe there is a bug in your code on this line: 

https://github.com/EleutherAI/lm-evaluation-harness/blob/543617fef9ba885e87f8db8930fbbff1d4e2ca49/lm_eval/models/openai_completions.py#L72

if I am understanding correctly, this should actually return the tokens, which would be accessed by this instead:

tokens = choice["logprobs"]["tokens"][ctxlen:-1]

Here is an example of a return from an openai request:

{
  "id": "***",
  "object": "text_completion",
  "created": 1725923867,
  "model": "gpt-3.5-turbo-instruct",
  "choices": [
    {
      "text": "\n\n",
      "index": 0,
      "logprobs": {
        "tokens": [
          "\n\n"
        ],
        "token_logprobs": [
          -0.8673025
        ],
        "top_logprobs": [
          {
            "\n\n": -0.8673025
          }
        ],
        "text_offset": [
          18
        ]
      },
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 1,
    "total_tokens": 6
  }
}

Hi! Thanks for catching that. Would appreciate a PR if you have the bandwidth!

@baberabb sure will send a PR by tomorrow.

thanks for calling this out!
Repository: EleutherAI/lm-evaluation-harness
Issue: A little typing issue
Root cause: Validation Gap

Hi! I think I have spotted a typing issue in the `simple_evaluate` function (see the relevant code [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L73)).
According to the docs, the different seed arguments can be set to `None` to  indicate the seed should not be set. However the type hint for these arguments is currently `int` instead of `Optional[int]` (or `int | None`).
Repository: EleutherAI/lm-evaluation-harness
Issue: Couldn't parse .yaml file for configuration
Root cause: Unimplemented Feature Gap

**Resource**
Hugging Face Repo - "benjamin-paine/stable-diffusion-v1-5"
**Background**
https://huggingface.co/benjamin-paine/stable-diffusion-v1-5/blob/main/v1-inference.yaml
**inference.yaml** has been used for the configuration which call the other config.json file which are present in the subfolder.
**Issue**
Couldn't find a way to parse .yaml for config in lm-evalution-harness

@lintangsutawika @haileyschoelkopf
Repository: EleutherAI/lm-evaluation-harness
Issue: Infer time by use library's external api is much longer than script
Root cause: Configuration Error

I tried to test llama2-7b on wikitext by script that follows your docs guide. Here is my code:
```
lm_eval --model hf \
    --model_args pretrained=/data/users/wangh/models/Llama-2/llama-2-7b-hf \
    --tasks wikitext \
    --batch_size 32 \
    --output_path ./eval_out/llama2_7b \
```
and infer time is just  cost 34 sec:

```
2024-09-11:15:08:55,718 INFO     [evaluator.py:465] Running loglikelihood_rolling requests
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 62/62 [00:34<00:00,  1.82it/s]

```
Then I tested same model and dataset following your library's external api doc. Here is my code:

```
from transformers import LlamaTokenizer, LlamaForCausalLM
import torch
import torch.nn.functional as F
from lm_eval.models.huggingface import HFLM
from lm_eval.utils import make_table
from lm_eval import evaluator
import lm_eval
import numpy as np
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"
seed = 619
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

@torch.no_grad()
def main():
    model_path = "/data/users/wangh/models/Llama-2/llama-2-7b-hf"
    tokenizer=LlamaTokenizer.from_pretrained(model_path,use_fast_tokenizer=True)
    model=LlamaForCausalLM.from_pretrained(model_path,device_map="cuda")
    model.eval()
    model.to("cuda")
    hf_model = HFLM(
        pretrained=model, 
        tokenizer=tokenizer,
        batch_size=32,
        device="cuda"
        )
  
    results = evaluator.simple_evaluate(
        hf_model,
        tasks=("wikitext"), 
        device="cuda",
    )
 
    print(make_table(results))
 

if __name__=="__main__":
    main()
```
and this time infer time is 5.03， much longer than before
```
2024-09-11:15:42:20,205 INFO     [evaluator.py:465] Running loglikelihood_rolling requests
100%|██████████████████████████████████████████████████████| 62/62 [05:03<00:00,  4.90s/it]
```
Is there any setting wrong? How to fix it?

Same issue, it seems stuck for a long time at the beginning
Repository: EleutherAI/lm-evaluation-harness
Issue: Using multi-GPU with accelerate is not working
Root cause: Environment Incompatibility

```
accelerate launch -m lm_eval \
    --model hf \
    --model_args pretrained=mistralai/Mistral-7B-v0.3 \
    --tasks winogrande \
    --num_fewshot 0 \
    --batch_size 16 \ 
```
    
I tried using accelerate to speedup evaluation latency for efficient test, but it not works.
I thought my script was wrong, so I tried adding 'parallelize=True', 'num_processes=9' or using other models,
but it stops running after building contexts as below.
I couldn't find any related issue, so I think I'm doing something wrong.
Can anybody help me?

![image](https://github.com/user-attachments/assets/0a9c2561-e0a1-4826-8c63-25c900e6591e)
![image](https://github.com/user-attachments/assets/868d9f00-a841-49c4-abf2-b89d960c8447)


Your code in the block works for me. It will also output something to inform that I'm using two processes because my machine has two GPUs. Not sure how you add num_processes, but you can try 

```
accelerate launch --num_processes 2 -m lm_eval \
    --model hf \
    --model_args pretrained=mistralai/Mistral-7B-v0.3 \
    --tasks winogrande \
    --num_fewshot 0 \
    --batch_size 16 
```

Or explicitly specify the GPU by
`export CUDA_VISIBLE_DEVICES=0,1`

Oh, thank you for the kind and quick response. It only works when I set num_processes to 2. I'm not sure why, but it seems to be an issue with the environment I'm using. On another server, I can set num_processes to 8 and it utilizes all 8 GPUs just fine.

@commmet-ahn, I have the same problem. Have you found what the issue is?
Repository: EleutherAI/lm-evaluation-harness
Issue: Worse evaluation performance with PEFT adaptors
Root cause: Configuration Error

Hello, thank you for providing this excellent model and repository. I encountered an issue while conducting my experiments with your codebase, and I’d appreciate your insights.

In my experiments, I found that the evaluation results using the original (vanilla) LLama model are better than those of the model fine-tuned with PEFT adapters. My understanding is that PEFT should enhance or at least maintain the performance of the base model. However, in my case, the vanilla model outperforms the PEFT-tuned model.

The command I used for evaluating the original (vanilla) model is as follows:
`lm_eval --model hf --model_args pretrained="../llama-7-hf",trust_remote_code=True,add_bos_token=True --batch_size 1 --seed 42 --device cuda:0 --output_path lm_eval_outputs/llama-7b-hf/ --tasks boolq,piqa,social_iqa,hellaswag,winogrande,arc_easy,arc_challenge,openbookqa --trust_remote_code`

The command I used for evaluating with peft adapter is as follows:
`lm_eval --model hf --model_args pretrained="../llama-7b-hf",trust_remote_code=True,add_bos_token=True,peft="{path_of_peft_adapter}" --batch_size 1 --seed 42 --device cuda:0 --output_path lm_eval_outputs/llama-7b-hf-prompt/ --tasks boolq,piqa,social_iqa,hellaswag,winogrande,arc_easy,arc_challenge,openbookqa --trust_remote_code`

Peft is trained with reference to https://github.com/AGI-Edgerunners/LLM-Adapters. The evaluation performance utilizing the code provided by LLM-Adapters does not have this problem, but the accuracy is much lower than lm_eval.

Environment Information:
huggingface-hub     0.24.6
lm_eval                     0.4.3
peft                           0.12.0
python                      3.9.16
torch                         2.0.0
transformers            4.44.2




You can try to add `--log_samples --verbosity DEBUG` in both commandlines to get logs and debug info to help you find important differences in model replies.
You may also want to use the latest 0.4.4 release of lm_eval and some newer version of torch.

experienced same also, after several hours and days of finetunning with peft, accuracy does not improve. Is it an issue with lm-evaluation-harness or??
Repository: EleutherAI/lm-evaluation-harness
Issue: the log is end,the gpu is not calculate,but is storing,the result is not getting,is it normal?
Root cause: Documentation Deficiency

![image](https://github.com/user-attachments/assets/fc7317ca-a868-4d2b-b867-08465b7232d8)
![image](https://github.com/user-attachments/assets/292fc646-971b-4eef-93dc-0e3b3de7f3b2)
 how to resolve it , thanks a lot.

Hi! Can you provide the full command you used?
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the Natural Language Inference (NLI) evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> Natural Language Inference (NLI) [Fyo00] concerns the ability to understand the relationship between two sentences. In practice, this task is usually structured as a two or three class classification problem where the model classifies whether the second sentence logically follows from the first, contradicts the first sentence, or is possibly true (neutral).
> SuperGLUE includes an NLI dataset, RTE, which evaluates the binary version of the task. On RTE, only the largest version of GPT-3 performs convincingly better than random (56%) in any evaluation setting, but in a few-shot setting GPT-3 performs similarly to a single-task fine-tuned BERT Large. We also evaluate on the recently introduced Adversarial Natural Language Inference (ANLI) dataset [NWD+19]. ANLI is a difficult dataset employing a series of adversarially mined natural language inference questions in three rounds (R1, R2, and R3). Similar to RTE, all of our models smaller than GPT-3 perform at almost exactly random chance on ANLI, even in the few-shot setting (∼ 33%), whereas GPT-3 itself shows signs of life on Round 3. Results for ANLI R3 are highlighted in Figure 3.9 and full results for all rounds can be found in Appendix H. These results on both RTE and ANLI suggest that NLI is still a very difficult task for language models and they are only just beginning to show signs of progress.

- [x] Data processing code implemented
- [ ] Evaluation implemented

This should be modeled after the `BoolQ` task in `lm_eval/tasks/suerglue.py`

I'm pretty sure NLI is a category rather than a single evaluation. Closing for now.