zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Configuring Azure OPENAI
Root cause: Documentation Deficiency

Hi,
I want to evaluate gpt4o and other gpt related models on SWE bench dataset. Can someone guide by giving what steps to follow?
I have azure openai access key but it is not getting configured. So, I am not able to access openai-based gpt models.
Also, is there a way to club swe-bench git repo with lm-evaluation-harness repo, so that i can test llm model on swe-bench dataset.
Does lm-eval supports azure openai key access support?


yep, it can support azure openai key

Can you please let me know the steps/code for it?
I searched extensively on GitHub but could not find one.
Thank you!

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: eyuansu62 ***@***.***>
Sent: Thursday, September 19, 2024 8:05:07 AM
To: EleutherAI/lm-evaluation-harness ***@***.***>
Cc: Sudhanshu Mishra ***@***.***>; Author ***@***.***>
Subject: Re: [EleutherAI/lm-evaluation-harness] Configuring Azure OPENAI (Issue #2302)


yep, it can support azure openai key

—
Reply to this email directly, view it on GitHub<https://github.com/EleutherAI/lm-evaluation-harness/issues/2302#issuecomment-2359858953>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BLAGALVE2VZFJELNSN2JZZTZXIZ5XAVCNFSM6AAAAABOIW3OQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJZHA2TQOJVGM>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


you can request by setting the parameter: base_url="xxx".

@sudhanshu-myl Did you manage to use it? I am also interested
Repository: EleutherAI/lm-evaluation-harness
Issue: Missing key in dictionary when loading tasks.
Root cause: Validation Gap

Here, at L274, the item `"task"` has been popped from the dictionary in some execution paths :
https://github.com/EleutherAI/lm-evaluation-harness/blob/fb963f0f0a5b28b69763590bb59676072cf43a01/lm_eval/tasks/__init__.py#L267-L276

Specifically, this one:
https://github.com/EleutherAI/lm-evaluation-harness/blob/fb963f0f0a5b28b69763590bb59676072cf43a01/lm_eval/tasks/__init__.py#L337

I can open a small PR.
Repository: EleutherAI/lm-evaluation-harness
Issue: New Task: `openai_mmmlu` professionaly translated by OpenAI as part of o1 release
Root cause: Unimplemented Feature Gap

From the [OpenAI o1 System Card](https://assets.ctfassets.net/kftzwdyauwt9/67qJD51Aur3eIc96iOfeOP/71551c3d223cd97e591aa89567306912/o1_system_card.pdf):

>"we translated MMLU’s[39] test set into 14 languages using professional human translators. This approach differs from the GPT-4 Paper where MMLU was machine translated with Azure Translate [14]. Relying on human translators for this evaluation increases confidence in the accuracy of the translations"

The datasets are included in their library here -> https://github.com/openai/simple-evals .

Is anyone working on this? I'd be interested in adding these to lm-evaluation-harness. What's a good way to structure this new task in terms of co-existing with the already present `mmlu` versions (`kmllu, cmmlu, arabicmmlu, ...`)

Tagging @baberabb 😄 

Hi! This would be great! We should be able to use the same [nomenclature](https://github.com/openai/simple-evals/blob/2df1a92bbddb8c89fbeb3670e2dd125b10632bca/run_multilingual_mmlu.py#L74) as they use here, maybe prepending it with `openai`? 

This [script](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mmlu/_generate_configs.py) to generate the task boilerplate should come in handy, and let me know if I can help!

Hi @giuliolovisotto @baberabb and all 😊
Any news on this issue ? I see it's PR in https://github.com/EleutherAI/lm-evaluation-harness/pull/2312 blocked, is there a plan to release it ?

Thanks !! 😌

Hey! @giuliolovisotto @baberabb 
Thanks a lot for your effort! Do you still have a plan to release this mmmlu dataset? 
I believe it is promising and will help a lot of people!
Please let me know if there's any progress.
Thanks again!!
Repository: EleutherAI/lm-evaluation-harness
Issue: Running multiple processes on a shared outlines cache database
Root cause: Resource Mishandling

When using the caching feature of outlines with multiple running processes, the cache database may get accessed simultaneously, resulting in disk I/O or database disk image errors:
```
sqlite3.OperationalError: disk I/O error
```

```
sqlite3.DatabaseError: database disk image is malformed
```

This is further described in [https://github.com/vllm-project/vllm/issues/4193](https://github.com/vllm-project/vllm/issues/4193) and [https://github.com/dottxt-ai/outlines/issues/827](https://github.com/dottxt-ai/outlines/issues/827).

A workaround for users, who don't use guided decoding, is described in [https://github.com/vllm-project/vllm/pull/7831](https://github.com/vllm-project/vllm/pull/7831).

Another possible workaround is to set a unique cache directory for each process using the environment variable `OUTLINES_CACHE_DIR`.

An implementation in this framework ([here](https://github.com/EleutherAI/lm-evaluation-harness/blob/fb963f0f0a5b28b69763590bb59676072cf43a01/lm_eval/evaluator.py#L219)) creates a separate cache database for each model "rank". **Does the `use_cache` argument here aim to solve this issue, or does it have a different use?**

Hi! The `use_cache` currently implemented here is for caching evaluation results, so that you can continue where you left off in case of an error.

Okay, thank you. 

Would there be interest in integrating a workaround (e.g., dynamically setting the `OUTLINES_CACHE_DIR`) into the framework?
Repository: EleutherAI/lm-evaluation-harness
Issue: Chat templates
Root cause: Documentation Deficiency

Doesn [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) support default chat templates or should i include in the prompts?

i wanted to know if [this](https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/) is happening by default?
Repository: EleutherAI/lm-evaluation-harness
Issue: AttributeError: 'GPT2TokenizerFast' object has no attribute 'default_chat_template'. Did you mean: 'get_chat_template'?
Root cause: Validation Gap

how to properly use chat templates

```
lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 1 \
    --limit 5 \
    --system_instruction "you are an llm " \
    --apply_chat_template 
```



output 

```
2024-09-17:14:51:24,638 INFO     [evaluator.py:279] Setting fewshot random generator seed to 1234
Traceback (most recent call last):
  File "/home/azime/anaconda3/envs/latest/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/home/azime/Documents/SSD2/latest/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/azime/Documents/SSD2/latest/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/azime/Documents/SSD2/latest/lm_eval/evaluator.py", line 297, in simple_evaluate
    chat_template=lm.chat_template(apply_chat_template),
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/azime/Documents/SSD2/latest/lm_eval/api/model.py", line 434, in chat_template
    template = self.tokenizer.chat_template or self.tokenizer.default_chat_template
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'GPT2TokenizerFast' object has no attribute 'default_chat_template'. Did you mean: 'get_chat_template'?
```

Hi! This is because `gpt-j-6B` is a base model and the HF tokenizer doesn't include a chat template.

but it shouldn't throw error it should not use it according to the documentation. does the same thing with llama 3 too.

yeah, the docs say a default chat template will be applied, but we'll need to update that. According to the HF [docs](https://huggingface.co/docs/transformers/v4.44.2/chat_templating#advanced-how-do-chat-templates-work), that should be the case but doesn't seem to be working:

```python
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
tokenizer.apply_chat_template([{"role": "user", "content": "hello"}])
```
```
ValueError: Cannot use apply_chat_template() because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Comma breaks __repr__ for write-out
Root cause: Validation Gap

Hi @lintangsutawika , this comma seems a formatting artifact but it breaks `--write-out` because `ConfigurableTask.__repr__` now returns a tuple instead of a string. Could you remove it? It was added a few days ago with this [commit](https://github.com/giuliolovisotto/lm-evaluation-harness/commit/fb963f0f0a5b28b69763590bb59676072cf43a01)

https://github.com/EleutherAI/lm-evaluation-harness/blob/fb963f0f0a5b28b69763590bb59676072cf43a01/lm_eval/api/task.py#L1592-L1598

Traceback:

```bash
Traceback (most recent call last):
  File "/Users/badq/.pyenv/versions/3.10.14/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/badq/.pyenv/versions/3.10.14/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/badq/repos/lm-evaluation-harness/lm_eval/__main__.py", line 461, in <module>
    cli_evaluate()
  File "/Users/badq/repos/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/Users/badq/repos/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/Users/badq/repos/lm-evaluation-harness/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
  File "/Users/badq/repos/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/Users/badq/repos/lm-evaluation-harness/lm_eval/evaluator.py", line 460, in evaluate
    print_writeout(task)
  File "/Users/badq/repos/lm-evaluation-harness/lm_eval/evaluator_utils.py", line 197, in print_writeout
    f"Task: {task}; document {inst.doc_id}; context prompt (starting on next line):\
TypeError: __str__ returned non-string (type tuple)```

Thanks. Should be fixed now.
Repository: EleutherAI/lm-evaluation-harness
Issue: GSM8K Problem On Colab With Finetuned Phi3.5 mini model
Root cause: Documentation Deficiency

Hi, I just fine-tuned my microsoft/Phi-3.5-mini-instruct, and LM_eval is working for metrics MMLU, hellaswag, but not gsm8k. I'm testing it on Google Colab with A100, and I did some real inferences directly from gsm8k questions, and it seems like my model is answering them correctly. However, why gsm8k metric is giving me score of 0?

Here is my test [Colab](https://colab.research.google.com/drive/1Y2Y7Nooeuqq78isobCiR2JHDOS8v5ucO?usp=sharing)

![48000202@2x](https://github.com/user-attachments/assets/ed88246d-a65c-4a42-99e5-3160bc982ecc)
![12000203@2x](https://github.com/user-attachments/assets/b24b1e3c-f3f8-48d3-8708-6561f699c88d)


It's probably because you aren't using `--apply_chat_template` to wrap the prompt in the chat format like in your example. You can also add `--log_samples` (use with `--output_path <dir_name>`) to log the per sample generated outputs.

Thanks for your reply!

Do you mean I should do something similar to this? I'm kind of confused, because I didn't use `--apply_chat_template` when I ran mmlu tests, and it worked perfectly fine. Why do I need to do this in this case?

```
!lm_eval --model hf \
    --model_args pretrained=SongTonyLi/Phi-3.5-mini-instruct-SFT-D_chosen-dpo-mix-shuffled5 \
    --tasks gsm8k \
    --device cuda:0 \
    --apply_chat_template \
    --batch_size 8
```

Thanks, `apply_chat_template` resolves the issue. Closing the issue now.
Repository: EleutherAI/lm-evaluation-harness
Issue: Multiple generations (sequential) per question
Root cause: Unimplemented Feature Gap

Hi lm-eval maintainers! I'm relatively new to using this library, so any pointers will be greatly appreciated.

I am trying to elicit two sequential answers from the model for each question. I want to do this by providing the question, recording an initial answer, then appending that answer to the original question and using that as a second prompt, and recording the second (and final) answer to do some post-processing.

After reading through the task_guide and new_task_guide I did not see anything directly related to my endeavor - is there a way to setup this workflow by modifying the relevant yaml config file? 

Or is the preferred method to call the log parser from a custom filter function?

Thank you



Hi! currently the way to do this is a bit involved and will require two `lm_eval` calls (and two task yamls):
  1. The first using `--predict_only` which logs the per sample generations (and docs) to file without computing metrics.
  2. The second yaml which [parses](https://github.com/EleutherAI/lm-evaluation-harness/issues/2257#issuecomment-2318655490) that file to a HF dataset using something like:
  ```yaml
  dataset_path: json
  dataset_kwargs:
    data_files: /test.jsonl
  ```
  and then you should be able to structure your task as normal (the `resps` and `filtered_resps` fields hold the model outputs).

We are currently looking at ways of supporting [lm judge-like](https://github.com/EleutherAI/lm-evaluation-harness/pull/2251) tasks, but its very much a work in progress.
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation of MMLU tasks using the OpenAI API
Root cause: Validation Gap

"Hello, I'm trying to evaluate the GPT-4o model using the MMLU dataset, but I'm encountering an error. Could you advise me on how to proceed?"

"This is the command I used:

lm_eval --model openai-chat-completions \
    --model_args model=gpt-4o \
    --tasks mmlu_anatomy \
    --apply_chat_template \
    --output_path ./result \
    --log_samples \
    --show_config \
    --limit 10

Is there anything I should add or change?"
Let me know if you need any more adjustments!

whats your error?

Traceback (most recent call last):
  File "C:\Users\KT\miniconda3\envs\gpt_test\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\KT\miniconda3\envs\gpt_test\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\KT\miniconda3\envs\gpt_test\Scripts\lm_eval.exe\__main__.py", line 7, in <module>
  File "C:\Users\KT\Desktop\conda_GPT\lm-evaluation-harness\lm_eval\__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "C:\Users\KT\Desktop\conda_GPT\lm-evaluation-harness\lm_eval\utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "C:\Users\KT\Desktop\conda_GPT\lm-evaluation-harness\lm_eval\evaluator.py", line 301, in simple_evaluate
    results = evaluate(
  File "C:\Users\KT\Desktop\conda_GPT\lm-evaluation-harness\lm_eval\utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "C:\Users\KT\Desktop\conda_GPT\lm-evaluation-harness\lm_eval\evaluator.py", line 476, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "C:\Users\KT\Desktop\conda_GPT\lm-evaluation-harness\lm_eval\models\openai_completions.py", line 168, in loglikelihood
    raise NotImplementedError(
NotImplementedError: Loglikelihood is not supported for chat completions. Consider using the completions API instead.

here are the solution: "NotImplementedError: Loglikelihood is not supported for chat completions. Consider using the completions API instead."
Repository: EleutherAI/lm-evaluation-harness
Issue: how to get lm_eval version 4.2 
Root cause: Documentation Deficiency

hello, is it possible to  install  lm_eval version 4.2?

Hi! You can do that either with `pip install lm-eval==0.42` or:
```bash
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
git fetch --tags
git checkout tags/0.42
```
Closing now but feel free to open!
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug in the float limit handling
Root cause: Algorithmic Error

Hi, LM eval team,

In this line: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L439

The limit is reassigned to an int if the original limit flag is a float. However, this does not count the case if I'm running two tasks together.

E.g. TASK A: Full data samples 100, TASK B: Full data samples 1000. And I use a limit of 0.1.

I'm expecting the eval to run 10 samples from TASK A and 100 samples from TASK B.

However, the current logic does not process the TASK B limit compute as the limit is set to 10 already after processing TASK A.

So it ends with we run only 10 samples for TASK B as well.

This should be an easy fix, could you help update your code? Thanks

Hi! nice catch! yes, a PR will be appreciated!

@baberabb could you help review the fix above?

@baberabb friendly ping? This should be an easy fix

Hi! @cjluo-omniml . left a comment in the PR earlier. The script uses `limit` again downstream as well so need to handle that as well.

@baberabb fixed. Could you review again?

@baberabb friendly ping?
Repository: EleutherAI/lm-evaluation-harness
Issue: Failed to add a new metric
Root cause: Documentation Deficiency

Hello, 
I tried to add a new metric to an existing multiple-choice task, but it seems that the metric was not added. 
I edited MedQA:

```
task: medqa_4options
dataset_path: GBaker/MedQA-USMLE-4-options-hf
output_type: multiple_choice
training_split: train
validation_split: validation
test_split: test
doc_to_text: !function preprocess_medqa.doc_to_text
doc_to_target: !function preprocess_medqa.doc_to_target
doc_to_choice: [ 'A', 'B', 'C', 'D' ]
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: acc_norm
    aggregation: mean
    higher_is_better: true
  - metric: !function preprocess_medqa.precision_fn
    aggregation: !function preprocess_medqa.precision_metric
    higher_is_better: true

```

In addition I edited `preprocess_medqa.py` and added the new metric: 
```
from sklearn.metrics import classification_report

def doc_to_text(doc) -> str:
    option_choices = {
        "A": doc["ending0"],
        "B": doc["ending1"],
        "C": doc["ending2"],
        "D": doc["ending3"],
    }
    answers = "".join((f"{k}. {v}\n") for k, v in option_choices.items())
    return f"Question: {doc['sent1']}\n{answers}Answer:"


def doc_to_target(doc) -> int:
    return doc["label"]

def precision_fn(items):
    print("in precision_fn")
    return items

def precision_metric(items) -> float:
    print("in precision metric !!")
    return 0.5

```


However, I don't find the prints or the new metric in the results of lm-eval. 
```
Running loglikelihood requests: 100%|████████████| 5092/5092 [00:50<00:00, 100.08it/s]
2024-09-23:07:19:53,957 INFO     [evaluation_tracker.py:269] Output path not provided, skipping saving results aggregated
hf (pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
|    Tasks     |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|--------------|-------|------|-----:|--------|---|-----:|---|-----:|
|medqa_4options|Yaml   |none  |     0|acc     |↑  |0.4501|±  |0.0139|
|              |       |none  |     0|acc_norm|↑  |0.4501|±  |0.0139|

``` 

The run command was: 
`lm_eval --model hf  --model_args pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct --apply_chat_template  --tasks medqa_4options`

Can you please help me? 
Thanks! 



I had the same issue specifically for custom metrics for multiple choice tasks. My workaround was to override the `process_result` function. I did so by defining a custom `process_result` function in `utils.py` and calling it in my `task.yaml` setup.

See https://github.com/EleutherAI/lm-evaluation-harness/blob/69ecd0b929701d346c1119d5cd0563ade4ab3536/lm_eval/api/task.py#L1665 to understand the default `process_result` function. Then override it to include your new desired metric. 

I'm not sure if this is the "correct" approach, but I got it working this way.
Repository: EleutherAI/lm-evaluation-harness
Issue: Confusion over the model outputs
Root cause: Documentation Deficiency

Hi team,

Thanks a lot for this package! I've been running the bbh benchmarks. Logging the model responses, I'm seeing something weird: The data seems to indicate that the model is outputting lower scores for the correct answers. I'm wondering if there's something I'm missing (e.g. is there a negative sign put into the scores for some reason?). For example, [for the phi-3.5 model](https://huggingface.co/datasets/open-llm-leaderboard/microsoft__Phi-3.5-mini-instruct-details), under doc_id 0 the label is False and the model response is:

```
[ [ "-3.5762786865234375e-07", "True" ], [ "-15.0", "False" ] ]
```

This implies that the logits are higher for True, but the acc_norm column seems to mark the model response as correct. Am I misreading the scores that come with the model responses? [I've tried creating a discussion thread here](https://huggingface.co/datasets/open-llm-leaderboard/microsoft__Phi-3.5-mini-instruct-details/discussions/1), but figured I'd just ask the team directly. 

Thanks,
Linh
Repository: EleutherAI/lm-evaluation-harness
Issue: Locally reproducible HF-Leaderboard evals
Root cause: Documentation Deficiency

Hi folks,

I am trying to run HF-Leaderboard (v2) evals locally, and according to the blog https://huggingface.co/spaces/open-llm-leaderboard/blog the scores are normalized and random prediction accuracy is subtracted for all tasks where this is applicable.
Is this by any chance supported in lm-evaluation-harness, and if not, do you have any pointers where to look in order to add this behaviour?

Hi! Their [documentation](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about) links to this [branch](https://github.com/huggingface/lm-evaluation-harness/tree/adding_all_changess)

I have already tried, but the reported scores are not matching the ones in https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard , and also their branch crashes with `--apply_chat_template` + vllm backend. 
At the same time, this works just fine on `main` here, which is why I hoped to add these aggregation scores so the leaderboard is fully reproducible locally with `lm-evaluation-harness`.

I also failed to use their repo/branch via vllm. The results are very off!
Repository: EleutherAI/lm-evaluation-harness
Issue: AttributeError: 'dict' object has no attribute 'has_test_docs'
Root cause: Interface Contract Mismatch

Traceback (most recent call last):
  File "/Users/sshubam/miniforge3/envs/ai4b/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/sshubam/miniforge3/envs/ai4b/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/sshubam/Desktop/lm_eval_harness/lm-evaluation-harness/scripts/write_out.py", line 102, in <module>
    main()
  File "/Users/sshubam/Desktop/lm_eval_harness/lm-evaluation-harness/scripts/write_out.py", line 73, in main
    if set == "test" and task.has_test_docs():
AttributeError: 'dict' object has no attribute 'has_test_docs'


I am getting this error on running:

python -m scripts.write_out \
    --output_base_path lm_eval_harness/OUTPUTS \
    --tasks mmlu \
    --sets test \
    --num_fewshot 5 \
    --num_examples 5
Repository: EleutherAI/lm-evaluation-harness
Issue: gpt2 evaluation
Root cause: Documentation Deficiency

Can I evaluate GPT-2 on the GLUE dataset using lm-eval? If so, could you explain how to do it?

Also, are there any other popular benchmarks for GPT-2 that I can evaluate using lm-eval?

Thank you!

The example code in the readme will run for GPT-2.