zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Is there something wrong with 'google/gemma-1.1-2b-it' ?
Root cause: Configuration Error

I test google/gemma-1.1-2b-it on gsm8k with following command
`
CUDA_VISIBLE_DEVICES=3  lm_eval --model vllm \
    --model_args pretrained=gemma-1.1-2b-it
,dtype=auto,gpu_memory_utilization=0.8, \
    --tasks gsm8k \
    --batch_size auto
`
My result is :
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/88258534/f830b1d1-e8f5-4d45-86c0-cd5a9bb828ad)

I think this is a weird result since gemma-2b reported a score close to 18 while here google/gemma-1.1-2b-it  only get less than 10...
Any idea?  😢 

The problem is relevant with gemma and vllm. Use naive method to inference can get kind of normal result like(noticed here I use 8-shot):
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/88258534/afeb86ac-7e2e-4204-bcb1-68a86bf29b60)


Hi! Could you rerun with `add_bos_token = True`? The HF model type adds a BOS token, and notes this in the logs--just added in #1857 a change to the default behavior so that VLLM should match this.

For reasons unclear to me, Gemma performance is dramatically lower when it does not receive a BOS token.

> Hi! Could you rerun with `add_bos_token = True`? The HF model type adds a BOS token, and notes this in the logs--just added in #1857 a change to the default behavior so that VLLM should match this.
> 
> For reasons unclear to me, Gemma performance is dramatically lower when it does not receive a BOS token.

Thanks, happy to know this flag, will try it later. But I still have some question about gemma, one is it version dramatically worse than base version (10 points lower on gsm8k), the other  is gemma model get lower score on gsm8k-cot than gsm8k. 

However  I have tried a gsm8k script from gemma official github, which do show considerable benifit from cot prompt.  I tried to align hf-eval and deepmind script but in vain. Would you help like to check this two strange question?  Thanks for your brilliant job here.

Hi, @haileyschoelkopf  I have tried add_bos_token=True like this

```bash
 lm_eval --model vllm    --model_args pretrained=gemma-2b,add_bos_token=True --tasks  gsm8k-cot   --batch_size auto
```
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/88258534/5ddac1b5-2c1d-4981-adb4-78635f7772d7)
This score still does not match the official score of gemma-2b (approximately 18), however, using the scripts from the gemma repository can.  https://github.com/google-deepmind/gemma/blob/main/colabs/gsm8k_eval.ipynb
I modified it with vllm , get 19.18119787717968


Using lm_eval harness hf model to test gsm8k_cot can get -> 16.76(19.26)   which is quite similar to my result. 
So there might exists other problem with vllm+gemma
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue with Dataset Loading for Task 'polemo2'
Root cause: Configuration Error

Error Message:
`ValueError: BuilderConfig 'klej-polemo2-out' not found. Available: ['default']`

Command Used:
`accelerate launch -m lm_eval --model hf --model_args pretrained=Qwen1.5-0.5B --tasks polemo2 --batch_size 32`

Observations:
I have checked the GitHub repository at [lm-evaluation-harness/tasks/polemo2](https://github.com/EleutherAI/lm-evaluation-harness/tree/1980a13c9d7bcdc6e2a19228c203f9f7834ac9b8/lm_eval/tasks/polemo2) for guidance.
After a preliminary investigation, I suspect that this issue may not be isolated to the 'polemo2' task alone. There may be multiple tasks with similar problems.
I plan to conduct a more thorough review of the tasks and will compile a list of any additional issues I encounter.
I will follow up with this issue to provide updates on my findings.


I appreciate any guidance or assistance you can provide to help resolve this issue. Thank you for maintaining this valuable tool for the community.

Best regards,

zhabuye

@haileyschoelkopf @lintangsutawika 

我之前加载其他数据集也遇到类似的问题，原因是数据集加载路径的问题，每个数据集下都有一个.py文件，py文件会有_URL变量,这个需要修改为自己的路径

I resolved the issue by changing `dataset_name: klej-polemo2-out` to `dataset_name: null` in the `polemo2_out.yaml` file. What is the principle behind this problem?
@haileyschoelkopf @lintangsutawika 

Looks like it's just that the dataset doesn't have a data subset name and somehow was mistyped.

https://huggingface.co/datasets/allegro/klej-polemo2-out

Using `dataset_path` to `allegro/klej-polemo2-out` is enough.

> Looks like it's just that the dataset doesn't have a data subset name and somehow was mistyped.
> 
> https://huggingface.co/datasets/allegro/klej-polemo2-out
> 
> Using `dataset_path` to `allegro/klej-polemo2-out` is enough.

Okay, I have submitted a Pull request for this
Repository: EleutherAI/lm-evaluation-harness
Issue: Llama2-hf-q40.gguf model got very poor results on lambada_openai tasks, but was fine on other tasks.
Root cause: Algorithmic Error

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/105184542/ebf87f13-7a0e-4733-b6f6-ac55c44031ce)
The model used is from [HF](https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/tree/main)
The steps for testing are as follows：
 - pip install 'llama-cpp-python[server]'
 - python3 -m llama_cpp.server --model llama-2-7b-chat.Q4_0.gguf  --host 0.0.0.0
 - lm_eval --model gguf --tasks lambada_openai --model_args model=local_gguf_model,base_url=http://0.0.0.0:8000
Repository: EleutherAI/lm-evaluation-harness
Issue: Add more math evaluation tasks
Root cause: Unimplemented Feature Gap

Regarding math evaluation, it seems that the current repo only supports gsm8k, math, two problem solving benchmarks. There are more fantastic evaluation suites being proposed recently, like using PoT, python code interpreter, formal proof, etc. Maybe we could integrate tasks from this repo(https://github.com/ZubinGou/math-evaluation-harness) for more comprehensive evaluation?

We are very interested in including more math evaluation tasks. The linked repo is a good reference--If there are other specific tasks you would like definitely drop links to them here.
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation MC Questions
Root cause: Documentation Deficiency

I have a question regarding evaluating LLMs on mc questions using loglikelihood of tokens. From existing implementations like MMLU etc, the code snippet would look like this:
```
# Create the model
model = AutoModelForCausalLM.from_pretrained(...)
tokenizer = AutoTokenizer.from_pretrained(...)

# Tokenize the inputs
inputs = tokenizer(...)

# Get the logits
with torch.no_grad():
    outputs = model(input_ids=inputs["input_ids"].to(model.device), attention_mask=inputs["attention_mask"].to(model.device))
logits = outputs.logits[:, -1, :]

# Compute probabilities
probs = (
    torch.nn.functional.softmax(
        torch.tensor(
            [
                logits[tokenizer(" A").input_ids[-1]], # or sometimes tokenizing "A" instead of " A" etc
                logits[tokenizer(" B").input_ids[-1]],
                logits[tokenizer(" C").input_ids[-1]],
                logits[tokenizer(" D").input_ids[-1]],
            ]
        ).float(),
        dim=0,
    )
    .detach()
    .cpu()
    .numpy()
)
prediction = {0: "A", 1: "B", 2: "C", 3: "D"}[np.argmax(probs)]
```

However, the result from doing this is usually very different from lm evaluation harness. For example, running evaluations on llama would result in 2 different numbers. I looked into the code snippet within this repo but cannot be sure what is the difference. I understand that this repo has more wrappers around, but under the hood, is the idea the same as the code snippet? I am a bit confused since I get vastly different results from lm eval harness by coding in this way.

Yes, the result should be the same (at least for the single-token completions case). Have you compared the logits you get to the observed per-sample values? (the ones we log won't have the softmax applied to them though, we just select the largest one)



Something to flag is that occasionally model predictions can flip due to things as minor as running with a different batch size, if logits on two answer choices were very very close or the same floating point value. But there should not be large divergences
Repository: EleutherAI/lm-evaluation-harness
Issue: Use local data sets
Root cause: Documentation Deficiency

Hello!
I downloaded the allenai/ai2_arc dataset using the huggingface hub, as shown in Figure 1. You also set dataset_kwargs, as shown in Figure 2. But he still reports an error, as shown in Figure 3. Why is this?

![微信截图_20240523184327](https://github.com/EleutherAI/lm-evaluation-harness/assets/64513343/24fd1ffd-7fc8-494b-8839-19ebc2efdc15)

![微信截图_20240523184256](https://github.com/EleutherAI/lm-evaluation-harness/assets/64513343/7c12a8f0-e971-41a8-a856-bcdc16c5d500)

![微信截图_20240523184412](https://github.com/EleutherAI/lm-evaluation-harness/assets/64513343/e3f96af9-3a61-4eaf-8d8e-5365617024aa)


Hi @claire360 , you can avoid this connection error by setting the environment variable `HF_DATASETS_OFFLINE=1` , I believe.

@haileyschoelkopf  Thank you, this method is very useful, problem solved
Repository: EleutherAI/lm-evaluation-harness
Issue: Release schedule - more regular PyPI releases?

As a researcher, having more frequent releases on PyPI would greatly aid in referencing specific versions for reproducibility.

Currently, some bug fixes are only available in the main branch, making it necessary to reference specific commits. This can complicate reproducing results due to unclear changes between versions.

Thank you for this amazing library! 🙏

Thanks for the interest and for pointing this out!

We'll try to get back on track for more frequent PyPI releases. Likely will see about cutting a new version release once #1873 lands and has settled a bit!
Repository: EleutherAI/lm-evaluation-harness
Issue: test_docs of scrolls dataset
Root cause: Documentation Deficiency


Why do all datasets in `tau/scrolls` set the `has_test_docs` to False?  The dataset `narrativeqa` has a test split. 

```python
class _SCROLLSTask(Task):
    ....
    ....
    def has_training_docs(self):
        return True

    def has_validation_docs(self):
        return True

    def has_test_docs(self):
        return False

    def training_docs(self):
        for doc in self.dataset["train"]:
            yield from self._process_doc(doc)

    def validation_docs(self):
        for doc in self.dataset["validation"]:
            yield from self._process_doc(doc)
```

It leads to evaluation based on the validation set by default. What does not use the test set? 

Hi! This is because Scrolls does not share the test set labels, similar to GLUE. 

If you have ideas for improving the clarity of this (perhaps with a note in the task's readme under `lm_eval/tasks/scrolls`?) then that would be very helpful!
Repository: EleutherAI/lm-evaluation-harness
Issue: eval with Alpaca template
Root cause: Documentation Deficiency

Do you have any PRs or examples that **wrap around the [Alpaca template](https://github.com/tloen/alpaca-lora/blob/main/templates/alpaca.json)**?

I know this issue has been covered before (#896), but I was wondering if it has been updated since then.

#1873 will add support for arbitrary Chat Templates via Huggingface tokenizers soon, and you could try that PR out now!
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Regression Testing
Root cause: Unimplemented Feature Gap

As per the issue title. 

It would be great to have regression tests for some core tasks set up, checking that 1. some key tasks' scores don't regress or change and 2. that the printouts for results stay the same (results tables).


ideally, we'd have:
- on-CPU tests (with `limit=10`) that can always run
- full on-GPU tests that can be run when desired for larger changes.

If any contributors are interested in taking this on, it'd be a huge help! Otherwise, I hope to get to this.

@haileyschoelkopf 
1. Which tasks would you consider as core?
2. Which models do you believe are most effective/efficient for doing this?

Hello @haileyschoelkopf, I would be interested in helping with this.

Thanks all for the interest! 

@clefourrier @KonradSzafer are tackling some regression testing now actually--if anything can be passed off maybe they can share. 

models that would be useful to test would be
- gpt2
- a small model which uses a Llama tokenizer / sentencepiece

and core tasks would be
- `mmlu`
- `arc_easy`, `arc_challenge`
- `lambada_openai`
- `wikitext` 
- other Open LLM Leaderboard tasks
- more could be added based on computational cost of running these tests.


Something that would be helpful which isn't currently being done is checking equivalence of the printed results tables, to ensure that the formatting there or printing of task groupings does not get modified by a new PR. This could be done by extending the current `tests/test_evaluator.py` tests. 

Something which could be taken on and is not yet being worked on would be to implement a greater number of tests for the `evaluate()` and `simple_evaluate()` functions, or to mock the CLI and test `cli_evaluate()` with various options to ensure that these components are functioning as intended. I could describe a number of tests that would be useful if this is of interest! the CLI testing would be a big one.

@haileyschoelkopf could you expand on the new tests that you consider as useful?
Repository: EleutherAI/lm-evaluation-harness
Issue: can we add C4 and PTB tasks for PpL?
Root cause: Unimplemented Feature Gap

Hello, can it add PPL for C4 and PTB tasks? I see that Wikitext is already supported, and I have added the YAML for C4 based on Wikitext. I have changed the dataset understanding and path, which is not correct. Can you help me support these two tasks?

We'd welcome support for these tasks, so long as only the val set for these tasks needs to be downloaded to run them!

(Don't want the train set of C4 to erroneously be downloaded and take up users' disk space)

> Hello, can it add PPL for C4 and PTB tasks? I see that Wikitext is already supported, and I have added the YAML for C4 based on Wikitext. I have changed the dataset understanding and path, which is not correct. Can you help me support these two tasks?
@123wujiao Hello, have you successfully run the tests for the C4 dataset? Could you please share your YAML file? Thank you!



Hi! @123wujiao @haileyschoelkopf @rippleD030 I want to try to add the C4 task and will submit a PR soon.

> Hi! [@123wujiao](https://github.com/123wujiao) [@haileyschoelkopf](https://github.com/haileyschoelkopf) [@rippleD030](https://github.com/rippleD030) I want to try to add the C4 task and will submit a PR soon.

Just submitted a PR. Feel free to review it!
Repository: EleutherAI/lm-evaluation-harness
Issue: ImportError: cannot import name 'HfApi' from 'huggingface_hub'
Root cause: External Dependency Breakage

The main script fails to run for me with this import error. Tried on 3.11 and 3.12 and also with a new environment. Full traceback below:
```bash
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "../lm-evaluation-harness/lm_eval/__main__.py", line 3, in <module>
    import logging
  File "../lm-evaluation-harness/lm_eval/logging/__init__.py", line 1, in <module>
    from .evaluation_tracker import EvaluationTracker
  File "../lm-evaluation-harness/lm_eval/logging/evaluation_tracker.py", line 8, in <module>
    from huggingface_hub import HfApi
ImportError: cannot import name 'HfApi' from 'huggingface_hub' (../lm-evaluation-harness/.venv2/lib/python3.11/site-packages/huggingface_hub/__init__.py)
```
huggingface-hub==0.23.1


Does #1858 fix your issue by chance?

> Does #1858 fix your issue by chance?

yup working! closing the issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError of scrolls_narrativeqa
Root cause: Validation Gap

The error occurred when I tried to test with `--tasks scrolls_narrativeqa`. My command was as follows:
```shell
lm_eval --model hf     --model_args pretrained=princeton-nlp/Sheared-LLaMA-2.7B,parallelize=True  --tasks scrolls_narrativeqa    --batch_size auto     --output_path /lm_eval_out/Sheared-LLaMA-2.7B    --use_cache /lm_eval_out/Sheared-LLaMA-2.7B/eval_cache    --num_fewshot 0
```
```shell
"/lm-evaluation-harness/lm_eval/evaluator.py", line 596, in <dictcomp>
"original": len(task_output.task.eval_docs),
TypeError: object of type 'generator' has no len()
```
I do some changes in `evaluator.py` then the evaluator can work smoothly
I want to make sure if these changes are allowed here
```shell
from collections.abc import Generator
...
# "n-samples": {
#     task_output.task_name: {
#         "original": len(task_output.task.eval_docs),
#         "effective": min(
#             limit if limit else len(task_output.task.eval_docs),
#             len(task_output.task.eval_docs),
#         ),
#     }
#     for task_output in eval_tasks
# },

"n_samples": {
        task_output.task_name: {
            "original": len(list(task_output.task.eval_docs)) if isinstance(task_output.task.eval_docs, Generator) else len(task_output.task.eval_docs),
            "effective": min(
                limit if limit else (len(list(task_output.task.eval_docs)) if isinstance(task_output.task.eval_docs, Generator) else len(task_output.task.eval_docs)),
                len(list(task_output.task.eval_docs)) if isinstance(task_output.task.eval_docs, Generator) else len(task_output.task.eval_docs),
            ),
        }
        for task_output in eval_tasks
    },
```
also some changes in `evaluator_utils.py`
```shell
# def get_sample_size(task, limit: Optional[int]) -> Union[int, None]:
#     if limit is not None:
#         limit = (
#             int(math.ceil(len(task.eval_docs) * limit)) if limit < 1.0 else int(limit)
#         )
#     return limit

def get_sample_size(task, limit: Optional[int]) -> Union[int, None]:
    if limit is not None:
        eval_docs_length = len(list(task.eval_docs)) if isinstance(task.eval_docs, Generator) else len(task.eval_docs)
        limit = int(math.ceil(eval_docs_length * limit)) if limit < 1.0 else int(limit)
    return limit
```
Repository: EleutherAI/lm-evaluation-harness
Issue: GPU memory very high and unbalanced when testing Gemma
Root cause: Configuration Error

I'm running Gemma-7B with RTX 4090, It takes up a lot of GPU memory, but llama3-8B can run normally under the same prompt. When I start the tensor parallel strategy, the GPU memory is very unbalanced (as shown in the figure below). Is this a Gemma bug or is there something wrong with my usage?

My platform：Linux
Python version: 3.10.14
lm-evaluation-harness version: 0.4.2

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/79445290/2239f9f2-09f4-4891-a25d-19e5ccb41243)


> I'm running Gemma-7B with RTX 4090, It takes up a lot of GPU memory, but llama3-8B can run normally under the same prompt. When I start the tensor parallel strategy, the GPU memory is very unbalanced (as shown in the figure below). Is this a Gemma bug or is there something wrong with my usage?
> 
> My platform：Linux Python version: 3.10.14 lm-evaluation-harness version: 0.4.2
> 
> ![image](https://private-user-images.githubusercontent.com/79445290/334124667-2239f9f2-09f4-4891-a25d-19e5ccb41243.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY4MTgxOTEsIm5iZiI6MTcxNjgxNzg5MSwicGF0aCI6Ii83OTQ0NTI5MC8zMzQxMjQ2NjctMjIzOWY5ZjItMDlmNC00ODkxLWEyNWQtMTllNWNjYjQxMjQzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI3VDEzNTEzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJjZmM3NzBiYjdlMDhmNDViYWNkYzEwOWYzMzQ3NjQ0ZTFlYTI5OTZmNWQ4MzZjZDY3ZTcxNTEyNTU1MGY2ODYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.Dq9K5dsUBWpKlPUAXdFXqhfXqWp9fgtt7XySVRruqi8)

The figure is running with batch_size 1

hi! Is this using `--model hf` with `parallelize=True`?

If so, I think this is sometimes expected behavior from HF. adding `device_map_option='balanced'` to `--model_args` may fix this? or simply upping the batch size.
Repository: EleutherAI/lm-evaluation-harness
Issue: llama3-base gsm8k score
Root cause: Documentation Deficiency

I met a problem relevant to #1799. 
The scores of LLaMA3-8B-base on the GSM8K benchmark are significantly lower than the scores reported by the official sources. 

I use gsm8k_cot task in lm_eval_harness get same score 50+ as https://github.com/EleutherAI/lm-evaluation-harness/issues/1799#issuecomment-2099958349  while official reported is 79.6
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/88258534/a8b13fe1-e0e8-40c8-bc46-e2ddd1e081d0)

Any idea about this ?

The 79.6 GSM8k number reported by Meta comes from their instruct model. I'm not certain if they've reported "official" GSM8k base model scores

Sorry for not paying attention to the subtitle， yes 79.6 is llama3-8B-it score : )


Hello,
could you please share what you ran to get 50+ on llama3 8b base model? My command is 
accelerate launch -m lm_eval --model hf --model_args pretrained=.../llama-3-8b --tasks gsm8k_cot --num_fewshot 8 --gen_kwargs max_gen_toks=512
and the result I get is this
hf (pretrained=.../llama-3-8b), gen_kwargs: (max_gen_toks=512), limit: None, num_fewshot: 8, batch_size: 1
|  Tasks  |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|---------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_cot|      3|flexible-extract|     8|exact_match|↑  |0.3427|±  |0.0131|
|         |       |strict-match    |     8|exact_match|↑  |0.2494|±  |0.0119|
Repository: EleutherAI/lm-evaluation-harness
Issue: model_comparator.py broken
Root cause: External Dependency Breakage

Referencing this issue https://github.com/meta-llama/llama-recipes/pull/488/files where it seems that the `evaluator` class is no longer used.

I could not get the model_comparator.py script to run as it throws the same attribute error `AttributeError: module 'lm_eval.tasks' has no attribute 'initialize_tasks'` so my guess is that the script is broken.

I actually don't understand the point of the script, isn't it more straightforward to eval using predict-only at temperature 0 then directly compare the results? since at any non zero temperature the results would not tally across different backends.

thanks for taking the time to read.