zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: All responses are considered false?
Root cause: Documentation Deficiency

I am evaluating several LLMs on my custom dataset using negative log likelihood and I always thought that the response with the highest likelihood is paired with 'true' while the others are paired with 'false' in the resulting json files. However, in some of my experiments I realized that for some quesitons, the likelihoods for all of the options are paired with 'false'. I wonder whether this is due to a bug and also whether this affects the model evaluation.

"resps": [
      [
        [
          -2.6447083950042725,
          false
        ]
      ],
      [
        [
          -2.3947083950042725,
          false
        ]
      ],
      [
        [
          -2.2697083950042725,
          false
        ]
      ],
      [
        [
          -4.644708633422852,
          false
        ]
      ],
      [
        [
          -2.5197083950042725,
          false
        ]
      ]
    ],

Just check the code producing and processing it. These booleans are not what you think they are ;)


Hi! Also replying to #1672 here: 

The format of these saved outputs are what is returned at https://github.com/EleutherAI/lm-evaluation-harness/blob/9ae96cdfd816ded3c7c10f625f56aadfce708bdf/lm_eval/models/huggingface.py#L1095 . 

The first `float` number in each tuple is a floating-point value that represents the sum of the per-token loglikelihoods. because these are log-probabilities, **they will always be <= 0** since we'd take `exp(log-probability)` to get a probability between 1 and 0.

The second value in each of these tuples here is a `bool` value that is `False` if the given continuation string's tokens would *not* be produced via starting from the input context and then greedily decoding for the appropriate number of tokens. If it would be produced via greedy decoding, the value will be `True`. (for `lambada_openai`, the `acc` metric maps directly from this value.)

> Multiple options have exactly the same likelihood
Multiple options have exactly the same decimal number (whole 15 decimal points) with different whole numbers
Multiple options have the same decimal number up to some number of digits and they differ at smaller decimal locations (e.g., -2.7487025260925293 and -1.7487024068832397)
Multiple options have the same decimal number starting after some number of digits (This one is particularly strange since I cannot think of a natural mathematical operation that will yield numbers with different initial n decimal digits followed by the same digits such as -2.2487025260925293 and -2.7487025260925293)
The reason for these oddities should be because the logits are returned in a limited-precision floating point representation (fp16, bf16, fp32). So due to rounding errors certain values may pop up more than one might expect--not every real number is representable without error in these floating-point formats.


Hope this is helpful!
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for sequence tagging tasks
Root cause: Unimplemented Feature Gap

We are trying to evaluate Named Entity Recognition and Part of Speech tagging tasks, but it is unclear to us how to do that.
We've noticed that `aclue` include a Named Entity Recognition task but it is treated as a multiple choice not a sequence tagging task.

Hi!

If I'm remembering right, the question of supporting sequence labeling tasks has come up before--The primary reason we haven't added any is that we're focusing on prompted evaluation of autoregressive LMs, and I'm not aware of a popular + effective framing of NER or POS tagging tasks as a prompted evaluation.

Maybe there have been proposals recently for this though that have gotten traction, if so we'd be happy to accept additions of such tasks!

Hello, I've been working on using llms for NLP tasks like NER and relationship extraction and would be interested in adding these tasks and datasets to the harness. You can see here several papers and datasets on this:   
* [Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023). Is ChatGPT a General-Purpose Natural Language Processing Task Solver? arXiv preprint arXiv:2302.06476.](https://arxiv.org/abs/2302.06476)  
* [Wei, X., Cui, X., Cheng, N., Wang, X., Zhang, X., Huang, S., Xie:, P., Xu, J., Chen, Y., Zhang, M., Jiang, Y., & Han, W. (2023a). Zero-Shot Information Extraction via Chatting with ChatGPT. arXiv preprint arXiv:2302.10205.](https://arxiv.org/abs/2302.10205)  
* [Xu, X., Zhu, Y., Wang, X., & Zhang, N. (2023). How to Unleash the Power of Large Language Models for Few-shot Relation Extraction? arXiv preprint arXiv:2305.01555.](https://arxiv.org/abs/2305.01555)    

What do you think? 

Any option of evaluation on these NLP tasks anytime soon?
Repository: EleutherAI/lm-evaluation-harness
Issue: Add a docs FAQ section
GitHub labels: documentation

There's a lot of content in our docs but not all of it is easily discoverable for new users. We get some questions asked in issues frequently, which mainly means we need to improve discoverability.

Adding a `docs/faq.md` which contains QnA for common footguns / pitfalls / errors and how to resolve would be very useful.

Some things for which an FAQ would be very very helpful: 

- "how is evaluation {request_type} implemented"? --> link to our new paper's appendix describing these details (https://arxiv.org/abs/2405.14782)
- Common mistakes when running the library offline without access to HuggingFace
- TL;DR on using saved-locally datasets
- ...
Repository: EleutherAI/lm-evaluation-harness
Issue: List of `until` tokens in cli args not supported
Root cause: Validation Gap

`gen_kwargs` are parsed via [`simple_parse_args_string()`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L141). This parsing function assumes that `gen_kwargs` can be split into arg-value strings by `,`, i.e. `"do_sample=false, temperature=1.0"` -> `["do_sample=false", "temperature=1.0"]`. See https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/utils.py#L72

This assumption doesn't hold when there are multiple tokens in the until list, e.g.

```
python -m lm_eval --tasks=xnli --gen_kwargs=until=["\n\n", "<\\s>"], do_sample=fal
se, temperature=1.0 --model=pretrained=mistralai/Mistral-7B-v0.1
```

will result in `__main__.py: error: unrecognized arguments: <\s>], do_sample=false, temperature=1.0`

The commit I ran with: `b177c82cb3ef4a8f03ff77341970ab9d77ffadb4`

I hope the issue description is clear. Otherwise, I'm very willing to further explain it.

Hi! 

Thanks for the report! 

Our current recommendation for working around issues like this is to run the library's `simple_evaluate()` function with the intended args passed within a python script of your own (could be a patched version of __main__.py). If there are troubles with this or it still doesn't support what you want, happy to help fix!

In the hopefully-near future we will be investigating the addition of a new entrypoint to allow for functionality we see people commonly avoiding the CLI to add, such as sweeps over a few settings / models, and things like your issue where the args you'd like to pass are too complex to directly parse from CLI arg strings. See #1622 for future discussions there!

Hi @haileyschoelkopf , thanks for your response!

I'm not sure I understand your proposal with `simple_evaluate()`. This function accepts `gen_kwargs` as a `str` and parse it with `simple_parse_args_string()`. This parsing function, `simple_parse_args_string()`, doesn't accommodate `gen_kwargs` with complex data strcture, e.g. when `until` is a list of tokens. Other than that, `simple_evaluate()` and `cli_evaluate()` both work just fine.

#1622 is what our team is doing, we have our own way of managing different settings and we pass the args as cli args to `lm-eval`.

```
lm_eval_main = importlib.import_module("lm_eval.__main__")
lm_eval_main.cli_evaluate(lm_eval_args)
```

Our current setup works pretty well except for when `until` contains a list as opposed to a single stop token.

I see sorry--will make it possible to pass an arg string or a dict, as we support for model args (https://github.com/EleutherAI/lm-evaluation-harness/blob/7852985b2b5352df147067e01a121c52297f8821/lm_eval/evaluator.py#L167), when I'm back from vacation!
Repository: EleutherAI/lm-evaluation-harness
Issue: Incorrect evaluation results when using 'mps' backend
Root cause: External Dependency Breakage

When using lm_eval with `cpu` backend on my Macbook, I can reproduce the results published in https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.

`
$ lm_eval --device cpu --model hf --model_args pretrained=openai-community/gpt2,revision=11c5a3d --tasks truthfulqa_mc2  --batch_size 1
`
|    Tasks     |Version|Filter|n-shot|Metric|Value |   |Stderr|
|--------------|------:|------|-----:|------|-----:|---|-----:|
|truthfulqa_mc2|      2|none  |     0|acc   |0.4069|±  |0.0149|

However, after switched the backend to `mps`, the results are completely different:

`
$ lm_eval --device mps --model hf --model_args pretrained=openai-community/gpt2,revision=11c5a3d --tasks truthfulqa_mc2  --batch_size 1
`
|    Tasks     |Version|Filter|n-shot|Metric|Value |   |Stderr|
|--------------|------:|------|-----:|------|-----:|---|-----:|
|truthfulqa_mc2|      2|none  |     0|acc   |0.5391|±  |0.0167|

Notably, there are lots of pytorch `mps` correctness issues being reported: https://github.com/pytorch/pytorch/issues?q=is%3Aissue+is%3Aopen+mps

Thanks!

I am using MacOS 14.4.1 on a M3 Max MBP. pytorch version is 2.2.2.

Just tried pytorch's nightly build and the problem appears to be fixed. Could be https://github.com/pytorch/pytorch/commit/05289a278c3eaca271061649982f38c435b50674, but I am not sure.

Hi! Glad that Pytorch nightly seems to have fixed your issue. 

Not much we can do from our side on this--but I'll make a note that MPS backend in pytorch is still early-stage so others are aware to take caution!
Repository: EleutherAI/lm-evaluation-harness
Issue: Add docstring for HFLM's many keyword args
GitHub labels: documentation

help wanted

feature request

good first issue

We have a lot of things being added to the HFLM class, such as the ability to add a BOS token, control said BOS token, the reuse of logits for 1-token loglikelihood tasks, and more.

A docstring explaining what each keyword argument to the initialization would be great.

I'd like to work on this, if it's still an issue
Repository: EleutherAI/lm-evaluation-harness
Issue: "--log_sample" will log the output twice when evaluate gsm8k
Root cause: Algorithmic Error

Hi! Thanks for your amazing project!

When I try to evaluate the gsm8k bench and log the output sample with "--log_sample", I find the output JSON file including the response twice.

For example, the gsm8k bench has 1319 test samples for the testing set, while the output JSON file includes 2638 logs. I also find that the last 1319 is a copy of the first 1319 logs.

My question is:
- **Is that a bug or did I make some mistake in the evaluation?**

Here is my command:
```
model_name="mistralai/Mistral-7B-v0.1"

lm_eval --model hf \
    --model_args pretrained=$model_name \
    --tasks gsm8k \
    --device cuda:7 \
    --batch_size 8 \
    --output_path "output/gsm8k/Mistral-7B-v0.1" \
    --log_samples
```

The config file of gsm8k is as same as the [gsm8k.yaml](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/gsm8k/gsm8k.yaml).

Thank you in advance!

Best wishes.



I believe I have solved it.

During the logging of samples, the code will traverse the filter_list in the YAML configuration. For each filter in the filter list, it will evaluate the metrics and output a log.

Specifically, when applied to gsm8k, the filter_list includes "strict-match" and "flexible-extract". Consequently, the number of logged samples is twice that of the original testing samples.


 



Hi! This may be a problem if you want to use the output, since the json contains two versions for each document but doesn't mention anything about the filters (if I am not missing anything). An additional field for which filter the respective outputs are, or something similar, could be useful!
Like in `evaluator.py`
```
example = {
                        ...
                        "filter_key": filter_key,
```
Repository: EleutherAI/lm-evaluation-harness
Issue:  After converting the Qwen1.8b HF model to the GGUF model, the context length dropped from 8182 to 512.
Root cause: Configuration Error

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/54474696/67dd7aee-8b99-4211-ae56-8e920d221d68)

after converting：
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/54474696/727b5be4-ea99-431d-bcb9-928933055882)
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/54474696/b45bbd90-d283-4147-a592-f466f0b8aa0a)
Repository: EleutherAI/lm-evaluation-harness
Issue: fatal: not a git repository (or any parent up to mount point /)
Root cause: Validation Gap

After run:
lm_eval --model hf \
    --model_args pretrained=./models/opt-6.7b \
    --tasks arc_easy \
    --batch_size auto:1 \
    --output_path ./eval_out/opt-6.7 \
    --use_cache ./eval_cache

it raises an error:
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

and also get a result:

hf (pretrained=./models/opt-6.7b), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:1
| Tasks  |Version|Filter|n-shot| Metric |Value |   |Stderr|
|--------|------:|------|-----:|--------|-----:|---|-----:|
|arc_easy|      1|none  |     0|acc     |0.6561|±  |0.0097|
|        |       |none  |     0|acc_norm|0.6006|±  |0.0101|

what's the matter with it?

I am also having this issue. My question is whether this would affect the result in the end. 

Hi!

This should not affect results, just whether the git repo's hash gets correctly logged to saved results json files! We'll see about making this confusing error not get logged though.
Repository: EleutherAI/lm-evaluation-harness
Issue: [TruthfulQA] update rouge-score version or add a way to suppress tokenizer logging
Root cause: External Dependency Breakage

When running truthfulqa evaluation i get a lot of unnecessary info/debug prints from the rouge scorer (usually hundreds if not thousands of lines of logging) each looking like this:
```
2024-04-09:15:38:18,284 INFO     [rouge_scorer.py:83] Using default tokenizer.
```

Eval harness has a dependency to the package rouge-score > 0.0.4
In the latest version on PyPi the Initialization of the scorer prints an info log each time it is called with the default tokenizer.
```
if tokenizer:
  self._tokenizer = tokenizer
else:
  self._tokenizer = tokenizers.DefaultTokenizer(use_stemmer)
  logging.info("Using default tokenizer.")
```

Truthfulqa uses a loop to re-initialize the scorer each time it wants to score something which leads to a lot of prints that are supposed to be debug prints but are still stuck at info level due to the outdated version of rouge-scorer (taken from [utils.py](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/truthfulqa/utils.py)).
```
rouge_scores = [rouge([ref], [completion]) for ref in all_refs]
```

That behaviour was changed 8 months ago with commit 6e232b36782739b234922e2a65e92d7c2651758b. (see history of https://github.com/google-research/google-research/blob/master/rouge/rouge_scorer.py).
The latest package of the rouge-scorer on Pypi however is from july 2022 and doesn't have these changes yet.

Is there a way to get a more updated version of rouge-scorer into the dependencies or a different way to suppress those prints or reduce the amount of them happening (like moving the initialization of the rouge scorer so it isn't called with each ietration of the loop)?

Does running with `--verbosity=WARNING` help? https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/__main__.py#L193

![grafik](https://github.com/EleutherAI/lm-evaluation-harness/assets/81179144/791dd420-ee2d-4402-b18b-30c5d5f3a1d3)
Ran with verbosity warning, but still:
![grafik](https://github.com/EleutherAI/lm-evaluation-harness/assets/81179144/5a250cf3-142e-4d23-8c29-ce7d6712feef)


I am also finding this logging output cumbersome!

Adding a fix for this in #2090
Repository: EleutherAI/lm-evaluation-harness
Issue: Is it possible to shuffle the dataset?
Root cause: Unimplemented Feature Gap

Hi. I only want to use a subset of a dataset. For that I am using --limit. But I will be repeating tests multiple times for parameter search and I don't want to overfit for to small subset so I want to take different subsets at different times. I thought changing seeds might shuffle the dataset but looking at the outputs I got by using --log_samples, it doesn't. So, how can I shuffle the dataset?

If no one comes with better idea, you can write custom process_docs with shuffling of dataset inside (new seed from environment variables, for example). Quick untested example
```python
def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:
    return dataset.shuffle(seed=int(os.environ("DATA_SEED")))
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for other languages - Translated datasets
Root cause: Unimplemented Feature Gap

Hi,

I am trying understand the best way to provide a separate LLM benchmark specifically evaluating against Turkish datasets.

This can be done by automatically translating the datasets references by tasks in `lm_eval/tasks`. This is the approach taken by

- (OpenLLMTurkishLeaderboard)[https://huggingface.co/spaces/malhajar/OpenLLMTurkishLeaderboard]

making changes to files like `lm_eval/tasks/arc/arc_easy.yaml` as below:

https://github.com/malhajar17/lm-evaluation-harness_turkish/blob/main/lm_eval/tasks/arc_tr/arc_easy.yaml

In this file, a reference to the new dataset is given. However, `doc_to_text` and `doc_to_target` fields still includes English keywords like `Question:`.

What is the best practice for supporting other languages? Can we add more datasets into arc_easy.yaml and make it use a different processing method for each different dataset?
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the CoQA evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive, multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset.
> GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI+18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions. On DROP [DWD+19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL+19]. On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to slightly outperform the best fine-tuned result in the original paper. On RACE [LXL+17], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA.

- [X] Data processing code implemented
- [ ] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Creating docs and doc->text is done for CoQA, the only thing left to complete is the evaluate function in https://github.com/EleutherAI/lm_evaluation_harness/blob/master/lm_eval/tasks/coqa.py
Repository: EleutherAI/lm-evaluation-harness
Issue: GSM8K Error result produced by gpt-3.5-turbo
Root cause: Documentation Deficiency

Thank you for your outstanding work! 
Recently, I have been trying to test the performance of GPT-3.5 turbo on gsm8k, but I have received poor results, as shown in the following figure:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/91206112/ea338777-2401-483b-8208-daddfb0fe050)

This is the command I am using, and I would like to ask, what is the reason for this?
```lm_eval --model openai-chat-completions     --model_args model="gpt-3.5-turbo"     --tasks gsm8k```

Looking forward to your help!
Repository: EleutherAI/lm-evaluation-harness
Issue: Why is stderr calculated when calculating the evaluation score?
Root cause: Documentation Deficiency

From the result summary, we can see `acc` and `acc_stderr` for each task.
Can anyone clarify how to calculate `acc_stderr`?

```
"results": {
  "arc_challenge": {
    "acc": 0.20051194539249148,
    "acc_stderr": 0.011700318050499373,
    "acc_norm": 0.21928327645051193,
    "acc_norm_stderr": 0.012091245787615723
}
```

Hey! 

`*_stderr` denotes the Standard Error calculated for metric `*`. It can be used to calculate confidence intervals for statistical significance. See https://github.com/EleutherAI/lm-evaluation-harness/issues/1524 for a more full description!

Lmk if this helps.
Repository: EleutherAI/lm-evaluation-harness
Issue: RACE dataset?
Root cause: Documentation Deficiency

Hi,

I have a quick question about the RACE dataset. It appears that the code is evaluating on the EleutherAI/RACE dataset, which contains approximately 1000 examples. However, the original RACE dataset consists of two subsets: "high" and "middle". Each has about 4k examples. I noticed that in the dataset card this is the test set for "high" subset. Can you explain why there is a mismatch here? Thanks!