zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Qwen's zero-shot evaluation results vary greatly from the Qwen technical report.
Root cause: Documentation Deficiency

Hello, I conducted an evaluation of Qwen-7B using the latest code, but the results of the evaluation show a significant discrepancy from those in Qwen's technical report, as detailed in the figure below. Is this result to be expected?
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/73322319/382e28bb-f91d-40b4-a552-c33173696f1b)


Hi, I’ll look into this. Have you gotten any results on 3-shot BBH or on MMLU as well?

> Hi, I’ll look into this. Have you gotten any results on 3-shot BBH or on MMLU as well?

I have only tested 5-shot MMLU on Qwen-7B, and the result I obtained is 56.87, whereas the technical report for Qwen cites 58.2.

Unfortunately, Qwen does not seem to provide their prompts or eval code for ARC, as best as I could tell: https://github.com/QwenLM/Qwen/tree/main/eval

so it's difficult to say why the scores differ. If they used a task description (e.g. "Answer the following questions:") this often significantly boosts zero-shot performance.

I'm able to fairly closely match LAMBADA accuracy with Qwen-1.8B here.


EDIT: I see actually that Qwen mentions using OpenCompass, so I will try to see if I can look at how OpenCompass does ARC when I can.
Repository: EleutherAI/lm-evaluation-harness
Issue: Organize / Cleanup Logging + Levels
GitHub labels: documentation

feature request

We should do a pass through the codebase and unify log printing styles + re-check log levels. 

Ideally a novice user should only see WARNING levels if they are doing something unintended / footgun-y and INFO levels should (theoretically) be ignorable if you're not a power user.  DEBUG should refer to logs that are useful when trying to add a new task and not useful to see every run otherwise. We should try not to print *too* many logs in the default setting, except where we think it helps the average user (there should be an easily-accessible "almost-silent" mode.)

Listing some changes that pop out if I run into them.

- `lm_eval/api/task#L701` should get a more informative log message? 
- Phase out `--write_out` in favor of just logging some sample instances instead when using DEBUG log level?
Repository: EleutherAI/lm-evaluation-harness
Issue: Group Alias issue
Root cause: Architectural Constraint

A group alias of another task is used instead of the actual group name #1198

Potential solution: have the benchmark YAML config, rather than individual tasks' configs, contain `task_aliases` and `group_alias` fields for how to print its subtasks when they are part of the group (task_alias) and when they are part of a larger group (group_alias) respectively?

Since task_alias and group_alias will be dependent on which group they are called as part of.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the HendrycksTest evaluation
Root cause: Unimplemented Feature Gap

> We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

https://arxiv.org/abs/2009.03300
Repository: EleutherAI/lm-evaluation-harness
Issue: help with multi-gpu run
Root cause: Configuration Error

I am trying to do an eval of a 13B model using accelerate launch command. However, the model does not split across the gpus, and gives me a OOM error. Any help?

Command:
```
accelerate launch --no_python lm_eval --model hf --batch_size 8 --model_args pretrained=<a_13B_model> --tasks truthfulqa,mmlu,arc_easy,arc_challenge,hellaswag,winogrande
```

Accelerate Config.
```
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: true
  fsdp_offload_params: true
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

We've had some problems with accelerate's FSDP and generation--I'd recommend trying vLLM with tensor parallelism, or if that still doesn't fit on your GPUs, then to use

 `lm_eval --model hf --batch_size 8 --model_args pretrained=<a_13B_model>,parallelize=True --tasks truthfulqa,mmlu,arc_easy,arc_challenge,hellaswag,winogrande`

Note also you may need to reduce the batch size for things to fit.

Thanks ! It works great !
Repository: EleutherAI/lm-evaluation-harness
Issue: Documentation about tasks <> request types mapping
Root cause: Documentation Deficiency

This table in the readme outlining which request type is supported for each API is super useful.
<img width="841" alt="Screenshot 2023-12-22 at 11 06 07" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/1911631/7d4b4b39-633f-4f41-be8c-264543edfe59">

However, as a user I usually want to run a set of benchmarks (tasks) against a set of APIs. Not knowing how the tasks map to request types means I don't know until I actually try to run the benchmark whether or not the run is going to succeed.

For example, running hellaswag with openai chat completions fails because `loglikelihood` and `loglikelihood_rolling` are not implemented due to lack of support for logits. 

Will complete as part of #1160 ! This is very useful feedback.
Repository: EleutherAI/lm-evaluation-harness
Issue: Incompatible argument for openai-completion in bbh_cot_fewshot
Root cause: Interface Contract Mismatch

https://github.com/EleutherAI/lm-evaluation-harness/blob/9fb2ebababd78986f445cbe10e97c499213c407a/lm_eval/tasks/bbh/cot_fewshot/_cot_fewshot_template_yaml#L15

```
TypeError: create() got an unexpected keyword argument 'max_gen_toks'
```

replication cmd:
```
lm_eval --model openai-completions --model_args model=gpt-3.5-turbo-instruct --tasks bbh_cot_fewshot_causal_judgement --log_samples --output_path results/gpt_3.5_turbo_instruct_bbh_cot_fewshot_causal_judgement
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the ETHICS evaluation
Root cause: Unimplemented Feature Gap

> We show how to assess a language model's knowledge of basic concepts of morality. We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. Models predict widespread moral judgments about diverse text scenarios. This requires connecting physical and social world knowledge to value judgements, a capability that may enable us to steer chatbot outputs or eventually regularize open-ended reinforcement learning agents. With the ETHICS dataset, we find that current language models have a promising but incomplete ability to predict basic human ethical judgements. Our work shows that progress can be made on machine ethics today, and it provides a steppingstone toward AI that is aligned with human values.

https://arxiv.org/abs/2008.02275

I'll work on this one tomorrow
Repository: EleutherAI/lm-evaluation-harness
Issue: how to eval the megatron-lm
Root cause: Unimplemented Feature Gap

I'm tring to evaluate a model pretrained with megatron-lm, and I have tried to convert it into huggingface format first, but it seems that there maybe some bugs I haven't found out because when I used the same prompt in GSM8K to infer the megatron-format model, I got totally different results. So I'm wondering if there's one way to directly use megatron-lm to run lm-eval

I don’t believe there is one at the moment, but there is an integration with gpt-neox (supports current version, maintained by us): https://github.com/EleutherAI/gpt-neox/blob/main/eval_tasks/eval_adapter.py

and a rather custom one in NeMo created by nvidia:
https://github.com/NVIDIA/NeMo-Megatron-Launcher/tree/master/launcher_scripts/nemo_launcher/collections/eval_harness

Having one for megatron-lm would be quite nice and I’d be happy to review one if so, though we won’t be prioritizing it ourselves any time soon, sorry!


> I don’t believe there is one at the moment, but there is an integration with gpt-neox (supports current version, maintained by us): https://github.com/EleutherAI/gpt-neox/blob/main/eval_tasks/eval_adapter.py
> 
> and a rather custom one in NeMo created by nvidia: https://github.com/NVIDIA/NeMo-Megatron-Launcher/tree/master/launcher_scripts/nemo_launcher/collections/eval_harness
> 
> Having one for megatron-lm would be quite nice and I’d be happy to review one if so, though we won’t be prioritizing it ourselves any time soon, sorry!

Thanks a lot! I will check the demos in the links later and wish you a happy new year!
Repository: EleutherAI/lm-evaluation-harness
Issue: Promblems concerning Mixtral evaluation
Root cause: External Dependency Breakage

When testing Mixtral 8x7b with 2xA100-80G GPUs, I employed the model vllm with its most recent package (vllm 0.2.6). However, I encountered a CUDA Out Of Memory (OOM) error, even when setting the batch size to 1. I came across <a herf="https://github.com/QwenLM/Qwen/issues/819">https://github.com/QwenLM/Qwen/issues/819</a> which suggests that the error might stem from the latest update of vllm. Subsequently, downgrading vllm to version 0.2.5 resolved the issue, allowing for error-free inference. You may update your code to support the updated version of vllm.

Thanks for the report!

Thanks ！ help a lot ！ So, why there's error? is the issue with vllm?

When I ran MMLU with vLLM 0.2.7, I found the it leads to OOM even for a 7B model when I set batch_size='auto'. Either adjusting batch_size to 1 or using a small gpu_memory_utilization value such as 0.6 could solve this OOM problem, but it tends to be slower.

After some debugging, I found it is related to the generation parameters prompt_logprobs=2. Whenever we enable this feature, we have a higher chance of OOM. For a normal generation task without prompt_logprobs, I can even run a llama2-70b without issues.

Not sure how frequently we use this prompt_logprobs feature, but for tasks like MMLU we can actually use logprobs instead of prompt_logprobs.
Repository: EleutherAI/lm-evaluation-harness
Issue: Found a bug
Root cause: Configuration Error

<img width="297" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/79445290/7c801f88-64a5-4df8-b409-359bc05d18eb">

The configuration file of siqa has to be changed into ``default.yaml``, otherwise this task can not be recognized.
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluating Arc Easy Got NonMatchingSplitsSizesError
Root cause: External Dependency Breakage

Hi, when evaluating with arc easy challenge with the script below:
```
lm-eval \
    --model hf \
    --model_args trust_remote_code=True,pretrained=$ckpt \ 
    --tasks arc_easy \
    --num_fewshot 0 \
    --device cuda:0 \
    --output_path "./eval_scripts/arc_easy.json"
```
I got `datasets.utils.info_utils.NonMatchingSplitsSizesError`.
I've noticed that the arc_easy repo in Huggingface has been updated 5 days ago. Are there some problems with it?
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/36473484/216b128d-538c-4179-9461-3b5c34e37115)


I'm not getting this. Maybe try clearing the dataset in the cache and try again?

I updated my `datasets` package from 2.12.0 to 2.16.0 and the issue disappeared. Perhaps this should be added to the dependencies. Thanks anyway.

We may need to pin datasets to >= 2.16.0 or 2.17.0 following #1135 .


Probably other HF datasets will also be migrated in the backend in a similar way. It shouldn’t change their contents though, HF is just phasing out dataset loading scripts.

This dataset used to be defined using a dataset script and we recently converted it to Parquet to enable the `datasets` security features. However because of this change `datasets` 2.14 is now needed to load this dataset, sorry for the inconvenience.

Alternatively it's possible to load the old version of this dataset (with the dataset script) by specifying its old revision before the change, see at https://huggingface.co/datasets/allenai/ai2_arc/commits/main)
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the GenericsKB evaluation
Root cause: Unimplemented Feature Gap

> We present a new resource for the NLP community, namely a large (3.5M+ sentence) knowledge base of *generic statements*, e.g., "Trees remove carbon dioxide from the atmosphere", collected from multiple corpora. This is the first large resource to contain *naturally occurring* generic sentences, as opposed to extracted or crowdsourced triples, and thus is rich in high-quality, general, semantically complete statements. All GenericsKB sentences are annotated with their topical term, surrounding context (sentences), and a (learned) confidence. We also release GenericsKB-Best (1M+ sentences), containing the best-quality generics in GenericsKB augmented with selected, synthesized generics from WordNet and ConceptNet. In tests on two existing datasets requiring multihop reasoning (OBQA and QASC), we find using GenericsKB can result in higher scores and better explanations than using a much larger corpus. This demonstrates that GenericsKB can be a useful resource for NLP applications, as well as providing data for linguistic studies of generics and their semantics. GenericsKB is available at this https URL.

https://arxiv.org/abs/2005.00660

This doesn't technically come with a task but we can make one just by looking at, say, bits-per-byte on the data.
Repository: EleutherAI/lm-evaluation-harness
Issue: CoQA's implementation only predicts the last answer of each text
Root cause: Algorithmic Error

For CoQA, in coqa/utils.py, only the last answer of each text (i.e. the answer for the last turn_id, with all the previous questions and answers in the context window) is predicted. On the website of the authors of CoQA, they seem to consider all turn_id (see their [sample prediction file](https://nlp.stanford.edu/data/coqa/drqa-pgnet-coqa-dev-hist1.txt.json) where they have answers for every turn_id, and the official [evaluation script](https://nlp.stanford.edu/data/coqa/evaluate-v1.0.py) where they do an average of the result for every turn_id).

Here is an excerpt from the paper (https://arxiv.org/pdf/1808.07042.pdf) : 

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/47578089/5eb6f6ea-17bd-4ce3-8e6c-15a98d86ca2d)

I haven't found how it's implemented with other popular LLM evaluation frameworks, but I'm pretty sure that predicting only the answer to the last question is not what is intended by the authors of CoQA.

I think it probably makes sense to support both versions of this task, but we should make it clear that the one described in the OP is the official one.
Repository: EleutherAI/lm-evaluation-harness
Issue: Only a single `filtered_resps` is logged for repeat > 1 for each sample
Root cause: Validation Gap

For repeat > 1, model outputs (`resps`) for each model call are saved to file when using `--log_samples` but only a single `filtered_resp` (the first one?).

Is this independent of the take-first filter being the default setting?

This was without setting an explicit filter in the yaml while testing `predict_only`. Also doesn't just picking the first generation defeat the whole point of repeat? Why is it even a thing. I'm probably missing something obvious. 

repeat is a thing for the purpose of things like Maj@K / self-consistency--if someone wants to use this, they'll set `repeats: K` and explicitly set some other filter, such that `filtered_resps`  will end up with one response, namely the most-frequently outputted answer.

The idea is that `resps` contains all `K` outputs from the model, and the filter pipelines distill it into a single output that can be scored. (or in multiple_choice's case, a single set of `N_CHOICES` outputs.) This could plausibly be improved or made more easy to understand in the output files for sure and maybe we do want a different default behavior for `repeats > 1`.

Aah that makes more sense. Wouldn't pass@k be more appropriate for the default? IMO that's a bit more intuitive ( + warning if just greedy sampling).


I am also encountering this issue. More generally, I want to evaluate a sample multiple times and automatically average the results of these evaluations. Can adding a new ```filter``` achieve this requirement?

I also want to know

Hi! @StellaAthena @haileyschoelkopf @yanrudan @baberabb  I just encountered the same issue today when evaluating with lm_eval + vllm, and I noticed there is a critical bug [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/69ecd0b929701d346c1119d5cd0563ade4ab3536/lm_eval/models/vllm_causallms.py#L674).

Snippet:
```python
cont = self._model_generate(
    requests=context_encoding_truncated,
    generate=True,
    sampling_params=sampling_params,
)

# cache generations
for output, context in zip(cont, context):
    generated_text: str = output.outputs[0].text
    # use secondary stop seqs to cut off should-have-been-stopped content post-hoc
    generated_text = postprocess_generated_text(
        generated_text, until, self.think_end_token
    )
    res.append(generated_text)
    self.cache_hook.add_partial(
        "generate_until", (context, gen_kwargs), generated_text
    )
    pbar.update(1)
```

Specifically, when you set `n` in `gen_kwargs` to some number > 1 (say 8), cont contains multiple generations, and `output.outputs` is a list of length 8. However, lm_eval only takes the first generated_text. I think that's probably why the issue mentioned above happened.


Additionally, there have been reports in vLLM that `n>1` generation fail. See [here](https://github.com/vllm-project/vllm/issues/17173) and [here](https://github.com/vllm-project/vllm/pull/16863). But I believe this issue has been fixed since v0.8.5. I don't know if other model classes (other than `vllm_causallm`) have the same problem.
Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
Root cause: Validation Gap

When I use vllm to evaluate a model on dataset "ceval-valid", at the final stage of the program's execution, this error occurred. 
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/97608046/d4443590-804f-45b9-86a9-49aa3d50158b)

I suspect it's because the `logprob_dict.get()` function is returning `NoneType`. However, I'm unsure how to modify the code to correctly avoid this issue. Moreover, when using vllm to evaluate other models and datasets, I encounter this error with some, while others execute correctly and produce evaluation results.
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/97608046/999b1c81-2894-4105-a022-17374dc4d57d)


I tried this with `pythia-410m` on `ceval-valid` but couldn't reproduce this. Might be a tokenizer issue? What model are you using? Looks like the token is either being truncated when we index out the answer or there might be a bug with how we index the log-probabilities.
I think something like 
```python
        try:
            continuation_logprobs = sum(
                logprob_dict.get(token)
                for token, logprob_dict in zip(
                    tokens[ctxlen:], continuation_logprobs_dicts[ctxlen:]
                )
            )
        except TypeError:
            print(f"ctxlen: {ctxlen}")
            print(f"Tokens: {tokens}")
            print(f"outputs: {outputs}")
```
would be an easy way to debug.

> I tried this with `pythia-410m` on `ceval-valid` but couldn't reproduce this. Might be a tokenizer issue? What model are you using? Looks like the token is either being truncated when we index out the answer or there might be a bug with how we index the log-probabilities. I think something like
> 
> ```python
>         try:
>             continuation_logprobs = sum(
>                 logprob_dict.get(token)
>                 for token, logprob_dict in zip(
>                     tokens[ctxlen:], continuation_logprobs_dicts[ctxlen:]
>                 )
>             )
>         except TypeError:
>             print(f"ctxlen: {ctxlen}")
>             print(f"Tokens: {tokens}")
>             print(f"outputs: {outputs}")
> ```
> 
> would be an easy way to debug.

I use the model `meta-llama/Llama-2-13b-hf`, can you get it? And the model `wenge-research/yayi-13b-llama2` also get the same problem on ceval-valid.  Oh, sorry to emphasize, it only appears in 5-shot, 0-shot is normal
Repository: EleutherAI/lm-evaluation-harness
Issue: Did you abandon the support for 'hf-causal'?
Root cause: Documentation Deficiency

Hi, thanks for your great contributions!
I am trying to evaluate Llama2-7b on your repo.
I run the command:
`lm_eval --model hf-causal  --model_args pretrained=/path/to/llama2-7b  --tasks lambada_standard   --device cuda:0  --batch_size 16`
However, I got an error:
`ValueError: Attempted to load model 'hf-causal', but no model for this name found! Supported model names: hf-auto, hf, huggingface, openai-completions, openai-chat-completions, local-chat-completions, textsynth, dummy, anthropic, gguf, ggml, vllm, mamba_ssm`
What should I do?

try using hf-auto instead of hf-causal

`hf-causal` has been renamed (and `hf-causal-experimental` has been deprecated) to be usable through any of `hf`, `hf-auto`, `huggingface`. This can be used for both causal models and seq2seq models.