zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Iinconsistent API and docs for loglikelihood_rolling
Root cause: Interface Contract Mismatch

Seems like the API for `loglikelihood_rolling` is inconsistent. The abc expects a sequence of (`loglikelihood`, `is_greedy`) tuples, but the implementations seem to return only a sequence of `loglikelihood` scalars.

The documentation at https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md is also inconsistent where the API code defines:

```python
def loglikelihood_rolling(self, requests: list[Instance]) -> list[tuple[float, bool]]:
  ...
```

while the docs suggest otherwise:

> It should return (ll,) : Tuple[float] , a.k.a. solely the loglikelihood of producing each piece of text given no starting input.

Would be great to clarify and make consistent. 

Thanks for pointing this out, I've updated the abc return type and docstring in #1532 .
Repository: EleutherAI/lm-evaluation-harness
Issue: overwrite evaluator for particular task
Root cause: Unimplemented Feature Gap

I have a task that requires context for requests. This leads to changes in evaluator as long as while processing the requests I do not have access to the answers to update the new requests so that they include "the context" - the questions and model answers for the previous ones. Now I want to use the designed evaluator instead of the original one only for this task. Is it possible to somehow overwrite the evaluator through yaml file of the task? Some kind of function that substitutes the original evaluator with my own so that I still can use the original one for other tasks (each task is launched separetely)?
Repository: EleutherAI/lm-evaluation-harness
Issue: Question about prompt formatting issue
Root cause: Unimplemented Feature Gap

Hi, thank you for such a convenient tool for evaluating llm. 

I was testing different models on my own custom tasks and find out that the performance was way lower than I expected. I logged the results and using the same prompt to do a sanity check using `Mixtral 8x7b Instruct` model. However, I found out that the results are some what different. 

I suspect that it is because that some special tokens are not included in the huggingface LMM `tok_encode` function or `tok_batch_encode` function because in these functions the strings are directly tokenized by the tokenizer. However, in lots of model, the prompt needs to apply template for example

```python
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
```

In `Mixtral 8x7b Instruct`, the special tokens `
` will be added. I wonder if there could be a feature similar to custom task that  we can add a `include_model_path` and then we inherit from the hf LLM to overwrite `tok_encode` or `tok_batch_encode` function to achieve best performance. I believe this can be more flexible for user to add new models without cloning the whole repo.

Thank you!


May be connected with
https://github.com/EleutherAI/lm-evaluation-harness/issues/1098
https://github.com/EleutherAI/lm-evaluation-harness/issues/1209
https://github.com/EleutherAI/lm-evaluation-harness/pull/1287
https://github.com/EleutherAI/lm-evaluation-harness/issues/1490


Check this PR, please, if it is good for you: https://github.com/EleutherAI/lm-evaluation-harness/pull/1578

Hi, @LSinev. Currently hardcode my chat template for models I want to test. Would also check this PR and would like to ask if this will be in the next release for pip?

I think, you can add pro and cons for particular PRs in the next release to the discussion here: https://github.com/EleutherAI/lm-evaluation-harness/issues/1560#issuecomment-1997673127
Repository: EleutherAI/lm-evaluation-harness
Issue: Questions about evaluation metrics
Root cause: Documentation Deficiency

Thanks a lot for your great work on providing the standard for evaluation. I am trying to understand a bit better the metrics used, in particular perplexity and accuracy.

I see that for perplexity calculations, there are three options: `perplexity`, `word_perplexity`, and `byte_perplexity`. I'm trying to understand how `perplexity` is averaged. Is it over the number of tokens, the number of characters, or something else entirely?

For accuracy, I have two questions:
1. Is `acc` soft or hard accuracy? I.e. per-example it can either be 0 or 1 depending on the most likely choice is the right one?
2. Similar to `perplexity`, I wonder how `acc_norm` normalized? 

Apologies if this has been asked previously, I wasn't able to find definitive answers. 

Hi!

For perplexity tasks, we typically report `word_perplexity`, `byte_perplexity`, and `bits_per_byte`, which you can find some details on in https://arxiv.org/pdf/2101.00027.pdf (word and byte perplexity normalize by the number of words (whitespace-separated) or number of bytes per string respectively). 

The `perplexity` (which as far as I'm aware is only reported in Lambada) metric is calculated by getting the loglikelihoods of every target string predicted (~one word for each lambada data point) and applying this function to average across the dataset and take exp. https://github.com/EleutherAI/lm-evaluation-harness/blob/9e6e240229429d2214bc281bed7a4e288f5169a1/lm_eval/api/metrics.py#L37-L38

`acc` is hard accuracy, and `acc_norm` takes the loglikelihoods from the multiple choice choices and divides by the length of each choice string in bytes (approach 3 in https://blog.eleuther.ai/multiple-choice-normalization/). 

Hope this helps!

Thank you. This answers it for me. I was indeed looking at Lambada, hence some of the confusion regarding `perplexity`.
Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval: error: unrecognized arguments: --wandb_args
Root cause: Environment Incompatibility

Hi, I'm getting this error when trying to run lm_eval with wandb arguments. I followed the correct installation, and can see the argument in *lm-evaluation-harness/lm_eval/__ main __.py*.

`usage: lm_eval [-h] [--model MODEL] [--tasks task1,task2]
               [--model_args MODEL_ARGS] [--num_fewshot N]
               [--batch_size auto|auto:N|N] [--max_batch_size N]
               [--device DEVICE] [--output_path DIR|DIR[/file.json](http://localhost:8888/file.json)]
               [--limit N|0<N<1] [--use_cache DIR]
               [--decontamination_ngrams_path DECONTAMINATION_NGRAMS_PATH] [--check_integrity] [--decontamination_ngrams_path DECONTAMINATION_NGRAMS_PATH]
               [--check_integrity] [--write_out] [--log_samples]
               [--show_config] [--include_path DIR] [--gen_kwargs GEN_KWARGS] [--verbosity CRITICAL|NAMS_PATH].
               [--verbosity CRITICAL|ERROR|WARNING|INFO|DEBUG]
lm_eval : error : unrecognized arguments : --wandb_args project=llm-compression-project`

But when I type *lm_eval --help*, I don't see any wandb_args arguments. The command was working few days ago. I really don't understand.

Hi! What is the current commit that you're working off of?

I'm unable to replicate this but perhaps there's some difference in our setups.

I'm working on the commit 9e6e240229429d2214bc281bed7a4e288f5169a1

The issue was related to the PATH environment variable prioritizing a different lm_eval installation (~/.local/bin).
Thanks for the incredibly quick response @haileyschoelkopf
Repository: EleutherAI/lm-evaluation-harness
Issue: Local fewshot support
Root cause: Documentation Deficiency

Hi,

Following #1224 , I managed to load my local data for my task. However, is there a way to load local fewshot examples as well? Supposed I have a file hellaswag_local/fewshot/data-00000-of-00001.arrow, how can I load them as the fewshot examples that the prompts use to sample from? Thanks

Hi, you can do this by putting them in a split of your existing local dataset, and then setting `fewshot_split` to the correct split name!

Great, thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: New Task Request: InflectionAI's Physics GRE
Root cause: Unimplemented Feature Gap

There is a new dataset of Physics GRE exams constructed by Inflection AI: https://github.com/InflectionAI/Inflection-Benchmarks?tab=readme-ov-file#physics-gre

Adding this task would be a nice addition :)

@haileyschoelkopf I am interested in contributing here by adding this task if possible.
Repository: EleutherAI/lm-evaluation-harness
Issue: Clarifying Sample Count and Evaluation Process in MultiMedQA
Root cause: Documentation Deficiency

The MultiMedQA benchmark contains a few datasets, as listed below:

dataset Splits:

# PubMedQA
Train: 450
Validation: 50
Test: 500

# MedMCQA
Train: 182,822
Validation: 4,183
Test: 6,150

#MedQA
Train: 10,178
Validation: 1,272
Test: 1,273

#MMLU-Anatomy
Test: 135
Validation: 14
Dev: 5

#MMLU-Clinical Knowledge
Test: 265
Validation: 29
Dev: 5

#MMLU-College Medicine
Test: 173
Validation: 22
Dev: 5

#MMLU Medical Genetics
Test: 100
Validation: 11
Dev: 5

#MMLU-Professional Medicine
Test: 272
Validation: 31
Dev: 5

#MMLU-College Biology
Test: 144
Validation: 16
Dev: 5

While doing the evaluation, the total number of samples is 27,680. I wanted some clarification on how the samples are being evaluated. Are we taking the training set into account?

Hi! What may be causing confusion is that for loglikelihood-based multiple choice, the total number of requests is **multiplied by the number of choices per document**, so e.g. MMLU's progress bar will show 4 x number of total MMLU documents. 

This is because we have to perform one LM call per target answer string to calculate its conditional loglikelihood. For cases like MMLU where each target string is only a single token, under the hood we only perform 1 LM call per document (and get each target's loglikelihood from the logit of the respective token in the vocab), but still display the progress bar as in the typical loglikelihood fashion with number of choices times documents shown. 

Thank you, Got it, here is for more clarification.

PubMedQA Test Set Multiplied by 3: (as three options Yes, No, Maybe)
500 * 3 = 1,500

MedMCQA Validation Set Multiplied by 4 (correcting the instruction):
4,183 * 4 = 16,732

For the rest of the datasets (Test Sets Multiplied by 4):

MedQA:
1,273 * 4 = 5,092

MMLU-Anatomy:
135 * 4 = 540

MMLU-Clinical Knowledge:
265 * 4 = 1,060

MMLU-College Medicine:
173 * 4 = 692

MMLU Medical Genetics:
100 * 4 = 400

MMLU-Professional Medicine:
272 * 4 = 1,088

MMLU-College Biology:
144 * 4 = 576

Total with the corrected specification:

1,500 (PubMedQA) + 16,732 (MedMCQA Validation) + 5,092 (MedQA) + 540 (MMLU-Anatomy) + 1,060 (MMLU-Clinical Knowledge) + 692 (MMLU-College Medicine) + 400 (MMLU Medical Genetics) + 1,088 (MMLU-Professional Medicine) + 576 (MMLU-College Biology) = 27,680
Repository: EleutherAI/lm-evaluation-harness
Issue: Whitespace before label in MultipleChoiceTask causes wrong label probability prediction
Root cause: Architectural Constraint

Hi,

When constructing the requests for a MultipleChoiceTask (pre-refactored version), an implicit white space is added before the labels:
```
def construct_requests(self, doc, ctx):
      lls = [
          rf.loglikelihood(ctx, " {}".format(choice))[0] for choice in doc["choices"]
      ]

      return lls
```
When using fine-tuned models for evaluation, which expect a specific format, the added white space causes the wrong prediction of label probabilities. How should I handle this for my custom fine-tuned llms?
Repository: EleutherAI/lm-evaluation-harness
Issue: How to use generate_until function for the multiple_choice problem.
Root cause: Documentation Deficiency

I want to test the chatgpt model on the multiple_choice problem. However, ChatGPT only supports the generate_until function instead of the loglikelihood. Is there a way to use generate_until function for the multiple_choice problem?

Please take a look at the discussion https://github.com/EleutherAI/lm-evaluation-harness/issues/1196#issuecomment-1948246171
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue in gsm8k and inability to access Huggingface
Root cause: Validation Gap

I've encountered an issue when running the provided usage example on both versions 0.4.0 (main branch) of your project. The example command I'm using is:
```python
accelerate launch -m lm_eval --model hf \
    --model_args pretrained=/mnt/workspace/wujk/models/zephyr-7b-sft-full \
    --tasks hellaswag \
    --output_path output/zephyr-7b-sft-full \
    --batch_size 16
```
However, this results in a FileNotFoundError as shown below:

FileNotFoundError: Couldn't find a module script at /models/software/lm-evaluation-harness/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.

It seems likely that the problem was caused by my inability to access Huggingface, leading to the failure to download the 'exact_match' module. After fixing by https://github.com/EleutherAI/lm-evaluation-harness/issues/1071#issuecomment-1847346533 , a new error occurs at gsm8k datasets:
```python
accelerate launch -m lm_eval --model hf \
    --model_args pretrained=/mnt/workspace/wujk/models/zephyr-7b-sft-full \
    --tasks gsm8k \
    --output_path output/zephyr-7b-sft-full \
    --batch_size 16
```

ValueError: Evaluation module inputs are missing: ['predictions1', 'predictions2']. All required inputs are ['predictions1', 'predictions2']



@haileyschoelkopf 

I have addressed it by setting `export HF_ENDPOINT=https://hf-mirror.com`. I will close the issue.

Hi, I had the same problem, was the problem solved?
Repository: EleutherAI/lm-evaluation-harness
Issue: get_task_dict should raise an error on incorrect types
Root cause: Validation Gap

This is mostly an issue for QoL enhancement that would've saved me a lot of debugging time. 

When calling `get_task_list`, I was passing in what I thought was a List[str], expecting to get a populated dict back, but ended up getting an empty dict:

<img width="840" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/8039770/5f7e7f9b-4c70-41a4-acd0-288f3afa318f">

This ultimately turned out to be an issue on my end, as the first list was actually a `class 'omegaconf.listconfig.ListConfig` that I failed to instantiate properly. Although, raising error on incorrect type in `get_task_list` instead of returning empty dict would have helped root cause the bug much faster in this particular case. thanks for your consideration!




Thanks for reporting this! Will take a look.
Repository: EleutherAI/lm-evaluation-harness
Issue: `--cache_requests` sometimes errors when run with `accelerate launch`
Root cause: Resource Mishandling

being tracked in https://github.com/EleutherAI/lm-evaluation-harness/pull/1372#issuecomment-1989902849 but creating an issue for this. Once it's closed, will likely push version v0.4.2 to PyPI (#1560 .)
Repository: EleutherAI/lm-evaluation-harness
Issue: Expose Configuration Options for Perplexity calculations
Root cause: Unimplemented Feature Gap

At some point we should add options for PPL calculations, such as the ability to determine whether the windows overlap with each other, and .

Note that this heavily impacts the speed / compute required for evaluations in addition to scores.


Preferred implementation approach:
- convert the `generation_kwargs` TaskConfig field to `request_kwargs`, leaving back-compatibility for providing `generation_kwargs` if `request_kwargs` is not in a given config
- allow `loglikelihood_rolling` requests to take these request_kwargs, and implement new helper functions different from the current creation of context windows for loglikelihood rolling, and add unit tests for these helper fns
- add some example task using these options
Repository: EleutherAI/lm-evaluation-harness
Issue: How can i evaluate the local model
Root cause: Documentation Deficiency

I want to evaluate my local LLM, Does the following command work?
```
python lm-evaluation-harness/main.py \
      --model hf \
      --model_args pretrained=$LOCAL_MODEL_PATH \
      --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
      --no_cache \
      > "mlt.log" 2>&1
```

this command should not work, because there is no main.py in lm-evaluation-harness  directory 

local models can be run, check documentation

i work with the code
```    
model = AutoModelForCausalLM.from_pretrained(
        model_path,
        low_cpu_mem_usage=True,
        trust_remote_code=True,
        torch_dtype=torch.bfloat16,
    ).to(device)
    tokenizer = AutoTokenizer.from_pretrained(
        model_path,  trust_remote_code=True,)
    print(model)
    lm = HFLM(pretrained=model, tokenizer=tokenizer, dtype=torch.bfloat16, max_length=tokenizer.model_max_length,
              batch_size=batch_size, trust_remote_code=True)
```
```
       results = lm_eval.simple_evaluate(
            model=lm,
            tasks=["mmlu", "gsm8k", "triviaqa"],
            task_manager=lm_eval.tasks.TaskManager(),)
```

> this command should not work, because there is no main.py in lm-evaluation-harness directory
> 
> local models can be run, check documentation

thanks, i solved the problem

> i work with the code
> 
> ```
> model = AutoModelForCausalLM.from_pretrained(
>         model_path,
>         low_cpu_mem_usage=True,
>         trust_remote_code=True,
>         torch_dtype=torch.bfloat16,
>     ).to(device)
>     tokenizer = AutoTokenizer.from_pretrained(
>         model_path,  trust_remote_code=True,)
>     print(model)
>     lm = HFLM(pretrained=model, tokenizer=tokenizer, dtype=torch.bfloat16, max_length=tokenizer.model_max_length,
>               batch_size=batch_size, trust_remote_code=True)
> ```
> 
> ```
>        results = lm_eval.simple_evaluate(
>             model=lm,
>             tasks=["mmlu", "gsm8k", "triviaqa"],
>             task_manager=lm_eval.tasks.TaskManager(),)
> ```

thaks for your answer

Closing as this seems to be answered!
Repository: EleutherAI/lm-evaluation-harness
Issue: I modified the structure of the llama, and I want to evaluate this model, how should I proceed?
Root cause: Documentation Deficiency

My computational graph is different from the original llama, how should I change it to successfully evaluate my own model?


accelerate launch -m lm_eval --model hf \
    --model_args pretrained=${model_path},trust_remote_code=true,dtype=bfloat16 \
    --tasks lambada \
    --batch_size auto \
    --output_path ./eval_out/${model_id}/lambada 

the model_path do not have the my_model.py

Hi, we support passing in pre-initialized HF models into the HFLM initialization—that would be one option!

Another would be to make use of HF’s trust_remote_code feature and bundle the modeling code of your model with your saved checkpoint.



> Hi, we support passing in pre-initialized HF models into the HFLM initialization—that would be one option!
> 
> Another would be to make use of HF’s trust_remote_code feature and bundle the modeling code of your model with your saved checkpoint.

Hello, thank you for your contribution! 
Is there any example or tutorial about the implementation of "passing in pre-initialized HF models into the HFLM initialization"?
That would be grateful for a novice like me.  ：）

@Ga-Lee  see `docs/interface.md` for a pseudocode sample skeleton script showing how to first initialize your model, then pass it into `simple_evaluate()` or `evaluate()`!
Repository: EleutherAI/lm-evaluation-harness
Issue: What it the origin of Flan-COT prompt?
GitHub labels: asking questions

Hi, 
Thanks for the nice framework.

I'm using MMLU Flan-CoT task but could not find the exact origin of the CoT prompt.
Where does it come from? 

Thanks.

cc @lintangsutawika who implemented this task and has experience working with FLAN's datasets.

@HonoMi got them from https://github.com/jasonwei20/flan-2
It's mentioned in Appendix D.1 of the [Scaling Instruction-Finetuned Language Models](https://arxiv.org/abs/2210.11416) paper.
Repository: EleutherAI/lm-evaluation-harness
Issue: 0.4.1 release does not contain TaskManager
Root cause: Unimplemented Feature Gap

0.4.1 release notes have

> A new TaskManager object and the deprecation of lm_eval.tasks.initialize_tasks(), for achieving the easier registration of many tasks and configuration of new groups of tasks

But neither code attached to the release notes, nor the pypi package contain TaskManager.

Could you please either release a fix or verify, that it is present in the new release? 

Closed: TaskManager was announced to be in the release after 0.4.1
Repository: EleutherAI/lm-evaluation-harness
Issue: (Question) How can I fully utilize the number of cores in my CPU?
Root cause: Documentation Deficiency

My current hardware configuration consists of an 8-core CPU with 32 GB of RAM. When I run the lm-eval command-line script for inference on the CPU, I observe that only a single core is being utilized (as indicated by %CPU reaching 100% in the top utility). However, when using the simple_evaluate() API within Python, I find that all CPU cores are fully utilized (with my %CPU showing up to 800%). Therefore, I would like to inquire about how I can best leverage all of my CPU cores while running the lm-eval command through the command line. Thank you very much！！
Repository: EleutherAI/lm-evaluation-harness
Issue: Regarding the question why hellaswag 0 shot has a higher score than 10 shot
Root cause: Documentation Deficiency

When I use the model to test hellaswag, 0 shot can achieve 50 points, but using 10 shot can only achieve 20 points. Why can few-shot affect the model? Thanks.

Hi, it's difficult to say reasons for this without more info. This can at times happen depending on the model, although a 30-point decrease seems unexpected. If you trained your model specifically to handle shorter inputs or with data in a zero-shot format that could influence this.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add duplication eliminating module
Root cause: Unimplemented Feature Gap

The following usecases should be in mind for the refactor:

- It should actually not be a pile of spaghetti
- It should be possible to implement modules that consider that one LM query may be usable to answer multiple requests - for example, if the same query is repeated, or if there are two loglikelihood queries that each have the same context and different single-token-long-responses (there are a *load* of these - anything that continues with " yes" and " no" fits the bill). While it would be a violation of the abstraction to actually look at how many tokens the continuation is, it still makes sense to aggregate stuff with the same context and let LM know somehow that these are potentially-optimizable - and even it it's not single-token continuation, it might still be possible to cache the context or something. (this will require moderate changes to LM interface) I'm thinking tentatively maybe a flag it can pass along per-request to say "hey, this is something that should be cached for x times (evaluator counts how many times it expects it to be reused)" so only the things that get reused actually get cached, and get evicted when no longer necessary. This does have the chance that a new LM impl might not handle exactly the same and get out of sync on the count, but that just introduces inefficiency rather than breaking anything. Any better idea proposasls are welcome.

On second thought, the second point would probably depend on `LM` internals. Probably best to implement it as a utility that LM impls can build off.