zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Local API, Getting "No support for logits". How to run command line with generate_until only?
Root cause: Documentation Deficiency

I'm trying to run this against a local instance of oobabooga exposing openai-like API:

```
OPENAI_API_KEY=x lm_eval --model local-chat-completions --model_args model=mymodel,base_url=http://127.0.0.1:5000/v1 --tasks lambada_openai
```

I'm getting

```
  File "/home/lxe/lm-evaluation-harness/lm_eval/models/openai_completions.py", line 499, in loglikelihood
    raise NotImplementedError("No support for logits.")
NotImplementedError: No support for logits.
```

I see in the docs that `loglikelihood` request type isn't supported, but I can't find anywhere in the docs about how to configure the harness to only do `generate_until` using the command line interface. 



Did you manage to solve the problem?

https://github.com/EleutherAI/lm-evaluation-harness/pull/1174#issuecomment-1867113258

Is it now possible to run tests on self-hosted llm instances that either do or don't return logits?
I'm getting the same error and can't figure out how to advance.
Repository: EleutherAI/lm-evaluation-harness
Issue: Speed up openai API calls 
Root cause: Unimplemented Feature Gap

I'm trying to run openai API on gsm8k like this` lm_eval --model openai-chat-completions --model_args model=gpt-3.5-turbo-0125 --tasks gsm8k`. But I've noticed the speed of calling API is so slow. Is there a way to use queues/not wait for API to return here?

I often use this example openai parallel processing code with great results:

https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py

We could implement similar here? Would require the user to input their own token and request rate limits to get the best out of it

> I often use this example openai parallel processing code with great results:
> 
> https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py
> 
> We could implement similar here? Would require the user to input their own token and request rate limits to get the best out of it

Adding this to our openai implementation would be awesome!

> > I often use this example openai parallel processing code with great results:
> > https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py
> > We could implement similar here? Would require the user to input their own token and request rate limits to get the best out of it
> 
> Adding this to our openai implementation would be awesome!

I can take a stab when I have a minute!
Repository: EleutherAI/lm-evaluation-harness
Issue: loglikelihood for local-completions using vllm?
Root cause: External Dependency Breakage

Hi, I'm trying to run local-completions using a openai compatible vllm server.

for the [oa_completion function](https://github.com/EleutherAI/lm-evaluation-harness/blob/756eeb6f0aee59fc624c81dcb0e334c1263d80e3/lm_eval/models/openai_completions.py#L43), and the following request in  _loglikelihood_tokens,  

```python
 response = oa_completion(
                client=self.client,
                model=self.model,
                prompt=inps,
                echo=True,
                max_tokens=0,
                temperature=0.0,
                logprobs=10,
                seed=self.seed,
            )
```
i get an  '500 Internal Server Error' because vllm does not support the "echo" variable but only support prompt_logprobs, which i cannot send through the openai api. 

Are there any suggestions on how i could fix this problem?

`vllm` does support [echo](https://github.com/vllm-project/vllm/blob/3711811b1d2956e83e626c72f0e1607f2dfbc8fb/vllm/entrypoints/openai/protocol.py#L118)! They added it in `0.2.4`. I think you'll also need to set `max_tokens`= 1 + `prompt_logprobs` = 1 (don't think you need `logprobs` here)

thanks for the kind reply. I've been working on this and it seems that echo=False works fine but echo=True raises an error. I think this is more of an VLLM issue, so I'll be closing the issue here.
Repository: EleutherAI/lm-evaluation-harness
Issue: How do I specify `generation_kwargs` in python `Task`s?
Root cause: Documentation Deficiency

Hi all! I defined some new tasks from scratch purely in python by inheriting `Task`, e.g., https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard/blob/main/src/backend/tasks/cnndm/task_v2.py#L63

How do I specify which `generation_kwargs` to use? I'd like to use the following ones:

```yaml
generation_kwargs:
  until:
    - "\n"
    - "."
    - ","
  do_sample: false
  temperature: 0.0
```

I'm asking because in some cases I'm getting the following:

```
ValueError: `temperature` (=0.0) has to be a strictly positive float, otherwise your next token scores will be invalid. If you're looking for greedy decoding strategies, set `do_sample=False`.
```

Looks like you can set them in `construct_requests`. Pass them as a dict to index 1 of the `arguments` tuple. 

@baberabb thanks! giving it a try

@lintangsutawika @haileyschoelkopf  shall we add this to SQuADv2 and Scrolls as well? these have no `generation_kwargs` atm

Here it is:

```python
    def __init__(self):
        super().__init__(config={'metadata': {'version': self.VERSION},
                                 'generation_kwargs': {'do_sample': True, 'temperature': 5.0}})
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Current recommended way to get weight avg MMLU score
Root cause: Documentation Deficiency

I know there are many discussions and evolutions around mmlu score in this repo, but it is unclear to me what's the latest recommended way to get a single score for weighted avg mmlu.

In the past I used to leverage this [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/922) in the big-refactor branch. But since that branch is merged into main I have switched to main and latest main does not have this avg score in the reporting([current](https://github.com/EleutherAI/lm-evaluation-harness/pull/922#issuecomment-1935897037) vs. [old](https://github.com/EleutherAI/lm-evaluation-harness/pull/922#issuecomment-1790173897)). 

I am wondering what's the latest way of doing this.



Thanks. Recently fixed by #1414

@lintangsutawika thanks for the quick action.  I can confirm it is working now.
Repository: EleutherAI/lm-evaluation-harness
Issue: The batch size parameter is invalid in dataset gsm8k.
Root cause: Validation Gap

python main.py     --model hf    --model_args pretrained=Qwen-1_8B-Chat-judge-alpaca --tasks gsm8k     --device cuda:0  --batch_size 64 --num_fewshot=0

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/76865636/854e1dec-017b-40b8-933d-0913ffa937a1)


Hi, thanks for opening this—we’ll check it out!

Hi, I got the same problem. It took 15 hours to finish the evaluation of GPT-J-6B with single A100 GPU.

I used the version (b281b09) described in OpenLLM Leaderboard.

Do you know how to solve this problem?

> Hi, I got the same problem. It took 15 hours to finish the evaluation of GPT-J-6B with single A100 GPU.
> 
> Do you know how to solve this problem?

I had problem with the version by huggingface llm leader board.
https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463
![Screenshot_2024-02-20-09-24-57-144_com opera browser](https://github.com/EleutherAI/lm-evaluation-harness/assets/76865636/b543d255-ffc8-4cd1-b3d8-7e46fc86c13f)


Just realized the comment opening this issue is about the HF leaderboard commit. 

On the `main` branch, batching is fully supported for all request types in Huggingface models! 

On the Open LLM Leaderboard commit, `--model hf-causal-experimental` supports batching better than `hf-causal`.
Repository: EleutherAI/lm-evaluation-harness
Issue: lm-evaluation-harness computes MMLU results differently than original OpenAI/hendrycks method
Root cause: Documentation Deficiency

Thank you to the EleutherAI team for supporting lm-evaluation-harness.

I have been computing MMLU scores on various LLMs (Llama-2-7B, Mistral-7B, Mixtral-8x7B) using the original OpenAI/hendrycks method:  https://github.com/hendrycks/test 
My results for Llama-2-7B agree well with the HuggingFace Leaderboard (lm-evaluation-harness) results.  But for Mistral-7B and Mixtral-8x7B, my results are about 7% lower than the lm-evaluation-harness results.  More details here:  
https://www.linkedin.com/posts/lloyd-watts-5523374_ai-llm-machinelearning-activity-7162209571627806720-q4en

OpenAI/hendrykcs computes a single answer for each question.  This method can output garbage (not A, B, C, or D) on some questions.
Eleuther/lm-evaluation-harness computes 4 loglikelihoods for each question, and chooses the best one.  This requires four times as much computation.  I presume it was done as a robustness mechanism, to "choose the best of A, B, C, or D, even when the greedy next token might not be one of those."  Is my understanding correct?

Is there any documentation about the decision to implement lm-evaluation-harness this way?  It looks like lm-evaluation-harness is different from OpenAI/hendrycks in this important way - potentially higher-quality results, but at the expense of four times the computation.  I've been searching and have not found any discussion of this important and interesting point.

And I notice that there appears to be no option to use lm-evaluation-harness on MMLU in a strictly hendrycks-compatible way, which I believe would be four times faster.

And there is the question of whether the lm-evaluation-harness method is really higher-quality.  It looks like it artificially boosts the apparent performance of the LLM under test.  If the LLM was going to give some other non-ABCD answer, the lm-evaluation-harness method forgives it and lets it give its best choice from ABCD.  So, is that really representative of real-world performance, i.e. is the benchmark fair?

Thank you in advance for taking time to consider these questions.



Not an answer to any of your questions, but a comparison of different approaches (sort of): https://huggingface.co/blog/evaluating-mmlu-leaderboard
And also an idea to check fairness (not sure if any benchmark supports it yet), and probably leakage of test to model training data, is testing with proper non-ABCD answer: ask to reply with ♠︎♧♥︎♢ (for example) and check that symbol (not ABCD).


Thank you @LSinev.  In fact, this discussion
https://huggingface.co/blog/evaluating-mmlu-leaderboard
does answer my main questions.  It acknowledges that the lm-evaluation-harness method is known to be different from the  "original" OpenAI/hendrycks implementation, and can produce substantially different results.  Since my goal was to be able to reproduce accepted benchmarks, it is clear that I must adopt the widely-used method used in lm-evaluation-harness.  I will do that!
Thank you,
Lloyd
Repository: EleutherAI/lm-evaluation-harness
Issue: `math_word_problems` group fails
Root cause: Validation Gap

`math_word_problems` group included both generate-until and likelihood tasks therefore when collating all results we get a KeyError of the wrong metric (since all tasks don't have the same metric)

`metrics = [results[task][metric] for task in task_list] # TODO: copy?
KeyError: 'exact_match,get-answer'`

There is already a TODO for this in the code. https://github.com/EleutherAI/lm-evaluation-harness/blob/b69c67c178ed101e857275f8d4d1a5769ced7210/lm_eval/evaluator.py#L515
Including an actual occurrence of the bug. 

Thank you for reporting this, and I'm sorry for the bug! 

This issue is also the same as #1423 I believe.

If @tcapelle isn't able to do so, I can PR a fix for this.
Repository: EleutherAI/lm-evaluation-harness
Issue: Truncation
Root cause: Unimplemented Feature Gap

Is it possible to change truncation strategy?
For example let's say that I want to remove whole few-shot sample or truncate each few-shot sample from left/right by a fair amount of tokens.

To add more context, with @mdocekal we found that harness truncates task description, when having n-shot prompt.  At least for GPT-2-XL and accelerate model. We would like to truncate the content of "shots" (so if it is 10-shot, and it won't fit, we want to change the particular example to e.g., 9-shot, or truncate from the last example, and not to truncate the preceding task description).



Might be a bit tricky. The task description is prepended to the fully constructed fewshot string here: https://github.com/EleutherAI/lm-evaluation-harness/blob/620d6a152b2eb9216138495bfd6db4ea770dec26/lm_eval/api/task.py#L853

If you only care about a specific model, one way could be to use a custom [sampler](https://github.com/EleutherAI/lm-evaluation-harness/blob/620d6a152b2eb9216138495bfd6db4ea770dec26/lm_eval/api/samplers.py#L20) and override the `get_context` method to condition the few-shots how you want by adding a tokenizer.

we found our "hacky" way to do what we wanted [here](https://github.com/DCGM/lm-evaluation-harness/commit/9f43d701cdcf673202744b0a1987cb109221b116#diff-28ae7c27276e965d9e744d9a91db657af52f4b0a74c2baa3976b81511f8777a5R108). The question remains whether we should try, implement and pull request such a thing into lm-harness. We are developing a benchmark, and were hoping people could use harness for its evaluation.


Do you think the truncation strategy could be specified with user function in yaml?

Chat template will be truncated too, am I correct?
Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval: error: unrecognized arguments in okapi folder for more than one task
Root cause: Validation Gap

This commands with two tasks that comes from the same root directory give errors:

`lm_eval --model hf     --model_args pretrained=mii-llm/maestrale-chat-v0.3-alpha  --tasks m_mmlu_it, arc_it     --device cuda:0 --batch_size 8`

<img width="1479" alt="Screenshot 2024-02-16 at 09 08 27" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/2806256/4b82a5e5-7870-44ab-8b3c-08783b2ea454">

but the command for  only one of the two tasks works. It seems that the it has problems going deeps into directories folders.

Hi! I believe this is because of the space after the comma in your argument: `--tasks m_mmlu_it,arc_it` should work as usual.
Repository: EleutherAI/lm-evaluation-harness
Issue: how to add tasks with requests based on the answers for the previous requests?
Root cause: Unimplemented Feature Gap

I have a task that I want to add. The task is about answering binary questions. The only difference from other tasks is that the second question requires knowing the response for the first one. I use loglikelihood strategy, so I pass two possible continuations and for question 1 I decide on the answer of the model based on output log-probs of these continuations. Then a put this question and the model answer into the prompt for the next question. Like storing history of model answering on questions. Each time I pass into the model the question and all previously answered questions with answers the model considered more probable.

Is there a way to implement such a task without adding any code outside task.py or task.yaml?

I have similar problem when i implement GPQA with CoT.In the paper, CoT needs to obtain the output of the first model first and add it to the prompt as the final prompt. The details are as follows:
![1](https://github.com/EleutherAI/lm-evaluation-harness/assets/92366232/c1c9895e-eca0-49ed-ad65-b9da15ce90fc)
Repository: EleutherAI/lm-evaluation-harness
Issue: logit_bias for openai.chat.completions
Root cause: Unimplemented Feature Gap

Have you tried using logit_bias param for openai.chat.completions? Like gpt-4. This param allows for generating only those tokens that have been previously biased (with quite a large value). Like the model makes forward and get logits for the currecnt token, then some lavue is added to the particular logits (for example to logits of tokens of letters A, B, C, D for mmlu), then log-probs are computed. gpt-4 does not return the probs for input tokens. But if you pass only ctx, bias possible continuation logits with quite large value, the model will exclusively generate tokens with these biased logits. gpt-4 can return log-probs for generated tokens. 

So, it may be the way to make the openai chat model generate only those tokens that we want and get their log-probs that will still be comparable (bias all tokens for the same value).
Repository: EleutherAI/lm-evaluation-harness
Issue: Cannot install lm_eval without changing other packages' version
Root cause: Configuration Error

Hi, Im trying to install lm_eval via 'pip install -e .' in the repo directory. But it always uninstall my old package and install a new one such as `torch`... however i dont want to change my env settings.

For `torch` as an example, I've already installed 'torch=1.12.1+cu113', and i check the 'pyproject.toml' of this repo, only requires `torch>=1.8`. However, when i tried 'pip install -e .', it show like this 
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/68947207/82f91362-bc0d-4b8c-900a-f373359b1d47)

How can i solve it, anybody helps?

And i tried to reinstall `torch=1.12.1`, but it did not satisfy the requirements of 'perf' 😂 
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/68947207/c4a49266-05ce-4d44-8ef5-a423db0b7bf0)
its a dilemma now 😂 

Does using `pip —upgrade-strategy to-satisfy-only` fix the initial problem?

https://pip.pypa.io/en/stable/development/architecture/upgrade-options/

Re: peft: https://github.com/huggingface/peft/blob/main/requirements.txt it does not seem to require torch 1.13, so this is odd to me.

> —upgrade-strategy to-satisfy-only

Thanks for reply. 

I tried `pip install --upgrade --upgrade-strategy=to-satisfy-only -e .` in repo directory, and it shows `option --upgrade-strategy: invalid choice: 'to-satisfy-only' (choose from 'only-if-needed', 'eager')`. In the `pip` docs you provided, the option `to-satisfy-only` is undocumented, which seems should be avoided. Moverover, `This is actually the “default” upgrade strategy when --upgrade is not set` said in the doc. so i think `--upgrade --upgrade-strategy=to-satisfy-only` is already the default setting. 

The problem of dependency seems to be weird and complicated 😂 Maybe upgrading torch is the easiest way to solve.

Yes, you should be able to fix this by force-reinstalling torch without any of its dependencies as an override.

Closing as I think this isn't lm_eval specific any more, but feel free to post on discord or if you find a workaround for this, feel free to post it here so that others who might experience it can see the fix!

Hello, I want to ask how exactly to operate
Repository: EleutherAI/lm-evaluation-harness
Issue: Task assigned to only one group when multiple groups are run
Root cause: Algorithmic Error

Not sure if this is an issue or if this is the expected way, but if we have a task assigned to two groups in the config and run both groups, for e.g with `gsm8k`:
```yaml
group:
  - math_word_problems
  - new_group
```
and `--tasks math_word_problems,new_group`

the `task_dict` is:
```python
... ('gsm8k', ('math_word_problems', <lm_eval.api.task.ConfigurableTask object at 0x1760df5c0>)), ...
```
Haven't tested if it's different in benchmarks.
Repository: EleutherAI/lm-evaluation-harness
Issue: llama / gguf interface broken?
Root cause: Interface Contract Mismatch

Only information i found on this issue was a post on discord:
```
Armando Diaz:
Hi!

I'm working on a research project and we were wondering if it is possible to use the lm-evaluation-harness to evaluate gguf models.

I've already done the following (only for testing purposes):

1) Host a quantized model from the TheBloke using llama.cpp
2) Execute lm_eval --model gguf --model_args base_url=http://localhost:8080 . This command gave me an error when sending a POST request because the endpoint is set as {self.base_url}/v1/completions and llama.cpp receives requests to http://localhost:8080/completion. That tells me that --model gguf expects an OpenAI-like completion. 
3)  Then, I ran a python script to map llama.cpp to an API like OpenAI following the README on llamacpp
4) After that, I did the same experiment lm_eval --model gguf --model_args base_url=http://localhost:8081, but now pointing to the OpenAI-like completion.
This gave me a bunch of errors because the script for lm_eval --model gguf --model_args  expects the keys token_logprobs and tokens that are never generated by a gguf quantized model. I can get top_logprobs but the other two keys are not available as part of the API Request.

Does somebody have successfully gotten this evaluation framework with gguf and if so, what am I missing? Do you have any examples running with gguf? Cannot find good documentation on the README
```

I went into the same issue, resulting in this error in logs:
`WARNING  [[gguf.py:90](http://gguf.py:90)] Invalid logprobs data. Expected 'logprobs' to contain 'token_logprobs' list. `

I hacked around with the gguf model implementation & made it talk to llama.cpp server directly. but idk about these logprobs.. 

I meet same error at [#1637](https://github.com/EleutherAI/lm-evaluation-harness/issues/1637)

Adding this to my Todo list. We should probably allow people to toggle on/off whether they are using the chat endpoint.

Hi,
I’ve been using lm_eval with gguf for the TheBloke-llama-7b.Q4_K_M.gguf model. It works, but it’s really slow.
I've installed the llama-cpp-python with CUDA:
`CMAKE_ARGS="-DLLAMA_CUDA=on" pip install llama-cpp-python[server] `
and used: 
```
python -m llama_cpp.server --host 0.0.0.0 --port 8080 --model /hf_models/llama-7b.Q4_K_M.gguf
lm_eval --model gguf --model_args base_url=http://0.0.0.0:8080 --tasks hellaswag --limit 5
```

About another issue, tasks like wikitext fail because loglikelihood_rolling isn't implemented.
Can we also fix this?


Hello, I also tried to perform some evals using ooba text-generation-webui (https://github.com/oobabooga/text-generation-webui)and a gguf template as backend but I also found issues with reading logprobs 
https://github.com/oobabooga/text-generation-webui/issues/5954

Specifically, the following error occurs:
`WARNING [gguf.py:92] Invalid logprobs data. Expected 'logprobs' to contain 'token_logprobs' list.`

However, as described, they are readable using the call to /v1/internal/logits
Repository: EleutherAI/lm-evaluation-harness
Issue: using additional scoring model to process results
Root cause: Architectural Constraint

I have a task that requires LM to generate some text as response. Then I use 3 other predefined LMs to score the generation. If I add loading this models as attributes to the class of the task, the models are loaded once the task is initialized utilizing GPU memory while still not needed. I need the scoring models right after the actual LM that I am using to generate texts has finished the runs. Nevertheless, I cannot load the models on process_results step, as in this case for each sample the models will be reloaded which is time consuming. 

Is it possible to implement loading some scoring models only once right after all the queries have finished (all greedy_until requests are finished) and use them to get the metrics for each sample?

What if you load the model only when evaluation starts? E.g. https://huggingface.co/spaces/hallucinations-leaderboard/leaderboard/blob/main/src/backend/tasks/xsum/task_v2.py#L68

You could also use one or multiple [filters](https://github.com/EleutherAI/lm-evaluation-harness/blob/a604f05c02d6b6d5c01887c61cb6a1b228f5b60a/lm_eval/filters/extraction.py#L6) for this. It has state and all the model responses are passed to it as a list. Might also be safe to  delete the model beforehand:
https://github.com/EleutherAI/lm-evaluation-harness/blob/a604f05c02d6b6d5c01887c61cb6a1b228f5b60a/lm_eval/evaluator.py#L390-L396