zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: can i evalute Qwen model，i'd like to use Qwen to be my base-model
Root cause: Validation Gap


Hi! We support evaluation on Qwen. You can use it through Huggingface via:

`lm_eval --model hf --model_args pretrained=Qwen/Qwen-1_8B,trust_remote_code=True --tasks ...`

And then following the instructions from there (e.g. installing `tiktoken`)

EDIT: due to Qwen's custom tokenizer you will currently need to manually set `tokenizer.pad_token = "
"` where `tokenizer.pad_token_id = tokenizer.eos_token_id` is typically set. I will be patching this in #1096 .


hi , i try to use the cmd to eavl the qwen model , but error rise "ValueError: Adding unknown special tokens is not supported",how can i solve it? thanks 

cmd :
`lm_eval --model hf \
    --model_args pretrained=qwen/be72f02dd47087f9035ee9bb5dea571b84785d27,trust_remote_code=True \
    --tasks ceval-valid_accountant \
    --device cuda `

error log :
Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/local/mnt2/workspace/shaocong/lm-evaluation-harness/lm_eval/__main__.py", line 207, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/local/mnt2/workspace/lm-evaluation-harness/lm_eval/utils.py", line 404, in _wrapper
    return fn(*args, **kwargs)
  File "/local/mnt2/workspace/lm-evaluation-harness/lm_eval/evaluator.py", line 102, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/local/mnt2/workspace/lm-evaluation-harness/lm_eval/api/model.py", line 136, in create_from_arg_string
    return cls(**args, **args2)
  File "/local/mnt2/workspace/lm-evaluation-harness/lm_eval/models/huggingface.py", line 245, in __init__
    self.tokenizer.add_special_tokens({"pad_token": "
"})
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py", line 948, in add_special_tokens
    added_tokens += self.add_tokens([value], special_tokens=True)
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py", line 999, in add_tokens
    return self._add_tokens(new_tokens, special_tokens=special_tokens)
  File "/usr2/shaocong/.cache/huggingface/modules/transformers_modules/be72f02dd47087f9035ee9bb5dea571b84785d27/tokenization_qwen.py", line 165, in _add_tokens
    raise ValueError("Adding unknown special tokens is not supported")
ValueError: Adding unknown special tokens is not supported

你把反斜杠“\”删了试试？

OK , it solved , the root reason is that path i set is "qwen/be72f02dd47087f9035ee9bb5dea571b84785d27", but in code , it judge
 `if "Qwen" in pretrained`

I change the code to 
`if "Qwen" in pretrained or "qwen" in pretrained:    
                # Qwen's trust_remote_code tokenizer does not allow for adding special tokens
                self.tokenizer.pad_token = "
"
            else:
                self.tokenizer.add_special_tokens({"pad_token": "
"})`
it work

Thanks,谢谢^_^ @WCSY-YG 

Thanks for reporting this! Will come up with a more generalized handling of this

Better fix for this in #1146 !
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the TriviaQA evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper:

> In this section we measure GPT-3’s ability to answer questions about broad factual knowledge. Due to the immense amount of possible queries, this task has normally been approached by using an information retrieval system to find relevant text in combination with a model which learns to generate an answer given the question and the retrieved text. Since this setting allows a system to search for and condition on text which potentially contains the answer it is denoted “open-book”. [RRS20] recently demonstrated that a large language model can perform surprisingly well directly answering the questions without conditioning on auxilliary information. They denote this more restrictive evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better and we test this hypothesis with GPT-3. We evaluate GPT-3 on the 3 datasets in [RRS20]: Natural Questions [KPR+19], WebQuestions [BCFL13], and TriviaQA [JCWZ17], using the same splits. Note that in addition to all results being in the closed-book setting, our use of few-shot, one-shot, and zero-shot evaluations represent an even stricter setting than previous closed-book QA work: in addition to external content not being allowed, fine-tuning on the Q&A dataset itself is also not permitted.

- [X] Data processing code implemented
- [x] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

_Note_: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/trivia_qa
Repository: EleutherAI/lm-evaluation-harness
Issue: Refactor main `evaluate()` loop into more readable sub-functions
GitHub labels: documentation

feature request


I can work on this! Will create a draft PR

Refactoring would include:
- [ ] Handling the many defaultdicts more cleanly such as via a dataclass
- [ ] Likely refactoring the requests-gathering (right now, lines 434 - 483 after PR 1390 merges) into a helper function 
- [ ] Cleaning up the fact that `alias` and `samples` keys are added to metrics and get popped from the dicts
- [ ] Placing `print_tasks` and the loop from L601-617 outside the main loop perhaps
Repository: EleutherAI/lm-evaluation-harness
Issue: Large discrepancy between Open LLM Leaderboard and running eval here
Root cause: Configuration Error

Using this model as a test: GeneZC/MiniMA-3B

I ran "hellaswag" on my local machine. It got a 0.25 (25) result. But on the Open LLM leaderboard it shows 68. Am I doing something wrong during testing?

The results on the model [page](https://huggingface.co/GeneZC/MiniMA-3B) for hellaswag are 10-shot while this the is the zero-shot result. You can use the
`--num_fewshot 10` arg to add the fewshot examples to the prompts! 

ahh.. thank you!

Any ideas why this thing may happen? And any solutions? I have observed similar phenomenon with Winogrande and already tried with `--num_fewshot 5`.

I have found the reason now. The reason lies in that previously harness does not take bos token into consideration (see Issue https://github.com/EleutherAI/lm-evaluation-harness/issues/401#issuecomment-1491242856) before the  introduction of Gemma (see recent PR https://github.com/EleutherAI/lm-evaluation-harness/pull/1465) while both Gemma and MiniMA is very much sensitive to this (perhaps due to the data format in design). And now you can use the bos feature as the above-mentioned PR has indicated.

As for the evaluation of Open LLM Leaderboard, I believe they have fixed this issue yet not explicitly showing that.

FYI, @l3utterfly 

Thanks for your interests in MiniMA and I hope this would not bother the use of MiniMA together with MiniChat.
Repository: EleutherAI/lm-evaluation-harness
Issue: general question: Is kv-cache actually not used in all the LLM-evaluation tasks?
Root cause: Documentation Deficiency

1. Is kv-cache actually **not used** in all the LLM-evaluation tasks, since those tasks usually takes **only one-step** attention calculation, not like language generating process which needs a lot of kv-cache since the words need to be generated one by one?

2. If this is true, **how to evaluate the quantization performace** if kv-cache needs to be quantized, if we want to accelerate LLM using something like [GPTQ](https://arxiv.org/pdf/2210.17323)?(since kv-cache is actually not used in normal evaluation tasks)

Hi!

1. Yes, loglikelihood-based evaluations (`loglikelihood`, `loglikelihood_rolling`, `multiple_choice`) are analogous to the "prefill" stage of decoding.
2. Could you share an example or link to code for what sort of quantizing of the KV cache you would want? We should support GPTQ-quantized models via AutoGPTQ: https://github.com/PanQiWei/AutoGPTQ#quantization-and-inference . 

Please feel free to reopen this if this does not answer your question.
Repository: EleutherAI/lm-evaluation-harness
Issue: `write_out.py` doesn't work for group tasks
Root cause: Validation Gap

Hi! First of all, thank you for such an amazing library! It help me a lot in benchmarking some base model.
I am using `scripts/write_out.py` to dump the input prompt of `ceval`. Here is the command I executed
```bash
python3 write_out.py --tasks ceval-valid --num_fewshot 5 --num_examples 0 --sets val --output_base_path output/path
```
Then the below error occurs:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/40186387/f38f2e5e-a4a8-4054-aad3-48d2622df352)

The reason seems to be that `get_task_dict()` called at https://github.com/EleutherAI/lm-evaluation-harness/blob/c9bbec6e7de418b9082379da82797522eb173054/scripts/write_out.py#L50 will return `Dict[Tuple[str, task]]` when benchmarking group task.
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for top-k metrics with num_return_sequences>1 
Root cause: Unimplemented Feature Gap

Hi, I wonder if lm-eval supports top-k evaluation when generating with beam_search with num_return_sequences>1 ?  

Hi! We don't currently support `num_return_sequences > 1`, and mostly focus on support for Top-P / Top-K temperature sampling, though we do support running generation K times per document to get K different completions via `repeats: K` in a task YAML.

Could you explain exactly what "top-K evaluation" means in this context?

If you'd be interested in opening a PR to help support this or streamline evaluation in the `num_return_sequences > 1` / beam search setting we'd be happy to accept it!


Thanks for the quick reply! I'm aiming to evaluate the top-k exact-match metric (check if there's a ground-truth hit in top-k lowest-perplexity candidates generated using beam-search  + num_return_sequences=k). 

In that case, I'll see if I can work on incorporating this feature and open a PR afterwards.

> Hi! We don't currently support `num_return_sequences > 1`, and mostly focus on support for Top-P / Top-K temperature sampling, though we do support running generation K times per document to get K different completions via `repeats: K` in a task YAML.
> 
> Could you explain exactly what "top-K evaluation" means in this context?
> 
> If you'd be interested in opening a PR to help support this or streamline evaluation in the `num_return_sequences > 1` / beam search setting we'd be happy to accept it!



> Thanks for the quick reply! I'm aiming to evaluate the top-k exact-match metric (check if there's a ground-truth hit in top-k lowest-perplexity candidates generated using beam-search + num_return_sequences=k).
> 
> In that case, I'll see if I can work on incorporating this feature and open a PR afterwards.
> 
> > Hi! We don't currently support `num_return_sequences > 1`, and mostly focus on support for Top-P / Top-K temperature sampling, though we do support running generation K times per document to get K different completions via `repeats: K` in a task YAML.
> > Could you explain exactly what "top-K evaluation" means in this context?
> > If you'd be interested in opening a PR to help support this or streamline evaluation in the `num_return_sequences > 1` / beam search setting we'd be happy to accept it!

Hi, sorry to disturb you, just wondering have you worked out this?
Repository: EleutherAI/lm-evaluation-harness
Issue: Stateful process_results
Root cause: Unimplemented Feature Gap

Hi, I'd like to use an external model for calculating some metrics, and I'm currently doing `process_results: !function utils.my_model`; is there a nice way to load the model only once, and free the memory afterwards? Should I be creating a Task class instead?

Hi! As currently set up, you could load the model once when `task/folder/utils.py` is first run, and then use the `process_results()` function referencing that single model, I'm pretty sure.  

a Task class would definitely work, or to add a method like `TaskSubclass.setup()` and `TaskSubclass.cleanup()` that run before and after the process results calls. 

If this is a common use case, we could support something like the above for if people are going to want to do e.g. inference using a reward model. Thoughts?

hey @haileyschoelkopf thanks for the feedback! I'll be going for a Task -- it seems the cleanest option (I can generate the reward model in `__init__` and associate its lifetime to the task object's)

> If this is a common use case, we could support something like the above for if people are going to want to do e.g. inference using a reward model. Thoughts?

What about a `process_all_results` that takes a list of instances and labels instead? Then one can do the reward model allocation and cleanup within that!

> What about a process_all_results that takes a list of instances and labels instead? Then one can do the reward model allocation and cleanup within that!

This sounds good to me and I think makes sense, can put this on the roadmap--would this mean you can still use a YAML for your purposes?

> would this mean you can still use a YAML for your purposes?

Yup, ideally, one could use the same syntax of `process_results`, with the difference that the provided function will be able to process a list of instances and results instead.

BTW I was digging a bit into the logic of lm-eval-harness, in case I could do a quick pull request for this, and it seems non-trivial to do -- will do a task class instead 🙂
Repository: EleutherAI/lm-evaluation-harness
Issue: Python task definitions are not imported by `include_task_folder`
Root cause: Unimplemented Feature Gap

As the title says -- I've tried copying the definition of `squadv2` (which is defined by a `task.py` file) in another folder, registering it under another name, and importing that folder via `include_task_folder`, but the new task does not get registered.

Yeah it looks like `include_task_folder` only considers yaml task definitions: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/__init__.py#L137 -- what's the best way to register python task definitions stored in different folders?

Python-based tasks still need to be added in the `__init__.py` manually.

https://github.com/EleutherAI/lm-evaluation-harness/blob/5133c9c4f5f3bf221d37dec2827aac0d5829d112/lm_eval/tasks/__init__.py#L18-L27

@lintangsutawika thanks, after importing them everything works like a charm!

BTW can we include all python tasks automatically within `initialize_tasks`?
The code would look something like this:

```python
import os
import importlib
import inspect

# Function to import all task classes decorated with @register_task in a given folder
def include_tasks_from_folder(folder_path):
    task_classes = []
    for filename in os.listdir(folder_path):
        if filename.endswith('.py') and not filename.startswith('__'):
            module_name = filename[:-3]
            module = importlib.import_module(f"{folder_path}.{module_name}")

            for name, cls in inspect.getmembers(module):
                if inspect.isclass(cls) and hasattr(obj, 'register_task'):
                    TASK_REGISTRY[name] = cls
                    ALL_TASKS.add(name)
                    func2task_index[cls.__name__] = name
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Is it possible to use BLEU with multiple references?
Root cause: Documentation Deficiency

I'm creating a new task and I would like to evaluate my generated output against N different references with BLEU, but the [code](https://github.com/EleutherAI/lm-evaluation-harness/blob/0bf683b4e6a9df359b3156ba9ba8d62bdd47e0c0/lm_eval/metrics.py#L122) appears to only pick up the first available reference, and I'm not sure how to map the `doc_to_target` in the task YAML to include multiple refs.



I'm assuming if you want to use the BLEU metric, then you would want to use the `generate_until` task type. In that case, you could also use the [HF's implementation of BLEU](https://huggingface.co/spaces/evaluate-metric/bleu). 

For `doc_to_target` we support it having more than 1 answer so you could make it that the dataset used has a gold feature that stores a list of references for each sample.

TriviaQA is one example of a dataset that uses multiple references! https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/triviaqa/default.yaml Please let us know if you have trouble mapping this onto BLEU.
Repository: EleutherAI/lm-evaluation-harness
Issue: Error evaluating openai models on benchmarks - ModuleNotFoundError for 'tiktoken' when already installed
Root cause: Environment Incompatibility

## Environment
* lm-evaluation-harness v0.4.0
* Python 3.9

## Description
I am encountering an error when attempting to evaluate GPT-4 on the HellaSwag benchmark using the lm-evaluation-harness. Despite having installed the necessary dependencies using `pip install "lm-eval[tiktoken]"` and pip install `"lm-eval[openai]"`, the code execution is failing with a `ModuleNotFoundError` for 'tiktoken'.

Running `pip freeze` I see `tiktoken==0.5.2`

## Steps to reproduce
1. Install dependencies using `pip install "lm-eval[tiktoken]"` and `pip install "lm-eval[openai]"`
2. Run the following command
```
lm_eval --model openai-chat-completions \
    --model_args model=gpt-4 \
    --tasks hellaswag
```

## Expected behavior
The evaluation should proceed without any module import errors, considering that the dependencies are reportedly installed correctly.

## Actual behavior
```
Traceback (most recent call last):
  File "/Users/me/lm-evaluation-harness/lm_eval/models/openai_completions.py", line 356, in __init__
    import openai, tiktoken  # noqa: E401
ModuleNotFoundError: No module named 'tiktoken'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/me/opt/anaconda3/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/Users/me/lm-evaluation-harness/lm_eval/__main__.py", line 207, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/Users/me/lm-evaluation-harness/lm_eval/utils.py", line 404, in _wrapper
    return fn(*args, **kwargs)
  File "/Users/me/lm-evaluation-harness/lm_eval/evaluator.py", line 102, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/Users/me/lm-evaluation-harness/lm_eval/api/model.py", line 136, in create_from_arg_string
    return cls(**args, **args2)
  File "/Users/me/lm-evaluation-harness/lm_eval/models/openai_completions.py", line 358, in __init__
    raise Exception(
Exception: attempted to use 'openai' LM type, but package `openai` or `tiktoken` are not installed.     please install these via `pip install lm-eval[openai]` or `pip install -e .[openai]`

```

I can't seem to replicate this--

if you install via `pip install -e .[openai]` from the root dir of this repo, this works on my end (fresh env, python3.9). 

The cause of this might be that we haven't pushed yet v0.4.0 to PyPI (we will be doing so very soon)? The above should work though.

Please feel free to reopen if the issue persists!
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the QA4MRE evaluation
Root cause: Unimplemented Feature Gap

> QA4MRE dataset was created for the CLEF 2011/2012/2013 shared tasks to promote research in question answering and reading comprehension. The dataset contains a supporting passage and a set of questions corresponding to the passage. Multiple options for answers are provided for each question, of which only one is correct. The training and test datasets are available for the main track. Additional gold standard documents are available for two pilot studies: one on alzheimers data, and the other on entrance exams data.

https://www.tensorflow.org/datasets/catalog/qa4mre
Repository: EleutherAI/lm-evaluation-harness
Issue: RuntimeError: Placeholder storage has not been allocated on MPS device!
Root cause: Resource Mishandling

Hi,
I'm running this on MPS and getting the error `RuntimeError: Placeholder storage has not been allocated on MPS device!`
Is there any way to resolve this?
Thank you

Hi! It sounds like we are not properly placing some variable in the code onto the `mps` device with `.to('mps')` (source: https://discuss.pytorch.org/t/torch-embedding-fails-with-runtimeerror-placeholder-storage-has-not-been-allocated-on-mps-device/152124 )--could you provide a full error trace to help debug this?

I don't have access to an MPS device to test this, but if you share the location in the code this error appears, and the devices of the objects, if any, involved in that line of code, then that should be very helpful.
Repository: EleutherAI/lm-evaluation-harness
Issue: CUDA OOM issue when comparing vLLM and HF models
Root cause: Resource Mishandling

Tried running [a script provided in ./scripts](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/scripts/model_comparator.py) to compare lm-evel results when using vLLM vs. HF backends, however, faced an OOM issue.

`torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 86.00 MiB (GPU 0; 31.75 GiB total capacity; 30.59 GiB already allocated; 47.94 MiB free; 30.67 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF`

It looks that after running vLLM, VRAM has not been cleared. 

if you replace the torch.cuda.enpty_cache() in the script with this utils function: https://github.com/EleutherAI/lm-evaluation-harness/blob/2a47159caff00135b026f724ace2a2011f3c7621/lm_eval/utils.py#L598 is the issue resolved?

This might also be a batch size issue. The default is 8 but you can choose a smaller one with `--batch`.

> clear_torch_cache()

@haileyschoelkopf thanks for your suggestion. I tried but still met the OOM issue. It looks that adding `gc.collect()` still cannot release GPU RAM occupied by vLLM.

> This might also be a batch size issue. The default is 8 but you can choose a smaller one with `--batch`.

Thanks. Tried this by reducing batch to be 1. However, the issue still exists.

hmm. You could try calling this function (from [here](https://discuss.pytorch.org/t/free-all-gpu-memory-used-in-between-runs/168202/2)) before and after `clear_torch_cache()` to check if the memory is released:
```python
def memory_stats():
    print(torch.cuda.memory_allocated()/1024**2)
    print(torch.cuda.memory_cached()/1024**2)
```
Also maybe change the order so that vllm is evaluated first?

Thanks. I have a graphical UI and can see GPU VRAM occupation rate in real time. I already switched the order -- running HF before vLLM , but still cannot release VRAM to make the second run start.

Sorry, I wasn't able to reproduce this. On a A100 with Llama-2-13b, I get:
```
allocated: 24874.125, cached: 72096.0
clear_torch_cache()
allocated: 9.125, cached: 22.0
```


On a V100 with a 7b LLaMa-2 model, I got
```
allocated: 23680.6484375
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/cuda/memory.py:416: FutureWarning: torch.cuda.memory_cached has been renamed to torch.cuda.memory_reserved
  warnings.warn(
cached: 24620.0
# run trorch.cuda.empty_cache()
allocated: 23680.6484375
cached: 23768.0
```

> 2024-01-05:01:14:42,822 INFO     [model_comparator.py:19] Memory allocated: 7040.6875, reserved: 9856
2024-01-05:01:14:43,443 INFO     [model_comparator.py:117] Memory stats cleared
2024-01-05:01:14:43,444 INFO     [model_comparator.py:19] Memory allocated: 8.125, reserved: 20

I also cannot replicate this. On 2080ti, batch size `auto`, Qwen/Qwen-1_8B. Going to close, but please let us know if the issue persists and you are able to provide a command to replicate this on a fresh machine!