zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: inference and memory using lm-eval-harness
Root cause: Documentation Deficiency

I need inference time for the given input prompt for whichever model i use and memory usage would be a plus . How to calculate this?

Are there particular functionalities that are not satisfied by our progress-bar projected time estimates (which are meant to act as *overestimates* of time required, if anything), or by existing resources on inference VRAM (https://vram.asmirnov.xyz/ , https://kipp.ly/transformer-inference-arithmetic/ , ... more linked in https://github.com/EleutherAI/cookbook )?

where is the progress bar i didn't find any ?
Where as the articles you shared about cookbook i liked the content but the quality is compromising.
could you help me finding the progress bar and also tell what all are displayed on that bar.

When you run our library on a task you will reach a TQDM progress bar reading `Running {loglikelihood / loglikelihood_rolling / generate_until} requests...` that tells you how much time each of those task types are expected to take when running your command in total. 

Further inference diagnostics displayed at runtime on our end will likely be considered out of scope for now and better implemented in an inference library itself, though if you give an example of exactly what information you would want exposed this would be useful info.

Closing as this should be better handled upstream.

I am still not finding any bar when i run a job. I am running the mmlu task on microsoft/phi-2 model .  At the end i get is accuracies. And i also find nothing in the output folder i specify in the script . I need a metric like time taken on a given prompt and memory usage etc. 

> I am still not finding any bar when i run a job.

![lm_eval_harness-tqdm](https://github.com/EleutherAI/lm-evaluation-harness/assets/19181718/e35b16cd-65d8-4e0d-913f-cbd9a170cd35)

The progress bar gives a running indication of how long the task will take.

@Akash08naik You can see #1763 for a discussion on including this in the harness (and perhaps pointers on how to instrument the code if you really want it).
Repository: EleutherAI/lm-evaluation-harness
Issue: Updating default model for openai completions
Root cause: Configuration Error

[According to OpenAI](https://openai.com/blog/gpt-4-api-general-availability), `text-davinci-003` (along with other models) are being deprecated on January 4 - we should replace the default https://github.com/EleutherAI/lm-evaluation-harness/blob/4d10ad56b1ffe569467eee2297e2317c99313118/lm_eval/models/openai_completions.py#L79 in the completions constructor  `gpt-3.5-turbo-instruct`

> Developers using other older completion models (such as text-davinci-003) will need to manually upgrade their integration by January 4, 2024 by specifying gpt-3.5-turbo-instruct in the “model” parameter of their API requests. gpt-3.5-turbo-instruct is an InstructGPT-style model, trained similarly to text-davinci-003. This new model is a drop-in replacement in the Completions API and will be available in the coming weeks for early testing.
Repository: EleutherAI/lm-evaluation-harness
Issue: always get acc,acc_norm, perplexity =1 on triviaqa task based on llama2 model
Root cause: Validation Gap

I use the following command to run triviaqa task
`
lm_eval --model hf \
    --model_args pretrained=../llama/models_hf/7B \
    --tasks triviaqa \
    --num_fewshot 1 \
    --device cuda:2 \
    --batch_size 8
`
I just get acc_norm=1, it's same when I use acc or perplexity indicator.
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/69768456/4231aed7-b04d-4286-b376-141aa50f87e1)

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/69768456/c73401b6-7b08-45af-b29c-3ef692cae7cb)

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/69768456/8b6adb9c-f7c8-470b-9505-6ea16f4a5259)


Hi! Could you provide the YAML file and codebase commit you are using to evaluate triviaqa?

This output seems quite strange given that triviaqa uses [none of these metrics](https://github.com/EleutherAI/lm-evaluation-harness/blob/25a15379676c8a2fa0b93ca9c4742b156e1fec39/lm_eval/tasks/triviaqa/default.yaml#L25) in its config. 

I can't seem to replicate your result when I try locally to run triviaqa on gpt2.

Hi, Thank you for your response. with the exact_match indicator, I get value=0.07.
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/69768456/b13414a2-4de1-47f9-9b8a-7efee632fe4e)

here is the YAML I used, I only changed the metric.
```
task: triviaqa
dataset_path: trivia_qa
dataset_name: rc.nocontext
output_type: generate_until
training_split: train
validation_split: validation
doc_to_text: "Question: {{question}}?\nAnswer:"
doc_to_target: "{{answer.aliases}}"
should_decontaminate: true
doc_to_decontamination_query: question
generation_kwargs:
  until:
    - "\n"
    - "."
    - ","
  do_sample: false
  temperature: 0.0
filter_list:
  - name: remove_whitespace
    filter:
      - function: remove_whitespace
      - function: take_first
target_delimiter: " "
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: true
metadata:
  version: 2.0

``` 

For the code, I used the following command and changed nothing except the above YAML file.
```
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
``` 



those metrics you used are currently only supported for loglikelihood or multiple_choice output_type tasks. In your case, this could be typically achieved by setting `output_type: loglikelihood`, but is complicated by triviaqa’s use of multiple gold standard answers.

We’ll make sure that running this will error out explicitly in future to avoid confusion.

We are also working on making metrics more easy to understand and add in the library in general!



Thanks a lot.
Repository: EleutherAI/lm-evaluation-harness
Issue: Granular control over few-shot prompt structure
Root cause: Unimplemented Feature Gap

Currently—if I understand correctly—few-shot evaluation is implemented by simply concatenating prompt-answer pairs with a separator, i.e. ```<P1><A1><sep><P2><A2><sep><P3><A3><P4>```. 

I can see a need for more granular control over the few-shot prompt, such as adding additional instructions before or after the prompt-answer pairs. For example, in some cases, the prompt template includes a long description of the task that you would only want to include once at the start and not pollute the context by repeating it for every example. Some models also need to be reminded to keep to a specific answer format, in which case you often want to include such instructions last.


This is an idea for the control I'd expect to have with an evaluation tool:
```<start text><P1><A1><sep>...<sep><Pn><An><Pn+1><end text>``` or 
```<start text><P1><A1><sep>...<sep><Pn><An><end text><Pn+1>```.

Am I missing a feature in the current codebase or is this on the agenda?

Hi! 

It's currently possible to add `<start text>` via the `description` config field.

Regarding the `<end text>` option, we don't currently expose it, but I have been thinking about how to combine things like `Let's think step by step.` with wrapping inputs in a chat template and we may want to expose such a field so that `<end text>` can go after the start of the model's response conversation turn.

Ah wonderful, thank you for this! I look forward to the addition of chat templates :)
Repository: EleutherAI/lm-evaluation-harness
Issue: [New Task] Upstream remaining Okapi multilingual tasks
Root cause: Unimplemented Feature Gap

#1228 adds support for the machine-translated HellaSwag task introduced in https://arxiv.org/abs/2307.16039 . 

We should add the other tasks translated in the Okapi paper, which are ARC, MMLU, and TruthfulQA. Adding these should be very straightforward and consist of following #1228 almost exactly but using a copy of the respective original benchmark's YAML as the template config. 

Code used to eval on these by the original authors can also be found at https://github.com/nlp-uoregon/mlmm-evaluation/blob/main/lm_eval/tasks/multilingual_arc.py . 

We'd welcome contributions adding any of these!

If anyone is working on it? if not, can i take it ?

Yes, go for it! I believe translated MMLU is being added in #1358 , but not the others yet

hello, I created #1358 using alexandrainst/m_mmul dataset that has some problems. Unfortunately the dataset for some languages is broken and the test does not pass. I don't know how to overcome the dataset issue creating a mirror dataset without the errors or ...? If you have suggestion let me know. I'm also planning next to integrate multilingual arc-c and arc-e.

Will try to take a look at #1358 asap if @lintangsutawika does not beat me to it!

I thought i merged that already lol

There's a problem in the HF dataset for it that needs checking, it seems

Will check tomorrow

Think these are all upstreamed now! Thank you @uanu2002 @giux78 and @JorgeDeCorte !
Repository: EleutherAI/lm-evaluation-harness
Issue: AutoGPTQ not working with HuggingFace accelerate (multi GPU)
Root cause: Configuration Error

If I run the following command:
```
accelerate launch -m lm_eval --model hf --model_args "pretrained=TheBloke/Llama-2-7B-Chat-GPTQ,gptq=True,load_in_4bit=True" --tasks "arc_challenge" --num_fewshot 25 --batch_size auto
```
I get the following error:
```
ValueError: You can't train a model that has been loaded with `device_map='auto'` in any distributed mode. Please rerun your script specifying `--num_processes=1` or by launching with `python {{myscript.py}}`.
```
If I add `device_map_option=balanced` to `--model_args`, I get the same issue.

But if I try an unquantized model with multi GPU e.g. `model_args="pretrained=meta-llama/Llama-2-7b-chat-hf"`, it works perfectly.

Is 4-bit AutoGPTQ compatible with multi GPU (accelerate)?

I installed AutoGPTQ from source, as follows
```
pip install "git+https://github.com/PanQiWei/AutoGPTQ.git@v0.6.0"
```
instead of `pip install -e ".[gptq]"`, because I encountered errors with the latter.

GPTQ with single GPU works fine.

What are the contents of your accelerate config?

```
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
gpu_ids: 0,1
machine_rank: 0
main_training_function: main
mixed_precision: 'no'
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

Note that we've changed the GPTQ argument name to `autogptq=True` to match the library, and that `load_in_4bit=True` should not be set if using GPTQ. Does making these changes solve your issue?


We will ideally be reexamining our quantization integrations as part of #1225 to make error messaging around these sorts of conflicts more intuitive.

I'm using v0.4.0, so I believe the argument is still `gptq=True`? Using `autogptq=True` threw an error anyways.

I tried removing `load_in_4bit=True` but the error still persists.

I see--I'd recommend trying with the current `main` branch installed from github and seeing if there is still an error. We'll put out a v0.4.1 on PyPI very soon.

@JeevanBhoot to narrow this down, could you try inserting `device_map={"":torch.cuda.current_device()}` into the `from_pretrained()` call?

It sounds as though, despite you not passing a `device_map` or using `parallelize=True` in the first snippet (this is our intended behavior + usecase), Huggingface or AutoGPTQ is inferring some device map nevertheless, which we don't want. (Again, `load_in_4bit=True` should also be removed here.) 

To which `from_pretrained()` calls should I add the `device_map` argument? I added it to all `from_pretrained()` calls in `huggingface.py` but the error persisted.

I have removed `load_in_4bit=True` and I am now working from `main`.
Repository: EleutherAI/lm-evaluation-harness
Issue: Same tasks, different models with same performance on multiple choice tasks, please HELP!
Root cause: Validation Gap

Hi,

Thanks for making and maintaining such a wonderful open-source project!

I am using the main branch, and clean conda env with just the requirement from the txt file.

I started to add in some other `output_type: multiple_choice` tasks, however, ran into the issue that I am getting same result from different models;

Then I roll back to check on other `output_type: multiple_choice` tasks, `cmmlu` and `hellaswag` both suffering the same problem.

I tried to run EleutherAI/pythia-70m,revision=step100000, EleutherAI/pythia-160m,revision=step100000, EleutherAI/pythia-410m, and meta-llama/Llama-2-13b-hf on hellaswag. 

Here are my results:

```
hf (pretrained=EleutherAI/pythia-70m,revision=step100000,dtype=float), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (64)
|    Tasks     |Version|Filter|n-shot|  Metric  | Value  |   |Stderr|
|--------------|-------|------|-----:|----------|-------:|---|-----:|
|hellaswag     |Yaml   |none  |     0|acc       |  0.2685|±  |0.0044|
|              |       |none  |     0|acc_norm  |  0.2730|±  |0.0044|
```

```
hf (pretrained=EleutherAI/pythia-160m,revision=step100000,dtype=float), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (64)
|    Tasks     |Version|Filter|n-shot|  Metric  | Value |   |Stderr|
|--------------|-------|------|-----:|----------|------:|---|-----:|
|hellaswag     |Yaml   |none  |     0|acc       | 0.2872|±  |0.0045|
|              |       |none  |     0|acc_norm  | 0.3082|±  |0.0046|
```

```
hf (EleutherAI/pythia-410m), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (64)
|  Tasks  |Version|Filter|n-shot| Metric |Value |   |Stderr|
|---------|-------|------|-----:|--------|-----:|---|-----:|
|hellaswag|Yaml   |none  |     0|acc     |0.2892|±  |0.0045|
|         |       |none  |     0|acc_norm|0.3114|±  |0.0046|
```

```
hf (meta-llama/Llama-2-13b-hf), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (64)
|  Tasks  |Version|Filter|n-shot| Metric |Value |   |Stderr|
|---------|-------|------|-----:|--------|-----:|---|-----:|
|hellaswag|Yaml   |none  |     0|acc     |0.2892|±  |0.0045|
|         |       |none  |     0|acc_norm|0.3114|±  |0.0046|
```

I also double check back to the [open-llm-leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) where the results on EleutherAI/pythia-410m and meta-llama/Llama-2-13b-hf are way higher than the number I got. 

I am a bit puzzled what is going on. Any help will be great!

Thanks!

command I am using is like this (really standard):


```
lm_eval --model hf \                                                                                
              --model_args EleutherAI/pythia-410m \
              --tasks hellaswag \
              --device cuda:0 \
              --batch_size auto
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Specify different few shot settings for tasks
Root cause: Documentation Deficiency

I know you can specify few shot settings for your entire task list with `—num_fewshot`. Is there a way to specify it for particular tasks? Eg. 25 shot for ARC, 10 shot for hellaswag.

Resolved:

1. Specify `num_fewshot: xxx` in your task yaml
2. Rebuild lm-eval harness: `pip install . -e`
Repository: EleutherAI/lm-evaluation-harness
Issue: HfHubHTTPError
Root cause: Environment Incompatibility

![微信图片_20240107222551](https://github.com/EleutherAI/lm-evaluation-harness/assets/136241945/73eec01b-084c-4ea2-a66b-dbc250ce44a5)


Hi, without more information we cannot help you with this error.

Could you tell us -- what command you run using our library that produces this error, any relevant information about your environment (e.g. versions of HF libraries, whether your machine has internet connection, ...) so we can narrow down whether this is a bug on our end or on HF's end?

Are you _really really sure_ that this isn't a credentials error?

Closing due to a likely credential error. Feel free to reopen if you can provide a reproducible example.
Repository: EleutherAI/lm-evaluation-harness
Issue: Maybe a bug of Winogrande task
Root cause: Documentation Deficiency

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/79445290/ab50233e-d15f-467c-b13c-4c172b6b6335)
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/79445290/1e3c7a17-3c05-4a44-8f01-3ad97367e8a3)
The above two images is the config and the preprocess code of Winogrande task. But it seems that the content of the  function ``doc_to_text`` and  ``doc_to_target`` needs to be switched

I came here because of the same question. I see it's unchanged - could you explain why you closed the issue?

> I came here because of the same question. I see it's unchanged - could you explain why you closed the issue?

You have to look into the Winogrande dataset, and you will find that that's the proper way to parse the dataset😂

I think my question is slightly different then. Why would the context be 0 or 1 in int, while it says the context is a str. I feel like this relies on some hidden assumptions of the evaluation pipeline. 

@haileyschoelkopf This seems a bit non-standard, in case you can clarify. Thanks in advance.

> I think my question is slightly different then. Why would the context be 0 or 1 in int, while it says the context is a str. I feel like this relies on some hidden assumptions of the evaluation pipeline.
> 
> @haileyschoelkopf This seems a bit non-standard, in case you can clarify. Thanks in advance.

You are right, if the text is represented by int, that is the index of the input list, you can check the code in this file: 
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the BioMRC evaluation
Root cause: Unimplemented Feature Gap

> We introduce BIOMRC, a large-scale cloze-style biomedical MRC dataset. Care was taken to reduce noise, compared to the previous BIOREAD dataset of Pappas et al. (2018). Experiments show that simple heuristics do not perform well on the new dataset, and that two neural MRC models that had been tested on BIOREAD perform much better on BIOMRC, indicating that the new dataset is indeed less noisy or at least that its task is more feasible. Non-expert human performance is also higher on the new dataset compared to BIOREAD, and biomedical experts perform even better. We also introduce a new BERT-based MRC model, the best version of which substantially outperforms all other methods tested, reaching or surpassing the accuracy of biomedical experts in some experiments. We make the new dataset available in three different sizes, also releasing our code, and providing a leaderboard.

https://arxiv.org/abs/2005.06376

Hi, I'm happy to attempt this

Edit: never mind, I see someone has self-assigned this one
Repository: EleutherAI/lm-evaluation-harness
Issue: About registering task as squadv2 /task.py
Root cause: Documentation Deficiency

Hi,

I followed the procedure outlined in "squadv2/task.py" to register a new task and have also imported it in the _init_.py. However, I encountered an error: "return TASK_REGISTRY[task_name](config=config) TypeError: __init__() got an unexpected keyword argument 'config'". 

Could you advise if I've overlooked any steps to registering a new task in a Python way? (this is more flexible to add my metric.)

Thank you.

I will close this bug, it's _init_ problem
Repository: EleutherAI/lm-evaluation-harness
Issue: exact_match computation very slow for multiple target tasks
Root cause: Algorithmic Error

TriviaQA has ~18,000 samples (with arbitrary number of targets) and currently it takes around 20 mins to compute the `exact_match` scores after the generations are completed (generations took ~ 3 mins). This is probably because each sample is called individually? We should definitely speed this up in #1167.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the HeadQA evaluation
Root cause: Unimplemented Feature Gap

> We present HEAD-QA, a multi-choice question answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. We then consider monolingual (Spanish) and cross-lingual (to English) experiments with information retrieval and neural techniques. We show that: (i) HEAD-QA challenges current methods, and (ii) the results lag well behind human performance, demonstrating its usefulness as a benchmark for future work.

https://arxiv.org/abs/1906.04701
Repository: EleutherAI/lm-evaluation-harness
Issue: Running lm-evaluation-harness on 1 example but progress bar runs to 4
Root cause: Documentation Deficiency

Hi folks, 

Thanks for a great tool. I want to run the evaluation harness on just 1 example, but the progress bar runs to 4, which I am confused by. Can you help me understand why, or point me to a relevant place in the documentation? 

Example code followed by the output is here below

```
paperspace@psgwzz6bpkub:~$ lm_eval --model hf     --model_args pretrained=gpt2     --tasks sciq    --device cuda:0     --batch_size 1 --limit 1
2024-01-13:14:13:59,717 INFO     [utils.py:148] Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-01-13:14:13:59,717 INFO     [utils.py:160] NumExpr defaulting to 8 threads.
2024-01-13:14:13:59,953 INFO     [config.py:58] PyTorch version 2.0.1+cu117 available.
2024-01-13:14:13:59,954 INFO     [config.py:95] TensorFlow version 2.9.2 available.
2024-01-13:14:13:59,956 INFO     [config.py:108] JAX version 0.4.8 available.
2024-01-13:14:14:08,172 INFO     [__main__.py:156] Verbosity set to INFO
2024-01-13:14:14:12,073 WARNING  [__init__.py:178] Some tasks could not be loaded due to missing dependencies. Run with `--verbosity DEBUG` for full details.
2024-01-13:14:14:15,790 WARNING  [__init__.py:178] Some tasks could not be loaded due to missing dependencies. Run with `--verbosity DEBUG` for full details.
2024-01-13:14:14:15,790 WARNING  [__main__.py:162]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2024-01-13:14:14:15,791 INFO     [__main__.py:229] Selected Tasks: ['sciq']
2024-01-13:14:14:15,804 WARNING  [logging.py:61] Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2024-01-13:14:14:15,804 INFO     [huggingface.py:146] Using device 'cuda:0'
2024-01-13:14:14:21,039 INFO     [task.py:337] Building contexts for task on rank 0...
2024-01-13:14:14:21,041 INFO     [evaluator.py:314] Running loglikelihood requests
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  5.21it/s]
```



I think I understood - because there are four options, the harness splits each question + options set into a question x options tuple and evaluates each question + option pair, creating 4x the number of questions.
Repository: EleutherAI/lm-evaluation-harness
Issue: Tasks on code evaluation
Root cause: Unimplemented Feature Gap

Hello, I've noticed that [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness.git) takes inspiration from this project. However, the implementation does not support vllm or other advanced inference methods. I hope to know if it's possible to add evaluation on coding tasks like mbpp or humaneval. 

We are currently discussing this here https://github.com/EleutherAI/lm-evaluation-harness/issues/1157