zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Inverse Scaling Tasks?
Root cause: Unimplemented Feature Gap

Apologies if this has been asked before, but I couldn't find the answer in `lm_evals/tasks` or any issues. Are there plans to add Inverse Scaling (https://github.com/inverse-scaling/prize) into the `lm-evaluation-harness`?

Hasn't been asked before!

Supporting these tasks as originally implemented would be very nice! We ourselves probably won't have the bandwidth for it soon, but if anyone wishes to contribute them we'd be happy to assist and review.

That implementation looks interesting, do you mind if I try it?

Yes, that'd be fantastic if you're interested!

Thank you for assigning. I'll get to work soon!

To address any possible issues, I'm currently asking the inverse scaling slack if it's okay to implement these tasks. I will start implementing them as soon as they are approved.

@RylanSchaeffer  @haileyschoelkopf  The initial implementation is done, all that's left is to test that it produces results like the paper. I'll make a pull request once I've verified the results.

Awesome :)

Cheers,
Rylan Schaeffer


On Fri, Mar 8, 2024 at 6:25 PM Hanwool Albert Lee ***@***.***>
wrote:

> @RylanSchaeffer <https://github.com/RylanSchaeffer> @haileyschoelkopf
> <https://github.com/haileyschoelkopf> The initial implementation is done,
> all that's left is to test that it produces results like the paper. I'll
> make a pull request once I've verified the results.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/EleutherAI/lm-evaluation-harness/issues/1442#issuecomment-1986697710>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACEHLCZBRHSEGU3IKRGC363YXJXLLAVCNFSM6AAAAABDOMNE6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWGY4TONZRGA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


@haileyschoelkopf 
Hi! I compared the scores shown in the paper with the scores obtained through lm-eval-harness using OPT 125M through 6.7B models, and there is a slight difference. I was wondering if lm-eval-harness tolerates a certain amount of score difference?

Or maybe I'll use the code utilized in the evaluation(of inverse scaling prize) as a custom metric. I didn't get anything from the inverse-scaling team, but I did find some related work in the authors' github.

@h-albert-lee That’s great progress!

Would you be able to open a PR with your implementation and resulting scores so we can discuss there? It’s hard to say without being able to look at the implementation differences/concrete numbers.



@haileyschoelkopf Thanks a lot!, I'll apply the pre-commit and post a pull request with my experimental results soon.
Repository: EleutherAI/lm-evaluation-harness
Issue: Error: BuilderConfig not found
Root cause: Configuration Error

I am trying to run the following command.

```
lm_eval --model hf \
    --model_args pretrained=mistralai/Mistral-7B-v0.1 \
    --tasks pile_openwebtext2 \
    --device cpu \
    --batch_size 10
```

And getting this error: ValueError: BuilderConfig 'pile_openwebtext2' not found. Available: ['all', 'enron_emails', 'europarl', 'free_law', 'hacker_news', 'nih_exporter', 'pubmed', 'pubmed_central', 'ubuntu_irc', 'uspto', 'github']

I have checked ``` lm-eval --tasks list``` and pile_openwebtext2 is available there.

Hi! This is a known issue ; I will push an update to the Pile tasks so that they reflect the fix described in #731 , but you will also need your own copy of the Pile downloaded locally to run these tasks.

closing as duplicative of #1338 for tracking purposes. please let us know if there are lingering questions though!
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluating on MMLU validation dataset
Root cause: Documentation Deficiency

Hello,

I'm trying to run the evaluation harness on the validation split of MMLU. So far, I understand that the `mmlu` task, which is registered with the harness, uses the test split of the `hails/mmlu_no_train`. 

I've verified that this dataset contains a validation split (on HF), so my strategy is to fork the repo and duplicate all the YAML files located at `lm-evaluation-harness/lm_eval/tasks/mmlu/default` and just change the `test_split: test` pair to `test_split: validation`.
Here are some steps I've done so far:

 * I think I need to ensure task name uniqueness, so I'll need to replace all instances of the `mmlu` prefix with `mmlu_validation` (my desired task name)
* I suspect the `_generate_configs.py` file is designed to generate new YAML files with a specified prefix. But attempting to run this from `lm_eval/tasks/mmlu/` complains that `No module named lm_eval.logger`. 

I apologize if this issue is easily rectifiable but I figured it would be easiest to ask! Or maybe there's an even simpler way to accomplish my goal. Thank you very much for your time!


> I suspect the _generate_configs.py file is designed to generate new YAML files with a specified prefix. But attempting to run this from lm_eval/tasks/mmlu/ complains that No module named lm_eval.logger.

I think it has been renamed to `lm_eval.utils.eval_logger`. You can also delete it or substitute it for print statements. I think it will save you a lot of time if you can re-use this script.

@lbertge let us know if you have further questions!
Repository: EleutherAI/lm-evaluation-harness
Issue: Strange behavior with repeats not set correctly in task in simple_evaluate
Root cause: Configuration Error

When I run simple_evaluate with hellaswag task, getting a strange error where the Task instance has repeats=None, even though it should be defaulting to 1.

I can see that the repeat is set correct as `repeats=1` in task.config:
```
(Pdb) task.config
TaskConfig(task='hellaswag', task_alias=None, group=['multiple_choice'], group_alias=None, dataset_path='hellaswag', dataset_name=None, dataset_kwargs=None, training_split='train', validation_split='validation', test_split=None, fewshot_split=None, process_docs=<function process_docs at 0x7f32f99e8ca0>, doc_to_text='{{query}}', doc_to_target='{{label}}', doc_to_choice='choices', process_results=None, use_prompt=None, description='', target_delimiter=' ', fewshot_delimiter='\n\n', fewshot_config=None, num_fewshot=0, metric_list=[{'metric': 'acc', 'aggregation': 'mean', 'higher_is_better': True}, {'metric': 'acc_norm', 'aggregation': 'mean', 'higher_is_better': True}], output_type='multiple_choice', generation_kwargs=None, repeats=1, filter_list=None, should_decontaminate=False, doc_to_decontamination_query=None, metadata={'version': 1.0})
```

But after `task.build_all_requests(limit=limit, rank=lm.rank, world_size=lm.world_size)` [[1](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/evaluator.py#L335)], the instance created has repeats set as None:

```
(Pdb) task.instances[0]
Instance(request_type='loglikelihood', doc={'ind': 24, 'activity_label': 'Roof shingle removal', 'ctx_a': 'A man is sitting on a roof.', 'ctx_b': 'he', 'ctx': 'A man is sitting on a roof. he', 'endings': ['is using wrap to wrap a pair of skis.', 'is ripping level tiles off.', "is holding a rubik's cube.", 'starts pulling up roofing on a roof.'], 'source_id': 'activitynet~v_-JhWjGDPHMY', 'split': 'val', 'split_type': 'indomain', 'label': '3', 'query': 'Roof shingle removal: A man is sitting on a roof. He', 'choices': ['is using wrap to wrap a pair of skis.', 'is ripping level tiles off.', "is holding a rubik's cube.", 'starts pulling up roofing on a roof.'], 'gold': 3}, arguments=('Roof shingle removal: A man is sitting on a roof. He', ' is using wrap to wrap a pair of skis.'), idx=0, metadata=(None, None, None), resps=[], filtered_resps={}, task_name=None, doc_id=None, repeats=None)
```

Which causes the run to fail later on when any code similar to `[req] * req.repeats` is run:

```
(Pdb) [req] * req.repeats
*** TypeError: can't multiply sequence by non-int of type 'NoneType'
```
Repository: EleutherAI/lm-evaluation-harness
Issue: janitor_util C++ splits multibyte characters into non-UTF bytes(?)
Root cause: Algorithmic Error

When I ran `janitor.decontaminate(input)` from `janitor.py` in C++ mode, my Python code threw `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 86: unexpected end of data.` I cleaned my `input` of non-UTF-8 characters before passing it to `decontaminate()` by running `input = input.encode("utf-32", errors="ignore").decode("utf-32", errors="ignore")`, but the error still showed. I suspect the error was caused by the string splitting in `janitor_util.cpp`'s `clean_ngram_with_indices`. When some multibyte UTF-8 characters are split, the resulting bytes are outside UTF range. For example, the en-dash is `\xe2\x80\x93` in UTF-8, but `\xe2` is not a UTF character.
Repository: EleutherAI/lm-evaluation-harness
Issue: log_samples File name too long. Need truncation or override
Root cause: Validation Gap

When setting --log_samples, the default logic for lm_eval.main is the following:

```
            if args.log_samples:
                for task_name, config in results["configs"].items():
                    output_name = "{}_{}".format(
                        re.sub("/|=", "__", args.model_args), task_name
                    )
                    filename = path.joinpath(f"{output_name}.jsonl")
                    samples_dumped = json.dumps(
                        samples[task_name],
                        indent=2,
                        default=_handle_non_serializable,
                        ensure_ascii=False,
                    )
                    filename.write_text(samples_dumped, encoding="utf-8")
```

If a user passes in a lengthy --model_args such as `pretrained=${model_path},tokenizer=${tokenizer},tensor_parallel_size=1,dtype=float16,gpu_memory_utilization=0.8,data_parallel_size=1`

The filename can become extremely length, especially if the pretrained is a local path nested in many directories, which leads to 

```
OSError: [Errno 36] File name too long: 
```


Either this should be truncated, or user should be allowed to specify a path_name to override args.model_args, or only set output_name to task_name and let output_path be the differentiator.
```
                    output_name = "{}_{}".format(
                        re.sub("/|=", "__", args.model_args), task_name
                    )
```

I believe this issue can be closed thanks to #1766
Repository: EleutherAI/lm-evaluation-harness
Issue: Run Gemma LM in Huggingface (simple patch)
Root cause: Algorithmic Error

Posting this here for visibility:

The following diff fixes Gemma performance:
```
diff --git a/lm_eval/models/huggingface.py b/lm_eval/models/huggingface.py
index e6ffc828..3f775dbb 100644
--- a/lm_eval/models/huggingface.py
+++ b/lm_eval/models/huggingface.py
@@ -786,8 +786,8 @@ class HFLM(LM):
             continuation = context[-n_spaces:] + continuation
             context = context[:-n_spaces]

-        whole_enc = self.tok_encode(context + continuation, add_special_tokens=False)
-        context_enc = self.tok_encode(context, add_special_tokens=False)
+        whole_enc = self.tok_encode(context + continuation, add_special_tokens=True)
+        context_enc = self.tok_encode(context, add_special_tokens=True)

         # whole_enc = self.tok_encode(context + continuation)
         # context_enc = self.tok_encode(context, add_special_tokens=False)
```        


Prior to this fix:
```
hf (pretrained=google/gemma-7b,attn_implementation=flash_attention_2), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)
|    Tasks     |Version|Filter|n-shot|  Metric  |     Value     |   |    Stderr    |
|--------------|------:|------|------|----------|--------------:|---|-------------:|
|lambada_openai|      1|none  |None  |perplexity|5909545606.3929|±  |899657349.2755|
|              |       |none  |None  |acc       |         0.0037|±  |        0.0008|
```
after:
```
hf (pretrained=google/gemma-7b,attn_implementation=flash_attention_2), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto (32)
|    Tasks     |Version|Filter|n-shot|  Metric  |Value |   |Stderr|
|--------------|------:|------|------|----------|-----:|---|-----:|
|lambada_openai|      1|none  |None  |perplexity|3.4640|±  |0.0677|
|              |       |none  |None  |acc       |0.6707|±  |0.0065|
```


Todos: 
- [ ] allow BOS / no BOS to be togglable by a user, and consider using it by default in most models?
- [ ] see how this impacts perf on Llama / Mistral / other BOS using models, and also on no-BOS models
Repository: EleutherAI/lm-evaluation-harness
Issue: Allow `--include_path` to import an externally-defined LM subclass
Root cause: Unimplemented Feature Gap

As requested by @LSinev , 

it might be helpful for some users to be able to run `lm_eval` or `lm_eval/__main__.py` without editing the file, but while using an `LM` subclass one has defined outside the library. This is already possible via either adding an import to `__main__.py` of the module containing the `register_model()` decorated class, or via passing an already-initialized LM class to `simple_evaluate()`, but that isn't as convenient in all cases.

We could thus allow, similar to the use of external task folders using `--include_path`, a way to import a desired python module at runtime, enabling users to import models / metrics defined outside the library into its registries.

Are there any pitfalls to avoid (based on recent experience with merging PRs with new models)? Or anything to remember? In case of starting the PR development to close this issue?
Repository: EleutherAI/lm-evaluation-harness
Issue: Add EQ-Bench
Root cause: Unimplemented Feature Gap

EQ-Bench (https://github.com/EQ-bench/EQ-Bench) is more and more popular. I think it should be possible to implement, do you agree?

We'd welcome a contribution for EQ-Bench!

I can have a look at doing this if no one is already

I will be happy to test it :)
Repository: EleutherAI/lm-evaluation-harness
Issue: how to run it on NPU
Root cause: Unimplemented Feature Gap


> _No description provided._
This runs too slowly on the CPU. I don't have a GPU here, or is there any other way to make it run faster on the CPU


I apologize if this issue is easily rectifiable but I figured it would be easiest to ask! Or maybe there's an even simpler way to accomplish my goal. Thank you very much for your time!

Hi! I am not familiar with NPUs or where they are used. Is this a device supported in stock Pytorch? is there a recommended typical way to run HF transformers models on NPU?

> Hi! I am not familiar with NPUs or where they are used. Is this a device supported in stock Pytorch? is there a recommended typical way to run HF transformers models on NPU?

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/7370936/48c59a1e-fc5c-4658-936a-6e160eaed138)

`import torch 
import torch_npu 
def test_cpu(): 
    input = torch.randn(2000, 1000).detach().requires_grad_() 
    output = torch.sum(input) 
    output.backward(torch.ones_like(output)) 
def test_npu(): 
    input = torch.randn(2000, 1000).detach().requires_grad_().npu() 
    output = torch.sum(input) 
    output.backward(torch.ones_like(output)) 
if __name__ == "__main__": 
    test_cpu() 
    torch_npu.npu.set_device("npu:0") 
    test_npu()`

Gotcha, thanks for sharing!--we would probably recommend that users initialize a HF model on NPU themselves and pass it into HFLM, and probably won't support it natively unless there's substantial community backing.

Hi @dqwtf117, I've submitted a [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/1886) that will help you run `lm_eval ` on Ascend NPU without any additional changes

This repo allows me to quickly and efficiently eval the chatglm series of models :grinning:. My lab recently brought hundreds of Ascend NPUs, and I wonder if you two can cooperate and help me eval our models natively:), thanks to you all! @lintangsutawika @haileyschoelkopf @statelesshz 

@statelesshz and @jiaqiw09 's PR has now been merged! NPU support through accelerate should be usable.

> This repo allows me to quickly and efficiently eval the chatglm series of models 😀. My lab recently brought hundreds of Ascend NPUs, and I wonder if you two can cooperate and help me eval our models natively:), thanks to you all! @lintangsutawika @haileyschoelkopf @statelesshz

Hi, here is a [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/1886) supporting NPU. Feel free to leave message if you need help.
Repository: EleutherAI/lm-evaluation-harness
Issue: evaluation extremely slow with llama_cpp/gguf
Root cause: External Dependency Breakage

Evaluation of gguf models via llama_cpp server is extremely slow. All the layers are offloaded to the GPU so normally it should work fine, but truthfulqa takes 10 hours, it should normally take ~40 minutes or even less.
`lm_eval --model gguf --model_args base_url=http://localhost:8000 --batch_size 16 --tasks truthfulqa_mc2 --num_fewshot 0`


> Note that for externally hosted models, configs such as --device and --batch_size should not be used and do not function. 

i have the same problem. What can I do to make it support batch processing？？

It seems to be an [issue ](https://github.com/ggerganov/llama.cpp/discussions/229)with llama.cpp.

> It seems to be an [issue ](https://github.com/ggerganov/llama.cpp/discussions/229)with llama.cpp.

So basically they say it's a problem with quantized models running with large prompts. 
That sounds strange because the impact of the dequantization step should actually matter much less with longer prompts and larger batches sizes.  


Hey,
Is there anything new on this topic. I am experiencing the same problems when evaluating a local llama.cpp server hosting a GGUF.

So I'm looking into our current implementation and looks like it's wrong for logliklihood/multiple-choice tasks as the llama.ccp server does not as yet return the logprobs of the prompt. Theres an open issue [here](https://github.com/ggerganov/llama.cpp/issues/8942). 

Looking into alternates. Whats the usual way people load gguf models? I'm trying [llama-ccp-python](https://github.com/abetlen/llama-cpp-python) (they also have a server implementation) but can't seem to get it working in my environment.
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue with `bigbench_gender_inclusive_sentences_german_multiple_choice`
Root cause: Validation Gap

While running this task I get the following error stack trace:

```
Traceback (most recent call last):
  File "/opt/conda/envs/fr-de-lb/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/__main__.py", line 288, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/evaluator.py", line 154, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 412, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(string_task_name_list)
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 253, in load_task_or_group
    collections.ChainMap(
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 164, in _load_individual_task_or_group
    return load_task(task_config, task=name_or_config, group=parent_name)
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 153, in load_task
    task_object = ConfigurableTask(config=config)
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/api/task.py", line 731, in __init__
    test_target = self.doc_to_target(test_doc)
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/api/task.py", line 962, in doc_to_target
    target_string = utils.apply_template(doc_to_target, doc)
  File "/home/ayushthakur/lm-eval/llm-leaderboard-fr-de/lm-evaluation-harness/lm_eval/utils.py", line 426, in apply_template
    return rtemplate.render(**doc)
  File "/opt/conda/envs/fr-de-lb/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/conda/envs/fr-de-lb/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
ValueError: 'Potsdam ist eine kreisfreie Stadt und mit gut 180.000 Einwohner*innen die bevölkerungsreichste Stadt und Hauptstadt des Landes Brandenburg.' is not in list
```

I am using the following command:

```
lm_eval --model hf --model_args pretrained=microsoft/phi-2,trust_remote_code=True --tasks bigbench_gender_inclusive_sentences_german_multiple_choice --device cuda:0 --batch_size 1 --output_path output/phi-2-mmlu-arc --limit 2 --wandb_args project=lm-eval-harness-integration --log_samples
```

cc: @haileyschoelkopf
Repository: EleutherAI/lm-evaluation-harness
Issue: Passing HF Llama Key
Root cause: Documentation Deficiency

Hello! 

I am currently trying to usie Harness with LLama 2, but I cannot appear to find a way to pass in my HF API key so that I can access the model. I was wondering if anybody could point me in the right direction? Or is this something I need to implement myself? 

Thank you in advance :) 

Cam

Hi! 

You should be able to run Llama or other gated models via first running `huggingface-cli login` and pasting your key in there. Is this not working?

Hello Hailey, 

That solved it... Sorry for the simple question, I'm still new to all of this! 

Thank you 😊

No problem, glad it worked!
Repository: EleutherAI/lm-evaluation-harness
Issue: use local model api
Root cause: Documentation Deficiency

When I use the following command I am not able to get any response：
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/53484050/aa1356e4-af1b-4906-bbcb-58363436f8e0)

my local model name is` gpt-3.5-turbo`
url is http://0.0.0.0:6006/v1

when i run python like this 
```
import openai
openai.api_base = "http://0.0.0.0:6006/v1" # 本地ip
# openai.api_base = "https://u17621-9cba-0e882657.westc.gpuhub.com:8443/v1" #远程访问ip
openai.api_key = "none"

# # 使用流式回复的请求
# for chunk in openai.ChatCompletion.create(
#     model="gpt-3.5-turbo",
#     messages=[
#         {"role": "user", "content": "你好"}
#     ],
#     stream=True
#     # 流式输出的自定义stopwords功能尚未支持
# ):
#     if hasattr(chunk.choices[0].delta, "content"):
#         print(chunk.choices[0].delta.content, end="", flush=True)

# 不使用流式回复的请求
response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "你好"}
    ],
    stream=False
)
# print(response.choices[0].message.content)
print(response)
```
I can get answer


Can you check your model hosted using a curl call first?
Like below
```
curl -X POST http://0.0.0.0:6006/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"system","content":"You are a helpful assistant. Please respond to the user request."}, {"role":"user","content":"What is 2 + 3 multiplied by 6."}], "max_tokens": 32, "stop":["
", "User:"], "temperature":0.0}'
```

If this does work then use lm-eval as 
```
OPENAI_API_KEY="key" lm_eval --model local-chat-completions --model_args model=MyModel,base_url=http://0.0.0.0:6006/ --tasks gsm8k
```

If the curl doesn't work then its an issue with your local model hosting.