zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: scrolls pyrouge import error
Root cause: External Dependency Breakage

when I try to run any `_SCROLLSSummaryTask` I get  
 (e.g. srolls_govreport)

ModuleNotFoundError: No module named 'datasets_modules.metrics.metricsscrolls_py.74e8c8051b3b18b1e72c49a033274c72e63ca708b628688199bb7001bb7a1884.rouge'

even after pip install -e.
I'm in a lightly modified fork with datasets==2.0 installed

any idea?


### Traceback

```python

  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/evaluator.py", line 110, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 425, in get_task_dict
    task_name_dict = {
  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 426, in <dictcomp>                                                                             task_name: get_task(task_name)()
  File "/home/sam/character-tech/lm-evaluation-harness/lm_eval/tasks/scrolls.py", line 148, in __init__
    load_metric(_download_metric(), config_name=self.DATASET_NAME)
  File "/home/sam/.conda/envs/cai-env-kurt2/lib/python3.10/site-packages/datasets/load.py", line 1395, in load_metric
    metric_cls = import_main_class(metric_module, dataset=False)
  File "/home/sam/.conda/envs/cai-env-kurt2/lib/python3.10/site-packages/datasets/load.py", line 106, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/sam/.conda/envs/cai-env-kurt2/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/sam/.cache/huggingface/modules/datasets_modules/metrics/metricsscrolls_py/74e8c8051b3b18b1e72c49a033274c72e63ca708b628688199bb7001bb7a1884/metricsscrolls_py.py", lin
e 8, in <module>
    from .rouge import compute_rouge, postprocess_text as rouge_postprocess_text  # From: https://huggingface.co/datasets/tau/scrolls/raw/main/metrics/rouge.py
ModuleNotFoundError: No module named 'datasets_modules.metrics.metricsscrolls_py.74e8c8051b3b18b1e72c49a033274c72e63ca708b628688199bb7001bb7a1884.rouge'


```

Is this happening on `main` or `big-refactor`?

main

Is there a summary of what's changed on `big-refactor` somewhere?
Repository: EleutherAI/lm-evaluation-harness
Issue: Warning on gsm8k
Root cause: Configuration Error

Hey all!

When I run this: `python main.py --model hf-causal --model_args pretrained=microsoft/phi-1_5,trust_remote_code=True --tasks gsm8k --device cuda:0`

I get the following warnings after each iteration: `A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.`

Is it a bug or I need to pass additional flags? I installed the library as described in the README with a new conda environment.

Thanks!

Hi! Do you experience the same issue when you run with either the `hf-causal-experimental` backend, or in the `big-refactor` branch?

Oh it works with `hf-causal-experimental`.
Thanks, and thanks for this great project!

sorry to hijack this post but what is the command line exactly?
```
(pt2.1.0) vincent@vince-ryzen:~/nlp/lm-evaluation-harness$ python main.py --model hf-causal-experimental --model_args pretrained="/mnt/InternalCrucial4/dataAI/phi-2-psy/",trust_remote_code=True,padding_side='left' --tasks "gsm8k" --device cuda:1 --batch_size 1 --num_fewshot 5 --no_cache
Selected Tasks: ['gsm8k']
Traceback (most recent call last):
  File "/home/vincent/nlp/lm-evaluation-harness/main.py", line 105, in <module>
    main()
  File "/home/vincent/nlp/lm-evaluation-harness/main.py", line 69, in main
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vincent/nlp/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/vincent/nlp/lm-evaluation-harness/lm_eval/evaluator.py", line 76, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vincent/nlp/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string
    return cls(**args, **args2)
           ^^^^^^^^^^^^^^^^^^^^
TypeError: HuggingFaceAutoLM.__init__() got an unexpected keyword argument 'padding_side'
```


There should be no need to set padding_side in model_args with hf-causal-experimental.

Have you checked out the main branch yet? it supports batched generation by default along with a bunch more features.


I am on main but what should I be looking at specifically ?
I'll try without the setting padding_side

I believe you're still using https://github.com/EleutherAI/lm-evaluation-harness/tree/master --
if you try out https://github.com/EleutherAI/lm-evaluation-harness/tree/main you'll need to change a few things

(`main.py` -> `lm_eval` or `lm_eval/__main__.py` , `--model hf` instead, `--no_cache` set by default... as a start)

but should have the opportunity to run things much faster, such as via using data-parallel inference (documented in the `main` branch readme) with `accelerate` or using vllm!
Repository: EleutherAI/lm-evaluation-harness
Issue: using "A:" replace "A: "
Root cause: Algorithmic Error

When I worked with the T5 series model (non-instruction tuned) on the Bigbench (e.g., sports_understanding), I noticed the generation is sensitive to space character in the output_prefix.

For instance, when using the input 'Determine whether an artificially constructed sentence relating to sports is plausible or not.\n\nQ: Is the following sentence plausible? "Bam Adebayo scored a reverse layup in the Western Conference Finals."\nA: yes (...omit) Q: Is the following sentence plausible? "Emmanuel Sanders got a base hit."\nA:', the model generates the correct response. However, altering the output prefix from "A:" to "A: " results in incomplete outputs like "ye".

I’m not sure why this occurs, but I wanted to share this observation. Feel free to correct me if I miss anything.

You're not missing anything at all :) This is a very common and unfortunate problem with evaluating models, especially ones that aren't instruction-finetuned.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to implement zero-shot cot (calling model twice?)
Root cause: Unimplemented Feature Gap

Hi,

Thanks for this great repo! I am new here and try to implement zero-shot CoT to the MMLU dataset. But I am confused by how can I let the model do the reasoning and then give the answer. Any pointer is welcome so I can diagest the code myself. Thank you!

Hi! For an example of this, see the BBH zero-shot CoT task implementations: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/bbh/cot_zeroshot

Chain-of-thought is implemented not as 2 LM calls, but as a single call where the model simultaneously generates natural language rationale followed by some concluding phrase such as `Therefore, the answer is ____`. Then, we use a post-processing (filter) step to extract the model's provided answer using a regex from that concluding phrase. 

This may not work very well when not done few-shot, as models may not give the format of answer expected or weaker models may fail to conclude their chain of thought correctly.

Feel free to ask more questions here as needed!

Thank you Hailey for your example! I’ve checked the example.I just wonder if that’s possible to use current framework if we do want to call a model many times (and probably in the 2nd time append the text like “therefore, the answer is X”)?Best,MarkOn Dec 4, 2023, at 07:48, Hailey Schoelkopf ***@***.***> wrote:﻿
Closed #1054 as completed.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

This is currently unsupported, but we would like to enable this in the future! (and do so efficiently). 

This would be a fairly major feature, but if you are interested in discussing it, then I would be happy to do so.
Repository: EleutherAI/lm-evaluation-harness
Issue: Does lm-evaluation-harness support AWQ quantized model testing?
Root cause: Unimplemented Feature Gap

Hi I noticed that Harness supports models that quantized with GPTQ, but it seems that Harness is not sympathetic to AWQ quantized model. Is there any solution to gauge the accuracy of AWQ quantized model with different evaluation dataset? Thanks

Hello! I believe that this is now supported, via `vllm` 's AWQ features, in the `big-refactor` branch (upcoming version release). 

Is there other AWQ or other quantization libraries that are often used, that you would want to see supported?

> GPTQ

@haileyschoelkopf  Hi, can I kindly know how to combine GPTQ and vllm together? I would like to load a model in 4-bit with VLLM. Maybe work with the command "--quantization awq"?

Hi @milliemaoo , I don't know if VLLM supports GPTQ quantization yet. (https://docs.vllm.ai/en/latest/models/engine_args.html#cmdoption-quantization)

If they don't support it, we don't support it yet either. If it's a recent addition, you can set vllm's quantization option via, for example, `--model vllm --model_args quantization=awq`, and should be able to use GPTQ via vllm with this flag.
Repository: EleutherAI/lm-evaluation-harness
Issue: AttributeError: can't set attribute 'pad_token'
Root cause: External Dependency Breakage

Hello! Thank you for your contribution. Currently, I have encountered the following error while testing my fine tuned model. What is the reason for this? How should I solve it? Looking forward to your reply!

```
Traceback (most recent call last):
  File "/home/path/lm-evaluation-harness/main.py", line 105, in <module>
    main()
  File "/home/path/lm-evaluation-harness/main.py", line 69, in main
    results = evaluator.simple_evaluate(
  File "/home/path/lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
  File "/home/path/lm-evaluation-harness/lm_eval/evaluator.py", line 76, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/path/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/path/lm-evaluation-harness/lm_eval/models/huggingface.py", line 211, in __init__
    self.tokenizer = self._create_auto_tokenizer(
  File "/home/path/lm-evaluation-harness/lm_eval/models/huggingface.py", line 539, in _create_auto_tokenizer
    tokenizer = super()._create_auto_tokenizer(
  File "/home/path/lm-evaluation-harness/lm_eval/models/huggingface.py", line 375, in _create_auto_tokenizer
    tokenizer.pad_token = tokenizer.eos_token
AttributeError: can't set attribute 'pad_token'
```
And the command is 
```
CUDA_VISIBLE_DEVICES=0 python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=THUDM/chatglm3-6b,peft=path \
    --tasks gsm8k \
    --device cuda:0
```

Hi! This no longer occurs on the new `main` branch:

```
lm_eval --model hf --model_args pretrained=THUDM/chatglm3-6b --tasks gsm8k
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Eval Harness Refactor Help

We recently pushed v0.4.0 of the LM Eval Harness (previously referred to as the `big-refactor` branch). Large refactors always come with some transition pains, so the purpose of this issue is to document issues people have and what the best solutions are. This post will be continuously updated with recommendations for fixing common problems.

**To update your local copy of this repo**
```
git branch -m big-refactor main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a
```

**For people using a fork of our library**
Before doing the above, go to Settings. In the Default branch section, select `Switch to another branch` and choose `big-refactor`. Then select `Rename branch` to and set it to `main`.

To update your local clone of this repository:

```
git branch -m big-refactor main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a
```

Hi all! What should we use instead of `lm_eval.evaluator.simple_evaluate` ?

That should still work! But is now also accessible in lm_eval.simple_evaluate() directly

Closing this as v0.4.0 has been live for a while and we just pushed v0.4.1 !

Further questions can be directed to Issues (use the 'asking questions' label) or to EleutherAI discord's #lm-thunderdome channel.
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for model instance in `HFLM.pretrained` argument
Root cause: Unimplemented Feature Gap

Hello EleutherAI team!

First of all, thanks for the amazing work on putting the `big_refactor` together and releasing it! We were eagerly waiting for it!

I have one question though. Is there anything similar to this class: https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/models/gpt2.py? This particular `HFLM` allowed a custom model to be passed to the `pretrained` argument, whereas `models.huggingface.HFLM` restricts `pretrained` to always be a string.

With the assertion, `models.huggingface.HFLM` is limited to load HF-based models from inside the class, whereas there might be some use cases to load a model from outside and pass it as an argument. For example, a transformer-based class that is not available with `AutoModelForCausalLM`, but follows the HF API.

Thanks for the attention and best regards,
Gustavo.

We were not aware we were breaking this compatibility. Can you elaborate on why the string restriction prevent loading models outside the class?

I'll push a fix for this shortly!

> We were not aware we were breaking this compatibility. Can you elaborate on why the string restriction prevent loading models outside the class?

Sure! Imagine that I have a custom class `TransformerX` which is a `torch.nn.Module` and have support for standard methods `lm-eval` is using, e.g., `generate(), forward(), etc`. However, it does not have support for `AutoModelForCausalLM`.

I used to use `models.gpt2.HFLM` as a hacky alternative because it didn't have the string assertion, but with the current `models.huggingface.HFLM`, the `pretrained` argument + string assertion will always require it to be used within the `AutoModelForCausalLM`. With the latter, models will always be loaded inside the class and no `torch.nn.Module` instances could be passed from outside the constructor.

This could be a really specific use case, but I just wanted to understand if there was some limitation on your side 😄. No worries about it though, we can always create a wrapper on our side.



See #1069 for a re-added version of this! It does however touch a lot of the initialization code, since our current HF implementation is more like those of `hf-causal-experimental` and `hf-seq2seq` combined than the old `master` `hf` / `gpt2` impls.

Note that if you want to do distributed, be wary of the code that handles data-parallel with Accelerate in `evaluator.py`--so currently passing a `PreTrainedModel` only works when `lm.world_size = 1`. 

If you have a use case that uses `lm_eval` as imported in an external library to use your pre-initialized model, and you run into difficulties, I'd love to hear about it! We would ideally want `evaluate()` to be drop-in for any distributed training loop.
Repository: EleutherAI/lm-evaluation-harness
Issue: Is there a current way to run lm-eval against a self-hosted inference server?
Root cause: Documentation Deficiency

We are interested in trying to run lm-eval on a low-resource machine and have it talk to models on a self-hosted inference server. We are not bound to any specific inference server, but some that we are interested in are vLLM, TGI, and ray-llm.

Is there a current way to do this out of the box? It looks like the `big-refactor` branch has support for loading models with vLLM, but only in the same process as the evaluation runner (which would require GPU resources on the machine). 

One option I was thinking about was to extend the `LLM` base class and implement bindings to one of our self-host inference servers. But I'm not sure if that is necessary if the library already supports that capability.

Thanks for the help!

Hi! vLLM does have a [drop in replacement](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#using-openai-completions-api-with-vllm:~:text=8000/v1/models-,Using%20OpenAI%20Completions%20API%20with%20vLLM,-%23) for the OpenAI API server which should work with minimal [modifications](https://github.com/EleutherAI/lm-evaluation-harness/blob/f0b9649146b30c966bf7fa159e3ad45133acc672/lm_eval/models/openai_completions.py#L58) by passing in the `base_url` and correcting the tokenizer to match. You could also use it as a template to support any custom server.

Would be interested to see how well this works!

I have a PR in several steps to address this, the first step is making sure the completions API works. It's here: https://github.com/EleutherAI/lm-evaluation-harness/pull/1141


Merged and resolved! https://github.com/EleutherAI/lm-evaluation-harness/pull/1174
Repository: EleutherAI/lm-evaluation-harness
Issue: Add task variants replicating Llama 1 / 2 evaluation numbers
Root cause: Documentation Deficiency

In some cases, Llama 1 (and 2, whose eval setups at times differ) paper results are not replicable by our implementations due to Meta’s custom undisclosed prompts or prepended task descriptions. 

However, for some tasks like Triviaqa, we have successfully found setups / reverse engineered prompts. Where we have done this we should add documentation and variants of the tasks for ease of use.

@haileyschoelkopf Is this documented yet somewhere? Even if it's in an informal capacity (eg a branch, a Google doc etc)
Repository: EleutherAI/lm-evaluation-harness
Issue: Genericize Arguments
Root cause: Architectural Constraint

Python allows us to pass arguments to [sub/super]classes elegantly by using the `*args` and `**kwargs` arguments. Right now we need to manually add support for every argument not only of our library but also every library that we call to build a LM. For example, we currently don't support `inject_fused_attention=False` despite the fact that `transformers` and `GPTQ` do.

This is best practices in Python and will save us substantial amounts of time and code by offloading work that other libraries are already doing to those libraries.

I would abstractly like to take this on but probably won't have time to do so this month due to NeurIPS + taking time off.

This is addressed in #1096
Repository: EleutherAI/lm-evaluation-harness
Issue: Upstream Mamba integration
Root cause: Unimplemented Feature Gap

Mamba, a new SSM architecture for language modeling, has released alongside pretrained models and an LM-Eval-Harness integration: 

https://github.com/state-spaces/mamba/blob/main/evals/lm_harness_eval.py

If Mamba won't be upstreamed to HF (any time soon) due to the custom kernels involved, we should upstream this LM class integration and point to it as a great example of using `lm_eval` externally.



Hi everyone,

Looks like there are hf-compatible mamba models now (https://huggingface.co/state-spaces/mamba-2.8b-hf).

However, I cannot use them either when using --model hf or --model mamba_ssm. Are there any plans to support these?

Hf-compatible mamba models would allow much more research into finetuning/PEFT. Having support for them in lm-evaluation-harness might be crucial.

Thanks!

Hi @BAJUKA ! 

Can you share what happens (e.g error and full traceback) if you try to run the HF versions of Mamba models using `--model hf` (ideally in a new Github issue)? These should be usable in the HF model type now that they've been added to `transformers` but if there are any compatibility issues, would be happy to fix.
Repository: EleutherAI/lm-evaluation-harness
Issue: Verify Stopsequences Don't Impact Scores
Root cause: Validation Gap

We should check on all tasks to ensure that they do not use stop sequences that might cut off valid answers too early, such as happened with DROP. We've already done so for some tasks such as GSM8k as of the new release, but should look at other generation-based tasks.

A notable culprit is using `\n` as a stop sequence when `\n\n` would be more correct.
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for finetuning

Hi, I'm currently trying to extend `lm-evaluation-harness`  to support finetuning on datasets easily. I'd want the API to look something like this:

```
lm_finetune --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --peft_type lora
    --output_dir './trained_models/gpt-lora'
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8
```

A new finetuning framework would be separate from `lm-evaluation-harness`. I'd greatly appreciate pointers on how I could  implement this; would be happy to contribute a PR!

Hi! We are not currently planning to support finetuning and would consider it out of scope for the library.

If, however, you'd like to create this on your own, I'd recommend integrating a call to `lm_eval.evaluate()` or `lm_eval.simple_evaluate()` into your training loop, and possibly sourcing your training data from the content of the requests created when calling `task.build_all_requests()`, in order to train on a identically-formatted copy of the training set of a given dataset.

Hi! I tried this and just would like to share that if using build_all_requests(), this will also build the requests with incorrect completions. This makes a ton of sense for lm_eval true purpose, but if you want to use it for fine-tuning then you will end up fine-tuning with incorrect data.
Repository: EleutherAI/lm-evaluation-harness
Issue: HF generations - discrepancy with Llama bsz > 1
Root cause: Algorithmic Error

There is a large inconsistency in the outputs — beyond what you would expect with normal batching processes — with, and without batch , on HF. For `gsm8k` on `Llama-2-7b` the exact_match accuracy is `0.08` with `batch=8` and `0.135` with `batch=1`. For reference the acc is `0.13` using vLLM. 

I suspect it's related to the padding and/or masking logic as many of the [instances](https://gist.github.com/baberabb/ed02df72ebb051a0ac7c57d8486f21c0) exclusively contain just the `<unk>` token. 

From @baberabb on discord: 

`self.tokenizer.pad_token_id = self.tokenizer.unk_token_id` instead of `self.tokenizer.pad_token_id = self.tokenizer.eos_token_id` fixes this. Addressed in #1096 .
Repository: EleutherAI/lm-evaluation-harness
Issue: [FEATURE REQUEST] Can we have HumanEval+ benchmark?
Root cause: Unimplemented Feature Gap

I think there are many good models now but it seems this `lm-eval` is lacking `HumanEval+` for testing the coding skill of the model.

Tracking progress towards code benchmarks in #1157 ! Closing this as duplicative of that tracker issue but please feel free to move discussion of desired code benchmarks / features into that issue. 

We've historically not supported codegen tasks due to not wanting to endorse unverified model-generated code execution but we will be supporting them in future after requiring offline scoring.