zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Merged models giving error during evaluation
Root cause: Validation Gap

I have merged the models and while using lm-evaluation I am getting following error:

`Loading checkpoint shards:   0%|                                                                                                     | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/root/miniconda3/envs/py3.10/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/workspace/axolotl/llm_exp/lm-evaluation-harness/lm_eval/__main__.py", line 341, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/workspace/axolotl/llm_exp/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/workspace/axolotl/llm_exp/lm-evaluation-harness/lm_eval/evaluator.py", line 180, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/workspace/axolotl/llm_exp/lm-evaluation-harness/lm_eval/api/model.py", line 134, in create_from_arg_string
    return cls(**args, **args2)
  File "/workspace/axolotl/llm_exp/lm-evaluation-harness/lm_eval/models/huggingface.py", line 202, in __init__
    self._create_model(
  File "/workspace/axolotl/llm_exp/lm-evaluation-harness/lm_eval/models/huggingface.py", line 540, in _create_model
    self._model = self.AUTO_MODEL_CLASS.from_pretrained(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3562, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3989, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/transformers/modeling_utils.py", line 813, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "/root/miniconda3/envs/py3.10/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 345, in set_module_tensor_to_device
    raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([32000, 4096]) in "weight" (which has shape torch.Size([32002, 4096])), this look incorrect.`

Hi!

Might need some more information to help here but it looks like there are 2 added tokens in your model's vocabulary, and either your saved model hasn't retained these 2 extra tokens, or its config file isn't set up to expect the extra tokens.

Hi, the error was due to a padded token and occurred during the merging of the models. I have removed those tokens and set the vocab size to 32000, it's working fine now.
Repository: EleutherAI/lm-evaluation-harness
Issue: Question: evaluate the custom torch model
Root cause: Documentation Deficiency

Hi, 

Give a simple model as example, and save it to local. Is it possible to use `lm_eval` to do evaluation as usage of the hf model?

> lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

```python

# Define model
class TheModelClass(nn.Module):
    def __init__(self):
        super(TheModelClass, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

# save the entire Model
model = TheModelClass()
torch.save(model, loacal_path)
```

Hi, 

We currently support passing already-initialized `pretrained` models to the HFLM class, if they share much of an interface.

Else, if your model requires custom logic to interact with it, and doesn't implement most of the same interface as a Huggingface pretrained model class or `trust_remote_code` model on Huggingface, I'd recommend extending the LM or HFLM classes to implement the different request types (described here: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md) for your model

> Hi,
> 
> Give a simple model as example, and save it to local. Is it possible to use `lm_eval` to do evaluation as usage of the hf model?
> 
> > lm_eval --model hf 
> >     --model_args pretrained=EleutherAI/gpt-j-6B 
> >     --tasks hellaswag 
> >     --device cuda:0 
> >     --batch_size 8
> 
> ```python
> # Define model
> class TheModelClass(nn.Module):
>     def __init__(self):
>         super(TheModelClass, self).__init__()
>         self.conv1 = nn.Conv2d(3, 6, 5)
>         self.pool = nn.MaxPool2d(2, 2)
>         self.conv2 = nn.Conv2d(6, 16, 5)
>         self.fc1 = nn.Linear(16 * 5 * 5, 120)
>         self.fc2 = nn.Linear(120, 84)
>         self.fc3 = nn.Linear(84, 10)
> 
>     def forward(self, x):
>         x = self.pool(F.relu(self.conv1(x)))
>         x = self.pool(F.relu(self.conv2(x)))
>         x = x.view(-1, 16 * 5 * 5)
>         x = F.relu(self.fc1(x))
>         x = F.relu(self.fc2(x))
>         x = self.fc3(x)
>         return x
> 
> # save the entire Model
> model = TheModelClass()
> torch.save(model, loacal_path)
> ```

Hi! Did you solve it? I get in trouble with it.
Repository: EleutherAI/lm-evaluation-harness
Issue: Passing a config file to wandb using CLI
Root cause: Unimplemented Feature Gap

Hi. I am using CLI to run lm eval harness and I logged into my wandb account. The problem is only the results are displayed there. But the models I am evaluating are custom generated so I want to attach a config file to those results in wandb. The config file is just a python dict named "config" or a json file saved at "config_loc". Is there a way to pass this config file to wandb? I have tried to pass the save location at --wandb_args but it didn't work. Chatgpt suggested some methods but they didn't work either.
Repository: EleutherAI/lm-evaluation-harness
Issue: Type error on stderr computation of accuracy
Root cause: Validation Gap

Hello,

I'm trying to evaluate on the `ai2_arc` task group, but I keep getting this exception.

```python
File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/evaluator.py", line 234, in simple_evaluate
    results = evaluate(
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/evaluator.py", line 514, in evaluate
    ] = lm_eval.api.metrics.pooled_sample_stderr(stderrs, sizes)
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/api/metrics.py", line 462, in pooled_sample_stderr
    sum([(size - 1) * stderr**2 * size for size, stderr in zip(sizes, stderrs)])
  File "/home/renatolg/.local/lib/python3.10/site-packages/lm_eval/api/metrics.py", line 462, in <listcomp>
    sum([(size - 1) * stderr**2 * size for size, stderr in zip(sizes, stderrs)])
TypeError: unsupported operand type(s) for ** or pow(): 'list' and 'int'
```

Some quick debugging shows `stderr = [[], []]`, while `results` in `evaluator.py:514` has value

```python
results = defaultdict(<class 'dict'>, {'arc_easy': {'acc,none': 0.7133838383838383, 'samples': 2376, 'acc_stderr,none': [], 'acc_norm,none': 0.6586700336700336, 'acc_norm_stderr,none': []}, 'arc_challenge': {'acc,none': 0.3890784982935154, 'samples': 1172, 'acc_stderr,none': [], 'acc_norm,none': 0.4087030716723549, 'acc_norm_stderr,none': []}, 'ai2_arc': {'acc,none': 0.6062570462232244}})
```

It seems harness is having trouble computing the standard deviation of accuracy when faced with several tasks.

Is this a bug or am I doing something wrong here? Is there an option to disable these statistics and only get accuracy or do I have to manually change the code so it skips this function?

Thanks

What is the command you are running?

Can't seem to reproduce with `lm_eval --model hf --tasks ai2_arc` , is this issue persisting?

Are there any changes you've made to the code?

I have managed to pinpoint exactly which flag causes this error. Here's a reproducible minimal example:

```python
import lm_eval, torch

mgr = lm_eval.tasks.TaskManager()
lm_eval.simple_evaluate(model=lm_eval.models.huggingface.HFLM(pretrained="google/gemma-2b", dtype=torch.float32),
                        device="cuda", tasks=["ai2_arc"], limit=10, bootstrap_iters=0)
```

The error disappears when omitting the option `bootstrap_iters=0`.

It now makes sense why the exception is being raised. However, it is unclear from documentation what `bootstrap_iters` is supposed to be without going through the code.

#1789 should fix this--let me know if not!
Repository: EleutherAI/lm-evaluation-harness
Issue: Pass doc to a custom metric
Root cause: Unimplemented Feature Gap

Good afternoon I encountered a problem when writing a custom metric for `output_type: generate_until`. As I noticed, all metrics receive the `items` parameter as input. Inside it are the answer llm and target. Is it possible to forward `doc` along with them? I see similar logic in the code for the `acc_all` metric. 


Bump on this, I'd also like an answer!

@BelousovIM I have gotten around this by using the utils.py function to "package" information from the `doc` into one `target`, to be unpacked later.  If you want, you could probably use `json.dumps` to package the relevant doc fields into a string, and load that in the aggregation/metric function, which is what I did.

In my case I used these [example](https://github.com/EleutherAI/lm-evaluation-harness/blob/d32ce5cf11608d59602e498d51f38d7d01f9d0ff/lm_eval/tasks/agieval/aqua-rat.yaml#L15) and it does work.
Repository: EleutherAI/lm-evaluation-harness
Issue: Single doc results of model inference not found 
Root cause: Documentation Deficiency

Hi~Referring to the doc, I think if I add '--output' and '--log_samples' argument I could get every inference result from one doc. I have executed the basic demo with these two arguments like this:`lm_eval --model hf     --model_args pretrained=EleutherAI/gpt-j-6B     --tasks hellaswag     --device cuda:0     --batch_size 8 --output_path /home/work/ais-harness-data-test/harness-res/0418/ --log_samples`
I have found 'pretrained\=EleutherAI__gpt-j-6B_hellaswag.jsonl' under the result output dir but I am confused of the result. The attachment is a result demo of one doc.  
[result_demo.txt](https://github.com/EleutherAI/lm-evaluation-harness/files/15019619/result_demo.txt)
I have found the dataset doc details and some results after computation. However, I do not find the inference result of the model. Has it been provided? If not, is there any way to obtain it or will it be supported in the future. I am looking forward to your reply, thanks a lot
Repository: EleutherAI/lm-evaluation-harness
Issue: GSM8K Accuracy Gap on Jamba
Root cause: Documentation Deficiency

Hi, thanks for this wonderful repo.
I have met a problem when running the evaluation task of GSM8K on the model Jamba. I ran both gsm8k and gsm8k_cot but got much lower accuracy than the released result.
I got:
|  Tasks  |Version|     Filter     |n-shot|  Metric   |Value |   |Stderr|
|---------|------:|----------------|-----:|-----------|-----:|---|-----:|
|gsm8k_cot|      3|strict-match    |     3|exact_match|0.2109|±  |0.0362|
|         |       |flexible-extract|     3|exact_match|0.2344|±  |0.0376|

But the result listed in the paper is 59.9 for 3-shot CoT. Could you please help to check? Thanks a lot.

I recommend asking the Jamba authors for more details on how they evaluated their models. Typically gaps like this are explained by undisclosed custom prompting in the paper.

@Mandy-77 

could you share the logs / outputs for your 3-shot `gsm8k_cot` run? because we hardcode the original CoT paper's 8-shot prompt in `gsm8k_cot` (https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/gsm8k/gsm8k-cot.yaml) the setting of `num_fewshot: 0` in its config file is meant to be un-overrideable since adding extra fewshots to this would mess with the formatting.

Aside from this, I'd recommend editing `gsm8k_cot`'s prompt to be 3-shot and seeing what you get. As Stella mentioned, if you can't reproduce it, you probably just need to ask the Jamba authors for eval details.
Repository: EleutherAI/lm-evaluation-harness
Issue: Question: implement and use a task outside of the repository?
Root cause: Configuration Error

Hello, can I implement a custom task (as a task folder) outside of the repository, and then use `lm_eval` to run it? This is useful if I want to pass the task around to others who have this library installed, without them having to put the task folder into the library installation location. 

There is an `--include_path` argument you can use to pass it around

--include_path does not work, as of today

@ZhangAustin could you file an issue for this? I'll investigate ASAP.
Repository: EleutherAI/lm-evaluation-harness
Issue: Accuracy fluctuation with same model and weights
Root cause: Configuration Error

Hi, I fine-tuned a model and evaluated it on a subset of the task one day before. After some time, I fine-tuned a few more models and tried to evaluate the old model. I found a 2-3% accuracy difference. I tried to use the same version but had difficulty reproducing the experiment. Previously, I was using an H100 80GB SXM5 for evaluation, and this time I evaluated on an A100. What could be the underlying reason? 

Is it the GPU version, floating-point precision, memory bandwidth, or other hardware-specific factors? I guess we are setting up the same random seed, so is there any other randomness in the model initialization that could affect the accuracy? Or could some changes made in the LM evaluation harness be the reason?

If you're changing the amount of VRAM on your GPUs, my go to thing to check would be if you're using a fixed batch size or an automatically determined one. Changing the batch size can result in variation in results for very short evaluation as it changes which examples are a shown in-context.

What task did you see this happen on, out of curiousity? How do the score differences stack up to the reported stderrs?

>  floating-point precision
if you're running in a different precision I would definitely expect some small differences.

this and batch size can affect results. See #1645 #1625 for more discussion, unfortunately, without a framework + hardware explicitly designed around bitwise reproducibility, it is very very hard to guarantee these things. 

If your loglikelihoods are massively different though (as can be checked by looking at samples from `--log_samples`) then definitely let us know. Some small variation is expected ; but massive changes are not.

>  as it changes which examples are a shown in-context.
if you observe this happening, please file a bug report! It is not meant to occur, due to the random seeding.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add support to azure-openai deployed models
Root cause: Unimplemented Feature Gap

Apparently Azure OpenAI deployed models can't be reached with the `openai.OpenAI` client instead it requires the `openai.AzureOpenAI` client which has a slightly different interface (see [link](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/switching-endpoints) for further details).

I was able to do what I needed by adding this rough solution in the "lm_eval/models/openai_completions.py":

```python
@register_model("azure-openai-chat-completions")
class AzureOpenaiChatCompletionsLM(OpenaiChatCompletionsLM):
    def __init__(
        self,
        model: str = os.getenv("AZURE_OPENAI_DEPLOYMENT_NAME"),  
        base_url: str = os.getenv("AZURE_OPENAI_ENDPOINT"),
        api_version: str = "2023-12-01-preview",
        truncate: bool = False,
        **kwargs,
    ) -> None:
        super().__init__()
        try:
            import openai  # noqa: E401
        except ModuleNotFoundError:
            raise Exception(
                "attempted to use 'openai' LM type, but package `openai` or `tiktoken` are not installed. \
    please install these via `pip install lm-eval[openai]` or `pip install -e .[openai]`",
            )
        self.model = model
        self.base_url = base_url
        self.truncate = truncate
        self.client = openai.AzureOpenAI(azure_endpoint=base_url, api_version=api_version)
```

Therefore, I believe that a more general/better thought solution won't require much effort.



Thanks Bruno. What task were you evaluating? I get this error when I run it with lambada_openai:
  NotImplementedError: No support for logits.
I've heard that the azure code slightly lag the openai code so may not be surprise that something hasn't been implemented yet

wow its wonderful

vote for this!!

This OpenaiChatCompletionsLM has not been defined and needs to be rewritten？ @bcarvalho-via 

> This OpenaiChatCompletionsLM has not been defined and needs to be rewritten？ [@bcarvalho-via](https://github.com/bcarvalho-via)

AFAIK, openai api & axure openai api are slightly different, just rewrite above part will make compatiblility to the azure openai api.
Repository: EleutherAI/lm-evaluation-harness
Issue: Process hangs when using `tensor_parallel_size` and `data_parallel_size` together
Root cause: Resource Mishandling

Hello,

I noticed that my process hangs at [`results = ray.get(object_refs)`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/vllm_causallms.py#L220) when I use `data_parallel_size` as well as `tensor_parallel_size` for vllm models.


For example, this call would hang.
```
lm_eval --model vllm --model_args pretrained=gpt2,data_parallel_size=2,tensor_parallel_size=2 --tasks arc_easy --output ./trial/  --log_samples --limit 10
```

These would not. 
```
lm_eval  --model vllm --model_args pretrained=gpt2,data_parallel_size=1,tensor_parallel_size=2 --tasks arc_easy --output ./trial/  --log_samples --limit 10
```

```
lm_eval  --model vllm --model_args pretrained=gpt2,data_parallel_size=2,tensor_parallel_size=1 --tasks arc_easy --output ./trial/  --log_samples --limit 10
```


Does anyone else face similar problem?

Hi! What version of vLLM are you running with?

@baberabb has observed some problems like this before with later versions ( >v0.3.3 I believe) of vllm. 

I am on vllm `0.3.2`.

Is this a vllm problem? Should I be raising an issue on that repo?

Hey. Have you tried caching the weights by running with DP=1 until they are downloaded? I found it prone to hang with DP otherwise. 

Yes, the weights are cached. The process is hanging after  `llm.generate` returns results.

> Yes, the weights are cached. The process is hanging after `llm.generate` returns results.

hmm. It's working for me with `0.3.2`. Have you tried running on a fresh virtual environment?


Just tried it on a separate server and new env still face the same issue. What version of ray do you have? Mine is `ray==2.10.0` 

> Just tried it on a separate server and new env still face the same issue. What version of ray do you have? Mine is `ray==2.10.0`

Probably the latest one. I installed it with `pip install -e ".[vllm]"` on runpod with 4 GPUs.

I had the same problem on vllm==0.8.2 and fixed it by downgrading to vllm==0.7.3


vllm-0.7.3 still got the same problem.
Repository: EleutherAI/lm-evaluation-harness
Issue: Need wide & small task for fast evaluation
Root cause: Unimplemented Feature Gap

Hi all,

I want to do a large-ish study on quanting methods and their effect on model performance. For this, I need an evaluation that's (i) "wide" (ie covers a broad set of tasks / topics) and small (so it's quick and cheap to run).

Iiuc, currently there is no task for that.

I suggest we add the [dharma2 dataset](https://huggingface.co/datasets/pharaouk/dharma-2) (samples from 8 tasks, incl MMLU ; 300 examples in total); or alternatively [big-bench lite](https://huggingface.co/datasets/bigbench) (samples from 24 tasks).

I'll fork this repo and add dharma2. If there's interest, I'd be happy to submit a PR. 

We always welcome more task PRs. Additionally, if there's a broad task that meets your needs you can use the `--limit` flag
Repository: EleutherAI/lm-evaluation-harness
Issue: Scrolls individual tasks are not available
Root cause: Unimplemented Feature Gap

Example:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/56915448/479761a5-5171-434e-88c4-4f56e1abec82)

Also, they are not available in the tasks list:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/56915448/fbe974bc-575f-42af-b112-669ba740a8b2)
Repository: EleutherAI/lm-evaluation-harness
Issue: Error when running lm_eval with piqa task with EleutherAI/gpt-j-6b
Root cause: Environment Incompatibility

# Descrption
An error occurs when running `lm_eval` with the piqa task using EleutherAI/gpt-j-6b as follows:

`UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte`

# Steps to Reproduce:

1. Start evaluation

```
lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6b,parallelize=True,load_in_4bit=True,peft=nomic-ai/gpt4all-j-lora \
    --tasks piqa \
    --device cuda:0
```


hii sorry for the bothering. 
how to solve this?

Thanksss.

lm_eval can not preprocess piaq automatically, you can visit https://huggingface.co/datasets/piqa and download piqa.py, preprocess the piqa into .cache dir via datasets package before you run lm_eval.

"I encountered the error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte while using `hellaswag`. My solution is the following:"

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/33996924/38c1ef05-6295-46a5-9ae2-068348d0e995)
Repository: EleutherAI/lm-evaluation-harness
Issue: [Feature Request] pre-built Docker image support
Root cause: Unimplemented Feature Gap

Is there any plan to support pre-built docker image on dockerhub or github package?

I think It will be very helpful and become more accessible to a lot of people.

We'd be glad to support this! It's a great idea.

Won't be currently prioritizing it so if anyone is able to contribute this we'd welcome contributions. Else, will put it on the todo list. 

An official lm-evaluation-harness Docker image would be great. I just found that BigScience hosts the following Docker image: https://github.com/orgs/bigcode-project/packages/container/package/evaluation-harness

+1

+1

I’d love to take this on! Let me know if I can start working on this. @haileyschoelkopf
Repository: EleutherAI/lm-evaluation-harness
Issue: [Feature Request] Metrics that require knowledge of input. 
Root cause: Unimplemented Feature Gap

For generate_until output type tasks, I would like to be able to register metrics that are computed using the input prompt to the LLM as well as as the reference and prediction. 

I have a niche use case for this but I think it would be useful more broadly. For example LLM-as-Judge style metrics would require the input prompt also. 

Also happy to submit a PR if you can point me in the right direction. 

Have you been able to solve it? I am encountering the same problem