zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Runtime Error with QGPT for EleutherAI/gpt-j-6B
Root cause: Interface Contract Mismatch

Hi all,
after using auto-gptq to generate quantized models for EleutherAI/gpt-j-6B; 
I load this model into the framework using this initializer: 
`
model = AutoGPTQForCausalLM(path_to_quantized_model, device_map=None, trust_remote_code=True, use_safetensors=True, use_triton=False).
`

when trying to run simply_evaluation on this model with hellaswag task; i get this error

`
Traceback (most recent call last):
  File "/mnt/task_runtime/main.py", line 200, in <module>
    main()
  File "/mnt/task_runtime/main.py", line 153, in main
    lm(**data)
  File "/mnt/task_runtime/lm_bolt/models/huggingface.py", line 574, in __call__
    return self.model(*args, **kwargs)
  File "/opt/python/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/python/lib/python3.9/site-packages/auto_gptq/modeling/_base.py", line 418, in forward
    return self.model(*args, **kwargs)
  File "/opt/python/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/python/lib/python3.9/site-packages/transformers/models/gptj/modeling_gptj.py", line 854, in forward
    transformer_outputs = self.transformer(
  File "/opt/python/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/python/lib/python3.9/site-packages/transformers/models/gptj/modeling_gptj.py", line 689, in forward
    outputs = block(
  File "/opt/python/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/python/lib/python3.9/site-packages/transformers/models/gptj/modeling_gptj.py", line 309, in forward
    attn_outputs = self.attn(
  File "/opt/python/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/python/lib/python3.9/site-packages/auto_gptq/nn_modules/fused_gptj_attn.py", line 206, in forward
    attn_output = F.scaled_dot_product_attention(
RuntimeError: Expected query, key, and value to have the same dtype, but got query.dtype: float key.dtype: float and value.dtype: c10::Half instead.
`


what is the issue here?

Hi! Do you have a command that would replicate this (which pretrained model if available on HF, etc?) Or are you running the library within your own codebase?

Can take a look if I'm able to replicate and you're running via `main.py`! It seems as though this might have something to do with `trust_remote_code` from HF? https://github.com/huggingface/text-generation-inference/issues/372
Repository: EleutherAI/lm-evaluation-harness
Issue: How to deal with load_in_8bit error?
Root cause: Environment Incompatibility

Hi, 
I'm trying to evaluate gpt2 to hellaswag dataset.

but, I got unexpected key word error 'load_in_8bit'

<img width="517" alt="스크린샷 2023-06-20 오후 4 22 02" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/78716763/d1ea82ab-af52-45bd-a62e-805673285bdc">

how can I deal with this error?

What is the exact command you are running? Are you sure you’re putting that flag in the right place?

> What is the exact command you are running? Are you sure you’re putting that flag in the right place?

<img width="451" alt="스크린샷 2023-06-21 오후 5 50 29" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/78716763/2bb18bb5-f68f-4fb3-a32f-223f2d6ef355">

This is my command. I tried to evaluate opt-350m model. This also happen with other models(gpt2 etc.)

There is no `load_in_8bit` there.

<img width="702" alt="스크린샷 2023-06-21 오후 9 07 54" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/78716763/f240887d-0303-4e96-a65c-a821e19847a2">

yes, but I get this 'load_in_8bit' error if I run that code.

How can I fix this?

I do not. Have you confirmed that you're up to date with the master branch and on the latest commit?

> I do not. Have you confirmed that you're up to date with the master branch and on the latest commit?

Yes. It is master branch on the latest commit. Anyone else struggling with this problem?

Hi! What version of `transformers` are you using? Can you report that, then run `pip install --upgrade transformers` and try to run again?

if you're trying to run a model without quantizing--this may be the culprit. If you're trying to run a quantized model-- `--model hf-causal-experimental` is more feature complete for quantization options at the moment.

Closing as I can't replicate, please reopen if you are still experiencing the issue on the newest `master` commit!
Repository: EleutherAI/lm-evaluation-harness
Issue: create single hendrycksTest task to evalulate all hendrycksTest-{sub} tasks
Root cause: Unimplemented Feature Gap

Currently, there are 57 hendrycksTest-{sub}  sub-tasks, and it is super inconvenient to evaluate all of them use a single accuracy score. Is it possible to create a new "hendrycksTest" task to average over all 57 hendrycksTest-{sub} tasks? 

strongly agree!!!

> Currently, there are 57 hendrycksTest-{sub} sub-tasks, and it is super inconvenient to evaluate all of them use a single accuracy score. Is it possible to create a new "hendrycksTest" task to average over all 57 hendrycksTest-{sub} tasks?

Hey,bro.Look at the create _all_tasks() method of hendrycks_test.py,and you can set the task to hendrycksTest-*

@huayicong23 This is great!

Hello! I've met the same problem, how to mix them to just a task to compute the average score?
Repository: EleutherAI/lm-evaluation-harness
Issue: max_length is not in effect
Root cause: Validation Gap

Hi,

`max_length` doesn't seem to be applied. 

I tested it on gsm8k (greedy_until) and hellaswag (loglikelihood) by adding max_length in arguments like below
```
python main.py --model hf-causal-experimental --model_args pretrained=openlm-research/open_llama_13b,use_fast=False,dtype=bfloat16,max_length=512 --tasks hellaswag --device cuda --output_path results/openllama-13b/hellaswag_25-shot.json --batch_size auto --no_cache --num_fewshot 25
```

I print the shape of input tensors and they are not truncated. Should these tokenization lines ([L407-417](https://github.com/EleutherAI/lm-evaluation-harness/blob/72b7f0c00a6ff94632c5b873fc24e093ae74fa47/lm_eval/models/huggingface.py#L409)) include `truncation=True`?

Best,
Rui

Hmmm. Is there a reason we should want to use max_length? When would someone want to use this?

We handle `max_length` manually in the new `main`, previously `big-refactor` branch.
Repository: EleutherAI/lm-evaluation-harness
Issue: ValueError bug in `greedy_until`  "too many values to unpack"
Root cause: Validation Gap

Some tokenizers (Llama, Alpaca) return more than one token for "\n" even with `add_special_tokens=False`. It causes a value error in: https://github.com/EleutherAI/lm-evaluation-harness/blob/72b7f0c00a6ff94632c5b873fc24e093ae74fa47/lm_eval/base.py#L420

One can replace it with

```python
primary_until = self.tok_encode(until[0])[0]
```

Thanks for raising this, and apologies for the bug!

In our upcoming version release, we handle multi-token stop sequences in a more principled+unified way ([see here](https://github.com/EleutherAI/lm-evaluation-harness/blob/ef4a26f6e3bc9fe4979aa86c4fa1081e10e3d21e/lm_eval/models/huggingface.py#L275)). 

I've patched this + added a warning for the `hf-causal` model in #628 , and confirmed it doesn't crop up in the `hf-causal-experimental` case.
Repository: EleutherAI/lm-evaluation-harness
Issue: do you support to evaluate rwkv?
Root cause: Unimplemented Feature Gap

I try to run
python main.py --device cuda --no_cache --model hf-causal-experimental --model_args pretrained=/RWKV-4-Raven-7B-v12-Eng98%-Other2%-20230521-ctx8192.pth,trust_remote_code=True,use_accelerate=True --tasks truthfulqa_mc --num_fewshot 0
to evaluate RWKV, but fail with errors:
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte
and
    OSError: It looks like the config file at
'/data/leaderboard/weights/BlinkDL/RWKV-4-Raven-7B-v12-Eng98%-Other2%-20230521-ctx8192.pth' is not a valid JSON file.

Hi! We do support RWKV via Huggingface's implementation, but only the models that are converted to Huggingface's format. 

For example, I'm able to run https://huggingface.co/RWKV/rwkv-raven-7b without issue, but it seems like you're trying to pass a specific .pth file, which isn't supported by default. To evaluate this model, either you'll need to integrate it into your own `LM` subclass or you'll need to convert it to Huggingface's format, which I believe can be done via this script: https://github.com/huggingface/transformers/blob/main/src/transformers/models/rwkv/convert_rwkv_checkpoint_to_hf.py
Repository: EleutherAI/lm-evaluation-harness
Issue: [big-refactor] Question about running single gpu task on multi-gpu machine
Root cause: Validation Gap

I have a machine with 8xA100, and I tried to run on a single gpu:
```bash
python main.py \
    --model hf \
    --model_args pretrained=/path/to/model \
    --tasks super-glue-lm-eval-v1 \
    --device cuda:0 \
    --batch_size 8
```
and I got this warning:
```
[huggingface.py:102] Passed device 'cuda:0', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
```
That is not right because I didnt using accelerate or parallelize.
And if I:
```bash
CUDA_VISIBLE_DEVICES=0 python main.py \
    --model hf \
    --model_args pretrained=/path/to/model \
    --tasks super-glue-lm-eval-v1 \
    --device cuda:0 \
    --batch_size 8
```
I got this error:
```
Traceback (most recent call last):
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/main.py", line 131, in <module>
    main()
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/main.py", line 80, in main
    results = evaluator.simple_evaluate(
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/utils.py", line 318, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/evaluator.py", line 90, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/api/model.py", line 117, in create_from_arg_string
    return cls(**args, **args2)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/models/huggingface.py", line 90, in __init__
    device = int(device)
ValueError: invalid literal for int() with base 10: 'cuda:0'
```

Thanks for bringing the warning to our attention.

For setting inference on a specific GPU, use `--device 0` instead of `--device cuda:0`. but this also doesn't seem to apply correctly. 

Will look into this.
Repository: EleutherAI/lm-evaluation-harness
Issue: Run lm-eval for a large LLM that doesn't fit into GPU
Root cause: Documentation Deficiency

Hi,

I have a relatively smaller GPU (RTX 3060 - 12 GB of VRAM). I want to evaluate a quantization technique on a fairly large model. I have the following workflow:

1. Load the model in CPU
2. Perform layer-wise post-training quantization - by moving each layer on GPU, performing quantization, and moving it back to CPU
3. Perform lm-eval

I can do steps 1 and 2, but am getting stuck at step 3. I can't fully move the entire model to GPU (even after quantization), since it's too big for the GPU. Ideally, I would use something like `accelerate` and `max_memory` to load the model on GPU as much as possible and then offload the remaining layers to CPU. 

Is there an example of `lm_eval` for such a model with some layers on GPU and some on CPU?
Thanks!

I think the best way to do this would probably be to do your steps 1. and 2. outside the lm-eval main `evaluator.simple_evaluate()` loop, and then also load your model after it has been placed across GPU and CPU via passing the loaded HF PreTrainedModel into the HFLM class: https://github.com/EleutherAI/lm-evaluation-harness/blob/13014b2c060200eb47d7af157967f3577d7f00e7/lm_eval/models/gpt2.py#L41 , https://github.com/EleutherAI/lm-evaluation-harness/pull/601 .

For an example of how to load a model across CPU and GPU, we support this via the `hf-causal-experimental` type, by passing `device_map` and related keyword arguments to the instantiation of a Huggingface model.


These model keyword arguments are created here: 
https://github.com/EleutherAI/lm-evaluation-harness/blob/13014b2c060200eb47d7af157967f3577d7f00e7/lm_eval/models/huggingface.py#L21

Hey @haileyschoelkopf,

Thanks, I figured out a way to do this.

Actually, there's a way to move an HF model from CPU to GPU without needing to store it on the disk and reload it.

You can load a model using usual `AutoModelForCausalLM.from_pretrained` by passing the model name and state dictionary and setting `device_map="auto"`. The device map will take care of loading as many layers as possible on GPU and offload remaining to CPU. I combined this with `max_memory` parameter to leave some room on the GPU to run large sequences during eval, if needed.

Thanks for the help!
Repository: EleutherAI/lm-evaluation-harness
Issue: Are there any NLU datasets (NLU for task-oriented dialogue systems) included?

Hi, I know this question sounds stupid since NLU tasks require pre-defined ontology and potentially require at least some few-shot learning. But I just need to confirm that no NLU benchmarks, e.g. ATIS, SNIPS, NLU++ are included in this repo, right?

Hi!

We don't include any of those 3 datasets in this repo. From a glance, it seems like 2 of them include speech components if I'm looking at the right papers, in which case those would be out-of-scope for this repo, and at least one seems to be a sequence tagging dataset? I'm not aware of any good translations of sequence-tagging datasets into a prompted format that are standard, but if you are aware of such a technique in the literature, feel free to contribute these benchmarks!

Hope this helps!
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] Revamp Testing / CI pipeline
Root cause: Unimplemented Feature Gap

cc @baberabb , @ethanhs

An issue to track progress on reworking the testing to be in line with the new version.

Things we definitely want to test for are: 

- [x] Test that `evaluator.simple_evaluate` and `evaluator.evaluate()` behave as expected / their interface does not change (https://github.com/EleutherAI/lm-evaluation-harness/blob/master/tests/test_evaluator.py)
-  [x] Test that Task interface doesn't change without us intending it (https://github.com/EleutherAI/lm-evaluation-harness/blob/master/tests/test_tasks.py)
- [ ] Test that all LM subclasses function as intended
- Mock out the different completion types
- https://github.com/EleutherAI/lm-evaluation-harness/blob/0fa2c7df05cb7a96a4e42a3d888ef5de8628da66/tests/test_ggml.py is a good self-contained test of an LM
- [ ] Test some of our utils.py functions
- https://github.com/EleutherAI/lm-evaluation-harness/blob/master/tests/test_utils.py
- [ ] Test for changes to "primary" tasks supported
- https://github.com/EleutherAI/lm-evaluation-harness/blob/master/tests/test_version_stable.py : **somewhat addressed: currently we test that tasks whose files have been changed do successfully run. We should test also that inputs/outputs for this task don't change.** 

- [ ] Longer-running regression test (not done every PR unless we call for it):
- Test a task on Pythia-70m with limit=1000 , and check that the metrics reported on some core representative tasks are identical to what they should be **partially addressed: currently support a CPU-only short-running test using Pythia-70m but we don't check the metrics that this saves.**

Many of these tests already exist in the repo in some form but must be modified for any interface changes introduced. 

Closing and will open separate new issues for specific test additions / todos.
Repository: EleutherAI/lm-evaluation-harness
Issue: Multiple choice task prefers shorter answers
Root cause: Documentation Deficiency

If I'm reading the code correctly, the multiple choice task tries each choice as a completion of the question takes the answer to be the completion with the lowest sum of log likelihoods over the tokens. Do I have that right? If so, doesn't this method bias strongly toward shorter answers?

@ishmandoo I think this article may answer your question to some extent: [evaluating-mmlu-leaderboard](https://huggingface.co/blog/evaluating-mmlu-leaderboard)

Thanks for the link @SingL3 . I see this comment, but I can't find the later discussion:

> For numerical stability we gather them by summing the logarithm of the probabilities and we can decide (or not) to compute a normalization in which we divide the sum by the number of tokens to avoid giving too much advantage to longer answers (more on this later). 

> If I'm reading the code correctly, the multiple choice task tries each choice as a completion of the question takes the answer to be the completion with the lowest sum of log likelihoods over the tokens. Do I have that right? If so, doesn't this method bias strongly toward shorter answers?

This is a recurring issue with language model evaluation. We currently support two options:
1. Saying “oh well” and computing accuracy normally
2. A metric we call `acc_norm` which normalizes with respect to the length of the input. You can learn more about the exact method and why we chose the one we chose in [our blog post](https://blog.eleuther.ai/multiple-choice-normalization/).

IMO the third feasible option is to define a list of stop tokens and say that the correct answer is the answer *followed by a stop token*. However I’m personally kinda low on this and we never got around to implementing it. One of the challenges with doing so is that the list of stop tokens to consider is tokenizer (and often training dataset!) dependant.
Repository: EleutherAI/lm-evaluation-harness
Issue: how to compute MMLU average scores?
Root cause: Documentation Deficiency

Hello, I've run the model on the various tasks in MMLU and got many scores, but how to mix them and get the avg score? just add them and avg? or is there any suggestions

maybe you can read this
https://github.com/hendrycks/test/blob/master/evaluate_flan.py#L166-L174
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Big Bench Hard (BBH) and AGIEval
Root cause: Unimplemented Feature Gap

Hi,

Since Microsoft released [Orca paper](https://arxiv.org/pdf/2306.02707.pdf), we are likely to see models instruct-tuned using Orca techniques that'll most likely outperform the current crop of models such as Vicuna and Wizard.

It would be good to add some of the harder evaluation benchmarks mentioned in the Orca paper such as Big Bench Hard (BBH) and AGIEval.

Thanks!

Hi, thanks for raising this issue! If you've got the bandwidth to help contribute some of these tasks, especially chain of thought variants of our existing multiple choice tasks, that would be greatly appreciated, though no problem if not!

On the BBH side, we currently support almost all BigBench Hard tasks, which are equivalent to the *full Bigbench* versions of those tasks, not the truncated versions used in BBH -- https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/bigbench.py

I think @dmahan93 has an AGIEval implementation in their fork -- @dmahan93 , would you be interested in contributing this upstream by chance?

Yup! Was going to wait until the big refactor was done to port it over, right now it's in the old style of evaluations.

Thanks!

Hey @dmahan93 , I just wanted to let you know that the big refactor branch was merged to main. Would you still be interested in merging it upstream? I've been using your branch to do agieval so far.

@dmahan93 AGI eval would be much appreciated :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Add Mac Support
Root cause: Documentation Deficiency

I created a branch `add-mac` with the goal of adding Mac support. While I was able to iron out some obvious issues, I'm not sure what to try next. The current error code is:

```
loc("varianceEps"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/c2cb9645-dafc-11ed-aa26-6ec1e3b3f7b3/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":228:0)): error: input types 'tensor<1x147x1xf16>' and 'tensor<1xf32>' are not broadcast compatible
LLVM ERROR: Failed to infer result type(s).
zsh: abort      /opt/homebrew/bin/python3 main.py --model hf --model_args  --tasks hellaswag 
stellabiderman@RP9FP91FMV lm-evaluation-harness % /opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
```

Regarding my system, I have:
```
Python 3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'2.0.1'
>>> torch.backends.mps.is_available()
True
```

Adding `,dtype=float32` to `model_args` allows it to run. It seems that torch's autocasting doesn't work right on MPS, so I'll add some warning flags and submit a PR tomorrow.

This is implemented in #680 . However, we should document this `--device mps` option in the readme, so leaving this open.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] bug(maybe) in Instance
Root cause: Documentation Deficiency

Hey. I'm working on the CI tests for the tasks classes and getting an error running the following:

```
import lm_eval.tasks as tasks
# Get task and task class(callable)
x = [x for x in tasks.TASK_REGISTRY.items() if x[0] == 'arc_easy']
taskname, task_class = x[0]
task = task_class()
# get test split
arr = task.test_docs()

reqs = [task.construct_requests(doc, task.doc_to_text(doc)) for doc in arr]
```
Basically metadata is defined as `metadata: tuple = Tuple[str, int, int]`lol so I get an error while initiating the Instance class. Maybe I'm missing a step?

Setting it to `none` doesn't pass the info on so it's a lot of `task_name=None, doc_id=None, repeats=None`.

This is a silly mistake on my part sorry! Fixing in #676 .

lol I thought it was a deliberate hacky solution! #676 still doesn't pass on the metadata info while constructing requests using the above code. Am I missing something? The api seemed to work fine on the terminal but I was getting a TypeError on this.



Hm let me try this again post-676 ! 

Though this may still be a bug in need of fixing, the missing piece here I think is that `task.build_all_requests()` is now what’s meant to be exposed and called by a user, and so we should test that that function’s behavior is consistent. The build_all_requests function will handle the passing of metadata.

Great! Closing the issue. Thanks!