zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: 请问这个框架可以保存模型生成每一条数据的输出吗
Root cause: Documentation Deficiency

请问这个框架可以保存模型生成每一条数据的输出吗，最后的结果只有一个指标。
我不知道要怎么才能看到模型生成的结果

Supported since v0.4.0 via `--log_samples` !
Repository: EleutherAI/lm-evaluation-harness
Issue: 请问这个框架可以添加有害性、偏见这样的评价指标吗
Root cause: Unimplemented Feature Gap

请问这个框架可以添加有害性、偏见这样的评价指标吗

This is on our radar! if you have any particular tasks you would like added, please feel free to open an issue!
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the Penn Tree Bank evaluation
Root cause: Unimplemented Feature Gap

> We calculate zero-shot perplexity on the Penn Tree Bank (PTB) [MKM+94] dataset measured in [RWC+19]. We omit the 4 Wikipedia-related tasks in that work because they are entirely contained in our training data, and we also omit the one-billion word benchmark due to a high fraction of the dataset being contained in our training set. PTB escapes these issues due to predating the modern internet. Our largest model sets a new SOTA on PTB by a substantial margin of 15 points, achieving a perplaexity of 20.50. Note that since PTB is a traditional language modeling dataset it does not have a clear separation of examples to define one-shot or few-shot evaluation around, so we measure only zero-shot.

not free - https://catalog.ldc.upenn.edu/LDC99T42 / https://stackoverflow.com/questions/8949517/is-there-any-treebank-for-free 
I suggest leaving this one out

Dropping because it isn’t free.

We received confirmation from OA that they use the version of PTB with all the `<unk>`s. So we are going to be using PTB now.

> We received confirmation from OA that they use **the version of PTB with all the s.** So we are going to be using PTB now.

What does this mean?



Edited comment for clarity, apparently github dropped the `<unk>` because it thought it was a html tag or something, and i never noticed.

eric hallahan is working on this
Repository: EleutherAI/lm-evaluation-harness
Issue: Does lm-harness support BloomZ model?
Root cause: Documentation Deficiency

I run the model evaluation task with passing `--model hf-causal --model_args pretrained="/disk1/bloomz-176b"`, but it seems `gpt2.py` is not suitable for `bloomz `model, since it will return errors as below:

```python
AttributeError: 'BloomConfig' object has no attribute 'n_ctx'
AttributeError: 'BloomConfig' object has no attribute 'max_position_embeddings'
``` 

Seems we need to specify `--models` as `hf-causal-experimental`.

> Seems we need to specify `--models` as `hf-causal-experimental`.

This is correct.
Repository: EleutherAI/lm-evaluation-harness
Issue: Where are the results of evaluations?
Root cause: Documentation Deficiency

Thanks for this work. Sorry for simple question but I couldn't find any benchmark docs similar to [HELM](https://crfm.stanford.edu/helm/latest/). Where can I find the results evaluated by this library?

The result is in the result dictionary, and the result is in the main. py file

I mean, do you have the benchmark results published somewhere online or in this repo? Can I see the results without running the code just like in HELM benchmark website?

We are working on uploading systematic evaluations to https://github.com/EleutherAI/lm-evaluation-harness/tree/master/results. You can also find evaluations using this library in our Pythia and GPT-NeoX-20B papers
Repository: EleutherAI/lm-evaluation-harness
Issue: Upload test set function
Root cause: Unimplemented Feature Gap

Our laboratory is planning to develop software based on your code. May I ask when your upload test set function will be implemented?

I would greatly appreciate it if you could answer!

I’m not sure what the question is. If you have a custom evaluation benchmark you’re welcome to submit a PR. We generally recommend hosting the data on the HuggingFace Hub, but if you have an independent hosting server you can also add some custom loading scripts.
Repository: EleutherAI/lm-evaluation-harness
Issue: What are the models that can be evaluated？

Does it support evaluating all types of large models when evaluating local models? If not, what are the models that can be evaluated？

For local models we primarily support the `transformers` library. The Megatron-Deepspeed, GPT-NeoX, and Mesh-Transformer-Jax libraries also have integrations with the eval harness, but you’ll need to call them from the respective codebase instead of this one.
Repository: EleutherAI/lm-evaluation-harness
Issue: Creating custom task for evaluation
Root cause: Documentation Deficiency

I'd like to evaluate some [HF Open LLM models](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) on a set of prompts for relation extraction from the [ChemProt dataset](https://huggingface.co/datasets/bigbio/chemprot), and I was thinking it might be straightforward to add a custom task script and use the infrastructure you already have set up.

I'm not sure that relation extraction falls within the purview of things that this library intends to evaluate, especially given the evaluation frameworks available. I'd like to use an F1 score to evaluate whether or not a model is able to recover (subject, predicate, object) triples from text, as compared to a gold standard annotation (which can be anchored to entity annotations in the text, or the triples can be unanchored and evaluated separately from their location in the text). With that in mind: 

(1) Do you think it makes sense to add a custom task;
(2) How can I use an F1 score within the harness framework; and
(3) Should I work on this on a fork for myself and leave it at that, or should I consider opening a PR for it when I'm done?

Thanks!
Serena

EDIT: Added details about evaluation paradigm

Decided that my task didn't quite fit the framework
Repository: EleutherAI/lm-evaluation-harness
Issue: webqustions wrong
Root cause: Validation Gap

发生异常: IndexError
list index out of range
  File "/root/lm-evaluation-harness/lm_eval/base.py", line 257, in _loglikelihood_tokens
    _, context_enc, continuation_enc = re_ord.get_reordered()[0]
  File "/root/lm-evaluation-harness/lm_eval/base.py", line 186, in loglikelihood
    return self._loglikelihood_tokens(new_reqs)
  File "/root/lm-evaluation-harness/lm_eval/base.py", line 866, in fn
    rem_res = getattr(self.lm, attr)(remaining_reqs)
  File "/root/lm-evaluation-harness/lm_eval/evaluator.py", line 247, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/root/lm-evaluation-harness/lm_eval/utils.py", line 182, in _wrapper
    return fn(*args, **kwargs)
  File "/root/lm-evaluation-harness/lm_eval/evaluator.py", line 86, in simple_evaluate
    results = evaluate(
  File "/root/lm-evaluation-harness/lm_eval/utils.py", line 182, in _wrapper
    return fn(*args, **kwargs)
  File "/root/lm-evaluation-harness/main.py", line 81, in main
    results = evaluator.simple_evaluate(
  File "/root/lm-evaluation-harness/main.py", line 110, in <module>
    main()
IndexError: list index out of range

when i use 
"args": [
                "--model", "hf-causal",
                "--model_args", "pretrained=EleutherAI/gpt-j-6B" ,
                "--tasks","webqs" ,
                "--device", "cuda:0"
            ]
got wrong

Hi, thanks for opening an issue! Does this issue still persist for you? I've tested your exact command on the most up to date version of the code, and I don't seem to replicate this error.
Repository: EleutherAI/lm-evaluation-harness
Issue: Can we use it to evaluate bloomz with lora?
Root cause: Validation Gap

hello, I evaluate the bloomz after finetune it by lora(https://github.com/tloen/alpaca-lora), the command is :
"""
python main.py \
    --model hf-causal-experimental \
    --model_args pretrained=/mnt/data/LLM/bloomz-7b1-mt/,peft=/mnt/data/lora_bloomz-7b1-mt_alpaca52k/ \
    --tasks piqa \
    --device cuda:0
"""
however, the result is the same as the origin bloomz, can you help me? thx!




Thank you for the bug report. Can you find some example strings where the finetuned and base models return different outputs (when run for normal generation in the HF `transformers` library) and try evaluating on a toy dataset made up of those strings? That’ll help us figure out if you’re just having problems loading the model or if something deeper is going on.

@zanussbaum wrote the PEFT implementation and may have some insights.

Hm that's interesting the results are exactly the same! Do you mind sharing the output from the `lm-eval-harness` from the original model and the finetuned model? 

One thing to note is that I used `hf-causal` instead of `hf-causal-experimental` before the experimental class was implemented but a quick look seems like it's been implemented correctly. I'll have a closer look later today to see if I can see what might be the root cause

Hi @wangzhao88 , does this issue persist for you, and for multiple Loras? If it does, if you upload the LoRA for which this occurs we can look into it, else it's difficult for us to test this.

Does the LoRA-ed model also remain at the same score across tasks beside `piqa`?

Closing this, but please feel free to reopen if you're willing to upload the models so that we can test this out!
Repository: EleutherAI/lm-evaluation-harness
Issue: Will you support HumanEval
Root cause: Unimplemented Feature Gap

This task: https://github.com/openai/human-eval

We have historically opposed doing so because human eval requires the fundamentally unsafe practice of having a LLM generate code and then executing that code. While this can be done safely in a sandbox, we do not have a sandbox we can currently offer publicly that is safe enough to do things like this in.
Repository: EleutherAI/lm-evaluation-harness
Issue: Do you support running models in fp16?
Root cause: Documentation Deficiency


Hi,
When I tried evaluate a pythia-12b model, I encountered CUDA out-of-memory issue.  Is it possible to set the model to run in fp16 mode in --model_args? Thanks

Have you tried deepspeed inference? That should help.

This was previously supported in `--model hf-causal-experimental` but not well-documented ; just extended this support to `hf-causal` model type and documented usage in readme!
Repository: EleutherAI/lm-evaluation-harness
Issue: Error in few-shot scenarios introduced in a previous commit.
Root cause: Algorithmic Error

I have made a comment in the commit explaining the issue.
[Link to the comment.](https://github.com/EleutherAI/lm-evaluation-harness/commit/d145167959c2b1826d900524912cb99c44d5fb30#commitcomment-114410327)

I am pasting it also here for clarity:

------------

This commit has introduced several datasets prompted in a mGPT fashion but they will only work in zero-shot scenarios. An example prompting one-shot PAWS-X using the scripts.write_out functionality:

```
!!@@##@@!! -- Example 0
The government was led by Gordon Coates from the United Party , with George Forbes of Reform as finance minister ., right? [MASK], The government was led by Gordon Coates of the United Party , with George Forbes of Reform as Minister of Finance . No

The exception was between late 2005 and 2009 when he played in Sweden with Carlstad United BK , Serbia with FK Borac Čačak and Russian FC Terek Grozny ., right? [MASK], The exception was between late 2005 and 2009 , when he played in Sweden with Carlstad United BK , Serbia with FK Borac Čačak and the Russian FC Terek Grozny .
```

As you can see the [MASK] token is not substituted in the examples and the labeled response token is written out of context at the end of the prompt.

Besides, as if done in PAWS-X you compute the request as follows `ll_yes = rf.loglikelihood_rolling(ctx.replace("[MASK]", self.YES))` you would be substituting not only the MASK of the final test sample but also of all the few-shot examples.

Please, this behavior should be corrected to avoid divergences between zero-shot and few-shot scenarios.

There would have to be a discussion on whether to change the type of prompt to one in which the completion of the answer is done at the end or if a more profound modification of the repository should be done in order to indicate if a response should be put at the end of a sentence or not.

Thanks!


@muxitox thanks for bringing this up. Have made a PR to fix this. Are the tasks impacted only XNLI and PawsX?

Thanks!

I think so, yes.

Okay, merged a fix for both.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add TheoremQA
Root cause: Unimplemented Feature Gap

This is awesome work and we should include it. It may be a bit tricky as it involves external calls to the WolframAlpha API for answer evaluation though.

https://github.com/wenhuchen/TheoremQA

Hey @StellaAthena wondering if this issue is still open/required. If so I can take it on. Could you give me a brief overview of what has to be done

@lintangsutawika I am interested in contributing here.

That's awesome! Feel free to start a PR anytime. 

If this is still open can I start work on this @baberabb
Repository: EleutherAI/lm-evaluation-harness
Issue: [Question] Pointers for another framework implementation
Root cause: Documentation Deficiency

Hi,
I am looking at implementing this for the OpenNMT-py framework.
I see 3 solutions:
1) poorman solution:
Get prompts in text files, process them in our framework, get the outputs, eval them with a script here. Is there any such scripts ready to do this ?

2) develop an API to serve a OpenNMT-py model in a "openai-like" structure. 

3) Write an adapter, but I am unclear on where to start to do such a thing.

thanks.


For #1, you can modify `write_out.py` to do this.

For #2 and #3, it depends a bit on what you’re looking to accomplish.

**If you want to evaluate models while they’re being actively used in your library** You can import this library into a file in your library that runs evaluations (multiple examples are linked in the readme, see GPT-NeoX in particular).

**If you want to evaluate models that have been developed in your library and are “finished.”** then you can just add an integration for the current API that your models expect to this library. The HuggingFace implementation is the best reference for this.
Repository: EleutherAI/lm-evaluation-harness
Issue: hf-seq2seq doesn't run on perplexity tasks
Root cause: Validation Gap

It is calling a function which isn't defined anywhere:

https://github.com/EleutherAI/lm-evaluation-harness/blob/095d84064b02fc7b126ed293ea97e9dba4299ad2/lm_eval/models/huggingface.py#L597-L601

Thank you for pointing this out. This is a known issue, but it seems like we don’t have an issue for it yet.

Closed as we have solved this problem.
Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError: Union[arg, ...]: each arg must be a type. Got True.
Root cause: Validation Gap

models/huggingface.py includes some type indicators of the form

```python
quantized: Optional[Union[True, str]] = None,
```

which in a clean Python 3.10.11 conda installation leads to an error

```
TypeError: Union[arg, ...]: each arg must be a type. Got True.
```

In Python 3.11 no error is thrown. 

Possible solutions:
- Set python_requires=">=3.11"
- Change the type indicator to Union[bool, str]


Hi,

I also got this error. I was running a command that worked perfectly fine 2 weeks ago, but I pulled the latest code now and the same command is throwing error!

Is there any quick fix other than reverting to the branch I worked on!

Will be very much appreciated :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Add support for openai ChatCompletion models
Root cause: Unimplemented Feature Gap

It would be nice to support ChatLLMS, starting with openai ones (gpt3.5-turbo and gpt4), to set baselines on these models as well.

This will likely be similar to
https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/models/gpt3.py , but using ChatCompletion instead of Completion and adjusting input/output processing.

Thank you for this wonderful repo!

it's not possible since prediction log_probs are not available: https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/models/gpt3.py#L24 for chat completion

Thanks for the answer, I did not catch this indeed.
So there is not way to evaluate GPT-4 without log_probs, even on specific tasks like Multi-choice + adding some stop criterion?

We'll be looking into adding support for this and for Claude, at first for generation tasks only and then considering approaches for full task support!

Contributions on this end would be welcome though if anyone is interested and has credits, as I do not currently have access to Claude and haven't frequently used the GPT-4 API yet.

Closing as duplicative
Repository: EleutherAI/lm-evaluation-harness
Issue: Convert return format of rf.loglikelihood from list to dict for custom metrics
Root cause: Architectural Constraint

I recommend converting the return format of rf.loglikelihood() from a tuple to a dictionary. This change is necessary because there may be a need to include additional metrics in the future. For instance, I would like to incorporate a metric called "token-length normalized loglikelihood" alongside the existing "byte-length normalized loglikelihood." However, modifying the code is challenging because it would impact all other tasks.

- As Is
```
log_probs = [rf.loglikelihood(ctx, " {}".format(choice))[0] for choice in doc["choices"]]
is_exact_matches = [rf.loglikelihood(ctx, " {}".format(choice))[1] for choice in doc["choices"]]
```


- To Be
```
log_probs = [rf.loglikelihood(ctx, " {}".format(choice))['log_prob'] for choice in doc["choices"]]
is_exact_matches = [rf.loglikelihood(ctx, " {}".format(choice))['is_exact_match'] for choice in doc["choices"]]
```

This has been added.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add support for evaluating Anthropic's Claude
Root cause: Unimplemented Feature Gap

Feature request for adding support for evaluating one of the best closed source models, Anthropic's Claude, this should help to,
1. Set a baseline when comparing open-source models
2. Effectively gauge the performance of open-source models against the proprietary models available at the moment

https://www.anthropic.com/index/introducing-claude

It looks like it has the same issue as GPT-4 (https://github.com/EleutherAI/lm-evaluation-harness/issues/541), not providing logprobs, so it can only be used in generation tasks.

We now support Anthropic after #562 on generation tasks! In future, we will look into how to provide support on more tasks for this and other generation-only APIs.
Repository: EleutherAI/lm-evaluation-harness
Issue: Problem when running with hf-seq2seq
Root cause: Validation Gap

Hi @StellaAthena and @haileyschoelkopf,

I tried to evaluate BART-base using this repo but got CUDA error

Command:

`python main.py --model hf-seq2seq --model_args pretrained=facebook/bart-base --tasks boolq,piqa,winogrande --device cuda:0 --batch_size=32 > output.log`

Error when device is set to GPU:
```
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

I set the device to CPU to see the detailed error and it's the following:
`IndexError: index out of range in self`

I can place the entire error stack if needed, but as it's too long, here's the relevant code where the code broke

```
lm_eval/models/huggingface.py:643 in loglikelihood_tokens
outputs = self._model_call(inputs=inputs_tokens, labels=targets_tokens)  
```

When I tried to use hf-causal for the same model, it ran successfully i.e
`python main.py --model hf-causal --model_args pretrained=facebook/bart-base --tasks boolq,piqa,winogrande --device cuda:0 --batch_size=32 > output.log` but with a warning that a lot of layers are not initialized properly, which is expected as it's an encoder-decoder model!

Let me know if I am missing something here or if the support for Seq2Seq Models is not clear?

Can run it as `pretrained=facebook/bart-base,max_length=1024`, to respect its [`max_position_embeddings`](https://huggingface.co/facebook/bart-base/blob/main/config.json#L42).

Can I know what's the version of transformers you are using?

Apparently, I pulled the latest branch and it threw a different error

Command: `python main.py --model hf-seq2seq --model_args pretrained=facebook/bart-base,max_length=1024 --tasks boolq,piqa,winogrande --device cuda:1 --batch_size=32 > output.log`

Error:
```
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

But changing device flag to `cuda:0` is working. 

Can I know why this is the case? I do have 8 GPUs and this issue is reported in other repos but with no possible fix around. I know this is not exactly related to this repo

Thanks

Can't reproduce with `pip install --upgrade torch` and `pip install git+https://github.com/huggingface/transformers`, works either way. Something with the system? Device ids should be in `nvidia-smi`, then there are environment variables like `CUDA_DEVICE_ORDER` (e.g. `PCI_BUS_ID`) and `CUDA_VISIBLE_DEVICES` (or `NVIDIA_VISIBLE_DEVICES`, e.g. `all`), then can check it in the process with `torch.cuda.device_count()` and `torch.cuda.get_device_properties(id)`.
Repository: EleutherAI/lm-evaluation-harness
Issue: Log Sample Predictions on Big-Refactor
Root cause: Unimplemented Feature Gap

Originally from [this issue](https://github.com/EleutherAI/lm-eval2/issues/6)

The log could be something like this.
```
{'doc_id': 0, 'context': "`` carlos rafael wilson . '' ...", 'targets': ' carlos', 'ppl': -7.824697017669678, 'acc': 0}
{'doc_id': 1, 'context': "`` you would n't d.        ...", 'targets': ' delilah', 'ppl': -1.4850026369094849, 'acc': 0}
```
Saved into a jsonl file accessible to the user.

Closed by #563 .