zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Bedrock Model API
Root cause: Unimplemented Feature Gap

Hi, does this library support the Amazon Bedrock API (for example, using Claude through Bedrock)?

Hi, we don't currently support the Amazon Bedrock API, though we'd gladly accept a PR adding an LM class for this. 

Are there any sample code snippets invoking the bedrock API? in an ideal world, if the Bedrock API supported an OpenAI-like API, we could run it through the existing `local-completions` model type to minimize overheads in maintenance but this may not be possible.

Hi @haileyschoelkopf thanks for your response. The Bedrock API can be called through the `bedrock-runtime` client using `boto3` SDK for Python. Here's an example:

```python
import boto3

bedrock_runtime_client = boto3.client(service_name="bedrock-runtime")

enclosed_prompt = "Human: " + prompt + "\n\nAssistant:"

body = {
	"prompt": enclosed_prompt,
	"max_tokens_to_sample": 200,
	"temperature": 0.5,
	"stop_sequences": ["\n\nHuman:"],
}

response = bedrock_runtime_client.invoke_model(
	modelId="anthropic.claude-v2:1", body=json.dumps(body)
)

response_body = json.loads(response["body"].read())
completion = response_body["completion"]
```

Docs: https://docs.aws.amazon.com/code-library/latest/ug/python_3_bedrock-runtime_code_examples.html


Thanks @austinmw ! Would you have any bandwidth / interest in contributing this?
Repository: EleutherAI/lm-evaluation-harness
Issue: How can we do evaluation few-shot in Chat-template?
Root cause: Documentation Deficiency

1. 
prompt = """
system
You are a helpful assistant.

user
Qus1;.......

assistant
Ans;1.............

user
Qus;2.......

assistant
Ans;2.............
"""

or 

2.
prompt = """
system
You are a helpful assistant.

user
Qus1;.......

assistant
Ans1;.............



system
You are a helpful assistant.

user
Qus2;.......

assistant
Ans2;.............

"""

lm-evaluation-harness didnt support the first format when i do few shot they support on second format that why i am not sure which format are used when eval Chat-template (Chat Model) ?

I don't know how we do it theoretically. Somebody can help me?
Repository: EleutherAI/lm-evaluation-harness
Issue: Adding new Task 'EXAMS'
Root cause: Unimplemented Feature Gap

Hi, 
I want to add [EXAMS](https://aclanthology.org/2020.emnlp-main.438/)  benchmark to the framework, as a new task. 
This benchmark is a multi languages, contain 16 languages.
I've already prepare the task for Arabic EXAMS, this what I am interested in.
Can I make a pull request to add only Arabic EXAMS as a new task to the framework.
This benchmark has been used with multiple Arabic models and I want to be available for me and other using lm-eval framework.

Hi @khalil-Hennara , we'd love to support this task! 

It would be great to support all languages in it, but you don't have the bandwidth for this perhaps another contributor can pick it up?

Ok great, I will make a pull request, Thanks @haileyschoelkopf
Repository: EleutherAI/lm-evaluation-harness
Issue: Output format of samples has been changed
Root cause: Algorithmic Error

Unless I am missing something [this](https://github.com/EleutherAI/lm-evaluation-harness/commit/a007bacb1941356233b0aba49dbcea9bec10429e#diff-45b7cf15ed225746696faea6973a591c9526c8a78f486e28b11f63635e543666R229) commit changed the format of the saved samples (when `log_samples` is true.) Previously the samples were saved as jsonlines in a `jsonl` file but now they are saved as a serialized json list.  A fix that maintains the improved serialization of the above commit but also maintains the jsonlines format should be pretty simple; something like: 
`json_lines = [json.dumps(sample, default=_handle_non_serializable, ensure_ascii=False, ) for sample in samples[task_name]]`
`filename.write_text(json_lines, encoding="utf-8")`

I am happy to implement the fix but I wanted to confirm what the intent was here. Do you want to save as jsonlines or a json list.

Hi! Thanks for reporting this. Indeed Jsonlines is what is desired--if you could implement the fix that would be fantastic!

We'd have to do a quick check to confirm that our [Zeno integration](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/scripts/zeno_visualize.py) and WandB integration which log samples are not broken by this behavior though.
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug in some model init
Root cause: Validation Gap

https://github.com/EleutherAI/lm-evaluation-harness/blob/2d0a646020db092a0e994a254bb9e5557a02e63b/lm_eval/models/mamba_lm.py#L59-L62

Here, if a user passes one of these kwargs, say, `tokenizer=x`, then this constructor will essentially become:
`init(..., tokenizer=x,... tokenizer=x)` as the tokenizer will be passed again in the `**kwargs`, thus causing error
```bash
got multiple values for keyword argument 'tokenizer'.
```

we should `pop` these args, rather than `get`.

Other place(s) suffer the same problem: https://github.com/EleutherAI/lm-evaluation-harness/blob/cc771ecac8964f0afab76e95e264e69528430ff2/lm_eval/models/optimum_lm.py#L29-L32
Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval not found
Root cause: Environment Incompatibility

Hello, 
I followed the required steps, 
git clone 
cd lm-eval-harness
pip install -e .

and when im trying to run lm_eval --model ... etc., it says lm_eval not found. I tried lm-eval too and it says not found. How do I fix this?


@obhalerao97 How did you fix it?

It was an issue with my PATH variable, made that change in .bashrc file
Repository: EleutherAI/lm-evaluation-harness
Issue: The document related to the Task Manager is no longer available.
GitHub labels: asking questions

Evaluation of the LM that showed some logging information to provide a document for the task manager is no longer available.

```bash
# Execution command
NUMEXPR_MAX_THREADS=72 lm_eval --model hf --tasks tmmluplus --model_args "pretrained=$WORK/weight/tinyllama-epoch-0-step-12414/"
```
```markdown
2024-02-29:11:50:47,252 INFO     [evaluator.py:139] get_task_dict has been updated to accept an optional argument, `task_manager`Read more here: https://github.com/EleutherAI/lm-evaluation-harness/blob/recursive-groups/docs/interface.md#external-library-usage
```

Here is the URL:
https://github.com/EleutherAI/lm-evaluation-harness/blob/recursive-groups/docs/interface.md#external-library-usage

Hi @ZoneTwelve , if you pull from main, the message should be updated to point to the correct docs pages (on `main` branch now!) https://github.com/EleutherAI/lm-evaluation-harness/blob/b177c82cb3ef4a8f03ff77341970ab9d77ffadb4/lm_eval/evaluator.py#L193-L196

Hi @haileyschoelkopf, thank you for pointing that out. I appreciate your assistance in ensuring we have the correct documentation referenced. I'll make sure to update the source code on my end as soon as possible. Thanks again for your understanding, and I apologize for any inconvenience caused by the oversight.

No need to apologize! Thank you for pointing out issues :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the ARC Challenge evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> ARC [CCE+18] is a dataset of multiple-choice questions collected from 3rd to 9th grade science exams. On the “Challenge” version of the dataset which has been filtered to questions which simple statistical or information retrieval methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in the zero-shot setting, 53.2% in the one-shot setting, and 51.5% in the few-shot setting. This is approaching the performance of a fine-tuned RoBERTa baseline (55.9%) from UnifiedQA [KKS+20]. On the “Easy” version of the dataset (questions which either of the mentioned baseline approaches answered correctly), GPT-3 achieves 68.8%, 71.2%, and 70.1% which slightly exceeds a fine-tuned RoBERTa baseline from [KKS+20]. However, both of these results are still much worse than the overall SOTAs achieved by the UnifiedQA which exceeds GPT-3’s few-shot results by 27% on the challenge set and 22% on the easy set.

- [X] Data processing code implemented
- [x] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Source: https://allenai.org/data/arc
Repository: EleutherAI/lm-evaluation-harness
Issue: Add New Lambada Translations
Root cause: Unimplemented Feature Gap


@haileyschoelkopf I couldn't find which datasets this issue is referring to. Could you please provide links?


Of course!

It should be https://huggingface.co/datasets/marcob/lambada_multilingual . 

More info in the StableLM 2 tech report if I'm recalling correctly: https://arxiv.org/abs/2402.17834

and some discussions here: https://discord.com/channels/729741769192767510/755950983669874798/1212754311969251338
Repository: EleutherAI/lm-evaluation-harness
Issue: [Bug] UnboundLocalError: lm acessed before assignment
Root cause: Validation Gap

https://github.com/EleutherAI/lm-evaluation-harness/blob/b177c82cb3ef4a8f03ff77341970ab9d77ffadb4/lm_eval/evaluator.py#L155

This particular line seems like it should be an if and not an elif. Otherwise, if model is str and model_args are None, lm object does not get assigned

Thanks for the report, you're correct! Would you be willing to PR this patch?

Oh yes, I will be glad to raise a PR over the weekend
Repository: EleutherAI/lm-evaluation-harness
Issue: concurrent api request to accelerate evaluation
Root cause: Unimplemented Feature Gap

Is there any plan on extending the current api-based evaluation to handle more requests concurrently for acceleration?

Yes, we'd like to do so but are bandwidth constrained--if you are interested in working on this it'd be very helpful!

#1460 provides concurrency for OpenAI models, and I need to review it--if you would also like to review it feel free to!

#2008 should substantially improve things here for the moment, please let us know if things are too slow and we can look into improving them further!
Repository: EleutherAI/lm-evaluation-harness
Issue: [Question] Caching the tokenized evaluation sets?
Root cause: Unimplemented Feature Gap

What's the recommended way (and is there a simple way) to cache the tokenized evaluation set so that running any eval for models using the same tokenizer (e.g. when fientuning the same base model) doesn't require tokenization?

Thanks

Hi! We now supply a `--cache_requests <true,refresh,delete>` that can be used to cache the building of requests, so that time-to-inference after the first run can be significantly faster.

As far as I'm aware, it is the building of requests, not their tokenization which happens in the modeling code, that's the bottleneck--but let me know if that's different in your case, or this `--cache_requests` feature doesn't help!

Thanks, Hailey!  I saw the `cache_requests`, which is great. I was hoping to avoid retokenize on the fly every time, but I guess I'll have to the LM/model to allow for that. I'm closing the issue but think it would be great to have that directly in lm-eval.

Got it! If you implement this on your side and see significant speedups, let me know— I don’t think we’ll plan to add it because I expect it’ll be very minimal speedups for added implementation complexity, but having numbers to prove that assumption wrong would change things!
Repository: EleutherAI/lm-evaluation-harness
Issue: ValueError: Tasks not found: persona_desire-for-acquiring-eval-results.
Root cause: Interface Contract Mismatch

I'm receiving the following error:

```
ValueError: Tasks not found: persona_desire-for-acquiring-eval-results. Try `lm-eval --tasks list` for list of available tasks, or '--verbosity DEBUG' to troubleshoot task registration issues.
```

However, when I try `lm-eval --tasks list`, this task is present:

```
 - persona_believes-it-knows-better-than-humans-about-how-to-help-humans
 - persona_believes-life-has-no-meaning
 - persona_cognitive-enhancement
 - persona_conscientiousness
 - persona_desire-for-acquiring-compute
 - persona_desire-for-acquiring-eval_results
 - persona_desire-for-acquiring-power
 - persona_desire-for-acquiring-wealth
 - persona_desire-for-advancing-technology-to-achieve-goals
 - persona_desire-for-being-rated-HHH-over-actually-being-HHH
 - persona_desire-for-computational-efficiency
```

But it doesn't appear to exist here: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/model_written_evals/persona

I have meet a similar problem, how do you deal with it?
Repository: EleutherAI/lm-evaluation-harness
Issue: ValueError: Input length of input_ids is 494, but `max_length` is set to 128. This can lead to unexpected behavior. You should consider increasing `max_length` or, better yet, setting `max_new_tokens`.
Root cause: Configuration Error

Hi,

I have tried to run
```
accelerate launch -m lm_eval --model hf \
    --model_args pretrained=${MODEL_PATH}/llama2_7b,max_length=2048 \
    --tasks bigbench_generate_until \
    --batch_size auto \
    --limit 10 \
    --show_config \
    --use_cache ${BASE_PATH}/cache \
    --output output/glue_mmlu/ \
    --log_samples
```
but I received the error like `ValueError: Input length of input_ids is 494, but max_length is set to 128. This can lead to unexpected behavior. You should consider increasing max_length or, better yet, setting max_new_tokens.` Could you please give me some hints about how to fix this? Is this due to the yaml file where the `max_length` is set to 128? How can I overwrite this?

Thank you very much!


+1 seems the model misused the generation `max_length` as the input `max_length`

Hi @haileyschoelkopf, What argument to pass to fix this error?

`ValueError: Input length of input_ids is 158, but max_length is set to 128. This can lead to unexpected behavior. You should consider increasing max_length or, better yet, setting max_new_tokens.`

I am running this:

```
!lm_eval \
    --model hf \
    --model_args pretrained=bigcode/starcoderbase-3b,trust_remote_code=True \
    --max_new_tokens=512
    --tasks bigbench_code_line_description_generate_until,bigbench_code_line_description_multiple_choice,code2text_go,code2text_java,code2text_javascript,code2text_php,code2text_python,code2text_ruby,codexglue_code2text \
    --device cuda:0 \
    --batch_size auto \
    --output_path output/starcoderbase-3b \
    --limit 50 \
    --wandb_args project=llm_eval_benchmark \
    --log_samples
 ```
 
 * Passing `--model_args pretrained=bigcode/starcoderbase-3b,max_length=512,trust_remote_code=True \` does not work.
 * Passing `--model_args pretrained=bigcode/starcoderbase-3b,max_new_tokens=512,trust_remote_code=True \`
 gives an error: `TypeError: GPTBigCodeForCausalLM.__init__() got an unexpected keyword argument max_new_tokens`.
Repository: EleutherAI/lm-evaluation-harness
Issue: Unused tests in test_tasks.py
GitHub labels: documentation

feature request

While working on some other code in the library, I noticed that we have a bit of unused test code in `test_tasks.py`, starting on line 125. On quick inspection, it looks like this code was rewritten into the `class TestNewTasks` above and is not currently called anywhere. Is it ok to remove this code? If so, happy to open a PR for it and do some cleanup. 

https://github.com/EleutherAI/lm-evaluation-harness/blob/e5e35fca0898ea4c026711a16cafd43fa969d7a4/tests/test_tasks.py#L125

That should be fine and would be much appreciated! cc @baberabb 

Have been meaning to cleanup + add further (regression) testing to the lib!
Repository: EleutherAI/lm-evaluation-harness
Issue: what does this log mean？
GitHub labels: asking questions

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/29994840/360d0386-1425-4154-9160-e15b22aaa3c6)

when eval task mmlu，the processing bar shows that `toal=56168 ` ，what does this num mean？  
At first, I thought it's total samples of mmlu task(which is 14042), but it seems not. ops... 

14042 * 4 = 56168
And 4 is the number of choices (ABCD).

This seems to be a frequently-asked question, so it would be good to signpost this more in the documentation most likely.
Repository: EleutherAI/lm-evaluation-harness
Issue: humaneval task
Root cause: Unimplemented Feature Gap

Hi, I find that one important code generation task for evaluating llms is missing in the repo. Is there any plan on adding this task?

Please, take a look at or take part in the discussion https://github.com/EleutherAI/lm-evaluation-harness/issues/1157

See #1157 for an update on this!
Repository: EleutherAI/lm-evaluation-harness
Issue: [Question] command to run open_llm_leaderboard ?
Root cause: Documentation Deficiency

Hi, Thanks for the library 💯  

Is there a script or command to run the Open LLM Leaderboard (at least approximately)? I tried using the following flags with `Mistral-7B-v0.1` loaded from HF:

```
--tasks arc_challenge --num_fewshot 25
--tasks truthfulqa_mc2 --num_fewshot 0
--tasks hellaswag --num_fewshot 10
--tasks winogrande --num_fewshot 5
--tasks mmlu --num_fewshot 5
--tasks gsm8k --num_fewshot 5
```

but I'm getting wildly different results. E.g. `hellaswag` gets only `acc_norm=0.24` and `arc_challenge` gets only `acc_norm=0.29`

Sorry if this is out of script for `lm_eval` but I think it would be a great example of command to run :) 

cc @StellaAthena @l3utterfly @GeneZC @fahadh4ilyas  who seem to have worked or asked about that.



@YannDubs It seems that by default there is not bos token inserted before the input. And from the description in this PR https://github.com/EleutherAI/lm-evaluation-harness/pull/1465, you may control BOS token presence via the `--model_arg add_bos_token=True`.

I don't think they use the BOS token on the OpenLLM [branch](https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463), or least they didn't used to? These numbers do seem wildly off though. What command are you using? Should be something like:
```zsh
lm_eval --model hf --model_args pretrained=mistalai/... --tasks arc_challenge  --num_fewshot 25
```
Could be a bug in the latest commits.

This is the command I'm using to run the eval:

```
accelerate launch -m lm_eval --model hf \
    --model_args pretrained=/home/layla/src/text-generation-webui/models/phi-2-layla-v1-out \
    --tasks mmlu \
    --device cuda:0 \
    --batch_size 4 \
    -f 5
```

This gives almost the same results as the open llm leaderboard

Thanks, everyone for your fast responses, I'm closing because I was able to reproduce the results with the parameters from the initial message. The issue was on my side!

I still think that it would be great provide a simple and more prominent example to run (approximately) the open llm leaderboard!

> @YannDubs It seems that by default there is not bos token inserted before the input. And from the description in this PR #1465, you may control BOS token presence via the `--model_arg add_bos_token=True`.

Thanks that's actually really useful to know and keep track of!
Repository: EleutherAI/lm-evaluation-harness
Issue: Multiple Choice Question Standard Deviation
Root cause: Documentation Deficiency

I saw that the multiple choice type evaluation would compute the metrics along with standard deviation. From my understanding, multiple choice answer is chosen from the choice with highest probability, so I thought this is some greedy selection that does not result in randomness Does anyone know how is the std computed in this case?

Hi! We compute the stderr for accuracies / scores which aggregate via taking the mean of the documents' metrics via computing the Standard Error. This gives a sense of the expected variance if we subsampled / observed a different set of documents with the same distribution of accuracies, and is largely impacted by 1) how the per-document scores are distributed (the mean would be more variable if we're closer to 50% accuracy, but less likely to vary if the model scores 99% on the benchmark) and 2) the sample size or number of docs used to measure a task's accuracy.

Relevant function(s): https://github.com/EleutherAI/lm-evaluation-harness/blob/c5acce0c7485a3bc8cbb7d80bf6a577a6922bafa/lm_eval/api/metrics.py#L231 https://github.com/EleutherAI/lm-evaluation-harness/blob/c5acce0c7485a3bc8cbb7d80bf6a577a6922bafa/lm_eval/api/metrics.py#L445

Here's a good article on the standard error of the mean: 
https://statisticsbyjim.com/hypothesis-testing/standard-error-mean/


We recently fixed how the standard errors across subtasks (e.g. how we compute stderr for the entire `mmlu` group) as they were previously aggregated incorrectly: see https://github.com/EleutherAI/lm-evaluation-harness/pull/1427 https://github.com/EleutherAI/lm-evaluation-harness/pull/1390 for reference!

I hope this is helpful!

Thank you for the comprehensive response! Just try to understand this a bit better. For example, for evaluating an mmlu task that contains only a single document (a single json file that contains a list of multiple choice questions). The stderr is computed by dividing questions into a few groups, computing the acc of each group, and then applying the formula to compute stderr from these acc scores. Is this process correct? If it is, then is there a fixed way to divide these groups, such as determining how many samples are partitioned into one group?

Hi!

The stderr is computed for each of the 57 MMLU subtasks, based on the `mean_stderr` formula (computing the std err by computing the sample std deviation and dividing by the square root of sample size, by definition of standard error).

We then aggregate these 57 subtasks' stderrs using the formula, weighting the subtasks by the number of documents they each contain.

For task groupings which would not weight aggregated metrics by number of samples in each group, we would likely want to treat each of them as having "size 1" when doing this stderr aggregation to get the stderr for the entire task grouping.

Thank you again for the clear explanation! Your insights solved my confusion.
Repository: EleutherAI/lm-evaluation-harness
Issue: coqa not working
Root cause: External Dependency Breakage

This happened once before and got fixed: https://github.com/EleutherAI/lm-evaluation-harness/issues/898

But now it seems not working again with same error, at least on my end.

```bash
  File "/home/lchu/.conda/envs/main/lib/python3.9/site-packages/datasets/builder.py", line 1726, in _prepare_split_single
    for key, record in generator:
  File "/lustre/.cache/huggingface/modules/datasets_modules/datasets/EleutherAI--coqa/9ee0502938f8c0d66e6801438f1d1814a99e5a6cbe1e4298869bfb62c2c7596d/coqa.py", line 183, in _generate_examples
    data = json.load(f)
  File "/home/lchu/.conda/envs/main/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/home/lchu/.conda/envs/main/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/home/lchu/.conda/envs/main/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/lchu/.conda/envs/main/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```

Thanks for the report, will have to take a look at the dataset!

@haileyschoelkopf Still experiencing the same error on CoQA, are there any updates on this one?