zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Add tasks for performance on long context lengths
Root cause: Unimplemented Feature Gap

There are a couple of papers I see with benchmarks for really long context lengths that don't seem to be available in lm-evaluation-harness. It would be great to have one of these or something similar for measuring ability to extract information from long context windows,  important for RAG.

* LongBench: https://arxiv.org/abs/2308.14508
* ∞ Bench: https://arxiv.org/abs/2402.13718
* Are there any others that might be better?


Needle-in-a-haystack might also be a nice-to-have though I think more difficult / "natural" long-context evals should be prioritized.

any plans for this feature release？

Interested here as well

Longbench and longbench 2 have both been added

https://github.com/EleutherAI/lm-evaluation-harness/tree/0c9072191e790204c4dfeb3285925b586a548c2e/lm_eval/tasks/longbench
https://github.com/EleutherAI/lm-evaluation-harness/tree/0c9072191e790204c4dfeb3285925b586a548c2e/lm_eval/tasks/longbench2
Repository: EleutherAI/lm-evaluation-harness
Issue: Accuracy gap between single GPU and multiple GPUs
Root cause: Resource Mishandling

I'm using lm-eval v0.4.2 to evaluate Llama 7b on the open llm leaderboard benchmark.
I found that there are accuracy gaps between single GPU and multiple GPUs as below. (I used data parallel)

|  | average | ARC-c | HellaSwag | MMLU | TruthfulQA | WinoGrande | GSM8K|
|-- | -- | -- | -- | -- | -- | -- | --|
|4 GPUs (batch size 4) | 46.58 | 50.85 | 78.13 | 35.14 | 34.08 | 71.82 | 9.48|
|4 GPUs (batch size 1)| 46.61 | 50.85 | 78.12 | 35.17 | 34.08 | 71.9 | 9.55|
|1 GPU (batch size 4)| 46.37 | 50.43 | 77.82 | 35.14 | 34.08 | 71.74 | 9.02|
|1 GPU (batch size 1)| 46.42 | 50.43 | 77.83 | 35.17 | 34.08 | 71.74 | 9.25|

Single GPU got overall lower accuracies. ARC-c, hellaswag and GSM8K drop 0.3~0.5.
I thought that data-parallel only speeds up the evaluation. Where did the difference come from?

Below is the command line I used for ARC-c.
Use CUDA_VISIBLE_DEVICES to control the number of GPUs.

```
accelerate launch --main_process_port $PORT -m lm_eval \
        --model hf \
        --model_args pretrained=huggyllama/llama-7b \
        --tasks arc_challenge \
        --num_fewshot 25 \
        --output_path $output_path \
        --batch_size $batch_size
```


Thank you for your efforts! Great table with results to compare! 

> Where did the difference come from?

Please check other issues/discussions about speed, batches and multiple GPU usage for ideas. For example (but not limited to),
https://github.com/EleutherAI/lm-evaluation-harness/issues/1625
https://github.com/EleutherAI/lm-evaluation-harness/issues/704#issuecomment-1670189773

Thank @LSinev for the quick reply.
I've checked the issues you linked and also searched for some others by myself.
Most of the issues focus on the difference between **different batch sizes** which is usually very small.
I understand there will be some order difference so the loglikelihood might be slightly different and this small difference is acceptable to me.

However, I didn't find the issue about the difference between different numbers of GPUs.
In my case, the accuracy drops 0.3~0.5 when using a single GPU. I think the drop is quite large. Is this also an expected result?

> Is this also an expected result?

No idea. According to your results from table it is also task dependent issue. 
You may want to further research this case with deep diving in code an logging. May be even use not yet merged PRs like https://github.com/EleutherAI/lm-evaluation-harness/pull/1731 to check consistency of tokenization (may be some adding of special tokens happens incorrectly), propagation of seeds, splitting (and restoring original order) of batches and so on.

If even just batch size makes difference, I suppose multiple GPU may propose even more difference.
Repository: EleutherAI/lm-evaluation-harness
Issue: Pytorch profiling Error In Megatron-DeepSpeed/tasks/eval_harness/evaluate.py
Root cause: Environment Incompatibility

I have error when use profile on evaluate.py(1 node 8 gpus) https://github.com/microsoft/Megatron-DeepSpeed/blob/main/tasks/eval_harness/evaluate.py .

```python
from torch.profiler import profile, record_function, ProfilerActivity
def main():
    start = time.time()
    model = load_ds_checkpoint_and_setup_megatron(extra_args_provider=tasks_args)

    args = get_args()
    if args.deepspeed and args.adaptive_seq_len:
        # adaptive_seq_len hack #1:
        # CL automatically enables reset_activation_shape() which allows us to change input shapes
        # and it also reshapes the attenion scores in attention_mask_func
        args.curriculum_learning_legacy = 1

    task_list = ALL_TASKS if args.task_list == 'all' else args.task_list.split(',')
    task_dict = tasks.get_task_dict(task_list)

    model.module.activation_checkpoint_interval = 0
    model._compute_loss = False
    model.fwd_outputs = []

    tokenizer = get_tokenizer()
    adaptor = EvalHarnessAdaptor(model, tokenizer)
    with profile(activities=[
        ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
        results = evaluator.evaluate(adaptor, task_dict, False, args.num_fewshot, None)
    prof.export_chrome_trace("trace.json")
    if mpu.is_pipeline_last_stage() and mpu.get_tensor_model_parallel_rank() == 0:
        print(json.dumps(results, indent=2))
        with open(args.results_path, 'w') as outfile:
            json.dump(results, outfile, indent = 4)
    end = time.time()
    print("evaluation of {} ends in {:.2f} sec, or {:.2f} min, or {:.2f} hr".format(args.task_list, end-start, (end-start)/60.0, (end-start)/3600.0))
```
But I get an error.
```bash
output_json.cpp:468 failed to rename trace.json.tmp to trace.json
```
How should I use pytorch.profile tools to fix this.
After the error, I get a file "trace.json" , but it can not be used in chrome://tracing/

Don't you think https://github.com/microsoft/Megatron-DeepSpeed/issues is better place for this issue?
Repository: EleutherAI/lm-evaluation-harness
Issue: New Task Request: LegalBench
Root cause: Unimplemented Feature Gap

https://arxiv.org/abs/2308.11462



Should the 162 individual tasks that LegalBench includes, be separate tasks?

Up for debate--one option would be to do the broader categories shown here in the LegalBench paper

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/65563625/33954590-2377-4eeb-a524-fd2cca0dc093)

Another option is to do whatever HELM and HELM-lite do for LegalBench already (https://github.com/stanford-crfm/helm/blob/460f2433ae8e33eeafe89adaa3c12cde6c9ea229/src/helm/benchmark/scenarios/legalbench_scenario.py#L52).
Repository: EleutherAI/lm-evaluation-harness
Issue: No inference time is returned in results
Root cause: Unimplemented Feature Gap

I think that it would be useful to return the inference time needed for the requests in the result.

Do you think that such a feature would be useful? If yes I can open a PR.

Just connecting other issue (not sure if it is the only one) about inference time report: https://github.com/EleutherAI/lm-evaluation-harness/issues/1236

Good, since there is demand for this feature I will open a PR.
Repository: EleutherAI/lm-evaluation-harness
Issue: HellaSwag with UnicodeDecodeError
Root cause: External Dependency Breakage

When I was trying to evaluate HellaSwag using:
`
lm_eval --model hf 
    --model_args pretrained=HuggingFaceH4/zephyr-7b-beta,dtype="bfloat16" 
    --tasks hellaswag 
    --device cuda:0 
    --num_fewshot 10 
    --batch_size auto 
    --trust_remote_code
`
I met the error:
File "/root/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/load.py", line 2587, in load_dataset
    builder_instance = load_dataset_builder(
  File "/root/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/load.py", line 2259, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/root/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/load.py", line 1910, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/envs/lm_eval/lib/python3.10/site-packages/datasets/load.py", line 1862, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
  File "/root/miniconda3/envs/lm_eval/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte

How can I solve this error?

same problem

encounter the same issue in local environment

same issue

same issue

same issue
update: it is working now

Cannot initially seem to replicate on a fresh HF cache... perhaps did something wrong though? Is the connection to the HF Hub working for those facing this problem?

> Cannot initially seem to replicate on a fresh HF cache... perhaps did something wrong though? Is the connection to the HF Hub working for those facing this problem?

It seems not this problem, the connection is well on my machine.

same problem
update: it is working now

> same problem update: it is working now

So what changes did you make?

same with drop

Waiting for solution...

same problem, someone solved it?

download dataset from hf to local,then modify yaml files will success

A workaround is to downgrade `datasets` to `2.14.6`.

For more details, please refer to this issue: https://github.com/huggingface/datasets/issues/6760#issuecomment-2041390144 .

Version: 2.16.0 is also ok
Repository: EleutherAI/lm-evaluation-harness
Issue: Output constrained support
Root cause: Unimplemented Feature Gap

I'd like to build a benchmark and I'd like to set up some rules based on which I can constrain the responses the LLM provides - then eval with pre-determined code (taking into consideration the predetermined constrained rules) the response.

I'm thinking at integrations with libraries like [outlines](https://github.com/outlines-dev/outlines) or [guidance](https://github.com/guidance-ai/guidance).

Hey @Mihaiii , I've been working with Outlines lately and it looks like a really good resource for this.

they have this interesting blogpost on how structured generation helps with benchmarks (https://blog.dottxt.co/performance-gsm8k.html).

Is this something you are still interested in looking at, i'd love to work on this as well.

Hey @ToluClassics ,

I was interrsted in building a particular benchmark that needed this functionality in lm-eval. I never aimed at introducing that functionality in lm-eval.

If you want to make the code changes in lm-eval, that would be awesome :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Does this support the model to use generate functions to eval not likelihood?
Root cause: Unimplemented Feature Gap

I saw some details in the code.As default, put the inp and cont togrther into the model and get the logits.Then throught the argmax to get the predictions.But it did not use the generate function and some code I wrote is in the generate code.
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug in yaml parsing
Root cause: Validation Gap

Hi, when I'm using a custom m_mmlu task, there is an error like this

```
Generating test split: 12909 examples [00:00, 512686.14 examples/s]
Traceback (most recent call last):
  File "/output/lm-evaluation-harness/lm_eval/__main__.py", line 297, in <module>
    cli_evaluate()
  File "/output/lm-evaluation-harness/lm_eval/__main__.py", line 236, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/output/lm-evaluation-harness/lm_eval/utils.py", line 425, in _wrapper
    return fn(*args, **kwargs)
  File "/output/lm-evaluation-harness/lm_eval/evaluator.py", line 122, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "/output/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 255, in get_task_dict
    task_obj = get_task_dict(task_name)
  File "/output/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 275, in get_task_dict
    task_name: get_task(task_name=task_element, config=config),
  File "/output/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 217, in get_task
    return TASK_REGISTRY[task_name](config=config)
  File "/output/lm-evaluation-harness/lm_eval/api/task.py", line 688, in __init__
    test_choice = self.doc_to_choice(test_doc)
  File "/output/lm-evaluation-harness/lm_eval/api/task.py", line 962, in doc_to_choice
    return ast.literal_eval(utils.apply_template(doc_to_choice, doc))
  File "/usr/lib/python3.10/ast.py", line 64, in literal_eval
    node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
  File "/usr/lib/python3.10/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    ["британців другого покоління з південно-азіатського півострова", "білих американців, які хотіли конвертуватися в іслам", "афроамериканців, які відчували себе виключеними з "етнічного тигелька" в США", "афрокарибських жителів внутрішніх міст, які мали виразну культуру молоді"]
                                                                                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: invalid syntax. Perhaps you forgot a comma?
```

I think this is caused by some special chars (like ") in the text, how can we solve this?
Repository: EleutherAI/lm-evaluation-harness
Issue: Seq2Seq Models with Batch Size `auto`
Root cause: Validation Gap

Whenever you run `lm-eval` with any sequence to sequence model specifying `--batch_size=auto`, the following error is raised:

```
Traceback (most recent call last):
  File "/usr/local/bin/lm-eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/__main__.py", line 341, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 251, in simple_evaluate
    results = evaluate(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 390, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/huggingface.py", line 1162, in generate_until
    batch_size = self._detect_batch_size()
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/huggingface.py", line 691, in _detect_batch_size
    batch_size = forward_batch()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/memory.py", line 142, in decorator
    return function(batch_size, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/models/huggingface.py", line 671, in forward_batch
    length = max(max_context_enc, max_cont_enc)
NameError: free variable 'max_context_enc' referenced before assignment in enclosing scope
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Support OpenAI's Batch API
Root cause: Unimplemented Feature Gap

How complex would be adding OpenAI's Batch API? For now, GPT-4 benchmark of Polish tasks would cost me $1500 :(

It's not very difficult for the most part it's replacing these [lines](https://github.com/EleutherAI/lm-evaluation-harness/blob/59cf408ab4a6d9f2473234ef06cb5aa5c97e9f96/lm_eval/models/openai_completions.py#L448)

such as 

```
                response = oa_completion(
                    client=self.client,
                    chat=True,
                    messages=inps,
                    model=self.model,
                    **kwargs,
                )
```

with the following

```
with open("open_ai_batch.jsonl", "a") as f:
    f.write(json.dumps(inps))

# Then after the json file is written
....
# upload the file
batch_input_file = client.files.create(
    file=open("batchinput.jsonl", "rb"),
    purpose="batch"
)

batch_input_file_id = batch_input_file.id

client.batches.create(
    input_file_id=batch_input_file_id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={
      "description": "nightly eval job"
    }
)
```

We need a new way (function) to get back and check the results

```
# check results
client.batches.retrieve(batch_input_file_id)

# retrieve results once done
content = client.files.content("output_file")
```

The devil is in the details though as the jsonl file can only be called for 1 model type and how do you handle multiple files/batches etc.  For the simple case of evaluating 1 model 1 batch it's pretty simple change.

Do you know if there has been any progress on this?
Repository: EleutherAI/lm-evaluation-harness
Issue: Same results - different models
Root cause: Configuration Error

I've been testing several benchmarks to compare the performance of our trained models, however different models give me the exact same results for a given dataset. I've repeated the test with other public models:
- EleutherAI/gpt-j-6B
- meta-llama/Meta-Llama-3-8B
- HuggingFaceH4/zephyr-7b-beta
- our llama2-13b private models...

Regarding the benchmarks, I've been using: arc_easy, arch_challenge, gsm8k, hellaswag and truthfulqa

In case it was related to files stored in the cache I've set:
export HF_DATASETS_OFFLINE=0
export HF_HOME=<cache directory>
export XDG_CACHE_HOME=<cache directory>

and I'm deleting <cache directory> after every execution. The order of the results in the table change but not the values.

Am I missing something?

Hey! 

What code commit are you using and could you share the precise command you are using to invoke these evaluations? Also, what are the repeated scores?

my initial guess is that you're passing `--model_args pretrained=[model repo path]` after having passed `--model_args` already and it's not picking up on the model name, or something similar. But hard to say without more information first.

I've been using the code for a couple months and currently using: f64e72f565b9a8fe09690b0a3216dc67a6ca91cb
Regarding the complete command, I have the opposite situation:

lm_eval --model hf --model_args pretrained=HuggingFaceH4/zephyr-7b-beta --model_args parallelize=True --tasks arc_easy --batch_size 16 --output_path output/zephyr_arc_easy --log_samples

I've only used this command changing only the path in pretrained and the task (and output name)

What happens when you do `--model_args pretrained=HuggingFaceH4/zephyr-7b-beta,parallelize=True` ?

Yes it works! Thank you 😄
Repository: EleutherAI/lm-evaluation-harness
Issue: How to filter to see only generate_until: lm-eval --tasks list
Root cause: Unimplemented Feature Gap

When running lm-eval --tasks list, is there a way to filter to see only the generate_until type?
Also, it'd be great to get a brief description for each task.
Thanks!

It would be nice as well to be able to filter a tasks or list of tasks to only run generate_until evaluations, when using the lm_eval command line, for example when the target model doesn't support logprobs so loglikelihood tasks can't be used.

It could be something like
`lm_eval --task leaderboard --filter_output_type generate_until`
Repository: EleutherAI/lm-evaluation-harness
Issue: Sorting task output alphabetically
Root cause: Unimplemented Feature Gap

When we run `lm-eval`, our output looks like this, where the tasks are scrambled every run:
```
hf (pretrained=meta-llama/Meta-Llama-3-8B), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 8
|    Tasks     |Version|Filter|n-shot|  Metric  |Value |   |Stderr|
|--------------|------:|------|-----:|----------|-----:|---|-----:|
|piqa          |      1|none  |     0|acc       |0.7960|±  |0.0094|
|              |       |none  |     0|acc_norm  |0.8069|±  |0.0092|
|arc_easy      |      1|none  |     0|acc       |0.8013|±  |0.0082|
|              |       |none  |     0|acc_norm  |0.7778|±  |0.0085|
|arc_challenge |      1|none  |     0|acc       |0.5026|±  |0.0146|
|              |       |none  |     0|acc_norm  |0.5341|±  |0.0146|
|winogrande    |      1|none  |     0|acc       |0.7309|±  |0.0125|
|sciq          |      1|none  |     0|acc       |0.9630|±  |0.0060|
|              |       |none  |     0|acc_norm  |0.9390|±  |0.0076|
|boolq         |      2|none  |     0|acc       |0.8113|±  |0.0068|
|hellaswag     |      1|none  |     0|acc       |0.6018|±  |0.0049|
|              |       |none  |     0|acc_norm  |0.7910|±  |0.0041|
|lambada_openai|      1|none  |     0|perplexity|3.0953|±  |0.0571|
|              |       |none  |     0|acc       |0.7557|±  |0.0060|
|openbookqa    |      1|none  |     0|acc       |0.3500|±  |0.0214|
|              |       |none  |     0|acc_norm  |0.4500|±  |0.0223|
```

This makes it difficult to compare different models, since the tasks must be in the same order. I understand that random sorting creates a bias toward alphabetically-earlier tasks, but the tradeoff would be worth it for an option, since we have to sort them anyway.

Addressing in #1791 !

Hooray, thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: error in eval-tracker : 'Namespace' object has no attribute 'push_results_to_hub'
Root cause: Validation Gap

Running the following command:

`lm_eval --model vllm --model_args pretrained=mistralai/Mixtral-8x7B-Instruct-v0.1,tensor_parallel_size=2 --tasks hellaswag --num_fewshot 5`

leads to error:

![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/8822956/3614a897-5074-461c-b68a-bb1915a73a60)


likely caused by this : https://github.com/EleutherAI/lm-evaluation-harness/pull/1766

Should be fixed by #1777
Repository: EleutherAI/lm-evaluation-harness
Issue: Data preprocess is slow for mmlu
Root cause: Documentation Deficiency

Hi, every time I evaluate models on MMLU dataset, the data needs to be prepared for a long time like this:
```
2024-05-04:19:26:20,464 INFO     [task.py:395] Building contexts for mmlu_prehistory on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 324/324 [00:21<00:00, 15.16it/s]
2024-05-04:19:26:41,892 INFO     [task.py:395] Building contexts for mmlu_high_school_us_history on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 204/204 [00:13<00:00, 15.19it/s]
2024-05-04:19:26:55,365 INFO     [task.py:395] Building contexts for mmlu_international_law on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 121/121 [00:07<00:00, 15.21it/s]
2024-05-04:19:27:03,346 INFO     [task.py:395] Building contexts for mmlu_professional_law on rank 0...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1534/1534 [01:40<00:00, 15.21it/s]
2024-05-04:19:28:44,445 INFO     [task.py:395] Building contexts for mmlu_high_school_world_history on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 237/237 [00:15<00:00, 15.12it/s]
```
Since there are many subject in MMLU, the whole data preprocess can take up to tens of minutes. How to avoid doing this every time?

Hi! 

`--cache_requests true` should allow you to reuse these computations after the first time.