zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Task.construct_requests cannot be a generator (i.e., must return requests, not yield them)
Root cause: Documentation Deficiency

The [documentation](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_guide.md#whats-a-request-whats-a-doc) on task creation suggests that `Task.construct_requests` does not have to return a list, but can be a generator. Is that intended? 

It seems that implementations of `construct_requests` mostly return tuples. Grepping for `yield` in `lm_eval/tasks/` does not show any cases of `construct_requests` implemented as a generator.

When I `yield` values from my implementation of `construct_requests`, I get an error:
```
Traceback (most recent call last):
  File "/home/tanner/git/club/build/git/lm-evaluation-harness/main.py", line 116, in <module>
    main()
  File "/home/tanner/git/club/build/git/lm-evaluation-harness/main.py", line 83, in main
    results = evaluator.simple_evaluate(
  File "/home/tanner/git/club/build/git/lm-evaluation-harness/lm_eval/utils.py", line 264, in _wrapper
    return fn(*args, **kwargs)
  File "/home/tanner/git/club/build/git/lm-evaluation-harness/lm_eval/evaluator.py", line 92, in simple_evaluate
    results = evaluate(
  File "/home/tanner/git/club/build/git/lm-evaluation-harness/lm_eval/utils.py", line 264, in _wrapper
    return fn(*args, **kwargs)
  File "/home/tanner/git/club/build/git/lm-evaluation-harness/lm_eval/evaluator.py", line 253, in evaluate
    requests[req.request_type].append(req)
AttributeError: 'generator' object has no attribute 'request_type'
```
This leads me to believe that implementing `construct_requests` as a generator is not really supported. Is that correct?

Thanks for raising this! This does seem to be correct, so I've updated the documentation to reflect this. 

If the task you are implementing would require using a generator, then happy to discuss why this might be to better understand the use case, and to add this as a feature!

It wasn't a requirement. Based on the docs, I implemented the function as a generator, and so I was confused when it didn't work. With the docs updated, I think it's fine to just call this done. I don't have a feature request or anything. Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Error while downloading translation dataset
Root cause: Interface Contract Mismatch

```
  File "D:\github\2023\EleutherAI\lm-evaluation-harness\lm_eval\evaluator.py", line 86, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\github\2023\EleutherAI\lm-evaluation-harness\lm_eval\tasks\__init__.py", line 390, in get_task_dict
    task_name_dict = {
                     ^
  File "D:\github\2023\EleutherAI\lm-evaluation-harness\lm_eval\tasks\__init__.py", line 391, in <dictcomp>
    task_name: get_task(task_name)()
               ^^^^^^^^^^^^^^^^^^^^^
  File "D:\github\2023\EleutherAI\lm-evaluation-harness\lm_eval\tasks\translation.py", line 112, in __init__
    super().__init__(dataset, language_pair)
  File "D:\github\2023\EleutherAI\lm-evaluation-harness\lm_eval\tasks\translation.py", line 126, in __init__
    super().__init__()
  File "D:\github\2023\EleutherAI\lm-evaluation-harness\lm_eval\base.py", line 477, in __init__
    self.download(data_dir, cache_dir, download_mode)
  File "D:\github\2023\EleutherAI\lm-evaluation-harness\lm_eval\tasks\translation.py", line 130, in download
    self.src_file, self.ref_file = sacrebleu.download_test_set(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)
```
full traceback here: https://gist.github.com/RomaKoks/b7f5fcc47296f31e5a19860febf06e46

<img width="747" alt="2023-06-09_124148" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/8873242/1d543d20-9b09-49b5-acd1-f8ae3b9e51fa">

as we can see in the screenshot, `get_files` returns a list of 5 files, but tuple of two were expected


Closed as we have solved this problem.
Repository: EleutherAI/lm-evaluation-harness
Issue: Save the output predictions in addition to accuracy values
Root cause: Documentation Deficiency

Hi @StellaAthena, @haileyschoelkopf 

I am able to run this repo and thanks for your help and wonderful setup.

I would like to save the output predictions (in addition to the final accuracy values) for all the examples for a particular dataset! Does your repo support that?

Thanks
Srinath

Hi! Very glad you like the library!

We are currently working on a major overhaul in the background on the “big-refactor” branch. This WIP PR: https://github.com/EleutherAI/lm-evaluation-harness/pull/563/files is targeted toward that branch but the contribution should be directly port-able to the main branch if you would like to use this locally and log any quantities beyond what are in that PR per example!

Thanks for the response @haileyschoelkopf 

Sure, I can make the changes from the WIP branch in my local machine to log the outputs. And will update you incase I face some issues.

A quick question - Can I know when this PR might be merged though? Incase I want to pull and work on the updated PR

> Thanks for the response @haileyschoelkopf
> 
> Sure, I can make the changes from the WIP branch in my local machine to log the outputs. And will update you incase I face some issues.
> 
> A quick question - Can I know when this PR might be merged though? Incase I want to pull and work on the updated PR

We’re hoping to open it up for usage this month and completely replace the current implementation over the summer. One of the major lifts will simply be implementing each evaluation task in the new format.

@NamburiSrinath I think what you need is --write_out --output_path. They are already implemented. For example, python main.py --model gpt3 --model_args engine=text-davinci-003 --tasks hendrycksTest-* --num_fewshot 5 --write_out --output_path "gpt3-mmlu"

Closing as this was implemented.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to run eval on multi devices？
Root cause: Documentation Deficiency

Hi, how to run evaluation on multi-GPU or multi-nodes?
Set model to "hf-causal-experimental" and?

I tried this:
```
torchrun --nproc_per_node=8 main.py --model hf-causal-experimental ....
```
and only the first gpu is used.

I have the same question. It looks that the code does not support data parallel. Maybe we should try https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html by ourselves?

Hi! We support naive pipeline parallelism via `--model hf-causal-experimental` and `--model_args use_accelerate=True,...` 
 and launching via main.py without torchrun, though this is slow and will be replaced.

We have data parallel support via `accelerate` in the big-refactor branch here: https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor though this is a rework of the library that is not yet fully feature-complete.

Hope this helps!

> Hi! We support naive pipeline parallelism via `--model hf-causal-experimental` and `--model_args use_accelerate=True,...` and launching via main.py without torchrun, though this is slow and will be replaced.
> 
> We have data parallel support via `accelerate` in the big-refactor branch here: https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor though this is a rework of the library that is not yet fully feature-complete.
> 
> Hope this helps!

@haileyschoelkopf 
Should I run with acclerate?
```
accelerate launch main.py --model hf-causal-experimental --model_args use_accelerate=True,...
```
Or just:
```
python main.py --model hf-causal-experimental --model_args use_accelerate=True,...
```

On the `master` branch: just `python main.py --model hf-causal-experimental --model_args use_accelerate=True,...`

On the `big-refactor` branch: use `accelerate launch main.py --model hf-causal ...` !
Repository: EleutherAI/lm-evaluation-harness
Issue: Puzzling behavior, larger models take less time to evaluate
Root cause: Configuration Error

Hello,

Thank you very much for this awesome evaluation tool.

I've been using it to evaluate my models and reproduce some results on Open LLM Leaderboard. I'm having a kind of mystery that I'm trying to wrap my head around: 

1. I'm evaluating LLaMA base models. I noticed that for 30B models with `hf-causal-experimental` takes quarter of the time needed for 13B models with `hf-causal`. 

2) 30B model throws an OOM when running with `hf-causal` but runs fine with `hf-causal-experimental`. 

Any input would be much appreciated!

Sorry, silly mistake on my side. It turns out my 13B models were float32 whereas my 30B models are float16. I still do not know the source of (2) though. Also, running with `hf-causal-experimental` takes less CPU RAM.
Repository: EleutherAI/lm-evaluation-harness
Issue: Can't read json file with pip install or git
Root cause: Architectural Constraint

Hi, when install this pkg with pip:
```
pip install lm-eval
```
or through git:
```
pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git
```
This will not package the file in bigbench_resources, and raise exception.
That means the only way to install this pkg is editable installation, any better way?

Hi, thanks for pointing this out! The best solution to this is to package the files in `bigbench_resources` with a dataset loading script and host them as a HuggingFace dataset. We can try to do this soon, or if you are interested in taking it on in the meantime happy to provide guidance!

@haileyschoelkopf 
OK, please give me some guidance. Thanks!

Of course! Huggingface provides a tutorial on creating new datasets at https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md . The first thing you'll have to do is make a dataset loading script (https://huggingface.co/docs/datasets/dataset_script) , with a different config for each Big-bench task that's supported-- https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/datasets/hendrycks_math/hendrycks_math.py may be a good example of a loading script to build on! Let me know if you run into any big issues.

Hi, @haileyschoelkopf 
Thanks for you guidance!
I will work on it.
I notice that there is [tasksource/bigbench](https://huggingface.co/datasets/tasksource/bigbench) and it is an archive. Maybe it is from the origin repo of [bigbench](https://github.com/google/BIG-bench/). I found there is some minus differences between the bigbench of this repo and the origin repo and I think the number of the bigbench tasks in this repo is likely to grow. So my thought is to get the json files list from this repo and download(no need to update loading script every time the tasks update), but I didnt find any way to do that using `datasets`.
Any solutions or I just compress these json files into an archive?

Hi @SingL3 , I apologize for not replying sooner! For the current set of Big-bench tasks we support, I'd prefer if we stick to the existing JSON files as possible, because @yurodiviy did take care to confirm parity between this code and the big-bench evaluation code. It seems like the files in that repo have some divergences like truncating numbers of examples which I'm concerned about.

Hi, @haileyschoelkopf 
I think #653 is enough? No need to use a datasets for now.

Thanks for the contribution! This is fine for now, but in future we'll also want to use HF datasets to not host this data in the repository.
Repository: EleutherAI/lm-evaluation-harness
Issue: do you have a plan to support hf_hub_ctranslate2 ? 


Is there a reason we should? I’ve never heard of this before today.

> 

There's a trend quantified models are more likely to be used in practice. Like GPTQ, Ctranslate2 is another quantification framework, and hf_hub_ctranslate2 is for connecting Transformers on HuggingfaceHub with Ctranslate2 . Guess it should be a good feature.

You are welcome to open a PR submitting this.

Closing because we don't currently have support for this library on the roadmap, but we would welcome a contribution integrating this library as an LM type!
Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] Collecting Feedback + Bug reports

We have a revamp of the Evaluation Harness library internals staged on the `big-refactor` branch! It is far along in progress, but before we start to move the `master branch` of the repository over to this new design, we'd like to ensure that it's been tested by outside users and there are no glaring bugs!

We’d like your help to test it out! you can help by:
1.Trying out your current workloads on the big-refactor branch, and seeing if anything breaks or is counterintuitive,
2. Porting tasks supported in the previous version of the harness to the new YAML configuration format described in https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/docs/new_task_guide.md !

If you choose to port a task not yet completed according to https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/lm_eval/tasks/README.md , then you can contribute it by opening a PR containing [Refactor] in the name with: 
- A shell command to run the task in the master branch, and what the score is
- A shell command to run the task in your PR branch to big-refactor, and what the resulting score is, to show that we achieve equality between the two implementations!

Lastly, we'll no longer be accepting new feature requests beyond those that are already open to the master branch as we carry out this switch to the new version over the next week, though we will be accepting bugfixes to master branch and PRs to big-refactor!

Happy to answer any questions you may have! 

This issue is intended to collate the problems encountered by users when trying out the `big-refactor` branch. Please feel free to reply down here with any confusions , errors, bugs, or otherwise encountered when using the `big-refactor` branch!

Closing as this is a bit stale, please get in touch with us at our discord in the `#lm-thunderdome` channel if you have any feedback on the `big-refactor` branch or are interested in helping port the last of the unported tasks to that new version.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Big-Refactor] `greedy_until` only compatible with HF Evaluate metrics
Root cause: Unimplemented Feature Gap

The `greedy_until` output_type only works with metrics found in HF Evaluate. Need alternative cases where other metrics is possible.

https://github.com/EleutherAI/lm-evaluation-harness/blob/24e3e3fa9d9cb5180212d4818f0c9b9bf6ceacbe/lm_eval/api/task.py#L895-L898

This is a strange bug to have reported, as this function predates the HF Evaluate library. What happens when you try to call another metric? 

Oh this particular few lines of implementation was written by me for the big-refactor. It's a lack of feature completeness actually, I only took to account HF Evaluate Library for evaluating `greedy_until` and left out the original metrics that were already implemented in lm-eval. This implementation also does not allow cases where a user might want to introduce their own metrics. 

The issue is meant to keep track of things to do. 

#648 closes this.
Repository: EleutherAI/lm-evaluation-harness
Issue: Labels in few-shot examples for `multiple_choice` are casted as Integers
Root cause: Algorithmic Error

In the `big-refactor` branch, when `num_fewshot` argument is non-zero, the labels in the in-context examples prepended to `multiple_choice` output_type are cast as integers instead of the string value. See the below example in sciq with `num_fewshot=1` (inputs are truncated)

```32.2 Pollination and Fertilization For fertilization to occur in angiosperms, pollen has to be transferre.......ovary form the fruit in which the seeds develop. The seed, when mature, will germinate under favorable conditions and give rise to the diploid sporophyte.
Question: For fertilization to occur in angiosperms, pollen has to be transferred to the stigma of what?
Answer:3

But transgenic animals just have one novel gene. What about an animal with a whole new genome? Could a clone , a genetically exact copy of an organism, be developed using techniques associated with biotechnology? It could be argued that human cloning is one of the inevitable outcomes of modern biotechnology. ........But animal cloning is arguably a different story.
Question: What term in biotechnology means a genetically exact copy of an organism?
Answer:
---------------------
 clone
 ```
Models are asked to evaluate the loglikelihood of the string continuation below the dash line. However this will affect model prediction if the in-context examples have integer-value answers that models are not able to understand. @lintangsutawika 

@RosenZhang I made a patch for this. It uses the argument `gold_alias` to help differentiate between the gold target and the string version to fill the few-shot samples.

https://github.com/EleutherAI/lm-evaluation-harness/pull/598

I think this was addressed in #598 ? Please feel free to reopen, @RosenZhang , if not! Thanks :)
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the LAMBADA evaluation
Root cause: Unimplemented Feature Gap

> The LAMBADA dataset [PKL+16] tests the modeling of long-range dependencies in text – the model is asked to predict the last word of sentences which require reading a paragraph of context. It has recently been suggested that the continued scaling of language models is yielding diminishing returns on this difficult benchmark. [BHT+20] reflect on the small 1.5% improvement achieved by a doubling of model size between two recent state of the art results ([SPP+19]and [Tur20]) and argue that "continuing to expand hardware and data sizes by orders of magnitude is not the path forward”. We find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.



Original dataset is available at the link below:

https://zenodo.org/record/2630551

this is the dataset openAI used for evaluating GPT2 https://github.com/EleutherAI/lm_evaluation_harness/pull/39
(reference: https://github.com/openai/gpt-2/issues/131)

@anishthite is this done/are you still working on this?

I think I'll be taking over this one
Repository: EleutherAI/lm-evaluation-harness
Issue: inference gets slower and slower
Root cause: Algorithmic Error

I observe that the inference time become longer and longer even after accounting for the request sequence length? Did you observe the same pattern? Is it related to GPU memory caching usage?

Hi, sorry for the slow response on this!

Do you have a specific task for which this occurs, so that I can see if I can replicate this myself?

For loglikelihood_rolling and loglikelihood (multiple choice tasks, as well as anything calling `rf.loglikelihood` or `rf.loglikelihood_rolling`, this is definitely unintended behavior--we sort in descending order by sequence length which should have the effect of speeding things up as time goes on.

For `greedy_until` (generation-based tasks), I noticed that we mistakenly do not yet sort in descending order, so fixing this in #616 . Can follow up further, if you are facing slowdowns beyond what might be expected in this PR!
Repository: EleutherAI/lm-evaluation-harness
Issue: Trying to extract raw per-question data
Root cause: Documentation Deficiency

Hi folks, I am trying to have the evaluation drop me the raw per "question" metrics in some format like {task, question, model, metric, score} into an output file (ideally 1 file per task). 

I'm having a little bit of trouble getting it to work, it seems like I want to just dump the contents of vals in https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/evaluator.py#LL321C13-L321C52 but I don't see any way of knowing exactly which question is being answered. 

I wonder if there is already an easy way to do this (either have index of the row in the original dataset or a hash of the question text or something) without hacking up the entire codebase.

Thanks!


Hi, I'm sorry for the slow response on this! 

We now support this in the `big-refactor` branch, which will be our newest version release soon. You can use it now via `--log_samples` on that branch: https://github.com/EleutherAI/lm-evaluation-harness/blob/1710b42d52d0f327cb0eb3cb1bfbbeca992836ca/main.py#L80 I hope this helps, and if you want any additional contents/metrics to be logged per-sample, definitely let us know, it'd be useful info to keep in mind!

> Hi, I'm sorry for the slow response on this! 
> 
> We now support this in the `big-refactor` branch, which will be our newest version release soon. You can use it now via `--log_samples` on that branch: https://github.com/EleutherAI/lm-evaluation-harness/blob/1710b42d52d0f327cb0eb3cb1bfbbeca992836ca/main.py#L80 I hope this helps, and if you want any additional contents/metrics to be logged per-sample, definitely let us know, it'd be useful info to keep in mind!

Hey guys, @haileyschoelkopf 
Along similar lines - 

What's the easiest way to get the model's `per sample predicted & decoded token` for tasks like Lambada? So basically, to compare `predicted word` vs `target word` for each sample.

Instead of just a resp of True/False? 

Hi @shikhar-srivastava ,

Have you found a solution?
I am also looking for the solution for logging predicted words.

Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Slow evaluation for 7B model using A100 multi-gpu
Root cause: Documentation Deficiency

Hello, I want to evaluate some 7B models using multi-gpu on a cluster of tasks. Right now I use the [master branch's latest commit ](https://github.com/EleutherAI/lm-evaluation-harness/commit/b281b0921b636bc36ad05c0b0b0763bd6dd43463) and run the following commands:
```shell
MODELS=(
  'EleutherAI/gpt-j-6b'
  'yahma/llama-7b-hf'
  "gpt2-xl"
  "togethercomputer/RedPajama-INCITE-7B-Base"
  'facebook/opt-6.7b'
  "stabilityai/stablelm-base-alpha-7b"
)
for model in ${MODELS[@]}; do
    for shot in 0 1 5; do
        echo $model $shot
        python main.py \
            --model hf-causal-experimental \
            --model_args pretrained=$model,use_accelerate=True,dtype="bfloat16" \
            --tasks anli_r1,anli_r2,anli_r3,arc_challenge,arc_easy,boolq,cb,copa,headqa,hellaswag,multirc,record,rte,wic,wsc,lambada_openai,lambada_standard,logiqa,winogrande,sciq,openbookqa,piqa \
            --batch_size 64 \
            --device auto \
            --output_path results/$model/$shot.json \
            --num_fewshot $shot
    done
done
```
Basically it runs every model in `MODELS` with 0,1,5 shots on several tasks. Running gpt-j-6b 1 shot gives me 231k total examples to evaluate and takes about 8 hours to finish 0 shot on 8 A100-40G GPUs (each takes about 34G memory usage, around 8.7 instances per second). I wonder if this is an expected speed since it seems to be slow (e.g. running all might takes 8x3x6 = 144 hours). I tried bf16 (by `dtype="bfloat16"`) but I am not yet sure if bf16 is used. I also made `use_accelerate=True`. Is there any way to speed up this process?
Thank you!

Yeah that sounds about right. You have a huge list of tasks, after all.

Are there other contexts where you get substantially more than 8.7 forward passes through a 6B model per second?

Hi @StellaAthena thanks I am still new to the field so you probably is right that this is an expected speed. Just want to check if I enable all accelerating configurations. Thanks. I am closing this issue.