zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Issue with Running Example in README
Root cause: Documentation Deficiency

I encountered an issue while trying to run the example provided in the "Model APIs and Inference Servers" section of the README. Specifically, I get an error when attempting to call a hosted model using the provided example commands.

When I run
```
lm_eval --model openai-completions \
    --model_args engine=davinci \
    --tasks lambada_openai,hellaswag
```

I get an error:
```
Traceback (most recent call last):
  File "/Users/lenni/coding-projects/lm-evaluation-harness/lm_eval/api/registry.py", line 34, in get_model
    return MODEL_REGISTRY[model_name]
KeyError: 'openai-completions'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lenni/anaconda3/envs/llm-evals/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/Users/lenni/coding-projects/lm-evaluation-harness/lm_eval/__main__.py", line 207, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/Users/lenni/coding-projects/lm-evaluation-harness/lm_eval/utils.py", line 404, in _wrapper
    return fn(*args, **kwargs)
  File "/Users/lenni/coding-projects/lm-evaluation-harness/lm_eval/evaluator.py", line 102, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/Users/lenni/coding-projects/lm-evaluation-harness/lm_eval/api/registry.py", line 36, in get_model
    raise ValueError(
ValueError: Attempted to load model 'openai-completions', but no model for this name found! Supported model names: hf-auto, hf, huggingface, gooseai, openai-chat-completions, textsynth, dummy, anthropic, gguf, ggml, vllm
```

Switching to `openai-chat-completions` for the `--model` arg I then get an error notifying me that 
```
TypeError: __init__() got an unexpected keyword argument 'engine'
```

Are there any OpenAI models with log probabilities currently supported? If so it would be helpful to provide an updated example in the README for new users like myself.  Thanks for all you do!


Hi! The completions end-point was renamed `gooseai` because OpenAI are depreciating the completions end-point next month. There were some errors but fixed them in this [PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/1139).

fixed in #1139 #1141 !
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the BioASQ evaluation
Root cause: Unimplemented Feature Gap

> This article provides an overview of the first BIOASQ challenge, a competition on large-scale biomedical semantic indexing and question answering (QA), which took place between March and September 2013. BIOASQ assesses the ability of systems to semantically index very large numbers of biomedical scientific articles, and to return concise and user-understandable answers to given natural language questions by combining information from biomedical articles and ontologies.


https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-015-0564-6
Repository: EleutherAI/lm-evaluation-harness
Issue: Consolidate Reorderer, Grouper and chunks
GitHub labels: feature request

I think we should consolidate these three into one data-loader. Will make the code a bit easier to follow.

>TODO: overhaul reorderer. It currently grouped requests by content but we don't want this

@haileyschoelkopf Just starting on this. I don't quite understand what this means in Reorderer. Why wouldn't we want to order by content? Also what would be a good way to overhaul it.

iirc, before I made a change to it, if there were 2 requests with the same input + output, reorderer would compress those into one request to run them and get the same output.

For the case where we run with repeats > 1 in the config, we don’t want this (although we probably want to complain somewhere if a user tries this with temp=0 or with loglikelihood tasks).

Can't really figure out why we group the requests before sorting though in this. Seems a bit redundant but maybe a good reason behind it?

Are you referring to the insides of the reorderer? if so yeah it can definitely be simplified.
Repository: EleutherAI/lm-evaluation-harness
Issue: tqdm progress bar cannot be kept on the same line when using vllm
Root cause: Configuration Error

When using hf model, tqdm progress bar works well. However, after switching to vllm model, got the following weird pattern. Any quick solution to force the progress bar is updated on the same line?

using the latest code, v0.4.0


```bash
Processed prompts: 100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 64.84it/s]
Processed prompts: 100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 64.69it/s]
Processed prompts: 100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 65.09it/s]
Processed prompts: 100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 65.38it/s]
Processed prompts: 100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 65.17it/s]
Processed prompts: 100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 65.16it/s]
100%|██████████████████████████████████████████████████████████| 1997/1997 [00:32<00:00, 61.29it/s]
```

It seems that using `--batch=auto` will solve this issue. 

Without using it, `n` in `utils.chunks()` won't be its default value 0. Following on this, a separate tqdm bar will appear.

https://github.com/EleutherAI/lm-evaluation-harness/blob/bd0f2414585df9209db2f4ab0430912ed000f6a0/lm_eval/models/vllm_causallms.py#L348
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the emrQA evaluation
Root cause: Unimplemented Feature Gap

> We propose a novel methodology to generate domain-specific large-scale question answering (QA) datasets by re-purposing existing annotations for other NLP tasks. We demonstrate an instance of this methodology in generating a large-scale QA dataset for electronic medical records by leveraging existing expert annotations on clinical notes for various NLP tasks from the community shared i2b2 datasets. The resulting corpus (emrQA) has 1 million question-logical form and 400,000+ question-answer evidence pairs. We characterize the dataset and explore its learning potential by training baseline models for question to logical form and question to answer mapping.

https://arxiv.org/abs/1809.00732

So it looks like, this dataset is behind a license agreement @leogao2, what do you want to do 


> So it looks like, this dataset is behind a license agreement @leogao2, what do you want to do

What is the license agreement?
Repository: EleutherAI/lm-evaluation-harness
Issue: Print "higher_is_better" in results table
Root cause: Unimplemented Feature Gap

We should add a column to the printed results table, that displays an arrow pointing up if the metric's `"higher_is_better"` value is True, arrow pointing down otherwise. 

Currently `higher_is_better` is not really used in the code.

This can be a part of #1158

@haileyschoelkopf I believe this issue should be closed now.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Discussion/Feedback] VLM + Multimodal benchmarking
Root cause: Unimplemented Feature Gap

This is an issue to discuss the feasibility / desirability of multimodal benchmarks to be included and supported in lm-eval. With the increase in multimodal (V)LMs and in benchmarks like MMMU designed to test these models, it's worth discussing either whether lm-eval is easily extensible to these tests or whether we wish to have these in scope for the library.

There is already a fork of lm-evalaution-harness built by the LLaVa team called [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main) that is focused on VLM evaluation that can serve a PoC. For what is worth, I already have a private fork of lm-eval that has added LLaVa support and is working with MMMU and the LLava 1.5 7B model through the original codebase. I think the main considerations of an implementation are the following:

- How tightly do we want to couple VLM and MM benchmarks? Should VLM only be able to run MM benchmarks should we enable both LM and VLM to run MM benchmarks. Do we need to ensure the VLM can run text -only benchmarks 
- Are VLM first party citizens? Do we need to focus on optimizations for these? For example VLLM just introduced VLM support a couple of days ago. Do we need to pull the latest benchmarks
- Significant effort needs to be invested in optimization of the VLMM benchmarks as image data/video data in particle is very big, at what point does it becomes a problem

If the project owners are aligned on these questions, I can clean up and submit a PR for my LLaVa implementation.

Wanted to follow up

Just wanted to give a heads up that I started the PR.
Repository: EleutherAI/lm-evaluation-harness
Issue: [Discussion] Add Major Code Benchmarks
Root cause: Unimplemented Feature Gap

Following #1152 we'll be willing to support code benchmarks, which we previously had not been supporting due to not wanting to endorse un-sandboxed model-generated code execution.


This issue is meant to collect discussion and solicit feedback on code generated benchmarks:

- What benchmarks are most crucial to support?
- What papers are best references for prompting / reference benchmark numbers / etc.
- Known drawbacks of these benchmarks (e.g. HumanEval's underspecified unit tests)


- Progress on implementing these benchmarks:
  - [ ] MBPP
  - [ ] HumanEval (+)
  - [ ] HumanEvalPack
  - [ ] Program-of-Thought
  - [ ] PAL (Program-Aided LMs) 
  - [ ] ???



Can we first support MBPP and HumanEval? These two have already been widely used and reported.

Let's look at supporting calling the [BigCode Eval Harness](https://github.com/bigcode-project/bigcode-evaluation-harness) instead of doing it from scratch ourselves. This could save time, but my primary motivation is wanting to not undermine their work and promote the ecosystem more broadly.

Spoke with @clefourrier regarding code tasks, and have some updates here:

the Bigcode harness is still used actively, but it will predominantly stay at its current version as the maintainers have other new responsibilities and projects. They've stated we are welcome to incorporate BigCode tasks/features into the Eval Harness and are happy to answer questions. 

So code tasks are now on the table--our preference would be to keep all model-generated code execution to be done offline, in a second step so that potentially-unsafe code is not run except on safe machines. The ideal path would be to first implement a general utility `lm_eval score`  to allow for re-scoring logged samples #1627 , then have user-defined metrics requiring model-generated code execution to be run at this stage on a separate machine if desired. 

cc @loubnabnl who was the core developer of the BigCode Harness
Repository: EleutherAI/lm-evaluation-harness
Issue: Unify "metric" and "aggregation" abstractions
Root cause: Architectural Constraint

@lintangsutawika is working on this right now so creating an issue to track it.

unifying the metric and aggregation functions / functionalities, so that one should only specify a single `metric` that will be called at the dataset level. 

Could go hand in hand with a revamp of `process_results()` that operates on a per-task/per-dataset level rather than, as requested by @pminervini .

Should reduce cognitive load on users, as well as make it easier to understand + add a new metric function within the code.

Closing as no longer planned, I don't think we want to break existing reported per-sample metrics in logged sample files unfortunately.
Repository: EleutherAI/lm-evaluation-harness
Issue: More Flexible Answer Extraction Code
Root cause: Unimplemented Feature Gap

In LM Evaluation Harness, we work to match the "original" / "default" methods used to evaluate datasets. This includes using whatever answer extraction / post-processing is done by the original code implementations if provided, even if such extraction is flawed and may miss correct-in-substance-but-not-form answers.


Where appropriate and requested, we may consider adding more flexible answer extraction code. This has been requested by many users. I think a good middle ground might be to support both a `standard` and `loose` filter pipeline or metric for various datasets, such as for GSM8k: scoring based only on the last number output from the model, as opposed to scoring `#### {number}` only as correct but not things such as `so, the answer is {number}`.

There's definitely a balance to be struck though between being too permissive and being sufficiently flexible s.t. benchmarks aren't just a test of performing the right formatting steps + incantations for a model.

This issue is to track our addition of such flexibility / improvements, and solicit requests or feedback on this.

Is this PR related -- https://github.com/EleutherAI/lm-evaluation-harness/pull/943 ?

Yup! This and the triviaqa ones are good examples of what we’ll want to handle.

Ideally we can use multiple filter pipelines for this purpose.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the BioREAD evaluation
Root cause: Unimplemented Feature Gap

> We present BioRead, a new publicly available cloze-style biomedical machine reading comprehension (MRC) dataset with approximately 16.4 million passage-question instances. BioRead was constructed in the same way as the widely used Children’s Book Test and
its extension BookTest, but using biomedical journal articles and employing MetaMap to identify UMLS concepts. BioRead is one of
the largest MRC datasets, and currently the largest one in the biomedical domain. We also provide a subset of BioRead, BioReadLite,
for research groups with fewer computational resources. We re-implemented and tested on BioReadLite two well-known MRC methods,
AS Reader and AOA Reader, along with four baselines, as a first step towards a BioRead (and BioReadLite) leaderboard. AOA Reader
is currently the best method on BioReadLite, with 51.19% test accuracy. Both AOA Reader and AS Reader outperform the baselines by
a wide margin on the test subset of BioReadLite. Our re-implementations of the two MRC methods are also publicly available.

https://www.aclweb.org/anthology/L18-1439/
Repository: EleutherAI/lm-evaluation-harness
Issue: Revamp + automate Task Table documentation feature
Root cause: Unimplemented Feature Gap

We used to have a docs page at https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_table.md that listed out all the tasks callable in lm-eval, along with some basic info about them. Its construction was also automated, so one could just run a Python script to regenerate it with new tasks included.

It'd be great to revitalize this now that we have a bunch more groups / subtasks / variants we've added or intend to add.

New features that would be awesome would be:
- [ ] List out all individual tasks, along with a few key config attributes (output_type, in particular)
- [ ] as a separate sub-table: list out all task groups that are supported
- [ ] (Stretch) allow one to specify a one-sentence `description` field within a YAML config's `metadata` field, that just says what the task is e.g. 'Abstract Algebra subject in the popular MMLU benchmark, as implemented by original paper' or 'variant of MMLU as implemented by FLAN 2022 paper.' or 'Benchmark for testing formal logic capabilities' that we could auto-populate a blurb column of this table with.


I could probably set this up if @lintangsutawika could provide me a stub to read all the configs from the task folder. 

Would this help?

https://github.com/EleutherAI/lm-evaluation-harness/blob/main/scripts/make_table_tasks.py

yeah this is good. But I was looking through the `tasks/__init__` file but couldn't find an easy way to read the configs without initialising the tasks. It's bound to be in there somewhere!

closed by #1929 for now
Repository: EleutherAI/lm-evaluation-harness
Issue: Error loading some HF models
Root cause: Documentation Deficiency

Hello, 

It seems that there is a problem loading some HF models. I got the following error using the model core42/jais-13b:

```
ValueError: Loading core42/jais-13b requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.
```
Is there a quick fix for this issue? 

Hi, you’ll need to include `—model_args trust_remote_code=True` to run this model.
Repository: EleutherAI/lm-evaluation-harness
Issue: Cannot use task `ifeval`
Root cause: Environment Incompatibility

I think this might be the reason -- working on it rn:

```bash
>>> import lm_eval
>>> from lm_eval.tasks.ifeval import instructions_util
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "XXX/lib/python3.10/site-packages/lm_eval/tasks/ifeval/instructions_util.py", line 22, in <module>
    import immutabledict
ModuleNotFoundError: No module named 'immutabledict'
```

just found that the missing dependencies can be installed with ".[ifeval]"
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the BMKC evaluation
Root cause: Unimplemented Feature Gap

> In this work, we introduced a new task of machine comprehension in the biomedical domain using a deep neural model. Since there was no large-scale dataset for training deep neural models in the biomedical domain, we created the new cloze-style datasets Biomedical Knowledge Comprehension Title (BMKC_T) and Biomedical Knowledge Comprehension Last Sentence (BMKC_LS) (together referred to as BioMedical Knowledge Comprehension) using the PubMed corpus. The experimental results showed that the performance of our model is much higher than that of humans. We observed that our model performed consistently better regardless of the degree of difficulty of a text, whereas humans have difficulty when performing biomedical literature comprehension tasks that require expert level knowledge.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5783222/
Repository: EleutherAI/lm-evaluation-harness
Issue: Inconsistency in Option Column Naming and Dataset Names in K-MMLU Evaluation Harness
Root cause: Interface Contract Mismatch

Column names for the options are named as A, B, C, and D here:
https://huggingface.co/datasets/HAERAE-HUB/K-MMLU-Preview/blob/main/data/Accounting-test.csv
But the utils.py file tries to read options from different names:
https://github.com/EleutherAI/lm-evaluation-harness/blob/7bc9b82bbfc1f8293518d24a648002bd484985e3/lm_eval/tasks/kmmlu/utils.py
```python
f"(1) {doc['option#1']}\n(2) {doc['option#2']}\n(3) {doc['option#3']}\n(4) {doc['option#4']}\n"
```

Also, dataset names include hyphens but the config files indicate names with spaces.
https://huggingface.co/datasets/HAERAE-HUB/K-MMLU-Preview/tree/4a310afe5771900afcdbbd43bfad7d316775a271/data
![Screenshot 2023-12-19 at 11 45 52 PM](https://github.com/EleutherAI/lm-evaluation-harness/assets/115020208/ff3a05c7-613e-4a50-85c8-f25d3b20625f)
https://github.com/EleutherAI/lm-evaluation-harness/blob/7bc9b82bbfc1f8293518d24a648002bd484985e3/lm_eval/tasks/kmmlu/kmmlu_agricultural%20sciences.yaml


Thank you for your contribution @seungduk-yanolja . But I am getting this error when running the current code. Is it only me?
<img width="771" alt="스크린샷 2023-12-20 오후 8 55 38" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/88217825/d6829a7f-3ff6-4852-8741-6b45aaec22ce">


> Thank you for your contribution @seungduk-yanolja . But I am getting this error when running the current code. Is it only me?
> <img width="771" alt="스크린샷 2023-12-20 오후 8 55 38" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/88217825/d6829a7f-3ff6-4852-8741-6b45aaec22ce">
> 

Can you provide the full error message please?

I dont have the full error message at this moment but that was just most of it. If this is a universal problem I have a working code on my branch that i can contribute. Can @seungduk-yanolja you try if you see the same error?

I've figured out the problem it seems to be an issue with the original implementation not this pr. To replicate the problem try:
```
lm_eval --model hf \
    --model_args pretrained=EleutherAI/polyglot-ko-1.3b,dtype=float16 \
    --num_fewshot 1 \
    --batch_size 32 \
    --tasks kmmlu\
    --log_samples\
    --output_path "results"
```

The num_fewshot option seems to cause the problem
Repository: EleutherAI/lm-evaluation-harness
Issue: Update Zeno Integration
Root cause: Unimplemented Feature Gap

There are a few things we'll need to do to fix edge cases in our Zeno integration.


- [ ] log info on the groups tasks may or may not have been called as part of, as well as which tasknames are themselves groups, as part of the results.json file we save
- [ ] Ensure Winograd Schema / minimal pair -like tasks (Winogrande, BLIMP, CrowS-Pairs) are handled intelligently in the Zeno export script
- [ ] Any other changes on our end to beautify Zeno projects created?
- [ ] Any tests we can set up to ensure Zeno support is not broken?

cc @lintangsutawika for your awareness because changing how metrics / aggregations are computed to unify them might harm Zeno's reliance on per-example metrics. 

cc @Sparkier !

Regarding tests, we could run a test on our end. E.g., we have integration tests set up where we have a project created on push. Would not alert you if anything breaks, though. I can also help set up a test that uploads a project to Zeno in your codebase if you want to.

> Any other changes on our end to beautify Zeno projects created?

We've seen some patterns in useful metadata recently. For example, the model output length in freeform answers is often interesting. We could think about additional metadata that would make sense per task to further enhance the created Zeno project.

For tests, see https://github.com/EleutherAI/lm-evaluation-harness/pull/1221

For additional metadata, see https://github.com/EleutherAI/lm-evaluation-harness/pull/1222
Repository: EleutherAI/lm-evaluation-harness
Issue: Using harness with a local internal API
Root cause: Documentation Deficiency

Hello there, 

Is there a way to run the evals with a local internal API URL of a model? Do I have to create a class under lm_eval/models and implement some specific models (if so what are they)? Any idea where to start? 

Thank you. 

Hi! Assuming that your local API can be called via the same interface as OpenAI via setting `openai.OpenAI(base_url=base_url)`, This will be addressed and documented in #1174 ! 

ChatCompletions support for arbitrary API providers is now merged. The same changes to `OpenAIChatCompletionsLM` will need to be ported to `OpenAICompletionsLM` to support non-chat models in this way--this is planned to be added asap, but if you'd like to take this on in the meantime we'd welcome a PR as well!

Also, if your internal API is different from the openai one you can still implement custom support! You can use that file as a guide... basically you need to implement the three request types (loglikelihood, loglikelihoodrolling, and generate_until).

+1, see https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md for a walkthrough on adding custom support.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the SciQ evaluation
Root cause: Unimplemented Feature Gap

> The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry and Biology, among others. The questions are in multiple-choice format with 4 answer options each. For the majority of the questions, an additional paragraph with supporting evidence for the correct answer is provided.

https://allenai.org/data/sciq