zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Possibe Implementation for Cross Evaluation
Root cause: Documentation Deficiency

Thank you so much for the amazing API! I am wondering whether there's a way to implement cross-evaluation since I have not found anything in the task-guide metrics section specifically. 
I have a few questions about the metrics:
1, I looked into the implementation and realized that f_1_fn and mcc_fn are all just pass-through functions, so I am planning on doing a similar thing to those since I am dealing with a multiple-choice task. Does that mean I need to modify the `Task.process_results` in `lm_eval/api/task.py`? 
2, Since cross-evaluation should also involve different splits for one dataset, I am planning to achieve this by presetting the different splits in the dataset and then getting the accuracy for five different tasks, then using the aggregation function to combine them. However, I don't quite understand what aggregation function is combining. Is it aggregating the results from each batch? Then if that is the case, is there a better way for me to implement cross-evaluation?

Sorry if any of these have been asked before, I am new to this API, so any suggestions will be very helpful to me. Thank you!



Hi! What exactly are you referring to as "cross-evaluation"-- K-folds?

Yes, currently adding multiple-choice metrics will require modifying `process_results()` for ConfigurableTasks--we plan to clean up the code so such metrics can be added without digging into the api's code for modification.

Aggregation for groupings of tasks is currently done by taking the mean of all subtasks' accuracies/other metrics, weighted by each subtask's number of documents. We also soon hope to have an improved API for better configuration of how task groupings do or do not aggregate their subtasks' scores together so that users have more control over this.

I hope this is helpful!

Thank you so much for the helpful reply Hailey! Yes, I was asking for K-folds, and I will try using grouping to get the metrics I want. Thanks again.
Repository: EleutherAI/lm-evaluation-harness
Issue: Sanity checking the semantic meaning of "perplexity" in code
Root cause: Documentation Deficiency

I noticed here https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L1178

that the  "perplexity" key is paired with the variable `ll`, short for log likelihood. Could someone please clarify whether the value of the variable the log likelihood or the perplexity?

This metric is currently only used for Lambada--each doc records the loglikelihood of the target string, and then it is aggregated by the `perplexity` aggregation https://github.com/EleutherAI/lm-evaluation-harness/blob/7d9922c80114218eaf43975b7655bb48cda84f50/lm_eval/api/metrics.py#L36-L38 to average and get perplexity from the per-instance loglikelihoods (in Lambada's case, the target strings are each approximately one word which is why this is used.)
Repository: EleutherAI/lm-evaluation-harness
Issue: XNLI weird result with gemma-2b
Root cause: Validation Gap

Evaluating gemma-2b with xcopa looks good, but the xnli result looks weird. 

xcopa result:
```
  "results": {
    "xcopa_zh": {
      "acc,none": 0.616,
      "acc_stderr,none": 0.021772369465547194,
      "alias": "xcopa_zh"
    },
    "xcopa_vi": {
      "acc,none": 0.674,
      "acc_stderr,none": 0.02098400956239357,
      "alias": "xcopa_vi"
    },
    "xcopa_tr": {
      "acc,none": 0.58,
      "acc_stderr,none": 0.02209471322976178,
      "alias": "xcopa_tr"
    },
    "xcopa_th": {
      "acc,none": 0.57,
      "acc_stderr,none": 0.022162634426652835,
      "alias": "xcopa_th"
    },
    "xcopa_it": {
      "acc,none": 0.618,
      "acc_stderr,none": 0.02175082059125084,
      "alias": "xcopa_it"
    },
    "xcopa_id": {
      "acc,none": 0.646,
      "acc_stderr,none": 0.021407582047916447,
      "alias": "xcopa_id"
    }
  },
```

xnli result:
```
| Tasks |Version|Filter|n-shot|Metric|Value |   |Stderr|
|-------|------:|------|-----:|------|-----:|---|-----:|
|xnli_zh|      1|none  |     0|acc   |0.3261|±  |0.0094|
|xnli_vi|      1|none  |     0|acc   |0.3594|±  |0.0096|
|xnli_tr|      1|none  |     0|acc   |0.3458|±  |0.0095|
|xnli_th|      1|none  |     0|acc   |0.3317|±  |0.0094|
|xnli_ru|      1|none  |     0|acc   |0.3390|±  |0.0095|
|xnli_hi|      1|none  |     0|acc   |0.3382|±  |0.0095|
|xnli_fr|      1|none  |     0|acc   |0.3297|±  |0.0094|
|xnli_es|      1|none  |     0|acc   |0.3418|±  |0.0095|
|xnli_en|      1|none  |     0|acc   |0.3554|±  |0.0096|
|xnli_de|      1|none  |     0|acc   |0.3450|±  |0.0095|
|xnli_ar|      1|none  |     0|acc   |0.3390|±  |0.0095|
```
Score around 0.33 is more like random guess?

Maybe @lintangsutawika has some ideas on this. 

As a whole (small?) LMs are pretty bad at the NLI task though I think. Maybe it's a matter of prompting?
Repository: EleutherAI/lm-evaluation-harness
Issue: Package conflicts on minerva_math task for OLMo-1B
Root cause: External Dependency Breakage

I try to use the following command to test OLMo-1B performance on minerva_math task:

`accelerate launch -m lm_eval --model hf --model_args pretrained=path/to/OLMo-1B,max_length=4096,trust_remote_code=True --tasks minerva_math --batch_size $BATCH_SIZE --output_path ./eval_out/${OUT_NAME}/${DATE}`

Then it came out the error like this

> 2024-03-15:17:23:41,455 ERROR    [utils.py:195] LaTeX parsing requires the antlr4 Python package, provided by pip (antlr4-python3-runtime) or conda (antlr-python-runtime), version 4.11

So I install antlr-python-runtime using `pip install antlr-python-runtime==4.11` and do the evaluate command mentioned above, it came out a pip dependency error:

> ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
omegaconf 2.3.0 requires antlr4-python3-runtime==4.9.*, but you have antlr4-python3-runtime 4.11.1 which is incompatible.

The omegaconf 2.3.0 is needed by OLMo: https://github.com/allenai/OLMo/blob/666da70fbd89b50141a3e521ecc0d4e27b351004/pyproject.toml#L18

After antlr-python-runtime==4.11 installed, I try to do the command above again, another exception happened:
> File "/usr/local/lib/python3.10/dist-packages/antlr4/atn/ATNDeserializer.py", line 50, in checkVersion
    raise Exception("Could not deserialize ATN with version " + str(version) + " (expected " + str(SERIALIZED_VERSION) + ").")
Exception: Could not deserialize ATN with version  (expected 4).

Since the antlr-python-runtime==4.11 is needed by sympy=1.12. I try to downgrade sympy==1.10.1, then it evaluated properly
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/45707768/22ea2bd8-325d-455f-b333-753d991a13a7)
However, during runtime there is a warning as follows, don't know if this will have a negative impact
> ANTLR runtime and generated code versions disagree: 4.9.2!=4.7.2

Hi, thanks for reporting this! I will have to take a look and see if there's any good way around this.

the latest dev releases of [omegaconf](https://pypi.org/project/omegaconf/#history) support antlr 4.11 but not sure how stable they are. Should be fine I think. Looks like Olma just uses omegaconf to parse the model configs.

Thank you for your suggestion! I'll try to figure it out.
Repository: EleutherAI/lm-evaluation-harness
Issue: How to randomize fewshots?
Root cause: Configuration Error

One of the tasks' yaml is as follows (it is not first_n sampler)
`
fewshot_split: train

fewshot_config:

  sampler: default
`
If I change the seed, and write_out how the input looks like, the fewshots do not seem to change.
How can I solve this?


So, how did you solve this issue?

I manually changed the seed from lm_eval/api/task.py

In the places mentioned here? https://github.com/EleutherAI/lm-evaluation-harness/pull/1412#issuecomment-1999830981
What do you think, will it be useful to add a PR with control of this seed by arguments, and not by changing code? At least for better reproducibility between different research teams.
Repository: EleutherAI/lm-evaluation-harness
Issue: Negative perplexity values
Root cause: Documentation Deficiency

Hey folks,

So, the perplexity values on a per sample/doc basis are **ALL** negative. 
Can someone explain why this is ?

This is using the `--log-samples` option.

Command:
```
lm_eval --model hf  
 --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float"  
 --tasks lambada_openai     --device cuda:0     --batch_size 32 
 --log_samples 
 --output_path test_nlu_eval/
```
Sample output:
```
{
    "doc_id": 0,
    "doc": {
      "text": "In my palm is a clear stone, and inside it is a small ivory statuette. A guardian angel.\n\n\"Figured if you're going to be out at night getting hit by cars, you might as well have some backup.\"\n\nI look at him, feeling stunned. Like this is some sort of sign. But as I stare at Harlin, his mouth curved in a confident grin, I don't care about signs"
    },
    "target": " signs",
    "arguments": [
      [
        "In my palm is a clear stone, and inside it is a small ivory statuette. A guardian angel.\n\n\"Figured if you're going to be out at night getting hit by cars, you might as well have some backup.\"\n\nI look at him, feeling stunned. Like this is some sort of sign. But as I stare at Harlin, his mouth curved in a confident grin, I don't care about",
        " signs"
      ]
    ],
    "resps": [
      [
        [
          -8.487064361572266,
          false
        ]
      ]
    ],
    "filtered_resps": [
      [
        -8.487064361572266,
        false
      ]
    ],
    "perplexity": -8.487064361572266,
    "acc": 0
  },
```

Hi! Sorry, been meaning to more clearly document the sample log formats and semantic meaning of per-sample metrics for e.g. perplexity.

Here the per-sample `perplexity` values are loglikelihoods of the target string for the document, and the list of these loglikelihoods goes through https://github.com/EleutherAI/lm-evaluation-harness/blob/dc90fecc8ba31b682d30e096980f678c18ddc435/lm_eval/api/metrics.py#L36-L38 to turn it into the dataset-level perplexity. Sorry for the confusion!

Thanks for clarifying!
Another great thing to add would be a `[predicted word]` along with the `[target word]` to the per-sample outputs. 

Any quick ways to get that too?

Ah, unfortunately not currently--though you can tell based on the `0/1` logged for acc on lambada whether the word was correctly predicted.

Ah, I see. Unfortunately, I need access to the `predicted word` for the per-sample output. 
How would you suggest possibly doing that?
Repository: EleutherAI/lm-evaluation-harness
Issue: Repeated Question in Arabic MMLU
Root cause: External Dependency Breakage

I was checking the English to Arabic translation of MMLU ([`ammlu`](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/ammlu)), and I noticed that a question related to masking IPs was repeated 1,568 times in the Arabic dataset. Each time, it overwrites the original question, while still maintaining the original ground-truth.

Interestingly, the question is phrased slightly differently each time, below is a sample with frequency counts:
```
count, question
  309, كيف يتم اكتشاف تزوير عنوان IP؟
  217, كيف يتم اكتشاف تزييف عنوان IP؟
   95, كيف يتم الكشف عن تزوير عنوان IP؟
   85, كيف يتم الكشف عن تزييف عنوان IP؟
   66, كيف يتم كشف تزوير عنوان IP؟
```
This does not appear in the original English dataset
```
count, question
    8, Which of the following is true?
    6, Which of the following statements is false?
    6, Which of the following statements is correct?

    4, Which of the following statements is true?
    4, Which of the following statements is correct?
```
My guess is that this might be an artifact from the GPT translation, where it might have been prompted with the IP masking question as an example.

I found that 1,568 out of all 14,042 questions contained a substring related to IP from the list `['IP', 'بروتوكول الإنترنت','آي بي']`.

I found two datasets on Hugging Face:
- [`Hennara/ammlu`](https://huggingface.co/datasets/Hennara/ammlu): The dataset used in LM Evaluation Harness. Claims that it was translated by GPT-4.
- [`FreedomIntelligence/MMLU_Arabic`](https://huggingface.co/datasets/FreedomIntelligence/MMLU_Arabic): Claims that it was translated by GPT-3.5-Turbo

However, both datasets seem identical.

@khalil-Hennara do you have any insights here?

It may be worth removing the task in light of this.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the OpenBookQA evaluation
Root cause: Unimplemented Feature Gap

From the GPT-3 paper

> On OpenBookQA [MCKS18], GPT-3 improves significantly from zero to few shot settings but is still over 20 points short of the overall SOTA. GPT-3’s few-shot performance is similar to a fine-tuned BERT Large baseline on the leaderboard.

- [X] Data processing code implemented
- [x] Evaluation implemented

The evaluation code should be modeled after the interface in `lm_eval/base.py` and the example of the `BoolQ` task in `lm_eval/tasks/suerglue.py`

Note: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/openbookqa

Working on adding this.

#42
Repository: EleutherAI/lm-evaluation-harness
Issue: Vllm model is broken
Root cause: Interface Contract Mismatch

The changes introduced by [this PR](https://github.com/EleutherAI/lm-evaluation-harness/pull/1584) broke the `VLLM` model.

The issue is that `VLLM` does not have the `AUTO_MODEL_CLASS` now required in the parent class when calling `def _encode_pair(self, context, continuation):`

Tagging @lintangsutawika for visibility. 

Thanks for raising an issue and I'm sorry for the bug. @lintangsutawika is on a plane right now but will get this merged ASAP once he's back to review.

> this PR

Revert `lm_eval/api/model.py`, then I can successfully run with VLLM.

@LuJunru @haileyschoelkopf  Thanks!

Patched in #1601 for now!
Repository: EleutherAI/lm-evaluation-harness
Issue: Make managing task variants / subversions easier
Root cause: Architectural Constraint

We now need to handle support for many many tasks, because tasks can have many slight variations that all need tracking. 

We should see if there is a way to easily handle these (e.g. different prompts / prompting style archetypes, different translations, etc.) s.t. we do not need to have each of those being a separate task--possibly some sort of tagging system or way to parameterize tasks? A way to define tasks as config stubs which can take a list of predefined mixin configs?

Would need consideration but this is an important problem to solve for future maintainability and configuration.
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug: `until` as a single `str`
Root cause: Validation Gap

In `lm_eval/models/huggingface.py` https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/huggingface.py#L1164-L1166
```
if "until" in kwargs.keys():
    until = kwargs.pop("until")
    if isinstance(until, str):
        until = [kwargs]
```

Is it supposed to be `until = [until]`? Otherwise, `until` has type `List[Dict]` as opposed to `List[str]`.

It is supposed to be `[until]`, thank you for the catch! Would you like to open a PR for this?

> It is supposed to be `[until]`, thank you for the catch! Would you like to open a PR for this?

Yes, I would like to make a PR. Seems like I don't have the permission?

> ERROR: Permission to EleutherAI/lm-evaluation-harness.git denied to tianyi-chen.
fatal: Could not read from remote repository.
> 
> Please make sure you have the correct access rights
and the repository exists.

@haileyschoelkopf any idea?

you'll have to fork and push to (a branch of) your fork-- then opening up this repo on github should give you the option to open a PR!

Ahh, looks like it was fixed in #1608 ?
Repository: EleutherAI/lm-evaluation-harness
Issue: OpenAI Completions unexpected kwarg "until"
Root cause: Interface Contract Mismatch

Using the latest release, 0.4.2 and openai==1.3.9 as in the pytoml, I am consistently reproducing the following error using openai completions and the existing gsm8k task with `simple_evaluate` and via the CLI.

```bash
2024-03-19:22:59:32,962 INFO     [evaluator.py:362] Running generate_until requests
  0%|                                                    | 0/20 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/zeus/miniconda3/envs/cloudspace/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/__main__.py", line 342, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/evaluator.py", line 234, in simple_evaluate
    results = evaluate(
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/evaluator.py", line 373, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/models/openai_completions.py", line 274, in generate_until
    response = oa_completion(
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/models/openai_completions.py", line 74, in oa_completion
    return completion()
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/models/utils.py", line 333, in wrapper
    return func(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lm_eval/models/openai_completions.py", line 72, in completion
    return client.completions.create(**kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/openai/_utils/_utils.py", line 303, in wrapper
    return func(*args, **kwargs)
TypeError: Completions.create() got an unexpected keyword argument 'until'
```

Hello, I'm sorry for this bug! I've created a patch in #1612 .
Repository: EleutherAI/lm-evaluation-harness
Issue: Vllm Broken? TypeError: Can't instantiate abstract class VLLM with abstract method prefix_token_id
Root cause: Interface Contract Mismatch

2024-03-19:23:25:01,218 INFO     [__main__.py:251] Verbosity set to INFO
2024-03-19:23:25:06,576 INFO     [__main__.py:335] Selected Tasks: ['gsm8k']
2024-03-19:23:25:06,577 INFO     [__main__.py:336] Loading selected tasks...
2024-03-19:23:25:06,579 INFO     [evaluator.py:131] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
Traceback (most recent call last):
  File "/root/miniconda3/envs/lm-eval/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/workspace/lm-evaluation-harness/lm_eval/__main__.py", line 342, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/workspace/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/workspace/lm-evaluation-harness/lm_eval/evaluator.py", line 164, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/workspace/lm-evaluation-harness/lm_eval/api/model.py", line 133, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class VLLM with abstract method prefix_token_id
Repository: EleutherAI/lm-evaluation-harness
Issue: Add better test coverage for models
Root cause: Unimplemented Feature Gap

We should improve our test coverage of the various model classes supported. 

Ideally we'd have at least https://github.com/EleutherAI/lm-evaluation-harness/blob/main/tests/models/test_huggingface.py tests for every model class. This would require, for API models, mocking out the API call with a dummy one, and probably something similar for VLLM, which currently requires CUDA to be run so we don't have the tests run on the CI.

Help or contributions on this welcome! Else, I'll look into this hopefully soonish.

Hi Hailey, I am interested in contributing to this great project and helping. Let me take spend some time on the docs and library to learn how it is setup and how the tests are designed, so I can help improve the test coverage. I will ask you more questions as I get going and run into any knowledge blocks :)

Hi @haileyschoelkopf , actually would you or one of your colleagues point me to a video or design doc that explains how the code base is setup and how to navigate it? from my quick look, it is complex even for an experienced coder like me, so a little kickstart wtih an existing developer would make it easier. I am happy to do a video chat also if that's easier.

Hey @raamana , I don't have the bandwidth to do a video chat unfortunately, but we'll be putting out a preprint soon that has in part a description of the high-level moving pieces / design!

If you try to understand the codebase but hit some snags, feel free to ping on discord in the #lm-thunderdome channel of EleutherAI's discord (link should be in readme) and I or someone else will probably be able to respond! It would be helpful to know what components are most confusing or should be better documented somewhere.
Repository: EleutherAI/lm-evaluation-harness
Issue: Allow registering custom LM implementations without requiring loading and modifying lm_eval code
Root cause: Unimplemented Feature Gap

Before commit 5627e81, it was possible to register the custom LMs with using --include_path. The code picked up by this argument was successfully registering custom LM implementations and allowed running tasks with custom LMs. 

Removing include_path, initialize_tasks from \_\_main\_\_.py resulted in inability to add a custom LM implementation during initialization. Since 0.4.2 there is no way to register a custom LM without loading and modifying the whole lm_eval code which is quite inconvenient.

Hi! I apologize, I wasn't aware that people were already using `--include_path` for import LM classes, as this wasn't originally intended behavior. Should be able to add it back!

Hi. I'm also using lmeval to evaluate my own model implementation. I would like to ask if there is any other more convenient method besides waiting for this issue to be fixed? @apetrov-msk @haileyschoelkopf 

Here is a workaround. Use a custom lm_eval script, e.g.: 
#!/opt/conda/envs/Python-3.10/bin/python
#-*- coding: utf-8 -*-
import re
import sys
from lm_eval.__main__ import cli_evaluate

sys.path.insert(0, "\<the path to your custom LM code\>")
import \<put your custom LM package here\>

#initialize()

if \_\_name__ == '\_\_main\_\_':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cli_evaluate())
Repository: EleutherAI/lm-evaluation-harness
Issue: Add alternate (configurable) launcher / orchestration + sweep functionality
Root cause: Unimplemented Feature Gap

People often need to build their own launching scripts to run many evals / get full access to certain args / loop through models or tasks.

We should investigate adding an alternate entrypoint to the current __main__.py which can provide more control and remove the need for this.


Things which should be supported include:
- all CLI functionality
- control over all args to simple_evaluate() or evaluate()
- The use of external tasks or LM classes
- running over multiple settings: e.g. many different fewshot values, multiple different models, …

perhaps useful to model off of : Torchtune allows for import of arbitrary classes and orchestration of different “recipes”. https://github.com/pytorch/torchtune/blob/main/recipes/configs/alpaca_generate.yaml though we’d want a single recipe that can be used generally