zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: High Number of Tokens for openai-completions Models
Root cause: Documentation Deficiency

I created the config file with hard-coded prompts to try out gpt-3.5-turbo on 5 examples, but the number of tokens is exceedingly high for this config file. What should I do to reduce the number of tokens?
The following is my config file:
```html
task: newyorker_caption_contest_zeroshot
dataset_path: csv
dataset_path: selinax10010/matching 
dataset_name: null 
dataset_kwargs: null 
output_type: multiple_choice
training_split: train 
validation_split: validation 
test_split: test 
process_docs: !function utils.process_docs
description: "role: system\nYou are CaptionContestGPT, an expert language model at understanding \
the famous New Yorker caption contest. You follow the contest each week, and \
understand what makes for a humorous caption for each cartoon. You are aware of\
the various theories of humor, and read/anaylze the caption contest entries and \
winners each week.\n\nSome things to remember:\n\n- You're well versed in the \
history of the New Yorker Caption contest, and the types of captions that are \
selected as finalists/winners vs. those that are not.\n- Provide the answer in \
the requested format. \n~~~\nrole: user\nI will describe a New Yorker cartoon \
to you. Then, I will give you 5 choices (labelled A-E) for captions. One of the \
captions was the winning caption for that cartoon, the other captions do not \
correspond to this cartoon. Your job is to find the correct match and respond \
with \"Answer: X\" where X is either A, B, C, D, or E.\n\n\n"
doc_to_text: "{{query}}\nChoices:\nA: {{choices_text[0]}}\nB: {{choices_text[1]}}\nC: \
{{choices_text[2]}}\nD: {{choices_text[3]}}\nE: {{choices_text[4]}}\n\nWhich of the 5 options \
(A, B, C, D, or E) is the caption that truly corresponds to the cartoon?\n~~~"
doc_to_target: "{{label}}"
doc_to_choice: "{{choices}}"
num_fewshot: 0
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
  - metric: acc_norm
    aggregation: mean
    higher_is_better: true
  - metric: mcc
    aggregation: matthews_corrcoef
    higher_is_better: true
metadata:
  version: 1.0
```
Repository: EleutherAI/lm-evaluation-harness
Issue: Format of Personal Defined Dataset for Evaluation
Root cause: Documentation Deficiency

Hi developers, 

I am wondering, if there's an instruction on how to create a self-defined eval dataset, like what the format should be, which could be executed by the lm_eval_harness ? 

Thanks for your help ~

Hi! in our documentation we provide a description of how to add and define a new task.


For your own dataset, the only precondition is that you turn this into a Hugging Face dataset loadable with the `datasets` library—then in your task definition you can use all the column values.
Repository: EleutherAI/lm-evaluation-harness
Issue: Regarding decontamination
Root cause: Documentation Deficiency

First, congratulations and thank you for the amazing project and the fantastic framework that is being used in almost all LLM evaluations. I appreciate you sharing it with us.


Regarding decontamination:

1. I wonder if data decontamination is still effective.
2. I am curious if there are plans or ongoing efforts to conduct strict data decontamination based on the current evaluation dataset.

Ultimately, if we strongly remove data contamination using an 8-gram(harder than 13-gram) approach through current data decontamination methods, can we ensure that the results on the current evaluation bench are free from contamination?

Many papers claim that they did not cheat or contaminate the data in their own ways, while some omit this process. Moreover, I believe this could ultimately raise issues regarding the reliability of the evaluation dataset. Although benchmark datasets can provide standardized quantitative metrics, the moment data contamination occurs, the benchmark's credibility will be significantly undermined. This might be fundamentally different from some forms of cheating in MMLU implementations. Additionally, methods that measure contamination based on the inference results of the model post hoc could also be misleading. Ultimately, a benchmark is needed for what the model has not been taught but can infer.

Is there any information that might be helpful for my brief perspective? If I use the current harness implementation for data decontamination, can I say that it sufficiently removes data contamination?
Repository: EleutherAI/lm-evaluation-harness
Issue: Results is weird for Qwen2-1.5B
Root cause: Environment Incompatibility

I try to test the results of some benchmarks for Qwen2-1.5B. But the score is too low:
```
| Tasks  |Version|Filter|n-shot| Metric |Value |   |Stderr|
|--------|------:|------|-----:|--------|-----:|---|-----:|
|arc_easy|      1|none  |     0|acc     |0.2647|±  |0.0091|
|        |       |none  |     0|acc_norm|0.2597|±  |0.0090|

|  Tasks   |Version|Filter|n-shot|Metric|Value |   |Stderr|
|----------|------:|------|-----:|------|-----:|---|-----:|
|winogrande|      1|none  |     0|acc   |0.5107|±  | 0.014|
```

Hi! Not sure when I might have time to dig deeper on this specifically, but it might be the case that Qwen uses MMLU-style prompting for ARC , a YAML config for which can be found in the Appendices of our recent paper https://arxiv.org/abs/2405.14782 . 

> Hi! Not sure when I might have time to dig deeper on this specifically, but it might be the case that Qwen uses MMLU-style prompting for ARC , a YAML config for which can be found in the Appendices of our recent paper https://arxiv.org/abs/2405.14782 .

Thanks for your reply. But the result of Qwen1.5-1B8 looks good and its acc_norm is 0.5901 for arc_easy. I am just wondering why Qwen2's result looks like a random guess, maybe it's about the prompt issue (special tokens etc..).

Ah, I see, thanks for reporting. 

We have seen some unique quirks / edge cases pop up for previous Qwen models--e.g. the tokenizer for Qwen1 does not quite follow the typical HF tokenizers interface. Would definitely recommend checking tokenization if you have the bandwidth to investigate this. 

How is the measured MMLU performance of this model? Do those match their reported results, or does MMLU also seem lower than anticipated when you eval it in the Harness?



> Ah, I see, thanks for reporting.
> 
> We have seen some unique quirks / edge cases pop up for previous Qwen models--e.g. the tokenizer for Qwen1 does not quite follow the typical HF tokenizers interface. Would definitely recommend checking tokenization if you have the bandwidth to investigate this.
> 
> How is the measured MMLU performance of this model? Do those match their reported results, or does MMLU also seem lower than anticipated when you eval it in the Harness?

The score of MMLU is also quite low.
|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.2295|±  |0.0035|
| - humanities     |N/A    |none  |     5|acc   |0.2421|±  |0.0062|
| - other          |N/A    |none  |     5|acc   |0.2398|±  |0.0076|
| - social_sciences|N/A    |none  |     5|acc   |0.2171|±  |0.0074|
| - stem           |N/A    |none  |     5|acc   |0.2125|±  |0.0073|

The reason for this issue is Qwen2-1.5B need inference with BF16 or FP32. Fp16 will cause this problem.

Ah, great catch!
Repository: EleutherAI/lm-evaluation-harness
Issue: The output of ceval is not as the same format at the official version? 
Root cause: Documentation Deficiency

As https://github.com/hkust-nlp/ceval/blob/main/README_zh.md shows:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/90491543/83b8c855-bfe3-4b55-ab3f-a171d11978b7)

but the result of ours is:
![image2024-6-11_11-15-20](https://github.com/EleutherAI/lm-evaluation-harness/assets/90491543/1779e867-7390-4b88-acdf-26eaed56173e)


The script is:
```
TASKS="ceval-valid*"
BATCH_SIZE=1
NUM_SHOTS=5

MODEL="" # model path
API=vllm
CUDA_VISIBLE_DEVICES=7 lm_eval \
    --model ${API} \
    --model_args pretrained=${MODEL},dtype="half", gpu_memory_utilization=0.9 \
    --tasks ${TASKS} \
    --batch_size ${BATCH_SIZE} \
    --num_fewshot ${NUM_SHOTS} \
    --trust_remote_code \
    --output_path ./result
```

How can I get the same ouput as the official version? 

Thanks!

Since GPT-4 and ChatGPT API replies may change every day, week or month, there may be no way to get the same output as reported 8 months ago (according to the last change date of the `README_zh.md` link you provided).

You can also try to find exact version of lm-eval code they used (probably some 0.3.0 like with updates) and check reproducibility.
Repository: EleutherAI/lm-evaluation-harness
Issue: Add MMLU-Pro Dataset
Root cause: Unimplemented Feature Gap

https://arxiv.org/abs/2406.01574

https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro

MMLU-Pro is an expanded version of MMLU which also includes a greater number of distractor answer choices.

Implementations should strive to match any "canonical" prompting setups introduced in the MMLU-Pro paper; it would be ideal to implement both a CoT generative (`generate_until`) variant and a loglikelihood-based (as with `mmlu` task grouping in our repo) variant.

Contributions would be welcome!
Repository: EleutherAI/lm-evaluation-harness
Issue: Check compatibility of `local-completions` with VLLM (returns logits) for `multiple_choice` tasks
Root cause: Interface Contract Mismatch

As raised in #1779 . We likely need separate behavior for OAI completions apis and similar ones that don't support `echo=True` and/or logprobs, and ones like VLLM that do support logprobs along with echo.
Repository: EleutherAI/lm-evaluation-harness
Issue: Plans for a new release?

Hi all - curious if there is another release planned soon? Looking at the [PyPI package](https://pypi.org/project/lm-eval/) it's been a few months since the last release, and I see from the [GitHub Release](https://github.com/EleutherAI/lm-evaluation-harness/releases/tag/v0.4.2) there's been about 92 commits since then.

Apologies if I missed any communications about this elsewhere - appreciate any info you can provide!

Hi! We'll be putting out a new release very soon! We were waiting on #1873 to merge and settle a little bit before doing so. 

Great news - thank you @haileyschoelkopf!

Please add in the new release also these fixes and improvements:
#1956
#1930
#1919
#1872
#1868
#1800
#1796
#1731
#1571

Some have been merged. But there's no need to rush them as we will be releasing with a shorter and more predictable period moving forward. 

v0.4.3 is live!
Repository: EleutherAI/lm-evaluation-harness
Issue: .ipynb_checkpoints causes eval harness to fail
Root cause: Validation Gap

Hi, just wanted to share with anyone running this on a jupyter notebook for some reason like me, that the eval-harness fails for tasks that have a custom python utils file if autosave for the project is turned on for the jupyter notebook / jupyterlab instance. This causes `.ipynb_checkpoint` directories to be created which I think the harness searches recursively and that causes an error.

you can check for instances of this and remove .ipynb_checkpoints directories from the task directory recursively, you can run
```find . -type d -name ".ipynb_checkpoints" -exec rm -r {} +``` in the lm_eval/tasks/ directory.

you should run the first find command before rm'ing anything as a sanity check.

edit: i had time to make a pr.
Repository: EleutherAI/lm-evaluation-harness
Issue: Cannot load model 'local-chat-completions' and 'local-completions' 
Root cause: Unimplemented Feature Gap

I want to use "lm-evaluation-harness" to test local model with OpenAI mirrored API. After reading the introduction, I choose to use 'local-chat-completions' model, but it reports no such model.

`!pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git@big-refactor`
`from lm_eval import api`
`!lm_eval \`
 `   --model local-chat-completions \`
`    --model_args model=xxx,base_url=http://xxx`
`    --include_path ./ \`
`    --tasks demo_boolq \`
`    --device mps\`
`    --limit 10`

ValueError: Attempted to load model 'local-chat-completions', but no model for this name found! Supported model names: hf-auto, hf, huggingface, gooseai, openai-chat-completions, textsynth, dummy, anthropic, gguf, ggml, vllm
Repository: EleutherAI/lm-evaluation-harness
Issue: Making torch dep optional?
Root cause: Architectural Constraint

Hi,

For those of us JAX cultists, any chance we could make torch an optional dependency? I think the only place the main framework needs torch is for the distributed evaluation stuff in evaluator.py, which can be import-guarded with a few lines' fix.

(I'd be happy to PR this if you all are interested)

Happy to review your PR. Also it would be great if there is some sort of example on how you using in jax for beginners, just an idea. 

Hey @dlwh ! Echoing this--happy to see a PR making this change, and we can discuss there whether the changes are self-contained enough to merge generally. Should be a good idea to support JAX.

@lintangsutawika Sounds like a good idea. I'm gonna PR the core change first just as a POC and once I have things working on my end I'll try to make a tutorial
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the APPS evaluation
Root cause: Unimplemented Feature Gap

> While programming is one of the most broadly applicable skills in modern society, modern machine learning models still cannot code solutions to basic problems. Despite its importance, there has been surprisingly little work on evaluating code generation, and it can be difficult to accurately assess code generation performance rigorously. To meet this challenge, we introduce APPS, a benchmark for code generation. Unlike prior work in more restricted settings, our benchmark measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code. Similar to how companies assess candidate software developers, we then evaluate models by checking their generated code on test cases. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. We fine-tune large language models on both GitHub and our training set, and we find that the prevalence of syntax errors is decreasing exponentially as models improve. Recent models such as GPT-Neo can pass approximately 20% of the test cases of introductory problems, so we find that machine learning models are now beginning to learn how to code. As the social significance of automatic code generation increases over the coming years, our benchmark can provide an important measure for tracking advancements.

https://github.com/hendrycks/apps

evaluation part is [done](https://github.com/EleutherAI/lm-evaluation-harness/pull/246), currently working on adding a sandbox.

Closing as no longer being done.
Repository: EleutherAI/lm-evaluation-harness
Issue: Multi-gpu evaluation with external library usage.
Root cause: Documentation Deficiency

Hi, great library!

I'm following this [doc](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage) to import lm_eval and evaluate my model. I noticed that it is possible to perform multi-gpu evaluation with the command-line interface, but I'm not sure if I can do this in my python script or how to do it(copy my model across all available gpus and evaluate with dataparallel).

Thanks for the help!

Hi! See #1935 -- you can do this with naive model parallel right now using `parallelize=True` in your HF model, or use vllm for data parallel, while still running in a single python script.
Repository: EleutherAI/lm-evaluation-harness
Issue: Error when chat template is not a string
Root cause: Validation Gap

E.g. Mixtral has more than one template https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1/blob/main/tokenizer_config.json
```
Traceback (most recent call last):
  File "/venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/lm-evaluation-harness/lm_eval/__main__.py", line 369, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/lm-evaluation-harness/lm_eval/utils.py", line 346, in _wrapper
    return fn(*args, **kwargs)
  File "/lm-evaluation-harness/lm_eval/evaluator.py", line 269, in simple_evaluate
    evaluation_tracker.general_config_tracker.log_experiment_args(
  File "/lm-evaluation-harness/lm_eval/loggers/evaluation_tracker.py", line 89, in log_experiment_args
    self.chat_template_sha = hash_string(chat_template) if chat_template else None
  File "/lm-evaluation-harness/lm_eval/utils.py", line 36, in hash_string
    return hashlib.sha256(string.encode("utf-8")).hexdigest()
AttributeError: 'dict' object has no attribute 'encode'
```
I guess just add `str()`. However, will it work later?

We should allow one to specify which chat template to use when there are multiple--potentially allowing for `--apply_chat_template {template_name}` as an arg format in addition to the current default?
Repository: EleutherAI/lm-evaluation-harness
Issue: How to use a vllm hosted model?
Root cause: Documentation Deficiency

Are there docs on best practices for using vllm hosted models?

I create a model using 
python -m vllm.entrypoints.openai.api_server --model model_path

and try running it as 
lm_eval --model local-chat-completions     --model_args model=model_path,base_url=http://localhost:8000/v1    --tasks /home/darshshah/lm_eval/tasks/financebench_inference_binary     --batch_size 12 --output_path /home/darshshah/lm_eval/tasks/financebench_inference_binary/outputs --log_samples

But get the following errors
openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable


OPENAI_API_KEY needs to be set to "EMPTY" for locally hosted models, so I usually launch lm_eval this way for my own, non OpenAI models:

```bash
% OPENAI_API_KEY=EMPTY lm_eval --model local-completions [..etc..]
```
Repository: EleutherAI/lm-evaluation-harness
Issue: TemplateLM#_encode_pair() only works for HF transformers auto-models
Root cause: Validation Gap

I invoked `lm_eval.evaluator.evaluate(harness)` with a harness which subclasses `TemplateLM`, on 0.4.2.

<img width="863" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/6141784/d7ce1072-130c-449c-a82a-67acd75a6939">


okay the `else` branch in latest commit looks like it'll fix it fine.  
https://github.com/EleutherAI/lm-evaluation-harness/blob/568af943e315100af3f00937bfd6947844769ab8/lm_eval/api/model.py#L352

I thought I was running latest commit already, but I was [misled by StackOverflow](https://stackoverflow.com/a/37398622/5257399).
Repository: EleutherAI/lm-evaluation-harness
Issue: OpenAI completions model not using OpenAI Completion API properly to extract LogProbs
Root cause: Algorithmic Error

The get_results method in the lm_eval/models/openai_completions.py is not properly using the OpenAI Completion API.

Below is the API definition for LogProbs (from [openai](https://github.com/openai/openai-python/blob/main/src/openai/types/completion_choice.py#L11)):

```python
class Logprobs(BaseModel):
    text_offset: Optional[List[int]] = None

    token_logprobs: Optional[List[float]] = None

    tokens: Optional[List[str]] = None

    top_logprobs: Optional[List[Dict[str, float]]] = None
```

However, get_results in that module is defined this way:

```python
def get_result(response) -> Tuple[float, bool]:
    is_greedy = True
    logprobs = response.logprobs.token_logprobs
    continuation_logprobs = sum(logprobs)

    for i in range(len(response.logprobs.token_logprobs)):
        token = response.logprobs.token_logprobs[i]
        top_tokens = response.logprobs.top_logprobs[i]
        top_token = max(top_tokens.keys(), key=lambda x: top_tokens[x])
        if top_token != token:
            is_greedy = False
            break

    return continuation_logprobs, is_greedy
```

It appears the method assumes response.logprobs.token_logprobs is both a list of log probabilities (it sets the **continuation_logprobs** variable to the sum of its values) _and_ a token since it sets values in the list to a variable named **token**.

Per the OpenAI type hints, it should be a list of floats and the corresponding token can be determined by extracting it from response.logprobs.tokens (a list of strings) at the same index as each item in response.logprobs.token_logprobs.

Is this solved at https://github.com/EleutherAI/lm-evaluation-harness/pull/1919 or is it some other issue?

> Is this solved at #1919 or is it some other issue?

It is a different issue.
Repository: EleutherAI/lm-evaluation-harness
Issue: Include support for evaluation of the GPT-Neo Models
Root cause: Documentation Deficiency

The evaluation harness can be run with GPT-2 and GPT-3 models. Would be useful to include support for the gpt-neo models to reproduce the result in https://huggingface.co/EleutherAI/gpt-neo-2.7B.

Additionally, it will be useful to implement a model.parallelize() method similar to the one implemented for GPT2 (in Huggingface).

> Additionally, it will be useful to implement a model.parallelize() method similar to the one implemented for GPT2 (in Huggingface).

Since the gpt2 model class in lm-eval-harness are just a wrapper around hf models, you should just be able to call model.parallelize() on your base model.

Either way, this is just a library for evaluation, so I think parallelization strategies are outside the scope here. (apart from maybe data parallel)

wrt to the inclusion of gpt-neo - since the HF api should be model agnostic, you should just be able to swap in GPTNeo HF model for gpt2.

adding `pretrained='EleutherAI/gpt-neo-2.7B'` [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/models/gpt2.py#L16) *should* work.

I meant that adding a model.parallelize() method to the gpt-neo implementation in huggingface will make the gpt-neo model more usable, but you are right, this is not the relevant repo for such a request - sorry :)

I am closing this. Thanks for updating the documentation.
Repository: EleutherAI/lm-evaluation-harness
Issue: incomplete task list
Root cause: Unimplemented Feature Gap

available tasks do not include hendrycks_math, running `lm-eval --tasks list ` or specifying `--tasks hendrycks_math` does not work. version is up-to-date 0.4.2.

There is no such task in 0.4.2, but there is one in the `main` branch. It was added with https://github.com/EleutherAI/lm-evaluation-harness/pull/1793 after the 0.4.2 release.

@LSinev is correct!