zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: The input format for XNLI seems wired?
Root cause: Documentation Deficiency

I try to test the XNLI results using the latest commit. And I find the inputs have a prefix_token which looks like:
```
[(('', ' 他说，妈妈，我回来了。, 正确? 是的, 校车把他放下后，他立即给他妈妈打了电话。'), [1], [55363, 2296, 55361, 2049, 55361, 55407, 7523, 55364, 55440, 55363, 2128, 56818, 5927, 55362, 55440, 52081, 55587, 5492, 6731, 55446, 55361, 55425, 1935, 3644, 2049, 7810, 878, 55364])]
```
This is the content of `requests` for `_loglikelihood_tokens`. The `context` is '' and `context_enc` is [1]. It is appended in this function from `lm_eval/api/model.py`:
```
    def loglikelihood(
        self, requests, disable_tqdm: bool = False
    ) -> List[Tuple[float, bool]]:
        new_reqs = []
        for context, continuation in [req.args for req in requests]:
            if context == "": 
                # BOS or EOS as context
                context_enc, continuation_enc = ( 
                    [self.prefix_token_id],
                    self.tok_encode(continuation),
                )
            else:
                context_enc, continuation_enc = self._encode_pair(context, continuation)
         
            new_reqs.append(((context, continuation), context_enc, continuation_enc))
         
        return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
```
As XNLI's config (xnli_zh.yaml) is this:
```
dataset_name: zh
doc_to_choice: '{{[premise+", 正确? 是的, "+hypothesis,premise+", 正确? 所以, "+hypothesis,premise+",                                                                                                                                                                                              
  正确? 不是的, "+hypothesis]}}'
doc_to_text: ''
include: xnli_common_yaml
task: xnli_zh
```
It merges both premise and hypothesis and has no `context`. So, the code will add a `prefix_token_id` to the input for any model. 
This input format seems wired for base model as most LLM do not use `bos` or `eos` in the start of inputs (maybe except Gemma).

cc @lintangsutawika 

@SefaZeng this is intentional. It's inspired by how XNLI was evaluated in the XGLM paper.
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/5774558/05633acc-e378-4e9e-8944-22128b19ab0d)
The `doc_to_choice` has 3 options which a decoder model is simply required to choose which is the most likely.
Repository: EleutherAI/lm-evaluation-harness
Issue: Getting error on lm-evaluation for merged models deployed on HF
Root cause: Validation Gap

Hi there,

Getting below error message while trying to evaluate fine-tuned and merged model. Any thoughts or solutions?

ValueError( ValueError: Trying to set a tensor of shape torch.Size([44312, 4096]) in "weight" (which has shape torch.Size([48353, 4096])), this look incorrect.

Hi, 

could you share more details about what you're trying to run? It looks like an embedding layer shape mismatch --potentially see #1707 for more info

Should be fixed by #1828 

Removing unnecessary padded tokens fixed the issue. Thanks!
Repository: EleutherAI/lm-evaluation-harness
Issue: Avoid slow testing due to network issues.
Root cause: Documentation Deficiency

This is just a reminder that in some cases where the network is restricted, **please remember to set the environment variable HF_DATASETS_OFFLINE to 1 to enable full offline mode**. This will prevent the program from waiting for the datasets' response, which has cost me a lot of time when testing MMLU. Hopefully this is helpful.

This would make a good addition to the documentation / README!

> This would make a good addition to the documentation / README!

I will submit a PR, hoping to be of some help.🤖
Repository: EleutherAI/lm-evaluation-harness
Issue: Add More Tests
Root cause: Unimplemented Feature Gap

We should add more tests to this repo to ensure we can catch breaking changes. 

An inexhaustive list:

- testing that prompts render correctly
- testing few-shot descriptions are applied correctly
- Testing the utilities in `scripts/`
- Better regression testing of commonly used tasks
- testing particular CLI functionality / sub-features or components of evaluate()
- testing the interface provided by LMs remains static
- testing the use of externally-included task configs
- ...
Repository: EleutherAI/lm-evaluation-harness
Issue: eval gsm8k from local dataset folder with the bug info "ValueError: BuilderConfig 'main' not found."
Root cause: Configuration Error

i have the same problems with this issue ( https://github.com/EleutherAI/lm-evaluation-harness/issues/1347 ) 

i just want to eval gsm8k from local dataset folder, as the web in China can't access huggingfaces during using lm-eval. 

i just follow the guide ( https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#beautifying-table-display ) to **Using Local Datasets**

I use the "dataset.save_to_disk()" to save gsm8k dataset into local folder, "llm/dataset/gsm8k". then i set gsm8k.yaml as
"
task: try_gsm8k
dataset_path: /mnt/nfs/vault/jiangp/llm/dataset/gsm8k
dataset_name: main
"
or
"
task: try_gsm8k
dataset_path: gsm8k
dataset_kwargs:
data_dir: /mnt/nfs/vault/jiangp/llm/dataset/gsm8k/
dataset_name: main
"
it doesn't work neither, and show the same bug info " File "/home/jiangp/.conda/envs/llm2/lib/python3.8/site-packages/datasets/builder.py", line 371, in init
self.config, self.config_id = self._create_builder_config(
File "/home/jiangp/.conda/envs/llm2/lib/python3.8/site-packages/datasets/builder.py", line 592, in _create_builder_config
raise ValueError(
ValueError: BuilderConfig 'main' not found. Available: ['default']"

However when i try to set the gsm8k.yaml as
"
task: gsm8k
dataset_path: arrow  # original gsm8k
dataset_kwargs:
  data_files:
    train: /mnt/nfs/vault/jiangp/llm/dataset/gsm8k/main/train/data-00000-of-00001.arrow
    test: /mnt/nfs/vault/jiangp/llm/dataset/gsm8k/main/test/data-00000-of-00001.arrow
dataset_name: main
"
it works, however it's not convenient as i also want to evaluate mmlu benchmark, which contain many tasks, it's not convenient to reset every subtask yaml with "data files in dataset_kwargs".

Want any help if possible

hi, i meet the same problem, have you found a solution to solve this?

same issue when using local dataset c-eval

same issue for gsm8k

> same issue for gsm8k

It seems to be solved by set dataset_name as null, directly using cache path.

dataset_path: your_cache_path (do not need save_to_disk())
dataset_name: null

Same issue here

Same issue
Repository: EleutherAI/lm-evaluation-harness
Issue: Errors when loading exact_match.py 
Root cause: Environment Incompatibility

The lm-eval suddenly raises errors. It used to run successfully. the error like this:
![image](https://github.com/EleutherAI/lm-evaluation-harness/assets/60814118/93d0a70e-5bf7-4e27-9790-1d796103f006)


I fixed the issue by changing export HF_ENDPOINT=https://huggingface.co/, I guess the problem may be from the https://hf-mirror.com/

It seems I can't connect to https://huggingface.co stably.
Is there other solution to the issue?
I installed the lm-eval by "pip install lm-eval" in command line. 
I guess the cause of the issue is  that lm-eval depends on exact_match.py, and at the runtime it can't get that file successfully.
@syx11237744 

Resolved the exact_match.py case in #2045 , although in general, most datasets and models must be either pre-downloaded from HF or a mirror offline, or downloaded online from the HF hub at runtime.
Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation results of llama2 with lm-evaluation-harness using wikitext-2

hi, kindly helper,
I am a newer of lm-eval and I want to test the llama2 model with lm-evaluation-harness using wikitext-2 dataset. The setps are as follows:

1. Download the llama2 model from hugfaceweb site.
2. install the lm-eval as descriped in this github of https://github.com/EleutherAI/lm-evaluation-harness?tab=readme-ov-file#install.
3. I downlaod the wikitext dataset from "https://huggingface.co/datasets/wikitext/tree/main/wikitext-2-raw-v1"
4. Then I condigue the lm_eval parameters and execute the test using the command :
"lm_eval --model hf --model_args pretrained=~/LLM-Models/Llama-2-7b-hf --tasks wikitext --device cuda  --batch_size 1 --output_path ./eval_harness/Llama-2-7b-hf-16b".

Then I get the results as follows:
hf-auto (pretrained=LLM-Models/Llama-2-7b-hf), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
| Tasks  |Version|Filter|n-shot|    Metric     | Value |   |Stderr|
|--------|------:|------|-----:|---------------|------:|---|------|
|wikitext|      2|none  |     0|word_perplexity|15.0550|±  |N/A   |
|        |       |none  |     0|byte_perplexity| 1.6875|±  |N/A   |
|        |       |none  |     0|bits_per_byte  | 0.7549|±  |N/A   |

Are the results nomal? I think the value is too high compared with the test results of "https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#quantization" 
Conld anyone help me?



about the dataset of wikitext, I downlaod from the url "https://huggingface.co/datasets/wikitext/tree/main/wikitext-2-raw-v1". The test results seem like not normal.

I think you should try to follow this issue https://github.com/EleutherAI/lm-evaluation-harness/issues/2301. My test result of llama2-7b in wikitext-2-raw-v1(from Salesforce/wikitext) is 5.4776.
Repository: EleutherAI/lm-evaluation-harness
Issue: sha256 for datasets or samples
Root cause: Unimplemented Feature Gap

Due to security reasons it may be helpful to keep sha256 of the entire dataset (dataset: List[dict] -> sha256(str(dataset))) or each sample (add new attr into each instance that keeps sha256 of the input doc). If you use lm-harness and want other users provide results for your dataset that are valid, one may want to check that the dataset that was passed into the model is exaclty the same dataset that is for leaderboard for example

comparing hashes seems quite easy to realize and may become one more way to defend against cheaters. Yes, they can still fake hashes, but it requires some work to be done with attention to details

or hash may include something else like adding to dataset some random info to get unique hash. This will require a cheater to dive deep into the code to fake hashes correctly as hashes from other models won't work

Anyway, what do you think about this measure to secure the integrity of the dataset?

seems done:)
Repository: EleutherAI/lm-evaluation-harness
Issue: Bug: wrong `until` default value for chat based model
Root cause: Configuration Error

In the `generation_until` mode, the until argument should be the default EOS of the language model itself. However, currently it is set to `fewshot_delimiter`, which further defaults to `\n\n`, if it is not provided. This makes chat LM to very quickly terminate the response in many cases. 

The problem is both present in the add-chat-template branch (below)

https://github.com/EleutherAI/lm-evaluation-harness/blob/495d50bf103d7db160a9244f16a73aa4ece46ca4/lm_eval/api/task.py#L102

and the main branch (below) 

https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L112

Some questions to consider.
What if model benchmarked is not a "chat model"?

There is a condition
`if "until" not in self.generation_kwargs`, so why chat model is scored without needed generation_kwargs provided in the first place?


Indeed, if the model is not a chat model, then it rarely knows to stop, so some `until` parameter is needed, such as the few shot delimiter. What I mean is that the base model and the chat model should have different default stopping criteria, because the chat model knows when to stop. 

I don't quite understand what you mean by "why chat model is scored without needed generation_kwargs provided in the first place?" In my opinion, the default behavior for chat models should be not to have any `until` parameter, because it knows when to stop via its native EOS token, which is also respected by the [generation code](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/huggingface.py#L1218). By the way, it looks like the `if not until:` condition is actually never triggered because we overwrite the `until` parameter in the task initialization. 

Hi guys, have you reached any agreement?

And by the way, if I am using a chat model and I want it to stop at its native `eos`, what should I set `until` to? I tried `null` but I am not quite sure it is working.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the QASPER evaluation
Root cause: Unimplemented Feature Gap

> Readers of academic research papers often read with the goal of answering specific questions. Question Answering systems that can answer those questions can make consumption of the content much more efficient. However, building such tools requires data that reflect the difficulty of the task arising from complex reasoning about claims made in multiple parts of a paper. In contrast, existing information-seeking question answering datasets usually contain questions about generic factoid-type information. We therefore present Qasper, a dataset of 5049 questions over 1585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers, motivating further research in document-grounded, information-seeking QA, which our dataset is designed to facilitate.

https://allenai.org/data/qasper

Hi, can I take this one?
Repository: EleutherAI/lm-evaluation-harness
Issue: How to use Zeno
Root cause: Documentation Deficiency

I would like to ask how to correctly use the script for uploading to Zeno. I use the same example as https://github.com/EleutherAI/lm-evaluation-harness/blob/main/examples/visualize-zeno.ipynb 

```bash
accelerate launch -m lm_eval \
    --model hf \
    --model_args pretrained=google/gemma-1.1-2b-it \
    --tasks imdb \
    --batch_size 32 \
    --device cuda \
    --log_samples \
    --output_path outputs/gemma-1.1-2b-it \
    --limit 10
```

```bash
python scripts/zeno_visualize.py --data_path outputs --project_name "Zeno test"

Traceback (most recent call last):
  File "/raid/data/david/lm-evaluation-harness/scripts/zeno_visualize.py", line 219, in <module>
    main()
  File "/raid/data/david/lm-evaluation-harness/scripts/zeno_visualize.py", line 48, in main
    tasks = set(tasks_for_model(models[0], args.data_path))
  File "/raid/data/david/lm-evaluation-harness/scripts/zeno_visualize.py", line 130, in tasks_for_model
    json.load(open(Path(dir_path, "results.json"), encoding="utf-8"))["configs"],
FileNotFoundError: [Errno 2] No such file or directory: 'outputs/gemma-1.1-2b-it/results.json'
```

The error seems weird to me because this is the directory structure:
```bash
outputs
└── gemma-1.1-2b-it
    └── google__gemma-1.1-2b-it
        ├── results_2024-05-14T17-07-12.477774.json
        └── samples_imdb_2024-05-14T17-07-12.477774.json
```


I created more outputs, renamed some files and folders, and got this error message:
```bash
FileNotFoundError: [Errno 2] No such file or directory: 'output/imdb_gemma_2b_it/google__gemma-1.1-2b-it/pretrained=google__gemma-1.1-2b-it,parallelize=True,load_in_4bit=True_imdb.jsonl'
```
Can anyone please suggest how to use those scripts correctly?


Hi @DavidAdamczyk!
I fixed the filename problem in #1926 - now the script will use the most recent result files for comparison. However, I'm not sure if the rest of the script is correct, there may have been other breaking changes introduced in the meantime. I fixed loading the data, but it still gives me errors later in the `generate_dataset()` function, so I suspect that part was broken before by changing the structure of the samples results files.

In the current version of the library, you no longer need to specify the model name in the output path, just use `--output_path outputs` and it will be created automatically. Then, when using the Zeno script, you just need to point to the `outputs` dir instead of a specific model.
Repository: EleutherAI/lm-evaluation-harness
Issue: how to select the --model parameter for the meta format checkpoints
Root cause: Unimplemented Feature Gap

I directly download the model from the meta website with the following file:
```
./Meta-Llama-3-8B/
|- original/
    |- consolidated.00.pth
    |- tokenizer.model
|- config.json
|- generation_config.json
|- model.safetensors.index.json 
|- special_tokens_map.json
|- tokenizer_config.json
|- tokenizer.json
|- USE_POLICY.md
|- README.md
|- .gitattributes
|- LICENSE
```
I want to run the following command, but it failed with the output:`ValueError: not enough values to unpack (expected 2, got 1)`
I don't know why , It might be I use the wrong `--model` parameter? cause the model is not hugging face format. But what is the correct parameter?
```
lm_eval --model hf \
    --model_args  Meta-Llama-3-8B/meta_model_0.pt\
    --tasks gsm8k_cot \
    --device cuda:0 \
    --batch_size 8
```

`--model_args  pretrained=Meta-Llama-3-8B/meta_model_0.pt \` ? The `--model_args` only accepts key, value pairs :)

> `--model_args  pretrained=Meta-Llama-3-8B/meta_model_0.pt \` ? The --model_args only accepts key, value pairs :)
I am sorry, I mean the `--model` parameter. I am trying to evaluate the meta formate model, I do not know how to select the parameter.



You'll probably have to use the HuggingFace-converted Llama checkpoints, or if you need to use the Meta-formatted ones for a specific reason, then that'll require edits to the code: we don't support this checkpoint format / codebase out of the box unfortunately.

If you've been approved for access on the Meta website, then requesting access to the HF model using a HF account with a matching email should do the trick. Or there are some HF-formatted mirrors I believe: https://huggingface.co/NousResearch/Meta-Llama-3-8B
Repository: EleutherAI/lm-evaluation-harness
Issue: --device cuda:3 not honored when using --model vllm
Root cause: Documentation Deficiency

Hi, I've just found the hard way that the `--device cuda:X` does not do anything when using `--model vllm`, the only way to force it to use an specific GPU is to pass 'CUDA_VISIBLE_DEVICES=1' to the "lm_eval" command. 

Hi! yeah that's the only way with `vllm` unfortunately. Would you like to add a logging message [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/a9eaaf46f1e246e5ce090e37f2f99fe1cfe5a919/lm_eval/models/vllm_causallms.py#L73) to inform other users about this limitation?

> Hi! yeah that's the only way with `vllm` unfortunately. Would you like to add a logging message [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/a9eaaf46f1e246e5ce090e37f2f99fe1cfe5a919/lm_eval/models/vllm_causallms.py#L73) to inform other users about this limitation?

+1, I had to skim through the code and it was only because this issue was raised that I could confidently run without accidentally causing errors (because i had other GPUs that were being used as well.)

for anyone else reading this, `CUDA_VISIBLE_DEVICES=1,2,3` as a comma separated list is how you would specify multiple GPUs

We could manually set `os.environ["CUDA_VISIBLE_DEVICES"]` in the code right?

I'm agnostic as to which solution -- setting the env variable ourselves, or just printing a warning log stating that `device` won't be respected -- open to feedback from users on which they would find more intuitive.
Repository: EleutherAI/lm-evaluation-harness
Issue: Implement the TweetQA evaluation
Root cause: Unimplemented Feature Gap

> With social media becoming increasingly popular on which lots of news and real-time events are reported, developing automated question answering systems is critical to the effectiveness of many applications that rely on real-time knowledge. While previous question answering (QA) datasets have concentrated on formal text like news and Wikipedia, we present the first large-scale dataset for QA over social media data. To make the tweets are meaningful and contain interesting information, we gather tweets used by journalists to write news articles. We then ask human annotators to write questions and answers upon these tweets. Unlike other QA datasets like SQuAD in which the answers are extractive, we allow the answers to be abstractive. The task requires model to read a short tweet and a question and outputs a text phrase (does not need to be in the tweet) as the answer.

https://tweetqa.github.io/
Repository: EleutherAI/lm-evaluation-harness
Issue: `--tasks list` does not work
Root cause: Environment Incompatibility

It just hung for a few seconds and returned nothing.

How to reproduce:
```shell
$ git clone https://github.com/EleutherAI/lm-evaluation-harness
$ cd lm-evaluation-harness/
$ pip install -e .
$ lm-eval --tasks list

```

Thanks for your help!

Just want to check, did you try installing this in a fresh environment?

No, I installed it in an existing environment.

On Sat, May 18, 2024, 01:56 Lintang Sutawika ***@***.***>
wrote:

> Just want to check, did you try installing this in a fresh environment?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/EleutherAI/lm-evaluation-harness/issues/1850#issuecomment-2118207911>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AKIL3E2CJQADIIZKUXS3WOLZCZHEPAVCNFSM6AAAAABH3OSW2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJYGIYDOOJRGE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Installing it on a new environment would probably be worth to try.

The problem was solved by your advice. Thanks!

@lintangsutawika what do you mean about fresh environment? when i install using the command "pip install -e . ", it doesnot generate the executing command lm-eval and lm_eval. whatis wrong

I meant by making a new venv or conda env.

@lintangsutawika I want to measure the precision using the vllm as the backend. so I need an environment containing torch, torchvision, vllm lm-eaval, of course the python is very fundamental. for example python 3.10. so whatis the order of creating an new environemnt