Repository: EleutherAI/lm-evaluation-harness
Issue: Error on kmmlu evaluation set with vLLM: KeyError: 'exact_match,none'
Root cause: Validation Gap

Hello, Thank you for this amazing repo. 

I trying out this new benchmark K-MMLU, and I got this error at the end of the evaluation.

My command is: 
```
$ lm_eval --model vllm \
--model_args pretrained=beomi/Yi-Ko-6B,dtype=auto,gpu_memory_utilization=0.9 \
 --tasks kmmlu \
--batch_size auto
```


and I got this error message at the end of the evaluation. 


```Processed prompts: 100%|__________________________________________________________________________________________| 4104/4104 [01:52<00:00, 36.63it/s]
100%|___________________________________________________________________________________________________________| 43238/43238 [10:37<00:00, 67.82it/s]
2024-02-23:15:47:12,103 INFO     [evaluator.py:369] Running loglikelihood requests
Processed prompts: 100%|_______________________________________________________________________________________| 16416/16416 [01:05<00:00, 249.75it/s]
100%|__________________________________________________________________________________________________________| 16416/16416 [01:06<00:00, 246.17it/s]
Traceback (most recent call last):
  File "/opt/conda/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())                                                                                                                            File "/root/lm-evaluation-harness/lm_eval/__main__.py", line 288, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/root/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper                                                                              return fn(*args, **kwargs)
  File "/root/lm-evaluation-harness/lm_eval/evaluator.py", line 190, in simple_evaluate
    results = evaluate(
  File "/root/lm-evaluation-harness/lm_eval/utils.py", line 288, in _wrapper
    return fn(*args, **kwargs)
  File "/root/lm-evaluation-harness/lm_eval/evaluator.py", line 535, in evaluate
    metrics = [                                                                                                                                         File "/root/lm-evaluation-harness/lm_eval/evaluator.py", line 536, in <listcomp>                                                                        results[task][metric] for task in task_list
KeyError: 'exact_match,none'
```

Please help me with this

In addition, when evaluating the subset of it, It works well.

```
lm_eval --model vllm     --model_args pretrained=beomi/Yi-Ko-6B,tensor_parallel_size=1,dtype=auto,gpu_memor
y_utilization=0.9,data_parallel_size=1     --tasks kmmlu_direct_machine_design_and_manufacturing     --batch_size auto                                2024-02-23:16:37:52,784 INFO     [__main__.py:209] Verbosity set to INFO
2024-02-23:16:37:52,785 INFO     [__init__.py:358] lm_eval.tasks.initialize_tasks() is deprecated and no longer necessary. It will be removed in v0.4.2 release. TaskManager will instead be used.
2024-02-23:16:37:56,687 INFO     [__main__.py:285] Selected Tasks: ['kmmlu_direct_machine_design_and_manufacturing']                                  2024-02-23:16:37:56,687 INFO     [__main__.py:286] Loading selected tasks...
2024-02-23:16:37:56,687 INFO     [evaluator.py:95] Setting random seed to 0                                                                           2024-02-23:16:37:56,687 INFO     [evaluator.py:99] Setting numpy seed to 1234
2024-02-23:16:37:56,687 INFO     [evaluator.py:103] Setting torch manual seed to 1234                                                                 INFO 02-23 16:37:57 llm_engine.py:73] Initializing an LLM engine with config: model='beomi/Yi-Ko-6B', tokenizer='beomi/Yi-Ko-6B', tokenizer_mode=auto,
 revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=None, seed=1234)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.                                 INFO 02-23 16:38:03 llm_engine.py:222] # GPU blocks: 58851, # CPU blocks: 4096
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.                                 2024-02-23:16:38:05,756 INFO     [evaluator.py:150] get_task_dict has been updated to accept an optional argument, `task_manager`Read more here:https:
//github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage                                                      2024-02-23:16:38:12,878 INFO     [task.py:361] Building contexts for kmmlu_direct_machine_design_and_manufacturing on rank 0...
2024-02-23:16:38:13,745 INFO     [evaluator.py:369] Running generate_until requests                                                                   Processed prompts: 100%|__________________________________________________________________________________________| 1000/1000 [00:12<00:00, 77.88it/s]
100%|_____________________________________________________________________________________________________________| 1000/1000 [00:12<00:00, 77.76it/s]vllm (pretrained=beomi/Yi-Ko-6B,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=1), gen_kwargs: (None), limit: None, n
um_fewshot: None, batch_size: auto                                                                                                                    |                    Tasks                    |Version|Filter|n-shot|  Metric   |Value|   |Stderr|
|---------------------------------------------|------:|------|------|-----------|----:|---|-----:|                                                    |kmmlu_direct_machine_design_and_manufacturing|      2|none  |None  |exact_match|    0|_  |     0|
```

Thanks for reporting this! @h-albert-lee did you observe this occurring? if not, or if you don't have the time to check this out, I can investigate myself.

@haileyschoelkopf I don't think I've seen that issue in my implementation, but I've reproduced it on my personal laptop and I get that error. It's probably a difference between the forked version and the current version, or it's an operating system issue. The implementation was worked on and tested on ubuntu, but the reproduction is on Windows, so there might be a difference. 

It also seems unusual that the issue is happening on vLLM, so I'll check that out as well(I haven't tested it in the implementation  to see if it works with vLLM). I hope the information I just shared will help us solve our problem. I'll do a little more investigation.

May be closed by #1441 ?

I think so

Confirmed no longer occurring on most recent code version!

@h-albert-lee Thank you for sharing this benchmark.
If I want to reproduce results table2 in the [paper](https://arxiv.org/pdf/2402.11548.pdf) is it correct to use `kmmlu_direct` instead of `kmmlu` for the task?
It seems if I set task to `kmmlu` , it evaluates on all 4 groups(direct, hard, hard_direct, hard_cot). I think the error @caisarl76 shared occurs because 4 groups have different evaluation metrics(e.g. `exact_match,none`, `acc_norm,none`, ...)

@yoonniverse  If you want to reproduce what is presented in the paper, it is correct to use only kmmlu_direct. Also, as stated in the readme, you can reproduce the exact values by taking the macro average.