Repository: EleutherAI/lm-evaluation-harness
Issue: Missing keys in metrics
Root cause: Algorithmic Error

I am getting an error when running evals on `main` that I don't get on `release`.

I am using the following command:

```bash
python -m lm_eval \
  --model hf \
  --model_args pretrained=mistralai/Mistral-7B-v0.1,,dtype="bfloat16" \
  --tasks truthfulqa \
  --device cuda:0 \
  --batch_size 1 \
  --limit 10
```

and I get this:

```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/lm_eval/__main__.py", line 298, in <module>
    cli_evaluate()
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/lm_eval/__main__.py", line 241, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/lm_eval/evaluator.py", line 174, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/lm_eval/evaluator.py", line 519, in evaluate
    metrics = [
              ^
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/lm_eval/evaluator.py", line 520, in <listcomp>
    results[task][metric] for task in task_list
    ~~~~~~~~~~~~~^^^^^^^^
KeyError: 'acc,none'
```
Some PDB magic:
```
  File "/home/ubuntu/miniforge3/envs/pt/lib/python3.10/site-packages/lm_eval/evaluator.py", line 520, in <listcomp>
    results[task][metric] for task in task_list
KeyError: 'acc,none'
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/ubuntu/miniforge3/envs/pt/lib/python3.10/site-packages/lm_eval/evaluator.py(520)<listcomp>()
-> results[task][metric] for task in task_list
(Pdb) task
'truthfulqa_gen'
(Pdb) metric
'acc,none'
(Pdb) 
```

I don't get an error on release. (`pip install lm_eval`)

The issue is coming from  `truthfulqa`, specifically from `truthfulqa_gen`

The issue is coming from this commit: https://github.com/EleutherAI/lm-evaluation-harness/pull/1385

There is also a comment that probably is the issue [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/bfbd0325d3beea7482b866919bc6d1ab05dd7578/lm_eval/evaluator.py#L531)

I can try to fix it with some directions

Thank you for opening this, and I'm sorry for the bug!

To fix this, instead of `for metric in ....` we'd want to iterate over `metric_list` where `metrics = <union of all keys in results[task].keys() for task in task_list, except if ending with "_stderr" or is "alias" or "samples"`

Additionally, https://github.com/EleutherAI/lm-evaluation-harness/blob/bfbd0325d3beea7482b866919bc6d1ab05dd7578/lm_eval/evaluator.py#L535-L539 these lines would need to be made s.t. they don't use any metrics, sample counts, or stderrs from tasks in `task_list` which don't report that metric key.