Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluation on Scrolls Tasks Error
Root cause: Validation Gap

I am trying to evaluate Mistral Model on Scrolls Tasks. I have encountered the following error. 

```
LLM-BenchMarking/lm-evaluation-harness$ lm_eval --model hf --model_args pretrained=mistralai/Mistral-7B-v0.1 --tasks scrolls --device cpu --limit 5 --num_fewshot 2
2023-12-05:15:01:50,795 INFO     [utils.py:145] Note: detected 224 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-12-05:15:01:50,795 INFO     [utils.py:148] Note: NumExpr detected 224 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-12-05:15:01:50,795 INFO     [utils.py:160] NumExpr defaulting to 8 threads.
2023-12-05:15:01:52,509 INFO     [__main__.py:132] Verbosity set to INFO
2023-12-05:15:01:55,762 WARNING  [__main__.py:138]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2023-12-05:15:01:55,762 INFO     [__main__.py:205] Selected Tasks: ['scrolls']
2023-12-05:15:01:55,763 WARNING  [evaluator.py:93] generation_kwargs specified through cli, these settings will be used over set parameters in yaml tasks.
2023-12-05:15:01:55,765 INFO     [huggingface.py:120] Using device 'cpu'
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:06<00:00,  3.37s/it]
2023-12-05:15:02:12,560 WARNING  [evaluator.py:143] Overwriting default num_fewshot of scrolls_contractnli from None to 2
2023-12-05:15:02:12,561 WARNING  [evaluator.py:143] Overwriting default num_fewshot of scrolls_govreport from None to 2
2023-12-05:15:02:12,561 WARNING  [evaluator.py:143] Overwriting default num_fewshot of scrolls_narrativeqa from None to 2
2023-12-05:15:02:12,561 WARNING  [evaluator.py:143] Overwriting default num_fewshot of scrolls_qasper from None to 2
2023-12-05:15:02:12,561 WARNING  [evaluator.py:143] Overwriting default num_fewshot of scrolls_qmsum from None to 2
2023-12-05:15:02:12,561 WARNING  [evaluator.py:143] Overwriting default num_fewshot of scrolls_quality from None to 2
2023-12-05:15:02:12,561 WARNING  [evaluator.py:143] Overwriting default num_fewshot of scrolls_summscreenfd from None to 2
2023-12-05:15:02:12,561 INFO     [task.py:355] Building contexts for task on rank 0...
Traceback (most recent call last):
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/__main__.py", line 207, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/evaluator.py", line 152, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/evaluator.py", line 285, in evaluate
    task.build_all_requests(limit=limit, rank=lm.rank, world_size=lm.world_size)
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/api/task.py", line 368, in build_all_requests
    inst = self.construct_requests(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/tasks/scrolls/task.py", line 239, in construct_requests
    request_list = [
                   ^
  File "/home/user1/LLM-BenchMarking/lm-evaluation-harness/lm_eval/tasks/scrolls/task.py", line 247, in <listcomp>
    for i, choice in doc["choices"]
        ^^^^^^^^^
ValueError: too many values to unpack (expected 2)
```

cc @lintangsutawika 

Will take a look