Repository: EleutherAI/lm-evaluation-harness
Issue: Error for AGIEval when using fewshot
Root cause: Validation Gap

Hi, I meet the following error when I evaluate on AGIEval with num_fewshot=3. However, everything works normally with 0-shot.

```
2024-09-19:13:03:47,189 DEBUG    [cache.py:33] requests-agieval_jec_qa_kd-3shot-rank0-world_size1-tokenizer is not cached, generating...
2024-09-19:13:03:47,189 INFO     [task.py:423] Building contexts for agieval_jec_qa_kd on rank 0...
 12%|████████████▍                                                                                            | 119/1000 [00:00<00:01, 480.91it/s]
Traceback (most recent call last):
  File "/home/baliao/.conda/envs/cluster/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/baliao/.conda/envs/cluster/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/__main__.py", line 468, in <module>
    cli_evaluate()
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/__main__.py", line 389, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/evaluator.py", line 420, in evaluate
    task.build_all_requests(
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/api/task.py", line 446, in build_all_requests
    fewshot_ctx = self.fewshot_context(
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/api/task.py", line 1088, in fewshot_context
    labeled_examples += self.sampler.get_context(doc, num_fewshot)
  File "/data/chatgpt/data/baliao/cluster/04_lm_eval/lm-evaluation-harness/lm_eval/api/samplers.py", line 89, in get_context
    str(doc_target[0])
IndexError: list index out of range
```

Here is how I run the code:

MODEL=/path/to/Llama-2-7b-hf

```
accelerate launch --num_processes 1 -m lm_eval \
    --model hf \
    --model_args pretrained=$MODEL,trust_remote_code=True \
    --batch_size 16 \
    --verbosity DEBUG \
    --tasks agieval \
    --num_fewshot 3
```

Version: lm_eval 0.4.4


Hi! So the dataset we are using is missing the fewshot split. It uses the test split for the fewshot samples and looks like one of the rows in `agieval_jec_qa_kd` is missing the answer field. We have some logic to handle that when its an evaluation question, but not when its in the fewshot. I'll try looking into it!

> Hi! So the dataset we are using is missing the fewshot split. It uses the test split for the fewshot samples and looks like one of the rows in `agieval_jec_qa_kd` is missing the answer field. We have some logic to handle that when its an evaluation question, but not when its in the fewshot. I'll try looking into it!

Hi Mr Abbasi, thank you so much for the dedication. I've been trying to reproduce the evaluation of llama3-8b-base on agieval English. In the huggingface page, llama3-8b-base claimed it has a 45.9 acc on agieval English cot 3-5 shot, and they mentioned that they were using an internal version of [AGIEval official repo](https://github.com/ruixiangcui/AGIEval) for evaluation. However when I use lm-harness for evaluating llama3-8b-base on agieval_en 5 shot, it only has an average acc of 32.71. 

Since lm-harness and the official repo are using different fewshot examples, and the performance discrepancy is so large between these two, do you think it is necessary to settle the fewshot example of the lm-harness to the same as the official repo?

If you think it is worth looking into, I'm happy to see what I can do to help fix this.

I found that the fewshot prompt is also wrong.
I checked model input with following command.
```
lm_eval --model hf --batch_size 16 --model_args pretrained=meta-llama/Llama-3.2-3B,trust_remote_code=True --write_out --output_path results --log_samples --tasks agieval_en
```
It seems this is the input prompt that is going into the model.
```
Q: What value of $x$ satisfies the equation $3 x+3=27$ ? Answer Choices: (A)3 (B)8 (C)10 (D)27
A: Among A through D, the answer is 1

Q: If $3(c+d)=5$, what is the value of $c+d ?$ Answer Choices: (A)$\frac{3}{5}$ (B)$\frac{5}{3}$ (C)3 (D)5
A: Among A through D, the answer is 1

Q: $$h=-4.9 t^{2}+25 t$$The equation above expresses the approximate height $h$, in meters, of a ball $t$ seconds after it is launched vertically upward from the ground with an initial velocity of 25 meters per second. After approximately how many seconds will the ball hit the ground? Answer Choices: (A)3.5 (B)4.0 (C)4.5 (D)5.0
A: Among A through D, the answer is 3

Q: If $\frac{x-1}{3}=k$ and $k=3$, what is the value of $x ?$ Answer Choices: (A)2 (B)4 (C)9 (D)10
A: Among A through D, the answer is
(end of prompt on previous line)
target string or answer choice index (starting on next line):
[3]
(end of target on previous line)
```
I guess `A: Among A through D, the answer is 1` part should be `A: Among A through D, the answer is (B)8.`