Repository: EleutherAI/lm-evaluation-harness
Issue: `RuntimeError` when running eval on Winogrande train set
Root cause: Algorithmic Error

When running lm_eval on the training set of Winogrande, I run into the following error:

```bash
2025-05-14:02:00:06 INFO     [evaluator_utils:210] Request: Instance(request_type='loglikelihood', doc={'sentence': "Ian volunteered to eat Dennis's menudo after already having a bowl because _ despised eating intestine.", 'option1': 'Ian', 'option2': 'Dennis', 'answer': '2'}, arguments=("Ian volunteered to eat Dennis's menudo after already having a bowl because Dennis", ' despised eating intestine.'), idx=1, metadata=('winogrande_train', 0, 1), resps=[], filtered_resps={}, task_name='winogrande_train', doc_id=0, repeats=1)
2025-05-14:02:00:06 INFO     [evaluator:559] Running loglikelihood requests
Running loglikelihood requests:   1%|          | 513/80796 [00:04<09:31, 140.53it/s]Passed argument batch_size = auto:1. Detecting largest batch size
Determined largest batch size: 512
Traceback (most recent call last):
  File "/mnt/nfs1/ameya/hubble-lm-eval-env/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/mnt/nfs1/ameya/hubble-lm-evaluation-harness/lm_eval/__main__.py", line 449, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/mnt/nfs1/ameya/hubble-lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/nfs1/ameya/hubble-lm-evaluation-harness/lm_eval/evaluator.py", line 338, in simple_evaluate
    results = evaluate(
  File "/mnt/nfs1/ameya/hubble-lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/nfs1/ameya/hubble-lm-evaluation-harness/lm_eval/evaluator.py", line 570, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/mnt/nfs1/ameya/hubble-lm-evaluation-harness/lm_eval/api/model.py", line 382, in loglikelihood
    return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
  File "/mnt/nfs1/ameya/hubble-lm-evaluation-harness/lm_eval/models/huggingface.py", line 1254, in _loglikelihood_tokens
    logits = torch.gather(logits, 2, cont_toks.unsqueeze(-1)).squeeze(
RuntimeError: Size does not match at dimension 1 expected index [1, 8, 1] to be no larger than self [1, 1, 128256] apart from dimension 2
Running loglikelihood requests:   1%|          | 839/80796 [00:05<08:25, 158.32it/s]
```

This issue doesn't occur when evaluating on the validation set

Task config
---

Modify `lm_eval/tasks/winogrande/default.yaml` and add
```python
test_split: train
```

Library config
---

- Using `main` branch at commit `1c03af331dc5127c618b347439b0e02890102a3c`
- Installed in a fresh venv with `pip install -e .[wandb]`

I narrowed down the issue to this pair of examples in the train_xl split of Winogrande

```
{"qID": "3Z56AA6EK6EDSJVUUMQ1H3DPRQ7M6K-2", "sentence": "Kathy's dog was having a seizure, so she put a pillow between his head and the floor, and the pillow helped his head not hit the _ .", "option1": "pillow", "option2": "floor", "answer": "2"}
{"qID": "3Z56AA6EK6EDSJVUUMQ1H3DPRQ7M6K-1", "sentence": "Kathy's dog was having a seizure, so she put a pillow between his head and the floor, and the _ helped his head not hit the floor.", "option1": "pillow", "option2": "floor", "answer": "1"}
```

The issue seems to be the collator that groups these two requests but truncates the logits based on the shorter "continuation".


Edit: Fixed the actual erroneous example