Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError: Evaluating on squad_v2
Root cause: Interface Contract Mismatch

Hi,

Thanks for the great work on evaluation LMs. 

I am trying to evaluate GPT2 on squad_v2 dataset but got the following error. 

```
Traceback (most recent call last):
  File "/nfs/projects/lm-evaluation-harness/main.py", line 108, in <module>
    main()
  File "/nfs/projects/lm-evaluation-harness/main.py", line 79, in main
    results = evaluator.simple_evaluate(
  File "/nfs/projects/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/nfs/projects/lm-evaluation-harness/lm_eval/evaluator.py", line 86, in simple_evaluate
    results = evaluate(
  File "/nfs/projects/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper
    return fn(*args, **kwargs)
  File "/nfs/projects/lm-evaluation-harness/lm_eval/evaluator.py", line 247, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/nfs/projects/lm-evaluation-harness/lm_eval/base.py", line 820, in fn
    rem_res = getattr(self.lm, attr)(remaining_reqs)
  File "/nfs/projects/lm-evaluation-harness/lm_eval/models/huggingface.py", line 319, in greedy_until
    stop_sequences = request_args["stop_sequences"]
TypeError: list indices must be integers or slices, not str
```
I printed out `request_args`, and it looks like this: `['\n']`

I tried copa dataset, and there is no problem. Do you have any idea about this error?

I am using these versions:
transformers==4.27.4
torch==2.0.0
datasets== 2.11.0

Best,
Chiyu


the same when applying Llama to gsm8k.

Same problem with `qasper`. I have a suspicion that all datasets that have metrics more complex than `acc` are failing with this error.

Same with `math_*` but haven't identified which `*` it is 

EDIT:
Crashes with `math_algebra` and more, does not crash with `mathqa`.

I got the same problem. Explored it a little more, and discovered that the huggingface model expects greedy requests to be a tuple of context string and a dict with the fields (`def greedy_until(self, requests: List[Tuple[str, dict]])`). While these datasets construct requests as a tuple of str and just a list of stop sequences, like `rf.greedy_until(ctx, ["\n"])`. But I'm not familiar with the library enough to understand which of these formats is correct.

After I've fixed this with the crutch, the huggingface.py module produce another error: `AttributeError: module 'lm_eval.utils' has no attribute 'select_continuation_from_batch_left_padding'`.
And indeed, there is no such function anywhere in the project source code:
```
$ grep -rn 'select_continuation_from_batch_left_padding' .
./lm_eval/models/huggingface.py:476:        return utils.select_continuation_from_batch_left_padding(
Binary file ./lm_eval/models/__pycache__/huggingface.cpython-310.pyc matches
```

#427 should fix this, tested locally! Note that the model names change in this PR, so now `hf-causal` calls what was formerly known as `--model hf` or `--model gpt2` and `--model hf-causal-experimental` calls the former `hf-causal`.

 Please let me know if there continue to be issues and I'll follow up right away. Sorry for the trouble, all!

Thank you for fixing this!

I tested it with `dolly-v2-12b` on `qasper` datasets (command `python main.py     --model hf-causal-experimental     --model_args pretrained=databricks/dolly-v2-12b     --tasks qasper     --device cuda:4`).

It gives tons of repeated messages like this:

```
Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
 46%|████████████████████████████████████████████████████████████████████████████▋                                                                                        | 165/355 

[07:02<07:25,  2.34s/it]Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
 47%|█████████████████████████████████████████████████████████████████████████████▏                                                                                       | 166/355 

[07:05<08:45,  2.78s/it]Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
 47%|█████████████████████████████████████████████████████████████████████████████▌                                                                                       | 167/355 

[07:06<06:53,  2.20s/it]Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
 47%|██████████████████████████████████████████████████████████████████████████████                                                                                       | 168/355 

[07:07<05:35,  1.79s/it]Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.
 48%|██████████████████████████████████████████████████████████████████████████████▌                                        
```

But I do get the metrics at the end in the table:

```
hf-causal-experimental (pretrained=databricks/dolly-v2-12b), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
| Task |Version|    Metric    |Value |   |Stderr|
|------|------:|--------------|-----:|---|-----:|
|qasper|      0|f1_yesno      |0.5403|±  |0.0416|
|      |       |f1_abstractive|0.1351|±  |0.0064|
```

However, when I run it on `alpaca-lora-7b` model this way `python main.py     --model hf-causal-experimental     --model_args pretrained=chainyo/alpaca-lora-7b     --tasks qasper  --device cuda:4`, I get another error:

```
Traceback (most recent call last):
  File "main.py", line 108, in <module>
    main()
  File "main.py", line 79, in main
    results = evaluator.simple_evaluate(
  File "/home/user/code/lm-evaluation-harness/lm_eval/utils.py", line 182, in _wrapper
    return fn(*args, **kwargs)
  File "/home/user/code/lm-evaluation-harness/lm_eval/evaluator.py", line 86, in simple_evaluate
    results = evaluate(
  File "/home/user/code/lm-evaluation-harness/lm_eval/utils.py", line 182, in _wrapper
    return fn(*args, **kwargs)
  File "/home/user/code/lm-evaluation-harness/lm_eval/evaluator.py", line 247, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/home/user/code/lm-evaluation-harness/lm_eval/base.py", line 820, in fn
    rem_res = getattr(self.lm, attr)(remaining_reqs)
  File "/home/user/code/lm-evaluation-harness/lm_eval/models/huggingface.py", line 395, in greedy_until
    token_context = self.tok_encode_batch(context)
  File "/home/user/code/lm-evaluation-harness/lm_eval/models/huggingface.py", line 354, in tok_encode_batch
    return self.tokenizer(
  File "/home/user/code/lm-evaluation-harness/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2538, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/home/user/code/lm-evaluation-harness/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2624, in _call_one
    return self.batch_encode_plus(
  File "/home/user/code/lm-evaluation-harness/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2806, in batch_encode_plus
    padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
  File "/home/user/code/lm-evaluation-harness/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2443, in _get_padding_truncation_strategies
    raise ValueError(
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
```

Same problem when I try alpaca with `squad2` dataset.

Is it related or you want me to create another issue for this?

Could you create another issue for this? I can address it there! 

This is on HF’s part, but there are workarounds we can use to fix it.

Sure, I've just created one https://github.com/EleutherAI/lm-evaluation-harness/issues/434

Thanks!