Repository: EleutherAI/lm-evaluation-harness
Issue: 'list' object has no attribute 'split' error for the `triviaqa` dataset.
Root cause: Validation Gap

When running `evaluator.simple_evaluate(llm_model, tasks=['triviaqa], no_cache=True)` with the opt-model, it report the following error. I found that it is because `term` variant is not a string but a list of tokens. 

```
Traceback (most recent call last):
  File "playground/inference.py", line 380, in <module>
    results = evaluator.simple_evaluate(llm_model, tasks=llm_tasks, no_cache=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "lm-evaluation-harness/lm_eval/evaluator.py", line 104, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "lm-evaluation-harness/lm_eval/utils.py", line 243, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "lm-evaluation-harness/lm_eval/evaluator.py", line 304, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lm-evaluation-harness/lm_eval/base.py", line 459, in greedy_until
    s = s.split(term)[0]
        ^^^^^^^
AttributeError: 'list' object has no attribute 'split'
```

I have inherited my model from lm_eval.base.BaseLM. I am using the latest code `bc10a390` from the `master` branch.

When running the code, the terminal prints 

```
max_gen_toks fn
max_gen_toks fn
max_gen_toks fn
```

Is that suppose to happen? Happy to provide any additional information if needed.

Hi! Would you be able to provide a script to reproduce how you're instantiating your model and calling `simple_evaluate` ? I'm not able to reproduce this after `bc10a390`, as far as I can tell. Thanks!

Hi, thank you for your response. I instantiate the model using the following code
```
class LLMClass(BaseLM):
    def __init__(self, model, tokenizer, batch_size=1):
        super().__init__()

        self._device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.batch_size_per_gpu = batch_size

        self.model = model
        self.seqlen = self.model.config.max_position_embeddings
        self.model.eval()

        # pretrained tokenizer for neo is broken for now so just hard-coding this to gpt2
        self.tokenizer = tokenizer
        self.vocab_size = self.tokenizer.vocab_size
        print("model vocab size: ", self.vocab_size)

    @property
    def eot_token(self) -> str:
        return self.tokenizer.eos_token

    @property
    def eot_token_id(self):
        # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*
        return self.tokenizer.eos_token_id

    @property
    def max_length(self):
        try:
            return self.gpt2.config.n_ctx
        except AttributeError:
            # gptneoconfig doesn't have n_ctx apparently
            return self.model.config.max_position_embeddings

    @property
    def max_gen_toks(self):
        print("max_gen_toks fn")
        return 256

    @property
    def batch_size(self):
        # TODO: fix multi-gpu
        return self.batch_size_per_gpu  # * gpus

    @property
    def device(self):
        # TODO: fix multi-gpu
        return self._device

    def tok_encode(self, string: str):
        return self.tokenizer.encode(string, add_special_tokens=False)

    def tok_encode_batch(self, strings):
        return self.tokenizer(
            strings,
            padding=True,
            add_special_tokens=False,
            return_tensors="pt",
        )

    def tok_decode(self, tokens):
        return self.tokenizer.batch_decode(tokens, skip_special_tokens=True)

    def _model_call(self, inps):
        """
        inps: a torch tensor of shape [batch, sequence]
        the size of sequence may vary from call to call
        returns: a torch tensor of shape [batch, sequence, vocab] with the
        logits returned from the model
        """
        with torch.no_grad():

            return self.model(inps)["logits"]

    def model_batched_set(self, inps):
        pdb.set_trace()
        dataset_logits = []
        for batch in inps:
            multi_logits = F.log_softmax(
                self._model_call(batch), dim=-1
            ).cpu()  # [batch, padding_length, vocab]
            dataset_logits.append(multi_logits)
        return dataset_logits

    def _model_generate(self, context, max_length, eos_token_id):
        return self.model.generate(
            context, max_length=max_length, eos_token_id=eos_token_id, do_sample=False
        )
```
The standard hugging face opt model and tokenizer are used to init this `LLMClass`
It is then passed to the evaluator with 
`results = evaluator.simple_evaluate(llm_model, tasks=llm_tasks, no_cache=True)`
I test the script with the latest commit `25dfd3f6`, and the problem stays the same. I would appreciate it if you could tell me what might be wrong. Thanks a lot

Closing because on the new `main` branch release I cannot replicate this, and TriviaQA runs well.