Repository: EleutherAI/lm-evaluation-harness
Issue: Failure to evaluate facebook/opt-350m model
Root cause: Validation Gap

I attempted to evaluate facebook/opt-350m model using the EleutherAI evaluation harness with the following command:
`python main.py --model gpt2 --model_args pretrained=facebook/opt-350m --tasks lambada_openai,hellaswag --device 0`
I got an assertion error:
`assert self.tokenizer.encode("hello\n\nhello") == [
AssertionError: [2, 42891, 50118, 50118, 42891]`

My understanding is that the string "hello\n\nhello" gets encoded by facebook/opt-350m's tokenizer as [2, 42891, 50118, 50118, 42891] and by gpt2's tokenizer as [31373, 198, 198, 31373], while both tokenizers are considered to be an instance of transformers.GPT2Tokenizer, so the assertion is checked and fails.

My question: what command should I use to evaluate facebook/opt-350m model?
Thank you very much for your help.

That's curious. I have evaluated OPT using the eval harness in the past, so this may have been a case of not realizing the consequences of an added assertion test. I would comment out the assertion for now, but we'll work on fixing this issue.

Just wanted to note to self and others that the OPT tokenizer prepends a BOS token but the `HFLM.tok_encode` method forcefully skips them https://github.com/EleutherAI/lm-evaluation-harness/blob/57f8ff24ca492b4cbd9c86fd9db7cfa519d79284/lm_eval/models/gpt2.py#L107-L108 If one wanted to maintain the BOS token behavior, this would have to be flipped to `add_special_tokens=True`. As a consequence, there are a few token manipulations you'd have to watch out for, mainly that `contexts` and `continuations` get tokenized separately  in a few places such as  https://github.com/EleutherAI/lm-evaluation-harness/blob/57f8ff24ca492b4cbd9c86fd9db7cfa519d79284/lm_eval/base.py#L174-L181
In this case, the continuation encodings must be stripped of their BOS tokens before they are concatenated back together.

I'm not sure of a general solution to this but will continue to mull it over. Thank you for reporting @underactuated 

> That's curious. I have evaluated OPT using the eval harness in the past, so this may have been a case of not realizing the consequences of an added assertion test. I would comment out the assertion for now, but we'll work on fixing this issue.

Good to know that I can safely comment out the assertion, thanks!

@underactuated the assertion is intended to prevent people from accidentally loading the wrong tokenizer and getting meaningless results. If you’re confident that you’re loading the correct tokenizer, its safe to remove.

@underactuated I’m going to leave this open as a reminder that someone should go figure out a general solution that doesn’t require hacking the codebase, but I’m going to consider this solved.

@StellaAthena Thanks a lot, yes, it solved my problem. I assume I don't need to formally mark it as solved, as I don't see such an option. (I am somewhat new to github, please let me know if any action is required on my part).

update: sorry, I closed the issue by mistake, so I reopened it, as I didn't see how to undo the closing.

Closing since we now merged a `hf-causal` LM class that does not use these assertions.