Repository: allenai/reward-bench
Issue: Potential Duplication of BOS Token
Root cause: Validation Gap

I noticed that, for default (sequence classification) models with chat template defined in the tokenizer, `scripts/run_rm.py` formats each conversation by `tokenizer.apply_chat_template` (via the function [`prepare_dialogue_from_tokenizer`](https://github.com/allenai/reward-bench/blob/bc72fb2a573fc31c614eef3405d354b398977b02/rewardbench/utils.py#L515)) and then uses the text classification `pipeline` to process the formatted conversations. Given that 1) many models' tokenizers (e.g., Llama-3 instruct series, Gemma-2 instruct series, etc.) define the `bos_token` in the chat template, and 2) the `pipeline` adds another `bos_token` during tokenization, does it mean these models read in two bos tokens in the forward pass?

I also realized that some models (e.g., ArmoRM) inherently avoids this potential issue via [customized pipeline](https://github.com/allenai/reward-bench/blob/bc72fb2a573fc31c614eef3405d354b398977b02/rewardbench/models/armorm.py#L24) by directly performing tokenization using `tokenizer.apply_chat_template` (as opposed to first formatting, then tokenizing).

Hey @chrisliu298 -- we abstracted away from the HuggingFace pipeline to our own one to simplify some of this. 
Does the HF Tokenizer add a `bos_token` by default, see here https://github.com/allenai/reward-bench/blob/main/rewardbench/models/pipeline.py

Ah okay, just ran an example, this seems right. Hmm looking.

Minimal example:
```
from transformers import AutoTokenizer
tokenizer= AutoTokenizer.from_pretrained("oobabooga/llama-tokenizer")
out = tokenizer.call("Testing my text")
print(out)
print(tokenizer.bos_token)
print(tokenizer.convert_ids_to_tokens(out['input_ids'])
```

@chrisliu298 -- it varies by model. Some examples (that should likely be fixed). 
* InternLM adds BOS token with chat template https://huggingface.co/internlm/internlm2-20b-reward/blob/6539b1df85e74019a3e36da43901e18205c572ea/tokenizer_config.json#L103
* Tulu 2 does not  https://huggingface.co/allenai/tulu-2-dpo-70b/blob/0ab5c875f0070d5aee8d36bc55f41de440a13f02/tokenizer_config.json#L35
* Llama 3 does not https://huggingface.co/NCSOFT/Llama-3-OffsetBias-RM-8B/blob/ddb9f337b80726b15eb16faa7c20d02050ca178a/tokenizer_config.json#L2053

I think the solution is in the standard inference pipeline, check if `bos_token` gets doubled

Example with the failure:
```
>>> chat = [
...    {"role": "user", "content": "Hello, how are you?"},
...    {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
...    {"role": "user", "content": "I'd like to show off how chat templating works!"},
... ]
>>>
>>> tokenizer.
KeyboardInterrupt
>>> chat = tokenizer.apply_chat_template(chat, tokenize=False)
>>> chat
"

user\nHello, how are you?
\n
assistant\nI'm doing great. How can I help you today?
\n
user\nI'd like to show off how chat templating works!
\n
"
>>> tokenizer(chat)
{'input_ids': [1, 1, 92543, 1008, 364, 9843, 328, 1392, 657, 629, 345, 92542, 364, 92543, 525, 11353, 364, 295, 2940, 3890, 2395, 281, 2745, 777, 489, 1638, 629, 3514, 345, 92542, 364, 92543, 1008, 364, 295, 4330, 1217, 442, 1620, 1147, 1392, 6392, 1708, 631, 1237, 4437, 346, 92542, 364, 92527], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

I'll submit a small fix for off by one errors in the default pipeline.

The top model with this issue that I've found is https://huggingface.co/weqweasdas/RM-Mistral-7B/blob/main/tokenizer_config.json, most models use custom code or do not have BOS token. The specific implementation of InternLM could be wrong, but I consider that separate.