Repository: EleutherAI/lm-evaluation-harness
Issue: bos token gets added 2 times for gemma models for loglikelihood_rolling
Root cause: Algorithmic Error

Hi, I observed that for gemma models the bos token gets added two times when using the loglikelihood (and probably also the loglikelihood_rolling). Which makes the perplexity scores higher than it should be. As I am not much familiar with the library not sure if it is also the case for other metrics.  Here is the explanation:

## Where BOS Token Gets Added

The BOS token gets added in **two places** during tokenization:

### 1. During Tokenization (`tok_encode`)

When `add_bos_token` is set to `True`, the tokenizer adds a BOS token during encoding.

For causal models, the `tok_encode` method adds the BOS token if `self.add_bos_token` is `True` (line 772).
**File:** lm_eval/models/huggingface.py (L 772-796)
```python
    def tok_encode(
        self, string: str, left_truncate_len=None, add_special_tokens=None
    ) -> List[int]:
        """ """
        # default for None - empty dict, use predefined tokenizer param
        # used for all models except for CausalLM or predefined value
        special_tokens_kwargs = {}

        # by default for CausalLM - false or self.add_bos_token is set
        if add_special_tokens is None:
            if self.backend == "causal":
                special_tokens_kwargs = {
                    "add_special_tokens": False or self.add_bos_token
                }
        # otherwise the method explicitly defines the value
        else:
            special_tokens_kwargs = {"add_special_tokens": add_special_tokens}

        encoding = self.tokenizer.encode(string, **special_tokens_kwargs)

        # left-truncate the encoded context to be at most `left_truncate_len` tokens long
        if left_truncate_len:
            encoding = encoding[-left_truncate_len:]

        return encoding
```


And for Gemma models this is hard-coded to set true at lines 214-218.
**File:** lm_eval/models/huggingface.py (L 214-218)
```python
  if "gemma" in getattr(self.config, "model_type", ""):
      self.add_bos_token = True
      eval_logger.info(
          f"Model type is '{self.config.model_type}', part of the Gemma family--a BOS token will be used as Gemma underperforms without it."
            )
```

### 2. As Prefix Token in `loglikelihood`

When the context is empty, the `loglikelihood` method in `TemplateLM` explicitly adds `prefix_token_id` as the context. 
**File:** lm_eval/api/model.py (L 362-378)
```python
    def loglikelihood(
        self, requests, disable_tqdm: bool = False
    ) -> List[Tuple[float, bool]]:
        new_reqs = []
        for context, continuation in [req.args for req in requests]:
            if context == "":
                # BOS or EOS as context
                context_enc, continuation_enc = (
                    [self.prefix_token_id],
                    self.tok_encode(continuation),
                )
            else:
                context_enc, continuation_enc = self._encode_pair(context, continuation)

            new_reqs.append(((context, continuation), context_enc, continuation_enc))

        return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)

```
And when `self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)` gets called, lets say for huggingface models the context and the continuation gets added together (may also be the case for other models) causing the sequence to start with 2 bos tokens (token number `2` in this case as seen in the output).
**File:** lm_eval/models/huggingface.py (L 362-378)
```python
    def loglikelihood(
        self, requests, disable_tqdm: bool = False
    ) -> List[Tuple[float, bool]]:
        new_reqs = []
        for context, continuation in [req.args for req in requests]:
            if context == "":
                # BOS or EOS as context
                context_enc, continuation_enc = (
                    [self.prefix_token_id],
                    self.tok_encode(continuation),
                )
            else:
                context_enc, continuation_enc = self._encode_pair(context, continuation)

            new_reqs.append(((context, continuation), context_enc, continuation_enc))

        return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm)
```
output:
```
============================================================
(2, 2, 150535, 44867, 532, 6812, 5645, 105657, 1274, 532, 90599, 651, 236764, 1015, 964, 8101, 3107, 684, 236761, 999, 38786, 611, 1156, 577, 834, 2712, 618, 531, 159854, 672, 880, 3121, 9377)
============================================================
```

**The issue occurs when both conditions are true:**

1. `add_bos_token=True` is set (hard-coded Gemma models)
2. The context is empty (which triggers the prefix token addition)

In this case:
- First BOS: Added by `tok_encode(continuation)` if `add_bos_token=True` 
- Second BOS: Added by `[self.prefix_token_id]` when context is empty

Here I showed the huggingface model but it is probably also the case for other models, as the problem happens before. I have very limited experience with this library so not sure which would be the best solution for this, my guess would be either by removing the hard-coded add_bos_token=True for gemma models or checking if it already exists before adding the prefix_token. 

Hi! Thanks for opening this issue and for the detailed report. The PR should fix this by basically checking the condition:
```python
context_enc, continuation_enc = (
                    ([self.prefix_token_id], continuation_enc)
                    if self.prefix_token_id != continuation_enc[0]
                    else (continuation_enc[:1], continuation_enc[1:])
                )
```
Appreciate you documenting this so clearly!

Hi! Thanks for the implementation.