Repository: EleutherAI/lm-evaluation-harness
Issue: Rolling window having disjoint windows
Root cause: Algorithmic Error

I've been reading at the rolling window implementation and as I understand it, it doesn't really have overlapping rolling windows, but rather "disjoint" windows, this is not to be confused with `utils.make_disjoint_window`, which just ensures context and continuation don't overlap.

I thought the impl would be following something like [this](https://huggingface.co/docs/transformers/perplexity), which gave me the impression that overlapping rolling windows are desirable to get a more precise "approximation of the fully-factorized perplexity", so it feels a bit odd that a method called `loglikelihood_rolling` would essentially compute ppl over disjoint chunks.

I've printed the indexes generated by "utils.get_rolling_token_windows" and I get a bunch of disjoint indexes (for wikitext):

```
Running loglikelihood_rolling requests
  0%|                                                                                                                                                                     | 0/62 [00:00<?, ?it/s]Input tokens: [prefix_token] + 0 : 2047
Input tokens: 2047 : 4095
Input tokens: 4095 : 6143
Input tokens: 5528 : 7576
  2%|██▌                                                                                                                                                          | 1/62 [00:01<01:49,  1.80s/it]Input tokens: [prefix_token] + 0 : 1446
  3%|█████                                                                                                                                                        | 2/62 [00:02<00:52,  1.15it/s]Input tokens: [prefix_token] + 0 : 2047
Input tokens: 2007 : 4055
  5%|███████▌                                                                                                                                                     | 3/62 [00:02<00:40,  1.44it/s]Input tokens: [prefix_token] + 0 : 2047
Input tokens: 2047 : 4095
Input tokens: 4095 : 6143
Input tokens: 5220 : 7268
...
```

Modified code (from [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/a2cada5d8e1534c636a6bd43ae98eb16eb6240a7/lm_eval/utils.py#L109)):
```
    assert 1 <= context_len <= max_seq_len
    if not token_list:
        return
    # +1 offset, going from input->preds
    pred_len = max_seq_len - context_len + 1
    predicted = 0

    # Special handling for first window: predict all tokens
    first_seq_len = min(max_seq_len, len(token_list))
    print("Input tokens: [prefix_token] +", 0, ":", first_seq_len - 1)
    #print("Pred tokens:", 0, ":", first_seq_len)
    yield ([prefix_token] + token_list[: first_seq_len - 1], token_list[:first_seq_len])
    predicted += first_seq_len

    while predicted < len(token_list):
        window_pred_len = min(len(token_list) - predicted, pred_len)
        window_end = predicted + window_pred_len

        print("Input tokens:", window_end - max_seq_len - 1, ":", window_end - 1)
        #print("Pred tokens:", window_end - window_pred_len, ":", window_end)
        yield (
            token_list[window_end - max_seq_len - 1 : window_end - 1],
            token_list[window_end - window_pred_len : window_end],
        )
        predicted += window_pred_len
```

Is there a particular reason to go for this implementation on the eval-harness? Is this a "standard" practice adopted in some paper and hence here too?




Closing as we've revamped this part of the codebase making this issue irrelevant.