Repository: EleutherAI/lm-evaluation-harness
Issue: Stopping Criteria depending on the batch size leads to different results
Root cause: Algorithmic Error

Using the huggingface class for generative tasks I observed scores being dependent on the batch size - where the value 1 gives the best results. 

Examining the stopping criteria code I see: https://github.com/EleutherAI/lm-evaluation-harness/blob/00209e10f6e27edf5d766145afaf894079b5fe10/lm_eval/models/huggingface.py#L748-L766

I wonder why is there a need to know/use the batch size when preparing the stopping patterns; put differently, why the need to stop should depend on other sentences in the batch (line 766)? these are different examples and not a variation on the same example, like in the multi-choice tasks. 

I think you're misunderstanding what this code does.

`self.done_tracker` begins as `[False, False, ..., False]` (w/ length equal to the batch size). Then we generate one token for each batch and move on to the for loop. If you put `print(i, done)` immediately after this loop, it would print out the following:
```
0 False
1 False
...
bs False
```
At each step of the loop it does the following:
1. check if we are already done (in which case we continue with the loop)
2. check if `self.sequence in lookback_tokens_batch[i]`. `i` is the current batch index, so this is just checking if the generation contains one of the stop sequences for this batch.
3. assign the value of `self.sequence in lookback_tokens_batch[i]` to `self.done_tracker[i]`. This means that if we encounter a stop token then the next time this function is called the code will know its stopped.
4. return `False not in self.done_tracker`, which is logically equivalent to returning `True` if all elements of `self.done_tracker` are `True` and returning `False` otherwise. This ensures that the entire batch isn't stopped until every sequence in the batch has met a stopping condition.

If increasing the batch size hurts performance, I don't think that this is (directly) why. I would look at the generations that come from the same batch in situations where one sequence in the batch hits a stop token much earlier than the other. One possible explanation for what you're seeing that I can't immediately rule out by looking at the code is that the model doesn't actually stop generating for the sequence that's supposed to stop early, filling it with junk that doesn't match the stopping criteria.

@StellaAthena I experienced the samething with MGSM. I've checked the generations and you are correct about 

> model doesn't actually stop generating for the sequence that's supposed to stop early, filling it with junk that doesn't match the stopping criteria.

Isn't this a major issue? As all generation tasks can be only be evaluated in batch size of 1.


> @StellaAthena I experienced the samething with MGSM. I've checked the generations and you are correct about
> 
> > model doesn't actually stop generating for the sequence that's supposed to stop early, filling it with junk that doesn't match the stopping criteria.
> 
> Isn't this a major issue? As all generation tasks can be only be evaluated in batch size of 1.

This seems correct. We'll need to figure out how to solve this while still preserving the ability to do batch evaluations.

On `big-refactor` branch, tested on Gsm8k with GPT2, and found that with BS = 1 and BS = 8 I received precisely the same outputs from the logged samples.

The fact that stopping conditions operate on a per-batch level is a bug / design choice on HF's end, and a fix is not planned: https://github.com/huggingface/transformers/issues/22340#issuecomment-1507459140 

In the `big-refactor` branch, which will soon be replacing this older code, I have pushed a fix for this at #1009 . 

1. We now pass all stop sequences to the StoppingCriteriaList , so this should speed up generation substantially where previously generation was only stopped early upon the primary stopping criteria (this does not affect the actual generations scored, due to the postprocessing done by `greedy_until` that strips all post-stopping-condition text for every stopsequence). 
2. **Functional Change**: previously, multi-token stopsequences might still slip through, due to tokenization messiness. For example, with the stop sequence `"\n\n"`, the stopping criteria would tokenize this as `["\n\n"]` and assume it only has to look at the last single token, but the model may generate a token sequence corresponding to `["\n", "\n"]`, and the stopping criteria will not catch this and correctly stop, because it only looks at the final token and sees `"\n" != "\n\n"`. The fix to this is to simply make sure we look back a few more tokens than the StoppingCriteria expects, which should almost always catch such behavior (and can be tweaked to ensure such)

Together, these should speed up generation-based tasks by stopping more effectively, but to my knowledge and in my testing, it should not change the resulting string outputs that have had the following truncation applied to them.

```
for term in until:
    if len(term) > 0:
        # ignore '' separator,
        # for seq2seq case where self.tok_decode(self.eot_token_id) = ''
        s = s.split(term)[0]
```

I've left a comment describing this new `MultiTokenEOSCriteria` behavior in `lm_eval/utils.py` there.


@MattYoon , if you have experienced substantially different scores when evaluating tasks with BS=1 and BS>1, is there a command you have to replicate this? Else, I think that with the postprocessing we do, any unstopped generations will have the junk-at-the-end cut off after the generation call finishes, meaning batch size should not impact the results' measured stopping points.