Title: Perplexity Speedup
Author: emibaylor (CONTRIBUTOR)
State: closed
Created: 2022-04-06T12:57:21Z

This PR makes necessary changes to perplexity such that:
- it runs much faster (via batching)
- it throws an error when input is empty, or when input is one word without <BOS> token
- it adds the option to add a <BOS> token

Issues:
- The values returned are extremely high, and I'm worried they aren't correct. Even if they are correct, they are sometimes returned as `inf`, which is not very useful (see [comment below](https://github.com/huggingface/datasets/pull/4108#discussion_r843931094) for some of the output values). 
  - If the values are not correct, can you help me find the error?
  - If the values are correct, it might be worth it to measure something like perplexity per word, which would allow us to get actual values for the larger perplexities, instead of just `inf`

Future:
- `stride` is not currently implemented here. I have some thoughts on how to make it happen with batching, but I think it would be better to get another set of eyes to look at any possible errors causing such large values now rather than later.

Comments:
WRT the high values, can you add some unit tests with some [string, model] pairs and their resulting perplexity code, and @TristanThrush can run the same pairs through his version of the code?

_The documentation is not available anymore as the PR was closed or merged._

I thought that the perplexity metric should output the average perplexity value of all the strings that it gets as input (not a perplexity value per string, as the new version does).
@lhoestq , @TristanThrush thoughts?

> I thought that the perplexity metric should output the average perplexity value of all the strings that it gets as input (not a perplexity value per string, as the new version does). @lhoestq , @TristanThrush thoughts?

I support this change from Emi. If we have a perplexity function that loads GPT2 and then returns an average over all of the strings, then it is impossible to get multiple perplexities of a batch of strings efficiently. If we have this new perplexity function that is built for batching, then it is possible to get a batch of perplexities efficiently and you can still compute the average efficiently afterwards.

Thanks a lot for working on this @emibaylor @TristanThrush :)

For consistency with the other metrics, I think it's nice if we return the mean perplexity. Though I agree that having the separate perplexities per sample can also be useful. What do you think about returning both ?
```python
return {"perplexities": ppls, "mean_perplexity": np.mean(ppls)}
```
we're also doing this for the COMET metric.

> Thanks a lot for working on this @emibaylor @TristanThrush :)
> 
> For consistency with the other metrics, I think it's nice if we return the mean perplexity. Though I agree that having the separate perplexities per sample can also be useful. What do you think about returning both ?
> 
> ```python
> return {"perplexities": ppls, "mean_perplexity": np.mean(ppls)}
> ```
> 
> we're also doing this for the COMET metric.

Thanks! Sounds great to me.

The CI fail is unrelated to your PR and has been fixed on master, feel free to merge the master branch into your PR to fix the CI ;)