Repository: huggingface/evaluate
Issue: Issue with Perplexity metric
Root cause: Validation Gap

## Describe the bug
For most of the model keys that are listed for perplexity score, the keys are not present.
https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForCausalLM

For using keys like `bert-base-multilingual-cased` or `xlm-roberta-large`, I get the following error

## Steps to reproduce the bug
```python
from datasets import load_metric
metric = load_metric("perplexity")

metric.compute(
    model_id="bert-base-multilingual-cased",
    input_texts=dataset
)
```

## Expected results
The model should have been able to process the request and produce a perplexity score
```
{'perplexity': XXX}
```

## Actual results
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_30901/2773184990.py in <cell line: 1>()
----> 1 metric.compute(
      2     model_id="bert-base-multilingual-cased",
      3     input_texts=flores_true_src
      4 )

~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/datasets/metric.py in compute(self, predictions, references, **kwargs)
    428             inputs = {input_name: self.data[input_name] for input_name in self.features}
    429             with temp_seed(self.seed):
--> 430                 output = self._compute(**inputs, **compute_kwargs)
    431 
    432             if self.buf_writer is not None:

~/.cache/huggingface/modules/datasets_modules/metrics/perplexity/35bbdc10965bddb5e3a69737699ecb19f4ddc4c115e0b929c94bb7d38897bd57/perplexity.py in _compute(self, input_texts, model_id, stride, device)
    110         special_tokens_masks = encodings["special_tokens_mask"]
    111 
--> 112         max_model_length = model.config.n_positions
    113 
    114         ppls = []

~/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/configuration_utils.py in __getattribute__(self, key)
    250         if key != "attribute_map" and key in super().__getattribute__("attribute_map"):
    251             key = super().__getattribute__("attribute_map")[key]
--> 252         return super().__getattribute__(key)
    253 
    254     def __init__(self, **kwargs):

AttributeError: 'BertConfig' object has no attribute 'n_positions'
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.4.0-1066-aws-x86_64-with-glibc2.10
- Python version: 3.8.12
- PyArrow version: 7.0.0
- Pandas version: 1.3.4

Hi @prabhakar267, thanks for reporting.

Please note that our old metrics module is in the process of being incorporated to a separate library called `evaluate`: https://github.com/huggingface/evaluate

I'm transferring your issue to the new repository.

cc @TristanThrush 

Hi there @prabhakar267, sorry for the delay. I've been traveling for over 2 weeks. It seems that the updated version of datasets and evaluate does not have this issue. I think it was fixed here: https://github.com/huggingface/datasets/commit/e90a7d41826967017c30d63cd46013a9272d3f77

Are you able to update your version? Let me know if there are any other issues.



Hi @prabhakar267 I'm closing this issue, because I think it's resolved? Let me know if I've missed something, and I'll reopen.