Repository: EleutherAI/lm-evaluation-harness
Issue: Very high perplexity values
Root cause: Algorithmic Error

I have found a problem with calculating perplexity. Maybe it is model dependent.

https://github.com/EleutherAI/lm-evaluation-harness/blob/7d9922c80114218eaf43975b7655bb48cda84f50/docs/model_guide.md?plain=1#L64
"Each request contains `Instance.args : Tuple[str]`, which is an input string to the model whose *entire* loglikelihood, conditioned on purely the EOT token, will be calculated."

Why an input string is conditioned on end of text/sentence and not on beginning?

https://github.com/EleutherAI/lm-evaluation-harness/blob/7d9922c80114218eaf43975b7655bb48cda84f50/lm_eval/models/huggingface.py#L358
Here we can see some comment to use EOT instead of EOS, but EOS is used.

https://github.com/EleutherAI/lm-evaluation-harness/blob/7d9922c80114218eaf43975b7655bb48cda84f50/lm_eval/api/model.py#L68
https://github.com/EleutherAI/lm-evaluation-harness/blob/7d9922c80114218eaf43975b7655bb48cda84f50/lm_eval/models/huggingface.py#L814
Here also EOT is mentioned.

From my testing much better results are with BOS at the beginning. For some llama model I obtain word-level perplexity with EOS 17832760.94 and with BOS 39.14.

Thanks for raising this! Indeed you're correct and a better choice is probably to use BOS if the model has one, else use EOS/EOT (EOS and EOT are two interchangeable names for EOS). If you'd want to PR that fix so we can credit you for it then that'd be great, else I will make it!

This is a holdover from when the library was predominantly evaluating models that didn't use BOS tokens--for example none of EAI's models are trained with BOS, nor gpt-2, BLOOM, and more. Llama using BOS seems to have started a trend though.


> For some llama model I obtain word-level perplexity with EOS 17832760.94 and with BOS 39.14.
What Llama model were you getting these results with? I've not observed extremely poor results like this with the Llama pretrained models as far as I'm aware, although Gemma does seem adversely affected by BOS-related sensitivities.

Thanks, I will prepare a PR.

The scores are obtained with Llama 13b continously pretrained on Polish tested with Polish sentence. Perplexity calculated with evaluate library is better than original model but here are exploding. 

Thank you! 

That's interesting--I still do not have a sense for what might be causing the explosion for Gemma and in your case this Llama continuous pretrain. 
Are you packing documents into contexts? masking out attention between the documents?  

This is the model: OPI-PG/Qra-13b

Tested on the first sentence of enelpol/poleval2018_task3_test_10k

Using metric: https://huggingface.co/spaces/evaluate-metric/perplexity/blob/main/perplexity.py
If we change in the above code BOS to EOS then the perplexity is very high.

Results of command before and after PR:
`time lm_eval --model hf --model_args pretrained=OPI-PG/Qra-13b --output_path tests_temp/ --tasks polish_poleval2018_task3_test_10k  --num_fewshot 0 --device cuda:0 --batch_size 4 --log_samples --limit 10`

Task definition: https://github.com/speakleash/lm-evaluation-harness/blob/main/lm_eval/tasks/polish_poleval2018_task3/polish_poleval2018_task3_test_10k.yaml

```
hf (pretrained=OPI-PG/Qra-13b), gen_kwargs: (None), limit: 10.0, num_fewshot: 0, batch_size: 4
|              Tasks              |Version|Filter|n-shot|    Metric     |    Value     |   |Stderr|
|---------------------------------|-------|------|-----:|---------------|-------------:|---|------|
|polish_poleval2018_task3_test_10k|Yaml   |none  |     0|word_perplexity|214593953.3901|±  |N/A   |
|                                 |       |none  |     0|byte_perplexity|       13.2910|±  |N/A   |
|                                 |       |none  |     0|bits_per_byte  |        3.7324|±  |N/A   |

hf (pretrained=OPI-PG/Qra-13b), gen_kwargs: (None), limit: 10.0, num_fewshot: 0, batch_size: 4
|              Tasks              |Version|Filter|n-shot|    Metric     | Value  |   |Stderr|
|---------------------------------|-------|------|-----:|---------------|-------:|---|------|
|polish_poleval2018_task3_test_10k|Yaml   |none  |     0|word_perplexity|114.0827|±  |N/A   |
|                                 |       |none  |     0|byte_perplexity|  1.8942|±  |N/A   |
|                                 |       |none  |     0|bits_per_byte  |  0.9216|±  |N/A   |
```

```
hf (pretrained=HuggingFaceH4/zephyr-7b-alpha), gen_kwargs: (None), limit: 10.0, num_fewshot: 0, batch_size: 4
|              Tasks              |Version|Filter|n-shot|    Metric     |  Value  |   |Stderr|
|---------------------------------|-------|------|-----:|---------------|--------:|---|------|
|polish_poleval2018_task3_test_10k|Yaml   |none  |     0|word_perplexity|5399.3146|±  |N/A   |
|                                 |       |none  |     0|byte_perplexity|   3.1866|±  |N/A   |
|                                 |       |none  |     0|bits_per_byte  |   1.6720|±  |N/A   |

hf (pretrained=HuggingFaceH4/zephyr-7b-alpha), gen_kwargs: (None), limit: 10.0, num_fewshot: 0, batch_size: 4
|              Tasks              |Version|Filter|n-shot|    Metric     |  Value  |   |Stderr|
|---------------------------------|-------|------|-----:|---------------|--------:|---|------|
|polish_poleval2018_task3_test_10k|Yaml   |none  |     0|word_perplexity|1433.5788|±  |N/A   |
|                                 |       |none  |     0|byte_perplexity|   2.6648|±  |N/A   |
|                                 |       |none  |     0|bits_per_byte  |   1.4140|±  |N/A   |
```

Facing (what I think is) the same issue, with perplexity on Lambada (reaching 10^9), and accuracy is just 0!
The model is Vicuña v1.5 7B

|     Tasks      |Version|Filter|n-shot|  Metric   |    Value     |   |   Stderr    |
|----------------|------:|------|------|-----------|-------------:|---|------------:|
|lambada_standard|      1|none  |None  |perplexity |803140990.3148|±  |90868900.4289|
|                |       |none  |None  |acc        |        0.0000|±  |       0.0000|
|wsc273          |      1|none  |None  |acc        |        0.5128|±  |       0.0303|
|winogrande      |      1|none  |None  |acc        |        0.5043|±  |       0.0141|
|webqs           |      2|none  |None  |exact_match|        0.0000|±  |       0.0000|
|arc_easy        |      1|none  |None  |acc        |        0.2492|±  |       0.0089|
|                |       |none  |None  |acc_norm   |        0.2521|±  |       0.0089|
|arc_challenge   |      1|none  |None  |acc        |        0.2304|±  |       0.0123|
|                |       |none  |None  |acc_norm   |        0.2833|±  |       0.0132|

@shikhar-srivastava could you try the fix in #1588 and see if it helps?

IIRC, Vicuna (or maybe it was Llama-1-7B mirrors on HF?) had some tokenizer config issues re: not having special tokens properly set up that might also be causing this.

Odd. Fix #1588 doesn't seem to have changed anything for me. @haileyschoelkopf @djstrong 

|       Tasks       |Version|Filter|n-shot|  Metric  |    Value     |   |   Stderr    |
|-------------------|-------|------|------|----------|-------------:|---|------------:|
|lambada            |N/A    |none  |None  |acc       |        0.0000|±  |       0.0000|
|                   |       |none  |None  |perplexity|502817149.2094|±  |47009091.2157|
| - lambada_openai  |      1|none  |None  |perplexity|202493306.0231|±  |24130095.8690|
|                   |       |none  |None  |acc       |        0.0000|±  |       0.0000|
| - lambada_standard|      1|none  |None  |perplexity|803140992.3958|±  |90868900.6265|
|                   |       |none  |None  |acc       |        0.0000|±  |       0.0000|



The results should be different because BOS is defined and it works better with this model on my small test with word perplexity. Metric `perplexity` works differently - maybe there nothing is prepended?

So, `perplexity` is not affected.
```
BOS
hf (pretrained=lmsys/vicuna-7b-v1.5), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 4
|    Tasks     |Version|Filter|n-shot|  Metric  |Value |   |Stderr|
|--------------|------:|------|-----:|----------|-----:|---|-----:|
|lambada_openai|      1|none  |     0|perplexity|3.2966|±  |0.0790|
|              |       |none  |     0|acc       |0.7112|±  |0.0063|

EOS
hf (pretrained=lmsys/vicuna-7b-v1.5,prefix_token_id=2), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 4
|    Tasks     |Version|Filter|n-shot|  Metric  |Value |   |Stderr|
|--------------|------:|------|-----:|----------|-----:|---|-----:|
|lambada_openai|      1|none  |     0|perplexity|3.2966|±  |0.0790|
|              |       |none  |     0|acc       |0.7112|±  |0.0063|
```