Repository: EleutherAI/lm-evaluation-harness
Issue: Dummy perplexity on LAMBADA
Root cause: Validation Gap

```
python main.py --model dummy --device 0 --tasks lambada,hellaswag
```
|  Task   |Version| Metric |Value |   |Stderr|
|---------|------:|--------|-----:|---|-----:|
|lambada  |      0|ppl     |1.6617|±  |0.0067|



That perplexity value is **extremely** low. GPT-2's PPL on LAMBADA ranges from 35 on 117M model down to 8.63 on 1.5B model. I get around 40 with `--model gpt2`.

Unless I misunderstand something calculation of LAMBADA perplexity is very broken at least for `dummy`.

I'm a bit confused about the evaluation. I was debugging the code (using the lambada task) and I came across this situation. The input is being cut in the wrong position?

Because in this example the word "drapes" appears as "drap" and the model is completing correctly with "es".
The target is "drapes" and because of the wrong cut, the match is different.

Someone can comment something about it?

Input text (inp):
Yanking back a fistful of the thief's hair and sinking her teeth into an unprotected throat. It was a strange, unsettling vision, but for a moment it almost seemed real. She became aware that Bonnie and Meredith were looking at her. "Well?" she said, feeling slightly uncomfortable. "I could tell you weren't listening," sighed Bonnie It can be anywhere from 4% up to 12%, with the average around 6% or 8%. The broker then turns around and shares his or her proceeds with the selling broker, who is the broker representing the buyer. (Confusing, I know.) In a net listing, however, the owner receives a specified — net — amount from the sale, with the excess going to the broker I wanted to shout myself hoarse at him, but I couldn't do it. I just wanted him gone. When I didn't move, he closed the heavy drapes around the bed, sealing me off in a dark little cage. 23 Antiquary I didn't sleep for hours. I could hear him at his desk, writing away, hidden from me only by the drap

Model output (greedy_tokens):
tensor([[259, 299]])
es

Target (cont_toks):
tensor([[95904, 299]])
drapes

FWIW we are also experiencing strange results from eval harness, specifically on lambada. Appears there may have been a regression? as the scores are a little lower compared to the results reported in the paper:

```
|  Task   |Version| Metric | Value |   |Stderr|
|---------|------:|--------|------:|---|-----:|
|hellaswag|      0|acc     | 0.2892|±  |0.0045|
|         |       |acc_norm| 0.3114|±  |0.0046|
|lambada  |      0|ppl     |40.0554|±  |1.4881|
|         |       |acc     | 0.3256|±  |0.0065|
```

compared to ~45% acc and ~35 ppl reported in the paper

edit: this for gpt2 to be clear

@lostmsu The dummy model doesn't do anything for computing loglikelihood but spit out a prandom value from `uniform(0, 1)`, so it's artificially low because the `ppl = exp(-mean(-uniform(0, 1)))` is in `[1.0, ~2.5]`.

We addressed @a-cavalcanti's question in the discord server, but I want to crosspost it here for future readers:

> lambada shouldn't be doing any greedy generation
> if this is unmodified then this is expected behavior, the loglikelihood code works by passing in tokens[:-1] to get logprobs on tokens[1:] and then it slices the logprobs to line up with the target

^ Issue Reference for awareness: #354

@sdtblck did you figure out why this was the case? I'm also getting ~10% lower scores on LAMBADA OpenAI for GPT2-XL (vs ~63% as reported in the paper)

```
lambada_openai|      0|ppl   |10.6341|±  |0.3305|
              |       |acc   | 0.5121|±  |0.0070|
```

(x-posting from discord; @sdtblck FYI)

Ah, this seems to be because the lambada_openai dataset evaluates whether the entire last **word** is predicted correctly, rather than just the last **BPE token**.

If I change L313 of base.py (https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/base.py#L313) to
```
                # Check if last BPE token is predicted correctly.
                max_equal = (greedy_tokens[:, -1] == cont_toks[:, -1])
```


then I get the expected accuracy (within 2%)
```
gpt2 (pretrained=gpt2-xl), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
|     Task     |Version|Metric| Value |   |Stderr|
|--------------|------:|------|------:|---|-----:|
|lambada_openai|      0|ppl   |10.6341|±  |0.3297|
|              |       |acc   | 0.6142|±  |0.0068|
```


Is this intended behavior? Based on https://github.com/EleutherAI/lm-evaluation-harness/issues/356 I was assuming that lambada_openai aims to match OpenAI's evaluation exactly, i.e. using both the detokenized dataset and the easier BPE token evaluation.

@jayelm thanks for catching this. The `LambadaOpenAI` tasks introduced in #357 were implemented to signal differences in the `LAMBADA` dataset and the BPE token eval was unintentionally left out.

On further thought, I don't believe last-BPE-token accuracy should be added. `lm_eval` is designed to be tokenization-agnostic such that models with two different tokenizers should not have scores directly influenced by their context/continuation tokenization inside the evaluation process. In this case, it's possible that the last tokens for two different model tokenizers are not the same, thus altering accuracy comparisons.

Note: If we do decide to break this assumption, we can't only change `L313: max_equal = ...` because other tasks rely on word/continuation-level accuracy through this (e.g. ASDiv, TriviaQA, and WebQs,).

I think I’m comfortable providing last-word evaluation and telling the user what OpenAI did if they want to try to replicate it. There is no particularly principled way to make cross-tokenizer comparisons fair, and to be blunt OpenAI cheated. Whether it was deliberate or not, they changed the evaluation protocol in a way that was substantially easier for their models and then compared to the mainline method for other models instead of reevaluating them.

While the original motivation for this library is “let’s replicate OAI’s analyses on our models” that’s not the driving thought anymore. We’ve build an open-source library that’s widely used enough that we should be thinking of its design in terms of standards-setting. If anyone other than OpenAI had done this we would have laughed and called the more common methodology the “correct” scores. I feel totally comfortable doing that here.

Thanks @StellaAthena and @jon-tow for looking into this! I agree with the idea to stick to last word prediction for this harness. While it's nice to know how to reproduce the original OpenAI results if needed, the last BPE token evaluation definitely trivializes a lot of the examples, especially when the final BPE token is highly dependent on the preceding token, e.g. when predicting the second half of a name (e.g. predicting "ana" after "Di", when "Diana" is clear from the context).

@jayelm @lostmsu is there anything more to be discussed here, or can this issue be closed.

A short document describing the evaluation method differences, the way to choose, and a sane default would be nice to have for the conclusion.

Personally, given the shenanigans, I'd settle on something universal (and more importantly, universally easy to understand) as the default. E.g. byte-based instead of word- or token-based.

> I think I’m comfortable providing last-word evaluation and telling the user what OpenAI did if they want to try to replicate it. There is no particularly principled way to make cross-tokenizer comparisons fair, and to be blunt OpenAI cheated. Whether it was deliberate or not, they changed the evaluation protocol in a way that was substantially easier for their models and then compared to the mainline method for other models instead of reevaluating them.
> 
> While the original motivation for this library is “let’s replicate OAI’s analyses on our models” that’s not the driving thought anymore. We’ve build an open-source library that’s widely used enough that we should be thinking of its design in terms of standards-setting. If anyone other than OpenAI had done this we would have laughed and called the more common methodology the “correct” scores. I feel totally comfortable doing that here.

To clarify for future readers on this issue, OpenAI did not do last BPE token evaluation for LAMBADA as claimed here. I believe this was based on a misunderstanding confusing Yaroslav's implementation (see: https://github.com/openai/gpt-2/issues/131#issuecomment-497136199) with ours. I suspect the difference between reported scores in the GPT-2 paper and those in this version of the LAMBADA eval is due to not using the stop word filter that we described in Section 3.3 of the GPT-2 paper.

An example implementation of this can be done by creating a bias vector that's added to the logits to prevent sampling stop words.

```
import torch
from transformers import AutoTokenizer
from spacy.lang.en.stop_words import STOP_WORDS

tokenizer = AutoTokenizer.from_pretrained("gpt2")
stop_word_mask = torch.zeros([50257])
token_to_idx = {tokenizer.decode([i]):i for i in range(50257)}
for word in STOP_WORDS:
    if ' '+word in token_to_idx:
        stop_word_mask[token_to_idx[' '+word]] = -float('inf')
```

This could be added as a flag / option to replicate GPT-2 results if desired, and works well to improve accuracy for smaller LMs, but with GPT-3 LAMBADA was changed into a fill-in-the-blank as well few-shot eval as more general way to specify the task is to predict the last word of the sentence as described in Section 3.1.2.

Closing as we don't have the ability to reproduce OpenAI's unreleased eval set up exactly.