Repository: stanford-crfm/helm
Issue: The first log probability and top choices are None
Root cause: External Dependency Breakage

The first log probability and top choices are None, so we skip. See openai_client.py

```
            for text, logprob, top_logprobs in zip(
                raw_data["tokens"], raw_data["token_logprobs"], raw_data["top_logprobs"]
            ):
                tokens.append(Token(text=text, logprob=logprob or 0, top_logprobs=dict(top_logprobs or {})))
```

Related: https://community.openai.com/t/token-logprobs-when-echo-is-true/9626


We should figure out how to evaluate perplexity of text then.  Close this once we implement that properly.

@percyliang Following https://community.openai.com/t/token-logprobs-when-echo-is-true/9626, what if we prepended `
` to the prompt in `openai_client.py` when `request.echo_prompt` is `True` and then just ignore the first token (which should be `
`) when we process the response?

Sounds reasonable.  Can we check this is what people do when they evaluate perplexity?

> Sounds reasonable. Can we check this is what people do when they evaluate perplexity?

@siddk pointed out this is what EleutherAI also does in their evaluation harness.

Looks like this is what EleutherAI does. https://github.com/EleutherAI/lm_perplexity/blob/24c5a04beb2f073b2230c7c7420bd1ec7776f513/lm_perplexity/models.py#L43 
@teetone The issue is that prepending `
` to the prompt does affect the logprob of the following words and might change the ranking of the words or the sampling results. Shall we always prepend the token to the prompt for all scenarios whether or not `request.echo_prompt` is `True` for the sake of consistency?

We have to be careful of not breaking backward compatibility.

Right now you can do unconditional generation from GPT-3...how would that work?

The other option is to put this in the `Adapter`...  I think we should only make deep changes in the client if it's really the right thing to do.

> We have to be careful of not breaking backward compatibility.
> 
> Right now you can do unconditional generation from GPT-3...how would that work?
> 
> The other option is to put this in the `Adapter`... I think we should only make deep changes in the client if it's really the right thing to do.

That makes sense. I agree. We should keep the client pure and make the change on the benchmarking side.

> Looks like this is what EleutherAI does. https://github.com/EleutherAI/lm_perplexity/blob/24c5a04beb2f073b2230c7c7420bd1ec7776f513/lm_perplexity/models.py#L43 @teetone The issue is that prepending `
` to the prompt does affect the logprob of the following words and might change the ranking of the words or the sampling results. Shall we always prepend the token to the prompt for all scenarios whether or not `request.echo_prompt` is `True` for the sake of consistency?

I'm seeing different log probabilities for the same token when `echo_prompt` is `True` vs `False`:

![Screen Shot 2022-01-12 at 12 57 16 PM](https://user-images.githubusercontent.com/16793796/149224489-f4c36f42-dbf2-4c90-bbd7-144ea931b33d.png)
![Screen Shot 2022-01-12 at 12 57 51 PM](https://user-images.githubusercontent.com/16793796/149224506-68f1b187-d10d-4ed1-a406-1bc17ba7fbc9.png)



> > Looks like this is what EleutherAI does. https://github.com/EleutherAI/lm_perplexity/blob/24c5a04beb2f073b2230c7c7420bd1ec7776f513/lm_perplexity/models.py#L43 @teetone The issue is that prepending `
` to the prompt does affect the logprob of the following words and might change the ranking of the words or the sampling results. Shall we always prepend the token to the prompt for all scenarios whether or not `request.echo_prompt` is `True` for the sake of consistency?
> 
> I'm seeing different log probabilities for the same token when `echo_prompt` is `True` vs `False`:
> 
> ![Screen Shot 2022-01-12 at 12 57 16 PM](https://user-images.githubusercontent.com/16793796/149224489-f4c36f42-dbf2-4c90-bbd7-144ea931b33d.png) ![Screen Shot 2022-01-12 at 12 57 51 PM](https://user-images.githubusercontent.com/16793796/149224506-68f1b187-d10d-4ed1-a406-1bc17ba7fbc9.png)

This does not sound reasonable to me. Why do you think echoing the prompt would affect the logprob?

Another issue is that whether we should also prepend the <eos> token to the input sentence when testing other LMs for a fair comparison of the PPL values.

> > > Looks like this is what EleutherAI does. https://github.com/EleutherAI/lm_perplexity/blob/24c5a04beb2f073b2230c7c7420bd1ec7776f513/lm_perplexity/models.py#L43 @teetone The issue is that prepending `
` to the prompt does affect the logprob of the following words and might change the ranking of the words or the sampling results. Shall we always prepend the token to the prompt for all scenarios whether or not `request.echo_prompt` is `True` for the sake of consistency?
> > 
> > 
> > I'm seeing different log probabilities for the same token when `echo_prompt` is `True` vs `False`:
> > ![Screen Shot 2022-01-12 at 12 57 16 PM](https://user-images.githubusercontent.com/16793796/149224489-f4c36f42-dbf2-4c90-bbd7-144ea931b33d.png) ![Screen Shot 2022-01-12 at 12 57 51 PM](https://user-images.githubusercontent.com/16793796/149224506-68f1b187-d10d-4ed1-a406-1bc17ba7fbc9.png)
> 
> This does not sound reasonable to me. Why do you think echoing the prompt would affect the logprob?
> 
> Another issue is that whether we should also prepend the token to the input sentence when testing other LMs for a fair comparison of the PPL values.

I don't know. I was also expecting the same log probabilities. We need to ask in the OpenAI community forum.

> > > > Looks like this is what EleutherAI does. https://github.com/EleutherAI/lm_perplexity/blob/24c5a04beb2f073b2230c7c7420bd1ec7776f513/lm_perplexity/models.py#L43 @teetone The issue is that prepending `
` to the prompt does affect the logprob of the following words and might change the ranking of the words or the sampling results. Shall we always prepend the token to the prompt for all scenarios whether or not `request.echo_prompt` is `True` for the sake of consistency?
> > > 
> > > 
> > > I'm seeing different log probabilities for the same token when `echo_prompt` is `True` vs `False`:
> > > ![Screen Shot 2022-01-12 at 12 57 16 PM](https://user-images.githubusercontent.com/16793796/149224489-f4c36f42-dbf2-4c90-bbd7-144ea931b33d.png) ![Screen Shot 2022-01-12 at 12 57 51 PM](https://user-images.githubusercontent.com/16793796/149224506-68f1b187-d10d-4ed1-a406-1bc17ba7fbc9.png)
> > 
> > 
> > This does not sound reasonable to me. Why do you think echoing the prompt would affect the logprob?
> > Another issue is that whether we should also prepend the token to the input sentence when testing other LMs for a fair comparison of the PPL values.
> 
> I don't know. I was also expecting the same log probabilities. We need to ask in the OpenAI community forum.

I followed up here: https://community.openai.com/t/difference-in-token-log-probabilities-when-echo-is-true-vs-false/14541