Repository: stanford-crfm/helm
Issue: AI21 models return 0.0 logprobs (at least for BoolQ)
Root cause: External Dependency Breakage

When runnnig `venv/bin/benchmark-run -r boolq:data_augmentation=canonical,model=ai21/j1-grande -m 50` I get a `logprob` of 0.0 for all instances (which can't be right since it means probability of 1.0). 

(When I was running the same thing yesterday I was getting a reasonable logprob (-0.something) for the `identity` perturbation and 0.0 for everything else, presumably due to earlier caching.)

Don't currently have quota to try more models, so not sure whether this is a widespread issue.

@dtsip Is this fixed now? 

Looking at both an LM and non-LM scenario, and at least the log probs are not 0 for either:
[BoolQ](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=boolq%3Amodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical) 
[The Pile - arXiv](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=the_pile%3Asubset%3DArXiv%2Cmodel%3Dai21_j1-jumbo)



For BoolQ, the scenario you linked has `logprob=-0.004` which seems suspiciously high (I suspect there are a lot of 0.0s in that average), but I don't know how to access the `scenario_state.txt` from the website.

When running `venv/bin/benchmark-run -r boolq:data_augmentation=canonical,model=ai21/j1-grande -m 50` on 8/19 I got all `0.0`s again.

kk makes sense - ya i had the same concern, though wasn't sure since it was only a token, though I guess assigning a probability of .996 on average might be too high (i.e. exp(-0.004)); I don't have good intuitions on the how the distribution looks like for these  types of contexts.

But ya, looking at `scenario_state.txt` seems better to make sure it is right, cool

@dtsip Can you check this again now?

Looking at the website results for language modeling, the results all seem reasonable for all AI21 models and of similar order as other models:
[The Pile](https://crfm-models.stanford.edu/static/benchmarking.html?group=the_pile)
[TwitterAAE](https://crfm-models.stanford.edu/static/benchmarking.html?group=twitter_aae)
[ICE](https://crfm-models.stanford.edu/static/benchmarking.html?group=ice)

@dtsip I am closing this, but please verify

The BoolQ scenario on the website ([link](https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=boolq%3Amodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical#metrics)) still shows a logprob of -0.004, which is suspicious.

Running `venv/bin/benchmark-run -r boolq:data_augmentation=canonical,model=ai21/j1-grande -m 50` still gives me 0.0 logprobs everywhere. This means the issue is still there. So, at least for BoolQ, the logprobs we are displaying on the website **are very likely wrong**.

Also, due to the randomness change, I am getting some cache misses which also give a logprob of 0.0, indicating that this is not due to caching requests at a time when things were not working properly.

At the same time, things look good on QuAC (logprob -2.976), naturalQA (-3.478), MMLU (-1.086). So this might be a BoolQ issue?

@YianZhang can you look into this 

Ok, we can actually look at IMDB to see this problem more acutely:
https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=boolq%3Amodel%3Dai21_j1-large%2Cdata_augmentation%3Dcanonical&runSpecRegex=.*boolq.*ai21.*canonical.*.*

So the model accuracy and other stats (e.g. robustness, fairness) all seem sensible, but the probabilities are not sensible (I think this is an issue for very short generations, which is why we see it happening for imdb/boolq but not other QA scenarios dimitris wrote above). 

I am not sure why this is happening (@teetone any thoughts?). 

Bumping this up again: 

One thing that is kind of surprising is that if I use the interface to load  the cached responses for one of the [demo questions](https://crfm-models.stanford.edu/static/index.html?prompt=Please%20answer%20the%20following%20question%20about%20geography.%0A%0AWhat%20is%20the%20capital%20of%20Germany%3F%0AA.%20Berlin%0AB.%20Bonn%0AC.%20Hamburg%0AD.%20Munich%0AAnswer%3A%20A%0A%0AWhat%20is%20the%20capital%20of%20Canada%3F%0AA.%20Montreal%0AB.%20Ottawa%0AC.%20Toronto%0AD.%20Vancouver%0AAnswer%3A&settings=temperature%3A%200%0Amax_tokens%3A%201%0Atop_k_per_token%3A%204%0Amodel%3A%20%24%7Bmodel%7D%20%20%23%20Try%20out%20GPT-3%20and%20Jurassic&environments=model%3A%20%5Bopenai%2Fdavinci%2C%20ai21%2Fj1-jumbo%5D), the AI21 logprobs seem reasonable. But if I simply permute the option order and re-run, I am getting 0s. Does this mean that things were fine before? Has something changed with how we query the AI21 models?

The problem here is that AI21 no longer returns logprobs if temperature = 0.  To fix this, we should modify `AI21Client` to check if temperature = 0, and if so, issue something with temperature = 1e-4 (exact value to be determined by experimentation), and then scale the log probabilities appropriately (which we need to do in any case to make them consistent with other models).  This shouldn't be that bad.

> The problem here is that AI21 no longer returns logprobs if temperature = 0. To fix this, we should modify `AI21Client` to check if temperature = 0, and if so, issue something with temperature = 1e-4 (exact value to be determined by experimentation), and then scale the log probabilities appropriately (which we need to do in any case to make them consistent with other models). This shouldn't be that bad.

@percyliang I tried {1e-4, 1e-3, 1e-2 and 0.1} and only 0.1 gave me non-zero log probs. I don't think we can set the temperature to 0.1, though, because we could get different outputs. An alternative is to send two requests, where the second request has the original prompt concatenated with the completion and `max_tokens=0`. The tokenization might make this a bit tricky.

Hmm, sounds like it's a bit non-trivial.  I'd suggest we table this for the release and send AI21 email saying explicit what the problem is and that we're not doing this for now.  We should definitely fix it after the release.