Repository: stanford-crfm/helm
Issue: Divide by zero error when calculating perplexity
Root cause: Validation Gap

```
Traceback (most recent call last):
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/bin/benchmark-present", line 33, in <module>
    sys.exit(load_entry_point('crfm-benchmarking', 'console_scripts', 'benchmark-present')())
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 153, in main
    runner.run()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 90, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 46, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 63, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 103, in run_one
    metric_result: MetricResult = metric.evaluate(scenario_state, self.metric_service)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metric.py", line 122, in evaluate
    -trial_stats[MetricName("logprob", split=split)].sum
ZeroDivisionError: division by zero
```

@teetone What is the command you ran?

@teetone @YianZhang Can I/someone close this?

@rishibommasani Probably not yet.

@teetone After a round of big run, I found that this problem seems to be specific to the `Wikifact` scenario. To reproduce the error, remove the [if statement](https://github.com/stanford-crfm/benchmarking/blob/070bba2b574020f410d5e509af62285359df2682/src/benchmark/metric.py#L138) and run the following:

` venv/bin/benchmark-run -r wikifact:k=5,subject=P19,model=openai/text-davinci-001 -m 10`

If you run the command, you will find that not only `logprobs`, `num_bytes`, and `num_perplexity_tokens` but also most other metrics are all `0` for this scenario. The model outputs are empty. e.g. 
`[split=test, sub_split=None, id=id509, perturbation=None] "Rick Ravanello was born in " => "Cape Breton", predicted "" [WRONG]`
`[split=valid, sub_split=None, id=id131, perturbation=None] "Gaius Servilius Geminus was born in " => "Ancient Rome", predicted "" [WRONG]`

The error does not seem to be model-specific. Running the following list of models on Wikifact all lead to this error:  `openai/davinci`, `openai/text-davinci-002`, `openai/text-curie-001`, `ai21/j1-jumbo`

@michiyasunaga Can you check why `Wikifact` causes this error? 

Hi folks, thanks for looking into this issue. Just to better understand, is the root cause that the model predicts empty string as the output (e.g. the model just generates "\n" right after the prompt)? If so, we can work on fixing this – e.g. make sure to return a non-empty answer among the top K predictions.

Hi @michiyasunaga, is it normal for a model to predict an empty string in Wikifact? If so, why would it? If you run the command, you will notice that it **always** predicts an empty string and `exact_match=0`.

This empty string phenomenon seems to occur to GPT-3 models (https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=wikifact%3Ak%3D5%2Csubject%3DP31%2Cmodel%3Dopenai_text-curie-001) but not to other models (https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=wikifact%3Ak%3D5%2Csubject%3DP31%2Cmodel%3Dai21_j1-jumbo). It seems that for some reason GPT-3 just generates "\n" right after the prompt, so the prediction becomes an empty string. @hyren is kindly looking into this issue. One solution might be to always filter out an empty string prediction and take the next top K predictions generated by models.

To get this resolved, I think we just keep the current fix and assume models shouldn't be immediately outputting a stop sequence (and if they do, they are wrong anyways). And not try to do anything fancier with multiple predictions.  

And @YianZhang what is the behavior in terms of LM-type metrics when this happens? Obviously the models will get no accuracy/etc. when they output a stop sequence, so that's fine. But just want to make sure the behavior is reasonable (e.g. will this end up yielding a ppl of 0 that artificially lowers the ppl of outputs?)

@YianZhang @michiyasunaga Does this sound good? If it sounds good, can you close the issue and thanks for the help on this!