Repository: stanford-crfm/helm
Issue: copyright_metrics: ZeroDivisionError: division by zero
Root cause: Validation Gap

`ai21/j1-jumbo`:

```
  Error when running copyright:model=text,datatag=n_books_1000-extractions_per_book_1-prefix_length_125:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 98, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 63, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 92, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 143, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 141, in evaluate
    results: List[List[Stat]] = parallel_map(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/general.py", line 177, in parallel_map
    if parallelism == 1:
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 70, in process
    self.metric.evaluate_generation(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/copyright_metrics.py", line 121, in evaluate_generation
    result = self.metric_fn(completion_tokens, truncated_reference_tokens, previous_best=result)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/copyright_metrics.py", line 56, in _edit_similarity
    esim = 1.0 - edist / max(len(s1), len(s2))
ZeroDivisionError: division by zero
```

`openai/text-davinci-002`:

```
  Error when running copyright:model=text,datatag=n_books_1000-extractions_per_book_1-prefix_length_125:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 98, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 63, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 92, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 143, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 141, in evaluate
    results: List[List[Stat]] = parallel_map(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/general.py", line 177, in parallel_map
    if parallelism == 1:
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 70, in process
    self.metric.evaluate_generation(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/copyright_metrics.py", line 121, in evaluate_generation
    result = self.metric_fn(completion_tokens, truncated_reference_tokens, previous_best=result)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/copyright_metrics.py", line 56, in _edit_similarity
    esim = 1.0 - edist / max(len(s1), len(s2))
ZeroDivisionError: division by zero
```