Repository: stanford-crfm/helm
Issue: `commonsense:model=full_functionality_text,dataset=hellaswag,method=multiple_choice_separate_calibrated` failing in `BasicMetric`
Root cause: Validation Gap

Failing for multiple models (e.g., `openai/text-curie-001`):

```
  Error when running commonsense:model=full_functionality_text,dataset=hellaswag,method=multiple_choice_separate_calibrated:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 126, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 75, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 89, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 126, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metric.py", line 103, in evaluate
    self.evaluate_references(adapter_spec, request_states, metric_service, eval_cache_path)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/basic_metrics.py", line 620, in evaluate_references
    reference_stats[reference_key] = compute_logprob_and_length(request_state)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/basic_metrics.py", line 598, in compute_logprob_and_length
    assert (
AssertionError
```

The same check is also failing for `blimp:model=full_functionality_text,phenomenon=quantifiers`

Can you provide a script that reproduces this error and the Git ID you are using? It seems that line 598 of the current master branch is not an assertion.

> Can you provide a script that reproduces this error and the Git ID you are using? It seems that line 598 of the current master branch is not an assertion.

@yuhui-zh15, you should be able to reproduce the error with the `main` branch and `max_eval_instances` set to 1000. It will fail here: https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/basic_metrics.py#L621

Also, `commonsense:model=full_functionality_text,dataset=hellaswag,method=multiple_choice_separate_calibrated` failed for `ai21/j1-large`, so maybe try AI21 instead  since we have less OpenAI credits.

More informative error:

```
  Error when running commonsense:model=full_functionality_text,dataset=hellaswag,method=multiple_choice_separate_calibrated:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 126, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 75, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 89, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 126, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metric.py", line 100, in evaluate
    self.evaluate_references(adapter_spec, request_states, metric_service, eval_cache_path)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/basic_metrics.py", line 641, in evaluate_references
    reference_stats[reference_key] = compute_logprob_and_length(request_state)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/basic_metrics.py", line 621, in compute_logprob_and_length
    assert span == reference, f"Expected: {reference}, Actual: {span}"
AssertionError: Expected: It is important to select the color that represents your hair type when you register your hair color. [substeps] Traditional semi-permanent dyes will generally not be available for hair color, like blow-dryers, curling irons, and appliances., Actual: ItĠisĠimportantĠtoĠselectĠtheĠcolorĠthatĠrepresentsĠyourĠhairĠtypeĠwhenĠyouĠregisterĠyourĠhairĠcolor.Ġ[substeps]ĠTraditionalĠsemi-permanentĠdyesĠwillĠgenerallyĠnotĠbeĠavailableĠforĠhairĠcolor,ĠlikeĠblow-dryers,ĠcurlingĠirons,ĠandĠappliances.
```

So, we need to detokenize. @yuhui-zh15, thinking about this a bit more, do we need this assertion?

Thanks a lot for the detailed errors @teetone! This is exactly what I expected.

This assertion is used for my debugging purpose during the development, so please feel free to remove it.

Another possibility is to weaken the comparison, could you try if the following codes work?

```python
printables = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
filtered_span = ''.join(list(filter(lambda x: x in printables, span)))
filtered_reference = ''.join(list(filter(lambda x: x in printables, reference)))
assert filtered_span == filtered_reference, f"Expected: {filtered_reference}, Actual: {filtered_span}"
```

Either way is fine, and should resolve this issue.

> Thanks a lot for the detailed errors @teetone! This is exactly what I expected.
> 
> This assertion is used for my debugging purpose during the development, so please feel free to remove it.
> 
> Another possibility is to weaken the comparison, could you try if the following codes work?
> 
> ```python
> printables = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
> filtered_span = ''.join(list(filter(lambda x: x in printables, span)))
> filtered_reference = ''.join(list(filter(lambda x: x in printables, reference)))
> assert filtered_span == filtered_reference, f"Expected: {filtered_reference}, Actual: {filtered_span}"
> ```
> 
> Either way is fine, and should resolve this issue.

@yuhui-zh15 Thank you, I think that should work. Can you take a look at this PR: https://github.com/stanford-crfm/benchmarking/pull/733?