Repository: stanford-crfm/helm
Issue: Remaining `TogetherClient` offline evaluation issues
Root cause: Validation Gap

# Remaining Issues

- [x] T0pp: exceed max output length since the value is so small
- [x] Models do not do well with prompts that end with space
- [x] OPT-66B: implement tokenizer
- [x] GPT-J: Missing results for 14 RunSpecs
- [x] GPT-NeoX: BoolQ
- [x] GPT-NeoX: [calibration issue](https://github.com/stanford-crfm/benchmarking/issues/629#issuecomment-1211367276). Also failing for other models: https://github.com/stanford-crfm/benchmarking/issues/724
- [x] Support `stop` sequences and early stopping
- [x] `token_logprobs=[]` and `top_logprobs=None` in the response when `echo=True` and `max_tokens=0`. Is `echo=True` supported?
- [x] No results for `entity_matching:model=text,dataset=Abt_Buy,data_augmentation=canonical` (GPT-NeoX). Most likely our fault because I don't see a scenario_state.json file.
- [x] `line` included in the request cache key ([PR to remove “line”.](https://github.com/stanford-crfm/benchmarking/pull/628))
- [x] together `together/gpt-neox-20b`: The first token is not included in the response at all, e.g., language modeling. (followed up)
- [x] Can't encode ThePile for the encoder-decoder Together models. echo=True is not supported for encoder-decoder models.

@teetone Is https://github.com/stanford-crfm/benchmarking/blob/main/scripts/together/together_import_results.py#L42 related? (Context: Trying to resolve any TODOs in codebase for #638). 

> @teetone Is https://github.com/stanford-crfm/benchmarking/blob/main/scripts/together/together_import_results.py#L42 related? (Context: Trying to resolve any TODOs in codebase for #638).

It's related. I will see if they got rid of it for the next batch of requests.

We're getting results for GPT-NeoX on language modeling tasks:

The Pile (ArXiv):

```
      MetricName(name='logprob', k=None, split='test', sub_split=None, perturbation=None)[min=-2516.910, mean=-2516.910, max=-2516.910, sum=-2516.910 (1)]
      MetricName(name='num_perplexity_tokens', k=None, split='test', sub_split=None, perturbation=None)[min=1922.686, mean=1922.686, max=1922.686, sum=1922.686 (1)]
      MetricName(name='num_bytes', k=None, split='test', sub_split=None, perturbation=None)[min=7215.856, mean=7215.856, max=7215.856, sum=7215.856 (1)]
      MetricName(name='num_output_tokens', k=None, split='test', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      MetricName(name='inference_runtime', k=None, split='test', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      MetricName(name='inference_idealized_runtime', k=None, split='test', sub_split=None, perturbation=None)[(0)]
      MetricName(name='inference_runtime_discrepancy', k=None, split='test', sub_split=None, perturbation=None)[(0)]
      MetricName(name='training_co2_cost', k=None, split='test', sub_split=None, perturbation=None)[(0)]
      MetricName(name='finish_reason_length', k=None, split='test', sub_split=None, perturbation=None)[min=1.000, mean=1.000, max=1.000, sum=1.000 (1)]
      MetricName(name='finish_reason_stop', k=None, split='test', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      MetricName(name='finish_reason_endoftext', k=None, split='test', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      MetricName(name='finish_reason_unknown', k=None, split='test', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      MetricName(name='num_in_context_examples', k=None, split='test', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      MetricName(name='perplexity', k=None, split='test', sub_split=None, perturbation=None)[min=3.703, mean=3.703, max=3.703, sum=3.703 (1)]
      MetricName(name='bits_per_byte', k=None, split='test', sub_split=None, perturbation=None)[min=0.503, mean=0.503, max=0.503, sum=0.503 (1)]
      MetricName(name='logprob_per_byte', k=None, split='test', sub_split=None, perturbation=None)[min=-0.349, mean=-0.349, max=-0.349, sum=-0.349 (1)]
      MetricName(name='estimated_num_tokens_cost', k=None, split=None, sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1954)]
      MetricName(name='num_prompt_tokens', k=None, split=None, sub_split=None, perturbation=None)[min=19.000, mean=2040.215, max=2049.000, sum=3986581.000 (1954)]
      MetricName(name='max_num_output_tokens', k=None, split=None, sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1954)]
      MetricName(name='num_requests', k=None, split=None, sub_split=None, perturbation=None)[min=1954.000, mean=1954.000, max=1954.000, sum=1954.000 (1)]
```

Getting results for Entity Matching (Abt Buy):

![Screen Shot 2022-08-10 at 3 46 09 PM](https://user-images.githubusercontent.com/16793796/184034470-bd7ba9f2-28a1-4444-bc04-30335ebb0015.png)


Calibration + GPT-NeoX issue:

```
  Error when running commonsense:model=full_functionality_text,dataset=hellaswag,method=multiple_choice_separate_calibrated:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 126, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 75, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 89, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 126, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metric.py", line 103, in evaluate
    self.evaluate_references(adapter_spec, request_states, metric_service, eval_cache_path)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/basic_metrics.py", line 620, in evaluate_references
    reference_stats[reference_key] = compute_logprob_and_length(request_state)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/basic_metrics.py", line 598, in compute_logprob_and_length
    assert (
AssertionError
```