Repository: stanford-crfm/helm
Issue: `num_prompt_tokens` > `num_output_tokens` for language modeling scenarios
Root cause: Validation Gap

## together/opt-175b


```
  Error when running the_pile:model=full_functionality_text,subset=Enron Emails:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 98, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 60, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 90, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 129, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 122, in evaluate
    return self.evaluate_language_modeling(scenario_state, metric_service, eval_cache_path)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 260, in evaluate_language_modeling
    request_stats = self.evaluate_generation(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/basic_metrics.py", line 618, in evaluate_generation
    metrics.extend(self.compute_efficiency_metrics(adapter_spec, request_state, metric_service))
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/basic_metrics.py", line 504, in compute_efficiency_metrics
    assert (
AssertionError: num_prompt_tokens (146) > num_output_tokens (57) for prompt: 
---------------------- Forwarded by Eric Bass/HOU/ECT on 03/13/2001 01:12 PM 
---------------------------
```

```
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 98, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 60, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 90, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 129, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 122, in evaluate
    return self.evaluate_language_modeling(scenario_state, metric_service, eval_cache_path)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 260, in evaluate_language_modeling
    request_stats = self.evaluate_generation(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/basic_metrics.py", line 618, in evaluate_generation
    metrics.extend(self.compute_efficiency_metrics(adapter_spec, request_state, metric_service))
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/basic_metrics.py", line 504, in compute_efficiency_metrics
    assert (
AssertionError: num_prompt_tokens (40) > num_output_tokens (15) for prompt: 
@iTweetBIG lolss \ud83d\ude02\ud83d\ude02\ud83d\ude02 yo cunta kintae  lookin ass !!
```

## together/bloom

```
  Error when running the_pile:model=full_functionality_text,subset=ArXiv:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 98, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 60, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 90, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 129, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 122, in evaluate
    return self.evaluate_language_modeling(scenario_state, metric_service, eval_cache_path)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 260, in evaluate_language_modeling
    request_stats = self.evaluate_generation(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/basic_metrics.py", line 618, in evaluate_generation
    metrics.extend(self.compute_efficiency_metrics(adapter_spec, request_state, metric_service))
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/basic_metrics.py", line 504, in compute_efficiency_metrics
    assert (
AssertionError: num_prompt_tokens (19) > num_output_tokens (11) for prompt: 
---
bibliography:
- 'bib.bib'
---

\#1



```

This is in the efficiency metric computation: https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/metrics/basic_metrics.py#L504. Any idea why this assertion would fail?

@LorrinWWW For the `bloom` and `OPT` models, the number of output tokens does not match the number of tokens in the original prompt when `echo=True`. Do you have any idea why that might be?

> @LorrinWWW For the `bloom` and `OPT` models, the number of output tokens does not match the number of tokens in the original prompt when `echo=True`. Do you have any idea why that might be?

Is this always the case or just a few examples, could you send me the samples where the assertion failed? I can re-run them once I find the reason.

> > @LorrinWWW For the `bloom` and `OPT` models, the number of output tokens does not match the number of tokens in the original prompt when `echo=True`. Do you have any idea why that might be?
> 
> Is this always the case or just a few examples, could you send me the samples where the assertion failed? I can re-run them once I find the reason.

I spot checked a few and it looks like it's not all of them. Here is one example for `together/opt-175b`:

<img width="1203" alt="Screen Shot 2022-09-03 at 7 57 03 PM" src="https://user-images.githubusercontent.com/16793796/188295076-599dec4d-62c6-4bd0-a726-0bd05518f7b6.png">


@teetone @LorrinWWW Just want to check in on the progress in resolving; also, for bookkeeping purposes, can we assign this issue to someone? 

I synced with @LorrinWWW, and for some models, the round trip is not the same. Here is an example using the tokenizer from HuggingFace outside of our implementations:

```python
prompt = 'Finance and Bizness: [header] How to wrk out salary increase percentage [title] Subtract yo old salary from ur new salary. [step] Say youu made $45 , 000 per year at yo old job and dat you accepted a new position makin $50 , 000 per year. Dis means yu would take $50 , 000 and subtract $45 , 000.  The remaining $70 , 000 would equal $100 , 000 in ur new salary. [substeps] If yu have less in ur previous salary den you did in ur current position, youu should subtract $100 , 000 from yo current salary and make da difference.'
tokenizer = AutoTokenizer.from_pretrained('EleutherAI/gpt-j-6B') # or other models
decoded_text = tokenizer.decode(tokenizer(prompt)['input_ids'])
assert len(prompt) == len(decoded_text)
```

We will see the warnings @deepakn94 put in place for certain prompts when `echo=True`. I'm unsure if there is anything we can do since it's the behavior of the model/tokenizer.

@teetone What's the plan for this then?

@teetone just checking in again wrt this

There's not much we can do. Closing this issue.