Repository: stanford-crfm/helm
Issue:  Rename `num_tokens` (language modeling) and also update the entry in schema.yaml
Root cause: Documentation Deficiency

For the language modeling scenarios, there seem to be discrepancies in the reported number of tokens in different places (`estimated_number_of_tokens`, `num_tokens`, `num_tokens_in_prompt`):

For example, with `wikitext_103`:
```
    Stats {
      ...
      MetricName(name='num_tokens', k=None, split='test', sub_split=None, perturbation=None)[min=1703.696, mean=1703.696, max=1703.696, sum=1703.696 (1)]
      ...
      MetricName(name='num_tokens_in_prompt', k=None, split='test', sub_split=None, perturbation=None)[min=1971.762, mean=1971.762, max=1971.762, sum=1971.762 (1)]
      ...
      MetricName(name='estimated_number_of_tokens', k=None, split=None, sub_split=None, perturbation=None)[min=3943.524, mean=3943.524, max=3943.524, sum=3943.524 (1)]
    } [0.0s]
```

I believe these should all be the same value, especially for language modeling tasks where we are not doing any generation.

For MMLU, on the other hand, things seem a bit more sane:
```
    Stats {
     ...
      MetricName(name='num_tokens', k=None, split='valid', sub_split=None, perturbation=None)[min=1.000, mean=1.000, max=1.000, sum=1.000 (1)]
     ...
      MetricName(name='num_tokens_in_prompt', k=None, split='valid', sub_split=None, perturbation=None)[min=356.471, mean=356.471, max=356.471, sum=356.471 (1)]
      ...
      MetricName(name='estimated_number_of_tokens', k=None, split=None, sub_split=None, perturbation=None)[min=354.977, mean=354.977, max=354.977, sum=354.977 (1)]
    } [0.0s]
```

`num_tokens` is the number of tokens in the first completion minus `num_conditioning_tokens`, so in most cases for language modeling tasks it should not equal `num_tokens_in_prompt`. I have no idea what `estimated_number_of_tokens` is. Do you know who implemented this?

I believe @teetone implemented `estimated_number_of_tokens`.

> num_tokens is the number of tokens in the first completion minus num_conditioning_tokens, so in most cases for language modeling tasks it should not equal num_tokens_in_prompt.

What are the conditioning tokens?

You can ignore `estimated_number_of_tokens` for the paper. It's used to estimate token usage.

Out of curiosity, how is it computed? Just trying to understand the discrepancy.

> Out of curiosity, how is it computed? Just trying to understand the discrepancy.

AI21: https://github.com/stanford-crfm/benchmarking/blob/main/src/proxy/tokenizer/ai21_token_counter.py#L24
OpenAI: https://github.com/stanford-crfm/benchmarking/blob/main/src/proxy/tokenizer/openai_token_counter.py#L32

For all the other models, the `estimated_number_of_tokens` is 0 because it's essentially free for us.

I think we should make the semantics/naming a bit clearer since there are two relevant quantities and it seems like there are some recurring confusion:
1. tokens in terms of cost for us; different models have different tokens that are mostly model/organization-dependent rather than scenario-dependent
2. tokens in terms of length of inputs/outputs; different models have slightly different counts due to tokenization/etc. nuances but this should be mostly scenario-dependent and marginally model/organization-dependent. 

I agree. I think `estimated_number_of_tokens` addresses point 1. I am not 100% sure what `num_tokens` is, but `num_tokens_in_prompt` is intended to be the number of input tokens, and `num_output_tokens` is intended to be the number of output tokens.

@deepakn94 @YianZhang What are the TODOs for resolving this issue? 

I think my recommendation is to get rid of `num_tokens`. Then we'll have:
- `num_prompt_tokens`: the number of tokens in the prompt.
- `num_output_tokens`: the number of generated tokens.
- `estimated_num_tokens_cost`: for cost tracking purposes. We could even re-name this to make it clearer.

@teetone Will renaming/removing break anything wrt existing results, etc? If so, can you take (I know you have a ton of stuff, but it seems very light and probably best if you take since you will know what the dependencies are)? 

I renamed `num_tokens_in_prompt` and `estimated_number_of_tokens` in https://github.com/stanford-crfm/benchmarking/pull/499. I don't think we can get rid of `num_tokens` because it's used for language modeling metrics.

@YianZhang: could you rename `num_tokens` so it's a bit more descriptive and also update the entry in `schema.yaml`?