Repository: stanford-crfm/helm
Issue: Perturbation=None metric that disagrees with identity perturbation
Root cause: Configuration Error

A `perturbation=None` metric appears in the `metrics.txt` file and I have no idea where it's coming from. The logic in `data_preprocessor` and `generate` should only produce `perturbation=...identity..` and perturbed inputs.

Moreover, the value of this metric is often different than the identity perturbation which seems quite strange.

Example: Running `venv/bin/benchmark-run -r boolq:data_augmentation=canonical,model=ai21/j1-grande -m 50` gives the following metrics in `metrics.txt`:
```
MetricName(name='exact_match', k=None, split='valid', sub_split=None, perturbation=None)[min=0.756, mean=0.756, max=0.756, sum=0.756 (1)]
MetricName(name='exact_match', k=None, split='valid', sub_split=None, perturbation=PerturbationDescription(name='identity', robustness=False, fairness=False))[min=0.800, mean=0.800, max=0.800, sum=0.800 (1)]
...
```

It would be good to clear this out sooner than later for the UI code. Part of the issue seems to originate from how `metric.py` initializes `MetricName`s. It can be easier to just change the default value of the `perturbation` field of `MetricName`. I can make a PR in a bit.

Ok, I think we figured out a bunch of stuff with @dilarasoylu. The main issue is that there are multiple places in the codebase that do not properly propagate/handle MetricName attributes. E.g.:
- We are producing metrics with `perturbation=None` here: https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/basic_metrics.py#L368
- We are computing perplexity while ignoring everything except the split here https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/metric.py#L165

(there are also other points where this happens)

In particular, this causes a bunch of metrics to be incorrect when perturbed inputs are there. For instance, a metric called `MetricName(name='perplexity', k=None, split='valid', sub_split=None, perturbation=None)` is produced that is grouping together perturbed and unperturbed inputs in some undefined manner. Thus, in scenarios where we evaluate perturbations, **certain metrics might be incorrect**.

The solution is to make sure that we keep `MetricName` attributes unchanged whenever we deal with them.

Unfortunately, I don't know enough about this part of the codebase to tackle it, but happy to chat and explain what I know.

Moving this to P1 until we decide how much we care.

The code snippet in the first bullet point is:
```
                perturbation: Optional[PerturbationDescription] = request_state.instance.perturbation
                reference_metrics.extend(
                    compute_metrics_helper(MetricName(metric_name, perturbation), metric_fn_mapping[metric_name])
                )

                perturbation: Optional[PerturbationDescription] = request_state.instance.perturbation
                if perturbation:
                    reference_metrics.extend(
                        compute_metrics_helper(
                            MetricName(metric_name, perturbation=perturbation), metric_fn_mapping[metric_name]
                        )
                    )
```
When we change it to be as follows:
```
                perturbation: Optional[PerturbationDescription] = request_state.instance.perturbation
                reference_metrics.extend(
                    compute_metrics_helper(MetricName(metric_name, perturbation=perturbation), metric_fn_mapping[metric_name])
                )
```
We get rid of some of the None perturbations, and left with these:
```
      MetricName(name='logprob', k=None, split='valid', sub_split=None, perturbation=None)[min=-0.094, mean=-0.094, max=-0.094, sum=-0.094 (1)]
      MetricName(name='num_perplexity_tokens', k=None, split='valid', sub_split=None, perturbation=None)[min=1.000, mean=1.000, max=1.000, sum=1.000 (1)]
      MetricName(name='num_bytes', k=None, split='valid', sub_split=None, perturbation=None)[min=3.605, mean=3.605, max=3.605, sum=3.605 (1)]
      MetricName(name='num_output_tokens', k=None, split='valid', sub_split=None, perturbation=None)[min=1.000, mean=1.000, max=1.000, sum=1.000 (1)]
      MetricName(name='inference_runtime', k=None, split='valid', sub_split=None, perturbation=None)[min=1.470, mean=1.470, max=1.470, sum=1.470 (1)]
      MetricName(name='inference_idealized_runtime', k=None, split='valid', sub_split=None, perturbation=None)[(0)]
      MetricName(name='inference_runtime_discrepancy', k=None, split='valid', sub_split=None, perturbation=None)[(0)]
      MetricName(name='training_co2_cost', k=None, split='valid', sub_split=None, perturbation=None)[(0)]
      MetricName(name='finish_reason_length', k=None, split='valid', sub_split=None, perturbation=None)[min=1.000, mean=1.000, max=1.000, sum=1.000 (1)]
      MetricName(name='finish_reason_stop', k=None, split='valid', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      MetricName(name='finish_reason_endoftext', k=None, split='valid', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      MetricName(name='finish_reason_unknown', k=None, split='valid', sub_split=None, perturbation=None)[min=0.000, mean=0.000, max=0.000, sum=0.000 (1)]
      
      MetricName(name='perplexity', k=None, split='valid', sub_split=None, perturbation=None)[min=1.099, mean=1.099, max=1.099, sum=1.099 (1)]
      MetricName(name='bits_per_byte', k=None, split='valid', sub_split=None, perturbation=None)[min=0.038, mean=0.038, max=0.038, sum=0.038 (1)]
      MetricName(name='logprob_per_byte', k=None, split='valid', sub_split=None, perturbation=None)[min=-0.026, mean=-0.026, max=-0.026, sum=-0.026 (1)]
      MetricName(name='estimated_num_tokens_cost', k=None, split=None, sub_split=None, perturbation=None)[min=1.000, mean=1.000, max=1.000, sum=43.000 (43)]
      MetricName(name='num_prompt_tokens', k=None, split=None, sub_split=None, perturbation=None)[min=898.000, mean=969.558, max=1185.000, sum=41691.000 (43)]
      MetricName(name='max_num_output_tokens', k=None, split=None, sub_split=None, perturbation=None)[min=1.000, mean=1.000, max=1.000, sum=43.000 (43)]
      MetricName(name='num_requests', k=None, split=None, sub_split=None, perturbation=None)[min=43.000, mean=43.000, max=43.000, sum=43.000 (1)]
```
The worry with the None perturbations is them being aggregate of metrics computed on instances with different splits, subsets, and perturbations; as seems to be the case in the second bullet point @dtsip linked.

Slightly related, it may also be good to always specify a value for `k`, rather than going from `None` to `1` (if I understand correctly)



@dilarasoylu the snippets are flipped?

Also, as you can see we miss metrics like perplexity.

Just to clarify, the code metrics for such scenarios (e.g., `exact_match`) are fine, things are weird mostly with the auxiliary info.

Oops, thanks for the heads up, they are in the correct order now. I have also added the part of the output that has the perplexity.

Agreed with the clarification - though the places where we saw metrics dealing with `k` etc. on their own may require taking a look.

What are the next steps for this?

Ideally, we want to fix the metric processing pipeline to properly propagate metric names.

Concretely, every piece of code that creates a new MetricName should either use `replace` to propagate an existing metric name (should solve most problems) or if that is not possible, make sure to populate all the relevant attributes (`split`, `perturbation`, etc.).

Maybe this is the easiest to do for the folks who actually implemented the metric logic (not sure if there are core infrastructure things to fix). 

(@rishibommasani not sure I have the capacity/understanding to actually address this issue, but happy to be a placeholder assignee till we figure it out)

The main issue seems to be that, as we added on to the `MetricName` class, some parts of the code-base that were working fine when they were implemented became outdated. To avoid this in the future, we should avoid directly instantiating `MetricName` fields as much as possible. Shared below are (possibly) some concrete steps that we can take to address the issue. This is based on my best judgment - I may very well be missing some important points, please feel free to make other suggestions and/or correct me!

* [x] **[Done] Getting rid of the duplicate stats with `None` perturbations:** [This](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/basic_metrics.py#L368) line is causing duplicate stats with `None` perturbations to be produced. The issue is addressed in #632.
* [x] **[PR Submitted] Updating how metric names are assigned in `evaluate_language_modelling` function:** This is where the `perplexity` is computed, as mentioned by @dtsip in an earlier comment. We should ensure that all the fields are passed properly to the newly created metric names [here](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/metric.py#L165). This is a quick fix. Addressed in #680.
* [x] **[PR Submitted] Post processing the return values of `evaluate_generation` and `evaluate_references` properly:** We want to ensure that all the properties of an instance (not just the `split`) are passed to the metric in the `evaluate` function of `metric.py` at this [line](https://github.com/stanford-crfm/benchmarking/blob/dbf3355af89c48996db63f6905deb5beada51cf6/src/benchmark/metric.py#L116). In case the fields of the `MetricName` class change again, we should not count on manually assigning the field values. Another quick fix. Addressed in #680.
* [x] **Ensuring that metric name comparisons account for all the metric name fields:** This is a follow up issue to the one immediately above, where the issue was caused by us creating a new `MetricName` to check the presence of other `MetricNames` with the same specifications in a list. This is likely an issue that is repeated in other parts of the code-base, so we should ideally scan for other such cases. We can possibly extend the `MetricName` class with a custom comparison method. Not a quick fix / requires triage.
* [x] **Refactor `evaluate`:** We should look through all the metrics and re-factor them as needed. Not a quick fix / requires triage.
* [x] **Look into all the instantiations of the `MetricName` class:** As a final step, we could look into all the instantiations of the `MetricName` class in the code-base and its used correctly.

I think that makes sense. Will start with the easy things and see how far we can go before requiring major restructuring.

Ok, so after #680 a lot of these should be fixed. I tested on BoolQ and thePile, and I'm quite happy with how metrics.txt looks like.

I took a pass over all the `src/*_metric*` files. The metrics that still override `evaluate` are
- [x] Bias (#700 should fix it) and BBQ (#699 should fix) which will be handled by @dilarasoylu
- [x] Edit by Dilara: ToxicityMetric seems to be editing the `sub_split` information, handled in #725.
- [x] TokenMetric (tokens_metric.py) which imho is fine. It computes coarse token-accounting metrics that we probably want to compute over all the Instances instead of grouping by context (e.g., split and perturbation). 
- [x] `MultipleRequestMetrics` (multiple_request_metrics.py), not super familiar with this part of the code but, according to @dilarasoylu , will be fixed when #613 is fixed.

If this makes sense, I think we can go ahead and close this.

From the list I made above, only partial item remaining is:
```
Disallow overriding evaluate: Now that we have pushed the multiple instance metric logic to the adapter side, there isn't any custom metric that requires having access to multiple instances at once, as far as I know. We should sift through all the metrics and re-factor them as needed. Not a quick fix / requires triage.
```
We will address all the special metrics after the issues linked above, but we didn't add logic to disallow overwriting evaluate.

Later Edit: It seems like the `TokenMetric` is using `evaluate`, so if we decide to leave it as is, we shouldn't disallow overriding evaluate.

Ok, that makes sense then. Since we want to leave `TokenMetric` as is we should not explicitly disallow. Closing 🔥