Repository: stanford-crfm/helm
Issue: Duplicate entries in per instance stats
Root cause: Algorithmic Error

I am expecting a single entry in `per_instance_stats_slim.json` for each `(instance_id, perturbation, train_trail_index)`. Instead I am seeing lots of duplicates, some with no stats listed. Reproduction:

Command:
```
curl -s https://storage.googleapis.com/crfm-helm-public/benchmark_output/runs/v1.0/boolq%3Amodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical/per_instance_stats_slim.json | jq '.[] | select(.instance_id == "id11746" and .perturbation == null and .train_trial_index == 0)
```

Output:
```
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": [
    {
      "name": {
        "name": "num_train_trials"
      },
      "mean": 3
    },
    {
      "name": {
        "name": "num_references"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "max_prob"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "exact_match"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "quasi_exact_match"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "prefix_exact_match"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "quasi_prefix_exact_match"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "logprob"
      },
      "mean": 0
    },
    {
      "name": {
        "name": "num_perplexity_tokens"
      },
      "mean": 2
    },
    {
      "name": {
        "name": "num_bytes"
      },
      "mean": 3
    },
    {
      "name": {
        "name": "num_prompt_tokens"
      },
      "mean": 930
    },
    {
      "name": {
        "name": "num_output_tokens"
      },
      "mean": 2
    },
    {
      "name": {
        "name": "inference_runtime"
      },
      "mean": 0.6877071857452393
    },
    {
      "name": {
        "name": "batch_size"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "training_co2_cost"
      }
    },
    {
      "name": {
        "name": "training_energy_cost"
      }
    },
    {
      "name": {
        "name": "inference_denoised_runtime"
      },
      "mean": 0.7194980468750001
    },
    {
      "name": {
        "name": "inference_idealized_runtime"
      },
      "mean": 0.35753124999999997
    },
    {
      "name": {
        "name": "finish_reason_length"
      },
      "mean": 0
    },
    {
      "name": {
        "name": "finish_reason_stop"
      },
      "mean": 0
    },
    {
      "name": {
        "name": "finish_reason_endoftext"
      },
      "mean": 0
    },
    {
      "name": {
        "name": "finish_reason_unknown"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "num_train_instances"
      },
      "mean": 5
    },
    {
      "name": {
        "name": "prompt_truncated"
      },
      "mean": 0
    }
  ]
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": []
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": []
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": []
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": []
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": []
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": []
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": [
    {
      "name": {
        "name": "expected_max_toxicity"
      },
      "mean": 0.019158829
    },
    {
      "name": {
        "name": "max_toxicity_probability"
      },
      "mean": 0
    },
    {
      "name": {
        "name": "toxic_frac"
      },
      "mean": 0
    },
    {
      "name": {
        "name": "num_completions"
      },
      "mean": 1
    }
  ]
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": [
    {
      "name": {
        "name": "estimated_num_tokens_cost"
      },
      "mean": 5
    },
    {
      "name": {
        "name": "num_prompt_tokens"
      },
      "mean": 930
    },
    {
      "name": {
        "name": "num_completions"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "max_num_output_tokens"
      },
      "mean": 5
    },
    {
      "name": {
        "name": "num_completion_tokens"
      },
      "mean": 2
    }
  ]
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": [
    {
      "name": {
        "name": "estimated_num_tokens_cost"
      },
      "mean": 5
    },
    {
      "name": {
        "name": "num_prompt_tokens"
      },
      "mean": 1022
    },
    {
      "name": {
        "name": "num_completions"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "max_num_output_tokens"
      },
      "mean": 5
    },
    {
      "name": {
        "name": "num_completion_tokens"
      },
      "mean": 3
    }
  ]
}
{
  "instance_id": "id11746",
  "train_trial_index": 0,
  "stats": [
    {
      "name": {
        "name": "estimated_num_tokens_cost"
      },
      "mean": 5
    },
    {
      "name": {
        "name": "num_prompt_tokens"
      },
      "mean": 952
    },
    {
      "name": {
        "name": "num_completions"
      },
      "mean": 1
    },
    {
      "name": {
        "name": "max_num_output_tokens"
      },
      "mean": 5
    },
    {
      "name": {
        "name": "num_completion_tokens"
      },
      "mean": 2
    }
  ]
}
```

Figured it out. For some reason, the perturbation gets stripped from the `estimated_num_token_cost` entries.

This also causes #1206. The should be three `estimated_num_token_cost`: the original, and two with a different perturbation. Since the perturbation is lost, it just shows up three times under the original instead, which makes the frontend JavaScript log a warning.

Apparently this was fixed at some point? I'm not getting this error message for runs generated using current `main`.

Edit: Never mind, I managed to reproduce it.

Still broken. It's also annoying that the perturbation needs to be stored in two places; once in [`PerInstanceStats.perturbation`](https://github.com/stanford-crfm/helm/blob/56068d6f0069ebd5394ed89e7dd69b1cb1e10380/src/helm/benchmark/metrics/metric.py#L33) and once in [`InstanceName.perturbation`](https://github.com/stanford-crfm/helm/blob/106113fc7fbd63e50404965ffdec1ed52a45586c/src/helm/benchmark/metrics/metric_name.py#L21)