Repository: stanford-crfm/helm
Issue: Add discussion section.

In github if you navigate to https://github.com/stanford-crfm/helm/graphs/community there is a button that will let you set up discussions. It might make sense for to enable that as a place for user questions that can be separated from software issues.

The reason I suggest this is because I have a question, which is more about how to interpret HELM results than it is an actual issue with the HELM software. My question is:

I'm looking at downloaded HEIM outputs, and I see examples in the per_instance_spec.json in each `perturbation` I sometimes see a "modifications" field, but I don't see this in the corresponding dataclass: `helm.benchmark.augmentations.perturbation_description.PerturbationDescription`. I checked older versions of HELM to see if it was removed, but either I missed it or it never existed. I'm wondering how this was ever written to disk. The reason I'm interested is because I have to differentiate the stats in `per_instance_spec` based on the instance_id and the permutation of the input text instance. When I load the json with the dataclasses this "modifications" field is dropped, so when I hash the `PerturbationDescription` I end up getting collisions where I shouldn't.

Thanks for the suggestion regarding discussions; I'll consider it.

Do you have the name of a leaderboard run that has the additional `modifications` field? I can look into this.

It was the HEIM leaderbord both v1.0 and v1.1

Here are some notes I took about it:

E.g.:

In v1.0.0/mscoco:model=AlephAlpha_m-vader,prompt=pixel_art,max_eval_instances=100 I have:

/data/crfm-helm-public/heim/benchmark_output/runs/v1.0.0/mscoco:model=AlephAlpha_m-vader,prompt=pixel_art,max_eval_instances=100/run_spec.json


```json
{
  "name": "mscoco:model=AlephAlpha_m-vader,prompt=pixel_art,max_eval_instances=100",
  "scenario_spec": {
    "class_name": "helm.benchmark.scenarios.mscoco_scenario.MSCOCOScenario",
    "args": {}
  },
  "adapter_spec": {
    "method": "generation",
    "global_prefix": "",
    "instructions": "",
    "input_prefix": "",
    "input_suffix": "",
    "reference_prefix": "A. ",
    "reference_suffix": "\n",
    "output_prefix": "",
    "output_suffix": "",
    "instance_prefix": "\n",
    "substitutions": [],
    "max_train_instances": 0,
    "max_eval_instances": 100,
    "num_outputs": 1,
    "num_train_trials": 1,
    "num_random_trials": 1,
    "sample_train": true,
    "model": "AlephAlpha/m-vader",
    "temperature": 1,
    "max_tokens": 0,
    "stop_sequences": [],
    "modifications": [
      "pixel art"
    ]
  },
  ...
}
```

I don't see where the "modifications" field is being written anywhere in HELM.


Also in

v1.0.0/mscoco:model=AlephAlpha_m-vader,data_augmentation=art,max_eval_instances=17

/data/crfm-helm-public/heim/benchmark_output/runs/v1.0.0/mscoco:model=AlephAlpha_m-vader,data_augmentation=art,max_eval_instances=17/run_spec.json

I have:

```
{
  "name": "mscoco:model=AlephAlpha_m-vader,data_augmentation=art,max_eval_instances=17",
  "scenario_spec": {
    "class_name": "helm.benchmark.scenarios.mscoco_scenario.MSCOCOScenario",
    "args": {}
  },
  "adapter_spec": {
    "method": "generation",
    "global_prefix": "",
    "instructions": "",
    "input_prefix": "",
    "input_suffix": "",
    "reference_prefix": "A. ",
    "reference_suffix": "\n",
    "output_prefix": "",
    "output_suffix": "",
    "instance_prefix": "\n",
    "substitutions": [],
    "max_train_instances": 0,
    "max_eval_instances": 17,
    "num_outputs": 4,
    "num_train_trials": 1,
    "num_random_trials": 1,
    "sample_train": true,
    "model": "AlephAlpha/m-vader",
    "temperature": 1,
    "max_tokens": 0,
    "stop_sequences": []
  },
```

But there is no modifications in the adapter_spec, they are in data_augmenter_spec.perturbation_specs


```
  "data_augmenter_spec": {
    "perturbation_specs": [
      {
        "class_name": "helm.benchmark.augmentations.style_perturbation.StylePerturbation",
        "args": {
          "modifications": [
            "oil painting"
          ]
        }
      },
      {
        "class_name": "helm.benchmark.augmentations.style_perturbation.StylePerturbation",
        "args": {
          "modifications": [
            "watercolor"
          ]
        }
      },
      {
        "class_name": "helm.benchmark.augmentations.style_perturbation.StylePerturbation",
        "args": {
          "modifications": [
            "pencil sketch"
          ]
        }
      },
      {
        "class_name": "helm.benchmark.augmentations.style_perturbation.StylePerturbation",
        "args": {
          "modifications": [
            "animation"
          ]
        }
      },
      {
        "class_name": "helm.benchmark.augmentations.style_perturbation.StylePerturbation",
        "args": {
          "modifications": [
            "vector graphics"
          ]
        }
      },
      {
        "class_name": "helm.benchmark.augmentations.style_perturbation.StylePerturbation",
        "args": {
          "modifications": [
            "pixel art"
          ]
        }
      }
    ],
    "should_augment_train_instances": false,
    "should_include_original_train": true,
    "should_skip_unchanged_train": true,
    "should_augment_eval_instances": true,
    "should_include_original_eval": true,
    "should_skip_unchanged_eval": true,
    "seeds_per_instance": 1
  },
```



I think your explanation is probably correct - this existed in a branch that was never merged into main. 

@teetone might know, but I'm don't think they're active here currently.