Repository: stanford-crfm/helm
Issue: Scenario instance caching
Root cause: Unimplemented Feature Gap

Currently, we generate and sample scenario instances from scratch for every run. Instance generation can be expensive; in particular, we usually load _all_ instances from a scenario into memory before down-sampling (this is a known problem, but out of scope for this issue).

An alternative would be to generate and sample instances just once per unique `ScenarioSpec`. We would then save these instances to a cache in a location such as `benchmark_output/instances`, the same way scenarios are saved to a cache under `benchmark_output/scenarios`. This would have the following advantages:

- We save a lot of time by not having to regenerate the instances in many situations, e.g.:
    - when we run `helm-run` with run specs with the same `ScenarioSpec` for different models
    - when we run `helm-run` with run specs with the same `ScenarioSpec` as past runs (e.g. when implementing new models)
    - when we have multiple parallel workers sharing a disk containing the cached instances (e.g. CodaLab)
- For developers adding new scenarios, they can view the instances during manual testing more easily without needing a full run.
- Minor: Currently each run has its own `instances.json`, which means that each version of the HELM evaluations contain many identical copies of the same `instances.json`. Eventually, we could upload the cached version instead and share it across multiple runs. However, this requires additional work.

We would also want a new flag that just writes the instances to disk and skips the rest of the run, or we could repurpose the existing `--dry-run` flag for this.

Note that we already generate a `instances.json` file, but this is reverse of this proposal; this is the _output_ rather than the _input_ to a model run. In particular, it is generated by `helm-summarize.

Open questions:

- What do we want to store? Should we store all instances, just sampled instances, or sampled instances along with data augmentations? 
    - Storing all instances would take up a lot of space, so we probably don't want to do this.
    - Storing sampled instances means that we would also need to add the value of the `AdapterSpec.max_train_instances` and `AdapterSpec.max_eval_instances` to the cache key
    - Storing sampled instances along with data augmentations means we also need to add the `PerturbationDescription` to the cache key. But this also makes the most sense because this is the finalized input to the model.
- How to index into the instances cache?
    - We could do something like `benchmark_output/instances/mmlu:subject=anatomy` e.g. for MMLU, but we would need to canonicalize the parameter order. Also we might need to add additional parameters such as `max_train_instances` and `max_eval_instances`. Also it's not clear how to serialize a `PerturbationDescription` into the path if it needs to be part of the cache key (it can be very big).
- Should we start introducing version numbers to scenarios?
    - We need to regenerate the instances cache if the scenario changes, so we don't use stale instances from the file.


cc @AndrewJGaut @percyliang this is one of the HELM on CodaLab improvements that we discussed.

Moved this into a public Google doc [here](https://docs.google.com/document/d/1KHw_1tcKy9ccJBwJITTVaD7aMVPa85x-vA5LkBbxch8/edit?usp=sharing)