Repository: stanford-crfm/helm
Issue: Plug-in architecture for custom scenarios and metrics
Root cause: Unimplemented Feature Gap

It would be nice to enable users to run `helm-run` using their own scenarios and metrics without having to fork the repository.

Currently this requires the user to fork the repository and add their own scenarios and metrics. They would then register a new run spec function in `run_specs.py`. This has the following disadvantages:

- Packaging: the user would have to maintain a forked repository and merge changes from upstream, rather than simply installing updated packages.
- Ease of use: adding a new scenario or metric is a [complicated multi-step process](https://github.com/stanford-crfm/helm/blob/2a59d7624f17481206bf8d0c1be0415338bbf63a/docs/code.md#adding-new-scenarios) that requires understanding and modifying existing code.
- Licensing encumbrance: the forked repository would be considered a revision and derivative work of the original, which would trigger licensing requirements; this would make HELM unusable in some corporate environments.
- Research code publication: to publish code for a paper, the user would either have to republish their entire fork, or they would have to work with us to merge their fork upstream. Republishing their entire fork makes it hard to understand by paper readers.
    - Merging: Generally, merging new scenarios is slow because we're busy and this is usually lower-priority. Also we may reject merge requests that are not generally useful enough, or take on too many additional dependencies.

Ideally, a user would just `pip install crfm-helm`, and then write their own module that contains their own scenarios and metrics.

We already use `ObjectSpec` for scenarios (`ScenarioSpec`) and metrics (`MetricSpec`), and `ObjectSpec` allows HELM to load any scenario or metric that is discoverable by Python. Therefore, the user can define their own module that contains their custom scenario or metric classes, and these scenarios and metrics would be usable by setting the class name in `ObjectSpec` to the user's classes.

The problem is that currently there is no way to run a `RunSpec` containing the `ObjectSpec` with the custom classes, so we need an API for that. My suggestion would be to have a `@run_spec_function` decorator, which will be used as such:

```python
@run_spec_function("myscenario")
def get_myscenario_spec(myparameter) -> RunSpec:
    scenario_spec = ScenarioSpec(
        class_name="mymodule.MyScenario",
        args={"myparameter": myparameter},
    )
    metric_specs = [MetricSpec(class_name="mymodule.MyMetric")]
    return RunSpec(
        name=f"myscenario:myparameter={myparameter}",
        scenario_spec=scenario_spec,
        adapter_spec=get_generation_adapter_spec(),
        metric_specs=metric_specs,
        groups=["myscenario"],
    )
```

Then a user could run `helm-run --local -m 10 --suite custom --run-specs myscenario:myparameter=42,model=openai/text-davinci-002 --run-spec-function-modules my_run_spec_functions`, where `my_run_spec_functions` is the module containing `get_myscenario_spec`.


cc @percyliang @teetone

This is the writeup for the plug-in scenario / metrics idea that I brought up yesterday.

Thanks, this looks good to me!  I think we could also use this mechanism to deal with `run_specs.py` so that we can split that into multiple files and start using the decorator as a convention everywhere.  Should we prototype it with some scenario?