Repository: stanford-crfm/helm
Issue: Feature request: HELM as a modular library that can be imported through scripts
Root cause: Unimplemented Feature Gap

Hi,
 
I have been going through HELM for a while now. HELM has done an excellent job when it comes to unifying all sorts of tasks and metrics in just a few commands with insightful visualizations. However, this feature could make it more extensible to integrate into various other workflows. So, I would like to propose this feature request where we can use this library in scrips by importing its different modules. 

Let's consider the scenario of the case of NeurIPS challenge. There I was using lit-gpt's implementation and here is how I was running the scoring

```python
entries: [{
description: "raft:subset=banking_77,model=neurips/local,data_augmentation=canonical", priority: 2}]
```

And I was using Lightning AI's implementation for the server side of things. I ran the benchmark with this command

```bash
helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 1000
```
Till then, things worked great, but when I was trying to do 

```bash
helm-summarize --suite v1
helm-serve
```
This will start a local host where we can visualize the dashboard in the raw-runs section. However, can an alternative path exist like this (pseudo code)

```python
from helm import evaluation_piepline

run = evaluation_piepline(
    entries = [{
        "description": "raft:subset=banking_77,model=neurips/local,data_augmentation=canonical", 
        "priority": 2
    }],
    suite = "v1",
    max_eval_instances = 1000
)

summary = run.summarize()

# Visualize in a pandas data frame
# summary will be a dict of data frames, where the keys will be different entries's description and the value will be
# the summary of the model's prediction containing columns like ['prompt', 'reference', 'prediction'], etc etc.

print(summary['raft:subset=banking_77'])
```

The above example is super high level, however, it makes things more accessible to people without running the interface through just the command line. So I would love to discuss the thought process here and whether this can be implemented or not.

Thanks for opening this issue. This has been on my roadmap for a while, but I haven't gotten around to it because I wasn't sure how many users need this.

There are actually a few components to this issue:

1. Supporting pandas for aggregations, which is much more friendly to users trying to do data analysis.
    - Relatedly, we've are considering rewriting the aggregation system in `helm-summarize`, and I think that we should just be using pandas for everything (cc @farzaank @percyliang for `helm-summarize` rewrite / "HELMlets")
2. Formalizing and documenting the external Python library API - it is already possible to call into HELM functions, but we should make some of this API external and stable.
3. Address how filesystem / sqlite caching works when HELM is in library mode - HELM in CLI mode writes to a number of local file paths and local sqlite files; we need to make these configurable.

Another related issue is #1704.

By the way, if all you want to do is call `helm-run` from within Python, you can play around with the [`run_benchmarking`](https://github.com/stanford-crfm/helm/blob/v0.2.4/src/helm/benchmark/run.py#L70) function. Again, this is not a great public API, so we probably want something better.

I see, that makes sense. Thanks for letting me know. I will try the `run_benchmarking` function. So is it safe to assume that I can still run in library model in HELM without any internal code changes, by just exposing some parts of its API?

Yes, with the caveat that these APIs might change in the future. But you can work around this by pinning the version of `crfm-helm` that you are using.

I see, thanks for the update 🙂

Hi @yifanmai I was about to write some more details regarding how I wanted to use HELM in PyTorch CI but @Anindyadeep did a great job already at communicating this

Here's another example Charles wrote using the Eleuther eval harness that might give you some ideas for what we're trying to get at https://gist.github.com/msaroufim/c941c73dbec102b0788a8994d9adb671

Ahh yes, this is what exactly I was looking for @msaroufim. However, I wanted to have the same for HELM, as I see it seems to be more structured in terms of categorization. 

Thank you! This does look like a use case we would want to support eventually, though we have not prioritized it yet on our roadmap for the next couple months. cc @percyliang @JosselinSomervilleRoberts 

Some HELM architecture notes:

1. It's more natural for the API user to pass in Client / WindowService / Tokenizer object directly, as in the code snippet above, but we don't permit this in HELM yet. Right now, we require users to provide a `ClientSpec`, which provides a class name from which the object is imported, because `ClientSpec` can be serialized to disk for outpu purposes.
2. It's more natural for the API to return the evaluation results to the user directly, but right now `run_benchmarking` does not return the results. It instead writes it to disk, and the user has to read back the results from disk.

I was also looking for something similar
I wanted to use helm as part of my existing scripts and not as a separate process to spawn and manage