Repository: stanford-crfm/helm
Issue: Questions about HELM Software Architecture
GitHub labels: user question

This might be more appropriate for a discussion page, but I don't see a discussion tab for this repo, so I'm writing this as an issue.

I am currently evaluating HELM for use in a large scale program for testing generalization claims about specific networks / tuning methods / etc. It seems to have 90% of what I need, and furthermore, we can likely use HELM as a backend service that handles the components of the system related to running a specific model on a specific scenario. It would be nice if we were able to contribute back to this project to patch in additional functionality that makes sense, but to do that I need a deeper understanding of the architecture. The current level of docs has gotten me pretty far, but I have a few questions I'm hoping someone can help me with here.

From what I'm seeing the output of running a set of scenarios is stored in a runs folder where the user picks the name of the folder to go there. One issue I'm seeing is that if two users run two different scenarios with the same name on different machines, then there isn't a straightforward way to combine them. Similarly if there are folders with different names, but the same fundamental content, then I don't see a de-duplication mechanism.

I'm wondering if helm is punting on this problem all together, or if there is handling of this that I'm not seeing. (perhaps some file in each user-named output directory has identifiers that would effectively recognize unique runs, so just renaming them all with UUIDs would address the issue out of the box?)

Similarly I'm seeing some database code, but not much related to it in the docs. I was wondering if these filesystem outputs are expected to be ingested into a database?

Additionally I see in the [Run Entries docs](https://crfm-helm.readthedocs.io/en/latest/run_entries/) run arguments are specified as long strings of function:key1=value1,...keyn=valuen, which can get cumbersome quickly. These seem to find there way into filenames, which will absolutely become a problem, and its one that I've solved in the past by hashing the parameters and storing the inverse mapping from hash to parameter values. I'm wondering if that approach has been considered (or if a hash-based path renaming triggers if the filenames do become too long)?

Some thoughts:

- If a run is run on two different machines, under some assumptions (e.g. identical version of HELM and dependencies), the two runs should be equivalent or identical (if the model is deterministic). Because of this, there is rarely a need for deduplication. `helm-summarize` does some kind of deduplication in the sense that for a given run name, it only uses the most recently seen run with that name.
- Run UUIDs are generally a good idea, and other frameworks (e.g. Inspect AI) take this approach. I think that it would be a good idea to do this if HELM was redesigned from the ground up. Hashing is also a reasonable approach. However, both approaches would make the folders less greppable.
- I agree that the folder names are cumbersome. They also cause some secondary issues (e.g. the `:` character is not allowed by Windows; the run name length may exceed the filesystem limit)
- We use databases for some other applications e.g. the request-response cache can be stored in a database (currently, only SQLite and MongoDB are supported).
- You can experiment with alternative run results storage schemes by implementing your own `Runner` class (which is responsible for storing run results) and then passing it as `--runner-class-name` to `helm-run`. This would allow you store the results in a database which your other infrastructure could read from. Note that downstream tools like `helm-summarize` won't work with results that are stored in this way.
- I have considered a results storage abstraction layer but this would be a lot of work. I also think that database storage is a good idea, but it is tricky to design a good database schema.
- Many design decisions was solidified before I joined the team, so I do not have the full context or fully agree with many of the decisions. HELM's design is optimized to allow academic researchers to collaborate by sharing folders over a network file system, which explains why the result storage is based around file trees and JSON files. We try to avoid breaking existing HELM users' workflows as much as possible.

Thanks for the feedback. I'm hoping that I'll be able to use and make meaningful contributions to HELM, and this s very helpful for my understanding about what the current limitations are and how to design a roadmap that is a net benefit while also not breaking user workflows. 

>  I think that it would be a good idea to do this if HELM was redesigned from the ground up. Hashing is also a reasonable approach. However, both approaches would make the folders less greppable.

I think that file based storage actually is the right place to start. From there it can ingested into a database. I can see a path to upgrade Helm's infrastructure without breaking existing workflows: 

* Add metadata files to each folder that contain UUIDs / config params. 
* Slowly migrate the filesystem based storage so the real folders use UUIDs, but they are symlinked to the current user-defined paths replicating existing behavior and maintaining search-ability (I've actually designed a robust filesystem based experiment storage that can be intuitively navigated and ). 

As an aside, I love using symlinks to map to the most recent run. Great design choice.

> They also cause some secondary issues (e.g. the : character is not allowed by Windows; the run name length may exceed the filesystem limit). 

I do have a solution here: [kwutil.util_path.sanitize_path_name](https://gitlab.kitware.com/computer-vision/kwutil/-/blob/main/kwutil/util_path.py?ref_type=heads#L466), although it may get confusing if a user starts small and gets interpretable file names, but then they change to hashes if they get big. However, it might be less irritating than having the system just break and not run the job.

> I have considered a results storage abstraction layer but this would be a lot of work. I also think that database storage is a good idea, but it is tricky to design a good database schema.

I think you start off with no schema and just map UUIDS to unstructured json. You don't need HELM itself to talk to a database, it just needs to produce something that can be ingested. If these concerns are kept separate, then both sides can evolve as needed. 

---

Another question I'm wondering about: 

I see logic for submitting jobs with slurm, but I don't see docs on how to use it. How well supported is slurm? What I would like to do is dispatch a lot of jobs - potentially over many different machines, and then collect all the results onto a centralized server. 

I've been working on a package: [cmd_queue](https://gitlab.kitware.com/computer-vision/cmd_queue), which is designed to run jobs in a pipeline similar to slurm. In fact, it has a backend that just wraps slurm. The key design that makes it different is that it doesn't need any complex configuration to _do something_. If you don't have slurm installed it can just run jobs one at a time, or use tmux to parallelize them. I have plans to extend it by adding an airflow backend, which would let it scale from single low power machine with no configurations all the way up to multi-node cloud clusters. Currently it can get up to the HPC level with the slurm backend. The only caveat is that jobs need to be runnable via independent bash commands. This has the upside of making thing very debuggable because you can always run a failing process independently.

Does helm have a way to run any particular job as an independent CLI invocation? I imagine if slurm is used, then there must be some mechanism to do so. If not, is there a path to adding this feature? Would it be a custom Runner?