Repository: stanford-crfm/helm
Issue: Contributing/Adding new model
Root cause: Documentation Deficiency

Apologies if this is already described somewhere, but how can one add a new model not in the library of models already?

I am particularly interested in OSS models and would like to add many of them. 

Given that the LLM area is very dynamic/fast-paced, what would be a good solution to make this a streamlined process where users can contribute easily?

@PhantomSpike - great idea.  I'd like to nominate StableVicuna and Mosic MPT-7B, for starters.  Since many of the OSS models are in HuggingFace, I wonder of HELM already has HF support, and we just need the eval compute.  Or perhaps, each model still needs some HELM adaption beyond HF.  

Hi,
Thanks for asking this. 
We are currently working on a standardized way of adding new models. In the meantime, here is the process to add a new model:
- First, you need to register the model in `schema.yaml` and `models.py`. Make sure to use the same `name`.
- Then you need a `Client`. Fo that you will have to go to `auto_client.py` and handle your new model in `_get_client()` and probably `_get_tokenizer_client()`. if your model comes from HuggingFace, it is probably already supported by our `HuggingFaceClient`. If not, you might need to implement a `Client` yourself. The amount of work this will require depends on how "compatible" the API of the model is with HELM. In the future, we hope to release an API doc that should be respected in order to be evaluated on HELM but this is not yet the case. If you need to write your own `Client`, you should take an example on simple clients like `palmyra_client.py`.
- Finally, you need a `WindowService` (this is something we hope to remove soon, but it's not a priority right now). To do so, you need to go to `window_service_factory.py` and add a case to handle your model in `get_window_service`. The `WindowService` defines properties of your model and the tokenizer associated with it such as the `max_request_length`, `tokenizer_name`, and special tokens, ... You probably can reuse an existing `WindowService`. Often we use `MODEL_TAGS` to identify the proper `WindowService`. Make sure to check out existing tags in `models.py` and add them to your model.
- If you've followed all the steps, normally you should be able to run your model. If your code is clean and you respected the previous steps, please consider sending a PR as it might interest other people to have this model added and we cannot add all of them ourselves

Let me know if you have more questions and feel free to add me as a reviewer to your PR.

Happy coding!

You might also find the existing documentation handy: [development setup](https://crfm-helm.readthedocs.io/en/latest/developer_setup/) and [adding new models](https://crfm-helm.readthedocs.io/en/latest/adding_new_models/).

As Josselin mentioned, we plan on improving the workflow and documentation soon.

Hi @JosselinSomervilleRoberts @yifanmai - I'm helping organize a competition to eval LLMs so was very interested in easier ways of adding models

1. If registering a new model requires updates to the repo then it might make more sense to recommend users `pip install -e .` since a common usecase for HELM is to evaluate some new model on a broad suite. Ideally there should be something like `helm register` that would make these changes on behalf of users
2. A new client feels necessary only if users are using some strange object store, in which case settling on something like `fsspec` should help reduce the number of clients
3. Can we just drop the `WindowService` abstraction? it feels like all the data should just be in the same place in the model registration

FWIW I'm OK on settling on a single client like HF Hub if they make 1 and 3 more seamless

EDIT: This already does what I need https://github.com/stanford-crfm/helm/blob/main/docs/huggingface_models.md

Hi @msaroufim,

1. Indeed; we support Hugging Face Hub, and we will soon support Hugging Face on local disk in #1505. 
2. If you need Hugging Face on Azure Blob Store / GCS / S3 support, we can add `fsspec` support.
3. If you use Hugging Face, we indeed auto-detect all this information from the model and tokenizer, so you don't need to write a window service.

It mostly depends on what kind of inference servers / libraries you want to support. I would be interested in adding integrations with other inference servers / libraries that work similarly to the Hugging Face one.

I'm also considering moving the Hugging Face flags into a config file, because it get very unwieldy if you have lots of models.

Thanks @yifanmai! 

Yes for our use cases a local eval and an eval on object stores would be really invaluable, I'd imagine many competitors would not want to publicly share their weights until the competition is complete

Regarding other libraries or inference servers that part is also interesting, on one hand, if all competitors settled on HF then that makes our eval simpler but we also want to make it easy for people who may not want to use HF to also also contribute. Maybe they're using torchserve/kserve/triton or maybe they dont like the HF abstractions and would prefer to use their favorite training loop provider lightning/mosaic/something custom, I'm still undecided what to do in this case. We figured maybe a custom helm client for the competition

And yes moving the HF flags to a file would be very helfpul



I added #1673 and #1674 for these requests. I also made the "competition" label; feel free to use this to tag competition-related issues.

> Maybe they're using torchserve/kserve/triton or maybe they dont like the HF abstractions and would prefer to use their favorite training loop provider lightning/mosaic/something custom, I'm still undecided what to do in this case. We figured maybe a custom helm client for the competition

For common things, we can add a client e.g. I think torchserve/kserve/triton would make sense.

If we want to be entirely framework agnostic, the submitter could provide a OpenAI API compatible HTTP server, or a batch script that outputs a request / response JSON file in a HELM-compatible format, and we could make HELM have better support for these.

Another thought on private model hosting: HELM already supports [private repositories](https://huggingface.co/docs/hub/repositories-settings)

Another thought: HELM already supports [private repositories on Hugging Face Hub](https://huggingface.co/docs/hub/repositories-settings#private-repositories). The user just needs to set the Hugging Face authentication as shell environment variables before running HELM. So that could be another way of uploading and hosting a private model.

Closing this - we now support many other ways to run private models, including Hugging Face models on disk, vLLM, OpenAI compatible servers, etc.