Repository: stanford-crfm/helm
Issue: Guidance on how to run HELM benchmarks against LLM models deployed on SageMaker Endpoints
Root cause: Documentation Deficiency

Can the authors provide a prescriptive guidance on how to run HELM benchmarks against LLMs deployed on SageMaker? There seem to be two obvious approaches:
a\front SageMaker Endpoint with an AWS API Gateway and provide and API_Key. What's not clear is where/how to provide API_URL. This approach is more scalable as it allows a SageMaker Endpoint to be invoked externally outside of AWS Account where the endpoint was deployed.
b\clone git repo into a machine with AWS Account credentials and hack the code to directly invoke SageMaker Endpoint instead of going the API_Key route. 

Any suggestions/experience with either one of the approaches?


AWS SageMaker support is something we definitely want to have in HELM eventually.

(a) and (b) are both reasonable approaches. I would suggest doing a fork and doing (b) if you have an urgent use case. For (a), you're welcome to open a PR to contribute a more general way of doing this.

I imagine that it would work similarly to how the [HuggingFace Hub integration](https://crfm-helm.readthedocs.io/en/latest/huggingface_models/) works with the `--enable-huggingface-models` flag where we specify an API endpoint URL through a flag passed to `helm-run`. The trickiness is that you'd have to also specify the model name and model length using this flag (which can't be auto-inferred from the URL, as far as I can tell).

I also think it would be good to move these things to a config file instead of flags eventually, but we don't have plans for that yet either.

I decided to follow option (b) as it is a more scalable approach. I deployed an AI21 Jurassic-J2-Jumbo model on a SageMaker endpoint and fronted it with an API Gateway + Lambda. API_URL and API_KEY can be used to invoke the model, eg: 
`curl -X POST -H "Content-Type: application/json" -H "X-API-Key: test" -d '{"prompt": "Who was Albert Einstein?"}' https://mkyfzfz3si.execute-api.us-west-2.amazonaws.com/completion`
- What is the format to use for API_KEY in credentials.conf?
- How should I provide API_URL?

At least at the 1st pass of integration, we aren't going to make all models in SageMaker Jumpstart available with public APIs (it would be too costly to host). 
I would rather a user create an LLM endpoint and front it with an API Gateway providing API_URL and (optional) API_KEY.
Then the user would specify these two parameters in a HELM config file (or command line params) and run applicable benchmarks. The assumption is that the user knows LLM's capabilities and would thus select an appropriate HELM benchmark to run. 


Amazon API Gateway URL is be different for every new deployed API. Specifically, the alphanumeric string following https:// (eg. https://mkyfzfz3si) as well as geo region (eg us-west-2). Each user would deploy a new API Gateway. It would not be possible to infer the URL from a model name. 
Therefore, if we follow the CLI flag proposal similar to --enable-huggingface-models, we would need to specify the API URL in the command line which would make it too long. Any thoughts of having a CLI flag (eg. --enable-amazon-sagemaker) which would then read the content of sagemaker.conf file? SageMaker API_URL and API_KEY would be stored in the sagemaker.conf.

The conf file option sounds reasonable to me:


```
[
  {
    "name": "mymodel",
    "url": "https://mymodelurl/",
    "api_key": "my_key"
  },
  {
    "name": "mymodel2",
    "url": "https://mymodelurl2/",
    "api_key": "my_key"
  }
]
```

We now support Titan on Bedrock using BedrockClient: see #2165.

For other model providers on Bedrock, we should use their first party Python library integrations:

- [Anthropic](https://docs.anthropic.com/en/api/claude-on-amazon-bedrock)
- [AI21](https://docs.ai21.com/docs/aws-bedrock)
- [Cohere](https://docs.cohere.com/docs/amazon-bedrock)