Repository: stanford-crfm/helm
Issue: API key for together models
Root cause: Documentation Deficiency

Hi,

Thank you for the great work.

I ran `helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10` and got an error like below.

My run specs: `entries: [{description: "mmlu:subject=abstract_algebra,model=together/h3-2.7b", priority: 1}]`

I might have to add an API key into `credentials.conf` to use the models from `together`. But, where can I get the API key?
```
Traceback (most recent call last):
  File "/prj/neo_lv/user/esuh/workspace/helm-dev/venv/lib/python3.8/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/prj/neo_lv/user/esuh/workspace/helm-dev/helm/src/helm/proxy/clients/auto_client.py", line 150, in make_request_with_retry
    return client.make_request(request)
  File "/prj/neo_lv/user/esuh/workspace/helm-dev/helm/src/helm/proxy/clients/together_client.py", line 96, in make_request
    response, cached = self.cache.get(cache_key, wrap_request_time(do_it if self.api_key else fail))
  File "/prj/neo_lv/user/esuh/workspace/helm-dev/helm/src/helm/common/cache.py", line 377, in get
    response = compute()
  File "/prj/neo_lv/user/esuh/workspace/helm-dev/helm/src/helm/proxy/clients/client.py", line 48, in wrapped_compute
    response = compute()
  File "/prj/neo_lv/user/esuh/workspace/helm-dev/helm/src/helm/proxy/clients/together_client.py", line 94, in fail
    raise RuntimeError(f"The result has not been uploaded to the cache for the following request: {cache_key}")
RuntimeError: The result has not been uploaded to the cache for the following request: {'request_type': 'language-model-inference', 'model': 'h3-2.7b-h3', 'prompt': 'The following are multiple choice questions (with answers) about abstract algebra.\n\nQuestion: Statement 1 | If aH is an element of a factor group, then |aH| divides |a|. Statement 2 | If H and K are subgroups of G then HK is a subgroup of G.\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer: B\n\nQuestion: Find all c in Z_3 such that Z_3[x]/(x^2 + c) is a field.\nA. 0\nB. 1\nC. 2\nD. 3\nAnswer: B\n\nQuestion: Find the characteristic of the ring 2Z.\nA. 0\nB. 3\nC. 12\nD. 30\nAnswer: A\n\nQuestion: Statement 1| Every function from a finite set onto itself must be one to one. Statement 2 | Every subgroup of an abelian group is abelian.\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer: A\n\nQuestion: Statement 1 | Every element of a group generates a cyclic subgroup of the group. Statement 2 | The symmetric group S_10 has 10 elements.\nA. True, True\nB. False, False\nC. True, False\nD. False, True\nAnswer: C\n\nQuestion: Compute the product in the given ring. (12)(16) in Z_24\nA. 0\nB. 1\nC. 4\nD. 6\nAnswer:', 'temperature': 0.0, 'n': 1, 'max_tokens': 1, 'best_of': 5, 'logprobs': 5, 'stop': None, 'echo': False, 'top_p': 1}
```

You will have to sign up on the [Together waitlist](https://api.together.xyz/) to get an API key.

It would be nice to be able to do local inference with a Hugging Face auto model for H3, or a HELM integration with the [reference H3 implementation](https://github.com/HazyResearch/H3), but currently neither exists.

cc @DanFu09

@yifanmai Hi, I encountered the same problem. So, if I want to do local inference, I have to use HF auto model integration, right?

Correct; please refer to the [Hugging Face Hub integration documentation](https://crfm-helm.readthedocs.io/en/latest/huggingface_models/).

It would be nice to be able just use the built in models with Hugging Face and without Together, so I filed #1755 for this.