Repository: stanford-crfm/helm
Issue: Error in running HEIM following the quick start guide
Root cause: Environment Incompatibility

Hi,

Thank you for publishing this amazing benchmark!

I am trying to use HEIM to benchmark text-to-image generation model, and I was following the quick start guide of HEIM in docs/heim.md and ran into this error message: ```ValueError: Unknown run spec name: mscoco```.

I tested the installation by running the quick start guide in HELM and it is working properly. 

The run_specs file I am using for HEIM looks like this:
 ```entries: [{description: "mscoco:model=huggingface/stable-diffusion-v1-4", priority: 1}]```

I encounter the error message when running HEIM because the name 'mscoco' is not included in the ```CANONICAL_RUN_SPEC_FUNCS``` dictionary, nor is any other of the other names used in HEIM (cub200, common_syntactic_processes,...).  Could you give some insights of what may be the cause of this error and possible solutions to it? 

Thank you!

This is the full error message:
```
main {
  Reading tokenizer configs from /home/xxx/miniconda3/envs/heim/lib/python3.8/site-packages/helm/config/tokenizer_configs.yaml...
  Reading model deployments from /home/xxx/miniconda3/envs/heim/lib/python3.8/site-packages/helm/config/model_deployments.yaml...
  Read 1 run entries from run_specs.conf
} [0.33s]
Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/heim/bin/helm-run", line 8, in <module>
    sys.exit(main())
  File "/home/xxx/miniconda3/envs/heim/lib/python3.8/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/home/xxx/miniconda3/envs/heim/lib/python3.8/site-packages/helm/benchmark/run.py", line 275, in main
    run_specs = run_entries_to_run_specs(
  File "/home/xxx/miniconda3/envs/heim/lib/python3.8/site-packages/helm/benchmark/run.py", line 42, in run_entries_to_run_specs
    for run_spec in construct_run_specs(parse_object_spec(entry.description)):
  File "/home/xxx/miniconda3/envs/heim/lib/python3.8/site-packages/helm/benchmark/run_specs.py", line 2673, in construct_run_specs
    raise ValueError(f"Unknown run spec name: {name}")
ValueError: Unknown run spec name: mscoco
```

Hi @gaojl19, thanks for reaching out! HEIM was not released yet as part of the latest package available on PyPI (`crfm-helm 0.4.0`). Sorry about that! 

Following the [local dev setup steps](https://crfm-helm.readthedocs.io/en/latest/developer_setup), could you check out the repo and run `./install-dev.sh`. Then try the [HEIM quick start guide](https://crfm-helm.readthedocs.io/en/latest/heim/) again.

Alternatively, if you are using `pip install crfm-helm[heim]`, you can instead replace this pip command with `pip install crfm-helm[heim]@git+https://github.com/stanford-crfm/helm.git@main` which will give you the HEIM capabilities.

@teetone I'm planning to do a PyPI release in the next couple of weeks, which will resolve this.

> Alternatively, if you are using `pip install crfm-helm[heim]`, you can instead replace this pip command with `pip install crfm-helm[heim]@git+https://github.com/stanford-crfm/helm.git@main` which will give you the HEIM capabilities.
> 
> @teetone I'm planning to do a PyPI release in the next couple of weeks, which will resolve this.

Thanks @yifanmai!

Hi @gaojl19, are you still facing an issue with running HEIM? If not, could we close this issue? Thank you!