Repository: stanford-crfm/helm
Issue: Issues from running quickstart
Root cause: Environment Incompatibility

I encountered three issues when running [quickstart](https://crfm-helm.readthedocs.io/en/latest/quick_start/):

### Error 1 
```
(helm) ivanzhou@jagupard36:~/dev/helm$ helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10


[nltk_data] Downloading package punkt to
[nltk_data]     /sailhome/ivanzhou/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
Traceback (most recent call last):
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/bin/helm-run", line 5, in <module>
    from helm.benchmark.run import main
  File "/sailhome/ivanzhou/dev/helm/src/helm/benchmark/__init__.py", line 74, in <module>
    from .metrics import summarization_metrics  # noqa
  File "/sailhome/ivanzhou/dev/helm/src/helm/benchmark/metrics/summarization_metrics.py", line 7, in <module>
    import spacy
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/__init__.py", line 14, in <module>
    from . import pipeline  # noqa: F401
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/pipeline/__init__.py", line 1, in <module>
    from .attributeruler import AttributeRuler
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/pipeline/attributeruler.py", line 6, in <module>
    from .pipe import Pipe
  File "spacy/pipeline/pipe.pyx", line 8, in init spacy.pipeline.pipe
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/training/__init__.py", line 11, in <module>
    from .callbacks import create_copy_from_base_model  # noqa: F401
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/training/callbacks.py", line 3, in <module>
    from ..language import Language
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/language.py", line 25, in <module>
    from .training.initialize import init_vocab, init_tok2vec
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/training/initialize.py", line 14, in <module>
    from .pretrain import get_tok2vec_ref
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/training/pretrain.py", line 16, in <module>
    from ..schemas import ConfigSchemaPretrain
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/site-packages/spacy/schemas.py", line 216, in <module>
    class TokenPattern(BaseModel):
  File "pydantic/main.py", line 299, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 411, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 342, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 451, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 545, in pydantic.fields.ModelField._type_analysis
  File "pydantic/fields.py", line 550, in pydantic.fields.ModelField._type_analysis
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/lib/python3.8/typing.py", line 774, in __subclasscheck__
    return issubclass(cls, self.__origin__)
TypeError: issubclass() arg 1 must be a class
```

Solution: `pip install typing-inspect==0.8.0`

### Error 2:
```
(helm) ivanzhou@jagupard36:~/dev/helm$ helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10
  File "/nlp/scr/ivanzhou/miniconda3/envs/helm/bin/helm-run", line 5, in <module>
    from helm.benchmark.run import main
  File "/sailhome/ivanzhou/dev/helm/src/helm/benchmark/run.py", line 16, in <module>
    from .slurm_runner import SlurmRunner
  File "/sailhome/ivanzhou/dev/helm/src/helm/benchmark/slurm_runner.py", line 14, in <module>
    from helm.benchmark.slurm_jobs import (
  File "/sailhome/ivanzhou/dev/helm/src/helm/benchmark/slurm_jobs.py", line 5, in <module>
    from simple_slurm import Slurm
ModuleNotFoundError: No module named 'simple_slurm'
```

Solution: `pip install simple-slurm`

### Error 3:
```
HuggingFace error: CUDA error: no kernel image is available for execution on the device
```

Solution: upgrade to PyTorch 2.0

----
My environment
- I installed locally from the source code with conda
- NLP Cluster (A6000, CUDA 11.7)

`pip install typing-inspect==0.8.0` seems not to work for the first error. BTW, thank you for the post.

For the first issue, I also did `typing_extensions==4.5.0`. Not sure if that's needed or not. 

- `typing-inspect`: not sure why this is needed, maybe a conda thing?
- `simple-slurm`: probably cauesd by working on a stale branch; this is in requirements [here](https://github.com/stanford-crfm/helm/blob/01ee95a8c5211e858116641db71a633834ad73b5/requirements.txt#L21) and [here](https://github.com/stanford-crfm/helm/blob/01ee95a8c5211e858116641db71a633834ad73b5/requirements-freeze.txt#L140) on the main branch. Though if it's still not installing on main, we should look into it.
- `typing_extensions` is needed (for dev only) and [already in the requirements](https://github.com/stanford-crfm/helm/blob/01ee95a8c5211e858116641db71a633834ad73b5/requirements-freeze.txt#L179)
- CUDA: We should upgrade to the latest minor version of PyTorch, which should also fix some security problems. I'm somewhat concerned that things will break when upgrading to PyTorch 2.0.

> simple-slurm: probably cauesd by working on a stale branch; this is in requirements [here](https://github.com/stanford-crfm/helm/blob/01ee95a8c5211e858116641db71a633834ad73b5/requirements.txt#L21) and [here](https://github.com/stanford-crfm/helm/blob/01ee95a8c5211e858116641db71a633834ad73b5/requirements-freeze.txt#L140) on the main branch. Though if it's still not installing on main, we should look into it.

@yifanmai That's odd. I just git pull yesterday and `pip install -e .` in the main branch. 

On my side, `typing_extensions==4.5.0` is required when using the pip crfm-helm package. 
\+ The code has worked well without further installation when following the [developer setup](https://crfm-helm.readthedocs.io/en/latest/developer_setup/).

This should be fixed by #1617.