Repository: stanford-crfm/helm
Issue: Dependecy Conflict while Installing Heim
Root cause: External Dependency Breakage

Hi there,

I ran into the following error while trying to install Heim in both Google Colab and our local server, according to the instructions mentioned in the "heim.md" file.
While I ran the following code:
`bash install-heim-extras.sh`

I receive this error below:
`ERROR: Ignored the following yanked versions: 0.4.32
ERROR: Could not find a version that satisfies the requirement jaxlib==0.3.25+cuda11.cudnn805 (from versions: 0.4.17, 0.4.17+cuda11.cudnn86, 0.4.17+cuda12.cudnn89, 0.4.18, 0.4.18+cuda11.cudnn86, 0.4.18+cuda12.cudnn89, 0.4.19, 0.4.19+cuda11.cudnn86, 0.4.19+cuda12.cudnn89, 0.4.20, 0.4.20+cuda11.cudnn86, 0.4.20+cuda12.cudnn89, 0.4.21, 0.4.21+cuda11.cudnn86, 0.4.21+cuda12.cudnn89, 0.4.22, 0.4.22+cuda11.cudnn86, 0.4.22+cuda12.cudnn89, 0.4.23, 0.4.23+cuda11.cudnn86, 0.4.23+cuda12.cudnn89, 0.4.24, 0.4.24+cuda11.cudnn86, 0.4.24+cuda12.cudnn89, 0.4.25, 0.4.25+cuda11.cudnn86, 0.4.25+cuda12.cudnn89, 0.4.26, 0.4.26+cuda12.cudnn89, 0.4.27, 0.4.27+cuda12.cudnn89, 0.4.28, 0.4.28+cuda12.cudnn89, 0.4.29, 0.4.29+cuda12.cudnn91, 0.4.30, 0.4.31, 0.4.33, 0.4.34, 0.4.35, 0.4.36, 0.4.38, 0.5.0, 0.5.1, 0.5.3, 0.6.0)
ERROR: No matching distribution found for jaxlib==0.3.25+cuda11.cudnn805`

And then when I try to solve them, I get stuck in a loop of version conflicts. I would be truly thankful if you could help me solve these issues.

Best regards,

`install-heim-extras.sh` is supported by @teetone - @teetone could you advise?

My own suggestion is to try installing everything with unpinned versions and see if pip (or uv) can resolve a set of compatible versions.

Thank you so much for your response. I have also tried to ignore installing `install-heim-extras.sh`, but still I receive an error while trying to run the following command:

`helm-run --run-entries mscoco:model=huggingface/stable-diffusion-v1-4 --suite my-heim-suite --max-eval-instances 10
`

And the error that I get is like below:

> Traceback (most recent call last):
  File "/usr/local/bin/helm-run", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/helm/benchmark/run.py", line 323, in main
    run_specs = run_entries_to_run_specs(
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/helm/benchmark/run.py", line 43, in run_entries_to_run_specs
    for run_spec in construct_run_specs(parse_object_spec(entry.description)):
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/helm/benchmark/run_spec_factory.py", line 57, in construct_run_specs
    run_spec_function = get_run_spec_function(name)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/helm/benchmark/run_spec.py", line 92, in get_run_spec_function
    discover_run_spec_functions()
  File "/usr/local/lib/python3.12/site-packages/helm/benchmark/run_spec.py", line 87, in discover_run_spec_functions
    importlib.import_module(name)
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/local/lib/python3.12/site-packages/helm/benchmark/run_specs/vlm_run_specs.py", line 10, in <module>
    from helm.benchmark.scenarios.vision_language.image2struct.image2struct_scenario import DIFFICULTY_ALL
  File "/usr/local/lib/python3.12/site-packages/helm/benchmark/scenarios/vision_language/image2struct/image2struct_scenario.py", line 5, in <module>
    from datasets import load_dataset
  File "/usr/local/lib/python3.12/site-packages/datasets/__init__.py", line 17, in <module>
    from .arrow_dataset import Dataset
  File "/usr/local/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 76, in <module>
    from .arrow_reader import ArrowReader
  File "/usr/local/lib/python3.12/site-packages/datasets/arrow_reader.py", line 32, in <module>
    from .download.download_config import DownloadConfig
  File "/usr/local/lib/python3.12/site-packages/datasets/download/__init__.py", line 9, in <module>
    from .download_manager import DownloadManager, DownloadMode
  File "/usr/local/lib/python3.12/site-packages/datasets/download/download_manager.py", line 33, in <module>
    from ..utils import tqdm as hf_tqdm
  File "/usr/local/lib/python3.12/site-packages/datasets/utils/__init__.py", line 17, in <module>
    from .info_utils import VerificationMode
  File "/usr/local/lib/python3.12/site-packages/datasets/utils/info_utils.py", line 5, in <module>
    from huggingface_hub.utils import insecure_hashlib
ImportError: cannot import name 'insecure_hashlib' from 'huggingface_hub.utils' (/usr/local/lib/python3.12/site-packages/huggingface_hub/utils/__init__.py)

I would suggest upgrading `huggingface-hub` and `transformers` and trying again.

Thank you for your response once again. However, I am still encountering conflicts with the `Jax` library. I am feeling really frustrated, as I have been dealing with these issues for days now. It seems like there is no light at the end of the tunnel.

I understand your frustration and am sorry that I'm not able to help more. Unfortunately, the HEIM code is more than a year old and has no continuous integration, and it is possible that it no longer works. Unless @teetone is able to help here, I would suggest treating the HEIM code as unmaintained and looking for other solutions.

It's unfortunate that we might not be able to execute this code, as we truly need a reliable tool to assess our model's performance on benchmarking. I believe your benchmark offers the best solution for that. One question I have is whether it is necessary to run Heim on a GPU, or can we also run it on our CPU?

I'm not sure - the originally HEIM paper was run using GPUs.

Thank you very much for all your support. However, I still haven't been able to resolve this issue. It's quite disappointing for me, especially since this project is intended to be a comprehensive tool for measuring the performance of image generation models. Hearing that it might be outdated is truly discouraging, as it could have been a significant help to those working in this field.

Thanks for your efforts in troubleshooting this. I'll leave this issue open in case I have time to work on this, and I'll let you know if I have any updates.