Repository: stanford-crfm/helm
Issue: Error running HEIM getting started example.
Root cause: External Dependency Breakage

I've been running into issues with running the example in the HEIM docs: https://crfm-helm.readthedocs.io/en/latest/heim/

The summary of what I've tried so far to overcome some errors is:


```bash
uv venv helm-venv-310 --python=python3.10 --seed
source ./helm-venv-310/bin/activate
uv pip install -e .[heim]
bash install-heim-extras.sh
uv pip install numpy==1.26.4
uv pip install jax==0.4.30
uv pip install flax -U numpy==1.26.4
uv pip install ml_dtypes==0.3.1

pip uninstall -y tensorflow ml_dtypes
pip cache purge
pip install "tensorflow==2.19.0" "ml_dtypes==0.5.1"

helm-run --run-entries mscoco:model=huggingface/stable-diffusion-v1-4 --suite my-heim-suite --max-eval-instances 10
```

The above script likely isn't minimal, but I'm just trying to document steps as I take them.


Details of errors I've been able to resolve with the above version pins:


<details>

```
An issue with diffusers / hugginface_hub:

```
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/diffusers/utils/dynamic_modules_utils.py", line 28, in <module>
    from huggingface_hub import HfFolder, cached_download, hf_hub_download, model_info
ImportError: cannot import name 'cached_download' from 'huggingface_hub' (/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/huggingface_hub/__init__.py)
```

There was also an issue with jax:

```
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/diffusers/models/unets/unet_2d_condition_flax.py", line 16, in <module>
    import flax
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/flax/__init__.py", line 22, in <module>
    from . import core
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/flax/core/__init__.py", line 16, in <module>
    from .frozen_dict import (
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/flax/core/frozen_dict.py", line 50, in <module>
    @jax.tree_util.register_pytree_with_keys_class
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'jax.tree_util' has no attribute 'register_pytree_with_keys_class'
```


After an upgrade of jax I got:

```
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/diffusers/models/unets/unet_2d_condition_flax.py", line 16, in <module>
    import flax
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/flax/__init__.py", line 18, in <module>
    from .configurations import (
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/flax/configurations.py", line 92, in <module>
    flax_filter_frames = define_bool_state(
                         ^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/flax/configurations.py", line 42, in define_bool_state
    return jax_config.define_bool_state('flax_' + name, default, help)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Config' object has no attribute 'define_bool_state'
```

```
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/diffusers/loaders/single_file_model.py", line 27, in <module>
    from .single_file_utils import (
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/diffusers/loaders/single_file_utils.py", line 52, in <module>
    from transformers import AutoImageProcessor
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2154, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2184, in _get_module
    raise e
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2182, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/home/joncrall/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py", line 27, in <module>
    from ...image_processing_utils import ImageProcessingMixin
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 22, in <module>
    from .image_transforms import center_crop, normalize, rescale
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/transformers/image_transforms.py", line 48, in <module>
    import tensorflow as tf
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/tensorflow/__init__.py", line 38, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/tensorflow/python/__init__.py", line 42, in <module>
    from tensorflow.python.saved_model import saved_model
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/tensorflow/python/saved_model/saved_model.py", line 20, in <module>
    from tensorflow.python.saved_model import builder
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/tensorflow/python/saved_model/builder.py", line 23, in <module>
    from tensorflow.python.saved_model.builder_impl import _SavedModelBuilder
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/tensorflow/python/saved_model/builder_impl.py", line 26, in <module>
    from tensorflow.python.framework import dtypes
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/tensorflow/python/framework/dtypes.py", line 37, in <module>
    _np_bfloat16 = pywrap_ml_dtypes.bfloat16()
TypeError: Unable to convert function return value to a Python type! The signature was
	() -> handle
```

This is a tensorflow error, that happens when I just import it. But can be fixed with a reinstall with pinned ml_dtypes.

```

</details>

However, there are errors I'm having trouble resolving:


First an error importing DiffusionPipeline:

```
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/retrying.py", line 259, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/joncrall/code/helm/src/helm/clients/auto_client.py", line 118, in make_request_with_retry
    return client.make_request(request)
  File "/home/joncrall/code/helm/src/helm/clients/image_generation/huggingface_diffusers_client.py", line 91, in make_request
    from diffusers import DiffusionPipeline
ImportError: cannot import name 'DiffusionPipeline' from 'diffusers' (/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/diffusers/__init__.py)
```

What is strange to me about this is I can import this module if I do it directly via:

```bash
python -c "from diffusers import DiffusionPipeline"
# or 
python -c "from helm.clients.image_generation import huggingface_diffusers_client"
```

Some sort of scheduling error in scheduling_pndm: 

```
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
                      ^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/helm/src/helm/clients/auto_client.py", line 118, in make_request_with_retry
    return client.make_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/helm/src/helm/clients/image_generation/huggingface_diffusers_client.py", line 179, in make_request
    results, cached = self._cache.get(cache_key, wrap_request_time(do_it))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/helm/src/helm/common/cache.py", line 198, in get
    response = compute()
               ^^^^^^^^^
  File "/home/joncrall/code/helm/src/helm/common/request.py", line 261, in wrapped_compute
    response = compute()
               ^^^^^^^^^
  File "/home/joncrall/code/helm/src/helm/clients/image_generation/huggingface_diffusers_client.py", line 155, in do_it
    image = diffuser(**raw_request).images[0]  # type: ignore
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 1060, in __call__
    latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs, return_dict=False)[0]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/diffusers/schedulers/scheduling_pndm.py", line 257, in step
    return self.step_plms(model_output=model_output, timestep=timestep, sample=sample, return_dict=return_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joncrall/code/helm/helm-env/lib/python3.11/site-packages/diffusers/schedulers/scheduling_pndm.py", line 380, in step_plms
    model_output = (1 / 24) * (55 * self.ets[-1] - 59 * self.ets[-2] + 37 * self.ets[-3] - 9 * self.ets[-4])
```

This error (or a similar one) happens multiple times. 


Still working on debugging, but maybe someone knows more about these errors. If I learn more I'll update the post. 

After a bit more poking around I found I could run with `--num-threads 1` to remove some parallelism and better pin down specific errors. In doing that I got a completely different exception: 

```
Traceback (most recent call last):
  File "/home/joncrall/code/helm/src/helm/benchmark/runner.py", line 216, in run_all
    self.run_one(run_spec)
  File "/home/joncrall/code/helm/src/helm/benchmark/runner.py", line 307, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/home/joncrall/code/helm/src/helm/benchmark/metrics/metric.py", line 143, in evaluate
    results: List[List[Stat]] = parallel_map(
  File "/home/joncrall/code/helm/src/helm/common/general.py", line 241, in parallel_map
    results = list(tqdm(map(process, items), total=len(items), disable=None))
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/home/joncrall/code/helm/src/helm/benchmark/metrics/metric.py", line 77, in process
    self.metric.evaluate_generation(
  File "/home/joncrall/code/helm/src/helm/benchmark/metrics/image_generation/clip_score_metrics.py", line 58, in evaluate_generation
    prompt = WindowServiceFactory.get_window_service(model, metric_service).truncate_from_right(prompt)
  File "/home/joncrall/code/helm/src/helm/benchmark/window_services/window_service_factory.py", line 17, in get_window_service
    model_deployment: Optional[ModelDeployment] = get_model_deployment(model_deployment_name)
  File "/home/joncrall/code/helm/src/helm/benchmark/model_deployment_registry.py", line 128, in get_model_deployment
    raise ValueError(f"Model deployment {name} not found")
ValueError: Model deployment openai/clip-vit-large-patch14 not found
```

I was able to run:

```python
from transformers import CLIPProcessor, CLIPModel

# Load model and processor from Hugging Face
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
```

So I think that means I can access the models, not sure why helm is not recognizing it yet. 


I think this might be an error introduced in c5633af07cdf881563d593b89034a5f390eee3eb 

Using `WindowServiceFactory.get_window_service` seems to want a model deployment, but `openai/clip-vit-large-patch14` is a tokenizer config. So when the WindowServiceFactory calls `get_model_deployment` of course it will error, because its not in the model deployment config.

I'm not familiar enough with the HELM architecture to know what the appropriate fix is. There seem to be many levels of indirection that makes it difficult to make minimal test examples outside running the entire system.

Hi @Erotemic, unfortunately, there is no official support for HEIM currently. The HEIM code was previously contributed by the paper's lead authors (including @teetone), but they are no longer actively answering questions in this repository.

Your best bet may be to try using the [v0.5.0 release](https://github.com/stanford-crfm/helm/releases/tag/v0.5.0), which was the first release after the HEIM code was added.

I will investigate the `WindowServiceFactory.get_window_service` issue.

That's helpful. Unfortunately v0.5.0 doesn't work, but it helps me narrow down my search space, so maybe I can make some progress. 

From what I can gather HEIM was added right after the v0.4.0 release in 346f765d9f03b866cb7454bbedc5aa7ed7199749 on 2023-12-20, and v0.5.0 is quite a bit after that on 2024-04-23. Using the older HEIM commit didn't work out of the box because `--run-entries` didn't exist yet. But the commit 46df083b218e0a326d5b09a4e51b0fd3cd0115b2 on 2024-03-01 is the one that adds it, and after playing with the install requirements a bit, I got it to do a bit more, but it still has the same issue, likely because it is after the c5633af07cdf881563d593b89034a5f390eee3eb (2024-01-09) commit which changed the way running CLIP scores was handled.

I will also continue to look into it and report back if I learn anything more. I would like to help get HEIM back to being fully supported and reproducible.

Note for my reference:

* ebbb3466025917e7340aad83effec0488d2d0f6d (2024-04-23) - v0.5.0
* 46df083b218e0a326d5b09a4e51b0fd3cd0115b2 (2024-03-01) - adds `--run-entries`
* c5633af07cdf881563d593b89034a5f390eee3eb (2024-01-08) - refactors CLIPWindowService
* 346f765d9f03b866cb7454bbedc5aa7ed7199749 (2023-12-20) - Add HEIM 
* 6b12aec0c07cdd7e58c62fe0008873a86686b6a7 (2023-12-20) - v0.4.0


On older versions of HELM, you should use `--run-specs` instead of `--run-entries`.

The `WindowServiceFactory.get_window_service()` issue should be fixed by #3750.

Thanks @yifanmai, I've been able to make some progress with some caveats. On the latest main, I've been able to run:

```bash
helm-run \
    --run-entries mscoco:model=huggingface/stable-diffusion-v1-4 \
    --suite my-heim-suite \
    --max-eval-instances 10 \
    --num-threads 1
```

without errors.

The caveats are:

1. I had to disable the `helm.benchmark.metrics.image_generation.nsfw_metrics.NSFWMetric` metric spec. I didn't see any obvious way to do this with a configuration file or the CLI, so I commented it out in src/helm/benchmark/run_specs/heim_run_specs.py. 

Details about the NSFWDetector metric issue:

<details>

The issues seems to be that the nsfw detector is relying on an older `clip_autokeras_binary_nsfw.zip` keras model, which has difficult requirements to satisfy. With the current set of dependencies I get:

```
Traceback (most recent call last):
  File "/home/joncrall/code/helm/src/helm/benchmark/metrics/image_generation/nsfw_detector.py", line 30, in load_safety_model
    import autokeras as ak
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/autokeras/__init__.py", line 17, in <module>
    from autokeras.auto_model import AutoModel
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/autokeras/auto_model.py", line 26, in <module>
    from autokeras import blocks
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/autokeras/blocks/__init__.py", line 27, in <module>
    from autokeras.blocks.heads import ClassificationHead
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/autokeras/blocks/heads.py", line 26, in <module>
    from autokeras import hyper_preprocessors as hpps_module
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/autokeras/hyper_preprocessors.py", line 16, in <module>
    from autokeras import preprocessors
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/autokeras/preprocessors/__init__.py", line 17, in <module>
    from autokeras.preprocessors.common import AddOneDimension
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/autokeras/preprocessors/common.py", line 18, in <module>
    from autokeras import keras_layers
  File "/home/joncrall/code/helm/helm-venv-310/lib/python3.10/site-packages/autokeras/keras_layers.py", line 20, in <module>
    from tensorflow.keras.layers.experimental import preprocessing
ModuleNotFoundError: No module named 'tensorflow.keras.layers.experimental'
```

However, I also spent a decent amount of time trying to make just this work with the older state of the requirements around the time HEIM was originally contributed, but I don't think the exact set of dependencies to make this model work will be easy to find without some help from the original HEIM authors. If we want to get HEIM running at all, I would recommend disabling this metric until everything else works smoothly, and then think about readding or replacing it with something more robust.

Here are some unstructured notes I took when  (unsuccessfully)  tried to get the model to load in a standalone way:

```bash
unzip benchmark_output/models/clip_autokeras_binary_nsfw.zip -d /tmp/unzipped_model

cd /home/joncrall/code/helm-heim
git co 346f765d9f03b866cb7454bbedc5aa7ed7199749
uv venv helm-heim-309 --python=python3.9 --seed
we helm-heim-309

sed -i 's|^torch|#torch|g' requirements.txt
uv pip install -r requirements.txt  --exclude-newer 2024-12-20
uv pip install --no-cache-dir --find-links https://download.pytorch.org/whl/torch_stable.html torch==1.12.1+cu113 torchvision==0.13.1+cu113 --exclude-newer 2024-12-20
uv pip install jax==0.3.25 jaxlib==0.3.25+cuda11.cudnn805 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html --exclude-newer 2024-12-20
uv pip install tensorflow numpy==1.23.3 autokeras==1.0.19 --exclude-newer 2024-12-20
uv pip install keras==2.11 tensorflow==2.11 --exclude-newer 2024-12-20
uv pip uninstall keras keras-core keras-nlp keras-hub
uv pip install autokeras==1.0.19 --no-deps
uv pip install keras==2.11 tensorflow==2.11 --exclude-newer 2024-12-20

#uv pip install keras-tuner==1.1.0
#uv pip uninstall keras-tuner
uv pip install keras-tuner==1.1.0 --exclude-newer 2024-12-20


python -c "if 1:
    from tensorflow import keras
    from tensorflow import keras
    import autokeras as ak

    # Path to the extracted SavedModel directory
    model_path = '/tmp/unzipped_model/clip_autokeras_binary_nsfw'
    model = keras.models.load_model(model_path)
"
```

--- end nsfw detector details ---

</details>


2. I had to update requirements:

```diff
-diffusers==0.24.0
+diffusers==0.34.0
-flax==0.6.11
+flax==0.10.7
-jax==0.4.38 ; python_full_version >= '3.10'
+jax==0.6.2 ; python_full_version >= '3.10'
-jaxlib==0.4.38 ; python_full_version >= '3.10'
+jaxlib==0.6.2 ; python_full_version >= '3.10'
-ml-dtypes==0.4.1
+ml-dtypes==0.5.1
```

Now, it looks like it's output some meaningful responses in the suite directory, but when I run the helm server, I can't view predictions.

Details about issues viewing HEIM results with the server:

<details>

On the official leaderboard I can see instances: https://crfm.stanford.edu/helm/heim/latest/#/runs/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base

On my local server, I can see an entry on my local predictions page:

<img width="1000" height="490" alt="Image" src="https://github.com/user-attachments/assets/e5adb506-79c2-4873-9435-8cfc7925bdf8" />

but when I click it I get a blank white screen. In my chrome console I do see a react error, but I'm not sure how meaningful it is:

```
Error: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=object%20with%20keys%20%7B%7D for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at Sl (react-f82877fd.js:30:12785)
    at C (react-f82877fd.js:30:17840)
    at Fe (react-f82877fd.js:32:1475)
    at ld (react-f82877fd.js:32:45620)
    at nd (react-f82877fd.js:32:39696)
    at Sg (react-f82877fd.js:32:39624)
    at fi (react-f82877fd.js:32:39479)
    at Su (react-f82877fd.js:32:35862)
    at Zp (react-f82877fd.js:32:34813)
    at T (react-f82877fd.js:17:1559)
```

I also see an issue which might be more relevant, but my web development experience is limited, so I'm still working to understand what potential paths forward there are:

```

Content Security Policy of your site blocks the use of 'eval' in JavaScript`
The Content Security Policy (CSP) prevents the evaluation of arbitrary strings as JavaScript to make it more difficult for an attacker to inject unathorized code on your site.

To solve this issue, avoid using eval(), new Function(), setTimeout([string], ...) and setInterval([string], ...) for evaluating strings.

If you absolutely must: you can enable string evaluation by adding unsafe-eval as an allowed source in a script-src directive.

⚠️ Allowing string evaluation comes at the risk of inline script injection.

1 directive
Source location	Directive	Status
script-src	blocked
Learn more: Content Security Policy - Eval
```

</details>

You might be able to run this with the React frontend in developer mode, which should give you a more useful error trace. I'll provide some instructions for how to do this shortly.

For these instructions, you will need to have the repository checked out. You will also need to have [installed Yarn](https://yarnpkg.com/getting-started/install).

Suppose you are running the following to view the web application using Python:

```sh
helm-summarize --suite my-suite --schema-path path/to/schema_heim.yaml
helm-server
```

Here are the corresponding instructions to view the web application with Yarn and React's developer mode:

1. Ensure that the `benchmark_output` and `prod_env` directories are siblings of `helm-frontend` (i.e. they all have the same parent directory).
2. Modify `helm-frontend/public/config.js` locally to the following (replacing `my-suite` appropriately):
```javascript
window.SUITE = "my-suite";
window.BENCHMARK_OUTPUT_BASE_URL = "/benchmark_output/";
```
3. With `helm-frontend` as your working directory, run `yarn install` to install the application dependencies
4. With `helm-frontend` as your working directory, run `yarn dev`.
5. Open http://localhost:5173/ in your web browser. You should see the web application React's developer mode.

I should probably add this to the developer documentation at some point!