Repository: stanford-crfm/helm
Issue: Where is the dataset for HEIM? 
Root cause: Documentation Deficiency

Where is the dataset that HEIM uses for its evaluations, the set of prompts that have been used to generate images, I am not able to find those. Please help.

@teetone Where is the dataset?

All the prompts and outputs can be found here: https://crfm.stanford.edu/helm/heim/latest/#/runs

> @teetone Where is the dataset?



@teetone May I know if there is a link to download all the evaluation results and generated images? And are all the evaluated images posted at the predictions page, like this [page](https://crfm.stanford.edu/helm/heim/latest/#/runs/common_syntactic_processes:phenomenon=ambiguity,model=huggingface_redshift-diffusion,max_eval_instances=100)?

You can download the evaluation results and generated images using the instructions [here](https://crfm-helm.readthedocs.io/en/latest/downloading_raw_results/).

> You can download the evaluation results and generated images using the instructions [here](https://crfm-helm.readthedocs.io/en/latest/downloading_raw_results/).

@yifanmai  Thanks for the reply. However, I couldn't find the images in the google cloud storage, only json files.
Below is an example result from ls command in one evaluation folders:
```
gcloud storage ls gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/
gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/display_predictions.json
gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/display_requests.json
gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/instances.json
gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/per_instance_stats.json
gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/run_spec.json
gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/scenario.json
gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/scenario_state.json
gs://crfm-helm-public/heim/benchmark_output/runs/v1.1.0/mscoco:model=huggingface_stable-diffusion-v1-4,max_eval_instances=100,groups=mscoco_base/stats.json
```
I find image path in the json file, but I have no idea how to access it.
```
{
      "instance": {
        "input": {
          "text": "A man is in mid air as he plummets towards the bed."
        },
        "references": [
          {
            "output": {
              "text": "",
              "file_path": "benchmark_output/scenarios/mscoco/data/val/COCO_val2014_000000468471.jpg"
            },
            "tags": [
              "correct"
            ]
          }
        ],
        "split": "valid",
        "id": "id564858"
      },
      "train_trial_index": 0,
      "request": {
        "model": "AlephAlpha/m-vader",
        "embedding": false,
        "prompt": "A man is in mid air as he plummets towards the bed.",
        "temperature": 1.0,
        "num_completions": 1,
        "top_k_per_token": 1,
        "max_tokens": 0,
        "stop_sequences": [],
        "echo_prompt": false,
        "top_p": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0
      },
      "result": {
        "success": true,
        "embedding": [],
        "completions": [
          {
            "text": "",
            "logprob": 0,
            "tokens": [],
            "file_location": "prod_env/cache/output/AlephAlpha/f181d35ad3564935a9781a92d4b886b3.jpg"
          }
        ],
        "cached": true,
        "request_time": 3.931143045425415
      },
      "num_train_instances": 0,
      "prompt_truncated": false,
      "num_conditioning_tokens": 0
    }
```
 

Hi @weiyi1991, for HEIM, the images are not actually stored in GCS as image files. Instead, you should access the `display_predictions.json` file, which will contain a field called `base64_images`, which is a string array in which each string is a base64 encoded image.

cc @teetone 

> Hi @weiyi1991, for HEIM, the images are not actually stored in GCS as image files. Instead, you should access the `display_predictions.json` file, which will contain a field called `base64_images`, which is a string array in which each string is a base64 encoded image.
> 
> cc @teetone

I see, thanks a lot!