Repository: stanford-crfm/helm
Issue: Can't success run demos of text-to-image evaluation when following HEIM Quick Start Docs
Root cause: Environment Incompatibility

Hi,

Thank you for publishing this amazing work!

I am trying to use HEIM to benchmark text-to-image generation model, and I was following the quick start guide of HEIM in docs/heim.md and ran into this error message: `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb2 in position 3009: invalid start byte` 

I tested the installation by running the quick start guide in HELM and it is working properly.

The run_specs file I am using for HEIM looks like this:
`entries: [{description: "i2p:model=huggingface/stable-diffusion-v1-4,category=hate,max_eval_instances=4", priority: 1}]`

I encounter the error message when running HEIM because UnicodeDecodeError in `i2p` and `mscoco` dataset, others not tried yet. Could you give some insights of what may be the cause of this error and possible solutions to it?

Thank you!

This is the full error message:
```

main {
  Reading tokenizer configs from /home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/config/tokenizer_configs.yaml...
  Reading model deployments from /home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/config/model_deployments.yaml...
  Read 1 run entries from run_heim.conf
  1 entries produced 1 run specs
  run_specs {
    RunSpec(name='i2p:category=hate,model=huggingface_stable-diffusion-v1-4,max_eval_instances=4', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.image_generation.i2p_scenario.I2PScenario', args={'category': 'hate'}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='', input_prefix='', input_suffix='', reference_prefix='A. ', reference_suffix='\n', output_prefix='', output_suffix='', instance_prefix='\n', substitutions=[], max_train_instances=0, max_eval_instances=4, num_outputs=8, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='huggingface/stable-diffusion-v1-4', model='huggingface/stable-diffusion-v1-4', temperature=1, max_tokens=0, stop_sequences=[], random=None, multi_label=False, image_generation_parameters=ImageGenerationParameters(output_image_width=None, output_image_height=None, guidance_scale=None, diffusion_denoising_steps=None)), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.image_generation.aesthetics_metrics.AestheticsMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.clip_score_metrics.CLIPScoreMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.efficiency_metrics.EfficiencyMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.fractal_dimension_metric.FractalDimensionMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.nsfw_metrics.NSFWMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.nudity_metrics.NudityMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.watermark_metrics.WatermarkMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': []}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['i2p'])
  } [0.0s]
  Running in local mode with base path: prod_env
Looking in path: prod_env
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
  AutoClient: file_storage_path = prod_env/cache
  AutoClient: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
  Found 1 account(s).
  Created cache with config: SqliteCacheConfig(path='prod_env/cache/ModerationAPI.sqlite')
  0%|                                                                                                        | 0/1 [00:00<?, ?it/s]  Running i2p:category=hate,model=huggingface_stable-diffusion-v1-4,max_eval_instances=4 {
    scenario.get_instances {
    } [12.193s]
    89 instances, 0 train instances, 4/89 eval instances
    DataPreprocessor.preprocess {
    } [0.0s]
    GenerationAdapter.adapt {
      4 instances, choosing 0/0 train instances, 4 eval instances
      Adapting with train_trial_index=0 {
        Sampled 0 examples for trial #0.
        Parallelizing computation on 4 items over 4 threads {
                                                                                                                                            Created cache with config: SqliteCacheConfig(path='prod_env/cache/openai.sqlite')                   | 0/4 [00:00<?, ?it/s]
          Created cache with config: SqliteCacheConfig(path='prod_env/cache/openai.sqlite')
          Created cache with config: SqliteCacheConfig(path='prod_env/cache/openai.sqlite')
          Created cache with config: SqliteCacheConfig(path='prod_env/cache/openai.sqlite')



  0%|                                                                                                        | 0/4 [00:00<?, ?it/s]

        } [0.077s]
      } [0.077s]
    } [0.077s]
  } [12.275s]
  0%|                                                                                                        | 0/1 [00:12<?, ?it/s]
} [12.547s]
Error in sys.excepthook:
Traceback (most recent call last):
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/linecache.py", line 47, in getlines
    return updatecache(filename, module_globals)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/linecache.py", line 137, in updatecache
    lines = fp.readlines()
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb2 in position 3009: invalid start byte

Original exception was:
Traceback (most recent call last):
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/runner.py", line 242, in run_all
    self.run_one(run_spec)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/runner.py", line 307, in run_one
    request_states: List[RequestState] = adapter.adapt(instances, self.executor.execution_spec.parallelism)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 63, in adapt
    self._adapt_trial_index(all_train_instances, train_trial_index, eval_instances, parallelism)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 86, in _adapt_trial_index
    results: List[List[RequestState]] = parallel_map(
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/common/general.py", line 235, in parallel_map
    results = list(tqdm(executor.map(process, items), total=len(items), disable=None))
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 83, in generate_requests_for_training_trial
    return self.generate_requests(eval_instance, train_trial_index, training_instances)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/adaptation/adapters/generation_adapter.py", line 37, in generate_requests
    prompt: Prompt = self.construct_prompt(
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 242, in construct_prompt
    prompt = self._make_prompt_fit(prompt)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 323, in _make_prompt_fit
    truncated_text = self.window_service.truncate_from_right(text, self.adapter_spec.max_tokens)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/window_services/image_generation/clip_window_service.py", line 8, in truncate_from_right
    result: str = self.decode(self.encode(text, truncation=True, max_length=self.max_request_length).tokens)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/window_services/local_window_service.py", line 45, in encode
    response: TokenizationRequestResult = self.service.tokenize(
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/window_services/tokenizer_service.py", line 23, in tokenize
    return self._service.tokenize(self._auth, request)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/proxy/services/server_service.py", line 153, in tokenize
    return self.tokenizer.tokenize(request)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/proxy/tokenizers/auto_tokenizer.py", line 72, in tokenize
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/proxy/retry.py", line 72, in <lambda>
    return lambda f: lambda *args, **kwargs: _retrying.call(f, *args, **kwargs)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/retrying.py", line 256, in call
    if not self.should_reject(attempt):
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/retrying.py", line 237, in should_reject
    reject |= self._retry_on_exception(attempt.value[1])
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/proxy/retry.py", line 60, in print_exception_and_traceback
    hlog("".join(traceback.format_exception(type(exception), exception, exception.__traceback__)))
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/traceback.py", line 120, in format_exception
    return list(TracebackException(
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/exceptiongroup/_formatting.py", line 96, in __init__
    self.stack = traceback.StackSummary.extract(
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/traceback.py", line 366, in extract
    f.line
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/traceback.py", line 288, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/linecache.py", line 47, in getlines
    return updatecache(filename, module_globals)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/linecache.py", line 137, in updatecache
    lines = fp.readlines()
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb2 in position 3009: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/taas/anaconda3/envs/crfm-helm/bin/helm-run", line 8, in <module>
    sys.exit(main())
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/run.py", line 320, in main
    run_benchmarking(
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/run.py", line 129, in run_benchmarking
    runner.run_all(run_specs)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/helm/benchmark/runner.py", line 247, in run_all
    hlog(f"Error when running {run_spec.name}:{traceback.format_exc()}")
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/traceback.py", line 167, in format_exc
    return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/traceback.py", line 120, in format_exception
    return list(TracebackException(
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/site-packages/exceptiongroup/_formatting.py", line 96, in __init__
    self.stack = traceback.StackSummary.extract(
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/traceback.py", line 366, in extract
    f.line
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/traceback.py", line 288, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/linecache.py", line 47, in getlines
    return updatecache(filename, module_globals)
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/linecache.py", line 137, in updatecache
    lines = fp.readlines()
  File "/home/taas/anaconda3/envs/crfm-helm/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb2 in position 3009: invalid start byte
```


cc @teetone for HEIM

The most likely cause is that you may have a corrupted file in your scenarios download. Could you try deleting your downloaded scenario files and try re-running `helm-run` (which will re-download the files)? In the case of i2p, the files will be in `./benchmark_output/scenarios/i2p/`.

Closing this due to staleness; feel free to reopen if you have further questions.