Repository: stanford-crfm/helm
Issue: RaceBias dataset access issue
Root cause: Environment Incompatibility

The RaceBias is a public MedHELM dataset and can be evaluated by everyone. However, it fails with:

docx.opc.exceptions.PackageNotFoundError: Package not found at '/share/pi/nigam/data/medhelm/race_based/race_based.docx'
Please guide how to retrieve it.


Complete Run Log

main {
  Reading tokenizer configs from /home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/config/tokenizer_configs.yaml...
  Reading model deployments from /home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/config/model_deployments.yaml...
  Reading tokenizer configs from ./config/tokenizer_configs.yaml...
  Reading model deployments from ./config/model_deployments.yaml...
  Read 1 run entries from ./entries-2.conf
  1 entries produced 1 run specs
  run_specs {
    RunSpec(name='race_based_med:model=Zaynoid_qwen2.5-7b-v2', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.race_based_med_scenario.RaceBasedMedScenario', args={}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='Answer A for yes, B for no.\n', input_prefix='', input_suffix='', reference_prefix='A. ', reference_suffix='\n', chain_of_thought_prefix='', chain_of_thought_suffix='\n', output_prefix="Respond with only 'A' for yes or 'B' for no. Do not add any other text, punctuation, or symbols: ", output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=0, max_eval_instances=1, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='Zaynoid/qwen2.5-7b-v2', model='Zaynoid/qwen2.5-7b-v2', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, reeval_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['race_based_med'], annotators=None)
  } [0.0s]
  Running in local mode with base path: ./config
Looking in path: ./config
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  AutoClient: file_storage_path = ./config/cache
  AutoClient: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  Found 1 account(s).
Looking in path: ./config
  AnnotatorFactory: file_storage_path = ./config/cache
  AnnotatorFactory: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  0%|                                                                                                            | 0/1 [00:00<?, ?it/s]  Running race_based_med:model=Zaynoid_qwen2.5-7b-v2 {
    scenario.get_instances {
    } [0.004s]
  } [0.162s]
  Error when running race_based_med:model=Zaynoid_qwen2.5-7b-v2:
Traceback (most recent call last):
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/runner.py", line 216, in run_all
    self.run_one(run_spec)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/runner.py", line 255, in run_one
    instances = scenario.get_instances(scenario_output_path)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/scenarios/race_based_med_scenario.py", line 123, in get_instances
    create_csv_from_word(word_file, data_path)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/scenarios/race_based_med_scenario.py", line 67, in create_csv_from_word
    doc = Document(doc_path)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/docx/api.py", line 27, in Document
    document_part = cast("DocumentPart", Package.open(docx).main_document_part)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/docx/opc/package.py", line 126, in open
    pkg_reader = PackageReader.from_file(pkg_file)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/docx/opc/pkgreader.py", line 22, in from_file
    phys_reader = PhysPkgReader(pkg_file)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/docx/opc/phys_pkg.py", line 21, in __new__
    raise PackageNotFoundError("Package not found at '%s'" % pkg_file)
docx.opc.exceptions.PackageNotFoundError: Package not found at '/share/pi/nigam/data/medhelm/race_based/race_based.docx'

100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.02it/s]
} [5.099s]
Traceback (most recent call last):
  File "/home/ubuntu/glm/MedHELM/client_env/bin/helm-run", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/run.py", line 341, in main
    run_benchmarking(
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/run.py", line 127, in run_benchmarking
    runner.run_all(run_specs)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/runner.py", line 225, in run_all
    raise RunnerError(f"Failed runs: [{failed_runs_str}]")
helm.benchmark.runner.RunnerError: Failed runs: ["race_based_med:model=Zaynoid_qwen2.5-7b-v2"]







This issue has been resolved with the latest commit