Repository: stanford-crfm/helm
Issue: Error while running medi_qa task
Root cause: Environment Incompatibility

entries: [
  {description: "medi_qa", priority: 1},
]


(client_env) ubuntu@159-13-59-117:/lambda/nfs/glm/MedHELM$ helm-run -c ./ss-s.conf \
    --suite Qwen/Qwen2.5-7B        \
    --max-eval-instances 1 \
    --local-path ./config \
    --models-to-run Qwen/Qwen2.5-7B     
main {
  Reading tokenizer configs from /home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/config/tokenizer_configs.yaml...
  Reading model deployments from /home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/config/model_deployments.yaml...
  Reading tokenizer configs from ./config/tokenizer_configs.yaml...
  Reading model deployments from ./config/model_deployments.yaml...
  Read 1 run entries from ./ss-s.conf
  1 entries produced 1 run specs
  run_specs {
    RunSpec(name='medi_qa:model=Qwen_Qwen2.5-7B', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.medi_qa_scenario.MediQAScenario', args={}), adapter_spec=AdapterSpec(method='generation', global_prefix='', global_suffix='', instructions='Answer the following consumer health question.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', chain_of_thought_prefix='', chain_of_thought_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=0, max_eval_instances=1, num_outputs=1, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='Qwen/Qwen2.5-7B', model='Qwen/Qwen2.5-7B', temperature=0.0, max_tokens=1024, stop_sequences=[], random=None, multi_label=False, image_generation_parameters=None, reeval_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.summarization_metrics.SummarizationMetric', args={'task': 'medi_qa', 'device': 'cuda', 'bertscore_model': 'distilbert-base-uncased', 'rescale_with_baseline': False}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': []}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.medi_qa_metrics.MediQAMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['medi_qa'], annotators=[AnnotatorSpec(class_name='helm.benchmark.annotation.medi_qa_annotator.MediQAAnnotator', args={})])
  } [0.0s]
  Running in local mode with base path: ./config
Looking in path: ./config
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  AutoClient: file_storage_path = ./config/cache
  AutoClient: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  Found 1 account(s).
Looking in path: ./config
  AnnotatorFactory: file_storage_path = ./config/cache
  AnnotatorFactory: cache_backend_config = SqliteCacheBackendConfig(path='./config/cache')
  0%|                                                                                                         | 0/1 [00:00<?, ?it/s]  Running medi_qa:model=Qwen_Qwen2.5-7B {
    scenario.get_instances {
    } [0.993s]
  } [0.996s]
  Error when running medi_qa:model=Qwen_Qwen2.5-7B:
Traceback (most recent call last):
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/runner.py", line 216, in run_all
    self.run_one(run_spec)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/runner.py", line 255, in run_one
    instances = scenario.get_instances(scenario_output_path)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/scenarios/medi_qa_scenario.py", line 91, in get_instances
    dataset = load_dataset("bigbio/mediqa_qa")
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/datasets/load.py", line 2606, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/datasets/load.py", line 2277, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/datasets/load.py", line 1923, in dataset_module_factory
    raise e1 from None
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/datasets/load.py", line 1875, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
  File "/usr/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.00s/it]
} [6.068s]
Traceback (most recent call last):
  File "/home/ubuntu/glm/MedHELM/client_env/bin/helm-run", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/run.py", line 341, in main
    run_benchmarking(
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/run.py", line 127, in run_benchmarking
    runner.run_all(run_specs)
  File "/home/ubuntu/glm/MedHELM/client_env/lib/python3.10/site-packages/helm/benchmark/runner.py", line 225, in run_all
    raise RunnerError(f"Failed runs: [{failed_runs_str}]")
helm.benchmark.runner.RunnerError: Failed runs: ["medi_qa:model=Qwen_Qwen2.5-7B"]
(client_env) ubuntu@159-13-59-117:/lambda/nfs/glm/MedHELM$ 

It's likely that you have a malformed local cached copy of the `bigbio/mediqa_qa` dataset. You can fix this by forcing a re-download of the dataset ([documentation](https://huggingface.co/docs/datasets/en/cache)).

```python3
from datasets import load_dataset
dataset = load_dataset(
    "bigbio/mediqa_qa",
    trust_remote_code=True,
    revision="9288641f4c785c95dc9079fa526dabb12efdb041",
    download_mode='force_redownload'
)
```

Also note that the LLM jury for `medi_qa` is currently tied to Stanford Health Care's internal environment and does not work for public users yet. cc @MiguelAFH 

Thanks