Repository: stanford-crfm/helm
Issue: `summarization_cnndm,model=openai_davinci` failed 
Root cause: External Dependency Breakage

Reproducible with a dry run:

Run: `benchmark-present --api-key-path <Path to text file with API key>  --max-eval-instances 1 --conf src/benchmark/presentation/run_specs2.conf --dry-run`

where `run_specs2.conf` has just these two lines:

```
"summarization_cnndm:model=default": {status: "READY"}
"summarization_xsum:model=default": {status: "READY"}
```

Error (fails 43 minutes in):
```
Downloading and preparing dataset cnn_dailymail/3.0.0 to /sailhome/tonyhlee/.cache/huggingface/datasets/ccdv___cnn_dailymail/3.0.0/3.0.0/0107f7388b5c6fae455a5661bcd134fc22da53ea75852027040d8d1e997f101f...
Downloading data files: 100%|██████████| 5/5 [00:00<00:00, 1022.50it/s]
Extracting data files: 100%|██████████| 5/5 [00:00<00:00, 127.93it/s]
Extracting data files:   0%|          | 0/5 [00:00<?, ?it/s]           } [42m30.517s]
    } [42m30.517s]
  } [42m30.52s]
  0%|          | 0/2 [42:30<?, ?it/s]} [42m30.565s]

Traceback (most recent call last):
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/bin/benchmark-present", line 33, in <module>
    sys.exit(load_entry_point('crfm-benchmarking', 'console_scripts', 'benchmark-present')())
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 153, in main
    runner.run()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 90, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 46, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 63, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 81, in run_one
    instances: List[Instance] = DataPreprocessor(run_spec.data_augmenter_spec).preprocess(scenario)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/data_preprocessor.py", line 30, in preprocess
    instances: Sequence[Instance] = scenario.get_instances()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/summarization_scenario.py", line 85, in get_instances
    dataset, article_key, summary_key = self._load_dataset(self.dataset_name)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/summarization_scenario.py", line 76, in _load_dataset
    dataset = load_dataset("ccdv/cnn_dailymail", "3.0.0")
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/datasets/load.py", line 1687, in load_dataset
    builder_instance.download_and_prepare(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/datasets/builder.py", line 696, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file.
Original error:
error closing file
```

Could you please direct me to the log file for this error? It seems different that last time. It's so weird, it was working when I test it locally.

> Could you please direct me to the log file for this error? It seems different that last time. It's so weird, it was working when I test it locally.

@fladhak I've run this multiple times on the NLP cluster and kept getting this error. I didn't see any other errors except what I included in the issue description.

OK no worries. I'll look into it and see what I can find.

@fladhak Can this be closed now based on the last PR (i.e. #363)?


Yes, it can be closed.