Repository: EleutherAI/lm-evaluation-harness
Issue: task: `headqa_en` not working - wrong key
Root cause: External Dependency Breakage

Using the eval harness tools from https://github.com/bigscience-workshop/Megatron-DeepSpeed I can download and eval on most tasks, but `headqa` (`headqa_en`):
```
python ./tasks/eval_harness/download.py --task_list headqa_en
2022-01-18 21:19:33.476640: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
WARNING: headqa is deprecated. Please use headqa_es or headqa_en instead. See https://github.com/EleutherAI/lm-evaluation-harness/pull/240 for more info.
Reusing dataset head_qa (/gpfswork/rech/six/commun/datasets/head_qa/en/1.1.0/d6803d1e84273cdc4a2cf3c5102945d166555f47b299ecbc5266d582f408f8e2)
```
and then at eval time:
```
WARNING:datasets.load:Using the latest cached version of the module from /gpfswork/rech/six/commun/modules/datasets_modules/datasets/head_qa/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9 (last modified on Sat Jan  8 18:38:49 2022) since it couldn't be found locally at head_qa.
Traceback (most recent call last):
  File "./tasks/eval_harness/evaluate.py", line 393, in <module>
    main()
  File "./tasks/eval_harness/evaluate.py", line 377, in main
    task_dict = tasks.get_task_dict(task_list)
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 320, in get_task_dict
    task_name_dict = {
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/lm_eval/tasks/__init__.py", line 321, in <dictcomp>
    task_name: get_task(task_name)()
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/lm_eval/tasks/common.py", line 11, in __init__
    super().__init__()
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/lm_eval/base.py", line 350, in __init__
    self.download()
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/lm_eval/tasks/common.py", line 14, in download
    self.data = datasets.load_dataset(path=self.DATASET_PATH, name=self.DATASET_NAME)
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/load.py", line 1604, in load_dataset
    builder_instance = load_dataset_builder(
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/load.py", line 1462, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/builder.py", line 1043, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/builder.py", line 266, in __init__
    info = self.get_exported_dataset_info()
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/builder.py", line 321, in get_exported_dataset_info
    return self.get_all_exported_dataset_infos().get(self.config.name, DatasetInfo())
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/builder.py", line 316, in get_all_exported_dataset_infos
    return DatasetInfosDict.from_directory(cls.get_imported_module_dir())
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/info.py", line 294, in from_directory
    dataset_infos_dict = {
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/info.py", line 295, in <dictcomp>
    config_name: DatasetInfo.from_dict(dataset_info_dict)
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/info.py", line 261, in from_dict
    return cls(**{k: v for k, v in dataset_info_dict.items() if k in field_names})
  File "<string>", line 20, in __init__
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/info.py", line 144, in __post_init__
    self.features = Features.from_dict(self.features)
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/features/features.py", line 1019, in from_dict
    obj = generate_from_dict(dic)
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/features/features.py", line 876, in generate_from_dict
    return {key: generate_from_dict(value) for key, value in obj.items()}
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/features/features.py", line 876, in <dictcomp>
    return {key: generate_from_dict(value) for key, value in obj.items()}
  File "/gpfswork/rech/six/commun/conda/hf-prod/lib/python3.8/site-packages/datasets/features/features.py", line 877, in generate_from_dict
    class_type = globals()[obj.pop("_type")]
KeyError: 'Image'
[2022-01-18 21:22:05,220] [INFO] [launch.py:160:sigkill_handler] Killing subprocess 95152
[2022-01-18 21:22:05,221] [ERROR] [launch.py:166:sigkill_handler] ['/gpfswork/rech/six/commun/conda/hf-prod/bin/python', '-u', './tasks/eval_harness/evaluate.py', '--local_rank=0', '--load', '/gpfsscratch/rech/six/commun/checkpoints/tr7d-1B3-alibi/checkpoints/global_step117000', '--results_path', 'tr7d-1B3-alibi-1-results.json', '--tensor-model-parallel-size', '1', '--pipeline-model-parallel-size', '1', '--vocab-file', '/gpfsssd/worksf/projects/rech/six/commun/code/eval/Megatron-DeepSpeed/data/gpt2-vocab.json', '--merge-file', '/gpfsssd/worksf/projects/rech/six/commun/code/eval/Megatron-DeepSpeed/data/gpt2-merges.txt', '--micro-batch-size', '6', '--no-load-optim', '--no-load-rng', '--inference', '--seq-length', '2048', '--adaptive_seq_len', '--eval_fp32', '--task_list', 'headqa_en', '--deepspeed', '--deepspeed_config', 'ds_config.json', '--num-layers', '-1', '--hidden-size', '-1', '--num-attention-heads', '-1', '--seq-length', '-1', '--max-position-embeddings', '-1'] exits with return code = 1
```

```
python -c "import datasets; print(datasets.__version__)"
1.15.1
```

which is the pinned version:

https://github.com/EleutherAI/lm-evaluation-harness/blob/1c52e917cf9196a72dee111ff3a107b115e9da96/setup.py#L25

Thanks.

Sorry for the late response. It's been resolved with a custom HuggingFace `datasets` loading script [lm_eval/datasets/headqa/headqa.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/datasets/headqa/headqa.py) which ignores the image features in the HuggingFace Hub version.

Thank you for reporting.