Repository: EleutherAI/lm-evaluation-harness
Issue: Error when evaluating TruthfulQA
Root cause: Environment Incompatibility

I'm using the command below:
```bash
lm-eval \
    --model hf \
    --model_args trust_remote_code=True,pretrained="$ckpt",parallelize=True \
    --tasks truthfulqa \
    --num_fewshot 0 \
    --device cuda \
    --output_path "$output_path/truthfulqa.json"
```
It worked fine a few days before, but today I got:
```
2024-01-05:05:35:17,853 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.        
2024-01-05:05:35:17,853 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.                           
2024-01-05:05:35:17,853 INFO     [utils.py:160] NumExpr defaulting to 8 threads.                                                                                            
2024-01-05:05:35:19,451 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmpr8ozuzzn                                                                     
2024-01-05:05:35:19,451 INFO     [instantiator.py:76] Writing /tmp/tmpr8ozuzzn/_remote_module_non_scriptable.py                                                             
[2024-01-05 05:35:25,268] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)                                                     
2024-01-05:05:35:29,214 INFO     [__main__.py:156] Verbosity set to INFO                                                                                                    
2024-01-05:05:35:34,147 WARNING  [__init__.py:178] Some tasks could not be loaded due to missing dependencies. Run with `--verbosity DEBUG` for full details.               
2024-01-05:05:35:38,894 WARNING  [__init__.py:178] Some tasks could not be loaded due to missing dependencies. Run with `--verbosity DEBUG` for full details.               
2024-01-05:05:35:38,895 INFO     [__main__.py:229] Selected Tasks: ['truthfulqa']                                                                                           
Downloading and preparing dataset None/generation to /root/.cache/huggingface/datasets/parquet/generation-c8e87328d182c991/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b
934492fdef7d8d6f236ec...                                                                                                                                                    
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1095.69it/s]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 403.49it/s]
Traceback (most recent call last):                                                                                                                                          
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1873, in _prepare_split_single                                                                   
    writer.write_table(table)                                                                                                                                               
  File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_writer.py", line 568, in write_table                                                                         
    pa_table = table_cast(pa_table, self._schema)                                                                                                                           
  File "/usr/local/lib/python3.10/dist-packages/datasets/table.py", line 2290, in table_cast                                                                                
    return cast_table_to_schema(table, schema)                                                                                                                              
  File "/usr/local/lib/python3.10/dist-packages/datasets/table.py", line 2248, in cast_table_to_schema                                                                      
    raise ValueError(f"Couldn't cast\n{table.schema}\nto\n{features}\nbecause column names don't match")                                                                    
ValueError: Couldn't cast                                                                                                                                                   
question: string                                                                                                                                                            
mc1_targets: struct<choices: list<item: string>, labels: list<item: int32>>                                                                                                 
  child 0, choices: list<item: string>                                                                                                                                      
      child 0, item: string                                                                                                                                                 
  child 1, labels: list<item: int32>                                                                                                                                        
      child 0, item: int32                                                                                                                                                  
mc2_targets: struct<choices: list<item: string>, labels: list<item: int32>>                                                                                                 
  child 0, choices: list<item: string>                                                                                                                                      
      child 0, item: string                                                                                                                                                 
  child 1, labels: list<item: int32>                                                                                                                                        
      child 0, item: int32                                                            
-- schema metadata --
huggingface: '{"info": {"features": {"question": {"dtype": "string", "_ty' + 382
to
{'type': Value(dtype='string', id=None), 'category': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'best_answer': Value(dtype='string', id=None), 'correct_answers': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'incorrect_answers': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'source': Value(dtype='string', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/lm-eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/share/yuanye/lm-evaluation-harness/lm_eval/__main__.py", line 231, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/share/yuanye/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/share/yuanye/lm-evaluation-harness/lm_eval/evaluator.py", line 122, in simple_evaluate
    task_dict = lm_eval.tasks.get_task_dict(tasks)
  File "/share/yuanye/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 239, in get_task_dict
    task_obj = get_task_dict(task_name)
  File "/share/yuanye/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 259, in get_task_dict
    task_name: get_task(task_name=task_element, config=config),
  File "/share/yuanye/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 201, in get_task
    return TASK_REGISTRY[task_name](config=config)
  File "/share/yuanye/lm-evaluation-harness/lm_eval/api/task.py", line 615, in __init__
    self.download(self.config.dataset_kwargs)
  File "/share/yuanye/lm-evaluation-harness/lm_eval/api/task.py", line 710, in download
    self.dataset = datasets.load_dataset(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1746, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1891, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

How to solve this problem?

I found that I forgot to upgrade my `datasets` package. Closed.