Repository: EleutherAI/lm-evaluation-harness
Issue: Task Configuration didn't work
Root cause: External Dependency Breakage

Hi, thanks for your great contributions!
When I run the command:
lm_eval  --tasks xxx(sst2,hellaswag,mmlu) --model hf  --model_args pretrained=/local/path/to/model --device cuda:1  --batch_size 20
I found the task to evaluate is anli, I guess maybe the task configuration didn’t work. There might be something wrong with my operation. Are there any suggestions?



Actually, I add some codes here:
https://github.com/EleutherAI/lm-evaluation-harness/blob/b93c3bcbf30109c40cd19ee862161702920b9c27/lm_eval/api/task.py#L716
to print the required dataset

But with the command: lm_eval --tasks xxx(sst2,hellaswag,mmlu) --model hf --model_args pretrained=/local/path/to/model --device cuda:1 --batch_size 20
the download function is to download anli task, I don't know why this happened.

Hi,

I'm not sure what you mean by --task xxx(sst2,hellaswag,mmlu)

Could talk more about what you wanted to try?

Thanks to your response. Because I need to use your code in a machine that can't access huggingface, so I want to download the task datasets needed. I add some print codes
in https://github.com/EleutherAI/lm-evaluation-harness/blob/b93c3bcbf30109c40cd19ee862161702920b9c27/lm_eval/api/task.py#L716 
to print self.DATASET_PATH and self.DATASET_NAME.
But when I run the command:
lm_eval --tasks sst2 --model hf --model_args pretrained=/local/path/to/model --device cuda:1 --batch_size 20
the printed information is: self.DATASET_NAME=None and self.DATASET_PATH=anli
I don't know why this happened.

Not sure why that is the case. Looks like a bug i should investigate.

But if you need to find the dataset_name and path you could always do a search in lm_eval/tasks/

sst2 is in glue/
hellaswag is hellaswag/
mmlu is in mmlu/default/

Actually, it seems to dowanload all the task datasets, I don't know why this happended. I just want to download the dataset for the task I want to evaluate.

2024-01-19:10:17:37,859 INFO     [utils.py:147] Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-01-19:10:17:37,859 INFO     [utils.py:159] NumExpr defaulting to 8 threads.
2024-01-19:10:17:38,714 INFO     [config.py:58] PyTorch version 2.0.1 available.
Namespace(model='hf', tasks='sst2', model_args='pretrained=/home/yangwanli/data/llama-7b', num_fewshot=None, batch_size='2', max_batch_size=None, device='cuda:1', output_path=None, limit=None, use_cache=None, decontamination_ngrams_path=None, check_integrity=False, write_out=False, log_samples=False, show_config=False, include_path=None, gen_kwargs=None, verbosity='INFO')
2024-01-19:10:17:45,621 INFO     [__main__.py:156] Verbosity set to INFO
None
anli
2024-01-19:10:18:38,294 WARNING  [__init__.py:185] Unexpected error loading config in
                                 /home/yangwanli/lm-evaluation-harness/lm_eval/tasks/benchmarks/t0_eval.yaml
                                 Config will not be added to registry
                                 Error: to_dict() got an unexpected keyword argument 'keep_callable'
                                 Traceback: Traceback (most recent call last):
  File "/home/yangwanli/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 173, in include_task_folder
    register_configurable_group(config, yaml_path)
  File "/home/yangwanli/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 75, in register_configurable_group
    base_config = task_obj._config.to_dict(keep_callable=True)
TypeError: to_dict() got an unexpected keyword argument 'keep_callable'

None
anli
2024-01-19:10:19:33,526 WARNING  [__init__.py:185] Unexpected error loading config in
                                 /home/yangwanli/lm-evaluation-harness/lm_eval/tasks/benchmarks/flan/flan_anli.yaml
                                 Config will not be added to registry
                                 Error: to_dict() got an unexpected keyword argument 'keep_callable'
                                 Traceback: Traceback (most recent call last):
  File "/home/yangwanli/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 173, in include_task_folder
    register_configurable_group(config, yaml_path)
  File "/home/yangwanli/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 75, in register_configurable_group
    base_config = task_obj._config.to_dict(keep_callable=True)
TypeError: to_dict() got an unexpected keyword argument 'keep_callable'

ARC-Easy
ai2_arc
Downloading readme: 100%|███████████████████████████████████████████████████████████████████████████████| 9.00k/9.00k [00:00<00:00, 4.11MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████| 331k/331k [00:02<00:00, 160kB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████| 346k/346k [00:01<00:00, 341kB/s]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████| 86.1k/86.1k [00:00<00:00, 108kB/s]
Generating train split: 100%|██████████████████████████████████████████████████████████████████| 2251/2251 [00:00<00:00, 71569.51 examples/s]
Generating test split: 100%|██████████████████████████████████████████████████████████████████| 2376/2376 [00:00<00:00, 128961.98 examples/s]
Generating validation split: 100%|██████████████████████████████████████████████████████████████| 570/570 [00:00<00:00, 119885.33 examples/s]
2024-01-19:10:20:01,050 WARNING  [__init__.py:185] Unexpected error loading config in
                                 /home/yangwanli/lm-evaluation-harness/lm_eval/tasks/benchmarks/flan/flan_arc.yaml
                                 Config will not be added to registry
                                 Error: to_dict() got an unexpected keyword argument 'keep_callable'
                                 Traceback: Traceback (most recent call last):
  File "/home/yangwanli/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 173, in include_task_folder
    register_configurable_group(config, yaml_path)
  File "/home/yangwanli/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 75, in register_configurable_group
    base_config = task_obj._config.to_dict(keep_callable=True)
TypeError: to_dict() got an unexpected keyword argument 'keep_callable'

2024-01-19:10:20:01,065 WARNING  [templates.py:384] Tried instantiating `DatasetTemplates` for gsmk boolq, but no prompts found. Please ignore this warning if you are creating new prompts for this dataset.
2024-01-19:10:20:01,068 WARNING  [templates.py:384] Tried instantiating `DatasetTemplates` for EleutherAI/asdiv, but no prompts found. Please ignore this warning if you are creating new prompts for this dataset.
anatomy
hails/mmlu_no_train
/home/yangwanli/anaconda3/lib/python3.9/site-packages/datasets/load.py:1429: FutureWarning: The repository for hails/mmlu_no_train contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hails/mmlu_no_train
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
Downloading builder script: 100%|███████████████████████████████████████████████████████████████████████| 5.86k/5.86k [00:00<00:00, 4.28MB/s]
Downloading readme: 100%|████████████████████████████████████████████████████████████████████████████████████| 420/420 [00:00<00:00, 196kB/s]
Downloading data:  42%|███████████████████████████████████▏                                               | 70.5M/166M [01:14<01:38, 973kB/s]Downloading data:  42%|███████████████████████████████████▏



Can you provide the actual CLI command you are running?

It looks like you are running the `t0_eval` grouping of tasks, which does include ANLI as a subtask.

It does look as if there is a separate bug from #1315 , cc @lintangsutawika 

I think this may have something to do with promptsource-based configs. I think we had a note in TaskConfig about this in the to_dict method.

Checking in on this--I believe this should be fully fixed. We had a bug where a couple extra datasets were getting downloaded upon the first time using the library, but this no longer occurs, and `keep_callable` as a config-loading kwarg has now been around for a little while.