Repository: EleutherAI/lm-evaluation-harness
Issue: PudMedQA tasks require trust_remote_code=True argument, but argument doesnt work
Root cause: Configuration Error

Error:

`The repository for bigbio/pubmed_qa contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/bigbio/pubmed_qa.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/utils.py", line 402, in _wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/evaluator.py", line 236, in simple_evaluate
    task_dict = get_task_dict(tasks, task_manager)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 618, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 414, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 314, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/tasks/__init__.py", line 280, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/task.py", line 822, in __init__
    self.download(self.config.dataset_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/lm_eval/api/task.py", line 929, in download
    self.dataset = datasets.load_dataset(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1849, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1731, in dataset_module_factory
    raise e1 from None
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1681, in dataset_module_factory
    ).get_module()
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 1331, in get_module
    trust_remote_code = resolve_trust_remote_code(self.trust_remote_code, self.name)
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 138, in resolve_trust_remote_code
    raise ValueError(
ValueError: The repository for bigbio/pubmed_qa contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/bigbio/pubmed_qa.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.

my code
`        model_args = {
            "pretrained": model_name,
            "revision": checkpoint,  
            "dtype": "float16",   
            "parallelize": True,
            "trust_remote_code": True
        }`

`        results = simple_evaluate(
            model="hf",                 
            model_args=model_args,
            tasks=tasks,
            batch_size=batch_size,
            confirm_run_unsafe_code=True,
        )`

I am running this in the background over multiple revisions so I cannot sit and enter Y each time. Why doesn't doing what it says fix the issue?

confirm_run_unsafe_code was added in a recent commit as an attempt to fix this, but it clearly didnt work. 0.4.7 (latest release) still has this broken

Hi! Have you tried `--trust_remote_code`? If you're using `simple_evaluate` as an entry point you can set the HF `datasets` environment variable  like [so](https://github.com/EleutherAI/lm-evaluation-harness/blob/9dda03d6be6c94cc803b6189302a8a148c5e4d12/lm_eval/__main__.py#L377)
The `confirm_run_unsafe_code` is mainly for code tasks, to run code generated by the model. I should make that clearer.

I am running into the same error even if I have passed ```--trust_remote_code=True```.

probably better to set `datasets.config.HF_DATASETS_TRUST_REMOTE_CODE = True` at the top of your script


Thanks for the reply! I ended up solving the problem by adding ```--trust_remote_code=True``` in ```api/task.py``` file when it loads dataset in.

> Thanks for the reply! I ended up solving the problem by adding `--trust_remote_code=True` in `api/task.py` file when it loads dataset in.

please in which part of the file did you add this?

> > Thanks for the reply! I ended up solving the problem by adding `--trust_remote_code=True` in `api/task.py` file when it loads dataset in.
> 
> please in which part of the file did you add this?

At here:
https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L931
Just add ```--trust_remote_code=True```

> > > Thanks for the reply! I ended up solving the problem by adding `--trust_remote_code=True` in `api/task.py` file when it loads dataset in.
> > 
> > 
> > please in which part of the file did you add this?
> 
> At here: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L931 Just add `--trust_remote_code=True`

Thank you!