Repository: EleutherAI/lm-evaluation-harness
Issue: --trust_remote_code does it actually do anything?
Root cause: Configuration Error

I'm running ` lm_eval --model hf --model_args pretrained=myllamamodel --device cuda:0 --batch_size 4 --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,winogrande,openbookqa --log_samples --write_out --trust_remote_code -v debug`

but it keeps on reporting this:

```
/home/ai/MiniConda3/envs/full-peft/lib/python3.10/site-packages/datasets/load.py:1486: FutureWarning: The repository for hellaswag contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/hellaswag
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
```
What am I doing wrong?

~~You aren't doing anything wrong. To suppress that warning `trust_remote_code=True` needs to be passed to the dataset constructor, which you don't have direct access to when querying the library. This is something that needs to be changed under the hood.~~

Never mind, I'm apparently wrong.


`--trust_remote_code` is meant to do this and suppress the warning, so something must be up. Reopening.

Thank you for taking a look at this!

Looks like it's because of this [line](https://github.com/huggingface/datasets/blob/af3acfdfcf76bb980dbac871540e30c2cade0cf9/src/datasets/load.py#L1491):
```python
if config.HF_DATASETS_TRUST_REMOTE_CODE and self.trust_remote_code is None:
    warnings.warn("....")
```
Not quite sure what they are going for here. Maybe the env variable currently defaults to `True`?

> Looks like it's because of this [line](https://github.com/huggingface/datasets/blob/af3acfdfcf76bb980dbac871540e30c2cade0cf9/src/datasets/load.py#L1491):
> 
> ```python
> if config.HF_DATASETS_TRUST_REMOTE_CODE and self.trust_remote_code is None:
>     warnings.warn("....")
> ```
> 
> Not quite sure what they are going for here. Maybe the env variable currently defaults to `True`?

Yea it's confusing!
Because I see that warning message, `self.trust_remote_code` must be `None`
This implies that the gated actions depending upon  `self.trust_remote_code` is not carried out.   So @haileyschoelkopf is right. It does seem like `--trust_remote_code` is likely a no-op :-(



I performed mmlu eval using datasets version 2.20.0. Even if I use the --trust_remote_code argument, a trust_remote_code related error occurs. In datasets version 2.19.2, only warnings related to trust_remote_code are raised and evaluation is performed. I recommend keeping the datasets version below 2.19.2 until internal code changes.

@baberabb
At [lm_eval/__main__.py](lm_eval/__main__.py), trust_remote_code arg was added to model_args.
and also setting HF_DATASETS_TRUST_REMOTE_CODE = True.
However, it seems that model_args is only used to load the model and not the dataset.
At [lm_eval/tasks/__init__.py](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/__init__.py), dataset configs are taken only from the yaml file of the task

Hi @abzb1 @devzzzero , I managed to track this down. Fix and description of what the issue is in #1998 !

> Hi @abzb1 @devzzzero , I managed to track this down. Fix and description of what the issue is in #1998 !

Thank you!!! W00T!

I'm running the following (for testing purposes) (on Linux):

```
accelerate launch -m lm_eval \
    --model hf \
    --model_args pretrained=openai-community/gpt2  \
    --tasks pile_openwebtext2,hellaswag,mmlu \
    --batch_size 64 \
    --trust_remote_code
```

But still get this:
```
[__main__.py:369] Passed `--trust_remote_code`, setting environment variable `HF_DATASETS_TRUST_REMOTE_CODE=true`
```

```
/home/freddy/.virtualenvs/lm-eval/lib/python3.10/site-packages/datasets/load.py:1298: FutureWarning: The repository for EleutherAI/pile contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/EleutherAI/pile
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
```
...

```
[rank4]: ValueError: BuilderConfig 'pile_openwebtext2' not found. Available: ['all', 'enron_emails', 'europarl', 'free_law', 'hacker_news', 'nih_exporter', 'pubmed', 'pubmed_central', 'ubuntu_irc', 'uspto', 'github']
```


I now see this is not an issue with `--trust_remote_code`, but with a `BuilderConfig` not found.
==> Ah, ok, it seems this doesn't work anymore, see issue #1338

> I performed mmlu eval using datasets version 2.20.0. Even if I use the --trust_remote_code argument, a trust_remote_code related error occurs. In datasets version 2.19.2, only warnings related to trust_remote_code are raised and evaluation is performed. I recommend keeping the datasets version below 2.19.2 until internal code changes.

Thanks, great quick fix :)