Repository: bigcode-project/bigcode-evaluation-harness
Issue: 'HumanEval' object has no attribute 'dataset'
Root cause: Validation Gap

When I evaluate human eval with llama 7b, I met this problem:
<img width="1311" alt="image" src="https://github.com/bigcode-project/bigcode-evaluation-harness/assets/60767110/23df4a5e-ce99-4810-a604-4b674ca2d563">

my script

accelerate  launch /cpfs01/shared/Group-m6/dongguanting.dgt/bigcode-evaluation-harness/main.py \
  --model "/path to my llama7b/llama-7b" \
  --tasks humaneval \
  --max_length_generation 512 \
  --do_sample True \
  --n_samples 200 \
  --batch_size 100 \
  --temperature 0.2 \
  --precision bf16 \
  --allow_code_execution \
  --use_auth_token


Hi, I can't reproduce the error, are you using `main`? maybe the dataset wasn't downloaded successfully could you try again, and make sure you don't have another package installed called HumanEval by any chance

I have faced the same problem. The problem appears to be at https://github.com/bigcode-project/bigcode-evaluation-harness/blob/68aea1805283b123a4e745e696e041c47cca4993/lm_eval/base.py#L27 If the `load_dataset` function raise no error only a warning message is produced, which is hidden by the traceback in the @dongguanting screenshot. In my case the problem was with access rights to the dataset folder. The fact that me as well as you @loubnabnl and @dongguanting  had not figured out what is wrong right away depicts that this part of the code should be refactored. The behavior of the `Task` class here is quite implicit, while Python Dzen tell us that "Explicit is better than implicit". The implication here is that the dataset attribute can be set outside of the base `Task` class. The only task which use this implication is MultiPLE https://github.com/bigcode-project/bigcode-evaluation-harness/blob/68aea1805283b123a4e745e696e041c47cca4993/lm_eval/tasks/multiple.py#L89
Note that the `load_dataset` function will be called twice here. Possible bug?

I believe the better way to allow to create the dataset outside of the `Task` is to make optional `dataset` argument in the `__init__`. The default value is `None`, if the dataset is passed then we do not execute `self.dataset = load_dataset(...)`.

To illustrate, this is how code looks like now:
```python
class Task(ABC):
    ...
    def __init__(self, stop_words=None, requires_execution=True):
        ...
        try:
            self.dataset = load_dataset(path=self.DATASET_PATH, name=self.DATASET_NAME)
        except Exception as e:
            warn(
                f"Loading the dataset failed with {str(e)}. This task will use a locally downloaded dataset, not from the HF hub."
            )
  ```
And here is what I propose:
```python
class Task(ABC):
    ...
    def __init__(self, stop_words=None, requires_execution=True, dataset=None):
        ...
        if dataset is None:
            self.dataset = load_dataset(path=self.DATASET_PATH, name=self.DATASET_NAME)
        else:
            self.dataset = dataset
  ```
  If this looks good - I am eager to double check everything and make a pool request (:

Error 
Traceback (most recent call last):
  File "main.py", line 396, in <module>
    main()
  File "main.py", line 380, in main
    results[task] = evaluator.evaluate(
  File "/workspace/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 95, in evaluate
    generations, references = self.generate_text(task_name, intermediate_generations=intermediate_generations)
  File "/workspace/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 45, in generate_text
    dataset = task.get_dataset()
  File "/workspace/bigcode-evaluation-harness/bigcode_eval/tasks/humaneval.py", line 62, in get_dataset
    return self.dataset["test"]
AttributeError: 'HumanEval' object has no attribute 'dataset'
Traceback (most recent call last):
  File "/opt/conda/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/opt/conda/lib/python3.8/site-packages/accelerate/commands/launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "/opt/conda/lib/python3.8/site-packages/accelerate/commands/launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python3.8', 'main.py', '--model', 'CodeLlama-7b-Instruct-hf', '--max_length_generation', '512', '--limit', '3', '--tasks', 'humaneval', '--temperature', '0.2', '--n_samples', '200', '--batch_size', '2', '--allow_code_execution']' returned non-zero exit status 1.

@pearlmary it seems you're having trouble loading the humaneval dataset you should've had this warning above the error
```bash
"Loading the dataset failed with {str(e)}. This task will use a locally downloaded dataset, not from the HF hub."
```
We can't throw an error because some datasets are local, hence the warning (but I'll try to make it more informative).
 Can you try installing the requirements again? we also faced this issue at one point due to a bug in a recent version of fsspec which is now fixed in the [requirements](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/bb1ed78aa5ce4f229ed87c4c6f0cbf50eb130f89/requirements.txt#L8)
 
@mu-arkhipov I'm not sure your solution addresses the issue, it's true that MultiPL-E calls the dataset again to get the stop words, but that shouldn't be an issue since it's a very small dataset and it will be cached for future calls.

I encountered the same issue as you, and I resolved it by executing 'huggingface-cli login'.

This is my problem, i try to eval 'humanevalsynthesize -python', but have a error, I don't know how to solve it, and I don't know why have this problem
```
  warnings.warn(
/root/autodl-tmp/sdft-main/bigcode-evaluation-harness/bigcode_eval/base.py:30: UserWarning: Loading the dataset failed with 'NoneType' object is not callable. This task will use a locally downloaded dataset, not from the HF hub.                 This is expected behavior for the DS-1000 benchmark but not for other benchmarks!
  warn(
Traceback (most recent call last):
  File "/root/autodl-tmp/sdft-main/bigcode-evaluation-harness/main.py", line 412, in <module>
    main()
  File "/root/autodl-tmp/sdft-main/bigcode-evaluation-harness/main.py", line 396, in main
    results[task] = evaluator.evaluate(
  File "/root/autodl-tmp/sdft-main/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 95, in evaluate
    generations, references = self.generate_text(task_name, intermediate_generations=intermediate_generations)
  File "/root/autodl-tmp/sdft-main/bigcode-evaluation-harness/bigcode_eval/evaluator.py", line 45, in generate_text
    dataset = task.get_dataset()
  File "/root/autodl-tmp/sdft-main/bigcode-evaluation-harness/bigcode_eval/tasks/humanevalpack.py", line 187, in get_dataset
    return self.dataset["test"]
AttributeError: 'HumanEvalSynthesize' object has no attribute 'dataset'
```

This issue is because of [this update](https://github.com/EleutherAI/lm-evaluation-harness/issues/1135#issuecomment-2165689312) to the datasets package which sets trust_remote_code to False as default. Unfortunately, trust_remote_code is not passed to the Task base class nor is it passed when [loading the dataset](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/6116c6a9a5672c69bd624373cfbc8938b7acc249/bigcode_eval/base.py#L28)