Repository: EleutherAI/lm-evaluation-harness
Issue: User cannot specify `data_dir` argument to `Task` constructor using the CLI 
Root cause: Documentation Deficiency

I am looking for guidance on how to use local data sources instead of the HuggingFace Hub.

Here's the approach I have thought up: I create a custom task by creating a subclass of `Task` as described in the [Task Guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/master/docs/task_guide.md), but I override the [download method](https://github.com/EleutherAI/lm-evaluation-harness/blob/23f30926f3ce738e3eee4e6be5c29fb3467e3a6e/lm_eval/base.py#LL483C9-L483C9) to load my data using `datasets.DatasetDict.load_from_disk(data_dir)`, where `data_dir` is the argument to the `Task` constructor which specifies the location on disk.

My issue is that it is not clear how to specify the `data_dir` argument to the `Task` constructor when I use the CLI in `main.py` as my entrypoint into the program.

Presumably, the arguments need to be passed to the constructor [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/23f30926f3ce738e3eee4e6be5c29fb3467e3a6e/lm_eval/tasks/__init__.py#LL391C36-L391C36), where the subclasses of `Task` are instantiated.

```python
def get_task_dict(task_name_list: List[Union[str, lm_eval.base.Task]]):
    task_name_dict = {
        task_name: get_task(task_name)(data_dir="/home/tanner/git/club/build/data/org-wide/promise/split")
        for task_name in task_name_list
        if isinstance(task_name, str)
    }
    task_name_from_object_dict = {
        get_task_name_from_object(task_object): task_object
        for task_object in task_name_list
        if not isinstance(task_object, str)
    }
    assert set(task_name_dict.keys()).isdisjoint(set(task_name_from_object_dict.keys()))
    return {**task_name_dict, **task_name_from_object_dict}
```

My question is whether this is the right approach to using local data sources, and whether this is a desired feature for the CLI.

As Task class defination, I think you could specify DATASET_PATH as "json" and set data_dir as your data_dir.  It's depends on download method of Task class  and HF datasets.
```
class Task(abc.ABC):
     ....
    def download(self, data_dir=None, cache_dir=None, download_mode=None):
       ....

        self.dataset = datasets.load_dataset(
            path=self.DATASET_PATH,
            name=self.DATASET_NAME,
            data_dir=data_dir,
            cache_dir=cache_dir,
            download_mode=download_mode,
        )

```

Does your dataset path change? If it’s at a fixed location you should be able to specify it with `DATASET_PATH = you/path/here` just like we specify datasets on the HF Hub.

@TannerSorensen were you able to figure this out? I am also interested in loading a local dataset to experiment with a custom task. Simply pointing to the local dataset in the task's .yaml doesn't work - did you find any success with your method?

@StellaAthena It would be lovely if easily loading local task datasets was possible, at least to test and debug before pushing to hub. Is this possible in the big-refactor yet?

This is now supported, though the documentation is a little lacking. Fixing the documentation is already on the to-do list, so I'll close this issue.