Repository: EleutherAI/lm-evaluation-harness
Issue: How to load local datasets?
Root cause: Documentation Deficiency

Hi there.

I wanted to load a local .json file as the dataset for evaluation, and followed the instructions provided by the official document.

> Using Local Datasets
> To load a local dataset for evaluation, you can specify data files in the dataset_kwargs field, such as the following for JSON files:
> 
> ```
> dataset_path: json
> dataset_name: null
> dataset_kwargs:
>   data_files: /path/to/my/json
> ```
```
dataset_path: json
dataset_name: null
dataset_kwargs:
  data_files: /home/lightweight_team/symbol-llm/data_for_testing/test_data/mtop/mtop.json
```

However, I encountered the following error when running the evaluation:
```
2025-05-19:19:06:50 WARNING  [api.task:132] mtop: No `generation_kwargs` specified in task config, defaulting to {'until': ['\n\n'], 'do_sample': False, 'temperature': 0}
self.config.dataset_kwargs: {'data_files': '/home/lightweight_team/symbol-llm/data_for_testing/test_data/mtop/mtop.json'}
self.config.custom_dataset: None
Generating train split: 4386 examples [00:00, 43661.94 examples/s]
2025-05-19:19:06:51 DEBUG    [api.task:883] No custom filters defined. Using default 'take_first' filter for handling repeats.
self.DATASET_PATH: json
Traceback (most recent call last):
  File "/home/lightweight_team/miniconda3/envs/symbol-llm/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/__main__.py", line 449, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/evaluator.py", line 264, in simple_evaluate
    task_dict = get_task_dict(
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 635, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 426, in load_task_or_group
    collections.ChainMap(
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 428, in <lambda>
    lambda task: self._load_individual_task_or_group(task),
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 326, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 286, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/api/task.py", line 921, in __init__
    self.task_docs = self.eval_docs
  File "/home/lightweight_team/symbol-llm/lm-evaluation-harness/lm_eval/api/task.py", line 693, in eval_docs
    raise ValueError(
ValueError: Task dataset (path=json, name=None) must have valid or test docs!
```

The dataset file exists and is in the correct JSON format (a list of {"prompt": ..., "completion": ...} objects), but the evaluation script still cannot identify the dataset properly.

Hi! You need to provide a `validation_split` or a `test_split` in the yaml.

iirc datasets defaults to creating a `train` split when loading from json, so:
```yaml
test_split: train
```

closing the issue but feel free to open!