Repository: EleutherAI/lm-evaluation-harness
Issue: task_name=None issue when loading local dataset.
Root cause: Validation Gap

I was evaluating a model on a [local dataset](https://github.com/EleutherAI/lm-evaluation-harness/blob/ebe7226ebfb8d11a9fb8d6b53eb65891f895c633/docs/new_task_guide.md#using-local-datasets), which have worked properly until recently, but I don't know if I changed or updated something in my enviroment, that now I get the following error:

```
024-08-28:15:30:50,370 INFO     [evaluator.py:469] Running loglikelihood requests
Running loglikelihood requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500/500 [00:01<00:00, 448.05it/s]
Traceback (most recent call last):
  File "/home/gurbizu/LLM/venv-lm-eval/bin/lm_eval", line 33, in <module>
    sys.exit(load_entry_point('lm-eval', 'console_scripts', 'lm_eval')())
  File "/home/gurbizu/LLM/lm-evaluation-harness/lm_eval/__main__.py", line 389, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/gurbizu/LLM/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/gurbizu/LLM/lm-evaluation-harness/lm_eval/evaluator.py", line 305, in simple_evaluate
    results = evaluate(
  File "/home/gurbizu/LLM/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/gurbizu/LLM/lm-evaluation-harness/lm_eval/evaluator.py", line 595, in evaluate
    results_agg, group_agg = prepare_print_tasks(task_dict, results)
  File "/home/gurbizu/LLM/lm-evaluation-harness/lm_eval/evaluator_utils.py", line 276, in prepare_print_tasks
    task_agg[name]["alias"] = tab_string + alias
TypeError: can only concatenate str (not "NoneType") to str
```

it seems that alias is None, which comes from task_name being None:
 {'winogrande_en': ConfigurableTask(task_name=None,output_type=multiple_choice,num_fewshot=None,num_samples=250)}


I have updated the enviroment and start from a clean harness instalation, but I have no idea what's failing.

The code works properly with the online version of winogrande.

Hi! Would you be able to share the config + how the local dataset is saved with us so we can better debug this?

Sure.

This is the yaml config file:
```
task: winogrande_en
dataset_path: json
dataset_name: null
dataset_kwargs: 
  data_files: /home/gurbizu/LLM/winogrande/winogrande_en_250.json
training_split: null
validation_split: null
test_split: train
output_type: multiple_choice
doc_to_text: !function preprocess_winogrande.doc_to_text
doc_to_target: !function preprocess_winogrande.doc_to_target
doc_to_choice: !function preprocess_winogrande.doc_to_choice
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0
```

and the json file looks like this:
```
{"sentence": "The woman kept the bikini but returned the top, because the _ was the right size.", "option1": "bikini", "option2": "top", "answer": "1"}
{"sentence": "The flour spills out when we use an ancient family recipe and pour three teacups of flour into our bowl. The _ is too small.", "option1": "teacup", "option2": "bowl", "answer": "2"}
{"sentence": "The wages at the factory were lower than the wages at the plant since it was more safe to work at the _ .", "option1": "plant", "option2": "factory", "answer": "2"}
{"sentence": "Betty had more burns on their hands than Amy because _ worked as a welder.", "option1": "Betty", "option2": "Amy", "answer": "1"}
{"sentence": "A background check was performed on Natalie by Lindsey due to _ being a nanny.", "option1": "Natalie", "option2": "Lindsey", "answer": "1"}
{"sentence": "Mary felt a great deal of guilt for stealing various items from Rachel so _ forgave her.", "option1": "Mary", "option2": "Rachel", "answer": "2"}
{"sentence": "The toddler kicked the hard furniture and soft toy, but only the _ caused pain.", "option1": "furniture", "option2": "toy", "answer": "1"}
```

the local dataset is working fine for my coworkers, and it was working fine for me a few weeks too, but I updated the enviroment, and it stopped working. not sure why. I'm using Python 3.8.10 btw.

Hi! Can't seem to reproduce the error (on python 3.12). I copied the example json samples you provided and it seems to run fine. Maybe a duplicate yaml with same task/group name somewhere in lm_eval/tasks folder? You could also try running with `--verbosity DEBUG` to check if any errors show up.

```yaml
task: winogrande_en
dataset_path: json
dataset_name: null
dataset_kwargs:
  data_files: /test.jsonl
training_split: null
validation_split: null
test_split: train
output_type: multiple_choice
doc_to_text: "{{ sentence }}"
doc_to_target: "{{answer|int - 1 }}"
doc_to_choice: |
  {% set options = [option1, option2] %}
  {{ options }}
metric_list:
  - metric: acc
    aggregation: mean
    higher_is_better: true
metadata:
  version: 1.0
```
  
  

Thanks for your taking the time.
I'll keep digging into that direction of folder structure and path names.

> Thanks for your taking the time. I'll keep digging into that direction of folder structure and path names.

oh. A simple check would be to change the `task` to test123 or something and see if that works. 

I found the reason I had that issue. I was calling to the dataset from another coworker home directory where the datasets were. I was passing `--tasks /home/ander/LLM/lm-evaluation-harness/lm_eval/tasks/mmlu_eu` but from my home directory and my own cloned lm-harness. I was testing with `task` test123 but I placed that outside `lm-evaluation-harness/lm_eval/tasks/` directory, and I kept getting the same error. But when I moved the task inside that folder it was solved.

I know I did not follow the [guidelines](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md) properly, and I did not use `--include_path` but the weird thing is that I did not get any error like "task not found", and the dataset config was read properly:
`2024-09-03:12:48:04,029 INFO     [__main__.py:383] Selected Tasks: [{'task': 'test444', 'dataset_path': 'orai-nlp/orai_multi_mmlu_eu', 'training_split': None, 'validation_split': None, 'test_split': 'test', 'output_type': 'multiple_choice', 'doc_to_text': '{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nErantzuna:', 'doc_to_choice': ['A', 'B', 'C', 'D'], 'doc_to_target': 'answer', 'metric_list': [{'metric': 'acc', 'aggregation': 'mean', 'higher_is_better': True}], 'metadata': {'version': 0.0}}]
2` 
and it was running requests:
`2024-09-03:12:48:10,256 INFO     [evaluator.py:471] Running loglikelihood requests
Running loglikelihood requests (or at least knows how many examples were in the : 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1080/1080 [00:08<00:00, 132.92it/s]
`
but then somewhere in between or while calculating/printing results it breaks.

It would be great to make the tool work properly once it was able to reads the config file, or make it that it raise an error earlier while reading the config file and print a more human-readable error.

But since the error I caused is solved, I'm closing this issue. 😊