Repository: huggingface/lighteval
Issue: [BUG] AttributeError: 'str' object has no attribute 'category'
Root cause: Validation Gap

## Describe the bug
Encounter attribute error when running narrativeqa evaluation,

## To Reproduce
1. Cloned the repo, ran the following command:
```
lighteval accelerate \
    --model_config_path="/mnt/c/Users/Documents/lighteval/lighteval/examples/model_configs/base_model.yaml" \
    --tasks "helm|narrativeqa|5|0" \
    --output_dir output_dir
```

Full log:
```
WARNING:lighteval.logging.hierarchical_logger:main: (0, Namespace(subcommand='accelerate', model_config_path='/mnt/c/Users/Documents/lighteval/lighteval/examples/model_configs/base_model.yaml', model_args=None, max_samples=None, override_batch_size=-1, job_id='', output_dir='output_dir', save_details=False, push_to_hub=False, push_to_tensorboard=False, public_run=False, results_org=None, use_chat_template=False, system_prompt=None, dataset_loading_processes=1, custom_tasks=None, tasks='helm|narrativeqa|5|0', cache_dir=None, num_fewshot_seeds=1)),  {
WARNING:lighteval.logging.hierarchical_logger:  Test all gather {
WARNING:lighteval.logging.hierarchical_logger:    Test gather tensor
WARNING:lighteval.logging.hierarchical_logger:    gathered_tensor tensor([0], device='cuda:0'), should be [0]
WARNING:lighteval.logging.hierarchical_logger:  } [0:00:01.915155]
WARNING:lighteval.logging.hierarchical_logger:  Model loading {
WARNING:lighteval.logging.hierarchical_logger:    Tokenizer truncation and padding size set to the left side.
WARNING:lighteval.logging.hierarchical_logger:    We are not in a distributed setting. Setting model_parallel to False.
WARNING:lighteval.logging.hierarchical_logger:    Model parallel was set to False, max memory set to None and device map to None
`low_cpu_mem_usage` was None, now set to True since model is quantized.
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████| 4/4 [00:10<00:00,  2.57s/it]
WARNING:lighteval.logging.hierarchical_logger:    Compiling the model
WARNING:lighteval.logging.hierarchical_logger:  } [0:00:14.170301]
WARNING:lighteval.logging.hierarchical_logger:  Tasks loading {
WARNING:lighteval.logging.hierarchical_logger:    If you want to use extended_tasks, make sure you installed their dependencies using `pip install -e .[extended_tasks]`.
WARNING:lighteval.logging.hierarchical_logger:    lighteval/narrative_qa_helm default
WARNING:lighteval.logging.hierarchical_logger:  } [0:00:00.006673]
WARNING:lighteval.logging.hierarchical_logger:} [0:00:16.121220]
Traceback (most recent call last):
  File "/home/project/anaconda3/envs/lighteval/bin/lighteval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/__main__.py", line 58, in cli_evaluate
    main_accelerate(args)
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/logging/hierarchical_logger.py", line 175, in wrapper
    return fn(*args, **kwargs)
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/main_accelerate.py", line 78, in main
    pipeline = Pipeline(
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/pipeline.py", line 125, in __init__
    self._init_tasks_and_requests(tasks=tasks)
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/pipeline.py", line 182, in _init_tasks_and_requests
    task_dict = Registry(cache_dir=self.pipeline_parameters.env_config.cache_dir).get_task_dict(
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/tasks/registry.py", line 152, in get_task_dict
    tasks_dict[task_name] = task_class()
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/tasks/registry.py", line 267, in __init__
    super().__init__(name, cfg, cache_dir=cache_dir)
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/tasks/lighteval_task.py", line 208, in __init__
    ignored = [metric for metric in self.metrics if metric.category == MetricCategory.IGNORED]
  File "/home/project/anaconda3/envs/lighteval/lib/python3.10/site-packages/lighteval/tasks/lighteval_task.py", line 208, in <listcomp>
    ignored = [metric for metric in self.metrics if metric.category == MetricCategory.IGNORED]
AttributeError: 'str' object has no attribute 'category'
```

## Expected behavior
Running evaluation on narrative qa. Was able to run quite a few tests before this.

## Version info
OS: Ubuntu (WSL)
Python: 3.10
Cuda: 12.2
Commit:[c83daef](https://github.com/huggingface/lighteval/pull/298/commits/c83daef84e4a95a49674685f46968e3532d777e0)

Hi @Vanessa-Taing , I believe this should be fixed with the latest release, can you check?

Hi @clefourrier, this issue still exists on Version: 0.9.2. Here is a snippet of the error log:

```
│   200 │   │   # Metrics                                                      │
│   201 │   │   self.metrics = as_list(cfg.metric)                             │
│   202 │   │   self.suite = as_list(cfg.suite)                                │
│ ❱ 203 │   │   ignored = [metric for metric in self.metrics if metric.categor │
│   204 │   │                                                                  │
│   205 │   │   if len(ignored) > 0:                                           │
│   206 │   │   │   logger.warning(f"Not implemented yet: ignoring the metric  │
╰──────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'str' object has no attribute 'category'
```

I was trying to run lighteval|glue:mrpc

hi ! just took a look, it's because of an issue [here](https://github.com/huggingface/lighteval/blob/c8714563a114acca57744cdb74b1878107eeef57/src/lighteval/tasks/default_tasks.py#L7676), the metric needs to be an object from the Metrics list, simply change it to:

```
Metrics.loglikelihood_f1
```

don't have bandwidth right now but would be happy to merge a quick PR fixing this :)