Repository: huggingface/lighteval
Issue: [BUG] pydantic throws error with custom evaluator
Root cause: Interface Contract Mismatch

## Describe the bug

When running lighteval with a custom evaluator, pydantic throws an error.

## To Reproduce

Custom class (per https://huggingface.co/docs/lighteval/en/evaluating-a-custom-model):
```python
from lighteval.models.abstract_model import LightevalModel

class DumpDataset(LightevalModel):
    def __init__(self, config):
        super().__init__(config)
        pass

    def greedy_until(self, requests, max_tokens=None, stop_sequences=None):
        for request in requests:
            print(request)

    def loglikelihood(self, requests, log=True):
        # Implement loglikelihood computation
        pass

    def loglikelihood_rolling(self, requests):
        # Implement rolling loglikelihood computation
        pass

    def loglikelihood_single_token(self, requests):
        # Implement single token loglikelihood computation
        pass
```

Command:
```bash
$ lighteval custom "dump" custom.py "lighteval|math_500"
[2025-05-20 20:59:18,048] [    INFO]: PyTorch version 2.6.0 available. (config.py:54)
INFO 05-20 20:59:21 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-20 20:59:21 [__init__.py:239] Automatically detected platform cuda.
╭─────────────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────────────╮
│ /home/aiscuser/.local/lib/python3.10/site-packages/lighteval/main_custom.py:112 in custom                                                                                │
│                                                                                                                                                                          │
│   109 │   )                                                                                                                                                              │
│   110 │                                                                                                                                                                  │
│   111 │   parallelism_manager = ParallelismManager.CUSTOM                                                                                                                │
│ ❱ 112 │   model_config = CustomModelConfig(model=model_name, model_definition_file_path=model_                                                                           │
│   113 │                                                                                                                                                                  │
│   114 │   pipeline_params = PipelineParameters(                                                                                                                          │
│   115 │   │   launcher_type=parallelism_manager,                                                                                                                         │
│                                                                                                                                                                          │
│ /home/aiscuser/.local/lib/python3.10/site-packages/pydantic/main.py:253 in __init__                                                                                      │
│                                                                                                                                                                          │
│    250 │   │   """                                                                                                                                                       │
│    251 │   │   # `__tracebackhide__` tells pytest and some other tools to omit this function fr                                                                          │
│    252 │   │   __tracebackhide__ = True                                                                                                                                  │
│ ❱  253 │   │   validated_self = self.__pydantic_validator__.validate_python(data, self_instance                                                                          │
│    254 │   │   if self is not validated_self:                                                                                                                            │
│    255 │   │   │   warnings.warn(                                                                                                                                        │
│    256 │   │   │   │   'A custom validator is returning a value other than `self`.\n'                                                                                    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: 2 validation errors for CustomModelConfig
model_name
  Field required [type=missing, input_value={'model': 'dump', 'model_...ile_path': 'custom2.py'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/missing
model
  Extra inputs are not permitted [type=extra_forbidden, input_value='dump', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden
```

## Expected behavior
This should run my custom evaluator.

## Version info
Ubuntu 22.04, Python 3.10.12

Versions:
```bash
$ pip list | egrep "pydantic|lighteval"
lighteval                                0.9.2
pydantic                                 2.11.4
pydantic_core                            2.33.2
```

Hey ! I opened a PR to fix this :)

Oh, you beat me! I had fixed this locally but didn't get through to pushing it up. I'll make comments over there.