Repository: huggingface/lighteval
Issue: [FT] Custom model to TransformersModel
Root cause: Validation Gap

Hello everyone,

First of all, thanks for the amazing work.

## Issue encountered
I have been trying to use lighteval but I'm facing an issue.

From my current understanding, it is only possible to pass a `pretrained` string to `TransformersModelConfig`, which means the model has to be already present on the hub and can't be modified in any way before using it for eval.

I tried passing `model` directly to `Pipeline` but I get the following error:
```python
AttributeError: 'TransformersModel' object has no attribute 'generation_config_dict'
```

## Solution/Feature
Instead of passing a `pretrained` string to `TransformersModelConfig`, I was wondering if it's possible to pass a `torch.nn.Module` and use that for evaluation purpose.
The idea is to pass to `lighteval` a transformed HF model (e.g., after applying quantization through third-party libraries).

Please let me know if I'm using the library wrong or misunderstanding something.

Many thanks,
Giuseppe 


Hi! Thanks for the kind words!

Easy way is simply to upload your quantized model to the hub, or save it locally and load from there.
You can also use lighteval programmatically (https://huggingface.co/docs/lighteval/using-the-python-api) and load a model directly in Python - you might need to inherit the LightevalModel abstract class :)

Yes I have been playing with it through the python API, passing `model` to `Pipeline` but I get the error below:

```python
AttributeError: 'TransformersModel' object has no attribute 'generation_config_dict'
```
This happens also when loading a model from HF and then passing it to pipeline (no quantization or any other modification really to a normal `AutoModelForCasualLM`)

After some experimenting, the issue seems to be due to a mismatch in how `TransformersModel` gets initalized when calling init compared to what appens with `from_model`.
There are a few configuration missing in `from_model` that cause the error and some numerical discrepancies once that error is fixed.

Also, unrelated to this, I noticed some smaller issues around some hyper-params like `override_batch_size`. If I just remove it, it goes to None and then there's an error when checking if `override_batch_size > 0`.

A similar issue seems to exist around the `accelerator` parameter. Related to this, in the API example above, the accelerator variable is created but never used.

I can post a script to reproduce but I basically copy-pasted the API tutorial you linked with some minor modifications here and there.

I second @Giuseppe5 's observations.
Encountered the same exception, when trying to initialize `TransformersModel` via `from_model` path (when provided a HF transformers module).
The root cause seems missing treatment of several aspects, one of which is `generation_config_dict`.
But I'm afraid there are other issues here, as I keep stumbling on additional exceptions.

@clefourrier, I definitely think it's a bug.

I ended up doing a very ugly looking fix in a personal fork, but I would not recommend that for anything if not to understand what's missing to get custom models to "work", but a lot of clean-up is needed starting from there.

I could try to put that into words if there's the chance to get a PR for that.

@Giuseppe5 ,
Did you manage to eventually produce a running pipeline that actually outputs correct results?
I did clean up the bugs to make it run, but the outputs do not match those obtained by a sanity run via the CLI (e.g., `lighteval vllm ...`).

Yes, this is the commit with my changes.
Again, take it for what it is, a brute-force attempt to fix the isssue:
https://github.com/huggingface/lighteval/commit/a9560964aa0cd0fd210d56ff47a0890ebd339fbc

Thank you for sharing the code!
My changes were similar, but it didn't help.

These changes were tested against results from `lm_eval` on same benchmarks and I got comparable numbers. I remember there were some small details to get from "running" to "numerically correct", but I can't quite recall what that was, but it's small things that can compromise the final output of the benchmarks

I would like to see this feature. Consider a case where I have a model which I am reluctant to share to HF Hub. I want to evaluate everything offline. In such cases, I am looking to pass the model (nn.Module) to this library to get the benchmark on different tasks. 

hey ! you can use the python API:

https://huggingface.co/docs/lighteval/en/using-the-python-api

I want to pass nn.Module instance not the model card name. 

you can pass an already initialized HF model:

```py
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.pipeline import Pipeline, PipelineParameters, ParallelismManager
from lighteval.models.endpoints.inference_providers_model import (
    InferenceProvidersModelConfig,
)
from lighteval.models.transformers.transformers_model import TransformersModel
import torch
from transformers import AutoModelForCausalLM, GenerationConfig

MODEL_NAME = "meta-llama/Meta-Llama-3-8B-Instruct"
PROVIDER = "hf-inference"
BENCHMARKS = "lighteval|gsm8k|0|0"

evaluation_tracker = EvaluationTracker(output_dir="./results")
pipeline_params = PipelineParameters(
    use_chat_template=True, launcher_type=ParallelismManager.NONE, max_samples=None
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3b-Instruct", torch_dtype=torch.bfloat16, device_map="auto"
)

# Configure generation parameters
generation_config = GenerationConfig(
    max_new_tokens=2048,
)
model.generation_config = generation_config
model = TransformersModel.from_model(model)
pipeline = Pipeline(
    model=model,
    pipeline_parameters=pipeline_params,
    evaluation_tracker=evaluation_tracker,
    tasks=BENCHMARKS,
)

pipeline.evaluate()
results = pipeline.get_results()["results"]
print(results)
```

However, this feature was still in development so you will have to change a few lines in the `from_model` function.
What needs to change can be found in this PR #714 

This seems to be fixed, so I'm closing this issue.