Repository: huggingface/lighteval
Issue: Question about Request Count and Generative Sampling
Root cause: Architectural Constraint

Hi everyone,

First off, thanks for this great library! I've been using it extensively recently and really appreciate the work that has gone into it. While analyzing the code, I came across a couple of questions and would love your insights.

1. Request Count for Metrics
I'm evaluating a custom community summarization task with the following metrics: BERTScore, METEOR, BLEU, ROUGE-1/2/L, Extractiveness, and a custom LLM-as-Judge metric.

Given a set of samples, I noticed that the number of requests being generated is twice the number of samples. I have two metric categories: `MetricCategory.GENERATIVE` and `MetricCategory.LLM_AS_JUDGE`. Does this mean the model gets called twice per sample, effectively doubling costs when using an endpoint? I see that LiteLLM has caching, but even when running the model locally, it seems like each sample is processed twice thus generating an overhead.

Would switching my judge metric to MetricCategory.GENERATIVE prevent this duplication, or is there a better way to ensure only one evaluation pass per sample? And would this change have any other side effect that I am not seeing currently

2. Generative Sampling Behavior
I want to use sampling (`do_sample=True`) when generating locally, but the setting gets overwritten unless the metric category is `MetricCategory.GENERATIVE_SAMPLING`. To work around this, I currently instantiate my metrics and then dynamically change the category, but this feels like a hack.

However, when I do this, I suddenly see three times the number of requests—one for `GENERATIVE`, one for `GENERATIVE_SAMPLING`, and one for `LLM_AS_JUDGE`. I'm guessing something is off in how I'm modifying the category dynamically.

Is there a cleaner way to enable sampling without unintended extra requests?

Thanks in advance for your help! I really appreciate your time and insights.

1. If the parameters for your 2 metrics (in terms of sampling/generation length/stop tokens/... are the same), it shouldn't but it's possible we messed up something. Good question for the category change, cc @NathanHB ?
2. You need a GENERATIVE_SAMPLING metric to do sampling - GENERATIVE will do greedy, so if you ask for both you'll get both behaviors (one in greedy and one in sampling, possibly with temperature change, etc). You could add your metrics as custom metrics, which would likely be the easiest. Can you give a code sample example?

Thank you for your time!

> 1. If the parameters for your 2 metrics (in terms of sampling/generation length/stop tokens/... are the same), it shouldn't but it's possible we messed up something. Good question for the category change, cc @NathanHB?
> 2. You need a GENERATIVE_SAMPLING metric to do sampling - GENERATIVE will do greedy, so if you ask for both you'll get both behaviors (one in greedy and one in sampling, possibly with temperature change, etc). You could add your metrics as custom metrics, which would likely be the easiest. Can you give a code sample example?

Currently I see that there are three different metric categories associated with the requests, the original GENERATIVE category of most of my metrics (even though I dynamically overwrite them), the GENERATIVE_SAMPLING category that I assign to metric.use_case and the LLM_AS_JUDGE category of my judge metric. When running three tasks with a maximum of 10 samples each, I get 90 requests to LiteLLM.

I see why having GENERATIVE and GENERATIVE_SAMPLING requests would result in different calls to the model since once it is greedy and once it should use sampling, however I would like to use the judge on the already generated samples without re-generating the output. I will try to make all metrics including the judge custom metrics of type GENERATIVE_SAMPLING and then report back if that worked, thank you for your advice.




# Update

## Overwriting the MetricCategory dynamically does not work well

**TL;DR:** It is better not to overwrite the `MetricCateogry` of the built-in metrics as there might be a reset somewhere in the code and you will end up with the original category. Better implement it as a custom metric as it was also suggested also by @clefourrier. 

I managed to set all of my metrics to the metric category GENERATIVE_SAMPLING and now the number of requests matches the number of samples in the pipeline. However now my LLM as Judge metric doesn't work anymore since `_get_metric_method_from_category` [1] returns the `apply_generative_metric` now instead of the `apply_llm_as_judge_metric`, which doesn't work with the output of the judge. Therefore, I switched the judge metric back to `LLM_AS_JUDGE`.

As soon as I did this, I had 90 instead of 30 requests again. The different request category metrics as observed in the in `_run_model` function in the pipeline [2] are of the following form:

```python
>>> pprint([r.metric_categories for r in requests])

[[<MetricCategory.GENERATIVE_SAMPLING: '5'>],
 [<MetricCategory.GENERATIVE: '3'>],
 [<MetricCategory.LLM_AS_JUDGE: '7'>],
 [<MetricCategory.GENERATIVE_SAMPLING: '5'>],
 [<MetricCategory.GENERATIVE: '3'>],
 [<MetricCategory.LLM_AS_JUDGE: '7'>],
...
 [<MetricCategory.LLM_AS_JUDGE: '7'>]]
```

Somehow adding a metric with the `LLM_AS_JUDGE` category resets the metric category of the built in metrics back to what they were originally. Using a custom metric worked for me in the end, thank you!

## LLM-as-Judge metric triggers additional requests to LLM

Now that I have only the two metric categories of interest (`GENERATIVE_SAMPLING` and `LLM_AS_JUDGE`), I still would like to have only one set of requests and not twice as much. For the LiteLLM experiments I should be fine because of the response caching feature. Is there something similar when using the `TransformersModel`?

Additionally, in case of the transformer models, I think this might lead to wrong results because if sampling is enabled, then the second request to the LLM that is used for generating the output to be evaluated by the LLM-as-Judge metric might be different than the initial output that was evaluated using the `GENERATIVE_SAMPLING` metrics.

---
References:
[1] [src/lighteval/tasks/lighteval_task.py#L508](https://github.com/huggingface/lighteval/blob/d332207bf65d70d3a1fe0538af91565d60cf47dd/src/lighteval/tasks/lighteval_task.py#L508)
[2] [src/lighteval/pipeline.py#L442](https://github.com/huggingface/lighteval/blob/d332207bf65d70d3a1fe0538af91565d60cf47dd/src/lighteval/pipeline.py#L442)


hey @rolshoven 
i have been trying to achieve something in lighteval can please guide me achieve it
1) I want to generate responses from my generation model use maj_at_5 as my metrics , i want to pick the most frequent response(i.e what maj_At_5 does) and send it to my Judge LLM.
2) Iam able to debug where the sampling responses are generated and everything my somehow lighteval is generating a response with num_samples as 1 and it is considering that as the response to be sent to judge llm as input