Repository: EleutherAI/lm-evaluation-harness
Issue: Passing sample based parameters for metric
Root cause: Unimplemented Feature Gap

Hi! I am implementing an LLM-as-judge metric for open-ended question answers. In the prompt that I am using for the evaluator model, I would like to provide three different variables:
- Correct answer
- Generated answer
- Question
Is there a way to pass the metric function the original data point used for generation? 

Hi! I've been meaning to add a model-based judge but haven't had the bandwidth.Simplest way would be to use a  [filter](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md#filters). Filters allow for stateful initialization, arbitrary kwargs can be passed from the [config](https://github.com/EleutherAI/lm-evaluation-harness/blob/d57e3d6573bbc347852ef65e1b55c56f31d0b9cc/lm_eval/tasks/gsm8k/gsm8k-cot-llama.yaml#L53-L54) and can be chained together. The model outputs along with their respective docs are then all passed to the filter's apply method for processing.

Happy to help review and chip in if you'd like to open a PR.

Otherwise can also overload process_results ([1](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/tasks/minerva_math/utils.py#L74), [2](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/tasks/minerva_math/minerva_math_algebra.yaml#L11)), but currently `process_results` is called on each (doc, result) pair individually.


Thank you for the answer! I understand the usage of filtering but I do not understand how they interact with the metrics. Is the output of a filter pass also to the metrics? Can I use the output of a filter as a metric? 

> Thank you for the answer! I understand the usage of filtering but I do not understand how they interact with the metrics. Is the output of a filter pass also to the metrics? Can I use the output of a filter as a metric?

Hi! apologies for the tardiness. The flow is model generations  -> filter -> metric. You can skip the metric calculation using the `--predict_only` flag (and save the samples to file using `--log_samples`). You can also overload `process_results` to process the outputs of the filter further (e.g. [humaneval](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/humaneval/humaneval.yaml)) (this will also skip the metric calculation, as they are called within `process_results`). 

Hope this helps!

Thank you for the answer! I have overloaded the process_results and it works, however doing sequential evaluation using `process_results` it is slow. Is there a way to have a batch (or also all) of the generations in `process_results?`

> Thank you for the answer! I have overloaded the process_results and it works, however doing sequential evaluation using `process_results` it is slow. Is there a way to have a batch (or also all) of the generations in `process_results?`

Currently not possible, but I've been meaning to add that in. A workaround it to use a filter (which takes in _all_ the requests along with their respective docs), and pass the outputs to a no-op process_results.

> Simplest way would be to use a [filter](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md#filters). Filters allow for stateful initialization, arbitrary kwargs can be passed from the [config](https://github.com/EleutherAI/lm-evaluation-harness/blob/d57e3d6573bbc347852ef65e1b55c56f31d0b9cc/lm_eval/tasks/gsm8k/gsm8k-cot-llama.yaml#L53-L54) and can be chained together

I'm hoping I understood correctly - does that mean we can somehow inject information from the doc (let's say the question, or context) into the metric using filters? I'm asking because I have multiple metrics which require the question to compute, and it gets really tedious to entirely override `process_results` every time the need arises.

> > Simplest way would be to use a [filter](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md#filters). Filters allow for stateful initialization, arbitrary kwargs can be passed from the [config](https://github.com/EleutherAI/lm-evaluation-harness/blob/d57e3d6573bbc347852ef65e1b55c56f31d0b9cc/lm_eval/tasks/gsm8k/gsm8k-cot-llama.yaml#L53-L54) and can be chained together
> 
> I'm hoping I understood correctly - does that mean we can somehow inject information from the doc (let's say the question, or context) into the metric using filters? I'm asking because I have multiple metrics which require the question to compute, and it gets really tedious to entirely override `process_results` every time the need arises.

Hi! yes! (but sort of). The filter call is defined here:
https://github.com/EleutherAI/lm-evaluation-harness/blob/bc811365ef3917a15da6aac9b7feafb4ba0fb42f/lm_eval/api/filter.py#L23

Filters are mainly for post-processing model outputs before they hit the metrics, but they don't directly pass doc information to the metric functions. However, you can use filters to augment the model output with doc information.