Repository: huggingface/lighteval
Issue: [FT] Problems with CorpusLevelMetricGrouping
Root cause: Documentation Deficiency


## Issue encountered
I’ve been working on setting up a custom evaluation task using a corpus-level metric, and while I’ve managed to get it working after a lot of trial and error, I’m still unclear on whether I’m doing it the right way — or just hacking around it.

Specifically, I have an external evaluation function that processes the entire dataset and returns multiple metrics. It works independently, and I’d like to integrate it into a custom task in lighteval. 

Since the evaluation is the fastest at the corpus level, I initially tried to use corpus_level_fn with CorpusLevelMetric. However, because the evaluator returns multiple metrics, I need some grouping. 
Therefore, CorpusLevelMetricGrouping felt like the correct solution — but I haven’t been able to get it working.

Below is what I currently have. This works, but I believe it’s not the intended approach, and it would be way better if i corpus_level_fn would.

```py
from typing import List

import numpy as np
from datasets import get_dataset_config_names
from evaluate import load
from lighteval.metrics.utils.metric_utils import (
    MetricUseCase,
    MetricCategory,
    SampleLevelMetricGrouping,
)
from lighteval.tasks.lighteval_task import LightevalTaskConfig
from lighteval.tasks.requests import Doc

symbolic_judge = load("AIML-TUDA/VerifiableRewardsForScalableLogicalReasoning")


def generate_pass_at_k_n_sample_level_metric(k: int, n: int):
    """Factory function to create pass@k metrics and detailed metrics"""

    assert k >= 1
    assert n >= k

    def prolog_data_preparator(
        predictions: List[str], golds: List[str], formatted_doc: Doc
    ):
        references = [
            {
                "validation_program": gold,
                "evaluation_config": {
                    "positive_predicate": "eastbound",
                    "negative_predicate": "westbound",
                },
            }
            for gold in golds
        ]
        results = symbolic_judge.compute(predictions=predictions, references=references)
        return {
            f"accuracy@{k}:{n}": results["accuracy"],
            f"partial_score@{k}:{n}": results["partial_score"],
            f"syntax_score@{k}:{n}": results["syntax_score"],
        }

    def top_k_mean_fn(values):
        return np.mean(np.sort(values)[-k:])

    metric_directions = {
        "accuracy": True,
        "partial_score": True,
        "syntax_score": True,
    }
    metric_names = [
        f"{metric}@{k}:{n}" for metric, direction in metric_directions.items()
    ]

    return SampleLevelMetricGrouping(
        metric_name=metric_names,
        sample_level_fn=prolog_data_preparator,
        category=MetricCategory.GENERATIVE,
        use_case=MetricUseCase.REASONING,
        corpus_level_fn=dict.fromkeys(metric_names, top_k_mean_fn),
        higher_is_better=dict.fromkeys(metric_names, True),
    )


def prompt_fn(line: dict, task_name: str):
    return Doc(
        task_name=task_name,
        query=line["prompt"],
        choices=[line["validation program"]],
        gold_index=0,
    )


available_subsets = get_dataset_config_names("ahmad21omar/MetaBench")

TASKS_TABLE = []


for subset in available_subsets:
    _task_name = f"V-LOL-Benchmark:{subset}"
    task_config = LightevalTaskConfig(
        name=_task_name,
        suite=["custom"],
        prompt_function=prompt_fn,
        hf_repo="ahmad21omar/SRL-Bench",
        hf_subset=subset,
        hf_avail_splits=["validation"],
        evaluation_splits=["validation"],
        few_shots_split=None,
        few_shots_select=None,
        metric=[
            generate_pass_at_k_n_sample_level_metric(1, 1),
            generate_pass_at_k_n_sample_level_metric(2, 8),
            generate_pass_at_k_n_sample_level_metric(4, 32),
        ],
        trust_dataset=True,
        generation_size=32768,
        version=1,
    )
    TASKS_TABLE.append(task_config)
```

## My questions
- Is CorpusLevelMetricGrouping the right tool here? If so, are there any working examples or documentation that I could look at?

## Motivation
In theory, this use case is simple: I have an evaluator that takes in the full corpus and returns a dict of scores. And I want lighteval to just calculates `"{metric}@{n}"`. 
This cant be that complicated, right?