Repository: huggingface/lighteval
Issue: [BUG] Num samples not respected
Root cause: Configuration Error

## Describe the bug
num samples metric_option is not respected

## To Reproduce
regardless of options set on task definition and metric settings, it's not generating multiple samples

## Expected behavior

setting
```
metric_options={
            "extractive_match": {
                "num_samples": 16,
            },
```

or 

```
# Define tasks
task = LightevalTaskConfig(
    num_samples=NUM_SAMPLES, ...
)
```
should respect the setting

## Version info
install
```
conda create -n eval python=3.11
pip install vllm==0.7.2
pip install git+https://github.com/huggingface/lighteval.git#egg=lighteval math-verify==0.5.2
```

@plaguss sorry for the random ping but do you have any tips? I was trying to follow the LCB approach for this.
tried the following as well as full custom SampleLevelMetric

```
latex_gold_metric = multilingual_extractive_match_metric(
    language=Language.ENGLISH,
    fallback_mode="first_match",
    precision=5,
    gold_extraction_target=(LatexExtractionConfig(),),
    # Match boxed first before trying other regexes
    pred_extraction_target=(ExprExtractionConfig(), LatexExtractionConfig(boxed_match_priority=0)),
    aggregation_function=max,
)
latex_gold_metric.metric_name = "extractive_match_pass@1:16"
```

Hi @rawsh, can you share the results you are obtaining, and the model you are using? With the updated name, it should set `num_samples=16` (it's done [here](https://github.com/huggingface/lighteval/blob/main/src/lighteval/tasks/lighteval_task.py#L218) at the Task level).

Hey @plaguss, it's just silently using n=1 completions. I've tried using metric names like `exact_match@16` and setting the pipeline with an override with

```
pipeline = Pipeline(
        tasks=tasks,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        model_config=model_config,
        metric_options={
            "extractive_match": {
                "num_samples": 16,
            },
        }
    )
```

I'm using the distilled r1 models (qwen 1.5B/7B)

hey ! can you sahre your exact command ? then I can reproduce and try a fix

I actually ended up finding a workaround, trick for me was figuring out it wouldn't respect num samples bc the category was GENERATIVE instead of GENERATIVE_SAMPLING

I think it would be good to have an easier way to work with multiple samples using `multilingual_extractive_match_metric` + setting multiple samples using metric_options, or passing to the metric setup fn

This is what I'm using:

```python
def create_custom_avg_metric(
    gold_extraction_target,
    pred_extraction_target,
    language=Language.ENGLISH,
    fallback_mode="first_match",
    extraction_mode="any_match",
    precision=5,
    timeout_seconds=5
):
    def sample_level_fn(golds: List[str], predictions: List[str], formatted_doc: Doc) -> float:
        print("\n\n")
        print(formatted_doc)

        # Get extraction regexes
        gold_extraction_regexes = get_extraction_regexes(formatted_doc, gold_extraction_target, language)
        pred_extraction_regexes = get_extraction_regexes(formatted_doc, pred_extraction_target, language)
        
        # Extract gold answers
        extracted_golds = [
            extract_target_from_pred(gold, gold_extraction_regexes, fallback_mode, extraction_mode, timeout_seconds)
            for gold in golds
        ]
        
        # Handle empty gold extractions
        if any(len(g) == 0 for g in extracted_golds):
            logger.warning(f"Could not extract gold in correct format. Gold: {golds}")
            extracted_golds = [[gold] for gold in golds]
        
        # Process each prediction separately and get individual scores
        prediction_scores = []
        
        for prediction in predictions:
            # Extract answers from this prediction
            extracted_preds = extract_target_from_pred(
                prediction, pred_extraction_regexes, fallback_mode, extraction_mode, timeout_seconds
            )
            
            # Skip empty predictions
            if len(extracted_preds) == 0:
                prediction_scores.append(0.0)
                continue
            
            # Check if ANY extracted gold matches ANY extracted prediction
            is_correct = False
            for pred in extracted_preds:
                for gold_set in extracted_golds:
                    for gold in gold_set:
                        try:
                            if compare_gold_target([gold], [pred], precision, timeout_seconds=timeout_seconds):
                                is_correct = True
                                break
                        except Exception as e:
                            logger.warning(f"Error comparing gold and prediction: {e}")
                    if is_correct:
                        break
                if is_correct:
                    break
            
            prediction_scores.append(1.0 if is_correct else 0.0)
        
        # Store for debugging
        if formatted_doc.specific is None:
            formatted_doc.specific = {}
        
        formatted_doc.specific["extracted_predictions"] = [
            str(pred) for preds in [extract_target_from_pred(
                p, pred_extraction_regexes, fallback_mode, extraction_mode, timeout_seconds
            ) for p in predictions] for pred in preds
        ]
        formatted_doc.specific["extracted_golds"] = [
            str(gold) for golds in extracted_golds for gold in golds
        ]
        
        # Calculate average score
        avg_score = np.mean(prediction_scores) if prediction_scores else 0.0
        
        # For debugging, still calculate and print max score
        max_score = max(prediction_scores) if prediction_scores else 0.0
        
        print(f"Individual prediction scores: {prediction_scores}")
        print(f"Average score: {avg_score}")
        print(f"Max score (best-of-N): {max_score}")
        
        # Return average score instead of max score
        return avg_score
    
    return sample_level_fn

# Create custom metrics for each task type
latex_metric_fn = create_custom_avg_metric(
    gold_extraction_target=(LatexExtractionConfig(),),
    pred_extraction_target=(ExprExtractionConfig(), LatexExtractionConfig(boxed_match_priority=0)),
    precision=5
)

expr_metric_fn = create_custom_avg_metric(
    gold_extraction_target=(ExprExtractionConfig(),),
    pred_extraction_target=(ExprExtractionConfig(), LatexExtractionConfig(boxed_match_priority=0)),
    precision=5
)

gpqa_metric_fn = create_custom_avg_metric(
    gold_extraction_target=[IndicesExtractionConfig(prefix_for_extraction="NativeLetters")],
    pred_extraction_target=[IndicesExtractionConfig(prefix_for_extraction="NativeLetters")],
    precision=5
)

# Create the actual metric objects
latex_avg_metric = SampleLevelMetric(
    metric_name=f"avg_accuracy@{NUM_SAMPLES}",
    sample_level_fn=latex_metric_fn,
    category=MetricCategory.GENERATIVE_SAMPLING,
    use_case=MetricUseCase.ACCURACY,
    corpus_level_fn=np.mean,
    higher_is_better=True,
)

expr_avg_metric = SampleLevelMetric(
    metric_name=f"avg_accuracy@{NUM_SAMPLES}",
    sample_level_fn=expr_metric_fn,
    category=MetricCategory.GENERATIVE_SAMPLING,
    use_case=MetricUseCase.ACCURACY,
    corpus_level_fn=np.mean,
    higher_is_better=True,
)

gpqa_avg_metric = SampleLevelMetric(
    metric_name=f"avg_accuracy@{NUM_SAMPLES}",
    sample_level_fn=gpqa_metric_fn,
    category=MetricCategory.GENERATIVE_SAMPLING,
    use_case=MetricUseCase.ACCURACY,
    corpus_level_fn=np.mean,
    higher_is_better=True,
)
```