Repository: prometheus-eval/prometheus-eval
Issue: Evaluation reproducibility issue with "Unbabel/M-Prometheus-7B" and temperature setting
Root cause: Validation Gap

Hello, thank you for your great work on Prometheus.

I'm using the "Unbabel/M-Prometheus-7B" model to evaluate LLM responses via PrometheusEval.absolute_grade(). In order to improve evaluation reproducibility, I set the temperature parameter to a lower value (e.g., between 0 and 0.9). However, when doing so, I frequently encounter the following issue: `Retrying failed batches`

This seems to happen more often when the temperature is low, and in some cases the evaluation fails entirely.

Could you clarify:
- Whether low temperature values are supported for inference-time deterministic scoring?
- If this issue is expected or avoidable?
- Whether there are recommended inference settings (e.g., temperature, top_p, etc.) for stable and reproducible scoring using the Prometheus model?

Any guidance would be greatly appreciated.
Thanks in advance!

```python
# Absolute Grading: Outputs score of 1 to 5
from prometheus_eval.vllm import VLLM
from prometheus_eval import PrometheusEval
from prometheus_eval.prompts import ABSOLUTE_PROMPT, SCORE_RUBRIC_TEMPLATE
from colorama import Fore, Style
import time
import pandas as pd
import argparse
import json
import os
import sys
current_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.abspath(os.path.join(current_dir, '..'))
sys.path.append(project_root)

from utils.utils import (
    MODEL_MAPPING, format_time, setup_logging,
    load_environment, save_json
)


def load_data(file_path):
    try:
        df = pd.read_json(file_path)
    except FileNotFoundError:
        print(f"Error: The file at {file_path} was not found.")
        sys.exit(1)
    return df

def load_results(output_path):
    if os.path.exists(output_path):
        with open(output_path, "r", encoding="utf-8") as f:
            results = json.load(f)
    else:
        results = []
    return results

def prometheus_eval(judge, instructions, responses, reference_answers):
    rubric_data = {
        "criteria": "질문에 답하기 위해 필요한 모든 필수 정보를 포함하고 있는 정도를 평가합니다.",
        "score1_description": "질문과 무관한 내용만 포함하거나, 답변으로 간주할 수 없을 정도로 매우 불완전합니다.",
        "score2_description": "핵심 정보가 거의 빠져 있거나, 질문에 거의 답하지 못해 유의미한 정보 전달이 어렵습니다.",
        "score3_description": "일부 중요한 정보가 누락되어 있어 답변의 이해나 활용에 제약이 있습니다.",
        "score4_description": "대부분의 중요한 정보를 포함하고 있으나, 일부 보완이 필요한 세부사항이 존재합니다.",
        "score5_description": "질문에 대한 답변이 매우 완전하며, 모든 핵심 정보와 관련된 세부사항까지 충실히 포함하고 있습니다."
    }
    
    score_rubric = SCORE_RUBRIC_TEMPLATE.format(**rubric_data)
    
    _, scores = judge.absolute_grade(
        instructions=instructions,
        responses=responses,
        rubric=score_rubric,
        reference_answers=reference_answers
    )
    avg_score = sum(scores) / len(scores)
    print(f"Avg prometheus score: {Fore.RED}{avg_score:.3f}{Style.RESET_ALL}")
    return avg_score

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="평가용 데이터 입력")
    parser.add_argument("--model_name", type=str, choices=list(MODEL_MAPPING.keys()), default="gpt-4o-mini")
    parser.add_argument("--shot", type=str, choices=["0", "1", "3", "6"], default="0")
    parser.add_argument("--input_format", choices=["preprocessed", "raw"], default="preprocessed")
    parser.add_argument("--answer_type", type=str, choices=["E", "NE", "ALL"], default="ALL")
    parser.add_argument("--use_finetuned_model", action="store_true")
    args = parser.parse_args()
    
    env = load_environment()
    logger = setup_logging()
    logger.info(f"MODEL NAME: {args.model_name}")
    logger.info(f"SHOT: {args.shot}")
    logger.info(f"INPUT FORMAT: {args.input_format}")
    logger.info(f"ANSWER TYPE: {args.answer_type}")
    logger.info(f"USE FINETUNED MODEL: {args.use_finetuned_model}")
    
    if args.use_finetuned_model:
        file_path = os.path.join(env["generated_answers_dir"],
                                 f"output_{args.model_name}_{args.shot}_{args.input_format}_{args.answer_type}.json")
    else:
        file_path = os.path.join(env["generated_answers_dir"],
                                 f"output_{args.model_name}_{args.shot}_{args.input_format}.json")
    
    start_time = time.time()
    df = load_data(file_path)
    logger.info(f"TOTAL SAMPLE COUNT: {len(df):,}")
    print("--------------------------------")
    
    prometheus_model = VLLM(
        model="Unbabel/M-Prometheus-7B",
        tensor_parallel_size=2
    )
    judge = PrometheusEval(model=prometheus_model, absolute_grade_template=ABSOLUTE_PROMPT)

    unique_categories = [("dog", "expert"), ("dog", "nonexpert"), ("cat", "expert"), ("cat", "nonexpert")]
    
    output_path = "./prometheus_results.json"
    results = load_results(output_path)
    
    endpoint = f"{args.model_name}_{args.shot}_{args.input_format}_{args.answer_type}"
    results.append({"id": endpoint})
    
    for animal_type, answer_type in unique_categories:
        filtered_df = df[(df['animal_type'] == animal_type) & (df['answer_type'] == answer_type)]
        
        category = f"{animal_type}-{answer_type}"
        print(f"Evaluating: {category} (Sample count: {len(filtered_df):,})")
        
        instructions = filtered_df['preprocessed_question'].tolist()
        responses = filtered_df['generated_answer'].tolist()
        reference_answers = filtered_df['preprocessed_answer'].tolist()

        avg_score = prometheus_eval(judge, instructions, responses, reference_answers)
        results.append({
            "category": category,
            "avg_score": avg_score
        })
    
    save_json(results, output_path)

    
    elapsed = time.time() - start_time
    print(f"TOTAL TIME: {format_time(elapsed)}")
    print("--------------------------------")


Additionally, I noticed that the retry logic for failed batches is capped at a maximum of 10 attempts. However, when evaluating around 10,000 Korean samples, 10 retries were not sufficient to complete all requests successfully.

As a workaround, I locally modified the retry loop to allow unlimited retries, using the following logic:

``` python
def batch_completions_with_retries(
    model,
    inputs,
    mode: str,
    max_retries: int = 10,
    params: dict = None,
):
...

# Retry logic with progress bar
retries = 0
while to_retry_inputs:
    retries += 1
    print(f"Retrying failed batches: Attempt {retries}")
...
```
I have two follow-up questions:

1. Is there a specific reason why the maximum retry count is limited to 10?
2. Would it be safe to use a higher or unlimited retry threshold like above, or could it cause any unintended issues (e.g., hanging processes, resource leaks)?

Thanks again for your support and for providing such a useful evaluation framework!

The key for picking a setting that works for you requires an understanding of what a failed output means and how the framework tries to fix it. A failure means that the output didn't match the intended format, so it couldn't be parsed. When this happens, the only strategy this repo allows is to _rerun the same prompt again_ and hope for a different output that can be parsed. If you make the generation greedy (temperature=0), then this strategy will fail because repeated generations from the same prompt will always produce the same output.

For this reason, if you want reproducibility I would recommend fixing the random seed(s).

I also encountered the same problem with the M-Prometheus models (even at 14B). I don't think *just* fixing the random seeds is the correct solution. I wonder if the released models were undertrained to follow the rubric format 🤔 I'm also curious as to what the recommended sampling parameters for the M-Prometheus models.