Repository: explodinggradients/ragas
Issue: can customize CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT when evaluate answer_correctnes
Root cause: Documentation Deficiency

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
can customize CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT when evaluate answer_correctnes

**Code Examples**
I found some statements inaccuracy. can custom CORRECTNESS_INSTRUCTIONS and CORRECTNESS_PROMPT?




<!-- Answer -->
Yes, you can customize `CORRECTNESS_INSTRUCTIONS` and `CORRECTNESS_PROMPT` when evaluating answer correctness.

The `CORRECTNESS_INSTRUCTIONS` and `CORRECTNESS_PROMPT` are defined as follows:

```python
CORRECTNESS_INSTRUCTIONS = """\
Given a ground truth and an answer statements, analyze each statement and classify them in one of the following categories:

- TP (true positive): statements that are present in answer that are also directly supported by the one or more statements in ground truth,
- FP (false positive): statements present in the answer but not directly supported by any statement in ground truth,
- FN (false negative): statements found in the ground truth but not present in answer.
Each statement can only belong to one of the categories. Provide a reason for each classification.
"""

CORRECTNESS_PROMPT = Prompt(
    name="answer_correctness",
    instruction=CORRECTNESS_INSTRUCTIONS,
    output_format_instruction=_output_instructions,
    examples=[
        {
            "question": """What powers the sun and what is its primary function?""",
            "answer": [
                "The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
                "The primary function of the sun is to provide light to the solar system.",
            ],
            "ground_truth": [
                "The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
                "This fusion process in the sun's core releases a tremendous amount of energy.",
                "The energy from the sun provides heat and light, which are essential for life on Earth.",
                "The sun's light plays a critical role in Earth's climate system.",
                "Sunlight helps to drive the weather and ocean currents.",
            ],
            "classification": AnswerCorrectnessClassification.parse_obj(
                {
                    "TP": [
                        {
                            "statement": "The primary function of the sun is to provide light to the solar system.",
                            "reason": "This statement is somewhat supported by the ground truth mentioning the sun providing light and its roles, though it focuses more broadly on the sun's energy.",
                        }
                    ],
                    "FP": [
                        {
                            "statement": "The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
                            "reason": "This statement is incorrect and contradicts the ground truth which states that the sun is powered by nuclear fusion.",
                        }
                    ],
                    "FN": [
                        {
                            "statement": "The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
                            "reason": "This accurate description of the sun’s power source is not included in the answer.",
                        },
                        {
                            "statement": "This fusion process in the sun's core releases a tremendous amount of energy.",
                            "reason": "This process and its significance are not mentioned in the answer.",
                        },
                        {
                            "statement": "The energy from the sun provides heat and light, which are essential for life on Earth.",
                            "reason": "The answer only mentions light, omitting the essential aspects of heat and its necessity for life, which the ground truth covers.",
                        },
                        {
                            "statement": "The sun's light plays a critical role in Earth's climate system.",
                            "reason": "This broader impact of the sun’s light on Earth's climate system is not addressed in the answer.",
                        },
                        {
                            "statement": "Sunlight helps to drive the weather and ocean currents.",
                            "reason": "The effect of sunlight on weather patterns and ocean currents is omitted in the answer.",
                        },
                    ],
                }
            ).dict(),
        },
        {
            "question": """What is the boiling point of water?""",
            "answer": [
                "The boiling point of water is 100 degrees Celsius at sea level"
            ],
            "ground_truth": [
                "The boiling point of water is 100 degrees Celsius (212 degrees Fahrenheit) at sea level.",
                "The boiling point of water can change with altitude.",
            ],
            "classification": AnswerCorrectnessClassification.parse_obj(
                {
                    "TP": [
                        {
                            "statement": "The boiling point of water is 100 degrees Celsius at sea level",
                            "reason": "This statement is directly supported by the ground truth which specifies the boiling point of water as 100 degrees Celsius at sea level.",
                        }
                    ],
                    "FP": [],
                    "FN": [
                        {
                            "statement": "The boiling point of water can change with altitude.",
                            "reason": "This additional information about how the boiling point of water can vary with altitude is not mentioned in the answer.",
                        }
                    ],
                }
            ).dict(),
        },
    ],
    input_keys=["question", "answer", "ground_truth"],
    output_key="classification",
    output_type="json",
)
```

You can modify the `CORRECTNESS_INSTRUCTIONS` and `CORRECTNESS_PROMPT` to suit your specific requirements <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu What does the complete code example look like?

<!-- Answer -->
Here is the complete code example for customizing `CORRECTNESS_INSTRUCTIONS` and `CORRECTNESS_PROMPT`:

```python
from ragas.llms.prompt import Prompt
from ragas.metrics._answer_correctness import AnswerCorrectnessClassification

CORRECTNESS_INSTRUCTIONS = """\
Given a ground truth and an answer statements, analyze each statement and classify them in one of the following categories:

- TP (true positive): statements that are present in answer that are also directly supported by the one or more statements in ground truth,
- FP (false positive): statements present in the answer but not directly supported by any statement in ground truth,
- FN (false negative): statements found in the ground truth but not present in answer.
Each statement can only belong to one of the categories. Provide a reason for each classification.
"""

CORRECTNESS_PROMPT = Prompt(
    name="answer_correctness",
    instruction=CORRECTNESS_INSTRUCTIONS,
    output_format_instruction=_output_instructions,
    examples=[
        {
            "question": """What powers the sun and what is its primary function?""",
            "answer": [
                "The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
                "The primary function of the sun is to provide light to the solar system.",
            ],
            "ground_truth": [
                "The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
                "This fusion process in the sun's core releases a tremendous amount of energy.",
                "The energy from the sun provides heat and light, which are essential for life on Earth.",
                "The sun's light plays a critical role in Earth's climate system.",
                "Sunlight helps to drive the weather and ocean currents.",
            ],
            "classification": AnswerCorrectnessClassification.parse_obj(
                {
                    "TP": [
                        {
                            "statement": "The primary function of the sun is to provide light to the solar system.",
                            "reason": "This statement is somewhat supported by the ground truth mentioning the sun providing light and its roles, though it focuses more broadly on the sun's energy.",
                        }
                    ],
                    "FP": [
                        {
                            "statement": "The sun is powered by nuclear fission, similar to nuclear reactors on Earth.",
                            "reason": "This statement is incorrect and contradicts the ground truth which states that the sun is powered by nuclear fusion.",
                        }
                    ],
                    "FN": [
                        {
                            "statement": "The sun is powered by nuclear fusion, where hydrogen atoms fuse to form helium.",
                            "reason": "This accurate description of the sun’s power source is not included in the answer.",
                        },
                        {
                            "statement": "This fusion process in the sun's core releases a tremendous amount of energy.",
                            "reason": "This process and its significance are not mentioned in the answer.",
                        },
                        {
                            "statement": "The energy from the sun provides heat and light, which are essential for life on Earth.",
                            "reason": "The answer only mentions light, omitting the essential aspects of heat and its necessity for life, which the ground truth covers.",
                        },
                        {
                            "statement": "The sun's light plays a critical role in Earth's climate system.",
                            "reason": "This broader impact of the sun’s light on Earth's climate system is not addressed in the answer.",
                        },
                        {
                            "statement": "Sunlight helps to drive the weather and ocean currents.",
                            "reason": "The effect of sunlight on weather patterns and ocean currents is omitted in the answer.",
                        },
                    ],
                }
            ).dict(),
        },
        {
            "question": """What is the boiling point of water?""",
            "answer": [
                "The boiling point of water is 100 degrees Celsius at sea level"
            ],
            "ground_truth": [
                "The boiling point of water is 100 degrees Celsius (212 degrees Fahrenheit) at sea level.",
                "The boiling point of water can change with altitude.",
            ],
            "classification": AnswerCorrectnessClassification.parse_obj(
                {
                    "TP": [
                        {
                            "statement": "The boiling point of water is 100 degrees Celsius at sea level",
                            "reason": "This statement is directly supported by the ground truth which specifies the boiling point of water as 100 degrees Celsius at sea level.",
                        }
                    ],
                    "FP": [],
                    "FN": [
                        {
                            "statement": "The boiling point of water can change with altitude.",
                            "reason": "This additional information about how the boiling point of water can vary with altitude is not mentioned in the answer.",
                        }
                    ],
                }
            ).dict(),
        },
    ],
    input_keys=["question", "answer", "ground_truth"],
    output_key="classification",
    output_type="json",
)
```

You can modify the `CORRECTNESS_INSTRUCTIONS` and `CORRECTNESS_PROMPT` to suit your specific requirements <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Here's an example on how to do it:

```py
from ragas.metrics import AnswerCorrectness
from ragas.metrics._answer_correctness import (
    CorrectnessClassifier,
    QuestionAnswerGroundTruth,
    ClassificationWithReason,
    StatementsWithReason,
)
from ragas.prompt import PydanticPrompt

class CustomCorrectnessClassifier(
    PydanticPrompt[QuestionAnswerGroundTruth,
ClassificationWithReason]
):
    instruction = """You are a strict evaluator. Classify
statements as:
    - TP: Only if the statement is EXACTLY stated in ground
truth
    - FP: Any statement not found verbatim in ground truth
    - FN: Ground truth statements missing from the answer
    Be very strict and penalize vague or approximate
matches."""

    input_model = QuestionAnswerGroundTruth
    output_model = ClassificationWithReason
    examples = [
        # Add your custom examples here
    ]

metric = AnswerCorrectness()
metric.set_prompts(correctness_classifier=CustomCorrectnessCl
assifier())

result = metric.compute(
    samples=[...],  # your samples
    run_config=RunConfig(...)
)
```

It seems the issue was answered, closing this now.