Repository: explodinggradients/ragas
Issue: How is factual correctness calculated?
Root cause: Algorithmic Error

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
I calculated the factual correctness of a certain data and found that the reference and response were exactly the same, but the score was 0.

Calculated data：
`
    {
        "user_input": "”收肺气，补气不足，升也。酸以收逆气，肺寒气逆“ 描述的哪一味中药？",
        "response": "五味子",
        "reference": "五味子"
    }
`

Claims of reference:
`
Output: , 模型回答为: {
    "claims": [
        "五味子 is a Chinese word."
    ]
}
`

Claims of response:
`
Output: , 模型回答为: {
    "claims": [
        "五味子 is a Chinese word."
    ]
}
`

The claims in the reference and response are the same, and the verdict in the output is also 1. But the final score is 0.0.

Please tell me why this is happening? @dosu

<!-- Answer -->
The issue you're experiencing, where the factual correctness score is 0 despite identical claims in the reference and response, could be related to several factors in the Ragas framework:

1. **Configuration Parameters**: The `mode` parameter, which can be set to `precision`, `recall`, or `f1`, affects the score calculation. If the mode is set to something other than `f1`, it might lead to unexpected results. Ensure that the mode is set correctly for your evaluation needs <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/available_metrics/factual_correctness.md)</sup>.

2. **Claim Decomposition Bug**: There is a known bug in Ragas version 0.2.11 related to the claim decomposition process, where the `traces` property only saves the last prompt per trace name. This could potentially lead to discrepancies in scoring <sup>[[2]](https://github.com/explodinggradients/ragas/issues/1871)</sup>.

3. **Verdict and Score Calculation**: The 'verdict' is a binary result indicating whether the evaluation criteria are met. If multiple checks are performed, the final score is determined by the majority vote of the verdicts. However, if strictness is set to 1, the score should directly reflect the verdict. Ensure that the strictness parameter is set correctly <sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_aspect_critic.py)</sup>.

4. **Noise Sensitivity and Other Parameters**: Parameters like `noise_sensitivity` and the weightage between factual and semantic similarity can also influence the final score. These should be configured appropriately to avoid unexpected variations <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/available_metrics/factual_correctness.md)</sup>.

Review these aspects in your setup to identify any misconfigurations or bugs that might be affecting the score calculation.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/fe2a7ad5-fa61-449e-859f-bbf7fc8c4640?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/fe2a7ad5-fa61-449e-859f-bbf7fc8c4640?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/fe2a7ad5-fa61-449e-859f-bbf7fc8c4640?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/fe2a7ad5-fa61-449e-859f-bbf7fc8c4640?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/fe2a7ad5-fa61-449e-859f-bbf7fc8c4640?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/fe2a7ad5-fa61-449e-859f-bbf7fc8c4640?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/fe2a7ad5-fa61-449e-859f-bbf7fc8c4640?feedback_type=other)</sup>


The claims in the reference and response are the same, and the verdict in the output is also 1. But the final score is 0.0. Why this is happening? @dosu 

<!-- Answer -->
The issue where the factual correctness score is 0 despite identical claims and a verdict of 1 could be due to several factors:

1. **Mode Parameter**: Ensure that the `mode` parameter is set correctly. The mode can be `precision`, `recall`, or `f1`, and it affects how the score is calculated. If it's not set to `f1`, it might lead to unexpected results <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/available_metrics/factual_correctness.md)</sup>.

2. **Known Bugs**: There might be a bug affecting the score calculation. For instance, in Ragas 0.2.8, there's a known issue where the `SimpleCriteriaScore` does not process the reference parameter correctly, which could lead to discrepancies in scoring <sup>[[2]](https://github.com/explodinggradients/ragas/issues/1757)</sup>.

3. **Strictness Parameter**: Check the `strictness` parameter. It determines the number of checks and the final score is based on a majority vote. If strictness is set to 1, the score should directly reflect the verdict. Ensure this parameter is configured correctly <sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/docs/concepts/metrics/available_metrics/aspect_critic.md)</sup>.

Review these aspects in your setup to identify any misconfigurations or bugs that might be affecting the score calculation.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/492bb555-59f6-46d7-b090-30594d84548a?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/492bb555-59f6-46d7-b090-30594d84548a?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/492bb555-59f6-46d7-b090-30594d84548a?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/492bb555-59f6-46d7-b090-30594d84548a?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/492bb555-59f6-46d7-b090-30594d84548a?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/492bb555-59f6-46d7-b090-30594d84548a?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/492bb555-59f6-46d7-b090-30594d84548a?feedback_type=other)</sup>


For your advice:
1. I use the default mode parameter, which is f1.
2. My Ragas version is 0.2.14, newer than 0.2.8.
3. I cannot find strictness parameter in class FactualCorrectness(), are you sure FactualCorrectness()  have this parameter?

I still don't understand the cause and solution of the problem. Can you give more advice?
@dosu

<!-- Answer -->
In Ragas version 0.2.14, the `FactualCorrectness` class does not have a `strictness` parameter. The issue you're experiencing, where the factual correctness score is 0 despite identical claims and a verdict of 1, might be influenced by other parameters or factors:

1. **Beta Parameter**: The `beta` parameter in the `FactualCorrectness` class affects the F1 score calculation. A beta value different from 1.0 can skew the balance between precision and recall, potentially affecting the score even if the claims are identical <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_factual_correctness.py)</sup>.

2. **Atomicity and Coverage**: These parameters determine how claims are decomposed and evaluated. If the decomposition is not aligned with the evaluation criteria, it might lead to unexpected scoring results. Ensure that these parameters are set appropriately for your use case <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_factual_correctness.py)</sup>.

3. **Potential Bugs or Changes**: While there are no known issues specifically documented for version 0.2.14, it's possible that changes in the claim decomposition process or other internal logic could affect the score. Reviewing the latest release notes or change logs might provide insights.

Consider adjusting the `beta`, `atomicity`, and `coverage` parameters to see if they influence the score. If the issue persists, further investigation into the specific implementation details of version 0.2.14 might be necessary.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*

---
<sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/a3a178d9-3587-49fd-8775-3d4c5359c100?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/a3a178d9-3587-49fd-8775-3d4c5359c100?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/a3a178d9-3587-49fd-8775-3d4c5359c100?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/a3a178d9-3587-49fd-8775-3d4c5359c100?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/a3a178d9-3587-49fd-8775-3d4c5359c100?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/a3a178d9-3587-49fd-8775-3d4c5359c100?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/a3a178d9-3587-49fd-8775-3d4c5359c100?feedback_type=other)</sup>


This issue is my mistake. close.