Repository: explodinggradients/ragas
Issue: RAGAS evaluation metrics faithfulness is not working well for true or false type of questions.
Root cause: Algorithmic Error

Looks like Ragas is actually not working well for any true or false questions which have false as the answer.
Here is an example from my testing to figure out what's happening.
In this scenario, I have generated 2 true or false questions. I'm sending the prompt as the question to Ragas and the entire set of generated question and answers to Ragas as it's 'answer'.

Generated quiz (sent completely at once as answer to ragas):
{
  "quiz": [
    {
      "question": "True or False: Hierarchical Softmax is less efficient than the normal softmax and is better for frequent words.",
      "answer": "false",
      "question_type": "true_or_false"
    },
    {
      "question": "True or False: In Hierarchical Softmax, each node of the graph (except the root and the leaves) is associated to a vector that the model is going to learn.",
      "answer": "true",
      "question_type": "true_or_false"
    }
]
}
Internal evaluation process by Ragas:
[Generation(text='[\n    {\n        "statement_1": "Hierarchical Softmax is less efficient than the normal softmax and is better for frequent words.",\n        "reason": "The context states that hierarchical softmax is a more efficient alternative to the normal softmax. It also mentions that hierarchical softmax tends to be better for infrequent words, not frequent words.",\n        "verdict": "No"\n    },\n    {\n        "statement_2": "In Hierarchical Softmax, each node of the graph (except the root and the leaves) is associated to a vector that the model is going to learn.",\n        "reason": "The context mentions that each node of the graph is associated with a vector that the model is going to learn.",\n        "verdict": "Yes"\n    }\n]', generation_info={'finish_reason': 'stop', 'logprobs': None})]

Verdict=No means a score of 0 and verdict=Yes means a score of 1. For the overall quiz, it gave a scoare of 0.5.

What's happening:
For the first question, Ragas didn't consider that the answer is false (statement 1). It directly compared the given false statement against the context and determined that it is incorrect and gave it a score of 0. In our case, since our quiz generated a false statement as expected, Ragas should have identified that and given a score of 1.

For the second question, Ragas directly compared the given statement against the context again (statement 2). But in this case, since it was a true statement and it was matching with the context, it gave the score of 1.

Basically, for all true statements, if our application generated it properly from the context, we're getting the score of 1. But for a false question, when our application changes the context slightly to generate the false statement(as expected), Ragas is directly evaluating it assuming that it's a correct statement instead of converting the false statement to a correct one before evaluation.

What's happening:
For the first question, Ragas didn't consider that the answer is false (statement 1). It directly compared the given false statement against the context and determined that it is incorrect and gave it a score of 0. In our case, since our quiz generated a false statement as expected, Ragas should have identified that and given a score of 1.

For the second question, Ragas directly compared the given statement against the context again (statement 2). But in this case, since it was a true statement and it was matching with the context, it gave the score of 1.

Basically, for all true statements, if our application generated it properly from the context, we're getting the score of 1. But for a false question, when our application changes the context slightly to generate the false statement(as expected), Ragas is directly evaluating it assuming that it's a correct statement instead of converting the false statement to a correct one before evaluation.

I tried reformatting the "answer" we send to ragas by adding a prefix like "The following is a set of true or false questions", reformatting the quiz to prefix the answer such as "False: Hierarchical Softmax is less efficient than the normal softmax and is better for frequent words." or even specifying if it's a true or false statement in the same sentences. However, none of the approaches are working for the false questions.

Apparently while converting the given answer to  statements for evaluation, ragas makes use of the question as well (although it looks like it ignored the prompt in the above scenario). So in the next exercise, I gave the generated question as the question itself and the generated answer as the answer to Ragas to see what happens. I used the same quiz as above.

This is how Ragas evaluated it internally:
[Generation(text='[\n    {\n        "statement_1": "Hierarchical Softmax is not less efficient than the normal softmax.",\n        "reason": "The context explicitly states that hierarchical softmax is a much more efficient alternative to the normal softmax.",\n        "verdict": "No"\n    },\n    {\n        "statement_2": "Hierarchical Softmax is not better for frequent words.",\n        "reason": "The context mentions that hierarchical softmax tends to be better for infrequent words, while negative sampling works better for frequent words.",\n        "verdict": "No"\n    }\n]', generation_info={'finish_reason': 'stop', 'logprobs': None})]
[Generation(text='[\n    {\n        "statement_1": "In Hierarchical Softmax, each node of the graph is associated to a vector.",\n        "reason": "The context explicitly states that each node of the graph (except the root and the leaves) is associated with a vector.",\n        "verdict": "Yes"\n    },\n    {\n        "statement_2": "The model is going to learn the vectors associated with each node.",\n        "reason": "The context mentions that the model is going to learn the vectors associated with each node of the graph.",\n        "verdict": "Yes"\n    }\n]', generation_info={'finish_reason': 'stop', 'logprobs': None})]

Looks like it managed to update the false question to correct statements. However, if we see statement 1 and 2 for the first generation (question/answer), even though the meaning of the statement is the same as the reason it provided, it scored it as 0.

Hey @soundaryabala123 , this is interesting. We had to fix a lot of similar edge case issues like this in the past. I could take this up, if you are interested in working in it let me know.

Yes, I’m interested in working in it.

Sure, I can assign it to you. If you like to have a short chat on how to solve this issue feel free to schedule a call [here](https://calendly.com/shahules/30min)
@soundaryabala123