Repository: explodinggradients/ragas
Issue: Unable to view Answer Relevancy questions
Root cause: Documentation Deficiency

Hi, 
I was trying to do a POC on different metrics of RAGAS but for that I need to make the black box things visible. I've tried my best to delve deep into the answer relevancy library and get ahold of the 3 questions that are generated by LLM for every row, but I couldn't.
Whatever I modify, the only output I get is - 
![image](https://github.com/user-attachments/assets/870fc2e8-54e4-468d-8de9-57efbbfe6351)

I'd be thankful if anyone can assist me regarding that.

Many Thanks!!

<!-- Greeting -->
Hey @knpunk! I'm here to assist you with any bugs, questions, or contributions. Let's dive into this together!

<!-- Answer -->
To access the three questions generated by the LLM for each row in the answer relevancy library in RAGAS, you need to look into the `_ascore` method in the `AnswerRelevancy` class. This method generates the questions using the LLM and the `question_generation` prompt. The number of questions generated is controlled by the `strictness` attribute, which is set to 3 by default.

Here is the relevant code snippet from the `_ascore` method:

```python
async def _ascore(self, row: t.Dict, callbacks: Callbacks, is_async: bool) -> float:
    assert self.llm is not None, "LLM is not set"

    prompt = self._create_question_gen_prompt(row)
    result = await self.llm.generate(
        prompt,
        n=self.strictness,  # This controls the number of questions generated
        callbacks=callbacks,
        is_async=is_async,
    )

    answers = [
        await _output_parser.aparse(result.text, prompt, self.llm)
        for result in result.generations[0]
    ]
    if any(answer is None for answer in answers):
        return np.nan

    answers = [answer for answer in answers if answer is not None]
    return self._calculate_score(answers, row)
```

In this method, `result.generations[0]` will contain the generated questions. The `strictness` attribute, which is set to 3, ensures that three questions are generated for each row <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_relevance.py)</sup>.

To make these questions visible, you can modify the `_ascore` method to print or log the generated questions. For example:

```python
async def _ascore(self, row: t.Dict, callbacks: Callbacks, is_async: bool) -> float:
    assert self.llm is not None, "LLM is not set"

    prompt = self._create_question_gen_prompt(row)
    result = await self.llm.generate(
        prompt,
        n=self.strictness,
        callbacks=callbacks,
        is_async=is_async,
    )

    questions = [result.text for result in result.generations[0]]
    print("Generated Questions:", questions)  # Add this line to print the questions

    answers = [
        await _output_parser.aparse(result.text, prompt, self.llm)
        for result in result.generations[0]
    ]
    if any(answer is None for answer in answers):
        return np.nan

    answers = [answer for answer in answers if answer is not None]
    return self._calculate_score(answers, row)
```

This will print the generated questions to the console, allowing you to see them during your POC.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu Sorry, but it's not doing the work for me, it just evaluates and stores the score in the dataframe, that's all. Do I need to make any change in evaluator function code as well?

<!-- Answer -->
Yes, you need to make changes in the `_ascore` method of the `AnswerRelevancy` class to print the generated questions. You can add a print statement to output the generated questions.

Here is the modified `_ascore` method with the added print statement:

```python
async def _ascore(self, row: t.Dict, callbacks: Callbacks, is_async: bool) -> float:
    assert self.llm is not None, "LLM must be set"

    question = row["question"]
    statements = {}
    for item in ["answer", "ground_truth"]:
        p_value = self._create_statements_prompt(row)
        item_statement = await self.llm.generate(
            p_value, callbacks=callbacks, is_async=is_async
        )
        statements[item] = await _statements_output_parser.aparse(
            item_statement.generations[0][0].text,
            p_value,
            self.llm,
            self.max_retries,
        )
        statements[item] = (
            statements[item].dicts() if statements[item] is not None else []
        )

    if not all([val == [] for val in statements.values()]):
        ground_truth = [
            statement
            for item in statements["ground_truth"]
            for statement in item["simpler_statements"]
        ]
        answer = [
            statement
            for item in statements["answer"]
            for statement in item["simpler_statements"]
        ]
        p_value = self.correctness_prompt.format(
            question=question,
            ground_truth=ground_truth,
            answer=answer,
        )
        is_statement_present = await self.llm.generate(
            p_value, callbacks=callbacks, is_async=is_async
        )
        result_text = is_statement_present.generations[0][0].text

        answers = await _output_parser.aparse(
            result_text, p_value, self.llm, self.max_retries
        )
        if answers is None:
            return np.nan

        f1_score = self._compute_statement_presence(answers)
    else:
        f1_score = 1.0

    if self.weights[1] == 0:
        similarity_score = 0.0
    else:
        assert self.answer_similarity is not None, "AnswerSimilarity must be set"

        similarity_score = await self.answer_similarity.ascore(
            row, callbacks=callbacks, is_async=is_async
        )

    score = np.average(
        [f1_score, similarity_score],
        weights=self.weights,
    )

    # Print the generated questions
    generated_questions = [statement["question"] for statement in statements["answer"]]
    print("Generated Questions:", generated_questions)

    return float(score)
```

This will print the generated questions each time the `_ascore` method is called <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_relevance.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@dosu I think you've confused the answer relevancy with answer correctness. Can any human answer me please?

Print statement isn't doing anything, I think we need to modify the evaluation.py code, but not exactly sure how to do it.

<!-- Answer -->
This is a tough nut to crack, could really use your help on this, @jjmachan.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@jjmachan the questions are being displayed now with the first solution given by @dosu, but all three are the same, can you help me out regarding that?

<!-- Answer -->
A maintainer has already been looped in.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*