Repository: explodinggradients/ragas
Issue: why answer_relavancy getting nan and sometime get 0. why is that?
Root cause: External Dependency Breakage

[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
why answer_relavancy getting nan and sometime get 0. why is that?

**Code Examples**

`from datasets import Dataset
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall
)
from ragas import evaluate`

`eval_data = [
    {
        "question": "What are the most valuable features of Canva according to the participants?",
        "reference": (
            '"The brand kit feature is essential for client work. We can set up each client\'s colors and fonts, and everyone on the team can access them instantly" - Jennifer Walsh (Current Customer); '
            '"The stock photos were definitely the best feature. Really high-quality images that I could use for product backgrounds" - Marcus Rodriguez (Churned Customer); '
            '"The API capabilities are crucial, though they could be more robust. For actual design work, the drag-and-drop editor is surprisingly intuitive" - David Kim (Partner); '
            '"Templates are helpful too for getting started, but I always felt limited by them" - Marcus Rodriguez (Churned Customer); '
            '"The collaboration tools are great too - clients can comment directly on designs instead of sending revision emails" - Jennifer Walsh (Current Customer)'
        )
    }
]`


`rows = []
for item in eval_data:
    docs = vector_store.similarity_search(item["question"], k=10)
    contexts = [doc.page_content for doc in docs]

    rows.append({
        "question": item["question"],
        "answer": get_chatbot_response(item["question"], vector_store),
        "reference": item["reference"],
        "contexts": contexts
    })`

`from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],

)
print(results)`



**Output Result**

<img width="1240" height="202" alt="Image" src="https://github.com/user-attachments/assets/0b9f9691-8520-41fa-bd3f-20a83299f37d" />


This seems like a bug. Thanks for reporting @Chandimam 

Still getting the same error

@Chandimam Did you try with the PR #2327 ? 

Hi @anistark I try the PR 2327 but I get the same error decribed by @Chandimam 

@anistark if I use Response Relevancy(strictness=1) I get {'answer relevancy': 1.0000}, it seems the LLM only generate one question instead three of them

hey @anistark I merged your PR but keeping this open since @rodyoukai reported it was still not fixed.

@rodyoukai could you share a bit more about 
1. which LLM you are using
2. traces if any

Hi again @anistark 6 @jjmachan  I use openai/gpt-4 and anthropic/claude-opus-4.1 (same behaviour) I do a downgrade to 0.3.2 versión and now I can use Relevancy(strictness=5)  and works perfetly

> Hi again [@anistark](https://github.com/anistark) I use openai/gpt-4 and anthropic/claude-opus-4.1 (same behaviour) I do a downgrade to 0.3.2 versión and now I can user Relevancy(strictness=5) and works perfetly

Thanks for the details. Will look into it more. 

> [@Chandimam](https://github.com/Chandimam) Did you try with the PR [#2327](https://github.com/explodinggradients/ragas/pull/2327) ?

yes

@Chandimam @rodyoukai Could you please check if PR #2343 fixes the issue for you? 

PR #2343 doesn't solve the problem. 

As you mentioned above, there is no problem in version 0.3.2. 
However, in version 0.3.5, there is a problem that answer relevancy continues to be null even if I modify it with the code you modified in PR #2343. 

Unusually, when evaluating with just one metric(=answer relevancy), it is evaluated without problem. 
However, as shown in the figure, the error only occurs when evaluating with other metrics. Only answer relevancy is stored as null.

<img width="643" height="17" alt="Image" src="https://github.com/user-attachments/assets/a0966b8a-b06c-4cd0-9d62-96bafe0db3c3" />