Repository: explodinggradients/ragas
Issue: Why faithfullness is NaN most of the times even when context and answer both makes sense.
Root cause: Validation Gap

I am using Bedrock for my RAG and faithfullness is NaN most of the times even when context and answer both makes sense. The same problem is also there for the amnesty dataset shared in ragas docs.

Ragas version: 0.1.0

**Code to Reproduce**
from datasets import load_dataset
from ragas import evaluate

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
result = evaluate(amnesty_qa["eval"],metrics=[context_precision, faithfulness, answer_relevancy])
result.to_pandas()

**Error trace**
![image](https://github.com/explodinggradients/ragas/assets/86981996/995b4d7d-1c97-4c18-abe5-b528bc25475f)

I see similar issue in my own dataset as well, but it's not clear why failthfulness is NaN. Could you please give some clarity what is the issue here.


I have experienced the same issue. `faithfullness` is always 0 when gpt-4 is used as judge, and NaN when gpt-3.5 is used as judge. version 0.1.0. The same behaviour occurs when running the 0.0.21 version and changing the llm to gpt-4 (default is 'gpt-3.5-turbo-16k') 

from ragas.llms.openai import OpenAI
faithfullness.llm = OpenAI(model="gpt-4")

hey @christdahl , I am not able to reproduce the issue using gpt4/3.5 
If you're trying to customise models, please follow the updated docs [here](https://docs.ragas.io/en/stable/howtos/customisations/llms.html)  for 0.1 version
<img width="1178" alt="Screenshot 2024-02-09 at 10 15 36 AM" src="https://github.com/explodinggradients/ragas/assets/25312635/24a5fa13-40c7-4f50-bf4a-1531e47158b2">


Hi @repl-mira-agarwal Found the issue with bedrock, but this has been known to us for a while now. Bedrock as model seems completely incapable of producing valid JSON outputs. for example even with clear description and demonstrations this is how bedrock response to a query in faithfulness. 

```
 Here are my responses:

statement_1: Criminalizing abortion disproportionately impacts marginalized groups, exacerbating existing inequalities and barriers to healthcare.
Verdict: Yes 
Reason: The context indicates that marginalized groups are disproportionately affected by abortion criminalization, supporting this statement.

statement_2: When abortion is illegal, marginalized individuals may resort to unsafe methods, facing severe health risks and even death.
Verdict: Yes
Reason: The context states that criminalization leads to unsafe abortions and increased maternal mortality, supporting this statement.

statement_3: The economic burden of accessing abortion increases for marginalized individuals when it is criminalized.
Verdict: Null
Reason: The context does not provide information to support or refute this statement.

statement_4: Criminalizing abortion limits reproductive autonomy and choices for marginalized groups. 
Verdict: Yes
Reason: The context indicates criminalization undermines access to services, supporting limited autonomy.

statement_5: It can lead to the criminalization of pregnancy itself, resulting in discrimination and punishment of marginalized women.
Verdict: Null 
Reason: The context does not directly address criminalization of pregnancy.

statement_6
```

Is there a way that you know of to improve this to help produce json outputs from bedrock? 


Hey guys, just made a couple of changes that improved the filled values. 
#588 #581 
feel free to try out by installing from source once merged. 

> feel free to try out by installing from source once merged.

Can we get a (minor?) release for these? I'm also seeing some problems with `faithfulness` and `context_recall` on gpt4 where often only the first item of the list gets parsed.

(This is probably due to the "find outermost json" function, actually a fix for this would be to run "find outermost json" recursively until no json is found -- I'd be happy to write up this feature if there's interest).

Hey @pberger514 we can do a minor release tomorrow. 

> I'd be happy to write up this feature if there's interest

that would be great, please do :) 

What model is being used for Bedrock? Also what prompt?

> Hey guys, just made a couple of changes that improved the filled values. #588 #581 feel free to try out by installing from source once merged.

Thank you ShahuI for looking into this. I see this PR is merged but when I try installing updated RAGAS in Bedrock, it is now failing in importing ragas.langchain which is a wrapper llm required to pass bedrock model to RAGAS evaluate function. Could you please point out what could be the issue here.
![image](https://github.com/explodinggradients/ragas/assets/86981996/29bc11c6-826f-4d71-baad-f925378d78b1)


 

> What model is being used for Bedrock? Also what prompt?

Model I used for bedrock is meta.llama2-70b-chat-v1 and prompt is ChatPromptTemplate.

hey there, I've tried version `v0.1.2` but still getting `nan` issues most of the time with `gpt-3.5-turbo-1106` and those very simple inputs:

```
    dataset = Dataset.from_dict(
        {
            "question": ["What is the capital of France?"],
            "answer": ["The capital of France is Paris."],
            "contexts": [["France is a country in Europe.", "Paris is a city in France."]],
            "ground_truth": [""],
        }
    )
```

if I remove the second context example (`"Paris is a city in France."`), then it works just fine, which is quite odd

for the meantime I'm reverting back to `v0.1.0` where it all works fine

edit: my bad, on v0.1.0 it also happens, maybe less, but I see issues in prod

@rogeriochaves @shahules786 thanks for your work.

When performing ragas test evaluation, using `gpt-4-0125-preview`, and likewise, sometimes I get a nan value for faithfulness metric. Is it a bug?

@shahules786 using Bedrock (more specifically any of the Anthropic Claude models v1-2.1) seems to always result in NaN as Claude is incapable of adhering to a specific output format. May have to get creative with that. Any ideas?

I am having this issue of nan faithfulness score with seemingly random inputs/outputs. I am using text-embedding-3-small. Is there any update on how to handle this?
Version 0.1.4, Python v 3.10.13

Hey @pberger514 can you update to the latest version, it should mostly fix the issue if you're using a good LLM. Feel free to close the issue if it's fixed.

@shabie @rogeriochaves @prufrock349 @jlonge4 @repl-mira-agarwal 

@shahules786 updated to v0.1.7, seems much more stable now, no nans so far, thanks!

I found for more complex local knowledge a lot od NaNs (recent version).  I performed quite extensive updates to the prompts used for synthetic test generation. Here are my draft changes - I'd be glad to discuss them. I managed to reduce NaNs to minimum. 

Still there are situations where for more complex questions and advanced context it's actually not easy to understand whether the verdict should be 1 or -1. For some specific queries/contexts witl already modified prompts I manually passed them to Claude 3 Opus and Llama 3 70B - Opus initially returned 1 with elaborated answer while Llama 3 returned -1 with justified answer (I modified prompt extra for this). Finally I pasted Llama 3 response to Claude Opus which stated that Llama 3 response was more nuanced and its -1 verdict is well justified ;)

https://github.com/explodinggradients/ragas/pull/936

one of the problems I noticed is that sentences saying it can't reply always fail with nan (default gpt-3.5-turbo-16k model), for example:

"I couldn't find any information on completing your account. Can I help you with anything else today?"

I'm pretty sure it's because of this example on the faithfulness prompt:

```
question: "Were Hitler and Benito Mussolini of the same nationality?"
answer: "Sorry, I can't provide answer to that question."
statements: ```[]```
```

I mean I get it, in both cases, either for safety or the model not knowing, what should even be the faithfulness score in this case? Is it 0 because there is nothing to verify against the context? Is it faithful because indeed, the context does not reply the question? But I mean, shouldn't be nan right, that seems just like a bug, or is it expected to be nan in cases it cannot be evaluated?

--- edit ---

actually even though it seems like it works by accident, getting `nan` on this cases is being helpful to us to identify entries where measuring faithfulness indeed doesn't make sense

Can you try this with latest ragas `metrics.collections` ?

[Docs](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/)

It seems the issue was answered, closing this now.