Repository: explodinggradients/ragas
Issue: Ragas scores look inconsistent
Root cause: Validation Gap

**Describe the bug**
We performed ragas evaluation on our dataset and came across the scores looking rather inconsistent, or maybe I'm not understanding the number correctly?

We are using these metrics:
```
from ragas.metrics import (
    context_precision, faithfulness, answer_relevancy, context_relevancy
)
```

A few lines of the result:
![image](https://github.com/explodinggradients/ragas/assets/1017308/bb16648f-2a1c-45dd-a78e-dacf50bd9f32)

As you can see the retrieval scores are super low, especially `context_precision` shows 0 a lot, while `faithfulness` and `answer_relevancy` are still high. So is there something wrong that we're doing incorrectly or this is just expected behavior? We're using GPT 4 for Ragas, which is a different model from what we're using for embedding and completion.

Ragas version: 0.0.22
Python version: 3.11.0


why i am getting {'faithfulness': nan} even when i am following the same steps 
https://docs.ragas.io/en/stable/howtos/customisations/llms.html for open -llm

Hi @nmklong , can you try out latest version by installing ragas from source? We did fix this issue 

@shahules786 thank you for the information, looks like `context_precision` requires ground truths now. So I'm using `context_utilization` instead, the scores look more consistent now. However `faithfulness` is now having the same problem, it's either 1 or 0, or sometimes blank.

![image](https://github.com/explodinggradients/ragas/assets/1017308/1cd0ff93-1eed-4669-817d-9abbdf3da616)

I took a look at the blank ones and they're because OpenAI sometimes don't give the response with the right format (having `verdict` for example), I guess we can try to fix by reinforcing in the prompt will help, something like "make sure your response has the format in  the example", Claude has this same problem and I adjusted the prompt so it can give a more correct response format.

But what do you suggest for `faithfulness` having only either 1 or 0?



Hi @nmklong , thank you. Interestingly, your faithfulness is either zero or one. Are your answers short, ie only containing one statement? If you like me to check it out, please share some data points. 

We already do json correction, you can see the code for it [here](https://github.com/explodinggradients/ragas/blob/da334769b85225953cc2a502c58b846b42401ab5/src/ragas/llms/json_load.py#L75). Please suggest any improvements if necessary. 


+ if you don't have ground truth data, feel free to try out our synthetic test generation feature :) @nmklong 

> why i am getting {'faithfulness': nan} even when i am following the same steps https://docs.ragas.io/en/stable/howtos/customisations/llms.html for open -llm

Just change max_token to 1000. I also encountered this problem and solved it like this.

> * if you don't have ground truth data, feel free to try out our synthetic test generation feature :) @nmklong

I would love to know more about this, do you have the link to it?

I'm closing the issue now as the problem seems to get fixed in the latest codes. Thank you for the help so far.

@riyaj8888 Did you resolve the issue, I am also getting {'faithfulness': nan} 
but I am using llama2 as llm and ollama embeddings

I have the same issue with ragas version `0.1.3` (Python version `3.10.13`).

Using metrics "faithfulness", "answer_relevancy", "context_relevancy", "context_recall", "context_precision",  and "answer_correctness" and OpenAI with the default model.
I have 69 questions, which results in 414 requests for this combination with metrics.

All metric results are floats except in "faithfulness", some are nan. 

My workaround currently is the following:

```
df_scores = results.scores.to_pandas()

df_scores['faithfulness'] = df_scores['faithfulness'].fillna(0)

results.scores = Dataset.from_pandas(df_scores)
```

I need to do this because I use validate.tonic.ai to visualize results over time and its `upload_results` method can't handle non-float values.

> @riyaj8888 Did you resolve the issue, I am also getting {'faithfulness': nan} but I am using llama2 as llm and ollama embeddings

Yes, Try to augment the `max_token` of your LLM, because when it pass the max_token it return `nan`