Repository: explodinggradients/ragas
Issue: Ragas metrics result output sometimes show "null", "nan" when using my own bedrock llm in evaluation
Root cause: Validation Gap

When i try the same evaluation pipeline with same questions but switching different llm which is azureopenAI and bedrock. AzureOpenAI gives me consistent metrics score for faithfulness, context_precision, answer_relevancy but when i switch to bedrock. Often I saw the output of those metrics turn to be "null", "nan".. 
Observed the below warning not sure is it related to warning below:
ragas    | evaluating with [context_precision]
100%|██████████| 1/1 [01:20<00:00, 80.19s/it]
  0%|          | 0/1 [00:00<?, ?it/s]us-west-2
ragas    | evaluating with [faithfulness]
100%|██████████| 1/1 [00:35<00:00, 35.79s/it]
  0%|          | 0/1 [00:00<?, ?it/s]us-west-2
ragas    | /usr/local/lib/python3.10/site-packages/ragas/evaluation.py:130: RuntimeWarning: Mean of empty slice
ragas    |   value = np.nanmean(self.scores[cn])

Is there any difference when setting up bedrock compared to azureopenai? cause azureopenai doesn't behave like this cause using the same pipeline and retriever and etc..

ragas version used: 0.0.22 which installed from pip

I get the same issue with another open-source LLM with the same ragas version. So I check the code of [context_precision].
```
 for response in grouped_responses:
                response = [
                    json_loader.safe_load(item, self.llm) for item in sum(response, [])
                ]
                response = [
                    int("yes" in resp.get("verdict", " ").lower())
                    if resp.get("verdict")
                    else np.nan
                    for resp in response
                ]
                denominator = sum(response) + 1e-10
                numerator = sum(
                    [
                        (sum(response[: i + 1]) / (i + 1)) * response[i]
                        for i in range(len(response))
                    ]
                )
                scores.append(numerator / denominator)
```
It seems that whereas there is no 'verdict' keyword in LLM's output, it will return NaN and the metric can't be calculated. And ''verdict' can be seen in template, which indicates the context's quality. Eg:
```
verification:
{{"reason":"This context is extremely relevant and useful for answering the question. It directly addresses the mechanisms of photosynthesis, explaining the key components and processes involved.", "verdict":"Yes"}}
```
So I guess it's related to LLM's performance. If the LLM I choose can not return the content which is formatted with 'verdict', it can't be counted.

I see. But it is inconsistent. With the same questions and same code, sometimes get values sometimes not..Even sometimes get "null" for it.

Have you ever checked the outputs of your LLM every time? Is there any difference? Maybe it's related to the parameter 'temperature'. I'm not sure...still trying... :sob:

Ya.. answer and context retrieved almost same.. Just that bedrock always give me null and nan while azureopenai give me better consistant output value

Hey @hteeyeoh @RuiNov1st , I'll checkout this issue. It is most probably the LLMs incompatibility to generate JSON output even with a retry mechanism in place. 

@shahules786 Sure. Thank you. It would be helpful. Even i sometimes noticed with AzureGPT4 model sometimes also will observed value with "null"

We will try to improve this but there is also another reason for the scores to be null, the API throwing something like timeout, token limit even after multiple retries. @hteeyeoh 

I see. But it would be great if can throw better error handling instead of just "null" so that user know what's causing the "null" to be appear in the metrics

Yes, I would love to add that. If you like to help us out, please feel free to raise a PR:)

> We will try to improve this but there is also another reason for the scores to be null, the API throwing something like timeout, token limit even after multiple retries. @hteeyeoh

hi @shahules786 , just to check how the token limit define? i think the scores turns out null easily. As i passed in my top_k=3 context as my contexts into RAGAS to evaluate.. And faithfulness often give me "null value.