Repository: explodinggradients/ragas
Issue: Metrics outputs may be list of dictionaries instead of the single dictionary that is expected.
Root cause: Validation Gap

**Describe the bug**
`AnswerRelevancy` (and probably other metrics) are not robust to the occasional generation of a list of dictionaries.

When calculating the answer relevancy for a test case, I got the following output from the [LLM evaluator](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_relevance.py#L123):

```
{"question":"What should be the focus of surveillance audits for economic operators to ensure compliance with the ISCC requirements?", "noncommittal":false}{"question":"What should be the focus of surveillance audits for economic operators under the ISCC requirements?", "noncommittal":false}{"question":"What should be the focus of surveillance audits for economic operators in compliance with ISCC requirements?", "noncommittal":false}
```

This was then passed to `json_loader.safe_load` who correctly fixed the syntax to a list of dictionaries. The problem is that (in this case) answer relevancy assumes the output is a valid dictionary, and thus the execution fail when we run [try to get the value for an attribute](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_relevance.py#L134)

Ragas version: 0.0.22
Python version: 3.11.4

**Code to Reproduce**
Even I have some problems reproducing this one. I've hit this issue a couple times before it finally worked. Completely unrelated but for this project especially we are seeing an increase variation in generated answers even when temperature is set to 0.

**Additional context**
I am happy to help implementing the solution to this issue.

I think the nice way to fix this is to add the functionality to `safe_load` to `force_single_dict`, and inside the function we would check if the output is a list or not.

I believe that modifying metrics' prompts trying to enforce a single dictionary output is not an option as the LLM may simply ignore the instruction and the error happen again.

**Expected behavior**
- Metrics should be able to handle all valid json outputs
- Generally, there should be a parameter to indicate whether to stop the evaluation or not in case of errors.

**Error trace**
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb) Cell 25 line 3
      [1](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=0) results_dataset = pa.Table.from_pandas(results)
----> [3](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=2) metrics = evaluate(
      [4](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=3)     Dataset(results_dataset),
      [5](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=4)     metrics=[
      [6](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=5)         context_precision,
      [7](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=6)         answer_relevancy,
      [8](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=7)         context_recall,
      [9](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=8)     ],
     [10](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=9) )
     [12](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=11) # mlflow.log_metric("ragas_score", metrics["ragas_score"])
     [13](vscode-notebook-cell:/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/qa-evaluation-ragas.ipynb#Y104sZmlsZQ%3D%3D?line=12) mlflow.log_metric("context_precision", metrics["context_precision"])

File [~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/evaluation.py:101](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/evaluation.py:101), in evaluate(dataset, metrics, column_map)
     [99](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/evaluation.py:99)         binary_metrics.append(metric.name)
    [100](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/evaluation.py:100)     print(f"evaluating with [{metric.name}]")
--> [101](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/evaluation.py:101)     scores.append(metric.score(dataset).select_columns(metric.name))
    [103](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/evaluation.py:103) # log the evaluation event
    [104](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/evaluation.py:104) metrics_names = [m.name for m in metrics]

File [~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/base.py:76](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/base.py:76), in Metric.score(self, dataset, callbacks)
     [74](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/base.py:74) with trace_as_chain_group(f"ragas_{self.name}", callback_manager=cm) as group:
     [75](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/base.py:75)     for batch in tqdm(self.get_batches(len(dataset))):
---> [76](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/base.py:76)         score = self._score_batch(dataset.select(batch), callbacks=group)
     [77](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/base.py:77)         scores.extend(score)
     [79](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/base.py:79) return dataset.add_column(f"{self.name}", scores)

File [~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:134](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:134), in AnswerRelevancy._score_batch(self, dataset, callbacks, callback_group_name)
    [132](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:132) scores = []
    [133](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:133) for question, result in zip(questions, results):
--> [134](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:134)     gen_questions = [item.get("question", "") for item in result]
    [135](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:135)     committal = np.any([item.get("noncommittal", False) for item in result])
    [136](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:136)     cosine_sim = self.calculate_similarity(question, gen_questions)

File [~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:134](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:134), in <listcomp>(.0)
    [132](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:132) scores = []
    [133](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:133) for question, result in zip(questions, results):
--> [134](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:134)     gen_questions = [item.get("question", "") for item in result]
    [135](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:135)     committal = np.any([item.get("noncommittal", False) for item in result])
    [136](https://file+.vscode-resource.vscode-cdn.net/Users/leonardoschettini/dev/omv/omv-sustainability-backend/evaluate/~/.pyenv/versions/3.11.4/envs/omv-sustainability-backend/lib/python3.11/site-packages/ragas/metrics/_answer_relevance.py:136)     cosine_sim = self.calculate_similarity(question, gen_questions)

AttributeError: 'list' object has no attribute 'get'
```

This problem extends bit further as the model some times outputs a nested json with additional and unexpected fields:

```
{
      "question": "What should be the focus of surveillance audits for economic operators to ensure compliance with the ISCC requirements?",
      "noncommittal": {
          "question": "What should be the focus of surveillance audits for economic operators in compliance with ISCC requirements?",
          "noncommittal": False,
      },
      "additionalQuestion": {
          "question": "What should be the focus of surveillance audits for economic operators under the ISCC requirements?",
          "noncommittal": False,
      },
  }
```

What is really interesting here is that this issue is persistent in this given sample example. Hence I believe it's important to give the option to allow evaluation to run regardless of errors

tagging @shahules786 to get his inputs here