Repository: explodinggradients/ragas
Issue: AttributeError: 'str' object has no attribute 'get' for context_recall 
Root cause: Validation Gap

Hello everyone, I am having this error "AttributeError: 'str' object has no attribute 'get'" while computing context_recall metric, 

Ragas version: 0.0.21
Python version: 3.10.13

**Code to Reproduce**
```
from langchain.embeddings import VertexAIEmbeddings
from langchain.chat_models import ChatVertexAI

from ragas.metrics import AnswerCorrectness, AnswerSimilarity
from ragas.llms import LangchainLLM
import ragas

import google.auth

project_id = ""

config = {
    "project_id": project_id,
}

creds, _ = google.auth.default(quota_project_id=project_id)

chat = ChatVertexAI(credentials=creds)
ragas_vertexai_llm = LangchainLLM(chat)

embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5") 

metrics = [
    ragas.metrics.faithfulness,
    ragas.metrics.context_relevancy,
    ragas.metrics.context_recall
]

for m in metrics:
    m.__setattr__("llm", ragas_vertexai_llm)

    if hasattr(m, "embeddings"):
        m.__setattr__("embeddings", embeddings)
        
        
answer_similarity = AnswerSimilarity(llm=ragas_vertexai_llm, embeddings=embeddings)
answer_correctness = AnswerCorrectness(llm=ragas_vertexai_llm, answer_similarity=answer_similarity)
metrics.insert(0, answer_correctness)

results = ragas.evaluate(
    ds_row, 
    metrics=metrics,
)
```

**Error trace**
```
evaluating with [answer_correctness]
100%|██████████| 1/1 [00:02<00:00,  2.25s/it]
evaluating with [faithfulness]
100%|██████████| 1/1 [00:08<00:00,  8.78s/it]
evaluating with [context_relevancy]
100%|██████████| 1/1 [00:01<00:00,  1.24s/it]
evaluating with [context_recall]
  0%|          | 0/1 [00:03<?, ?it/s]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[135], line 42
     39 answer_correctness = AnswerCorrectness(llm=ragas_vertexai_llm, answer_similarity=answer_similarity)
     40 metrics.insert(0, answer_correctness)
---> 42 results = ragas.evaluate(
     43     ds_row, 
     44     metrics=metrics,
     45 )

File /opt/conda/lib/python3.10/site-packages/ragas/evaluation.py:101, in evaluate(dataset, metrics, column_map)
     99         binary_metrics.append(metric.name)
    100     print(f"evaluating with [{metric.name}]")
--> 101     scores.append(metric.score(dataset).select_columns(metric.name))
    103 # log the evaluation event
    104 metrics_names = [m.name for m in metrics]

File /opt/conda/lib/python3.10/site-packages/ragas/metrics/base.py:76, in Metric.score(self, dataset, callbacks)
     74 with trace_as_chain_group(f"ragas_{self.name}", callback_manager=cm) as group:
     75     for batch in tqdm(self.get_batches(len(dataset))):
---> 76         score = self._score_batch(dataset.select(batch), callbacks=group)
     77         scores.extend(score)
     79 return dataset.add_column(f"{self.name}", scores)

File /opt/conda/lib/python3.10/site-packages/ragas/metrics/_context_recall.py:123, in ContextRecall._score_batch(self, dataset, callbacks, callback_group_name)
    121 response = json_loader.safe_load(response[0], self.llm)
    122 if response:
--> 123     response = [
    124         int(item.get("Attributed", "").lower() == "yes")
    125         if item.get("Attributed")
    126         else np.nan
    127         for item in response
    128     ]
    129     denom = len(response)
    130     numerator = sum(response)

File /opt/conda/lib/python3.10/site-packages/ragas/metrics/_context_recall.py:125, in <listcomp>(.0)
    121 response = json_loader.safe_load(response[0], self.llm)
    122 if response:
    123     response = [
    124         int(item.get("Attributed", "").lower() == "yes")
--> 125         if item.get("Attributed")
    126         else np.nan
    127         for item in response
    128     ]
    129     denom = len(response)
    130     numerator = sum(response)

AttributeError: 'str' object has no attribute 'get'
```

The error only appears for a specific row in my dataset. I tried to compare it with another row for which the process succeed, and there is legit no differences. 

Do you have any suggestions on this ? 

Hi @0ENZO , can you install ragas from source and try again? 

Hi @shahules786, I am working on a jupyter notebook hosted in GCP (Workbench). 
Not sure if it is the right way but I did installed ragas from source by doing :
```
!git clone https://github.com/explodinggradients/ragas
!cd ragas && pip install -e .
```

```
Building wheels for collected packages: ragas
  Building editable for ragas (pyproject.toml) ... done
  Created wheel for ragas: filename=ragas-0.0.23.dev26+gd466070-0.editable-py3-none-any.whl size=7445 sha256=b82b5ae1ad640055fc435ca7fc46cfcb76e4bef229337d0c654b5b5340ea2f7a
  Stored in directory: /var/tmp/pip-ephem-wheel-cache-yaqxkm8f/wheels/bc/ff/e5/310a849ca2fb5f3cfdcbe18dfceca9b75e9ae33433a12e96a2
Successfully built ragas
Installing collected packages: ragas
Successfully installed ragas-0.0.23.dev26+gd466070
```

LangchainLLM seems to be no longer supported :
`ImportError: cannot import name 'LangchainLLM' from 'ragas.llms' (/home/jupyter/ragas/src/ragas/llms/__init__.py)`
But if I use `LangchainLLMWrapper()` instead, I am now having a`openai_api_key` error since LangchainLLMWrapper use ChatOpenAI by default. 

```
def llm_factory(model="gpt-3.5-turbo-16k") -> BaseRagasLLM:
    return LangchainLLMWrapper(ChatOpenAI(model=model))
```

hey @0ENZO appologies for the hard time  - we are in the middle of a v0.1 release hence the breaking changes.
but could you try
```py
chat = ChatVertexAI(credentials=creds)
ragas_vertexai_llm = LangchainLLMWrapper(chat)
```
this will wrap the `ChatVertexAI` model into something ragas can use. Now you can pass it via
```py
results = ragas.evaluate(
    ds_row, 
    metrics=metrics,
    llm=ragas_vertextai_llm
)
```
could you try and let us know if it works - if not we share the error messages and we will get it sorted for you 🙂 

Hi @jjmachan, I am getting the following error, note that this does not occurs for the first row of my dataset and this time for a different row than the previous error. 
I can't explain it. Looking directly at my dataset, there's no causal link. 

Here is the full error : 

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~/ragas/src/ragas/evaluation.py:171, in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, max_workers, raise_exceptions, column_map)
    169         evaluation_rm.on_chain_error(e)
--> 171     raise e
    172 finally:

File ~/ragas/src/ragas/evaluation.py:154, in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, max_workers, raise_exceptions, column_map)
    152 try:
    153     # get the results
--> 154     results = executor.results()
    155     # convert results to dataset_like

File ~/ragas/src/ragas/executor.py:114, in Executor.results(self)
    113     if self.raise_exceptions:
--> 114         raise e
    115 finally:

File ~/ragas/src/ragas/executor.py:110, in Executor.results(self)
    109 try:
--> 110     r = future.result()
    111 except Exception as e:

File /opt/conda/lib/python3.10/concurrent/futures/_base.py:451, in Future.result(self, timeout)
    450 elif self._state == FINISHED:
--> 451     return self.__get_result()
    453 self._condition.wait(timeout)

File /opt/conda/lib/python3.10/concurrent/futures/_base.py:403, in Future.__get_result(self)
    402 try:
--> 403     raise self._exception
    404 finally:
    405     # Break a reference cycle with the exception in self._exception

File /opt/conda/lib/python3.10/concurrent/futures/thread.py:58, in _WorkItem.run(self)
     57 try:
---> 58     result = self.fn(*self.args, **self.kwargs)
     59 except BaseException as exc:

File ~/ragas/src/ragas/executor.py:36, in Executor.wrap_callable_with_index.<locals>.wrapped_callable(*args, **kwargs)
     35 def wrapped_callable(*args, **kwargs):
---> 36     return counter, callable(*args, **kwargs)

File ~/ragas/src/ragas/metrics/base.py:75, in Metric.score(self, row, callbacks)
     74         rm.on_chain_error(e)
---> 75     raise e
     76 else:

File ~/ragas/src/ragas/metrics/base.py:71, in Metric.score(self, row, callbacks)
     70 try:
---> 71     score = self._score(row=row, callbacks=group_cm)
     72 except Exception as e:

File ~/ragas/src/ragas/metrics/_context_recall.py:118, in ContextRecall._score(self, row, callbacks)
    116 response = json_loader.safe_load(result.generations[0][0].text, self.llm)
--> 118 return self._compute_score(response)

File ~/ragas/src/ragas/metrics/_context_recall.py:98, in ContextRecall._compute_score(self, response)
     97 if response:
---> 98     response = [
     99         int(item.get("Attributed", "0").strip() == "1")
    100         if item.get("Attributed")
    101         else np.nan
    102         for item in response
    103     ]
    104     denom = len(response)

File ~/ragas/src/ragas/metrics/_context_recall.py:100, in <listcomp>(.0)
     97 if response:
     98     response = [
     99         int(item.get("Attributed", "0").strip() == "1")
--> 100         if item.get("Attributed")
    101         else np.nan
    102         for item in response
    103     ]
    104     denom = len(response)

AttributeError: 'str' object has no attribute 'get'

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
Cell In[28], line 1
----> 1 context_recall_eval = ragas.evaluate(
      2     eval_ds, 
      3     metrics=[ragas.metrics.context_recall],
      4     llm=ragas_vertexai_llm
      5 )

File ~/ragas/src/ragas/evaluation.py:173, in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, max_workers, raise_exceptions, column_map)
    171     raise e
    172 finally:
--> 173     result = Result(
    174         scores=Dataset.from_list(scores),
    175         dataset=dataset,
    176         binary_columns=binary_metrics,
    177     )
    178     if not evaluation_group_cm.ended:
    179         evaluation_rm.on_chain_end(result)

File <string>:6, in __init__(self, scores, dataset, binary_columns)

File ~/ragas/src/ragas/evaluation.py:202, in Result.__post_init__(self)
    200 def __post_init__(self):
    201     values = []
--> 202     for cn in self.scores[0].keys():
    203         value = np.nanmean(self.scores[cn])
    204         self[cn] = value

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:2800, in Dataset.__getitem__(self, key)
   2798 def __getitem__(self, key):  # noqa: F811
   2799     """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 2800     return self._getitem(key)

File /opt/conda/lib/python3.10/site-packages/datasets/arrow_dataset.py:2784, in Dataset._getitem(self, key, **kwargs)
   2782 format_kwargs = format_kwargs if format_kwargs is not None else {}
   2783 formatter = get_formatter(format_type, features=self._info.features, **format_kwargs)
-> 2784 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
   2785 formatted_output = format_table(
   2786     pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   2787 )
   2788 return formatted_output

File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:583, in query_table(table, key, indices)
    581 else:
    582     size = indices.num_rows if indices is not None else table.num_rows
--> 583     _check_valid_index_key(key, size)
    584 # Query the main table
    585 if indices is None:

File /opt/conda/lib/python3.10/site-packages/datasets/formatting/formatting.py:526, in _check_valid_index_key(key, size)
    524 if isinstance(key, int):
    525     if (key < 0 and key + size < 0) or (key >= size):
--> 526         raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
    527     return
    528 elif isinstance(key, slice):

IndexError: Invalid key: 0 is out of bounds for size 0
```
I managed to check the source code so here are some intuitions  : 

The first error is caused by this list comprehension `response = [int(item.get("Attributed", "0").strip() == "1") if item.get("Attributed") else np.nan for item in response]` in _context_recall.py
The thing is, by doing `for item in response`, as response is a dict, you using .get() on keys which are str. 
It seems that you are originally expecting a list of dict but instead receive a dict. 

Indeed `_score()` returns a list of dict when it succeed and a dict only when the error happens.

In both cases `result` variable have same structure but the changes operate after this statement `response = json_loader.safe_load(result.generations[0][0].text, self.llm)`
Even if they have the same structure, result.generations[0][0].text have 3 statements when the errors happens but only 1 when it succeeds.  

@0ENZO thanks for sharing this issue and your analysis. Do you like to work on fixing the issue? It certainly looks like a significant one to prevent errors. I would add a check to verify the type and do [dict] if it is a dict instead of list.

@shahules786 
Inside the [_score() functions ](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_recall.py#L113) , it seems that the output returned by the LLM is not consistent, i.e when everything works well the LLM is returning such a text :

`[[ChatGeneration(text=' [{"statement_1": "Lorem ipsum.", "reason": "The context clearly states that lorem ipsum.", "Attributed": "1"}]', message=AIMessage(content=' [{"statement_1": "Lorem ipsum.", "reason": "The context clearly states that lorem ipsum.", "Attributed": "1"}]'))]] llm_output={} run=[RunInfo(run_id=UUID('...'))]`


But when the errors occurs, here is the returned result by the LLM call : 

`[[ChatGeneration(text=' [{"statement_1": "Lorem ipsum.", "reason": "Lorem ipsum.", "Attributed": "1"}, {"statement_2": "Lorem ipsum.", "reason": "Lorem ipsum.", "Attributed": "1"}, {"statement_3": "LVMH Staff Sales is a reason code for price override.", "reason": "The context lists Lorem ipsum.",', message=AIMessage(content=' [{"statement_1": "Lorem ipsum.", "reason": "Lorem ipsum.", "Attributed": "1"}, {"statement_2": "Lorem ipsum", "reason": "The context lists Lorem ipsum.", "Attributed": "1"}, {"statement_3": "Lorem ipsum.", "reason": "The context lists Lorem Ipsum.",'))]] llm_output={} run=[RunInfo(run_id=UUID('...'))]`

If you look closely, you will notice that the "text" is not closed properly `{"statement_3": "LVMH Staff Sales is a reason code for price override.", "reason": "The context lists Lorem ipsum.",', message=AIMessage`

This result is later send to `json_loader.safe_load`, which parse the text into a json object by using delimiter `[{` and `}]`, since `}]` is never found inside `text` key, the incomplete json is given though a new [LLM call ](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/json_load.py#L87) to "Rewrite the input into valid json". So the previous incomplete array of json become a simple json object.

One workaround would be, as you suggested, to check the type and go from dict to [dict]. 
I did make a [pull request](https://github.com/explodinggradients/ragas/pull/453) with the updates. 

Also, it might be a good idea to check if others metrics than context_recall relies on the consistency of LLM-evaluation output.