Repository: Giskard-AI/giskard-oss
Issue: KeyError: 'user_input' when calculating RAGAS metric
Root cause: External Dependency Breakage

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.15.3

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

```shell
ragas==0.2.2
```


### Current Behaviour?

When trying to evaluate a RAG assistant with some RAGAS metrics (context recall), the evaluation fails. See stacktrace below.
This happens when trying to provide the answer as `AgentAnswer`. We're not super clear about what should be in the `documents` parameter, the documentation doesn't give any clear example. We're using LlamaIndex so `agent_output.source_nodes` doesn't return a list of strings. Here's what we've tried:

### Standalone code OR list down the steps to reproduce the issue

```python
def answer_fn(question: str, history: List[dict] = []) -> AgentAnswer:
    
    chat_history = [ChatMessage(role=MessageRole.USER, content=msg["content"]) for msg in history]
    agent_output = chat_engine.chat(question, chat_history=chat_history)

    answer = agent_output.response
    # documents = agent_output.source_nodes
    documents = [node.text for node in agent_output.source_nodes if hasattr(node.node, 'text')]

    return AgentAnswer(message=answer,documents=documents)
```

```python
evaluate(
            answer_fn,
            testset=testset,
            knowledge_base=knowledge_base,
            metrics=[ragas_context_recall, ragas_context_precision, ragas_faithfulness, ragas_answer_relevancy]
        )
```
### Relevant log output

```log
  File "/evaluation_manager.py", line 310, in _run_giskard_evaluation_and_return_generated_report
    return evaluate(
           ^^^^^^^^^
  File "/giskard/rag/evaluate.py", line 105, in evaluate
    metrics_results[sample["id"]].update(metric(sample, answer))
                                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/giskard/rag/metrics/ragas_metrics.py", line 119, in __call__
    return {self.name: self.metric.score(ragas_sample)}
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ragas/utils.py", line 159, in emit_warning
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/ragas/metrics/base.py", line 121, in score
    raise e
  File "/ragas/metrics/base.py", line 117, in score
    score = loop.run_until_complete(self._ascore(row=row, callbacks=group_cm))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
           ^^^^^^^^^^
  File "/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/asyncio/tasks.py", line 314, in __step_run_and_handle_result
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/ragas/metrics/_context_recall.py", line 191, in _ascore
    return await super()._ascore(row, callbacks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ragas/metrics/_context_recall.py", line 156, in _ascore
    question=row["user_input"],
             ~~~^^^^^^^^^^^^^^
KeyError: 'user_input'
```

I'm experiencing the same issue, and I believe this is a quite critical bug. I couldn't run evaluations with additional metrics. Any update on this?

Hey @PierreMesure and @Snow31ind - This should be solved by https://github.com/Giskard-AI/giskard/pull/2052

Can you try again with Giskard latest release?

@alexcombessie Thank for replying to us. Let me input more context on this issue. The version of giskard and ragas in my `requirements.txt` file is:
```
giskard==2.15.3
ragas==0.2.2
```

I believe `2.15.3` is the latest release, and I still see the same error thrown as above.

After inspecting the stack traceback, I'm wondering if the [ragas sample](https://github.com/Giskard-AI/giskard/blob/9fe01f7f5deaa8271c4e4a30546bd29bae0bee86/giskard/rag/metrics/ragas_metrics.py#L113-L118) in the giskard ragas metric wrapper matches the required interface in the base ragas metric score [method](https://github.com/explodinggradients/ragas/blob/8929a46442bdc21a9d6c20a7593109c5abadfb42/src/ragas/metrics/_context_recall.py#L155-L159), as the sample doesn't contain the `user_input` key. That's why I strongly believe it's the root cause.

Could you help double check on that? And is there any tests being run to make sure there's no data interface mismatch?

I just reverted ragas to 0.1.21 and it works. 😊

@alexcombessie, I reported another problem fixed by #2052, I don't think this PR will fix it.
I think the problem stemmed from a change in the name of the parameters by RAGAS. In [this commit](https://github.com/explodinggradients/ragas/commit/495bbd1acf7fbc5a978edf22576f467cf2d5bc96#diff-00dfa8fa0e6faa859dca79a8508a9c0fe26f24ba7bfdc7d4bae6fa3947f380d3L3), you can see the change in the documentation. I think the change in variable names comes from [this PR](https://github.com/explodinggradients/ragas/pull/1222)

@PierreMesure Awesome! You make my day. Anyway, this issue is worth having a fixed soon. Thanks team!

Thanks! @henchaves could you have a look when you have time next week 🙏 ?

Hello @PierreMesure and @Snow31ind, thanks a lot for reporting this bug.
Indeed RAGAS v0.2 has changed the name of parameters, which are breaking the `RagasMetric` call.
I opened a PR to make it compatible with the old version (v0.1) and also the last one (v0.2): https://github.com/Giskard-AI/giskard/pull/2073

It should be reviewed and merged soon! 
Thanks again and sorry for the delay!