Repository: explodinggradients/ragas
Issue: Additional required fields for metrics
Root cause: Validation Gap

[ yes] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I keep getting missing required fields when trying to use `faithfulness`, `answer_relevancy` and `answer_correctness` metrics. From the documentation, there is no response field specified but I am getting an error that it is needed

Ragas version:
Python version:3.10

**Code to Reproduce**
```
def query_engine(query):
    query_engine_result= vector_db.max_marginal_relevance_search(query, k=1)
    model_response=final_chain.run(context=query_engine_result, question=query)
    return model_response,query_engine_result

def generate_response(query_engine,test_questions,test_answers):
    responses=[query_engine(question) for question in test_questions]
    answers=[]
    contexts=[]
    for index,r in enumerate(responses):

        answers.append(r[0])
        contexts.append([r[1][0].page_content])
        dataset_dict={
            "user_input":test_questions,
            "answer":answers,
            "retrieved_contexts":contexts,
        }
    if test_answers is not None:
        dataset_dict["reference"]=[test_answer[0] if isinstance(test_answer, list) else test_answer for test_answer in test_answers]
    ds=Dataset.from_dict(dataset_dict)
    return ds

results_ds=generate_response(query_engine,test_questions,test_answers)

from ragas import evaluate
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
    answer_correctness
)
metrics=[
    context_precision,
    faithfulness,
    answer_relevancy,
    answer_correctness,
    context_recall,
]
result = evaluate(
    results_ds,
    metrics=metrics
)
```

**Error trace**
```
{
	"name": "ValueError",
	"message": "The metric [faithfulness] that that is used requires the following additional columns ['response'] to be present in the dataset.",
	"stack": "---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[101], line 16
      2 from ragas.metrics import (
      3     answer_relevancy,
      4     faithfulness,
   (...)
      7     answer_correctness
      8 )
      9 metrics=[
     10     context_precision,
     11     faithfulness,
   (...)
     14     context_recall,
     15 ]
---> 16 result = evaluate(
     17     results_ds,
     18     metrics=metrics
     19 )
     21 result

File ~/G_RAG/ragas/src/ragas/_analytics.py:129, in track_was_completed.<locals>.wrapper(*args, **kwargs)
    126 @wraps(func)
    127 def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
    128     track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 129     result = func(*args, **kwargs)
    130     track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
    132     return result

File ~/G_RAG/ragas/src/ragas/evaluation.py:162, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map)
    159     dataset = EvaluationDataset.from_list(dataset.to_list())
    161 if isinstance(dataset, EvaluationDataset):
--> 162     validate_required_columns(dataset, metrics)
    164 # set the llm and embeddings
    165 if isinstance(llm, LangchainLLM):

File ~/G_RAG/ragas/src/ragas/validation.py:47, in validate_required_columns(ds, metrics)
     45 available_columns = ds.features()
     46 if not required_columns.issubset(available_columns):
---> 47     raise ValueError(
     48         f\"The metric [{m.name}] that that is used requires the following \"
     49         f\"additional columns {list(required_columns - available_columns)} \"
     50         f\"to be present in the dataset.\"
     51     )

ValueError: The metric [faithfulness] that that is used requires the following additional columns ['response'] to be present in the dataset."
}
```

**Expected behavior**
Expecting to see the results of the metrics defined


**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hey there, @Nanayeb34! I'm here to help you out with any bugs, questions, or contributions you have. Let's squash those bugs together! 🤖

<!-- Answer -->
To resolve the "missing required fields" error for the `faithfulness`, `answer_relevancy`, and `answer_correctness` metrics, you need to ensure that your dataset includes the required fields: `user_input`, `response`, and `retrieved_contexts`. In your current code, you are using `answer` instead of `response`, which is causing the error.

Here's how you can modify your `generate_response` function to include the correct fields:

```python
def query_engine(query):
    query_engine_result = vector_db.max_marginal_relevance_search(query, k=1)
    model_response = final_chain.run(context=query_engine_result, question=query)
    return model_response, query_engine_result

def generate_response(query_engine, test_questions, test_answers):
    responses = [query_engine(question) for question in test_questions]
    answers = []
    contexts = []
    for index, r in enumerate(responses):
        answers.append(r[0])
        contexts.append([r[1][0].page_content])
    
    dataset_dict = {
        "user_input": test_questions,
        "response": answers,  # Changed from "answer" to "response"
        "retrieved_contexts": contexts,
    }
    
    if test_answers is not None:
        dataset_dict["reference"] = [test_answer[0] if isinstance(test_answer, list) else test_answer for test_answer in test_answers]
    
    ds = Dataset.from_dict(dataset_dict)
    return ds

results_ds = generate_response(query_engine, test_questions, test_answers)

from ragas import evaluate
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
    answer_correctness
)

metrics = [
    context_precision,
    faithfulness,
    answer_relevancy,
    answer_correctness,
    context_recall,
]

result = evaluate(
    results_ds,
    metrics=metrics
)
```

Ensure that each entry in your dataset includes the `user_input`, `response`, and `retrieved_contexts` fields. This should resolve the "missing required fields" error <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_faithfulness.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


if i change to `response` and run it, i get an error that `faithfulness` requires `answer`.

currently, i am getting  the following error `ValueError: The metric [answer_relevancy] that that is used requires the following additional columns ['retrived_contexts'] to be present in the dataset.` which is quite strange. the spelling of `retrived_contexts` is even wrong. what is happening here? The docs don't even indicate `user_input` and `retrieved_contexts` are the keywords required.

<!-- Answer -->
The errors you're encountering are due to discrepancies between the code and the documentation, as well as a known typo in the codebase.

1. **Required Columns for Metrics**:
   - **Faithfulness**: Requires `user_input`, `response`, and `retrieved_contexts` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/experimental/metrics/_faithfulness.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_faithfulness.py)</sup>.
   - **Answer Relevancy**: Requires `user_input`, `response`, and `retrived_contexts` (note the typo) <sup>[[3]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_relevance.py)</sup>.
   - **Answer Correctness**: Requires `user_input`, `response`, and `reference` <sup>[[4]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_answer_correctness.py)</sup>.

2. **Typo in `retrived_contexts`**:
   - There is a known typo in the codebase where `retrieved_contexts` is misspelled as `retrived_contexts` <sup>[[5]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_entities_recall.py)</sup>.

To resolve these issues, you should:
- Ensure your dataset includes the correct columns: `user_input`, `response`, `retrieved_contexts` (or `retrived_contexts` due to the typo), and `reference` if applicable.
- Update your `generate_response` function to match these requirements.

Here's an updated version of your `generate_response` function:

```python
def query_engine(query):
    query_engine_result = vector_db.max_marginal_relevance_search(query, k=1)
    model_response = final_chain.run(context=query_engine_result, question=query)
    return model_response, query_engine_result

def generate_response(query_engine, test_questions, test_answers):
    responses = [query_engine(question) for question in test_questions]
    answers = []
    contexts = []
    for index, r in enumerate(responses):
        answers.append(r[0])
        contexts.append([r[1][0].page_content])
    
    dataset_dict = {
        "user_input": test_questions,
        "response": answers,  # Ensure this is "response"
        "retrieved_contexts": contexts,  # Ensure this is "retrieved_contexts"
    }
    
    if test_answers is not None:
        dataset_dict["reference"] = [test_answer[0] if isinstance(test_answer, list) else test_answer for test_answer in test_answers]
    
    ds = Dataset.from_dict(dataset_dict)
    return ds

results_ds = generate_response(query_engine, test_questions, test_answers)

from ragas import evaluate
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
    answer_correctness
)

metrics = [
    context_precision,
    faithfulness,
    answer_relevancy,
    answer_correctness,
    context_recall,
]

result = evaluate(
    results_ds,
    metrics=metrics
)
```

Make sure your dataset includes the columns `user_input`, `response`, `retrieved_contexts` (or `retrived_contexts`), and `reference` if needed. This should resolve the errors you're encountering.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Hey @Nanayeb34 I see that you're following column names as per new data representation #1222. We haven't added the documentation for it yet because we are revaming docs for #1009 release

To solve your issue please use the column names as per https://docs.ragas.io/en/stable/ 
```
 dataset_dict={
            "question":test_questions,
            "answer":answers,
            "contexts":contexts,
        }
    if test_answers is not None:
        dataset_dict["ground_truth"]
```

@shahules786 i have changed the generate_response function to mirror the documentation. 
```
def generate_response(query_engine,test_questions,test_answers):
    responses=[query_engine(question) for question in test_questions]
    answers=[]
    contexts=[]
    for index,r in enumerate(responses):

        answers.append(r[0])
        contexts.append([r[1][0].page_content])
        dataset_dict={
            # "user_input":test_questions,
            # "answer":answers,
            # "retrieved_contexts":contexts,
            "question":test_questions,
            "answer":answers,
            "context":contexts  
        }
    if test_answers is not None:
        dataset_dict["ground_truth"]=test_answers
        # dataset_dict["reference"]=[test_answer[0] if isinstance(test_answer, list) else test_answer for test_answer in test_answers]
    ds=Dataset.from_dict(dataset_dict)
    return ds
```

However, i still get an error with `context_precision` missing some required args. I realised #1222 was merged 4 days ago. I installed ragas 4 days ago as well. Might the issue have to do with the version of ragas i am using?
```
{
	"name": "ValueError",
	"message": "The metric [context_precision] that that is used requires the following additional columns ['user_input', 'retrieved_contexts', 'reference'] to be present in the dataset.",
	"stack": "---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[112], line 16
      2 from ragas.metrics import (
      3     answer_relevancy,
      4     faithfulness,
   (...)
      7     answer_correctness
      8 )
      9 metrics=[
     10     context_precision,
     11     faithfulness,
   (...)
     14     context_recall,
     15 ]
---> 16 result = evaluate(
     17     results_ds,
     18     metrics=metrics
     19 )
     21 result

File ~/minohealth/G_RAG/ragas/src/ragas/_analytics.py:129, in track_was_completed.<locals>.wrapper(*args, **kwargs)
    126 @wraps(func)
    127 def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
    128     track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 129     result = func(*args, **kwargs)
    130     track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
    132     return result

File ~/minohealth/G_RAG/ragas/src/ragas/evaluation.py:162, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map)
    159     dataset = EvaluationDataset.from_list(dataset.to_list())
    161 if isinstance(dataset, EvaluationDataset):
--> 162     validate_required_columns(dataset, metrics)
    164 # set the llm and embeddings
    165 if isinstance(llm, LangchainLLM):

File ~/minohealth/G_RAG/ragas/src/ragas/validation.py:47, in validate_required_columns(ds, metrics)
     45 available_columns = ds.features()
     46 if not required_columns.issubset(available_columns):
---> 47     raise ValueError(
     48         f\"The metric [{m.name}] that that is used requires the following \"
     49         f\"additional columns {list(required_columns - available_columns)} \"
     50         f\"to be present in the dataset.\"
     51     )

ValueError: The metric [context_precision] that that is used requires the following additional columns ['user_input', 'retrieved_contexts', 'reference'] to be present in the dataset."
}
```

@shahules786  this is a typo?! retrived_contexts

{
	"name": "ValueError",
	"message": "The metric [answer_relevancy] that that is used requires the following additional columns ['retrived_contexts'] to be present in the dataset.",
	"stack": "---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[155], line 9
      6 for column in ['retrived_contexts', 'retrieved_contexts', 'user_input', 'response']:
      7     print(column, len(prepared_dataset[column]))
----> 9 result = evaluate(
     10     dataset = prepared_dataset,
     11     metrics=[
     12         answer_relevancy,
     13         context_precision,
     14         context_recall,
     15         faithfulness,
     16     ],
     17 )

File ~/anaconda3/envs/ragas_api/lib/python3.10/site-packages/ragas/_analytics.py:129, in track_was_completed.<locals>.wrapper(*args, **kwargs)
    126 @wraps(func)
    127 def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
    128     track(IsCompleteEvent(event_type=func.__name__, is_completed=False))
--> 129     result = func(*args, **kwargs)
    130     track(IsCompleteEvent(event_type=func.__name__, is_completed=True))
    132     return result

File ~/anaconda3/envs/ragas_api/lib/python3.10/site-packages/ragas/evaluation.py:164, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map)
    161     dataset = EvaluationDataset.from_list(dataset.to_list())
    163 if isinstance(dataset, EvaluationDataset):
--> 164     validate_required_columns(dataset, metrics)
    166 # set the llm and embeddings
    167 if isinstance(llm, LangchainLLM):

File ~/anaconda3/envs/ragas_api/lib/python3.10/site-packages/ragas/validation.py:47, in validate_required_columns(ds, metrics)
     45 available_columns = ds.features()
     46 if not required_columns.issubset(available_columns):
---> 47     raise ValueError(
     48         f\"The metric [{m.name}] that that is used requires the following \"
     49         f\"additional columns {list(required_columns - available_columns)} \"
     50         f\"to be present in the dataset.\"
     51     )

ValueError: The metric [answer_relevancy] that that is used requires the following additional columns ['retrived_contexts'] to be present in the dataset."
}

https://github.com/explodinggradients/ragas/commit/2e6ac350c14e4886e121b9136789e4c8776d7742

Hey @rschumann @Nanayeb34 thanks for pointing this out. I thought I handled the v1 dataset case but somehow it slipped through. I have raised the above PR to fix it - you can either install ragas from source to use it once merged or wait for release that should happen in few hours.